Recognizing Vector Graphics without Rasterization

1 Introduction . Raster graphics have been commonly used for image recognition due to its easy accessibility from cameras . Most existing benchmark datasets are built upon raster graphics , from ImageNet [ 1 ] for classification to COCO [ 2 ] for object detection . However , due to its pixel-based fix-sized format , raster graphics may lead to aliasing when scaling up or down by interpolation . Fields like engineering design or graphic design require a more precise way to describe visual content without aliasing when scaling ( e.g. , graphic designs , mechanical drafts , floorplans , diagrams , etc ) , so another important image format emerges , namely vector graphics . Vector graphics achieve this powerful feature by recording how the graphics are constructed or drawn , instead of the color bitmaps represented by pixel arrays defined in the raster graphics ( first row in Figure 1 ) . Specifically , vector graphics contain a set of primitives like lines , curves and circles , which is defined with parametric equations in analytic geometry and some extra attributes . As shown in Figure 1 , such vector graphic is usually a document where every primitive is defined precisely and written in a line of textual command . Due to the analytic representation , with few parameters , vector graphics can represent an object at any scale or even in infinite resolution , making it potentially a lot more precise and compact image format than raster graphics . Also , instead of independent pixels , ∗This work was done when the authors were interns at MSRA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . vector graphics give higher level structural information on how low-level elements like points or curves group together to form high level shapes or structures . However , this powerful and widely used data format has been rarely investigated in previous computer vision literature . To explore this data format , this paper focuses on the fundamental recognition tasks : object localization and classification , with wide applications in vector graphics related field like automatic design audit , AI aided design , design graphics retrieval , etc . Existing raster graphics based methods [ 3 , 4 , 5 , 6 , 7 ] takes pixel arrays as input and can not be directly applied on vector graphics . There have been attempt [ 8 ] dealing with this format by rendering vector graphics into raster graphics first . However , rendering the vector graphics into raster graphics could result in a pixel array with super resolutions ( e.g. , thousands by thousands ) , which brings extremely large memory cost , and would be inefficient or even intractable for the traditional models to process . On the other hand , rendering a lower resolution image causes substantial information loss , and the object bounding boxes obtained from a low resolution image could be imprecise when scaled back to the original resolution . Furthermore , the rendering process results in a set of independent pixels and discards the high-level structural information within the primitives . Some of this information could be critical for recognition , such as corners in a shape or contours , etc . To address these issues , we resolve the tasks on vector graphics by introducing a model that does not need rasterization and takes the textual documents of vector graphics as input , called YOLaT ( You Only Look at the Text ) . Instead of rendering the vector graphics into raster graphics , we propose an efficient end-to-end pipeline which predicts objects from the raw textual definitions of primitives . YOLaT first transforms different types of primitives into a unified format . Then it constructs an undirected multi-graph to model the structural and spatial information from the unified primitives . Compared to rendering to raster graphics , this transformation is able to preserve more complete information . YOLaT generates object proposals directly from the vector graphics , which produces precise object bounding boxes . Finally , a dual-stream graph neural network ( GNN ) specifically designed for vector graphics is proposed to classify the graph contained in each proposal , with no extra regression needed for bounding box refinement . To evaluate our pipeline over vector graphics , we use two datasets . i.e. , floorplans and diagrams and show the advantages of our method over the raster graphics based object detection baselines . Without pre-training , our method consistently outperforms raster graphics based object detection baselines , with significantly higher efficiency in terms of the number of parameters and FLOPs . Even compared with the powerful ImageNet pretrained two-stage model , YOLaT achieves comparable performance with 25 times fewer parameters and 100 times fewer FLOPs . We also show visualizations to better demonstrate why looking at the text can capture more delicate details and predicts more accurate bounding boxes . 2 Related Work . Object Detection on Raster Graphics . Currently deep learning based object detection methods dominate the research field with the superior performance . Two-stage object detection methods first generate region proposals and classify and regress the proposals to give object predictions with a deep convolutional networks . R-CNN [ 9 ] and Fast-RCNN [ 3 ] use selective search for proposal generations . Faster-RCNN [ 10 ] speeds up the proposal generation by introducing a region proposal network . He et al . [ 11 ] proposed Mask-RCNN , adding a segmentation branch to the detection model for instance segmentation . To train a more translation-variant backbone , Dai et al . [ 12 ] proposed F-RCN – a new prediction head with position-sensitive score maps . Most two-stage object detection methods have large computation overhead of the proposal generation process , and require running a classification and regression sub-network on all the region proposals . One-stage object detection methods tackle this challenge by removing the proposal generation process and directly predict the object bounding boxes in an end-to-end fashion . Anchor-based methods like SSD [ 4 ] , YOLO series [ 5 , 13 , 14 , 15 ] , RetinaNet [ 6 ] densely tile anchor boxes over the image and conduct classification and bounding box coordinate refinement on each anchor box . Recently , anchor-free methods like CornerNet [ 16 ] , CenterNet [ 7 ] , FCOS [ 17 ] propose to directly find object without presets anchors . Graph Neural Networks . GNN has become a powerful tool for machine learning on graphs . It computes a state for each node in a graph , and iteratively updates the node states according to its neighbors . Spectral approaches [ 18 , 19 ] define a convolution operation in the Fourier domain . Spatial approaches [ 20 , 19 , 21 ] define convolutions directly on the graph . EdgeConv [ 22 ] applies GNN model for classification on 3D Cloud by taking the state difference between neighboring nodes as the input of the aggregation function . [ 23 ] further applies EdgeConv to the object detection task on 3D cloud data by integrating the GNN backbone into an anchor-free detection framework . The closest GNN model to our YOLaT is EdgeConv but YOLaT has extra upgrade specifically designed for vector graphics , including edge attributes , faster inference on densely connected edges , and dual-stream structure for multi-graph . Online Sketch and Handwriting Recognition . Online handwriting and drawing recognition [ 24 , 25 ] handles a data form that very similar to vector graphics , which contains a sequence of discrete points . Most of these methods use sequential models to handle this problem . For example , [ 26 ] proposes to convert the point sequences to a sequence of Bézier Curves and use a LSTM for sequential modeling . Compare to online handwriting , vector graphics contain more types of unordered shapes with more attributes and properties other than polylines , and hence need more general and non-sequential method . Vector Graphics Related Application . One of the most common application for vector graphics is design , such as architecture , graphic design , etc . Several methods in architecture drawing recognition propose to represent symbols in a floor-plan as graphs , and use rule-based graph matching method to classify and localize symbols , such as visibility graph [ 27 ] and attributed relational graph [ 28 , 29 ] . In this paper , we propose a novel scheme that directly construct graph from vector graphics based on Bézier Curve , and the object detection is conducted based on the prediction of GNN . Recent years , a few works develop deep learning based methods to automatically generate vector graphics for computer aided design or converting raster graphics to vector graphcis ( i.e . vectorization ) [ 30 , 31 , 32 , 33 , 34 , 35 ] , while to the best of knowledge , our paper is the first to focus on recognition task on vector graphics . Koch et al . [ 36 ] proposes a large 3D model dataset containing analytic representations , but it lacks semantic labeling to train recognition model . 3 Detection Model . In this paper , we study the problem of object detection leveraging the definitions of the vector graphics without rendering them . Here , we define the task as object localization and object classification . Specifically , the model needs to predict a set of bounding box coordinates as well as the category of the object within the bounding boxes . In this section , we describe our proposed YOLaT , which is an end-to-end efficient pipeline taking the raw definitions of the vector graphics as the input without further rendering the graphics . Figure 2 shows the overall pipeline of YOLaT . We convert the primitives like lines and curves as a universal format of Bézier curves . Based on the Bézier curves , un-directed multi-graphs are constructed to model both spatial and structural relationships among the key-points within a primitive and among different primitives . More details on how we build the graphs can be found in Section 3.1 . To fully explore the vector graphics based on the multi-graph , we propose a dual-stream GNN for graph feature extraction and classification . Section 3.2 shows the detailed design of the proposed dual-stream GNN . Compared to the complex prediction head commonly used in the object detection for raster graphics , YOLaT generates precise proposal bounding boxes directly from high resolution vector graphics . Hence each sub-graph in the proposals are fed into the dual-stream GNN classifier without further correction of the box coordinates . We show how to get the potential bounding boxes and predict their objectiveness and category in Section 3.3 . 3.1 Graph Construction . Universal formats of curves . Compared to the raster graphics represented by pixel arrays , vector graphics have more precise representations and no loss of quality and aliasing when resizing . The vector graphics consists of primitives defined by textual commands described in parametric equations , such as lines , curves , polygons and other shapes.Different primitives are described with different parametric equations . Here , like pixel in raster graphics , we want to find a unified way to describe all types of primitives . We chose Bézier Curve due to its generality and capability of modeling different shapes and curves . Bézier Curve is defined by a set of control points , { p0 , ... , pn } , where n is the order of the curve and pi∈ [ n+1 ] is a 2-d vector for the coordinates of point i . The first point and the last point are the end points of a curve while the rest of the control points usually do not sit on the curve and provide side information instead , such as directional information and curvature statistics of the curve from p0 to pn . We chose cubic Bézier Curve where n = 3 for the balance between modeling capability and computational complexity . Formally , the cubic Bézier curve B is defined as : B ( t ) = ( 1− t ) 3p0 + 3 ( 1− t ) 2tp1 + 3 ( 1− t ) t2p2 + t3p3 , 0 ≤ t ≤ 1 ( 1 ) where B ( t ) defines the position of a specific point at the scale rate of t on the curve from p0 to p3 . Next , we introduce our graph construction based on a set of Bézier Curves . Nodes . To improve efficiency , we only include the points from the set of start points and end points , denoted by P , into the collections of nodes on graphs . The rest of the control points will serve as the edge attributes as defined in the following paragraph . For a point p , the attributes x of the corresponding node include the coordinates of the point , the RGB color value c and stroke width w : x = concat ( px , py , c , w ) , p ∈ P ( 2 ) where px and py denote the coordinate value of the point p along the x axis and y axis respectively . These information is defined in the vector graphic documentation . Edges . We design the graph as a multi-graph containing two sets of edges , namely the stroke-wise edges and the position-wise edges . These two types of edges capture the node relationships from different perspectives . Stroke-wise edges capture the connections defined by the stroke in the vector graphics , which refers to the actual stroke drawn between the start and end point of each Bézier curve . This type of connections represents the structures and layouts of the objects in the vector graphics . Thus , an edge is built in-between two nodes if there is a Bézier Curve linking them : Es = { ( vi , vj ) : ( vi , vj ) ∈ S } ( 3 ) where S denotes the set of tuples containing the start point vi and end point vj of a Bézier Curve . Other than the connections between start and end points , other attributes of a cubic Bézier curve like curvature or other appearance are described by the off-curve control points . We use the coordinates of these off-curve control points as the attributes of the stroke-wise edges : xe = concat ( px , py ) , p /∈ P ( 4 ) The stroke-wise edges only model the long-term structural connection between the vertices based on strokes , which is irrelevant to the spatial vicinity . To further capture the spatial relationship between nodes , we generate another set of edges , called position-wise edges . Specifically , the position-wise edges are defined as the dense connections among nodes within a regional cluster Ck : Ep = { ( vi , vj ) : vi , vj ∈ Ck } , k ∈ { 1 , 2 , ... , m } , ( 5 ) A regional cluster is a set of nodes close to each other spatially , which can by obtained in different ways . In our implementation , we obtain regional cluster in three steps . First , given our graph representation of a vector graphic , we obtain all the connected components in the graph , based on the stroke-wise edges Es . Secondly , for each pair of connected components , obtain their expanded minimum bounding rectangles and the overlapping area of the rectangles . If the expanded area of two connected components overlap , they are spatially close and are merged to be one regional cluster Ck . The expand length is a hyper-parameter .

Summary: Update at the end of the discussion session:  Discussion among reviewers and with the authors made me increase my score by a total of 2 points.   The paper presents a graph-neural network based approach to detecting objects in vector graphics. The paper works directly on the vector graphics representation by first normalizing the vector graphics to bezier splines and then building a graph representation of the vector graphics. Object detection happens directly on this graph representation. Experiments show a comparison to standard computer vision models.