Recognizing Vector Graphics without Rasterization

1 Introduction . Raster graphics have been commonly used for image recognition due to its easy accessibility from cameras . Most existing benchmark datasets are built upon raster graphics , from ImageNet [ 1 ] for classification to COCO [ 2 ] for object detection . However , due to its pixel-based fix-sized format , raster graphics may lead to aliasing when scaling up or down by interpolation . Fields like engineering design or graphic design require a more precise way to describe visual content without aliasing when scaling ( e.g. , graphic designs , mechanical drafts , floorplans , diagrams , etc ) , so another important image format emerges , namely vector graphics . Vector graphics achieve this powerful feature by recording how the graphics are constructed or drawn , instead of the color bitmaps represented by pixel arrays defined in the raster graphics ( first row in Figure 1 ) . Specifically , vector graphics contain a set of primitives like lines , curves and circles , which is defined with parametric equations in analytic geometry and some extra attributes . As shown in Figure 1 , such vector graphic is usually a document where every primitive is defined precisely and written in a line of textual command . Due to the analytic representation , with few parameters , vector graphics can represent an object at any scale or even in infinite resolution , making it potentially a lot more precise and compact image format than raster graphics . Also , instead of independent pixels , ∗This work was done when the authors were interns at MSRA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . vector graphics give higher level structural information on how low-level elements like points or curves group together to form high level shapes or structures . However , this powerful and widely used data format has been rarely investigated in previous computer vision literature . To explore this data format , this paper focuses on the fundamental recognition tasks : object localization and classification , with wide applications in vector graphics related field like automatic design audit , AI aided design , design graphics retrieval , etc . Existing raster graphics based methods [ 3 , 4 , 5 , 6 , 7 ] takes pixel arrays as input and can not be directly applied on vector graphics . There have been attempt [ 8 ] dealing with this format by rendering vector graphics into raster graphics first . However , rendering the vector graphics into raster graphics could result in a pixel array with super resolutions ( e.g. , thousands by thousands ) , which brings extremely large memory cost , and would be inefficient or even intractable for the traditional models to process . On the other hand , rendering a lower resolution image causes substantial information loss , and the object bounding boxes obtained from a low resolution image could be imprecise when scaled back to the original resolution . Furthermore , the rendering process results in a set of independent pixels and discards the high-level structural information within the primitives . Some of this information could be critical for recognition , such as corners in a shape or contours , etc . To address these issues , we resolve the tasks on vector graphics by introducing a model that does not need rasterization and takes the textual documents of vector graphics as input , called YOLaT ( You Only Look at the Text ) . Instead of rendering the vector graphics into raster graphics , we propose an efficient end-to-end pipeline which predicts objects from the raw textual definitions of primitives . YOLaT first transforms different types of primitives into a unified format . Then it constructs an undirected multi-graph to model the structural and spatial information from the unified primitives . Compared to rendering to raster graphics , this transformation is able to preserve more complete information . YOLaT generates object proposals directly from the vector graphics , which produces precise object bounding boxes . Finally , a dual-stream graph neural network ( GNN ) specifically designed for vector graphics is proposed to classify the graph contained in each proposal , with no extra regression needed for bounding box refinement . To evaluate our pipeline over vector graphics , we use two datasets . i.e. , floorplans and diagrams and show the advantages of our method over the raster graphics based object detection baselines . Without pre-training , our method consistently outperforms raster graphics based object detection baselines , with significantly higher efficiency in terms of the number of parameters and FLOPs . Even compared with the powerful ImageNet pretrained two-stage model , YOLaT achieves comparable performance with 25 times fewer parameters and 100 times fewer FLOPs . We also show visualizations to better demonstrate why looking at the text can capture more delicate details and predicts more accurate bounding boxes . 2 Related Work . Object Detection on Raster Graphics . Currently deep learning based object detection methods dominate the research field with the superior performance . Two-stage object detection methods first generate region proposals and classify and regress the proposals to give object predictions with a deep convolutional networks . R-CNN [ 9 ] and Fast-RCNN [ 3 ] use selective search for proposal generations . Faster-RCNN [ 10 ] speeds up the proposal generation by introducing a region proposal network . He et al . [ 11 ] proposed Mask-RCNN , adding a segmentation branch to the detection model for instance segmentation . To train a more translation-variant backbone , Dai et al . [ 12 ] proposed F-RCN – a new prediction head with position-sensitive score maps . Most two-stage object detection methods have large computation overhead of the proposal generation process , and require running a classification and regression sub-network on all the region proposals . One-stage object detection methods tackle this challenge by removing the proposal generation process and directly predict the object bounding boxes in an end-to-end fashion . Anchor-based methods like SSD [ 4 ] , YOLO series [ 5 , 13 , 14 , 15 ] , RetinaNet [ 6 ] densely tile anchor boxes over the image and conduct classification and bounding box coordinate refinement on each anchor box . Recently , anchor-free methods like CornerNet [ 16 ] , CenterNet [ 7 ] , FCOS [ 17 ] propose to directly find object without presets anchors . Graph Neural Networks . GNN has become a powerful tool for machine learning on graphs . It computes a state for each node in a graph , and iteratively updates the node states according to its neighbors . Spectral approaches [ 18 , 19 ] define a convolution operation in the Fourier domain . Spatial approaches [ 20 , 19 , 21 ] define convolutions directly on the graph . EdgeConv [ 22 ] applies GNN model for classification on 3D Cloud by taking the state difference between neighboring nodes as the input of the aggregation function . [ 23 ] further applies EdgeConv to the object detection task on 3D cloud data by integrating the GNN backbone into an anchor-free detection framework . The closest GNN model to our YOLaT is EdgeConv but YOLaT has extra upgrade specifically designed for vector graphics , including edge attributes , faster inference on densely connected edges , and dual-stream structure for multi-graph . Online Sketch and Handwriting Recognition . Online handwriting and drawing recognition [ 24 , 25 ] handles a data form that very similar to vector graphics , which contains a sequence of discrete points . Most of these methods use sequential models to handle this problem . For example , [ 26 ] proposes to convert the point sequences to a sequence of Bézier Curves and use a LSTM for sequential modeling . Compare to online handwriting , vector graphics contain more types of unordered shapes with more attributes and properties other than polylines , and hence need more general and non-sequential method . Vector Graphics Related Application . One of the most common application for vector graphics is design , such as architecture , graphic design , etc . Several methods in architecture drawing recognition propose to represent symbols in a floor-plan as graphs , and use rule-based graph matching method to classify and localize symbols , such as visibility graph [ 27 ] and attributed relational graph [ 28 , 29 ] . In this paper , we propose a novel scheme that directly construct graph from vector graphics based on Bézier Curve , and the object detection is conducted based on the prediction of GNN . Recent years , a few works develop deep learning based methods to automatically generate vector graphics for computer aided design or converting raster graphics to vector graphcis ( i.e . vectorization ) [ 30 , 31 , 32 , 33 , 34 , 35 ] , while to the best of knowledge , our paper is the first to focus on recognition task on vector graphics . Koch et al . [ 36 ] proposes a large 3D model dataset containing analytic representations , but it lacks semantic labeling to train recognition model . 3 Detection Model . In this paper , we study the problem of object detection leveraging the definitions of the vector graphics without rendering them . Here , we define the task as object localization and object classification . Specifically , the model needs to predict a set of bounding box coordinates as well as the category of the object within the bounding boxes . In this section , we describe our proposed YOLaT , which is an end-to-end efficient pipeline taking the raw definitions of the vector graphics as the input without further rendering the graphics . Figure 2 shows the overall pipeline of YOLaT . We convert the primitives like lines and curves as a universal format of Bézier curves . Based on the Bézier curves , un-directed multi-graphs are constructed to model both spatial and structural relationships among the key-points within a primitive and among different primitives . More details on how we build the graphs can be found in Section 3.1 . To fully explore the vector graphics based on the multi-graph , we propose a dual-stream GNN for graph feature extraction and classification . Section 3.2 shows the detailed design of the proposed dual-stream GNN . Compared to the complex prediction head commonly used in the object detection for raster graphics , YOLaT generates precise proposal bounding boxes directly from high resolution vector graphics . Hence each sub-graph in the proposals are fed into the dual-stream GNN classifier without further correction of the box coordinates . We show how to get the potential bounding boxes and predict their objectiveness and category in Section 3.3 . 3.1 Graph Construction . Universal formats of curves . Compared to the raster graphics represented by pixel arrays , vector graphics have more precise representations and no loss of quality and aliasing when resizing . The vector graphics consists of primitives defined by textual commands described in parametric equations , such as lines , curves , polygons and other shapes.Different primitives are described with different parametric equations . Here , like pixel in raster graphics , we want to find a unified way to describe all types of primitives . We chose Bézier Curve due to its generality and capability of modeling different shapes and curves . Bézier Curve is defined by a set of control points , { p0 , ... , pn } , where n is the order of the curve and pi∈ [ n+1 ] is a 2-d vector for the coordinates of point i . The first point and the last point are the end points of a curve while the rest of the control points usually do not sit on the curve and provide side information instead , such as directional information and curvature statistics of the curve from p0 to pn . We chose cubic Bézier Curve where n = 3 for the balance between modeling capability and computational complexity . Formally , the cubic Bézier curve B is defined as : B ( t ) = ( 1− t ) 3p0 + 3 ( 1− t ) 2tp1 + 3 ( 1− t ) t2p2 + t3p3 , 0 ≤ t ≤ 1 ( 1 ) where B ( t ) defines the position of a specific point at the scale rate of t on the curve from p0 to p3 . Next , we introduce our graph construction based on a set of Bézier Curves . Nodes . To improve efficiency , we only include the points from the set of start points and end points , denoted by P , into the collections of nodes on graphs . The rest of the control points will serve as the edge attributes as defined in the following paragraph . For a point p , the attributes x of the corresponding node include the coordinates of the point , the RGB color value c and stroke width w : x = concat ( px , py , c , w ) , p ∈ P ( 2 ) where px and py denote the coordinate value of the point p along the x axis and y axis respectively . These information is defined in the vector graphic documentation . Edges . We design the graph as a multi-graph containing two sets of edges , namely the stroke-wise edges and the position-wise edges . These two types of edges capture the node relationships from different perspectives . Stroke-wise edges capture the connections defined by the stroke in the vector graphics , which refers to the actual stroke drawn between the start and end point of each Bézier curve . This type of connections represents the structures and layouts of the objects in the vector graphics . Thus , an edge is built in-between two nodes if there is a Bézier Curve linking them : Es = { ( vi , vj ) : ( vi , vj ) ∈ S } ( 3 ) where S denotes the set of tuples containing the start point vi and end point vj of a Bézier Curve . Other than the connections between start and end points , other attributes of a cubic Bézier curve like curvature or other appearance are described by the off-curve control points . We use the coordinates of these off-curve control points as the attributes of the stroke-wise edges : xe = concat ( px , py ) , p /∈ P ( 4 ) The stroke-wise edges only model the long-term structural connection between the vertices based on strokes , which is irrelevant to the spatial vicinity . To further capture the spatial relationship between nodes , we generate another set of edges , called position-wise edges . Specifically , the position-wise edges are defined as the dense connections among nodes within a regional cluster Ck : Ep = { ( vi , vj ) : vi , vj ∈ Ck } , k ∈ { 1 , 2 , ... , m } , ( 5 ) A regional cluster is a set of nodes close to each other spatially , which can by obtained in different ways . In our implementation , we obtain regional cluster in three steps . First , given our graph representation of a vector graphic , we obtain all the connected components in the graph , based on the stroke-wise edges Es . Secondly , for each pair of connected components , obtain their expanded minimum bounding rectangles and the overlapping area of the rectangles . If the expanded area of two connected components overlap , they are spatially close and are merged to be one regional cluster Ck . The expand length is a hyper-parameter .

Summary: The paper addresses the problem of object recognition, i.e., object localization and classification, in Vector Graphics. The input is a text document that describes the vector graphics, and the output is a bounding box corresponding to a localized object.  The proposed approach is supervised.  The main idea is to represent the input textual document describing the Vector Graphic (VG) object using a unified representation for different primitives described in the text document (such as a Bezier curve). The VG objects are represented using a Graph, where the nodes correspond to the start and end points of the Bezier curves.  A dual-stream Graph Neural Network (GNN) is proposed to aggregate feature representation for the input VG object, which is in-turn used for classification and localization.  The method is evaluated using Average Precision metric.   Two kinds of publicly available VG datasets are used: Floorplan and Diagrams. Comparison against object recognition techniques that employ a CNN has been done, and the efficiency of dual-stream GNN architecture is demonstrated.