Recognizing Vector Graphics without Rasterization
Authors: XINYANG JIANG
Lu Liu
Caihua Shan
Yifei Shen
Xuanyi Dong
Dongsheng Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vector Graphics
Object Detection
Image Recognition

Abstract: In this paper, we consider a different data format for images: vector graphics. In contrast to raster graphics which are widely used in image recognition, vector graphics can be scaled up or down into any resolution without aliasing or information loss, due to the analytic representation of the primitives in the document. Furthermore, vector graphics are able to give extra structural information on how low-level elements group together to form high level shapes or structures. These merits of graphic vectors have not been fully leveraged in existing methods.  To explore this data format, we target on the fundamental recognition tasks: object localization and classification. We propose an efficient CNN-free pipeline that does not render the graphic into pixels (i.e. rasterization), and takes textual document of the vector graphics as input, called YOLaT (You Only Look at Text). YOLaT builds multi-graphs to model the structural and spatial information in vector graphics, and a dual-stream graph neural network is proposed to detect objects from the graph. Our experiments show that by directly operating on vector graphics, YOLaT outperforms raster-graphic based object detection baselines in terms of both average precision and efficiency. Code is available at https://github.com/microsoft/YOLaT-VectorGraphicsRecognition. 

Recognizing Vector Graphics without Rasterization
Xinyang Jiang', Lu Liu"; Caihua Shan', Yifei Xuanyi Dong"; Dongsheng Li'
Microsoott Research Asia
(xinyangj iang, caihua.shan, dongsheng...
soa.gaaghhaganddddeeeeeeeng 1ii (11l@mmiirot cmiioosoot.ccmm
University of Technology Sydney
u. 1il cs@icloud. com, xuanyi muanyi.dxya dxx@gmailll...m com
'The Hong Kong University of Science and Technology
yshenaw&connect. ust.
Abstract
In this paper, we consider a different data format for images: vector graphics. In contrast to raster graphics which are widely used in image recognition, vector graphics can be scaled up or down into any resolution without aliasing or infor- mation loss, due the analytic representation Furthermore, vector graphics are able to give extra structural information on how low-level elements group together form high level shapes or structures. These merits of graphic vectors have not been fully leveraged in existing methods. To explore this data format, we target on the fundamental recognition tasks: object localization and classification. We propose an efficient CNN-free pipeline that does not render the graphic into pixels (i.c. rasterization), and takes textual document the vector graphics input, called YOLaT (You Only Look at Text). YOLaT builds multi-graphs to model the structural and spatial information in vector graphics, and graph neural network is proposed detect objects from the graph. Our experiments show that by directly operating on vector graphics. YOLaT out- performs raster- sitter-rappic based object detection baselines in terms of both average precision and efficiency. Code available at https:/github..com wwt.cccooo////ot/////oo/mo aOL.-- VectorGrapphissReconniie
the primitives in the document.
Introduction
Raster graphics have been commonly used for image recognition due to its casy accessibility from cameras. Most existing benchmark datasets are built upon raster graphics, from ImageNet [1] for classification to COCO [2] for object detection. However, due to its pixel-based fix-sized format, raster graphies may lead to aliasing when scaling up or down by interpolation. Fields like engineering design or graphic design require more precise way to describe visual content without aliasing when scaling (e.g.. graphic designs, mechanical drafts, floorplans, diagrams, etc), another important image format emerges, namely vector graphics.
Vector graphics achieve this powerful feature by recording how the graphics constructed drawn, instead of the color bitmaps represented by pixel arrays defined in the raster graphics (first row in Figure 1). Specifically, vector graphies contain a set of primitives like lines, curves and circles, which is defined with parametric equations in analytic geometry and some extra attributes. As shown in Figure 1, such vector graphic is usually a document where every primitive is defined precisely and written in a line of textual command. Due to the analytic representation, with few parameters, vector graphics can represent an object at any scale or even in infinite resolution, making potentially lot more precise and compact image format than raster graphics. Also, instead of independent pixels,
"This work was done when the authors were interns at MSRA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The from Borit servadl soid shalind
Vaghtty ritter grapiic
Wh's koring
0. 80, 50
80.
m0
Magajhattors select part of
VoT
The bex bier in the world Best served cold chillal
nhe a1-7120 ylanggee x2-7377" (2-T1044 siaker "tack
51-"273 yoant
you 1004 strok yollo o..
Magely vettor graphics
Why no slasing
scircle ce 2920" op- 26T (-1411 Wichson" stroks"Mack"
MAng-Busttos oelect suavote from poinsttees
Figure 1:
Difference between raster graphics (row 1) and vector graphics (row 2).
vector graphics give higher level structural information on how low-level elements like points or curves group together to form high level shapes structures. However, this powerful and widely used data format has been rarely investigated previous computer vision literature.
To explore this data format, this paper focuses on the fundamental recognition tasks: object lo- calization and classification, with wide applications in vector graphics related field like automatic design audit, Al aided design, design graphics retrieval, etc. Existing raster graphics based methods [3, 4, 5, 6, 7] takes pixel arrays as input and cannot be directly applied on vector graphics. There have been attempt [8] dealing with this format by rendering vector graphics into raster graphics first. However, rendering the vector graphics into raster graphics could result in a pixel array with super resolutions (e.g., thousands by thousands), which brings extremely large memory cost, and would be inefficient or even intractable for the traditional models to process. On the other hand, rendering lower resolution image causes substantial information loss, and the object bounding boxes obtained from a low resolution image could be imprecise when scaled back to the original resolution. Furthermore, the rendering process results in set of independent pixels and discards the high-level structural information within the primitives. Some of this information could be critical for recognition, such corners in shape contours, etc.
To address these issues, we resolve the tasks on vector graphics by introducing a model that does not need rasterization and takes the textual documents of vector graphics as input, called YOLaT(You Only Look at the Text). Instead of rendering the vector graphics into raster graphics, we propose efficient end-to-end pipeline which predicts objects from the raw textual definitions of primitives. YOLaT first transforms different types of primitives into a unified format. Then it constructs an undirected multi-graph Compared to rendering to raster graphics, this transformation is able to preserve more complete information. YOLaTT generates object proposals directly from the vector graphics, which produces precise object bounding boxes. Finally, a dual-stream graph neural network (GNN) specifically designed for vector graphics proposed classify the graph contained each proposal, with no extra regression needed for bounding box refinement.
model the structural and spatial information from the unified primitives.
To evaluate our pipeline over vector graphics, show the advantages our method over the raster graphics based object detection baselines. Without pre-training, method consistently outperforms raster graphics based object detection baselines, with significantly higher efficiency in terms the number of parameters and FLOPs. Even compared with the powerful ImageNet pretrained two-stage model, YOLaT achieves comparable performance with 25 times fewer parameters and 100 times fewer FLOPs. We also show visualizations to better demonstrate why looking at the text can capture more delicate details and predicts more accurate bounding boxes.
two datasets. i.e., floorplans and diagrams and
2 Related Work
Object Detection on Raster Graphics. Currently deep learning based object detection methods dominate the research field with the superior performance. Two-stage object detection methods first generate region proposals and classify and regress the proposals to give object predictions with a deep convolutional networks. R-CNN [9] and Fast-RCNN [3] use selective search for proposal generations. Faster-RCNN [10] speeds up the proposal generation by introducing a region proposal network. He et al. [11] proposed Mask-RCNN, adding segmentation branch to the detection model

---

for instance segmentation. To train more translation -ariiiannnn-niann backbone, Dai et al. [12] proposed F-RCN a new prediction head with position-sensse sensitive score maps.
Most two-stage object detection methods have large computation overhead of the proposal generation process, and require running classification and regression sub-network all the region proposals. One-stage object detection methods tackle this challenge by removing the proposal generation process and directly predict the object bounding boxes in an end-to-end fashion. Anchor-based methods like SSD [4], YOLO series [5, 13, 14, 15], RetinaNet [6] densely tile anchor boxes over the image and conduct classification and bounding box coordinate refinement on each anchor box. Recently, anchor-free methods like CornerNet [16]. CenterNet(7], FCOS [17] propose to directly find object without presets anchors.
Graph Neural Networks. GNN has become a powerful tool for machine learning on graphs.
computes state for each node in graph, and iteratively updates the node states according to its neighbors. Spectral approaches [18, 19] define a convolution operation in the Fourier domain. Spatial approaches [20, 19, 21] define convolutions directly on the graph. EdgeConv [22] applies GNN model for classification on 3D Cloud by taking the state difference between neighboring nodes as the input of the aggregation function. [23] further applies EdgeConv to the object detection task on 3D cloud data by integrating the GNN backbone into an anchor-free detection framework. The closest GNN model to our YOLaT EdgeConv but YOLaT has extra upgrade specifically designed for vector graphics, including edge attributes, faster inference on densely connected edges, and dual-stream structure for multi-graph.
Online Sketch and Handwriting Recognition. Online handwriting and drawing recognition [24, 25] handles a data form that very similar to vector graphics, which contains sequence of discrete points. Most of these methods use sequential models to handle this problem. For example, [26] proposes convert the point sequences to a sequence of Bézier Curves and use LSTM for sequential modeling. Compare to online handwriting, vector graphics contain more types of un- ordered shapes with more attributes and properties other than polylines, and hence need more general and non- sequentelt method.
Vector Graphics Related Application. One of the most common application for vector graphics is design, such as architecture, graphic design, etc. Several methods in architecture drawing recognition propose to represent symbols in floor-plan as graphs, and use rule-based graph matching method to classify and localize symbols, such as visibility graph [27] and attributed relational graph [28, 29]. In this paper, we propose a novel scheme that directly construct graph from vector graphics based on Bézier Curve, and the object detection is conducted based on the prediction of GNN. Recent years, few works develop deep learning based methods to automatically generate vector graphics for computer aided design or converting raster graphics to vector graphcis (i.e. vectorization) [30, 31, 32, 33, 34, 35], while to the best of knowledge, our paper the first to focus on recognition task on vector graphics. Koch et al. [36] proposes large 3D model dataset containing analytic representations, but it lacks semantic labeling to train recognition model.
3 Detection Model
In this paper, we study the problem of object detection leveraging the definitions of the vector graphics without rendering them. Here, we define the task as object localization and object classification. Specifically, the model needs to predict set of bounding box coordinates as well as the category of the object within the bounding boxes.
In this section, we describe our proposed YOLAT,, which is an end-to-end efficient pipeline taking the raw definitions of the vector graphics as the input without further rendering the graphics. Figure shows the overall pipeline of YOLAT. We convert the primitives like lines and curves as a universal format of Bézier curves. Based on the Bézier curves, un-directed multi-graphs are constructed to model both spatial and structural relationships among the key-points within a primitive and among different primitives. More details on how we build the graphs can be found in Section 3.1. To fully explore the vector graphics based on the multi-graph, we propose a dual-stream GNN for graph feature extraction and classification. Section 3.2 shows the detailed design of the proposed dual-stream GNN. Compared to the complex prediction head commonly used in the object detection for raster graphics, YOLaT generates precise proposal bounding boxes directly from high resolution vector graphics. Hence each sub-graph in the proposals are fed into the dual- dual-stream GNN classifier
3

---

Formal Unii aaliianioon
Graph Condruction
Progosal Gemeration
dog
Dual Stream GNN
pbject
Vestor Graphkcs
Cabie deear Curses
Gruti-Gruph
Propocit
object
Strake wise GNN Blaccbone
Lover
SKI CoFe
1-255
LKC CUFV F:0 Oboses
GRN Layprr
Poction-wis ENN Bacibone
GNN Blaccoone
MEP fleesterr
Figure 2: The overall pipeline of the proposed method.
without further correction of the box coordinates. We show how get the potential bounding boxes and predict their objectiveness and category in Section 3.3.
3.1 Graph Construction
Universal formats of curves. Compared to the raster graphics represented by pixel arrays, vector graphics have more precise representations and no loss of quality and aliasing when resizing. The vector graphics consists primitives defined by textual commands described in parametric equations, such as lines, curves, polygons and other Chapes.Differee parametric equations. Here, like pixel in raster graphics, we want to find unified way to descrihe all types of primitives. We chose Bézier Curve due to its generality and capability of modeling different shapes and curves. Bézier Curve is defined by a set of control points, Po. Pr. ), where n the order of the curve and Pic(n+11 is 2-d vector for the coordinates of point i. The first point and the last point are the end points of a curve while the rest of the control points usually do not sit on the curve and provide side information instead, such directional information and curvature statistics of the curve from Po to Pa. We chose cubic Bézier Curve where n = 3 for the balance between modeling capability and computational complexity. Formally, the cubic Bézier curve B is defined as:
primitives are described with different
B(t) = (1 - t) + 3(1 - t) + 3(1 - t)t'p2 + t 0
(1)
where (t) defines the position of a specific point at the scale rate of t on the curve from Po to P3- Next, we introduce our graph construction based on set of Bézier Curves.
Nodes. To improve efficiency, we only include the points from the set of start points and end points, denoted by P. into the collections of nodes graphs. The rest of the control points will serve as the edge attributes as defined in the following paragraph. For a point p. the attributes æ of the corresponding node include the coordinates of the point, the RGB color value and stroke width w:
concat(p", pp", C, w) P
(2)
where p" and pd denote the coordinate value of the point p along the axis and y axis respectively. These information is defined in the vector graphic documentation.
Edges. We design the graph as a multi-grap graph containing two sets of edges, namely the stroke-wiss edges and the position- wise edges. These two types of edges capture the node relationships from different perspectives.
Stroke-wise edges capture the connections defined by the stroke in the vector graphics, which refers the actual stroke drawn between the start and end point of each Bézier curve. This type of connections represents the structures and layouts of the objects in the vector graphics. Thus, edge is built in-between two nodes if there is a Bézier Curve linking them:
E. {v, Uz)= (vi, 0j) E ES}
where S denotes the set of tuples containing the start point and end point of a Bézier Curve.
Other than the connections between start and end points, other attributes of cubic Bézier curve like curvature or other appearance are described by the off-curve control points. We use the coordinates of these off-curve control points as the attributes of the stroke-wise edges:
concat(p" (p" ,p"),
(4)

---

The stroke wise edges only model the long-term structural connection between the vertices based on strokes, which is irrelevant to the spatial vicinity. To further capture the spatial relationship between nodes, we generate another set of edges, called position-wist wise edges. Specifically, the position-wise edges are defined as the dense connections among nodes within a regional cluster Ca:
Ep = {(v,, 11): U4 E Cc), k E f1,2,..., m.
(5)
regional cluster is a set of nodes close to each other spatially, which can by obtained in different ways. In our implementation, we obtain regional cluster in three steps. First, given graph representation of a vector graphic, we obtaín all the connected components in the graph, based on the stroke-wise edges & Secondly, for each pair of connected components, obtain their expanded minimum bounding rectangles and the overlapping area of the rectangles. If the expanded area of two connected components overlap. they are spatially close and are merged be one regional cluster CR. The expand length is
3.2 Feature Extraction with Dual-stream GNN
In the previous section, we introduced how to generate graphs, including building nodes, two sets of edges and attributes. To analyze the proposed multi-graph, YOLaT applies GNN network specifically designed for the graph built from vector graphics. Since the proposed graph is defined by two sets of edges hand, YOLaT uses dual-stream GNN structure where specific GNN branch is designed to update node representations based on each type of edges. The node representations extracted by the dual-stream GNN are able leverage the spatial and structural information in vector graphics, and can better guide the following head for the downstream tasks.
In the following section, we first elaborate the details of both streams in GNN. Then we intro- duce how to get the representation a specific region by leveraging multi-step node representations propagation and representation fusion.
Siseke-wise Stream. For the graphs with troke-wise edges, inspired from [37], this GNN takes the input of the concatenation of node representation, the difference of the representation to its
neighbor node, and the attributes on the edge. At time step & + 1, the representations h f++ for node i updated follows:
= ((b()
Σ JEN.
(' h3 h x(,)),
W
where the initialization calculated as Equation 2, i.e., h =x. x,, denotes the attributes on the stroke-wise wise edge between node and node j as defined in Equation 4. I' is a linear transformation function. .₁ denotes set of nodes adjacent the i-th node in terms of stroke-wise edges in the graph, and f denotes a transformation function which consists of a linear transformation, a ReLU activation function [38] and batch normalization layer [39] in our implementation.
Position-wise Stream. Since the position-wise edges are constructed densely as described in Sec- tion 3.1, the number of position-wise edges is significantly larger than that of the stroke-wise edges. To reduce the computational cost, we design simpler GNN model for graphs with position wise edges. At time step t++ the model only takes the representation of node and the node representation 1+1 is updated by considering the neighboring transformed representation:
H+1
1 WTT
Σ SP(2)).
SENTULO
where A.? denotes the neighbors of node defined by the positional edges (to maintain the information from v self-loop for each node is added). f is transformation function with the same structure but untied parameters as f². The initialization of the node representation also calculated as Equation 2, ie., 20 Compared significantly, because the updates of the nodes in the same regional cluster C₁ only needs to be computed once. More details of the efficient implementation for the GNN can be found in Section 4.
stroke-wise edge, the computation complexity of f" be reduced
Representation Fusion. The representation of a specific region in vector graphics is based on the cluster of nodes representations located within this region. Specifically, given the node representations
5

---

refined by the proposed dual-stream GNN, we average the node representations over the nodes inside this region V" get a region representation and concatenate the region representations for 7 steps:
concat(r., r 53
r2 ro rl
(8)
1 M Σ IIEE, r r
E M
(9)
where denotes the fused representation of a specific region and r r denote the region representa- tion time step from the graph with stroke-wiss wiie edges and position wise edges, respectively.
3.3 Prediction and Loss
Here we propose a vector graphics based proposal generation method. Given a vector graphic, first evenly slices each regional clusters C into grids. Then, permute all vertex pairs the grid mesh, each of which forms the top-left and bottom-right points of a rectangle region. The nodes, edges and corresponding primitives within each rectangle region forms proposed object, whose minimum bounding rectangle is the bounding box of the proposal. Note that proposals with size larger than a threshold is filtered. Compared to generating proposals on raster graphics, YOLaT produces much fewer negative samples, and operates at highest resolution to directly produce tightest bounding boxes around the proposed object. Hence, YOLaT requires no extra regression branch for bounding box refinement.
For each proposal, a proposal B's representation r is obtained by the representation fusion strategy described in previous section, which is then fed into a multi-layer perception to predict object category. During training, we only optimize the cross-entropy loss over the prediction and D's ground truth label y. In each image, for each ground truth object box B, its label is y/- We set the same as that of the ground truth B;, which has the largest Intersection over Union (IoU) with the proposal B. the largest IoU is below a threshold a, we regard this proposal as "no object" and set its ground truth label as the total number of classes C' (the class index is from 0). We minimize the cross entropy loss of each proposal:
min
log Pr(yl // B) min - log Pr(y/r),
(10)
arg y,y loU B, B,) if max(loU( B:)) >= C
(11)
else
where Y' is the set of all ground truth labels for the boxes {B:).
During evaluation, we regard the probability of the classification as the confidence level for this pro- posal and select the bounding boxes with the confidence level above a set threshold as the predictions.
4 Experiments
4.1 Implementation Details
Architecture. In the model used in our main results comparison, we build two-layer GNN for both position-wise stream and stroke-wise stream with dimension of all the hidden node representations set to 64. We observe no significant performance improvement with deeper GNN due to the over- smoothing effect. In our graph, the number of position connectivity within each regional cluster. To speed up the inference, we first pre-compute the transformation function fr on each node. Then for each regional cluster, we aggregate the obtained node representations with mocan-ooling. The aggregated representation for each regional cluster is then assigned to each node in the cluster as their new node representation. In this way, each node only requires one transformation operation and one mean-pooling operation. Furthermore, since the fully connected graph constructed by position-wis edges could cause severe over-smoothe sootthit problem, after run f" on each node, the mean aggregate operation is only conducted on the last layer of GNN. We use a three-layer MLP as classifier, where the dimension of middle layer output is 512 and 256.
edges is quite large due to its full
Graph Construction and Proposal Generation. Our experiments use a widely used vector graphics standards called Scalable Vector Graphies (SVG). All the primitives in SVG are first converted to cubic Bézier Curves. A circle is split equally into four parts and then each part is converted into Bézier

---

Table 1: Performance comparison on the floorplan dataset.
Methods
Pretrained
AP (%)
APTs (%)
mAP (%)
Inference time (ms)
Params(M)
GFL.PPs
Yolov3-tiny
X
75.23
60.97
53.24
1.2
8.7
13.0
Yolov3
X $8
80.44
72.98
8.2
61.6
155.2
Yolox3-spp
X
$7.38
79.66
71.61
8.3
62.7
156.1
Yolov4
X
93.04
87.48
79.59
11.7
70.3
165.5
Waster-renn-RS--FP
K
80.91
71.48
67.32
58.7
28.4
126.8
faster-renn-EE R34-FPN
Sesta-rrennR34-FPMM
X
80.50
72.18
65.89
61.9
38.5
157.3
faster-renn-R55
73.28
41.4
165.7
refinanet-R50-FPN
87,50
82.91
79.18
79.2
38.0
189.2
YOL.T (Ours)
K
98.83
94.65
90.59
1.3
1.6
1.5
faster-renn-Ee R50-FPN
71.2
41.4
165.6
Yolov3
74.61
60.33
53.76
8.2
61.6
155.2
curves. We also split curves at the intersection into multiple sub-curves to model delicate differences. For proposal generation, each region cluster is slices into grid with 10 columns and 10 rows.
Training. We use Adam optimizer with learning rate of 0.0025 and a batch size of 16. For data augmentation, we randomly translate and scale the vector graphics by at most 10% of the image width and height, and the transformed vector graphics are further rotated by a random angle. The model is trained for 200 epochs from scratch which takes around 2 hours on a Nvidia V V00 graphic card.
4.2 Datasets
We use SESYD, which is a public database containing different types vector graphic documents, with the corresponding object detection groundtruth, produced using the 3gT system'. Our experi- ments use the floorplans and diagrams collections.
Floorplans. This dataset includes vector graphics for floorplans. It contains 1,000 images with totally 28,065 objects in 16 categories, e.g., armchair, tables and windows. The images are evenly divided into 10 layouts. We divide half of the layouts as the training data and the other half for validation and test. The ratio of the validation and test data is 1:9.
Diagrams. This dataset includes vector graphics for diagrams. It contains 1,000 images with totally 1,4100 objects in 21 categories, e.g., diode and resistor. There are 10 layouts and 100 images for each layout. Note that scale variance of different objects is huge in this dataset. For example, resistor is often much smaller compared to transistor. We divide the training, validation and test set in way that objects from the same category are included both training and testing set. Thus, the dataset is split as 600, 41 and 359 images for training, validation and test stage.
4.3 Evaluation Metric
We evaluate the models in terms of both accuracy and efficiency. For accuracy, we use AP50. AP75 and mAP, where AP. represents the average precision with the intersection over union (IOU) threshold for counting as detected as 50%, and 75%. mAP is the mean of the average precision for the IOU threshold between 0.50 and 0.95. We also evaluate the efficiency because of the real-world requirements for real-time object detection. Specifically, we use GFLOPs (Giga (one billion) Floating point operations) and inference time for model efficiency and we also report the number of parameters to meet the scenarios of limited resources. The inference time is evaluated on Nvidia V100.
4.4 Comparison to Baselines
We compare YOLaT with two types object detection methods: one-stage methods, i.c., Yolov3 [14], Yolov4 [15, 40] and its variants, RetinaNet [6], and twoestage methods, i.e., faster-renn with Pyramid Network (FPN) [41] and its variants. For Yolov3, the -tiny variant smaller model and the sspp uses Spatial Pyramid Pooling. For Yolov4, we use a scaled Yolov4 [40] with slightly more parameters and potentially much better performance called Yo1ov4-P5. The faster-renn-PP RFFNN model series
http://mattieu.delllaadd htw.:aaawannd........oooooooeeea
7

---

Table 2: Performance comparison on the diagram dataset.
Methods
Pretrained
AP50 (%)
APTs (%)
mAP (
Inference time (ms)
Params (M)
GFLOPs
Yolov3-tiny
88.40
79.53
71.42
3.6
8.7
13.0
Yolov3
89.69
81.38
78.20
61.6
155.2
Yosp
90.29
84.51
78.68
10.8
62.7
156.1
Yolov4
88.71
84.65
76.28
11.1
70.3
165.5
Dsstr-reon-R18-777
92.79
85.89
34.4
28.4
121.1
Baaee--renn-334-11
90.47
88.74
85.21
36.0
38.5
150.0
faster-rean-K00-E
41.7
158.0
retinantt-50-FPN
91.33
83.17
82.79
47,9
38.0
179.5
YOLAT (Ours)
96.63
94.89
89.67
2.1
1.6
2.9
Bister-renn-55--F R55-EEN Yolov3
41.7
157.9
90.11
84.68
79.55
8.2
61.6
155.2
Table 3: Ablation study and variant analysis on the floorplan dataset.
(a) Ablation study graph construction.
(b) Ablation study and variant analysis GNN model.
Methods
APso(%)
(P)
mAP(3)
Methods
APso(%)
AP7a(%)
mAP(S.)
YOLaT
98.83
94.65
90.59
stroke-wise
w/o h
96.83
93.19
88.40
streum
w/o hj
92.90
w/o Ep
95.81
91.03
87.17
w/o
91.57
91.22
86.00
position-wise streum
early aggregate with h; hy
V5 95.82
91.64
87.90
w/o
94.57
90.76
86.25
98.67
94,46
GCN GAT
90.36
88.02
83.32
aggregation function
y1.20
89.46
83.92
GraphSage
92.70
91.17
85.26
use backbones of different scales, with ResNet18 [42]. ResNet34, ResNet50 for R18, R34, R50, respectively.
We choose these baselines because they are the most popular methods in object detection. On both datasets, YOLaT outperforms all baselines without pretraining on ImageNet terms of precision and efficiency as shown in Table and Table 2. We also include a baseline, i.e., faster-renn-RSo ---nn-R5--FPN which is pretrained on ImageNet, YOLaT shows competitive precision with around 100x less FLOPs and around 25 less model parameters. We also train Yolov3 with ImageNet pretrained backbone, do not observe performance improvement. We conduct 3 rounds of experiments with different random seeds and the standard error in terms of AP50 is 0.0003 on floorplan and 0.0008 on diagram.
For Yolov3, we use the implementation of ultralytics [43]. For Yolov4 we use the official pytorch implementation of Scaled Yolov4 [40]. Note that both the Yolov3 and Yolov4 implementation shows superior performance on COCO [2] when without ImageNet pretraining. For faster-renn and retina-net, we use the Detectron2 [44] library", For the non-pretrained model, we use the strategies of replacing Batch normalization Group Normalization following [45] improve the performance.
Broader Impact. Our YOLAT model may present promising solution for applications that have the input of vector graphics. Any deployment of the proposed model however should be preceded by an analysis the potential biases captured by the dataset sources used for training and the correction of any such undesirable biases captured by the pre-trained backbones and model.
4.5 Ablation Study and Variants Analysis
Graph Construction. We analyze the effectiveness of the Bézier based graph construction method in YOLaT on CYYSED-Floopplan dataset. Table 3a shows the results of the ablation study position- wise edges Ep (as defined in Eq. 7), stroke-wise edges E, (as defined in Eq. 6) and edge attributes xij (as defined in Eq. 2). The ablation of these components show significant drop in precision.
Dual-Stream GNN. As show Table 3b, we conduct several experiments analyze the effective- ness of our dual-stream GNN that is specifically designed for vector graphics recognition. We did the
Phtps://github.com http://gggiuuu.oom///uaalllyssoo

---

Yolo v3
Missing objects: Wrong predictions: 4
Imprecise box
Struggled with details
STR www.ororpteenn. wenentor weweerann prottarattttn www.ssttooooooo... 1:00
Caralsor-pm 1.00
Tranertor-pop 1:00
YOLaT
Missing objects: Wrong predictions: 0
precise box
Capture details
Figure 3: Visualizations of Yolov3 (upper line) and YOLaT (lower line) show that 1) [Leftt] Yolov3 has more missing objects (shaded boxes in YOLaT figure) and wrong predictions (shaded boxes in Yolov3 figure). 2) [Mid] The prediction boxes of YOLaT are tighter and more accurate. 3) [Right] Yolov3 can not distinguish the details of transistors, e.g., the direction of the arrows, leading wrong predictions.
ablation of YOLaT without the input h and h3 hf. For position-wise stream, feature aggregation for position-wise edges is conducted on every layer in GNN, instead of only last layer. the experiment results show that early aggregation hurts the performance, due to the over-smoothing caused by fast message passing along the fully connected edges. Due to the high computation complexity of aggregation function on fully connected position-wise edges, YOLaT discards neighboring feature difference in fP. This experiment shows that there is no obvious performance improvement by adding neighboring feature difference on f". Meanwhile, this method significantly increases the computation complexity by and increase GFLOPs by almost 60% from 1.5 to 2.4. The last three rows of Figure 3b shows the performance comparison between YOLaT and some other popular GNN aggregation methods. In this experiments, aggregation functions in GCN [19), GAT [21] and GraphSage [20]. Since some of these methods do not directly support edge attributes, similar to dual-stream GNN, treat extra dimensions of features of pair of adjacent nodes. The experiment shows that our GNN outperforms existing GNN methods, which further verifies the effectiveness of our vector graphics specific design.
replace our proposed aggregation function with the
4.6 Visualizations
We visualize the detection results for Yolov3 and YOLaT as in Figure 3. The prediction results in first two columns show that the bounding box predicted by Yolo is imprecise while the bounding box predicted by YOLaT is precise and sits exactly at the border of every object. For example, both models generate bounding box for the table in the middle of the figure, while YOLaT outputs tighter box for the object border. This is because YOLaT directly looks at the text and leverages the information of where the positions of the curves are, while Yolo only leverages the lower resolution pixel arrays rendered from the text. The imprecise predictions can affect the performance for higher standard detection which is reflected as the AP with higher IOU. This is why the gap between Yolo and YOLAT is bigger for mAP compared to that for AP50 as shown in Table 1. Also, Yolo gives more undetected cases under strict threshold, such the armchair and sofa in Figure 3.
The third column on Figure shows that Yolov3 fails to distinguish the object details (the direction of arrows in different types of transistors) due to the limited resolution of raster graphics, while by

---

directly operating on the vector graphics with each primitive precisely described by textual command, YOLaT is able capture the details at very small scale.
Conclusions
We propose efficient CNN-free pipeline does not need rasterization called YOLaT(You Only Look Text). YOLaT builds unified representations for all primitives in vector graphic with un-directed multi-graph and detect objects with dual-stream GNN specifically designed for vector graphics. The experiments show that YOLaT outperforms both one-stage and two-stage deep learning methods vector graphics,
with much better efficiency. Our work provides new direction for recognition and is able to draw more researchers' attention on exploring the merits of vector graphics. In the future, there is much work further improve YOLaT and recognition vector graphies in general, such as leveraging both vector graphic and raster graphic based methods, building a GNN model for vector graphics that supports deeper structure, large vector graphics dataset support backbone pre-training.
References
[1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in CVPR, 2009.
[2] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Zitnick, "Microsoft coco: Common objects in context," in ECCV, 2014.
[3] R. Girshick, "Fast r-cnn," in ICCV, 2015.
[4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, "Ssd: Single shot multibox detector," in ECCV, 2016.
[5] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in CVPR, 2016.
[6] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in ICCV, 2017.
[7] X. Zhou, D. Wang, and P. Krähenbühl, "Objects as points," in arXiv preprint arXiv:1904.07850
2019.
[8] A. Rezvanifar, M. Cote, and A. B. Albu, "Symbol spotting on digital architectural floor plans using a deep learning-based framework," in CVPR-W, 2020.
[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in CVPR, 2014.
[10] Ren, K. He, Girshick, and J. Sun, "Faster r-cnn: towards real-time object detection with region proposal networks," IEEE TPAMI, 2016.
[11] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," ICCV, 2017.
[12] Dai, Y. Li, K. He, and Sun, "R-fen: object detection via region- based fully convolutional
networks," in NeurlPS, 2016.
[13] J. Redmon and A. Farhadi, "Yolo9000: better, faster, stronger," CVPR, 2017.
[14] Redmon and F.Fahaadi, "Yolov3: An incremental improvement," arXiv preprint
1rXiv:1804.02767. 2018.
[15] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, "Yolov4: Optimal speed and accuracy of object detection." " arXiv preprint arXiv:2004. 10934, 2020.
[16] H. Law and J. Deng, "Comernet: Detecting objects paired keypoints," in ECCV, 2018.
[17] Z. Tian, C. Shen, H. Chen, and T. He, "Fcos: Fully convolutional one-stage object detection," in ICCV, 2019.
[18] Bruna, W. 7aremba, A. Szlam, and Y. Lecun, "Spectral networks and locally connected networks on graphs," in ICLR, 2014.
[19] N. Kipf and M. Welling, "Semi-superrisse classification with graph convolutional networks," in ICLR, 2017.
10

---

[20] W. L. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning on large graphs," in NeurIPS, 2017.
[21] Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, "Graph attention networks," arXiv preprint arXiv:/710. 10903, 2017.
[22] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, "Dynamic graph enn for learning point clouds," ACM TOG, 2019.
[23] W. Shi and R. Rajkumar, "Point-gnn: Graph neural network for 3d object detection in point cloud," " in CVPR, 2020.
[24] D. Ha and D. Bck, "A neural representation of sketch drawings," " arXiv preprini arXiv:1704.03477 2017.
[25] E. Aksan, T. Deselaers, A. Tagliasacchi, and O. Hilliges, "Cose: Compositional stroke embed- dings," in NeurlPS, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020.
[26] V. Carbune, P. Gonnet, Deselaers, H. A. Rowley, A. Daryin, M. Calvo, L.-L. Wang. D. Keysers, S. Feuz, and P. Gervais, "Fast multiilanguage lstm-based online handwriting recognition," IJDAR, vol. 23, no. 2. pp. 89-102, 2020.
[27] H. Locteau, S. Adam, Trupin, Labiche, and P. Héroux. "Symbol spotting using full visibility graph representation," in Workshop on Graphics Recognition, 2007.
[28] J.-Y. Ramel, N. Vincent, and H. Emptoz, "A structural representation for understanding line- drawing images," IJDAR, 2000.
[29] K. Santosh, B. Lamiroy, and L. Wendling, "Symbol recognition using spatial relations," Pattern Recognition Letters, 2012.
[30] A. Carlier, M. Danelljan, A. Alahi, and R. Timofte, "Deepsvg: A hierarchical generative network for vector graphics animation," in NeurIPS, 2020.
[31] Y. Ganin, S. Bartunov, Y. Li, E. Keller, and S. Saliceti, "Computer-aided design as language." arXiv preprint arXiv:2105. 02769, 2021.
[32] P. Reddy, M. Gharbi, M. Lukac, and N. J. Mitra, "Im2vec: Synthesizing vector graphics without vector supervision," arXiv preprint PrXiv:2102.02700 2021.
[33] R. G.G. Lopes, D. Ha, D. Eck, and J. Shlens, "A learned representation for scalable vector graphics," in ICCV, 2019, pp. 7930-7939.
[34] I.-C. Shen and B.-Y. Chen, "Clipgen: A deep generative model for clipart vectorization and synthesis," IEEE Transactions on Visualization and Computer Graphics, 2021.
[35] A. D. Parakkat, M.-P. R. Cani, and K. Singh, "Color by numbers: Interactive structuring and vectorization of sketch imagery," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1-11.
[36] S. Koch, A. Matveev, Z. Jiang, F. Williams, A. Artemov, E. Burnaev, M. Alexa, D. Zorin, and D. Panozzo, "Abc: A big cad model dataset for geometric deep learning," " in CVPR, 2019.
[37] G. Li, M. Muller, A. Thabet, and B. Ghanem, "Deepgens: Can gens go as deep as cnns?" in ICCV, 2019.
[38] X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifier neural networks," in AISTATS, 2011.
[39] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in ICML, 2015.
[40] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, "Scaled-yolov4: Scaling cross stage partial network," arXiv preprint arXiv:201 / .,8836 2020.
[41] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in CVPR, 2017.
[42] K. Hc, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in CVPR, 2016.
[43] G. Jocher, Y. Kwon, guigarfr, perry0418, J. Veitch-Michaelis. Tlayu, D. Suess, F. Baltaci, G. Bianconi, IlyaOvodov, Mare, e96031413, C. Lee, D. Kendall, Falak, F. Reveriano, Ful.in, GoogleWiki, J. Nataprawira, J. Hu, LinCoce, LukeAI, NanoCode012, NirZarrabi, O. Reda,
11

---

P. Skalski, SergioSanchezMoottsUUMM, S. Song, T. Havlik, and T. M. Shead, "ultralytics/yolox::
v9.5.0 YOLŌv5 v5.0 release compatibility update for YOLOv3," 2021.
[44] Y. Wu, A. Kirillov, F. Massa, W.-Y Lo, and R. Girshick, "Detectron2." 2019.
[45] K. He, R. Girshick, and P. Dollár, "Rethinking imagenet pre- wre-training," in 2019.
12


Reviews:
[{"review_id": "rw8nioLM9q1", "reviewer": "Reviewer_jgFb", "summary": "The paper present a method to perform object detection for vector graphics images directly from their description as order three Bézier curves. It proceeds as follows:\n1. it convert a vector graphics object as a set of order 3 Bézier cuvres\n2. it looks at this set of curves as a graph\n3. it process this graph using a GNN\n4. it aggregates the features of the GNN on proposal regions, generated form the vector description of the image and classify them.\n\nThe method is evaluated on two public datasets and compared to standard CNN baselines on the rendered images.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I don't know of any other paper performing object detection in vector images directly from the vector description. If this is indeed true, I think it gives the paper enough novelty to be accepted despite the weaknesses listed bellow.\n\nWeaknesses:\n1. I would like a clear ablation showing the performance of CNN classifers (ResNet/Fast-RCNN) on the proposal extracted using the method from the paper + the proposed GNN using proposals from another method. Right now, it's unclear how much of the boost is coming from better proposals.\n2. I found the technical part of the paper (part 4) very hard to parse. Notations could likely be improved, showing them on a picture might also help. \n3. Beyond readability, I felt many of the choices were kind of arbitrary, even if several are justified by ablations. Assuming this is the first paper in this area, I think it's ok even if not ideal.\n4. A transformer baseline/comparison would be very natural considering the structure of the data.\n5. Why not perform segmentation instead of detection? To me it would be much more natural considering the data\n6. It would be nice to include a quick literature review on vectorization, e.g. based on the related work of \"Shen, I. C., & Chen, B. Y. (2021). ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis. IEEE Transactions on Visualization and Computer Graphics.\" or \"Parakkat, A. D., Cani, M. P. R., & Singh, K. (2021, May). Color by numbers: Interactive structuring and vectorization of sketch imagery. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11).\"\n\nmisc. \n- l 155 \"an cubic\"\n- l 161-164 especially unclear\n- section 3 seems completely useless to me", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitations: it would be nice to discuss more the influence of the way the vector graphics was generated on the resulting Bézier curve-based representation as well as the errors made in the conversion.\nsocietal impact: ok", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "gyfpqwx4je", "reviewer": "Reviewer_NPkQ", "summary": "In this paper, the authors proposed one framework for the object detection in the vector graphics instead of raster graphics (i.e., pixel images), where the main contribution maybe is to employ the graph neural networks in this area. The extensive experiments on two datasets show the effectiveness of this method, especially comparing with the methods on raster graphics. However, I do not think this comparison is fair, since the proposed methods employ more information contained in the vector graphics while other methods only use pixel information.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work in this paper is for the vector graphics, which is interesting. One problem is that how to utilize the detailed information in the vector graphics. \n\nIn this paper, the authors transforms all the primitives into a unified format, where the Bézier Curve is used, then graph neural networks are employed for the object detection. The experimental results are good, and I think this work is significant for the research on vector graphics.\n\nHowever, some details are better to give:\n\n(1) In the Eq. (4), based on my understanding, there are several points P between the two points of this edge, so the x^e is obtained by concatenating all of these points?\n\n(2) It is better to give more details on how to get the regional cluster C_k, since this plays an important role in the following steps.\n\n(3) In the Eq. (7), are all of the z_i in the same cluster C_k the same? My understanding is that neighbors in the same cluster are the same (i.e., N_i^p) because they are dense connections. Please correct me if my understanding is wrong.\n\n(4) In the table 3b, it is not clear that the aggregation functions (last three rows) are replaced by three popular GNN methods in stroke-wise stream, or position-wise stream, or both. Moreover, why the performance of three popular GNN aggregation methods are reduced largely compared to the proposed two aggregation functions? BTW, in the second part (position-wise stream), I guess the h_j - h_i may be z_j - z_i because there is no variable h in this stream.\n\n(5) In the proposal generation, the authors mentioned that \"permute all vertex pairs on the grid mesh ... rectange region\". does it mean every two nodes in the C_k will get one rectangle region? if it does, there will be a large number of rectangle regions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not find any negative societal impact, and the suggestion are in the box of Main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Igq-beN1FrN", "reviewer": "Reviewer_z9Ww", "summary": "The paper addresses the problem of object recognition, i.e., object localization and classification, in Vector Graphics.\nThe input is a text document that describes the vector graphics, and the output is a bounding box corresponding to a localized object.\n\nThe proposed approach is supervised.\n\nThe main idea is to represent the input textual document describing the Vector Graphic (VG) object using a unified representation for different primitives described in the text document (such as a Bezier curve). The VG objects are represented using a Graph, where the nodes correspond to the start and end points of the Bezier curves.\n\nA dual-stream Graph Neural Network (GNN) is proposed to aggregate feature representation for the input VG object, which is in-turn used for classification and localization.\n\nThe method is evaluated using Average Precision metric. \n\nTwo kinds of publicly available VG datasets are used: Floorplan and Diagrams.\nComparison against object recognition techniques that employ a CNN has been done, and the efficiency\nof dual-stream GNN architecture is demonstrated.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\nI think the paper is addressing an interesting problem of object recognition in Vector Graphics input. The proposed idea, especially extracting the textual data into Bezier curves, is good. I have one major question here:\n1) The VG input is in text format. How do you extract Bezier curves from there? In other words, how do you parse the input textual document to understand what kind of primitives are described in there?\n\nThere needs to be more clarity about the \"Format Unification\" step shown in figure 2.\n\nQuality: \n\nThe proposed method has shown to significantly outperform object recognition methods that employ a CNN.\nSince no CNN is made use of in the proposed method, the model is relatively light-weight and yet, efficient.\nIn terms of writing and mathematical formulation, the paper is reasonably written. \n\nClarity:\n\nThe flow of the paper is easy to follow. I would suggest the following to help improve the exposition of the paper even more:\nState clearly what is the input. I think this is where the readers need to be educated. Let them know what VG format is and how is it described in general. Also describe how such a textual document of VG is parsed into a data structure that is fed to the employed neural network.\nExplain how the proposal generator works. I am still not very clear about the proposal generator. I think Section 4.3 can be broken down in a better way. Perhaps start of with a different paragraph about proposal generators and then describe the training loss.\nAnd finally, be specific about the representation of the output.\n\n\nSignificance:\n\nThe problem being addressed is important. Dependence on raster formats of data is both computationally heavy and resolution-constrained. Computational methods that can operate on VG data and perform visual recognition tasks as that of raster form of data are much needed and perhaps will become dominant in the next few years.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "CvhZj0RoG0w", "reviewer": "Reviewer_qaZw", "summary": "Update at the end of the discussion session: \nDiscussion among reviewers and with the authors made me increase my score by a total of 2 points. \n\nThe paper presents a graph-neural network based approach to detecting objects in vector graphics. The paper works directly on the vector graphics representation by first normalizing the vector graphics to bezier splines and then building a graph representation of the vector graphics. Object detection happens directly on this graph representation. Experiments show a comparison to standard computer vision models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Further updates after more discussion: \n\nincreased score to \"above decision threshold\" assuming: \n - title is changed\n - clarification about baselines not being strong baselines\n\n## Updated review following on discussion with other reviewers and authors: \n\nReview up'd by 2 points because: \npositive:\n - it's an important topic\n - the model is interesting\n\nnegative: \n - experiment baselines are lacking\n - online handwriting / drawing recognition work is relevant\n - the title is misleading\n\nThe paper presents an application of GNNs to detecting objects in vector graphics. \n\n\nREFERENCES AND RELATED WORK\nWhile working on vector graphics is not a very common research domain at the moment, the authors do not refer to the related work that has been looking at vector graphics, e.g. \n\n[1] A Learned Representation for Scalable Vector Graphics https://arxiv.org/abs/1904.02632\n\nfurther, the entire domain of online handwriting and drawing recognition also handles vector graphics, in in particular is often using a very similar representation: \n\n[2] SketchRNN  - https://arxiv.org/abs/1704.03477\n[3] CoSE: Compositional Stroke Embeddings - NeurIPS 2020\n[4] Fast Multi-language LSTM-based Online Handwriting Recognition - is using a Bezier representation - similar to what is proposed here for handling (vector graphics of) handwriting\n[5] BézierSketch: A generative model for scalable vector sketches - ECCV 2020 https://arxiv.org/abs/2007.02190\n\nFurther, in the domain of architectural drawings there was: \n[5] PlanIT: Planning and Instantiating Indoor Scenes with Relation Graph and Spatial Prior Networks - https://dl.acm.org/doi/pdf/10.1145/3306346.3322941\n\n\nBeyond that, I feel that the paper title and representation of the work is misleading: Just looking at the text would in my mind imagine just looking at the raw SVG file - however, as I am reading the paper it becomes clear that there is a lot of specialized preprocessing to normalize the vector graphics and create the graph. So, \"only look at text\" - is clearly not a good representation of the paper. \n\n\nTECHNICAL\nin section 4, is a description of how the graph is build which includes a step in which the input drawing is converted to bezier curves. It is not clear however, how this would work in the case of a very long polyline or e.g. for a circle which are both not easily represented as a Bezier curve. It feels like for this the shape need to be split into multiple bezier splines - but it is unclear from the paper how that is done. \n\n\nEXPERIMENTAL EVALUATION\n - it is not surprising that a computer vision model that was pre-trained on very different data and then fine tuned on ~600 images doesn't perform particularly well on objects that are mostly defined by their outlines rather than by their textures. \n\n\nMINOR ASPECTS\n - lots of abbreviations are badly capitalized throughout the paper, e.g. \"Yolov3\" instead of YOLOv3, \"faster-rcnn\",...\n - similarly the references are badly formatted\n - Figure 2 uses an image of a dog as example - I feel it would have been good to use an image (or object) of the type that is used in the evaluation later for better understanding. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I personally don't think the evaluation in the paper is sufficient to understand limitations. Experiments on two datasets of 1000 images of unknown difficulty don't give a good idea of what the limitations of the method are. ", "review_text": "", "extra_scores": {"ethical_concerns": "no concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have extensively discussed the paper with the authors and came to the conclusion that the paper studies an interesting and important problem. The reviewers have agreed to support the acceptance of the paper if some of the changes are incorporated in the final version. Specifically, the title should be changed to be more precise and less misleading, and there should be more discussion on the feasibility of potential baselines (including the ones coming from handwriting recognition).
Overall, I agree with the reviewers and recommend acceptance of the paper but I would like to encourage the authors to carefully incorporate all the raised concerns and suggestions to improve the paper.
Decision: Accept (Poster)