SketchGen: Generating Constrained CAD Sketches
Authors: Wamiq Reyaz Para
Shariq Farooq Bhat
Paul Guerrero
Tom Kelly
Niloy Mitra
Leonidas Guibas
Peter Wonka
Venue: NeurIPS 2021 Poster (2021)
Keywords: CAD
transformers
generative modelling
layouts

Abstract: Computer-aided design (CAD) is the most widely used modeling approach for technical design. The typical starting point in these designs is 2D sketches which can later be extruded and combined to obtain complex three-dimensional assemblies. Such sketches are typically composed of parametric primitives, such as points, lines, and circular arcs, augmented with geometric constraints linking the primitives, such as coincidence, parallelism, or orthogonality. Sketches can be represented as graphs, with the primitives as nodes and the constraints as edges. Training a model to automatically generate CAD sketches can enable several novel workflows, but is challenging due to the complexity of the graphs and the heterogeneity of the primitives and constraints. In particular, each type of primitive and constraint may require a record of different size and parameter types.
We propose SketchGen as a generative model based on a transformer architecture to address the heterogeneity problem by carefully designing a sequential language for the primitives and constraints that allows distinguishing between different primitive or constraint types and their parameters, while encouraging our model to re-use information across related parameters, encoding shared structure. A particular highlight of our work is the ability to produce primitives linked via constraints that enables the final output to be further regularized via a constraint solver. We evaluate our model by demonstrating constraint prediction for given sets of primitives and full sketch generation from scratch, showing that our approach significantly out performs the state-of-the-art in CAD sketch generation.


SketchGen: Generating Constrained CAD Sketches
Wamiq Reyaz Para'
Shariq Farooq Bhat
Paul Guerrero²
Tom Kelly
Niloy Mitra 2,4
Leonidas Guibas
Peter Wonka'
KAUST
Adobe Research
University of Leeds
University College London
5 Stanford University
Abstract
Computer-aided design (CAD) is the most widely used modeling approach for technical design. The typical starting point in these designs is 2D sketches which can later extruded and combined to obtain complex sirre-dimensional assem- blies. Such sketches typically composed of parametric primitives, such points, lines, and circular arcs, augmented with geometric constraints linking the primitives, such coincidence, parallelism, represented as graphs, with the primitives as nodes and the constraints as edges. Training model automatically generate CAD sketches can enable several novel workflows, but is challenging due to the complexity of the graphs and the hetero- geneity of the primitives and constraints. In particular, each type of primitive and constraint may require record of different size and parameter types. We propose SketchGen generative model based on transformer architecture to address the heterogeneity problem by carefully designing a sequential language for the primitives and constraints that allows distinguishing between different primitive constraint types and their parameters, while encouraging our model to re-use information across related parameters, encoding shared structure. A particular highlight of our work is the ability to produce primitives linked via constraints that enables the final output to be further regularized via a constraint solver. We evaluate our model by demonstrating constraint prediction for given sets of primitives and full sketch generation from scratch, showing that our approach significantly out performs the oatt--of-the-att in CAD sketch generation.
orthogonality. Sketches be
1 Introduction
Computer Aided Design (CAD) tools are popular for modeling in industrial design and engineering. The employed representations are popular due to their compactness, precision, simplicity, and the fact that they are directly "understood" by many fabrication procedures.
CAD models are essentially a collection of primitives (e.g., lines, ares) that are held together by set of relations (e.g., collinear, parallel, tangent). Complex shapes are then formed by a sequence of CAD operations, forming base shape in 2D that is subsequently lifted to 3D using extrusion operations. Thus, at the core, most CAD models are coplanar constrained sketches, i.c., a sequence of planar curves linked by constraints. For example, CAD models in some of the largest open-sourced datasets like Fusion360 [37] and SketchGraphs 27 are thus structured, or sketches drawn in algebraic systems like Cinderella [I] Such sequences not only useful for shape creation, but also facilitate easy editing and design exploration, as relations are preserved when individual elements are edited.
CAD representations are heterogeneous, First, different instructions have their unique sets of parame- ters, with different parameter counts, data types, and semantics. Second, constraint specifications involve links (i.e., pointers) to existing primitives or parts of primitives. Further, the same final shape
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

can be equivalently realized with different sets of primitives and constraints. Thus, CAD models neither come regular structures (e.g. image grids) nor can they be easily encoded using fixed length encoding. The heterogeneous nature of CAD models makes it difficult to adopt existing deep learning frameworks to train generative models. One easy-to-code solution is to simply ignore the constraints and rasterize the primitives into an image. While such a representation is easy to train, the output is either in the image domain, or needs to be converted to primitives via some form of differential rendering. In either case, the constraints and the regularization parametric primitives provide is lost, and the conversion to primitives likely to be unreliable. A slightly better option is to add padding to the primitive and constraint representations, and then treat them fixed-length encodings for each primitive/coontrrint However this only hides the difficulty, as the content of the representation is still heterogeneous, the generator now additionally has to learn the length of the padding that needs to be added, and the increased length is inefficient and becomes unworkable for complex CAD models.
We introduce SketchGen as generative framework for learning a distribution of constrained sketches. Our main observation is that the key to working with heterogeneous representation is well- structured language. We develop a language for sketches that is endowed with a simple syntax, where each sketch is represented as additional guidance for our generative model. Our architecture makes use of transformers to capture long range dependencies and outputs both primitives along with their constraints. We can aggressively quamize/approximaae the continuous primitive parameters in our sequences as long as the inter-primitive constraints can be correctly generated. A sampled graph can then be converted, if desired, to a physical sketch by solving a constrained optimization using traditional solvers, removing errors that were introduced by the quantization.
sequence of tokens. We explicitly use the syntax tree of a sequence
We evaluate SketchGen on one of the largest publicly-availabee constrained sketch datasets Sketch- Graph [27]. We compare with set of existing and contemporary works, and report noticeable improvement in the quality of the generated sketches. We also present ablation studies to evaluate the efficacy of our various design choices.
Related Work
Vector graphics generation. Vector graphics, unlike images, are presented in domain-specific languages (c.g., SVG) and are not in a format easily utilized by standard deep learning setups. Recently, various approaches have been proposed by linking images and vectors using differentiable renderer [19 26 or supervised with vector sequences (e.g., deepSVG 14. SVG- VAE (20), and Cloud2Curve ( DeepSVG, mostly closely related to our work, proposes hierarchical non- autoregressive model for generation, building on command, coordinate, and index embeddings.
Structured model generation. Geometric layouts are often represented as graphs encoding rela- tionships between geometric entities. Motivated by the success in image synthesis, several authors attempted to build on generative adversarial networks 17 22 or variational autoencoder [15]. Re- cently, autoregressive models like the transformer architecture [3]] emerged as an important tool for generative modeling of layouts, e.g. [34| 23 36). Closely related to our work are DeepSVG [ and Polygen [2]]. These solutions are also related modeling with variable length commands, but they simplify the representation so that commands are padded to a fixed length. Polygen is an nuto-regressive generative model for 3D meshes that introduces several influential ideas. One of the proposed ideas, that we also build on, is to employ pointer networks (32) to refer to previously generated objects (i.e., linking vertices form polygonal mesh faces). Unlike PolyGen, we work with heterogeneous primitives and constraints, and with edges that constrain the primitive parameters, motivating our constraint optimization step.
CAD Datasets and CAD generation. Recently, several notable datasets have emerged for CAD models: ABC [13], Fusion360 [37], and SketchGraphs [27]. Only the latter two include constraints and only SketchGraphs introduces a framework for generative modeling. There are several papers that focus on the boundary (surface) representation for CAD model generation without constraints such as UV-Net [9] and BRepNet [14. related topic is to recover (parse) a CAD-like representation
from unstructured input data, e.g. [28 6
30
18
33
35
3 29
40
7 10

---

primitive sequence
cinle
21 Ni
V1 M1 Ui 71 K2
Ci
22 1/2 U2
1/2
b
..
langent
coincident
syntax trees
loc. circle
dir.
loc.
dir.
ran.
line
midpoint
coincident
midpoint
suD
reD rel Sudb
constraint sequence A midppoots jai
d2
A mimicidous
112
Figure 1: A typical CAD sketch consists of primitives such as lines, circles and arcs, and constraints between primitives, shown as floating boxes. We represent them as primitive- and constraint sequences in a language that is endowed with a simple syntax. We show the syntax trees of the first two primitives and constraints. Tokens in the constraint sequence reference primitives (colored token background).
Concurrent work. Multiple papers, namely Computer-Aidest Design as Language [8) Engineering Sketch Generation for Computer-Ae Aidee Design (38), DeepCAD [39], have appeared on arXiv very recently, and should be considered to be concurrently works.
3 Overview
CAD sketches. A CAD sketch can be defined by graph S (P,C), where P is a set of parametric primitives, such as points, lines, and circular arcs and C is set of constraints between the primitives, such as coincidence, parallelism, or orthogonality. See Figure [T] for an example. Some of the primitives only used construction aids and are not part of the final 3D CAD model (shown dashed in Figure [ They can be used to construct more complex constraints; in Figure| [ for example, the center of the circle is aligned with the midpoint of the yellow line, which serves construction aid. Both primitives and constraints in the graph are heterogeneous: different primitives have different numbers and types of parameters and different constraints may reference different number of primitives. Primitives P E P are defined by a type and a tuple of parameters P = (T, KG, p). where is the type of primitive, K is binary variable which indicates if the primitive is construction aid, and is a tuple of parameters with a length dependent on the primitive type. A point, for example, has parameters p = (ar, See Figure 2 for a complete list of primitives and their parameter type. Constraints C E C are defined by a type and a tuple of target primitives C = (v, 7). where is the constraint type and y is tuple of references constraint type. Constraints can target either the entire primitive, or a specific part of the primitive, such as the center of a circle, or the start/end point of line. coincidence constraint, for example, (A1, #1. A2, (2) refers to two primitives Px, and Pxa. while μl, P2 indicate which part of each primitive the constraint targets, if targets the entire primitive.
primitives with a length dependent on the
CAD sketch generation with Transformers. We show how generative model based on the trans- former architecture [31] can be used to generate CAD sketches defined by these graphs. Transformers have proven very successful as generative models in wide range of domains, including geometric layouts 21 36. but require converting data samples into sequences of tokens. The choice of sequence representation is a main factor influencing the performance of the transformer and has to be designed carefully. The main challenge our setting is then find a conversion of our heterogeneous graphs into a suitable sequence of tokens.
To this end, design a language to describe CAD sketches that is endowed with a simple syntax. The syntax imposes constraints on the tokens that can be generated at any given part of the sequence and can help interpreting sequence of tokens. An extreme example is the famous natural language sentence "Will, will Will will Will Will's will?", which is a valid sentence that is easier to interpret given its syntax tree, as shown the inset. natural language processing, syntax is complex and hard to infer automatically from a given sentence, generative models usually only infer it implicitly in a data-driven approach. On the other end of the spectrum of syntactic complexity are geometric problems such as mesh generation 21 where the syntax consists of repeating triples of vertex coordinates or triangle indices that can easily be given explicitly, for example as indices from 1 to 3 that denote which element of the triple given token represents. In our case, the grammar
sentence
VP: varh phrasa NP: soun phrase
proper
peoper
wath
proper prssesiee noun
nome
Will, will Will will Will Will's will?
3

---

is more complex due to the heterogeneous nature of our sketch graphs, but can still be stated explicitly, We show that giving the syntax of a sequence as additional imput to the transformer helps sequence interpretation and increases the performance of our generative model for CAD sketches
We describe our language for CAD sketches in Section 4 In this language, primitives are described first, followed by constraints, We train two separate generative models, one model for primitives that we describe in Section 5.1 and a second model for constraints that we describe in Section 5.2
4 A Language for CAD Sketches
We define a formal language for CAD sketches, where each valid sentence is a sequence of tokens Q= (g), (2, with production rules for primitives on the left and for constraints on the right. Terminal symbols for primitives include {2, r, N; x. y; u, v, a, b) and for constraints (A, V, X, μt Q]. Each terminal symbol denotes variable that holds the numerical value of one token q, in the sequence Q. The symbols A and 52 are special constants; marks the start of a new primitive or constraint, while S marks the end of the primitive constraint sequence. T, V, K, X, and μ were defined in Section 3 and denote the primitive type, constraint type, construction indicator, primitive reference, and part reference type, respectively. The remaining terminal symbols denote specific parameters of primitives, please refer to the supplementary material for a full description.
(g,,?,,.. .... that specifies a CAD sketch. A grammar for our language is defined in Figure 2
Syntax trees. A derivation of a given sequence in our grammar can be represented with synlax tree, where the leafs are the terminal symbols that appear in the sequence, and their ancestors are non-terminal symbols. An example of a sequence for a CAD sketch and its syntax tree are shown in Figure I The syntax tree provides additional information about token sequence that we can use to 1) interpret given token sequence in order to convert it into a sketch, 2) enforce syntactic rules during generation to ensure generated sequences are well-formed, and 3) help our generative model interpret previously generated tokens, thereby improving its performance. Given a syntax tree T, we create two additional sequences Q3 and Q*. These sequences contain the ancestors of each token from two specific levels of the syntax tree. Q contains the ancestors of each token at depth ( (92) ). where ar7 is a function that returns the ancestor of q at level I: Q" = (91). of the syntax tree T, or a filler token if q does not have such an ancestor. Level 3 of the syntax tree contains non-terminal symbols corresponding to primitive or constraint types, such as point, line, or coincident, while level 4 contains parameter types, such as location and direction. The two sequences Q² and Q4 are used alongside Q as additional input to our generative model.
Parsing sketches To parse a sketch into a sequence of tokens that follows our grammar, we iterate through primitives first and then constraints. For each, we create corresponding sequence of tokens using derivations in our grammar, choosing production rules based on the type of primitive/constraintt and filling in the terminal symbols with the parameters of the primitive/conssrrintt. Concatenating the resulting per- per-primitive and per-constraint sequences separated by tokens A gives us the full sequence Q for the sketch. During the primitive and constraint derivations, we also store the parent and grandparent non-terminal symbols of each token, giving us sequences Q² and Q². The primitives and constraints can be sorted by variety of criteria. In our experiments, we use the order in which the primitives were drawn by the designer of the sketch 27 In this order, the most constrained primitives typically occur earlier in the sequence. The constraints are arranged based on prevalence in the dataset, constraints that are used more frequently in the dataset occur earlier.
Models
Following (21), we decompose our graph generation problem into two parts:
p(S)=
p(P) PCIP)
Primitive Constraint Model
Model
Both of these models are trained by teacher forcing with a Cross-Entropy loss. We now describe each of the models.

---

Figure 2: Grammar of the CAD sketch language. Each sentence represents syntactically valid sketch. The full grammar is given in the supplementary material.
S P.C
p = A, P. (A, PJ, Ω
A, C, [A, C}, 12
P point line circle arc
C
coincident parallel equal horizontal vertical midpoint perpendicular tangential v folncidom ref, sub, ref, sub v parallel reef. ref
point line line K. location, direction, range
T pain N2 location
coincident parallel
location 2,
direction u,
range a,
ref X sub 1
primitive generation
constraint generation A
KI
V1
di
by
2/1
by
A
location direccion
range
ref
sub
rref
sub
syntax free
line
Symax Iree
Q'
70
n Figure 3: Sequence generation approach. The sequence Q from 1...np describes the primitives and from np 1 nc the constraints of sketch. We two separate generators for the two sub-sequences (blue for primitives, green for constraints). The sequences Q4 and Q3 describe part of the syntax tree Q and are used as additional input.
5.1 Primitive Generator
Quantization. Most the primitive parameters are continuous and have different value distribu- tions. For example, locations are more likely to occur near the origin and their distribution tapers off outwards, while there is bias towards axis alignment in direction parameters. To account for these differences, we quantize each continuous parameter type (location, direction, runge, radius, rotation) separately. Due to the non-uniform distribution of parameter values in each parameter type, uniform quantization would be wasteful. Instead, we find the quantization bins using k-means clustering of all parameter values given parameter type in the dataset, with 256.
Input encoding. In addition the the three inputs sequences Q. Q3 and Qa described in Section 4 we use a fourth sequence Q' of token indices that provides information about the global position in the sequence. Figure 3 gives an overview of the input sequences and the generation process. We use four different learned embeddings for the input tokens, one for each sequence, that we sum up to obtain an input feature:
fi h= En. Exp Sot
S
where of E Q* and ** are learned dictionaries that are trained together with the generator.
Sequence generation. We use transformer decoder network 33 generator. As autoregres- sive model, it decomposes the joint probability p(Q) of a sequence Q into a product conditional probabilities: p(Q) II, p(gulaen). In case, the probabilities features p<)) = p(qnl fen) where feg denotes the sequence of input features up to (excluding) position Each step applies the network gº to compute the probability distribution over all discrete values for the next token 9;:
conditioned on the input
(
p(q: |(x)) ggcc).
(2)
At training time all input sequences are obtained from the ground truth. At inference time, the sequence Q is sampled from the output probabilities p(g: fei) using nucleus sampling, and sequences Q3 and Q' are constructed on the fly based the generated primitive type as shown in
5

---

Figure 3
In addition to providing guidance to the network, the syntax described in Q and Q* allows us limit generated token values valid range. For example, we correct the generated values for tokens A and Ω to the expected special values if different value has been generated.
5.2 Constraint Generator
The constraint generator implemented as a Pointer Network (32 where each step returns an index into a list of encoded primitives. These indices form the constraint part of the sequence 9>ngr where Up is the number of primitive tokens in Q.
Primitive encoding. We the same quantization for parameters described in the previous section, but use a different set of learned embeddings, one for each primitive terminal token. The feature liद for primitive j is the sum of the embeddings for its tokens:
h2 E,, + Cx, + + + Ey Su, Se, + + E, + Sa, + Eb,
where Tj. Kj..... are the tokens for primitive j. We use a special filler value for tokens that are missing in primitives. We follow the strategy proposed in PolyGen [2]] to further encode the context of each primitive into its feature vector using a transformer encoder:
hi e(hij H"),
(4)
where H" the sequence of all primitive features hi. The transformer encoder is trained jointly with the constraint generator.
Input encoding. In addition the sequence Q, the sequences Q' and Q' inputs, but do not use Q3 as we did not notice an increase in performance when adding it. The final input feature is then:
h4
hig
4CC Saz
IC 5
where M4 is the feature for the primitive with index qs, and eAC g/C are learned dictionaries that are trained together with the constraint generator.
Sequence generation. Similar to the primitive generator, the constraint generator outputs a proba- bility distribution over the values for the current token in each step: p(g: M,,,, <i), conditioned on the input features for the previously generated tokens the constraint sequence, denoted as h>ny<< < Unlike the primitive generator, follow PointerNetworks [32] in computing the probability as dot product between a the output feature of the generator network and the primitive features:
P(g: = =1gganes) = l1. (hynges),
(6)
where nC is the constraint generator. This effectively gives us a probability distribution over indices into our list of primitive embeddings. Constraints may also reference sub-parts of primitives, such as line endpoints, defined by the μ tokens of the constraint (see Figure For tokens, the indices into primitive embeddings are interpreted as IDs for the sub-part; each primitive can have up to sub-parts (see the supplementary for details), At training time all input sequences obtained from the ground truth. At inference time, the sequence Q is sampled from the output probabilities p(s using nucleus sampling, and the sequence Q* is constructed on the fly based on the generated constraint type v, shown in Figure Similar to primitive generation, the syntax in Q' provides additional guidance to the network and allows us to limit generated token values to a valid range.
Results
We evaluate our approach on two main applications. We experiment with generating sketches from scratch and also demonstrate an application that we call auto-constraining sketches, where we infer plausible constraints for existing primitives. We evaluate conditional models in the supplementary.

---

cincle per sketch
line per sketch
coincident per sketch
horizontal per sketch
data
0.2
0.33
1.36
probabilty
ours
0
0
Figure 4: Sketch statistics. We compare the the distribution of circle and line counts per sketch and the distribution of coincident and horizontal constraint counts in generated and data sketches.
Dataset. We train and evaluate on the recent Sketchgraphs dataset [27]. which contains 15 million real-world CAD sketches (licensed without any usage restriction), obtained from OnShape [2], web- based CAD modeling platform. These sketches are impressively diverse, however, simple sketches with few degrees of freedom tend to have many near-identical copies in the dataset. These make up ~84% of the dataset and would bias our results significantly. For this reason, we filter out sketches with less than 6 primitives, leaving with roughly 2.4 million sketches. Additionally sketches with constraint sequences of more than 208 tokens, which are typically over-constrained and constitute < 0.1% of the sketches. We focus the 4 most common primitive types and the most common constraint types (see Figure 2 for a full list), and remove any other primitives and constraints from our sketches. We keep aside random subset of 50k samples validation set and 86k samples as test set.
filter
Experimental Setup. We implemented our models in PyTorch [24], using GPT-2 [25] like Trans- former blocks. For primitive generation, we use 24 blocks, 12 attention heads, an embedding dimension of 528 and batch size of 544. For constraint generation, the encoder has 22 layers and the pointer network 16 layers. Both have 12 attention heads, an embedding dimension of 264 and use batch size of 1536. We use the Adam optimizer [12] with a learning rate of 0.0001. Training was performed for 40 epochs on 8 V100 GPUs for the primitive model and for 80 epochs 8 A100 GPUs for the constraint model. See the supplementary material for more details.
Baselines. We use the sketch generation approach proposed in SketchGraphs [27 as the main baseline, which is based on graph neural network that operates directly on sketch graphs. Due to the recent publication of the SketchGraphs dataset, to our knowledge this still the only established baseline for data-driven CAD Sketch generation. This baseline has two variants: SG-sketch generates full sketches and SG-constraint generates constraints only on given set of primitives. We re-train both variants on dataset. Additionally retrain DeepSVG 4 model the SketchGraphs dataset. Details of the retraining for both can be found in the supplementary material. As lower bound for the generation performance, also include random baseline, where token values in each step are picked with a uniform random distribution over all possible values. Additionally, for constraint generation, ablation, we compare to variants of our own model that do not use the syntax tree, or only part of the syntax tree input.
compare
nuto-constraining method with hand-crafted rules, and as
Table 1: Sketch generation. We compare the quality of our learned distribution over sketches to two baselines the left, and compare three variants of our method using statistics over generated sequences on the right.
(a) Metrics on the test set.
(b) Metries on generated sequences.
NLL. in bits per
method
Esynax
E
Exar
method
sketch
prim.
constr.
(p 1.0)
19.8
0.0058
0.0134 0.0442
0.0192
ours (p 0.9)
18.3
0.0185
0.0627
random
1020.73
70.97
24. 14
SG-sketch
158,90
2,42
DecpSVG
100.26
11.49
ours
88.22
8.60
0.61

---

Figure 5: Examples of generated sketches before (gray background) and after optimization to satisfy the generated constraints. Note that the optimization corrects quantization errors (red arrows point out a few changed details). The right-most example is perturbed testset sketch.
Sketch Generation Metries. We report sketch generation performance using three metrics. The negative log- likelihooood (NLL) of test set sketches in our models (in bits) measures how much the learned distribution of sketches differs from the test set distribution (lower numbers mean better agreement). In addition to this metric of the generator's s performance on the test set, we use two metrics that evaluate the quality of generated sequences. The syntactic error (E.ynaa)) measures the percentage of sequences with invalid syntax if we do not constrain tokens be syntactically valid. Lastly, the statistical errur (Esut), measures how much the statistics of generated sketches differ from the ground truth statistics. We compute Estar based on statistics like the number of point primitives sketch, the distribution of typical line directions, relative postioning errors, each represented as normalized histogram. Esta is then the earth mover's distance [16] (EMD) between the histograms of generated sketches and test set sketches, see the supplementary material for details. We further split up Esta into EP for statistics relating to primitives and EC for constraints.
Sketch Generation Results. Quantitative results are shown in Table [ Metries computed on the test set shown the left. We can see that method leads to the smallest NLL, indicating that the distribution of generated sketches more closely aligns with the test set distribution than both DeepSVG and SG-sketch. Since primitives show the NLL per primitive for that baseline. DeepS VG can only generate primitives, but as we see in the results, still has lower performance than model. All three methods perform far better than the upper bound given by the random baseline.
constructed using constraints in SG-sketch,
do not
On the right, metrics are computed on 15k generated sequences. We compare two different variants of our results, using two different nucleus sampling parameters p. Nucleus sampling clips the tails of the generated distribution, which tend to have lower-quality samples. Thus, without nucleus sampling (p = 1.0) we see an increase in the syntactic error, due to the lower quality samples in the tail, but decrease in the statistical error, since, without the clipped tails, the distribution more accurately resembles the data distribution. We show few of the statistics we used to compute Esta in Figure 4 We can verify that generated distributions closely align with the data distribution. Additional statistics are shown in the supplementary material.
The generated constraints can be used to correct errors in the primitive parameters that may arise, for example, due to quantization. In Figure 5 all sketches except the right-most sketch are examples of generated sketches before and after optimizing to satisfy the generated constraints, using the constraint solver provided by OnShape 2] We can see that our our generated sketches are visually plausible and that the constraint generator finds plausible set of constraints, that correctly closes gaps between adjacent line endpoints, among other misalignments.
Auto-constraining Sketches. Another potential application of our model to predict a plausible set of constraints for given set of primitives, for example to expose only useful degrees of freedom

---

Table 2: constraining sketches. We compare the quality of our learned distribution over constraints to two baselines on the left, and compare three variants of our method using various metries over the generated constraint sequences on the right.
(a) Metrics on the test set.
(b) Metries generated sequences.
NLL.I in bits per
method
Esyana
E +
method
seq.
constr.
ours (p 1.0)
1.56
0.0415
ours 0.9)
1.29
0.0442
random
259.40
24.14
SG conotratt
1.22 19.29
ours
8.28
0.61
Table 3: Ablation on the primitive model.
Table 4: Ablation on the constraint model.
method
NLL/seq.1 NLL/prim. NLL/token]
method
NLI./seq /consstt Nto//tken
ours
79.94
8.600
0.979
ours ours -Q"
8.28
0.610
0.110
ours -Q²
79.96
8.605
0.980
8.47
0.633
0.113
ours Q2" Q² 80.35
8.640
0.984
ours -shared
8.45
0.627
0.112
for editing the sketch, or to correct slight misalignments in the sketch. To evaluate this application, we separately measure the constraint generation performance given a set of primitives, using the metrics described previously.
Quantitative results are shown in Table On the left, we see similar results for constraint generation as for full sketch generation: our learned distribution over constraint sequences is significantly closer to the dataset distribution (lower NI..) than SG-constraint, and both are far from the upper bound given by the random baseline. On the right, we compute constraints for the primitives of 15k previously generated sketches, using two different nucleus sampling parameters p. Similar to the previous section, deactivating nucleus sampling results in increase of the syntactic error, but decrease the statistical error.
Constraints can also be generated for primitives that do not come from the primitive generator. We perturb the primitives in all test set sketches, generate constraints, and optimize the primitives to satisfy the generated constraints using the OnShape optimizer. Since we have ground truth constraints for the test set, we can measure the accuracy of the generated constraints. The average accuracy on the test set is 98.4%. An example is shown in the right-most shape of Figure 5 where the left version (gray background) the perturbed test set shape and the right version the same shape after optimization. Additional results are shown in the supplementary material.
Ablation. We ablate the primitive and constraint models separately. We aim show that our syntax provides prior knowledge about the the otherwise ambiguous structure of sketch sequence that improves the generative performance of our models. Table 3 we show the result removing sequences Q3 and/or Q', which contain information about the sequence syntax, from the input of the primitive generator. Removing only Q3 results in a slight performance degradation, while removing both results in a more significant drop. In Table we that removing Q* also causes significant performance drop in the constraint generator. Additionally, we show the importance of grouping the tokens by parameter type in the shared embeddings of the primitive encoder (see Eq. 3). Mixing tokens with different parameter types in the shared embeddings, denoted as ours -shared, causes a significant drop in performance.
7 Conclusion
In this work, we improve upon the state of the art in CAD sketch generation through the use of transformers coupled with carefully designed sketch language and an explicit use of its syntax. The SketchGen framework enables the full generation of CAD sketches, including primitives and constraints, or auto-constraining existing sketches by augmenting them with generated constraints.
We left a few limitations for future work. First, we chose only the most common types primitives and constraints in the dataset to avoid learning from the long tail of the dataset distribution. We can

---

easily incorporate more types by extending our grammar, and in future work it would be interesting experiment with using more complex and less common types, such as Bezier curves and distance constraints. Second, the autoregressive nature of our model prevents correcting errors in earlier parts of the sequence and we would like to explore backtracking to correct these errors in future work.
We are excited about future research into generating complex parametric geometry and in examining the explicit use of formal languages and generative language models.
8 Broader Impact
There are no foreseeable societal impacts specific to our method. There are societal impacts of generative modeling, machine learning, and deep learning in general that are shared by papers in these areas. The discussion of these broader topics is beyond the scope of this paper.
9 Acknowledgements
This work was suppported in part by ARL grant W911NF2120104, and a Vannevar Bush Faculty Fellowship. We would like to acknowledge gifts from Adobe, Autodesk and the UCL AI Centre. We thank the KAUST Supercomputing Lab (KSL) for providing compute infrastructure. Finally, we thank the anonymous reviewers for their constructive comments.
References
[1] Cinderella, bttps://www.iiddrellll.ccc
Accessed: 2021-05-28.
[2] Onshape. ftttps://www.ossaaee com/ Accessed: 2021-05-28.
[3] Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. Cose: Composi- tional stroke embeddings. In H. Larochelle. M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. volume 33, pages 10042-10052. Curran Associates, Inc., 2020. URI.
.dff
[4] Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hicrarchical generative network for vector graphics animation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16351-16361. Curran Associates, Inc., 2020.
[5] Ayan Das, Yongxin Yang. Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Cloud2eurve: Generation and vectorization of parametric sketches. arXiv preprint arXiv:2103.15536 2021.
[6] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B Tenenbaum. Learning to infer graphies programs from hand-drawn images. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6062-6071, 2018
[7] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar- Write, execute, assess: Program synthesis with a repl. In Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[8] Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer-aide design as language. arXiv preprint arXiv:2105.02766 2021.
[9] Pradeep Kumar Jayaraman, Aditya Sanghi, Joseph Lambourne, Thomas Davies, Hooman Shayani, and Nigel Morris. Uv-net: Learning from curve-networks and solids. arXiv preprint arXiv:2006. 102/1, 2020.
[10] R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. Shapeassembly: Learning generate programs for 3d shape structure synthesis. ACM Transactions Graphics (TOG), 1-0,, 2020.
[11] Kacper Kania, Macicj Ziçba, and Tomasz Kajdanowiez. Uesg-net- desg-net-unstin unsuuprrrvdd discovering of constructive solid geometry tree. arXiv preprint arXiv:2006.09/00 2020.
[12] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
10

---

[13] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev. Mare Alexa, Denis Zorin, and Daniele Panozzo. Abe: A big cad model dataset for geometric deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2019.
[14] Joseph G. Lambourne, Karl D.D. Willis, Pradeep Kumar Jayaraman, Aditya Sanghi, Peter Melizer, and Hooman Shayani. Brepnet: topological message passing system for solid models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B. Le, Haifeng Gong, Ming- Hsuan Yang, and Weilong Yang. Neural design network: Graphic layout generation with constraints. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV 2020, pages 491-506, 2020.
Ming-Hsuan
[16] E. Levina and Bickel. The earth mover's distance is the mallows distance: some insights from statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, 2001.
[17] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. In International Conference Learning Representations, 2018.
[18] Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas J Guibas. Supervised fitting of geometric primitives to 3d point clouds. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2652 2660, 2019.
[19] Tzu-Mao Li, Michal Lukáč, Michael Gharbi, and Jonathan Ragan-Kelley Kelley. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Gruphics (TOG), 39(6):1-1,, 2020.
[20] Raphael Gontijo Lopes, David Ha. Douglas Eck, and Jonathon Shlens. learned representation for scalable vector graphics. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
[21] Charlie Nash, Yaroslav Ganin, SM Eslami, and Peter W Battaglia. Polygen: An autoregressive generative model of 3d meshes. arXiv preprint arXiv:2002. 10880, 2020.
[22] Nelson Nauata, Kai-Hung Chang, Chin-Yi Cheng, Greg Mori, and Yasutaka Furukawa. House-gan: Relational generative adversarial networks for graph-constrained house layout generation. 2020.
[23] Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. arXiv preprint 1.X1v:2011.13417, 2020.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner. Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high high-performance deep learning library. In Advances in Neural Information Processing Systems. 2019.
[25] Alec Radford, Jeff Wu. Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[26] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra. Im2vec: Synthesizing vector graphies without vector supervision. arXiv preprint arXiv:2102.02798. 2021.
[27] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. SketchGraphs: A large-scale dataset for modeling relational geometry in computer-aided design. In ICML 2020 Workshop Object-Oriented Learning, 2020.
[28] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Csgnet: Neural shape parser for constructive solid geometry. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[29] Gopal Sharma, Difan Liu, Subhransu Maji. Evangelos Kalogerakis, Siddhartha Chaudhuri, and Radomir Méch. Parsenet: A parametric surface fitting network for 3d point clouds. In European Conference on Computer Vision, pages 261-276. Springer, 2020.
[30] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu, Learning to infer and execute 3d shape programs. In International Conference on Learning Representations, 2019.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
11

---

[32] Oriol Vinyals. Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, 2015.
[33] Homer Walke, R Kenny Jones, and Daniel Ritchie. Learning to infer shape programs using latent execution self training. arXiv preprint arXiv:2011.13045, 2020.
[34] Kai Wang, Manolis Savva, Angel X. Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions Graphics, 2018.
[35] Xiaogang Wang, Yuelang Xu, Kai Xu, Andrea Tagliasacchi, Bin Zhou, Ali Mahdavi-Amiri, and Hao Zhang. Pie-net: Parametric inference of point cloud edges. In Advances in Neural Information Processing Systems, 2020.
[36] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor generation with transformers. arXiv preprint arXiv:2012.0979; 2020.
[37] Karl D. D. Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, Joseph G. Lambourne, Armando Solar- Lezama, and Wojciech Matusik. Fusion 360 gallery: dataset and environment programmatic cad construction from human design sequences. ACM Transactions on Graphics (TOG), 40(4), 2021.
[38] Karl DD Willis, Pradeep Kumar Jayaraman, Joseph G Lambourne. Hang Chu, Yewen Pu, Engineering sketch generation for computer-aided design. arXiv preprint arXiv:2104.0962) 2021.
[39] Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: deep generative network computer-aided design models. arXiv preprint arXiv:2105.09992 2021.
[40] Xianghao Wenzhe Peng, Chin-Y Yi Cheng, Karl D. Willis, Daniel Ritchie, Inferring cad modeling sequences using zone graphs. In CVPR, 2021.
12


Reviews:
[{"review_id": "xRNN9FWakh", "reviewer": "Reviewer_BbqY", "summary": "This paper proposes a sequential language to convert the CAD sketches to syntax trees. With\nthe help of proposed grammar, each sketch is represented as a sequence of primitives (e.g.\nlines, arcs) and constraints (e.g. collinear, parallel, tangent). The authors propose a generative\nmodel based on the transformer architecture to generate primitives linked by constraints which\ncould be optimized by a constraint solver, provided by Onshape. The proposed method is\nevaluated on two tasks: a) generating sketches from scratch, and b) auto-constraining\nsketches.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*:   \nThe paper is well-written and the logic of the paper is smooth. Figure 1 and Figure 3 clearly\nstate the proposed sequential for representing the sketch and workflow of the generative\nmodel. The sequential language for representing the sketches as syntax trees is a novel way to\nsolve the heterogeneous nature between different sketch graphs. The result shows the statistical\nconsistency between the data sketches and the generated sketches via the proposed generative\nmodel. The generative constraints help obviously correcting the quantization error.\n\n*Weakness*:\n1. Would it be better to include other generative models for sketches as baselines make this paper\nstronger, e.g. Sketch-RNN. In this paper, the baseline is not sufficient.\n2. For your ablation study, I think adding the sequence Q3 and/or Q4 does not make a very\nsignificant difference. This makes me question the effectiveness of using this information.\n3. I think it is necessary to compare with some non-transformer-based methods to show the\neffectiveness of using this architecture in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the proposed sequential language for representing the sketches as syntax trees is\nhelpful for solving the heterogeneous nature between sketch graphs, and the proposed\ngenerative model shows good quantitative results, I think the experiments in this paper are not\nsufficient to show the superiority of the proposed method on CAD sketch generation. The\nablation study also does not show a very significant effect of different components of their\nmethods. ", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "MHqoFIy2WtF", "reviewer": "Reviewer_vzeX", "summary": "The paper proposes a generative model for CAD sketches. The task of modeling CAD format is challenging due to the heterogeneity of the primitives and the constraints. In other words, the CAD representation lacks a common interface. To deal with this problem, the paper presents a domain-specific language enabling tokenizing a CAD sketch. Being able to represent sketch primitives and constraints allows using sequence models such as Transformers.\nThe task of modeling CAD sketches is decomposed into the modeling of primitives and the constraints given the set of primitives. By using the predicted constraints, a post-processing optimization can be used to correct the alignment and positioning issues with the predicted primitives.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe paper proposes a domain-specific language and a task decomposition to model the primitives and the constraints separately. Both approaches have been used in different domains and shown to be effective. The paper successfully combines them for CAD sketches. The presented model, training objectives, and evaluations are not novel, but reasonable choices for this task. Modeling primitives and the relations among them is a common pattern followed in “layout modeling” tasks. In the related work section, the authors make connections to relevant works. Though, the long citation list (Ln. 86) is not useful as the “CAD-like representations” statement is too vague to summarize the listed works. [1, 2, 3] should also be included.\n\n**Quality**\n- The paper follows a symbolic approach in representing the CAD sketches rather than images. It is well motivated. The proposed grammar and the language disambiguate the tokens (i.e., primitive and constraint parameters) and provide semantic information to the model. \n\n- Due to the quantization of the 2D space, the primitive model often makes misaligned predictions. The results are arguably good as the model requires a post-processing optimization to correct the placement of the primitives with respect to the constraints. The constraint model, on the other hand, performs well in determining the constraints between the primitives (98% constraint prediction accuracy). This enables running the optimization with a high success rate. \n\n- The paper presents results on the quantized data only. Training the primitive model on real-valued data could be an interesting baseline. It could alleviate the issues with positioning due to the quantization. For example, in [2], the Transformer model makes real-valued predictions for stroke positions. \n\n- Both for the primitives and the constraints, the paper follows a sequential approach. Is the ordering of the shapes or constraints carry useful information (see [2])? Running the same model with a set assumption would be a useful ablation. \n\n- How are the $Q^3$ and $Q^4$ sequences processed by the model? Consider the example illustrated in Fig. 3. The “line” label starts at index 2 and continues until index 9. Are the line embeddings (i.e., $\\xi_{q_i^3}^3$) the same across these indices?  \n\n- The paper provides an interesting model design in the introduction section (Ln. 44-49). Having it among the baseline models would strengthen the story of the paper. Having the syntax labels as part of the padded representation (see my previous comment regarding $Q^3$ and $Q^4$), I do not see a reason for why this baseline fails. \n\n- Ablations: In Table 3, the model is evaluated with and without $Q^3$ and $Q^4$ input sequences. I do not think the difference is significant. What is the difference between NLLs of 79.94 and 79.96? \n- Qualitative comparison: The paper does not present qualitative results for the baseline SG-sketch. \n- NLL metric for the generated sketches: How is it calculated for the sketches generated from scratch? \n- The paper does not provide a measure for the positioning errors. Decomposing it into prediction (i.e., incorrect bin) and quantization errors (i.e., correct bin prediction) could be a useful metric. \n\n[1] Ellis et al., Learning to Infer Graphics Programs from Hand-Drawn Images \n\n[2] Aksan et al., CoSE: Compositional Stroke Embeddings\n\n[3] Wang et al., Deep Convolutional Priors for Indoor Scene Synthesis\n\nNote: I tried to open the manuscript both in Adobe Reader and Chrome browser. There was no gray background in the figures for me. I interpreted the results with failures as the ones with a gray background.\n\n**Clarity**\nThe paper is well organized. I find the grammar notation cluttered, and hence it is not straightforward to follow section 4. I took several passes. A toy example (like the one in Fig. 1) with numbers along with the symbols could be useful. \n\n**Significance**\nIt is an interesting and rather new problem for the community. The proposed approach is reasonable, yet it is not evaluated thoroughly due to the lack of self-comparisons. As I mentioned under the **Quality** section, more evidence could be provided to support the design choices. The provided ablations are not very informative. \n\n-------------------\n**Update:**\nI thank the authors for their response. After reading the other reviews and the rebuttal, I decided not to chance my rating. The paper presents an interesting problem and a technically solid approach. The submission could be stronger by supporting some of the design choices or the claims with experimental evidence. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work are addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BfdaCaJjJI_", "reviewer": "Reviewer_wERC", "summary": "This paper proposes a deep generative model for widely used computer-aided design (CAD) sketches, which are typically composed of parametric primitives (e.g. line, arc) and geometric constraints (e.g. coincidence, parallelism). The problem is challenging because of the complexity of the sketch graph and the heterogeneous nature of the primitives and constraints. To this end, the authors carefully develop a language for CAD sketches to represent each sketch as a sequence of tokens, and then make use of Transformers network to model the data distribution in an autoregressive manner. Evaluation is carried out on unconditional generation and sketch auto-constraining. The generated CAD sketch can be further optimized via an external solver to meet the generated constraints.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this work is clear to me, as a first attempt on generative modeling for engineering CAD sketches (together with some concurrent works), it leverages the similarity between CAD sketches and natural language processing and develops a language to properly represent both parametric primitives and geometric constraints. \n\nThe technical part looks sound and there are some interesting design choices, such as the three levels of syntax tree (line 157-158) and non-uniform quantization (line 179-180). Also, the post-processing with an external constraint solver further improves the results,  which is not considered in prior works.\n\nMy main concerns are as follows:\n1. Although learning a generative model for CAD sketches of engineering standard is interesting, random generation from scratch seems to have less practical importance. I can hardly think of a situation where random CAD sketch generation would be useful. On the other hand, auto-constraining is a more compelling use case, but is less touch upon in the paper. Besides, generation on conditional inputs (e.g. image, partial sketch) is another way to go. It will be better if the author could show some results on that.\n2. For the comparison part, it would be better to compare with more previous sketch generation works (e.g. DeepSVG), as the SG-sketch method is not a very strong baseline. I understand that DeepSVG does not model the constraints, but it's still possible to purely compare the sketch quality (using their proposed metrics). \n3. All the metrics are in the parametric space. Since the CAD language finally results in a visual 2D sketch, it would be better to include some measurements in the image space, e.g. the image reconstruction error.\n4. Another minor point is that the test set seems small, 86K out of 2.4M (3.6%). It might result in much variance during different training runs.\n\nSome confusing details:\n1. In line 168-169, what exactly does \"the frequency of constraints\" refer to? Is it the frequency in the whole dataset or the frequency in the single sketch.\n2. How is the token \"sub\" represented in the constraint generator? By definition, it should indicate some part of a primitive. I'm a bit confused how exactly it is represented as the pointer seems to refer only an entire primitive.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suggestions as above. Mostly, I hope the comparison to DeepSVG can be included. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5-6 hours"}, "extra_text": {}}, {"review_id": "7T8r_AYWM5Y", "reviewer": "Reviewer_Wjpj", "summary": "The paper proposes a new CAD sketches generation method, which designs a new language for CAD primitives and constraints and proposes a transformer-based CAD sketch generator. Results on sketch generation and auto-constraining sketches show the effectiveness of the proposed method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper decomposes CAD sketched into primitives and constraints and designs a new language for CAD sketches. The grammar is carefully designed.\n\n2. The generative model achieves successful sketch generation from scratch and generating constraints for primitives.\n\n3. The quantitative evaluation and ablation studies are sufficient. Ablation studies validate additional syntax tree information is helpful for both primitive and constraint generation.\n\n4. The paper is well-written, and the figures are clear and helpful for understanding.\n\nWeaknesses:\n\n1. The primitive and constraint types are quite limited and can only generate relatively simple sketches.\n\n2. The design of the generative model is similar to PolyGen[9], primitive generator corresponds to vertex model and constraint generator corresponds to face model. This limits the novelty of the model.\n\nConsidering the complex CAD sketch language design and limited novelty of the generative model, I regard this paper as marginally above the acceptance threshold.\n\n***\nUpdate:\n\nI thank the authors for their response. The response addressed my concerns about the primitive and constraints types, and the difference between the proposed method and PolyGen.\n\nAfter reading the other reviewers' comments and author rebuttals, I remain my original rating. The submission solves an interesting and challenging problem, and proposes a solid method to solve it. I recommend acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations and failure cases in the paper.\n\nThere are not much negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes a generative model for CAD sketches. In particular, the work proposes a language to address the heterogeneity in sketch primitives and constraints, which allows a CAD sketch to be tokenized such that it can be handled using sequence models (such as Transformers) followed by a constraint optimizer. Overall, the reviewers are positive about the paper. The reviewers think the work is well motivated and the paper is clearly written. While several reviewers think the experiments can be strengthened with more comparison, the reviewers generally agree the work is novel and makes reasonable modeling choices, which are valuable to the field. The reviewers raised a set of questions for clarification, which were mostly addressed by the authors in the rebuttal. The authors should further address these points in the revision.

 
Decision: Accept (Poster)