Vitruvion: A Generative Model of Parametric CAD Sketches
Authors: Ari Seff
Wenda Zhou
Nick Richardson
Ryan P Adams
Venue: ICLR 2022 Poster (2022)
Keywords: generative modeling
CAD
transformers
design
geometric constraints

Abstract: Parametric computer-aided design (CAD) tools are the predominant way that engineers specify physical structures, from bicycle pedals to airplanes to printed circuit boards. The key characteristic of parametric CAD is that design intent is encoded not only via geometric primitives, but also by parameterized constraints between the elements. This relational specification can be viewed as the construction of a constraint program, allowing edits to coherently propagate to other parts of the design. Machine learning offers the intriguing possibility of accelerating the design process via generative modeling of these structures, enabling new tools such as autocompletion, constraint inference, and conditional synthesis. In this work, we present such an approach to generative modeling of parametric CAD sketches, which constitute the basic computational building blocks of modern mechanical design. Our model, trained on real-world designs from the SketchGraphs dataset, autoregressively synthesizes sketches as sequences of primitives, with initial coordinates, and constraints that reference back to the sampled primitives. As samples from the model match the constraint graph representation used in standard CAD software, they may be directly imported, solved, and edited according to downstream design tasks. In addition, we condition the model on various contexts, including partial sketches (primers) and images of hand-drawn sketches. Evaluation of the proposed approach demonstrates its ability to synthesize realistic CAD sketches and its potential to aid the mechanical design workflow.

Published as a conference paper at ICLR 2022
VITRUVION: A GENERATIVE MODEL OF PARAMETRIC CAD SKETCHES
Ari Seff', Wenda Zhou Nick Richardson', & Ryan P. Adams! Princeton University, New York University, FFlatioon Institute {aseff, njkrichi raddoonn rpa]@princetonn. edu {wz224 enyu. edu
ABSTRACT
Parametric computer-aided design (CAD) tools are the predominant way that engi- neers specify physical structures, from bicycle pedals to airplanes to printed circuit boards. The key characteristic of parametric CAD is that design intent is encoded not only via geometric primitives, but also by parameterized constraints between the elements. This relational specification a constraint program, allowing edits to coherently propagate to other parts of the design. Machine learning offers the intriguing possibility of accelerating the de- sign process via generative modeling of these structures, enabling new tools such as autocompletion, constraint inference, and conditional synthesis. In this work. we present such an approach to generative modeling of parametric CAD sketches, which constitute the basic computational building blocks of modern mechanical design. Our model, trained on real-world designs from the SketchGraphs dataset, autoregressively synthesizes sketches as sequences of primitives, with initial coor- dinates, and constraints that reference back to the sampled primitives. As samples from the model match the constraint graph representation used in standard CAD software, they may be directly imported, solved, and edited according down- stream design tasks. In addition, we condition the model on various contexts, including partial sketches (primers) and images of hand-drawn sketches. Evalua- tion of the proposed approach demonstrates its ability to synthesize realistic CAD sketches and its potential aid the mechanical design workflow.
be viewed the construction of
INTRODUCTION
Parametric computer-aided design (CAD) tools are the core of the design process in mechanical engineering, aerospace, architecture, and many other disciplines. These tools enable the specifica- tion of two- and three-dimensional structures terized variations on their designs, while also providing a structured representation for downstream tasks such as simulation and manufacturing. In the context of CAD, parametric refers to the domi- nant paradigm for professionals, in which the software tool essentially provides a graphical interface constraint program which can then be solved to provide a geometric con-
way that empowers the user to explore parame-
to the specification figuration. This implicit programmatic workflow means that modifications to the altteration of dimensions, angles, etc. propagate across the construction, even it is an assembly with many hundreds thousands of components.
At an operational level, parametric CAD starts with the specification of two-dimensional geometric representations, universally referred as "sketches" in the CAD community (Shah, 1998; Camba et al., 2016; Choi et al., 2002). A sketch consists of a collection of geometric primitives (e.g., lines, circles), along with a set of constraints that relate these primitives to one another. Examples of sketch constraints include parallelism, tangency, coincidence, and rotational symmetry. Constraints are central the parametric CAD process as they reflect abstract design intent on the part of the
Figure I: CAD sketch generation conditioned on hand drawing. Our model first conditions on a raster image of hand-drawy sketch and generates sketch primitives. Given the primitives, the model generates constraints, and the resulting parametric sketch imported and edited in CAD software, ultimately producing a 3D model.

---

Published as a conference paper at ICLR 2022
Hand drawing
Interred sketch
Editing in CAD software
0IO
Figure 2:
Editing inferred sketches. We take raster images of hand-drawn sketches (left), infer the intended sketches using our anage-conditional primitive model and constraint model (center), then solve the resultant sketches in CAD software, illustrating edit propagation arising from the constraints (right).
user, making it possible for numerical modifications to coherently propagate into other parts of the design. Figure 2, which demonstrates capability made possible by our approach, illustrates how constraints are used to express design intent: the specific dimensions of the final model (shown on the right) can be modified while preserving the geometric relationships between the primitives. These constraints are the basis of our view of parametric CAD as the specification of a program; indeed the actual file formats themselves
indistinguishable from conventional computer code.
This design paradigm, while powerful, is often a challenging and tedious process. An engineer may execute similar sets of operations many dozens of times per design. Furthermore, general motifs may be repeated across related parts, resulting in duplicated effort. Learning to accurately predict such patterns has the potential to mitigate the inefficiencies involved in repetitive manual design. In addition, engineers often begin visualizing design by roughly drawing it by hand (Fig. 1). The automatic and reliable conversion such hand drawings, or similarly noisy inputs such as 3D scans, to parametric, editable models remains a highly sought feature.
In this work, we introduce Vitruvion, a generative model trained to synthesize coherent CAD sketches by autoregressively sampling geometric primitives and constraints (Fig. 3). The model employs self-attention (Vaswani et al., 2017) to flexibly propagate information about the current state of sketch a next-step prediction module. This next-step prediction scheme is iterated until the selection of stop token signals a completed sample. The resultant geometric constraint graph is then handed to a solver that identifies the final configuration of the sketch primitives. By generating sketches via constraint graphs, we allow for automatic edit propagation in standard CAD software. Constraint supervision for the model is provided by the SketchGraphs dataset Seff al. (2020), which pairs millions of real-world sketches with their ground truth geometric constraint graphs.
In addition to evaluating the model via likeliool and distributional statistics, we demonstrate the model's 's ability to aid in three application scenarios when conditioned on specific kinds of con- text. In autocomplete, we first prime the model with an incomplete sketch (e.g., with several primi- tives missing) and query for plausible completion(s). In autoconstrain, the model is conditioned on a set of primitives (subject to position noise), and attempts to infer the intended constraints. We also explore image-conditional synthesis, where the model is first exposed a raster image of hand- drawn sketch. In this case, the model is tasked with inferring both the primitives and constraints corresponding to the sketch depicted in the image. Overall, we find the proposed model capable of synthesizing realistic CAD sketches and exhibits potential to aid the mechanical design workflow.
2 RELATED WORK
When discussing related work. important to note that in computer-aiie aided design, the word "sketch" term of art that refers to this combination of primitives and constraints. In particular, it should
2

---

Published as a conference paper at ICLR 2022
CtrcleCx
Circle(s, Xi ?
Tongent(),
Coincident(C 3.store)
Line(x1, yl, x2, y2)
LineCat, x2, y2)
TongereCh,
EquaL
Primitive Model
Constraint Model
Figure 3: We factorize CAD sketch synthesis into two sequence modeling tasks: primitive generation (left) and constraint generation (right). The constraint model conditions on the present primitives and outputs sequence of constraints, serving as edges in the corresponding geometric constraint graph. A separate solver (standard in CAD software) is used to adjust the primitive coordinates, if necessary.
not be confused with the colloquial use of the word "sketch" as has sometimes appeared in the machine learning literature, as in Ha & Eck (2018) discussed below.
CAD sketch generation The SketchGraphs dataset (Seff et al., 2020) was recently introduced to target the underlying relational structure and construction operations found in mechanical design software. Specifically, the dataset provides ground truth geometric constraint graphs, and thus the original design steps, for millions of real-worke CAD sketches created in Onshape. Initial models are trained in Seff et al. (2020) for constraint graph generation using pesssage-passingg networks, but without any learned attributes for the positions of the primitives; the sketch configuration determined via constraints alone, limiting the sophistication of output samples.
Following SketchGraphs, several works concurrent to ours explore generative modeling of CAD sketches utilizing transformer-bas based architectures (Willis et al., 2021; Ganin et al., 2021; Para al., 2021; Wu et al., 2021). In Willis et al. (2021), modeling is limited to primitives, without any learned specification of geometric constraints; Vitruvion, however, enables edit propagation in standard CAD software by outputting constraints. Similar to our approach, both primitives and constraints modeled in Ganin et al. (2021); Para al. (2021). In contrast to our work, these do condition on hand-drawn sketches nor consider noisy inputs when training on the constraint inference task (which proves crucial for generalization in Section 4.4). Unlike the above works that solely target 2D sketches (including ours), Wu et al. (2021) attempts to model sequences of both 2D and 3D CAD commands, but, similarly to (Willis et al., 1021), does not output sketch constraints.
Geometric program synthesis CAD sketch generation may be viewed via the broader lens of ge- ometric program synthesis. This comprises a practical subset of program synthesis generally con- cerned with inferring set of instructions (a program) for reconstructing geometry from some input specification. Recent examples include inferring TikZ drawing commands from images (Ellis et al., 2018) and constructive solid geometry programs from voxel representations (Sharma et al., 2018). Related to these efforts is the highly-sought ability to accurately reverse engineer a mechanical part given underspecified and noisy observations of (e.g., scans, photos, drawings). One of the con- ditional generation settings we explore in this work is training version of our model that generates parametric sketches when given raster hand drawing. Note that previous work on geometric pro- gram synthesis have not generally considered constraints, which are critical to the CAD application.
Vector graphics generation Generative models have been successfully applied to vector graphics in recent years. Sketch-RNN (Ha & Eck, 2018) is trained on sequences of human drawing strokes to learn to produce vector drawings of generic categories (dog, bus, etc.). Note the common meaning of the word sketch differs slightly from its usage term of art in mechanical design (as it in this work). DVG (Carlier et al., 2020) models SVG icons, demonstrating flexible editing with latent space interpolation. Less recent tracing programs, which convert raster drawings to vector graphics and have been widely available for decades (e.g., Selinger (2003)). None this previous work considers constraints mechanism for specifying design intent as performed in
htttp::/www.onnhaap.e
3

---

Published as a conference paper at ICLR 2022
CAD. The inage-conditional version of Vitruvion also takes input raster image (hand-drawn) and can output vector graphics. Unlike tracing, which attempts to faithfully reproduce an input image closely as possible, our model attempts to infer the sequence of geometric primitives and constraints intended by the user, learning to ignore noise in the input.
Graph-structured modeling This work is also related modeling of graph structures, via our use of geometric constraint graphs. Graphs constitute natural representation of relational structure across variety of domains, including chemistry, social networks, and robotics. In recent years, peessag-ppsssing networks (Gilmer et al., 2017; Duvenaud et al., 2015; Kipf & Welling, 2017), building off of Scarselli et al. (2009), have been extensively applied to generative modeling of graph-structured data, such as molecular graphs (Jin et al., 2018: Liu et al., 2018). These networks propagate information along edges, learning node- or graph-level representations.
While message passing between adjacent nodes can be useful inductive bias, it can be an im- pediment to effective learning when communication between distant nodes is necessary, e.g., in sparse graphs (Kurin al., 2021). The constraint graphs in our work sparse, the constraints (edges) grow at most linearly in the number of primitives (nodes). Recent work leverages the flex- ible self-attention mechanism of transformers (Vaswani al., 2017) to generate graph-structured data without the limitations of fixed, generation (Nash et al., 2020). Our work similarly bypasses edge-specific message passing, using self-attention to train a generative model of geometric constraint graphs representing sketches.
communication paths, such as for mesh
3 METHOD
We aim to model a distribution over parametric CAD sketches. Each sketch is comprised of both sequence of primitives, P. and sequence of constraints, C. We note that these sequences, as opposed to unordered sets, due to the ground truth orderings included in the data format (Seff al., 2020). Primitives and constraints are represented similarly in that both have categorical type (e.g.. Line, Arc, Coincidence) as well as set of additional parameters which determine their place- ment/behavior in a sketch. Each constraint includes reference parameters indicated precisely which primitive(s) (or components thereof) must adhere to it. Rather than serving as static geometry. CAD sketches are intended to be aypamic--utomaticall eters. Likewise, equipping our model with the ability to explicitly constrain generated primitives serves to ensure the preservation various geometric relationships during downstream editing.
updating response to altered design param-
We divide the sketch generation task into three subtasks: primitive generation, constraint generation, and constraint solving. In our approach, the first two tasks utilize learned models, while the last step may be conducted by any standard geometric constraint solver. 2 We find that independently trained models for primitives and constraints allows for a simpler implementation. This setup is similar to that of Nash et al. (2020), where distinct vertex and face models are trained for mesh generation.
The overall sketch generation proceeds as: Po (P, C ctx) Po(C PP ctx)
S solve(P,C)
where:
P a sequence of primitives, including parameters specifying their positions;
* C is sequence of constraints, imposing mandatory relationships between primitives;
S is the resulting sketch, formed by invoking solve routine on the (P,C) pair; and
ctx is an optional context for conditioning, such as an image or prefix (primer).
This factorization assumes C is conditionally independent of the context given P. For example, in an imagc-conditional setting, access to the raster representation of a sketch is assumed to be superfluous for the constraint model given accurate identification of the portrayed primitives and their respective positions in the sketch. Unconditional samples may be generated by providing null context.
3.1 PRIMITIVE MODEL
The primitive model is tasked with autoregressive generation of sequence of geometric primitives. For each sketch, we factorize the distribution as product of successive conditionals,
No
Po
ctx)
II po (P. PP.11 (Pzljs,,cx))
(2)
We use D-Cubed (Siemens PLM) for geometric constraint solving via Onshape's public API.
4

---

Published as a conference paper at ICLR 2022
where Np is the number of primitives in the sequence. Each primitive is represented by tuple of tokens indicating its type and its parameters. The primitive type may be one of four possible shapes (here either arc, circle, line, or point), and the associated parameters include both continuous positional coordinates and an isConst ruct i on Boolean. 3 We use sequence to emphasize that our model treats sketches as constructions arising from step-by-step design process. The model is trained in order to maximize the log-likelihood of 0 with respect to the observed token sequences.
Ordering. Abstractly, the mapping from constructions sequences to final sketch ge- ometries is clearly non-injective; it is typi- cal for many construction sequences to re- sult in the same final geometry. Neverthe- less, empirically, we know that the design routes employed by engineers often admit evident patterns, e.g., greater involvement of earlier "anchor" primitives in constraints (Seff et al., 2020). In our training data from SketchGrrphss, the order of the design steps taken by the original sketch creators is preserved. Thus, we expose the model to this ordering, training our model to autoregressively predict subsequent design steps conditioned on prefixes. By following an explicit representation of this ordering, our model is amenable to applications in which conditioning on prefix of design steps is required, e.g., autocompleting a sketch. This approach also constrains the target distribution such that the model is not required to distribute its capacity among all orderings.
Arc (2i, Wiiih Yuiia 22. 12)
Circle (c. v.
Line (1. (a.g)
Point
Table 1: We convert the native Onshape numerical parame- ters to the above forms for modeling. Subscripts of 1, mid, and 2 indicate start, mid, and endpoints, respectively.
Parameterization. SketchGraphss uses the parameterization provided by Onshape. In order to ac- commodate the interface to the Onshape constraint solver, these are often vverpparameterizationn,, e.g. line segments are defined by six continuous parameters rather than four. We compress the orig- inal parameterizations into minimal canonical parameterizations for modeling (Table 1) and inflate them back to the original encoding prior to constraint solving.
Normalization. The sketches in SketchGraphs exhibit widely varying scales in physical space, from millimeters meters. In addition, because Onshape users are free place sketches anywhere in the coordinate plane, they are often not centered with respect to the origin in the coordinate space. To reduce modeling ambiguity, we modify the primitive coordinates such that each sketch's square bounding box has a width of one meter and is centered at the origin.
Quantization. The sketch primitives are inherently described by continuous parameters such as ar, Cartesian coordinates, radii, etc. As is now common approach in various continuous domains, we quantize these continuous variables and treat them as discrete. This allows for more flexible, yet tractable, modeling of arbitrarily shaped distributions (van den Oord et al., 2016). Following the sketch normalization procedure described in Section 3.1, we apply 6-bit uniform quantization to the primitive parameters. Lossy quantization is tolerable as the constraint model and solver ultimately responsible for the final sketch configuration.
Tokenization. Each parameter (whether categorical or numeric) in the input primitive sequence is represented using a three-tuple comprised of value, ID, and position token. Value tokens play a dual role. First, a small portion of the range for value tokens reserved types. Second, the remainder of the range is treated separately as set of (binned) values to specify the numerical value of some parameter, for example, the radius of a circle primitive. ID tokens specify the type of parameter being described by each value token. Position tokens specify the ordered index of the primitive that the given element belongs to. See Section C for further details on the tokenization procedure.
enumerate the primitive
Embeddings. We use learned embeddings for all tokens. That is, each value, ID, and position token associated with a parameter is independently embedded into fixed-length vector. Then, compute an element-wise sum across each of these three token embedding vectors to produce a single, fixed-length embedding to represent the parameter embedding. The sequence of parameter embeddings are then fed as input to the decoder described below.
Architecture. The unconditional primitive model is based on decoder-only transformer (Vaswani et al., 2017) operating on the flattened primitive sequences. To respect the temporal dependencies of the sequence during parallel training, the transformer decoder leverages standard triangular masking
"Construction or virtual geometry is employed in CAD software to aid in applying constraints to regular geometry. i ssoonat rust on specifies whether the given primitive is to he physically realized (when false) simply serve as reference for constraints (when true).
5

---

Published as a conference paper at ICLR 2022
during self-attention. The last layer of the transformer outputs final representation for each token that then passes through a linear layer to produce a logit for each possible value token. Taking softmax then gives the categorical distribution for the next-token prediction task. We discuss conditional variants in Section 3.3. See the appendix for further details.
3.2 CONSTRAINT MODEL
Given a sequence of primitives, the constraint model is tasked with autoregressive generation of sequence of constraints. As with the primitive model, we factorize the constraint model as a product of successive conditionals.
Ne (C P) = II (C (C)),,)P)
(3)
where Nç is the number of constraints. Each constraint is represented by a tuple of tokens indicat- ing its type and parameters. Like the primitive model, the constraint model is trained in order to maximize the log-likelihood of 0 with respect to the observed token sequences.
Here, we focus our constraint modeling on handling all categorical constraints containing one or two reference parameters. This omits constraints with numerical parameters (such as scale constraints) as well as "hyperedge" constraints that have more than two references.
Ordering. We canonicalize the order of constraints according to the order of the primitives they act upon, similar to Seff al. (2020). That is, constraints are sorted according to their latest member primitive. For each constraint type, we arbitrarily canonicalize the ordering of its tuple of parame- ters. Constraint tuples specify the type of constraint and its reference parameters.
Tokenization. The constraint model employs a similar tokenization scheme to the primitive model in order to obtain a standardized sequence of integers as input. For each parameter, a triple of value, ID, and position tokens indicate the parameter's 's value, what type of parameter it is, and which ordered constraint it belongs to, respectively.
Architecture. The constraint model employs an encoder-decoder transformer architecture which allows it to condition on the input sequence of primitives. A challenge with the constraint model that it must be able to specify references to the primitives that each constraint acts upon, but the number of primitives varies among sketches. To that end, utilize variable-length softmax distributions (via pointer networks (Vinyals et al., 2015)) over the primitives in the sketch to specify the constraint references. This is similar to (Nash al., 2020), but distinct in that constraint model is equipped to reference hierarchical entities (i.e., primitives or sub-primitives)
Embeddings. Two embedding schemes are required for the constraint model. Both the input primi- tive token sequence as well as the target output sequence (constraint tokens) must be embedded. The input primitive tokens are embedded identically to the standalone primitive model, using lookup ta- bles. A transformer encoder then produces a sequence of final representations for the primitive tokens. These vectors serve a dual purpose: conditioning the constraint model and embedding ref- erences in the constraint sequence. The architecture of the encoder is similar to that of the primitive model, except it does not have the output linear layer and softmax. From the tuple of representations for each primitive, segment) and each its sub-primitives (e.g.. a line endpoint) that may be involved in constraints.
extract reference embeddings for both the primitive as a whole (c.g., line
Noise injection. The constraint model must account for potentially imperfect generations from the preceding primitive model. Accordingly, the constraint model is conditioned on primitives whose parameters subjected independent Gaussian noise during training.
3.3 CONTEXT CONDITIONING
Our model may be optionally conditioned on a context. As described below Eq. (I), we directly expose the primitive model the context while the constraint model is only implicitly conditioned on it via the primitive sequence. Here, we consider two specific cases of upplication-relevan context: primers and images of hand-drawn sketches
generation. Here, a primer is sequence of primitives representing the prefix of an incomplete sketch. Conditioning on a primer consists of presenting the corresponding prefix to trained primitive model; the remainder of the primitives are sampled from the model until stop token is sampled, indicating termination of the sequence. This emulates a component of an "auto- complete" application, where CAD software interactively suggests next construction operations.
Primer-conditionn

---

Published as a conference paper at ICLR 2022
Image-conditional generation. In the image-conditional setting, we interested in accurately recovering a parametric sketch from a raster image of a hand-drawn sketch. This setup is inspired by the fact that engineers often draw a design views prior to investing time building it in a CAD program. Accurate inference of parametric CAD) models from images or scans remains highly-sought feature in CAD software, with the potential to dramatically reduce the effort of translating from rough paper scribble to CAD model.
paper or whiteboard from various orthogonal
When conditioning on an image, the primitive model is augmented with an image encoder. We lever- age an architecture similar to a vision transformer (Dosovitskiy et al., 2021). first linearly projecting flattened local image patches to a sequence of initial embeddings, then using a transformer encoder to produce a learned sequence of context embeddings. The primitive decoder cross-attends into the image context embedding. The overall model is trained to maximize the probability of the ground truth primitives portrayed in the image; we condition on sequence of patch embeddings. This is similar to the image-condee conge-conditioning route taken in Nash et al. (2020) for mesh generation, which also uses sequence of context embeddings produced by residual networks (He et al., 2016).
Hand-drawn simulation. We aim to enable generalization of the image-conditional model de- scribed above to hand-drawn sketches, which can potentially support a much wider array of ap- plications than only conditioning on perfect renderings. This requires a noise model to emulate the distortions introduced by imprecise hand drawing. as compared to the precise rendering of a para- metric sketch using software. Two reasonable noise injection targets are the parameters underlying sketch primitives and the raster rendering procedure. Our noise model takes a hybrid approach, sub- jecting sketch primitives to random translations/rotations in sketch space, and augmenting the raster rendering with a Gaussian process model. A full description is provided in Section B.
4 EXPERIMENTS
We evaluate several versions of our model in both primitive generation and constraint generation settings. Quantitative assessment is conducted by measuring negative log-likelihood (NLL) and predictive accuracy on held-out test set as well as via distributional statistics of generated sketches. We also examine the model's 's performance on conditional synthesis tasks.
4.1 TRAINING DATASET
Our models are trained on the SketchGraphs dataset (Seff et al., 2020), which consists of several million CAD sketches collected from Onshape. We use the filtered version, which is restricted to sketches comprised of the four most common primitives (arcs, circles, lines, and points), a maximum of 16 primitives per sketch, and non-empty set of constraints. We further filter out any sketches with fewer than six primitives to eliminate most trivial sketches (e.g., a simple rectangle). We randomly divide the filtered collection of sketches into 92.5% training, 2.5% validation, and 5% testing partition. Training is performed on server with four Nvidia V100 GPUs, and our models take between three and six hours train. See Section E.2 for full details of the training procedure.
Deduplication. As components of larger parts and assemblies, CAD sketches may sometimes be reused across similar designs. Onshape's native document-copying functionality supports this. In addition, similar sketches may result from users of Onshape following tutorials, importing files from popular CAD collections, or by coincidentally designing for the same use case. As a result, a portion of the examples in SketchGraphs may be considered to be duplicates, depending the precise notion of duplicate adopted. For example, two sketches may exhibit visually similar geometry yet differ in their ordering of construction operations, placement in the coordinate plane, constraints.
First, we normalize each sketch via centering and uniform rescaling. The numerical parameters in 64 x 64 grid. Then,
the primitive sequence quantized to six bits, mapping their coordinates for any set of sketches with identical sequences of primitives (including their quantized parameters), we keep only one, resulting in a collection of 1.7 million unique sketches. This procedure removes all sketches that have approximately equivalent geometry with the same ordering of construction operations.
4.2 BASELINES
We evaluate several ablation and conditional variants of our model as well as four baselines: 1) A uniform sampler that selects tokens with equal probability at each time step. 2) An LZMA compres-
7

---

Published as a conference paper at ICLR 2022
SkatchGraphe dataset
Figure 5: Average NLL per primitive posi- tion for unconditional primitive model. In general, the model achieves lower NLL for primitives later in sequence, ambi- guity decreases. However, staircase pat- tem of length four also apparent, due the high prevalence rectangles.
DitsPrim Bits/Sketch
Accuracy
Uniform
33.41
350.2
1.3
Compression
15.34
160.7
Unconditional
Unconditional (perm.)
8.15 (0.39)
85.5 (4.1) 60.3 (1.8S%
Image-conditional
3.50 (0.05)
35.1 (0.5) 82.2 (0.2)%
Table 2: Primitive model evaluation. Nega- tive test log-likelihood in bits/primitive and bits/sketch. Next-step prediction accuracy (per-token avg.) is also provided. Standard deviation across 5 replicates parentheses.
Figure 4: Random examples from SketchGraphs dataset (left) and random samples from our unconditional primitive model (right).
sion baseline (see Section F). 3) The models proposed in Seff et al. (2020) (see Section G). 4) The models proposed in Willis et al. (2021) (see Section H).
4.3 PRIMITIVE MODEL PERFORMANCE
Unconditional model. The unconditional variant of our model is trained to approximate the dis- tribution over primitive sequences without any context. primitive model achieves substantially lower NLL than both the uniform and compression base- lines. We employ nucleus sampling (Holtzman et al., 2020) with cumulative probability parameter of p 0.9 to remove low-probability tokens each step. Fig. 4 displays set of random samples.
shown in Table 2, the unconditional
To assess how the predictive performance of the model depends on the sequence order, we train an unconditional primitive model sequences where the primitives are shuffled, removing the model's exposure to the true ordering. As shown in Table 2, this results in a decline on next-step prediction. Importantly, the erasure of primitive ordering removes useful local hierarchical structure (e.g., high- level constructs like rectangles; see Fig. 5) as well as global temporal structure.
Image-conditional generation. Table displays the performance of the basic image-conditional variant of the primitive model. By taking visual observation of the primitives as input, this model able to place substantially higher probability on the target sequence.
In Table 3, evaluate the performance of several imagc-conditional models on set of human- drawn sketches. These sketches were hand-drawn on a tablet computer in 128x 128-pixee bounding box, where the drawer first eyeballs test set sketch, and then attempts to reproduce it. Likewise, each image is paired up with its ground truth primitive sequence, enabling log-likelihood evaluation.
We three versions of the image-conditional model on the hand drawings, each trained on a differ- ent type of rendering: precise renderings, renderings from the hand-drawn simulator, and renderings with random affine augmentations of the hand-drawn simulator. Both the hand-drawn simulation and augmentations substantially improve performance. Despite never observing hand drawing during training, the model able to effectively interpret these. Figure 6 displays random samples from the primitive model when conditioned on images of hand-drawn sketches.
Primer-condittonal generation. We assess the model's ability to complete primitive sequences when only provided with an incomplete sketch. Here, randomly select subset of test set sketches, deleting 40% of the primitives from the suffix the sequence. The remaining sequence prefix then serves to prime the primitive model.
8

---

Published as a conference paper at ICLR 2022
O
Figure 6: Inage-conditional samples from our primitive model. Raster images of real hand-drawn sketches (top) are input to the primitive model. Four independent samples are then generated for each input. The model makes occasional mistakes, but tends largely recover the depicted sketch with only few samples.
Training regimen
Bits/Prim
Bits/Sketch
Accuracy
Noiscless Teating
Noisy Testing
Precise rendering
22.80
292.3
47.0%
Model
Porplexity
Accuracy
Perplxitt
Accuracy
Hand-drawa sugmentation
10.59
135.5
65.5%
drawn
75.9%
Uniform
4,984
0.6%
4,984
0.6%
SketchGraphs
0.246
-9
0.900
Noiseless training
0.184
Table 3: Image-conditional primitive models evalu- ated on real hand-drawn sketches. NLL and predic- tive accuracy (per-token avg.) metrics are shown for three models trained with different data augmentation schemes.
Noisy training
0.198
90.3%
0.203
90.6%
Table Constraint model evaluation with per-token perplexity and accuracy. A model trained on noiseless data degrades substantially in the presence of noise.
Fig. 7 displays random examples of priming the primitive model with incomplete sketches. Because just over half of the original primitives are the primer, there is wide array of plausible comple- tions. In some cases, despite only six completions for each input, the original sketch is recovered. We envision this type of conditioning as part of an interactive application where the user may query for & completions from CAD program and then select one if it resembles their desired sketch.
4.4 CONSTRAINT MODEL PERFORMANCE
We train and test our constraint model with two different types of input primitive sequences: noise- less primitives and noise-injected primitives. Noise-injection proves to be crucial augmentation in order for the constraint model to generalize to imprecise primitive placements (such as in the image-conditional setting). As shown in Table and Fig. 8, while both versions of the model per- form similarly on noiseless test set, the model trained according to the original primitive locations fails to generalize outside this distribution. In contrast, exposing the model to primitive noise during training substantially improves performance.
Sketch editing. Figure illustrates an end-to-end workflow enabled by our model. Our model is first used to infer primitives and constraints from a hand-drawn sketch. We then show how the resulting solved sketch remains in a coherent state due to edit propagation.
5 CONCLUSION AND FUTURE WORK
In this work, we have introduced a method for modeling and synthesis of parametric CAD sketches, the fundamental building blocks of modern mechanical design. Adapting recent progress in mod- eling irregularly structured data, our model leverages the flexibility of self-attention to propagate sketch information during autoregressive generation. We also demonstrate conditional synthesis capabilities including converting hand- -daaiddg to parametric designs.
Avenues for future work include extending the generative modeling framework capture the more general class of 3D imputs; including modeling 3D construction operations and synthesis conditioned noisy 3D scans voxel representations.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
The goal of this work is to automate a tedious process and enable greater creativity in design, but we acknowledge the potential societal impact that could arise from any elimination of human labor. Moreover, there is the potential risk in high-stakes applications for automatically synthesizing me- chanical systems that do not meet human structural standards. We view this work, however, as being part of a larger human- in- mn-the-loop process which will still involve human expertise for validation and deployment.
REPRODUCIBILITY STATEMENT
In addition to the model descriptions in the main text, we have provided all details of the training procedure (optimizer, learning rate, hyperparameters etc.) in Appendix E. For code and pre-trained models, see https: / /ippppp..... cs https///ssss.....reettteeeeee//e
ACKNOWLEDGMENTS
The authors would like to thank Yaniv Ovadia and Jeffrey Cheng for early discussions of this project. Thanks to all members of the Princeton Laboratory for Intelligent Probabilistic Systems for provid- ing valuable feedback. Additionally, we would like to thank Onshape for the API access as well as the many Onshape users who created the CAD sketches comprising the SketchGraphs training data. This work was partially funded by the DataX Program at Princeton University through support from the Schmidt Futures Foundation and by the NSF Institute for Data-Driva Driven Dynamical Design (NSF 2118201).
REFERENCES
Jorge D Camba, Manuel Contero, and Pedro Company. Parametric CAD modeling: An analysis of strategies for design reusability. Computer-Aided Design, 74:18-31, 2016.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. DeepSVG: A hierarchical generative network for vector graphics animation. arXiv preprint wiXiv:2007.11300 2020.
Guk-Heon Choi, Du-Hwan Mun, and Soon- Hunn Han. Exchange of CAD part models based on the macro-parametric approach. International Journal of CAD/CAM, 2(1):13-21, 2002.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Hirzel, Alan Aspuru- Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learn- ing molecular fingerprints. Advances in Neural Information Processing Systems, 2015.
Timothy
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning infer graph- ics programs from hand-drawn images, In Advances in Neural Information Processing Systems, 2018.
Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer-aided design as language. In Advances in Neural Information Processing Systems, 2021.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, 2017.
David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), European Conference Computer Vision 2016.
10

---

Published as a conference paper at ICLR 2022
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. International Conference on Machine Learning, 2018.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Bochmer, and Shimon Whiteson. My body is a cage: the role of morphology graph-based incompatible control. In International Conference Learning Representations, 2021.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph varia- tional autoencoders for molecule design. In Advances in Neural Information Processing Systems, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representutions, 2019, 2019.
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. PolyGen: An autoregressive generative model of 3D meshes. International Conference on Machine Learning, 2020.
Wamiq Reyaz Para, Shariq Farooq Bhat, Paul Guerrero, Tom Kelly, Niloy J. Mitra, Leonidas J. Guibas, and Peter Wonka. Sketchgen: Generating constrained CAD sketches. In Advances in Neural Information Processing Systems, 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE Transactions on Neural Networks, 20:61-80, 2009.
Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. SketchGraphs: A large-scale dataset for modeling relational geometry in computer-aided design. In ICML 2020 Workshop Object- Oriented Learning, 2020.
Peter Sclinger. Potrace: polygon polsgon-based based tracing algorithm. In In http://potraceesourrcffoo..... 2003.
Jamil J Shah. Designing with parametric CAD: Classification and comparison of construction tech- niques. In International Workshop Geometric Modelling, pp. 53-68. Springer, 1998.
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. CSGNet: Neural shape parser for constructive solid geometry. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Leslie Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. arXiv preprint, 2018.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. WaveNet: A generative model for raw audio. In Arxiv, 2016.
A Vaswani, N Shazeer, N Parmar, J Uszkoreit, I. Jones, AN Gomez, I. Kaiser, and Polosukhin. Attention is all you need. In Advances in Newral Information Processing Systems, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Neural Information Processing Systems, pp. 2692-2700, 2015.
Karl D. D. Willis, Pradeep Kumar Jayaraman, Joseph G. Lambourne, Hang Chu, and Yewen Pu. Engineering sketch generation for computer-aided design. In CVPR 202/ Workshop Sketch- Oriented Deep Learning, 2021.
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer- aided design models. In Proceedings of the IEÉE/CVF International Conference Computer Vision (ICCV), 2021.
11

---

Published as a conference paper at ICLR 2022
Appendices
A ADDITIONAL EVALUATION
Original
Primer
Inferred
K
N
N
Figure 7: Primer-conditional generation. We take held-out sketches (left column) and delete 40% of their primitives, with the remaining prefix of primitives (second column from left) serving a primer for the primi- tive model. right are inferred completions, where the model queried for additional primitives until selecting the stop token. Note that only the primitive model is used in these examples.
Constraint model (trained w/out noise)
Constraint model (trained w/ noise)
Original
Perturbed
b
bb
b
Figure 8: Constraining primitives. We take test set sketches (leftmost column) and perturb the coordinates of the primitives with Gaussian noise (second from left). The constraint model trained without noise (second from right) often fails to correct the primitive locations, in constrast to the constraint model trained noise- augmented data (rightmost column).
A.1 DISTRIBUTIONAL STATISTICS
We quantitatively compare distributional statistics for generated sketches and those from the test set. 10K sketches are sampled from our full unconditional model (both primitives and constraints). In particular, compare the number of generated primitives, constraints, total degrees of freedom, degrees of freedom removed by constraints and net degrees freedom across generated sketches in Fig. 9.
12

---

Published as a conference paper at ICLR 2022
0.2
4.12
0.88
5.30
1.59
0325
0.86
Dul
1,50
0.055
0.00
25
Figure 9: Comparison of various statistics between between generated and dataset sketches. Shaded regions represent bootstrapped 95% confidence intervals.
A.2 ABLATION: TEST-TINE ORDERING SHUFFLING
Bits/Prim
Bits/Sketch
Accuracy
Uniform
33.41
350.2
1.3
%
Compression
15.34
160.7
%
Unconditional
6.35 (0.01)
66.6 (0.1)
71.7 (0.0)%
Unconditional (perm.)
8.15 (0.39)
85.5 (4.1)
60.3 (1.8)%
Unconditional (perm. test)
9.98 (0.03)
104.7 (0.3)
55.5 (0.2)%
Image-conditional
3.50 (0.05)
35.1 (0.5)
82.2 (0.2)%
Table 5: Additional evaluations of the primitive model. Unconditional (perm, test) denotes the
primitive model trained standard data but evaluated randomly permuted dataset.
We conduct additional ablation in order to better complement understanding of the depen- dency of the primitive model on the sequence order Table 5. We consider the standard uncondi- tional model, but evaluated on randomly permuted primitive sequences. We observe that this leads to a performance which is significantly worse (NLL of 104.7 bits/sketch compared to 66.6 bits/sketch on the original test set), underlining the fact that our model learns significant information about the primitive sequence order.
A.3 TYPICAL FAILURE CASES
For the hand-drawn conditional primitive model, a typical failure case involves non-axis aligned line segments (e.g., columns and 8 of Fig. 6), which are less prevalent in the data than axis-aligned ones. Data augmentation tailored to this scenario may be one route to alleviate this. For the constraint model, one typical failure case is when the model selects an incorrect reference, particularly when deployed on noise-injected primitives. We found using a tighter nucleus sampling parameter (0.7) to mitigate this substantially. We imagine model serving part of an interactive application where a user can select from a handful of output sketches, mitigating the effect an occasional incorrect prediction.
13

---

Published as a conference paper at ICLR 2022
B HAND-DRAWN NOISE MODEL
The noise model renders lines drawn from zero-mean Gaussian process with a Matérn-377 kernel that have been rotated into the correct orientation. Ares are rendered as Matérn Gaussian process paths in polar coordinates with angle as the input and the random function modulating the radius. No additional observation noise is introduced beyond the standard "jitter" term. The length-scale and amplitude are chosen, and the Gaussian process truncated, so that scales are consistent regardless of the length of the line arc.
C SYSTEM DETAILS
Value Token
Constraint Type
0
Pad
1
Start
2
Stop
Value Token
Primitive Type
3
Coincident
4
Concentric
Pad
5
Equal
Start
6
Fix
2
Stop
7
Horizontal
3
Arc
8
Midpoint
4
Circle
9
Normal
Line
10
Offset
Point
11
Parallel
12
Perpendicular
13
Quadrant
14
Tangent
15
Vertical
Table 6: Mapping for both primitive types (left) and constraint types (right) to the reserved set of value tokens for each model. The tokens 0, 1, and 2 are used in both models for padding. start, and stop, respectively.
Primitive and constraint types are represented using a pre-specified range of each model's value tokens. For example, for primitives, described in the Tokenization paragraph of Section 3.1, small set of the value tokens (four tokens for Arc, Line, Circle, and Point) are reserved for type specification. Three value tokens are also reserved for Start, Stop, and Pad.
32 bins
32 bins
-0.5
0.0 Normalized continuous value
0.5
Figure 10: Utilization of quantization bins by the different numerical parameters. I and coordinates may occupy the full range of bins (representing both negative and positive values) but the radius parameter only utilizes half of the bins (32 in our casc) since must be positive. This means the numerical interval represented by each bin invariant to which parameter is being considered.
14

---

Published as a conference paper at ICLR 2022
Arc
Circle
Line
Point
Parameterization
yt
Yerid
y2
Example numerical values
Value tokens
49
28
ID takens
(Non-numerical ID is
Figure 11: Mapping of primitive parameters to value and ID tokens. Example raw numerical values and the resulting value token that gets assigned to it shown for cach primitive parameter.
Stort
L
X2
Y1
X2
Y/2
y
X1
y1
X2
Y2
Stop
Podd..
Value
1
5
19
69
5
18
49
40
@...
ID
1
1
11
12
14
O...
Position
2
An example primitive sequence (consisting of a line, a point, and another line) and its associated sequences of value. ID, and position tokens.
Figure 12:
Constraint model output. The constraint model's output consists of both categorical types and references to primitives. At any given step during inference, the output dot product is computed between the decoder's output representation constraint types and the (sub-)primitive encodings. That is, the softmax output layer of the constraint model selects from the union both categoricals (for the constraint types) and the references. We may view this still as producing pointer, but the possible targets for the pointer not only the primitive references, but the set of constraint types as well.
concatenation of the learned embeddings for
Primitive and sub-primitive referencing. The constraint model must be able reference both individual primitives or subcomponents thereof (sub-primitives, consisting of specific points that may be the targets of constraints). We took a simple approach where, for each primitive type, a subset of the tuple of learned encodings (as output by the primitive model) are tasked with representing example, for a line, which consists of
either the entire primitive or a specific sub-primitive. tuple of length (type Line, x0, yO, x1, y1), we use the encoding at index 0 to represent the line segment as whole, index (the x coordinate of the first endpoint) represent the first endpoint, and index 3 (the coordinate of the second endpoint) to represent the second endpoint. We note that, after several layers of transformations in the primitive model, the representations at these indices do not solely represent the original token but instead learn to incorporate the information needed to represent the (sub-) primitive they were assigned. Note that this mapping arbitrary and there are other design routes that could produce equivalent results.
15

---

Published as a conference paper at ICLR 2022
For completeness, list the reference indices for each primitive type below where the index indi- cates which element of the primitive's tokenization is given a specific representation responsibility:
Arc: index 0 (whole arc), index (first endpoint), index (centerpoint), index (second endpoint)
Circe: index (whole circle), index (centerpoint)
Line: index (whole line segment), index (first endpoint), index 3 (second endpoint)
Point: index (whole point)
D QUANTIZATION STUDY
Quantization Bitwidth
2
3
4
7
Original
Reconstruction of SketchGraphs sketches at various quantization bit widths. Bit widths from bits (4 unique parameter values, leftmost column) to the original single-precision floating-poin point representation (rightmost column) illustrated above. The quantization used in model is 6 bits.
Figure 13:
E EXPERIMENTAL DETAILS
This section provides further details of the model architecture and training regime.
E.1 MODEL ARCHITECTURE
Our models all share main transformer responsible for processing the sequence primitives or constraints which is the target of the inference problem. This transformer architecture is identical across all of the models, using a standard transformer decoder with 12 layers, 8 attention heads, and performed.
total embedding dimension of 256. No significant hyper-parameter optimization The raw primitive generative model only equipped with the decoder on the primitive sequence, and trained in autoregressive fashion. The constraint model, in addition the transformer decoder
16

---

Published as a conference paper at ICLR 2022
on the constraint sequence, also performs standard cross-attention into an encoded representation of the primitive sequence.
The tagge-to-rrimitive model is additionally equipped with image encoder based vision transformer (Dosovitskiy et al., 2021), which is tasked with producing a sequence of image rep- resentations into which the primitive decoder performs cross-attention. We 128 128-sized images as input to the model. Non-overlapping square patches of size 16 x 16 are extracted from an input image and flattened, producing sequence 64 flattened patches. Each then undergoes linear transformation to the model's embedding dimension (in this case, 256) before entering standard transformer encoder. This transformer encoder has the same size as the decoder used in our other models.
E.2 TRAINING
Training was performed on cluster servers equipped with four Nvidia V100 GPUs. Total training time just under 2 hours for the primitive model, and 5.5 hours for the image-to-primitie model and the constraint model. The primitive model and the constraint model were each trained for 30 epochs, and the image-to-primitive model was trained for 40 epochs, although the total number of epochs did not appear to cause significant differences in performance. The models were trained using the Adam optimizer with "decoupled weight decay regularization" (Loshchilov & Hutter, 2019), with the learning rate set according to a "one-cycle" learning rate schedule (Smith & Topin, 3e-5 (at reference batch size 128, scaled linearly
2018). The initial initial learning rate was with the total batch size). The batch size was set according to the memory usage of the different models at 1024 / GPU for the raw primitive model, 512 GPU for the image-to-primitive model, and 384 / GPU for the constraint model.
E.3 IMAGE-TO-PRIMITVV DATA
Prior to training the image-to- toge-too-rimiitve model, we generate samples from our hand-drawn noise model (see Section B) in separate process. Five samples generated for each sketch in the dataset, taking a total of approximately 160 CPU-hours. This process is parallelized on a computing cluster. During each training epoch, one random image (out of the five generated samples) is selected to be used by the model. An additional data augmentation process is used during training, whereby chosen randomly
random affine transformations are applied to the images. The augmentation among all affine transformations which translate the image by at most 8 pixels, rotate by at most 10 degrees, shear the image by at most 10 degrees, and scale the image by at most 20%.
F COMPRESSION BASELINE
The compression baseline is computed by first tokenizing primitives according the tokenization scheme described in section 3.1, representing the token sequences as 6-bit integers, and compressing the given data using the LZMA compression algorithm. The compression is performed over the entire dataset (including training, validation and test splits), and the reported quantities are averages over the entire dataset. In particular, note that we may expect the entropy rate of the data be lower than reported.
G SKETCHGRAPHS BASELINE
We adapt the models proposed Seff et al. (2020) as a baseline comparison for some of the tasks studied in this paper. Namely, we evaluate the proposed autoconstrain model for the constraint inference task (conditional on primitives), and the proposed generative model on joint unconditional modelling of the primitives and constraints. We do attempt adapt these baseline models the image-comee conditiona or primer-condee crimer-conditional generation tasks also studied in the current work. In order ensure fair comparison, we adapt the SketchGraphs models operate on the featurization used in the current work, rather than the one proposed in Seff et al. (2020), as described in Sections 3.1 and 3.2.
G. GENERATIVE MODEL
We consider autoregressive graph neural network as raw generative model on the joint sequence of primitives and constraints as described in (Seff et al., 2020, Section 4.2). We note that by its nature, the graph neural network operates jointly on the primitives (nodes) and constraints (edges),
17

---

Published as a conference paper at ICLR 2022
and thus corresponds to different factorization of the distribution of the sketch the one proposed in the current work.
As the SketchGraphs model does not incorporate continuous primitive parameters such as position, we make some adaptations in order to model the same data as in the current work. The continuous parameters for primitives are input according the procedure described in (Seff et al., 2020, Ap- pendix D.2), using the quantization scheme in the current work. They are generated according to the scheme described for constraints in (Seff al., 2020, Appendix E.1). In addition, we filter out constraints with continuous parameters as these are not modelled by the current work.
We obtain an average log-likelihook of 124.0 bits per sketch for the SketchGraphs generative model. By combining the primitive model and the constraint model presented in this paper (Vitruvion), we obtain instead an average log-licelhhoo of 85.3 bits per sketch (66.6 bits per sketch for the primitives, and 18.7 bits per sketch for the constraints conditioned on the primitives).
G.2 CONSTRAINT MODEL
We adapt the "autoconstrain" model proposed in Seff et al. (2020) and compare it to our primitive- conditional constraint model. We also note here that the model operates on somewhat different factorization of the sequence of constraints, and thus we may only compare log-likelihood at the sketch level (i.e., for inferring all constraints in a sketch, see Table 4). For presentation purposes, we have rescaled that value to an equivalent per-token value, although the SketchGraphs model does not use the same tokenization, and hence no comparable accuracy metric can be computed.
H QUALITATIVE BASELINES
Seff et al. (2020). Samples from the SketchGraphs generative model (Seff al., 2020) may be viewed in Figure 11 from their work. Because this model does not learn to output primitive coordi- nates, solely relies on the constraint graph and Onshape solver to determine the final configuration. However, solving geometric constraints under poor coordinate initialization is difficult, and likewise the solver often fails for their model, limiting the sophistication of sketches that are generated.
Figure 5 from Seff et al. (2020) demonstrates autoconstraining and then editing sketch. This is very similar to our own Fig. 2. It is difficult to draw any conclusions about the qualitative performance as only one example is shown in their paper. They do include a few dimensional constraints (e.g.. distance constraint) in their model's 's targets, while in our work we model all categorical constraints.
Willis et al. (2021). Two models are introduced in Willis et al. (2021), CurveGen and TurtleGen. Both solely model primitives without outputting any constraints. CurveGen, has a two-stage fac- torization, like unconditional Vitruvion, except both stages are used for building primitives. first outputs vertex locations, and then a second model groups these vertices into primitives (line etc.). TurtleGen instead produces sketches as single sequence of pen drawing commands (within graph- ics program) in order to form closed loops. Both models utilize architectures based on transformers, similar to ours.
Figures 5, 8, and 9 from Willis et al. (2021) display some samples from their two models. Subjec- tively, the TurtleGinn samples tend to appear similar to the SketchGraphs model samples mentioned above, but with a bit more variety. CurveGen samples appear much more sophisticated, exhibiting the kinds of symmetries found in the SketchGraphs datasct. However, there are some sketches that exhibit a lack of coherence, where points that would otherwise be coincident are not (e.g., Fig. 5, second to last row, Fig. 9 second row). We note that Willis et al. (2021) ) does not model construction primitives while we do (they appear as dashed lines in our renderings), direct visual comparison is slightly muddled by that.
Willis et al. (2021) showcases how the auto-constrain tool of AutoCAD can apply parallel and perpendicular constraints to their output sketches in Figure 7. While this is useful tool, it requires manually setting the priority and tolerance for each individual constraint type. Instead, in our work, we propose a learned model that can adapt to the context of a given design automatically without the user toggling these settings.
None of the models above target condittooni generation, which we explore in our work (in- cluding conditioning on drannds.
18


Reviews:
[{"review_id": "mdIwJHbSxs", "reviewer": "Reviewer_rqPX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is somehow natural that after a release of an exciting new dataset multiple groups are trying to target core questions that can be tackled due to its release. There is already important work in this area, i.e. Ganin et al. (2021) and Para et al. (2021), that has substantial overlap with this submission. According to the rules of ICLR, I believe this previous work should count as independent publication. The authors also position their work as concurrently developed. \n\nOverall, a suitable engineering choice is to model sketches as sequences of tokens and to employ multiple transformers for sequence modeling. A strong inspiration for this (as well as the other papers) was PolyGen from Nash 2020 which is appropriately cited multiple times in the paper. Working with transformers to model sequences is difficult work and it typically takes quite a bit of skill and work to achieve good results, such as the ones shown in the paper. Working with such large datasets is also difficult in general.\n\nThe paper is generally very well written (language wise) and nicely illustrated. My main request for improvement is to add details in the writing. I would be interested to know more details of\n*) the architecture and training\n*) the data formats for sequence modeling (input as well as output)\nThere is a promise to release code, which should help clarify some details in the future. However, I would strongly request that these details are included in the paper. For example, the paper proposes to use references to primitives and sub-primitives. It is unclear how exactly this is done. The tokenization in 3.1 and 3.2 could be explained better with examples and the precise number of bits and precise number of choices that are being encoded. It seems that Ganin et al. and Para et al. have more content devloted to clarify details, but this submission does not have a corresponding appendix with details (e.g. appendix A and B in Ganin et al.). For example xCenter is an ID token, but what are all the other ones? What are the lookup table options for ID, Value, and Position (mainly ID and value are unclear). How many lookup tables (quantization tables) are there? Are they context dependent? There are multiple choices how exactly to do the encoding.\n\n6-bit quantization appears a bit low and I wonder if that is really a suitable choice to achieve very good results. Maybe there could be an ablation study. Intuitively, I really doubt that 6-bit is enough to encode details. I do not see how the constraints can fix that large loss of information.\n\nThe paper does provide a list of reasonable results, but at the same time there could be more comparisons and ablation studies to put the work into context. The results feel maybe a bit less extensive than the other two papers. It would also be informative if the paper could add more comparison to the previous papers to be helpful to future researchers in the field (even though they should not be required to do so). Running other code is probably tricky, but writing a table or a list comparing some of the design decisions in the text should be feasible.\n\nThe work does a good job conditioning on images, a result that is not present in Ganin et al. and Para et al. and a unique feature of this work.", "summary_of_the_paper": "The work builds on the SketchGraphs dataset that contains CAD sketches, consisting of primitives and constraints. This dataset opened many opportunities to tackle research problems that could not be easily be tackled before. One important question is how to define a generative model that can unconditionally sample from the distribution. Another important question is how to sample conditioned on some input, e.g. a hand drawn sketch. This paper looks at both of these questions.\n", "summary_of_the_review": "Assuming the authors are willing to add (a lot) more details, I would suggest that this paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HX5a30JLKyZ", "reviewer": "Reviewer_9htv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths** \n- The paper is well written, easy to follow. Section 2 provides a clear positioning of the work against prior efforts on similar problems. \n- The method itself is well motivated, the proposed architecture makes sense and is described in sufficient details. \n- The qualitative and quantitative evaluations suggest that the model performs well on the task.\n\n**Weaknesses**\n- I would have appreciated additional visual results on the task of inferring constraints over existing sketches (Fig.8) and on the task of generating CAD sketches from hand-drawn sketches (Fig.6). While the few results shown are very encouraging, I would have liked a discussion of typical failure cases (since I suspect some failure on such ambiguous tasks).\n- My main concern about this paper is that it appears very similar to [Ganin et al. 2021, Para et al. 2021], which will both appear at Neurips. Checking the ICLR reviewer instructions, I see that these papers are indeed considered concurrent (*if a paper was published (i.e., at a peer-reviewed venue) on or after June 5, 2021, authors are not required to compare their own work to that paper.*), which I interpret as not being a reason for rejecting this submission. \nStill, from a reader perspective, I would have liked a detailed discussion of [Ganin et al. 2021, Para et al. 2021] to understand if they are indeed very similar (both target the same problem, both rely on a Transformer+PointerNetwork architecture), or if the proposed model introduces significant differences. The submission mentions that these two concurrent papers do not attempt to train/test on hand-drawn / noisy input, but this seems more of a detail in the way the method is used than a fundamental difference in the actual model.\n", "summary_of_the_paper": "This paper presents a model capable of generating CAD sketches with constraints. Such sketches are typically formed as a sequence of 2D primitives (points, lines, arcs, circles) related by geometric constraints (parallelism, orthogonality, coincidence, etc). The model generates these sequences in an auto-regressive fashion, one primitive at a time, and then one constraint at a time. The model is trained on a large dataset of CAD sketches [Seff et al. 2020] that contains the necessary ordered sequences of CAD primitives and constraints.\n\nThe model is implemented as two Transformers, one to generate the sequence of primitive, and one to generate the sequence of constraints conditioned on the primitives. For the second part, a PointerNetwork mechanism is used to identify the primitives referenced by each constraint. \n\nThe model is used either to generate random CAD sketches that exhibit similar statistics as the training data (as measured by the number of primitives and the number of constraints used), or to auto-complete a partial CAD sketch by predicting missing primitives and constraints. In addition, an image-conditioned version of the model is also described, which can take as input an approximate bitmap drawing and turn it into a constrained CAD sketch.", "summary_of_the_review": "If the similarity to the concurrent work of Ganin et al. 2021 and Para et al. 2021 should not be accounted for, then I am in favor of acceptance because the described model is sound and effective. But I remain uncomfortable accepting a paper that shares so many similarities with papers that have already been reviewed and accepted, even if shortly before the deadline. I suspect that the authors have been inspired by these concurrent works, maybe in the development of their method, its description, its evaluation, which is why I have a hard time ignoring them in my evaluation.\n\nTo be clear, if [Ganin et al. 2021] and [Para et al. 2021] are to be ignored, then the technical novelty is of level 4, but if they should be considered, then the technical novelty drops to 1 or 2 and I would argue for rejection.\n\n--- Additional questions ---\n\nSince the constraint model is trained separately from the primitive model, is it trained with ground-truth primitive sequences from Seff et al., or from the (approximate) sequences predicted by the primitive model? Section 3.2 mentions that the constraint model must account for potentially imperfect generation from the primitive model, but I didn’t fully understand how this is achieved. Is it only achieved by perturbing the primitive parameters with Gaussian noise? What if the predicted primitive type is wrong, wouldn’t such error degrade all subsequent predictions?\n\nI appreciate the practical details provided in Section 3.1 and 3.2, and in Appendix C. Yet, I would have liked a diagram of the two Transformers to better understand the flow of information, and to illustrate how the sequences are fed to the networks. This would be particularly useful for the constraint prediction network, as the mechanism to cross-attend the primitive sequence and to point to its elements remains a bit vague to me.\n\nI didn’t fully understand the tokenization in Section 3.1. Since the ID token specifes the parameter type, why not also use it to specify the primitive type rather than allocate part of the value token for this purpose?\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "D_qTuNlPKnj", "reviewer": "Reviewer_ZmEf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The SketchGraphs dataset has resulted in quite a few concurrent CAD sketch generation methods that roughly follow a similar approach, with a few variations. The proposed method differs from prior work SketchGraphs and Willis et al. in its generation approach, and its ability to generate explicit constraints, respectively, and from both prior and concurrent work (SketchGen, DeepCAD and Ganin et al.) in its ability to successfully infer a CAD sketches from hand-drawn images. I would consider this to be a good contribution, but unfortunately the authors do not compare their results to any of these methods, making it hard to judge the relative quality of the generated CAD sketches. Comparing the qualitative results manually across papers, does seem to show at least a similar level of quality as the concurrent work, and better quality than prior work, so I lean towards accept, conditioned on the authors including at least qualitative comparisons in their revision, and ideally quantitative comparisons as well.\n\nDetails (in order of importance):\n- A comparison to prior work (SketchGraphs and ideally also Willis et al.) is needed. While both have a different generation approach than the proposed method, it should be possible to compare the constraint NLL and the full sketch NLL to SketchGraphs (as shown in Para et al. and Willis et al.), the primitive NLL to Willis et al., and the distributional statistics of generated sketches (possibly augmented with a few additional statistics to batter capture primitive and constraint quality) to both SketchGraphs and Willis et al. The qualitative results look quite good, so I would expect the quantitative results to be favorable as well. A comparison to concurrent work would be welcome, but is not necessary.\n- The following concurrent work is missing and should be discussed briefly in the related work:\nDeepCAD: A Deep Generative Network for Computer-Aided Design Models, Wu et al., ICCV 2021\n- The exposition is generally very clear, I am just missing two pieces of information that I can't find mentioned explicitly:\n--- Are types of primitives and constraints represented as separate 'type' tokens? If not, how are they represented? And if they are separate tokens, how are they represented in the constraint model, where the outputs are pointers rather than one-hot vectors?\n--- In the standalone primitive model, each primitive is represented by multiple tokens, but constraints only refer to either full primitives or sub-primitives. How is a reference to a full primitive obtained if only token embeddings are available? The authors only mention briefly: \"From the tuple of representations for each primitive, we extract reference embeddings for both the primitive as a whole (e.g., a line segment) and each of its sub-primitives (e.g., a line endpoint) that may be involved in constraints.\", but the authors don't mention how this is done.\n- For the primer-conditional generation, it might be good to discuss briefly if the primitives of an incomplete sketch need to be given in a similar order as during training (the typical order a designer would create the primitives in).\n- Willis at al. and Ganin et al. are missing the venue in the bibliography.", "summary_of_the_paper": "The authors present a generator for constrained CAD sketches that can also be used to convert hand-drawn line drawings to constrained CAD sketches or to auto-complete partial sketches. Two transformer models are used, the first model generates CAD primitives, optionally conditioned on a line drawing image or on a partial CAD sketch, and the second model generates constraints conditioned on the generated primitives. The authors show good qualitative results and quantitatively improved CAD generation performance over two simple baselines.", "summary_of_the_review": "The proposed methods contributes a new generation approach and explicit generation of both primitives and constraints over prior methods; and reconstruction of CAD sketches from hand-drawn images over both prior and concurrent work. The authors also show good qualitative results. However, as a big minus, the result are not compared to any related work. On the balance though, I am still leaning slightly towards acceptance, since the contributions are good, and the qualitative results are convincing enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "28OqAs_aOF", "reviewer": "Reviewer_DfnC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The task of using deep learning to generated structure representations of graphics useful in CAD applications is an interesting and open area of research that has potential for practical applications. The method proposed in this paper utilizes a reasonable approach based on state-of-the-art architectures and techniques. The paper is well-written, and the shown results look good given the dataset that is used for training.\n\nMy main concern about this paper has to do with its technical novelty. While it is true that the SketchGraphs dataset is fairly recent, and there have not been many papers published that utilize it, it is not completely obvious what are the high-level technical takeaways that can be useful for building on the proposed method. All of the components are certainly reasonable and demonstrate impressive engineering but seem like fairly standard applications of existing architectures. On the other hand, while the results are nice, I don't think they are ready to be put into a production-level system, in part due to the relative simplicity of the generated sketches. I would encourage the authors to clarify what is the broad novel insight for building learning-based systems that produce vectorized or CAD-style data and/or to provide some validation that the method can be practically helpful in a real-world CAD pipeline.\n\nThe other main issue that I see is with regards to comparison with related work. Some of the related work that's noted to be concurrent is in fact prior work. In particular, \"Engineering Sketch Generation for Computer-Aided Design\" [Willis et al. 2021] appeared at CVPR 2021 and, as the authors note, proposes a similar transformer-based CAD sketch generative model. It would be important to include a comparison to this work or at least a more detailed discussion. The authors mention that the main distinction is that Willis et al. do not produce the constraint graph as part of their output. This is true, but the evaluation of the learned constraint graphs is fairly limited---it is essentially only demonstrated in one small figure. Willis et al. show how the standard auto-constrain feature in AutoCAD can be used to infer constraints for sketches produced by their model. Do the learned constrain graphs here address certain cases that cannot just be handled by this sort of simple post-processing?\n\nA more minor issue is that it would be very helpful to include a diagram showing the network architecture---while the text is clearly written, it is a bit hard to keep track of all the moving parts.", "summary_of_the_paper": "This paper proposes a new deep generative model for 2D CAD sketches. The transformer-based model is able to generate both the primitives as well as the primitive constraints that make up a sketch, and its output can be then imported into standard CAD software. In addition to unconstrained sketch generation, the authors demonstrate experiments where they use the model to automatically complete partial sketches as well as produce sketches that resemble a rough human-drawn sketch that is given as input.", "summary_of_the_review": "This paper is well-written and tackles an interesting problem in a sensible way. However, due to limited novel technical insight and comparison to past work, I think it requires some improvement before it is ready to be published.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an approach for automatically generating CAD sketches, including both the primitives that describe the drawing, as well as the constraints that describe relationships between the primitives that need to be maintained even if the primitives are changed. This is an important problem that is starting to receive a lot of attention from the literature. 

Overall, the paper is very well executed and the results are quite compelling. 

There were some concerns about the relationship with the work by Willis et al. and other papers that were published around the time when this paper was submitted. There is still some novelty in this paper relative to those works as argued in appendix H, but it would have been really good to have a more quantitative comparison. However, the authors pointed out that this work was concurrent as opposed to prior work as per the ICLR reviewer guidelines. 

Overall, given the quality of this paper and the guidance given in the ICLR reviewer guide, most reviewers agree with the meta-reviewer that this paper should be accepted (the lowest reviewer still indicated it is above the acceptance threshold). However, there is some discomfort around not having an explicit comparison with very closely related work that ultimately was published before this paper.
Decision: Accept (Poster)