Computer-Aided Design as Language
Authors: Yaroslav Ganin
Sergey Bartunov
Yujia Li
Ethan Anderson Keller
Stefano Saliceti
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models
structured objects
computer-aided design
transformers
pointers
program synthesis

Abstract: Computer-Aided Design (CAD) applications are used in manufacturing to model everything from coffee mugs to sports cars. These programs are complex and require years of training and experience to master. A component of all CAD models particularly difficult to make are the highly structured 2D sketches that lie at the heart of every 3D construction. In this work, we propose a machine learning model capable of automatically generating such sketches. Through this, we pave the way for developing intelligent tools that would help engineers create better designs with less effort. The core of our method is a combination of a general-purpose language modeling technique alongside an off-the-shelf data serialization protocol. Additionally, we explore several extensions allowing us to gain finer control over the generation process. We show that our approach has enough flexibility to accommodate the complexity of the domain and performs well for both unconditional synthesis and image-to-sketch translation.

Computer-Aided Design as Language
Yaroslav Ganin Sergey Bartunoy'
Yujia Li
Ethan Keller
Stefano Saliceti
DDeppMind
OOnhape
Abstract
Computer-Aided Design (CAD) applications used in manufacturing model everything from coffee mugs to sports cars. These programs are complex and master. A component of all CAD
require years of training and experience models particularly difficult to make are the highly structured 2D sketches that lie the heart of every 3D construction. In this work, we propose machine learning model capable of automatically generating such sketches. Through this, we pave the way for developing intelligent tools that would help engineers create better designs with less effort. The core of our method is combination of a general- purpose language modeling technique alongside an off-the-shelf data scrialization protocol. Additionally, we explore several extensions allowing us to gain finer control over the generation process. We show that our approach has enough flexibility to accommodate the complexity of the domain and performs well for both unconditional synthesis and image-to-sketer translation.
1 Introduction
coincident
mirror
Li
Computer-Aiie Aidee Design (CAD) is used in the produc- tion of most manufactured objects: from to robots to stents to power plants. CAD has replaced pencil drawings with precise computer sketches, enabling paralleled precision, flexibility, and speed. Despite these improvements the CAD engineer must still de- velop, relate and annotate all the minutiae their de- signs with the same attention detail as their drafting- table forebears. CAD productivity might be improved by the careful application of machine learning to auto- mate predictable design tasks and free the engineer to focus on the bigger picture. The flexibility and power of deep learning is uniquely suited to the complexity of design.
orthogonal
tangent
Sketches are at the heart mechanical CAD. They are the skeleton from which three dimensional forms made. A sketch consists of various geometric entities (e.g., lines, arcs, splines and circles) related by specific constraints such as tangency, perpendicularity and sym- metry. Figure/ illustrates how entities and constraints work in tandem to create well-defined shapes. Geo- metric entities lie a single plane and together form enclosed regions used by subsequent construction oper- ations such as lofts and extrusions to generate complex
The anatomy of CAD sketch. Sketches are the main building block of ev- ery 3D construction. A sketch consists of entities (e.g., lines and ares) and constraints (e.g., tangent and mirror). The dotted curve shows what happens if we drop some of the constraints the design idea is lost.
Figure 1:
"Correspondence to: gaminodeepmind.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Interpreter states
line end.x
line. ead.y
(0, 0.6, False)
Transformer [33] Pointer Net [35]
0
line, Sine.atart.py yy: g
end 0.6
Tokens
tangent
Figure 2: Interpreter-guided generation of sketch. At each point in time, a Transformer [33] outputs raw value which is fed into an interpreter that decides which field of Protocol Buffers message this value corresponds to. Once the field is populated the interpreter communicates (-+) its decision back to the Transformer and transitions (~>) the next state.
3D geometry. Well-chosen sketch constraints are essential to properly convey design intent [2] and facilitate the sketch's 's resilience successive parameters modifications which is often understood as measure of the quality of a design document [8]. The dotted curve in Figure shows what happens when some of the constraints are dropped the design idea is lost.
The complexities of sketch construction are analogous those of natural language modeling. Select- ing the next constraint or entity in a sketch is like the generation of the next word in sentence. In both contexts, the selection must function grammatically (form consistent constraint system in the case of the sketch) and work towards some cohesive meaning (preserve design intent). Luckily, machine learning has proved highly successful in generating natural language especially the Transformer [33] trained on vast amounts of real-world data [26, 6]. It is therefore a promising choice for adapting the task of sketch generation. This work is our take this adaptation.
We make the following contributions: (1) We devise a method for describing structured objects using Protocol Buffers [32] and demonstrate its flexibility on the domain of natural CAD sketches. (2) We propose several techniques for capturing distributions of objects represented as serialized Protocol Buffers. Our approach draws inspiration from recent advances in language modeling while focusing on eliminating data redundancy. (3) We collect a dataset of over 4.7M of carefully preprocessed parametric CAD sketches and use this dataset to validate the proposed generative models. To our knowledge, the experiments presented in this work significantly surpass the scale of those reported in the literature both terms of the amount training data and the model capacity.
2 Related work
Datasets and generative models for CAD. Until recently there were very few parametric CAD datasets large and varied enough to serve training data for machine learning. This situation had started to change with the release of the ABC dataset [16] containing a collection of 3D shapes from the Onshape public repository [22]. Unfortunately, the main focus of [16] revolves around meshes and, as a result, the dataset is difficult to use for sketch modeling.
Several works concurrent with ours deal with the symbolic representation of CAD constructions. Seff et al. [31] center their attention on contributing better dataset of 2D sketches but also provide proof-of-coneept model predicting selected subset of object attributes. Willis et al. [37] use the

---

dataset from [31] to train a modification of [21] which demonstrates a boost in generation quality but designed to only work for sketch entities. The latter is addressed in our present work and in subsequent paper by Para et al. [24]. While [24] employ very similar ideas to ours they do not support certain features of the CAD data and their proposed model a more direct adaptation of [21] and therefore cannot handle arbitrary orderings of entities and constraints. In Section 5, we show that object ordering has a substantial impact on the performance.
Fusion 360 Gallery [36] attacks CAD data from a different angle. Here, the task is to recover a sequence of extrusion operations that gives rise to a particular target 3D shape. Despite dealing with 3D, this setting is deliberately limited: sketches are assumed to be given and the proposed model only decides on which sketch extrude and to what extent. [38] considers more general scenario where extrusion profiles are not provided and need to be synthesized from scratch. Although both of these works make initial steps towards full parametric CAD generation, they rely on significant simplifying assumptions and therefore it is unclear how well they will scale to more real-world scenarios. Our approach, on the other hand, is designed to be flexible and tomain-agnostic and is only limited by the data availability.
Vector image generation and inference. Synthesizing CAD sketches bears a lot of similarities with predicting vector graphics. In this field, several recent works Carlier et al. [7] and Reddy al. [30] using different vector object representations to define generative models of vector images. Egiazarian et al. [13] take a more traditional computer vision approach and propose a multi-stage pipeline for vectorizing technical drawings. All of these methods use highly domain-dependen architectures and, therefore, it would be a non-trivial task to adapt them for generation of complex sketch objects. CAD community has also been concerned with a similar task image to CAD conversion [20, 34, 10, 11], largely focusing on heuristic object recognition while our work relies more on learning the recognition from data.
Transformers for sequence modeling. In our work, we employ Transfomers [33] as a compu- tational backbone for the proposed approach. Due to its scalability and excellent performance [29, 9, 6, 27], this architecture has become the dominating approach in many sequence modeling applications. Our method can be seen as generalization of PolyGen [21], a Transformer-batt generative model for 3D meshes. Similarly to [21], we use Pointer Networks [35] to relate items in the synthesized sequence. Unlike PolyGen, however, our framework can handle non-homogeneous structures of arbitrary complexity. Moreover, we simplify the architecture to use single neural network to generate the entire object of interest. All these improvements make our approach good fit for modeling CAD sketches and potentially other components of CAD constructions.
3 Data
Formally, a CAD sketch is defined by two collections of objects: entities and constraints. Each object is generally represented as set of attribute-valus pairs where value can be either primitive (e.g., integer or floating-point or complex (e.g., an array or another object). Sketches that we use this work originate from the Onshape platform [22] which provides them in JSON format [25]. As the first step in our processing pipeline we convert JSON messages into Protocol Buffers (PB) [32]. In order to keep the pipeline domain-agnostic and as widely applicable possible, we aim to avoid any significant changes to the data and largely retain the original structures of objects. The benefit of converting into PB is twofold: the resulting data occupies less space because unnecessary information is removed, but also, unlike JSON, PB provide a convenient way to define precise specifications for structures of arbitrary complexity.
Listing shows how we represent the line entily and the mirror constraint (see Appendix for an extensive list of supported objects). The line specification is straightforward: we first need to decide whether our entity should be treated as a construction geometry² and then provide pairs of coordinates for the beginning and end of the segment. The MirrorConstraint is used to force an to be symmetrical with respect to
arbitrary number of pairs of geometries (i.e., mwrroree_ppaiieeee some axis (i.e., mirror). Constraints rely on the Pointer data type specify entities they act upon. In practice, a pointer is simply an index in the table of all the eligible pointees (i.e., entities and their parts).
Conntrucctonn geometries rendered in dashed style in the figures.
3

---

message LineEntity { bool is_coustrnttion 1; message Vector { // 2D coordinate. double x 1; double y = 2;
message MirrorConstraint {
Pointer mirror = 1; // Axis of symmetry.
message Pair { // Mirrored objects.
Pointer first 1;
Pointer second 2;
}
}
Vector start 2; // Start point.
repeated Pair mirrored_ pairs =
Vector 3; // End point.
}
}
(a) Entities and constraints have similar structures, Pointers refer to entities that constraints are applied to.
message Entity { oneof kind {
message Object { oneof kind { Entity entity 1; Constraint constraint 2; }
LineEntity 1ine 1; // And other entity types.
}
}
message Constraint
message Sketch {
// Defined similarly to Entity.
repeated Object objects = 1;
}
(b) A full sketch is defined as a sequence of objects each of which can he either an entity or a constraint.
Listing Examples of object specifications. We represent objects using Protocol Buffers. Protocol Buffers allow us to easily write specifications for structured objects of varying complexity.
Our ultimate goal is to build machine learning model of sketch objects. To that end, process the data even further and represent these objects as sequences of tokens. This allows us to pose sketch generation as language modeling (LM) and take advantage of the recent progress in this area [26, 6]. To achieve this, we pack first collections of entities and constraints into one Protocol Buffer message (see Listing 1b) assuming some ordering of objects. We discuss different orderings in Section 5.
There are few ways to obtaining a sequence of tokens from a sketch message. Arguably the most intuitive one is to format messages as text. For line entity connecting (0.0, 0.1) and (-0.5, 0.2) this will result in:
{ (i__conceee cossttruuuct::: true, start x: 0.0, y: 0.1 }, end { x: -0.5, y: 0.2
Since this format contains both the structure and the content of the data, the resulting sequences end up being prohibitively long. Additionally, the model would have to generate valid syntax, which would take up some portion of the model's capacity. To overcome these challenges, we work with two flavours of serialized PB messages.
The first one is sequences of bytes obtained by calling the SeriaaiizeoSStrngg! method of a message. Such sequences are much shorter since the structure is handled by an external parser automatically generated from the data specification. The parser's 's task is to interpret the incoming stream of unstructured bytes and populate the fields of PB messages. However, like the text format, not every sequence of bytes results in valid PB message.
Going one step further, we can utilize the structure of the sketch format more directly, and build a custom interpreter, that takes as input a sequence of tokens each representing valid choice at various decision steps [4] in the sketch creation process. We designed this interpreter in such a way that all sequences of tokens in this format lead to valid PB messages. More specifically, message as a sequence triplets (d,, Ct, fi) where is an index of the token. The majority of tokens describe basic fields of the sketch objects with each token representing exactly one field. The first two positions in each triplet are allocated for a discrete value and a continuous value respectively. Since each field in message is either discrete continuous only one of two positions is active time (the other one is set to a default zero value). The third component is a boolean flag signifying the end of repeated field which contains list of elements of the same type. An example sequence for sketch containing a line and a point placed at one of its ends is shown in Table (Triplet column).
represent
For additional details how we handle complex constructions of the PB language see Appendix A.

---

Triplet
Field
Triplet
Field
(0,0.0, 0.0, False)
objects. kind
(0, 0.O, False)
objects kind
2. (0, 0.0, False)
entity.
9.
(1, 0.0, False)
entity. kiid
Point
0.0, False)
line. iine.is_constr
10.
(0, 0.0, False)
point.is_cone
Line
(0, 0.0, False)
sine.start..
11.
(0, 0.0, False)
point.x
(0, 0.1, False)
y
12.
(0, 0.1, False)
point.y
(0, 0.55, False)
Line.end.x
13. (0, 0.0, True)
objects
biid kind
7. (0, 0.2 False)
y
Table 1: A triplet representation of a simple sketch. The sketch contains and a line and point. Within each triplet in the left column, the active component (the value that is actually used) is highlighted in bold. The right column shows which field of the object the triplet is associated with.
Given a sequence of such triplets, is possible to infer which exact field each token corresponds to. Indeed, the very first token (d1, c1 fa) is always associated with objects kind since it is the first choice that needs to be made to create a Sketch message (see Listing lb). The second field depends on the concrete value of d1. If d, token corresponds to entity kind. The rest of the sequence is associated in a similar fashion. Field identifiers along with their locations within object form the context of the tokens. We use this contextual information as an additional input for our machine learning models since it makes it easier to interpret the meaning of the triplet values and be aware of the overall structure of the data.
then the first object
entity which means that the second
4 Model
In order to estimate the distribution Pdata of 2D sketches in a dataset D, we decompose the joint distribution over the sequence of tokens [19] t = .t,....,ty ft)) ) in an autoregressive fashion, repre-
senting each conditional with neural network parameterized by 0 and pose the estimation of Pdata as maximization of the log-likelihood of DD i.e.,
N
p(t: 0) = II p(t t));
log p(t; 0) max, Σ EED
(I)
where is the length of the sequence and fay denotes all the tokens preceding t,-
More concretely, we employ the Transformer decoder architecture [33] that takes an embedding of the token - = embed, E RD and maps it into another vector hG of the same dimensionality. The latter is decoded into parameters of p(4; Les) by a learned mapping d-st;
Byte representation When dealing with the bytes of PB message, each token is simply a discrete value in the range {0, .., 255} U {EOS} and therefore p(t; fzz 0) can be modeled as categorical distribution similar to how it's done typical LM approaches [5]. In this setting, for each time step i of the sequence we have
embed ; (t:-1) V(t, e,
where [:] denotes the lookup operation and e/ pos is a position embedding for position i. Both V and pos are learned. Moreover, Vi dist; is the same linear projection into 237 (256 values and EOS) and the output is treated logits of the distribution.
Triplet representation In case of the triplet representation, we follow a slightly more involved procedure. As outlined in Section 3, tokens can be either discrete or continuous. Additionally, different discrete tokens may have different ranges of values. For example, there are only two possible token - either to an entity or a constraint. On the other hand, the range
values for the object.king of the entity. kind token has cardinality of 4 since support 4 different types of sketch entities. This means that we can't 't naively describe each conditional in Equation using the same template distribution like we did for the bytes. We circumvent this by introducing the notion of token groups.
token group G collection of related token types that can be handled in similar fashion. Specifically, we use the same embedding function and the same projection for every t such that
Note that use subscript for embed and dist since they can be different for different time steps.
5

---

L 000 A
(a) Unconditional byte model.
(b) Unconditional triplet model.
(c) Image-conditional triplet model.
(d) Solved samples.
(c) Data.
Figure 3: Synthesized sketches and data. (a)-(c) show samples from various proposed models (we use Nucleus Sampling with top-p = 0.9). (c), the output rendered slightly thicker style. (d) demonstrates samples from the unconditional model after applying predicted constraints (the output in blue). (e) shows examples from the dataset.
type(() E G. For instance, we might want to group all the tokens associated with coordinates. In the example from Table 1, tokens with indices 4-7 and 11-12 will all end up the same G³, Naturally, use the same functional form for the output distribution p(t. te 0) within each group.
We embed t that belongs to the group & as (note the difference with Equation 2):
[[field(t=-i) V [1,1]] + e c.fll
where field(!) returns the field of L (e.g., objects kind line. start.x ) and K is collec- tion of learnable embeddings for every possible field type. Unlike in Equation 2, instead of using global position embedding e,oo we describe the location with the index n of the current object as well as the relative position m 4 within object.
We handle the "end" tokens (i.e., ft = True) similarly to EOS in Section 4 the output projection produces additional logit used compute the probability of ending the repetition. Since fi is only expected to be True at particular points in the sequence (i.e., right after tokens forming a whole element of the list) mask oud the extra logit everywhere else. This ensures that the "end" token can't be predicted prematurely and also eliminates its unnecessary contribution to the optimized objective.
One significant difference between the byte setting and the triplet setting is how we process pointer fields. In the former, pointers do not get any special treatment and are generated just like any other integer field. We rely on the model's capability to make sense the entity part index and relate it to the corresponding locations in the sequence via attention weights. This seems to be viable strategy since Transformers have demonstrated an impressive referencing capacity recent works [6].
Since the triplet representation provides with direct access to the semantics of tokens it's possible relate pointers to their pointees more explicitly by using Pointer Networks [35]. The approach we
For the full list of token groups used in the model Appendix B.

---

are taking here is similar to [21]. In order to compute p(t; :0), we first project the output of the Transformer h, into the final pointer vector p: ptr h2. The conditional is then obtained as follows:
p(t points to tik tt<:0) softmax; (p: H:), H, [ha. hik..... haMl
(4)
where softmax; is the k-th element of the softmax vector and r; {fal..... fMM } is the set of tokens that & can point to. Naturally, T; only contains tokens from teg. This is different from [35, 21] where r; = r external the predicted sequence and remains immutable throughout the generation process.
In Onshape, constraints can be applied not only whole entities but also their subparts (e-g., the center of a circle or the end points of a line segment). To handle this, we introduce special referrable tokens decoupling pointees from the object attribute tokens. Referrables and have the same identifier within each entity type. In order to let the model distinguish between different subparts, we adjust Equation 3 to use learnable embedding of part index instead of [t] As referrable tokens do not need to be predicted the respective terms are removed from Equation For the same reason, in Equation 4, h5s corresponds to the time step where ti the outpur and not the input. This allows us to avoid having unused Transformer outputs.
injected after each entity
Finally, we need specify how embed pointers inputs to the Transformer network. Following [35, 21] we could reuse h2 for tokens that point to 1z. Unfortunately, this creates output-to-input connections which are extremely detrimental the efficiency of the Transformer architecture different time steps can no longer be processed in parallel during training. Instead, we opt for simpler solution and employ the standard embedding scheme for discrete tokens (Equation 3).
Sampling from the model Sampling from the byte model is identical to sampling from any typical Transformer- based LM. The triplet model, on the other hand, requires slightly more bespoke handling. Figure illustrates the procedure. We start by embedding and feeding a special BOS token into the Transformer. The Transformer then outputs collection of triplets, for each possible token group. In order to determine which concrete token needs to be emitted, we employ an interpreter (a state machine) automatically generated from the data specification. Knowing the current state allows us to choose the right token group and associate the active component of the triplet with a field in the synthesized object. Once the appropriate field populated the interpreter transitions the next state and produces an output token which is then fed back into the model. The process stops when the state machine receives the "end" triplet for the outermost repeated field (i.e., object. kind) ).
Conditional generation In addition to the unconditional model described above, we explore a variant that allows to translate bitmaps into sketches. Here, we simply let the main Transformer cross-attend to the features extracted from the input image by a ViT network [12]. The specific setup is detailed in Appendix C.
5 Experiments
Average bits per
Model
Sequence
We validate our proposed approaches on the data that we obtained from the repository of documents publicly available on the Onshape platform [22]. Following the standard evalua- tion methodology for autoregressive generative models [23, 21] we use log-likelihood as our primary quantitative metric. Additionally. we provide variety of random and selected model samples for qualitative assessment (Figure 3).
object
sketch
Uniform
unord, bytes
112.23
3683.56
unord, triplets
25.34
847.52
Text Byte
interleaved
4.622
139.687
concatenated
4.381
132.621
interlcaved
4.252
127.495
Triplet
concatenated
4.218
127.913
interleaved
4.103
123.213
Dataset Unlike the majority of the existing works dealing with CAD sketch generation [31, 37, 24] we do not rely on SketchGraphs [31]. Instead, we collect the largest to date dataset of engineering sketches addressing the main disadvantage of [31], severe data duplica- tion. The acquisition and the filtering procedures
Cond.
interleaved 1.570
53.730
Table 2: Test likelihoods of various models. The object column computed as the average number of bits per object in sketch averaged across test examples. The sketch column is similar except we do not divide by the number of objects.
7

---

# objects
entities
# constraints
0.10
0.30
0.15
0.20
0.10
0.06
0.10
0.05
0.00
0.00
0.00
50
100
10
20
30
20
40
60
Byte
Byte (0.9)
Triplet (1)
Triplet (0.9)
Data
Figure 4: Distribution of various sketch statistics for samples drawn from our unconditional models. The top-p parameter for Nucleus Sampling is shown in parentheses.
detailed in Appendix D. For our experiments we split the dataset randomly into 3 parts: 4,656,607 examples for the training set and 50,000 sketches for each the validation and the test set.
Unconditional generation. In this series of experiments, the goal is to determine how well our models capture the distribution of sketches in the dataset. We use the same network architecture for both the byte and the triplet settings." We compare two orderings of objects: in the first one (concatenated), constraints go after the last entity while in the second (interleaved), a constraint object injected immediately after the entities operates on. In both cases, the relative orderings within both the sequence of entities and the sequence of constraints are taken directly from the original JSON messages.
Table shows test log- ligllikeissodss obtained by different models. Unsurprisingly, our proposed methods (rows 4-7) significantly outperform the weak baselines (rows 1-2). The difference between the two uniform settings is due to the fact that the byte description of a PB message is usually longer than the triplet one: 239 vs. 456 tokens average with the maximum length of 959 1987. Additionally, the tokens in the triplet representation tend to have smaller range of values (i.e.. < 257).
These differences between representations may partially explain why triplet models demonstrate better performance on the hold-out test set. It's also worth emphasizing that the byte model does not receive any explicit information about the parsing state. This seem to make learning more challenging and as a result compared to the triplet model it takes roughly 3 times more network updates to reach the highest data likelihood on the validation.
Another important factor affecting the performance of the models is the choice of the object ordering. As it is evident from Table 2, the interleaved ordering consistently leads to better results. One explanation for this is that at any point in time, the model has more explicit information about the relations between the sketch entities produced so far.
In the row 3 of Table 2, we also provide the test logeliiel for conventional language modeling baseline trained on the text representation of PB messages (see Section 3). Here, we employ the SentencePiece tokenizer [17] with fairly aggressive settings (8000 words in the vocabulary; splitting at whitespaces switched off) to be able to keep the sequences within the budget of 1024 tokens. As can be seen from the table, the resulting Transformer model (of exactly the same architecture as other entries) despite being better than the uniform baselines is still substantially worse than the proposed methods.
In addition to measuring likelihoods, we sampled 10,000 sketches from the best performing byte and triplet models and computed distributions of various high-level statistics (Figure and Figure in the appendix). We repeated this procedure both with and without using Nucleus Sampling (NS) [15]. Both models follow the data distribution closely when we use samples from the unmodified model output. In this setting, however, a significant fraction of sketches is either malformed (e.g., the generated PB message cannot be parsed) or unsolvable: 36% for the byte model and 14% for the triplet model. NS with top-p = 0.9 skews the sample distribution and seems to have a more pronounced negative effect on the byte model. The upside that the resulting sketches become "cleaner": the percentage of invalid samples goes down to 25% and 6% for the byte and the triplet settings respectively.
'The dataset is available at
7 'llease refer to Appendix E for details.
8

---

Coincident
Coincident
Length:
Tangent
Hor isontal
Equal
Vertical
Longth
Parpendicalar
Parallel
Nidpolitt
Vertical
Perpendicular
Diatance
Dianeter
Mirror
Coincident
Horizontal
Perpendicular
Vertical
Parallel
Figure 5: Entities and constraints sampled from the unconditional (top) and the conditional (bottom) triplet models. The first column nodes represents different entities (all parts are folded into single node). The order of nodes (top to bottom) follows the generation order. The second column represents different constraints also ordered by their index the sequence. Finally, the third column is reserved for constraint types, from the most to the least frequent.
Figures 3a and 3b show renders of random samples from several proposed models. Overall, generated sketches look plausible and exhibit a lot of desired properties: closedness of regions, regularity, symmetry, a non-trivial amount of fine detail. We observe that the byte model produces slightly less complex samples with fewer open also note that the model does not always synthesize sensible sketches - just like any other typical autoregressive model trained with teacher forcing suffers from not being able to recover from mistakes made early on in the sequence [28]. This can potentially be addressed by fine-tuning using, for example, reinforcement learning.
but this could be side effect of partieular top-p value. We
We show a more detailed sample from the triplet model in Figure 5. While not perfect, the inferred constraints are reasonable most of the time. The model does good job connecting entities using Coincident constraints but also successfully detects more complex relations spanning more than two primitives (e.g., Mirror). Having access constraints gives opportunity correct mistakes in entity prediction by applying an external sketch solver (see Figure 3d). Although this aspect of sketching not the main focus of this work, believe that tighter integration between the model and the CAD software will lead to a significant boost in generation quality.
Conditional generation. As discussed in Section 4, also trained image-conditional model using the same regime as for the unconditional one. As expected, achieves significantly better fit (the last row of Table but the same time retains a non-trivial amount of uncertainty. The latter arising, in particular, from the fact that different permutations of entities result in the same rendered image. mmage-conditional samples can found in Figure 3c. The model was able nearly perfectly reconstruct simpler sketches and mostly made mistakes in the presence of a large number of fine details. Additionally, we compared our system against the nearest neighbour baseline in terms of visual reconstruction error as measured by Chamfer distance. We found that the proposed method reduces the error by 80% (see Appendix H). order test the -uitt--distiiuutionn performance, we supply the model with several made drawings. Surprisingly, after minor post-processing account for the short sequence hias the system manages to produce reasonable reconstructions (see the bottom example in Figure 5). We detail this experiment in Appendix I.

---

Discussion
In this work, we have demonstrated how a combination of general-purposs language modeling technique alongside an off-the-shelf data serialization protocol can be used to effectively solve generation of complex structured objects. We showcased the proposed system on the domain of 2D CAD drawings and developed models that can synthesize geometric primitives and relations between them both unconditionally as well as using bitmap as a reference. These are only initial proof-of-concept experiments and we are hoping to see more applications taking advantage of the flexibility of the developed interface: conditioning on various sketch properties, inferring constraints given entities and automatically completing drawings, to name few.
Although we focused our attention on a particular dataset we argue that the approach described in this paper is largely domain-agnostin In order to adapt the system to new kind of data, the algorithm designer only needs to provide an appropriate Protocol Buffer specification and if the PB language is too restrictive one can always replace it with more powerful interpreter. As straightforward direction for future work, we can consider extending the method to handle 3D. In Onshape, 3D operations bear a lot of similarities with sketch constraints just like constraints, they can be represented as nested messages containing references to the geometries existing in the scene. Thus, most of the ideas from the present work can be taken verbatim to this new setting.
We hope that this work will serve as a stepping stone for further advances in the field of automated CAD but also inspire new ideas and approaches to generative modeling of arbitrary structured data.
Acknowledgments and Disclosure of Funding
The authors would like to thank Charlie Nash, Georg Ostrovski, and Adam Kosiorek for helping with the manuscript preparation well Igor Babuschkin, David Choi, Nate Kushman, Andrew Kimpton, Jake Rosenfeld, Lana Saksonov, John Rousseau, Greg Guarriello, Andy Brock, Francesco Nori, Aäron van den Oord, and Oriol Vinyals for insightful discussions and support.
YG, SB, YL, and SS are funded by DeepMind. EK is funded by PTC.
References
[1] Alakuijala, A. Farruggia, P. Ferragina, E. Kliuchnikov, R. Obryk, Z. Szabadka, and L. Vande- venne. Brotli: A general-purpose data compressor. TOIS, 2018.
[2] H. K. Ault. Using geometric constraints to capture design intent. JGG, 1999.
[3] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. Technical report, SRI INTERNATIONAL MENLO PARK CA ARTIFICIAL INTELLIGENCE CENTER, 1977.
[4] R. Bavishi, C. Lemieux, R. Fox, K. Sen, and I. Stoica. AutoPandas: Neural-backed generators for program synthesis. In OOPSLA, 2019.
[5] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. JMLR, 2003.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[7] A. Carlier, M. Danelljan, A. Alahi, and R. Timofte. DeepSVG: A hierarchical generative network for vector graphics animation. arXiv preprint arXiv:2007. 2020.
[8] Company, F. Naya, M. Contero, and Camba. On the role of geometric constraints to support design intent communication and model reusability. Computer-Aided Design and Applications, 2019.
[9] Dhariwal, Jun, C. Payne, J. Kim, A. Radford, and Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.00344 2020.
10

---

[10] D. Dori and K. Tombre. From engineering drawings to 3d CAD models: are we ready now? Computer-Aides Design, 1995.
[11] D. Dori and L. Wenyin. Automated cad conversion with the machine drawing understanding system: concepts, algorithms, and performance. SMC, 1999.
[12] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19900, 1929, 2020.
[13] V. Egiazarian, O. Voynov, A. Artemov, D. Volkhonskiy, A. Safin, M. Taktasheva, D. Zorin, and E. Burnaev. Deep vectorization of technical drawings. In ECCV, 2020.
[14] Y. Ganin, Kulkarni, Babuschkin, S. A. Eslami, and Vinyals. Synthesizing programs for images using reinforced adversarial learning. In ICML, 2018.
[15] A. Holtzman, J. Buys, L.. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint
2019.
[16] S. Koch, A. Matveev, Z. Jiang, F. Williams, A. Artemov, E. Burnaev, M. Alexa, D. Zorin, and
D. Panozzo. ABC: big CAD model dataset for geometric deep learning. In CVPR, 2019.
[17] T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 2018.
[18] A. Meurer, C. P. Smith, M. Paprocki, O. Čertík. S. B. Kirpichev, M. Rocklin, A. Kumar, Ivanov, K. Moore, S. Singh, al. SymPy: symbolic computing in Python. PeerJ Computer Science, 2017.
[19] T. Mikolov, M. Karafiát, L. Burget, J. Čemocký, and S. Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010.
[20] V. Nagasamy and N. A. Langrana. Engineering drawing processing and vectorization system. CVGIP, 1990.
[21] C. Nash, Y. Ganin, S. A. Eslami, and Battaglia. PolyGen: An autoregressive generative model of 3D meshes. In ICML, 2020.
[22] Onshape developers. Onshape website. https://www https::/www.onnhhap.. com/. Accessed: 2021-03-10
[23] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with PixelCNN decoders. arXiv preprint arXiv:16006 053288 2016.
[24] W. R. Para, S. F. Bhat, P. Guerrero, T. Kelly, N. Mitra, I.. Guibas, and P. Wonka. SketchGin: Generating constrained CAD sketches. arXiv preprint arXiv:2106.02711, 2021.
[25] F. Pezoa, J. L. Reutter, F. Suarez, M. Ugarte, and D. Vrgoë. Foundations of JSON schema. In WWW, 2016.
[26] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[27] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprini 1rXiv:2102.12090 2021.
[28] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. arXiv preprint 1rXiv:1511.06733 2015.
[29] A. Razavi, A. v. d. Oord, and O. Vinyals. Generating diverse high-fidelity images with VQ- VAE-2. arXiv preprint arXiv: 00446, 2019.
[30] Reddy, M. Gharbi, M. Lukac, and Mitra. Im2Vee: Synthesizing vector graphies without vector supervision. arXiv preprint arXiv:2102.02798, 2021.

---

[31] A. Seff, Y. Ovadia, W. Zhou, and R. P. Adams. Sketchgraphs: A large-scale dataset for modeling relational geometry computer-aides design. arXiv preprint 2xXiv:2007.08506 2020.
[32] K. Varda. Google protocol buffers: Google's data interchange format. Technical report, 2008,
[33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. arXiv preprint T1X..07706603722 2017.
[34] Vaxiviere and K. Tombre. Celesstin: CAD conversion of mechanical drawings. Computer,
1992.
[35] O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. In NeurlPS, 2015.
[36] K. D. Willis, Y. Pu, J. Luo, H. Chu, T. Du, J. G. Lambourn,, A. Solar-Lezama, and W. Matusik. Fusion 360 gallery: A dataset and environment for programmatic CAD reconstruction. arXiv preprint vrXiv:2010.023922 2020.
[37] K. D. Willis, P. K. Jayaraman, J. G. Lambourne, H. Chu, and Y. Pu. Engincering sketch generation for computer-aided design. arXiv preprint wrXiv:2104.09621 2021.
(38] R. Wu, C. Xiao, and C. Zheng. DeepCAD: A deep generative network for computer-aided design models. In ICCV, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] The method presented in this work is general machine learning framework and it is not immediately obvious how could negatively impact the society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is proprietary but may be open- in the future. The dataset will be released shortly.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] The experiments in the paper are expensive to run. Moreover, we did not observe any significant fluctuations in the validation metrics between different training runs with the same hyperparameters.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use data coming
from publicly available platform and we give credit to the platform in the paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
12

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lbFvsVPWLT", "reviewer": "Reviewer_K3kn", "summary": "This paper provides a large dataset of structured 2D sketches and proposes transformer-based models capable of both unconditional and conditional generations of such sketches. The authors present a novel representation for structured sketches based on Protocol Buffers, encoding both objects and the constraint between them, and further devise a custom tokenization scheme to adapt it for the training and inference using a transformer decoder architecture, showing advantages compared to a simple byte tokenization. Their model can successfully generate valid structured sketches from scratch, and moreover infer valid objects and constraints from an input image (as a condition).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is mostly well-written and easy to follow. The proposed coding of structured 2D sketches is novel and seems useful, particularly for an effective correction of predicted objects based on constraints. The dataset, which the authors plan to make public, could be useful to other applications due to its quality and structured labeling, and the experiments, though not extensive, show that their proposed models work as expected. I have a few concerns and questions that I will elaborate below:\n\n1. Run-time and train-time of the models are missing, and this makes comparisons between models and an overall efficiency/practicality evaluation hard. For example, how far is the generation (unconditional and conditional) from real-time performance?\n\n2.  Comparison to similar models is missing, that is, how does the model compare in quality and speed with other vector image generation methods (conditional and unconditional)? (an extension of table 4 in appendix H).\n\n3. Whenever reporting average, it is good practice to report standard deviation as well, to provide a sense of confidence.\n\n4. It is important to highlight/discuss the types of failures (wrong shapes and wrong constraints) and the rates with which each happen, to give a better picture of the models’ capabilities.  Also, I think discussion of model failure, such as the compression failure discussed in appendix G, are very valuable to the more broad transformer research, and are a good idea to include and elaborate on in the main text.\n\nSide note: wrong syntax in Eq.1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Ok", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "aMXWMr6Ygmb", "reviewer": "Reviewer_p1Hz", "summary": "This paper proposes a deep learning model of automatically generating computer-aided design (CAD) sketches, which are typically composed of entities (e.g. line and arc) and constraints (e.g. tangent and mirror). The authors devise a method to describe CAD sketches as serialized Protocol Buffers, and further represent them as sequences of triplet tokens. Then they combine the devised data representation with language modeling techniques (Transformer + PointerNet) to capture the data distribution in an auto-regressive manner. The proposed method is evaluated mainly on unconditional generation and various design choices are ablated. Additional generation results are shown conditioned on binary images.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this work is clear to me. As one of very first attempts on generative modeling for engineering CAD sketches (there are some concurrent works), it devise a method to represent CAD sketch as a sequence of triplet tokens and combine this representation with general language modeling technique to capture both the geometric primitives and constraints. The proposed special embedding and pointer strategy for its triplet representation is also interesting. \n\nThe results in general look good and I like the colored visualization. Besides unconditional generation, one of the potential application, image-conditional generation is also thoroughly evaluated. \n\nMy concerns and suggestions are as follows:\n1. It does not compare to any prior works. I understand it's the very first work for CAD sketch generation, but there are prior works (e.g. DeepSVG) that generates vector graphics, which are also parametric sketches but without constraints. The authors can just compare results in the image domain. The same applies to the image-conditioned generation.\n2. The test set is too small. A test set of 50K is only 1% compared to a training set of 4.6M (line 239-241). This could result in much variance. 90%-10%-10% is the general practice. I hope the author to use a larger test set in the revision. If not, it's better to explain why. \n3. Consider to cite concurrent works on CAD generation: \n    - DeepCAD: A Deep Generative Network for Computer-Aided Design Models\n    - SketchGen: Generating Constrained CAD Sketches\n4. What exactly is external sketch solver (line 288)? I guess it's some sort of constraint optimizer. It's better to give some detailed descriptions (including in the supplement is OK).\n\nLastly, will the dataset be released? A large dataset is always beneficial to the community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suggestions are as above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "XrUnWTm2f6Z", "reviewer": "Reviewer_KVBt", "summary": "This paper proposes a novel method to represent structured CAD sketch objects via Protocol Buffers\nand demonstrate its generalization ability across the heterogeneous nature of sketches. They\nalso proposed a transformer-based generative model for generating the CAD sketches and evaluate\nits effectiveness on carefully collected and preprocessed data which consists of over 4.7M\nsketches. They show the capability of the generative model for both unconditional and conditional\ngeneration tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*:\n1. In general, the motivation of the paper is quite strong. There are few learning-based methods\nfor generating CAD sketches.\n2. The proposed data representation costs less space than the classic CAD sketch format by\nremoving unnecessary information.\n3. The proposed triplet and byte representation overcome the necessity of converting the raw\nsketch message into a text format that contains both the structure and content of the data.\n4. The quantitative (Table 2) and qualitative (Figure 3) experiment results demonstrate the better\nperformance of the triple representations than byte ones.\n5. The predicted constraints by the model are reasonable for making a closed-shape CAD sketch.\n6. The proposed representation methods lead to a high generation quality for both conditional\nand unconditional generation tasks.\n\n*Weakness*:\n1. The metric for evaluating the performance of the proposed method is not sufficient. How about the \nfidelity and diversity of the generation with respect to the training dataset? Those should be quantitatively evaluated also.\n2. The paper lack comparison with some existing generative models for sketch generation, such as Sketch-RNN.\n3. I think it is necessary to compare the proposed representation with the naïve text format.\n4. Line 141 has a typo. No Figure 1b.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper shows strong motivation for automated CAD sketches generation. The proposed\nsequential data representation takes the advantage of using the transformer-based architecture\nfor the sketch generation. Yet, this paper lacks some important evaluation metrics and\nbaseline comparisons, which is the main reason for my rating not being higher. \nThe experiment results show the high generation quality of the CAD\nsketches generation for both conditional, and unconditional tasks.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "F506RDdcgnl", "reviewer": "Reviewer_yGks", "summary": "This paper presents a learning-based model to generate 2D sketches that are widely used in 3D CAD applications. It considers 2D sketches consisting of entities (lines, curves, etc.) and constraints (mirror, coincide, perpendicular, etc.), a representation that is commonly used in modern CAD software. By design, 2D sketches are structured objects. To handle these structured data, this paper presents a linear representation using Protocol Buffers, which flattens the structured data into a sequence of triplets. Through this representation, the paper establishes similarities between natural language modeling and 2D sketches and proposes to use Transformer to train a generative model that captures the distribution of sketches in a large dataset. The paper reports quantitative and qualitative performance of the generative model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is dealing with a practical problem that has wide industrial applications. The high-level idea behind the proposed technical method (bridging 2D sketches and languages) makes sense to me. The quantitative and qualitative evaluation seems OK, although I think a few more experiments could be added to make the paper stronger (see below). Overall, I think this paper has contributed some technical ideas that this community can benefit from. However, I still have a few questions and would like to make my final decision after reading the authors’ answers:\n- A high-level comment on the motivation and application of the technical method: the paper attempts to make an analogy between 2D sketches and natural language modeling. They are indeed similar in some way but also have differences. In particular, 2D sketches in mechanical designs have a low tolerance of inaccurate or imprecise designs, while imprecise natural languages are quite acceptable in many applications. As an example, 2D sketches of a pair of a nut and a bolt must have compatible radii, but a generative model like this paper might generate designs with roughly the same radii which, while looking similar, won’t work at all in practice. To what extent can we extend this Transformer idea to generate precise 2D sketches that are really usable in real-world mechanical design applications?\n- The paper mentioned that long sequences of data are a challenge, motivating the usage of PB messages. From what I understand, the proposed data representation still duplicates geometric information, e.g., line 4-5 and line 11-12 in Table 1 store the same point (x, y) twice. Why not separate topology and geometry information like in a boundary representation (B-rep), or did I misunderstand your data representation? What is fundamentally novel in your Protocol Buffer representation if I compare it with B-rep?\n- The paper mentioned “from coffee mugs to sports cars” twice in the main paper and its supplemental material, yet the 2D sketches shown in the paper are not even close to the complexity of a sports car. In particular, line 513-515 seems to imply some of your sketches are from sports cars. If you have impressive results from these sophisticated models, please add them to the main paper, otherwise, I would suggest toning down this description and state what examples your 2D sketches can handle in a more factual way.\n- One thing that makes 2D sketches really useful in CAD applications is that they are parametric designs: the (entity, constraint) combo not only defines a particular design itself but also provides a family of similar designs if you perturb the continuous parameters (e.g., length of a line, angle of an arc) in these entities. In practice, it is surprisingly not uncommon for these perturbations to result in invalid sketches that break some constraints if constraints are not defined properly. I would love to see some small experiments on the 2D sketches generated by this paper, i.e., mildly perturbing the entity parameters in the resultant 2D sketches, and see how often they lead to invalid sketches that break the constraints. I understand this may be out of the scope of this paper, but I think having such an experiment would be helpful for us to fully understand how well these generated 2D sketches can be used for parametric designs.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see my review above for limitations. I do not see any major issues with negative societal impact from this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new approach to generating constrained CAD sketches. The key challenge in this problem is generating constraints that relate all the different strokes in a sketch. At a high level, the main idea in this paper is to treat this as a language generation problem, since the sketches with their constraints can be represented in a language. The key new idea for this paper is that instead of learning directly in the language of CAD sketches, they first encode the strings to a very compact and dense representation, which allows the network to learn more efficiently. Versions of this idea have been used in other settings but the idea is new in this context. 

Additionally, the paper introduces a new dataset for this problem which should be of significant value to the community.

Decision: Accept (Poster)