SketchGen: Generating Constrained CAD Sketches

1 Introduction . Computer Aided Design ( CAD ) tools are popular for modeling in industrial design and engineering . The employed representations are popular due to their compactness , precision , simplicity , and the fact that they are directly ‘ understood ’ by many fabrication procedures . CAD models are essentially a collection of primitives ( e.g. , lines , arcs ) that are held together by a set of relations ( e.g. , collinear , parallel , tangent ) . Complex shapes are then formed by a sequence of CAD operations , forming a base shape in 2D that is subsequently lifted to 3D using extrusion operations . Thus , at the core , most CAD models are coplanar constrained sketches , i.e. , a sequence of planar curves linked by constraints . For example , CAD models in some of the largest open-sourced datasets like Fusion360 [ 37 ] and SketchGraphs [ 27 ] are thus structured , or sketches drawn in algebraic systems like Cinderella [ 1 ] . Such sequences are not only useful for shape creation , but also facilitate easy editing and design exploration , as relations are preserved when individual elements are edited . CAD representations are heterogeneous . First , different instructions have their unique sets of parameters , with different parameter counts , data types , and semantics . Second , constraint specifications involve links ( i.e. , pointers ) to existing primitives or parts of primitives . Further , the same final shape 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can be equivalently realized with different sets of primitives and constraints . Thus , CAD models neither come in regular structures ( e.g. , image grids ) nor can they be easily encoded using a fixed length encoding . The heterogeneous nature of CAD models makes it difficult to adopt existing deep learning frameworks to train generative models . One easy-to-code solution is to simply ignore the constraints and rasterize the primitives into an image . While such a representation is easy to train , the output is either in the image domain , or needs to be converted to primitives via some form of differential rendering . In either case , the constraints and the regularization parametric primitives provide is lost , and the conversion to primitives is likely to be unreliable . A slightly better option is to add padding to the primitive and constraint representations , and then treat them as fixed-length encodings for each primitive/constraint . However this only hides the difficulty , as the content of the representation is still heterogeneous , the generator now additionally has to learn the length of the padding that needs to be added , and the increased length is inefficient and becomes unworkable for complex CAD models . We introduce SketchGen as a generative framework for learning a distribution of constrained sketches . Our main observation is that the key to working with a heterogeneous representation is a wellstructured language . We develop a language for sketches that is endowed with a simple syntax , where each sketch is represented as a sequence of tokens . We explicitly use the syntax tree of a sequence as additional guidance for our generative model . Our architecture makes use of transformers to capture long range dependencies and outputs both primitives along with their constraints . We can aggressively quantize/approximate the continuous primitive parameters in our sequences as long as the inter-primitive constraints can be correctly generated . A sampled graph can then be converted , if desired , to a physical sketch by solving a constrained optimization using traditional solvers , removing errors that were introduced by the quantization . We evaluate SketchGen on one of the largest publicly-available constrained sketch datasets SketchGraph [ 27 ] . We compare with a set of existing and contemporary works , and report noticeable improvement in the quality of the generated sketches . We also present ablation studies to evaluate the efficacy of our various design choices . 2 Related Work . Vector graphics generation . Vector graphics , unlike images , are presented in domain-specific languages ( e.g. , SVG ) and are not in a format easily utilized by standard deep learning setups . Recently , various approaches have been proposed by linking images and vectors using differentiable renderer [ 19 , 26 ] or supervised with vector sequences ( e.g. , deepSVG [ 4 ] , SVG-VAE [ 20 ] , and Cloud2Curve [ 5 ] ) . DeepSVG , mostly closely related to our work , proposes a hierarchical nonautoregressive model for generation , building on command , coordinate , and index embeddings . Structured model generation . Geometric layouts are often represented as graphs encoding relationships between geometric entities . Motivated by the success in image synthesis , several authors attempted to build on generative adversarial networks [ 17 , 22 ] or variational autoencoder [ 15 ] . Recently , autoregressive models like the transformer architecture [ 31 ] emerged as an important tool for generative modeling of layouts , e.g . [ 34 , 23 , 36 ] . Closely related to our work are DeepSVG [ 4 ] and Polygen [ 21 ] . These solutions are also related to modeling with variable length commands , but they simplify the representation so that commands are padded to a fixed length . Polygen is an auto-regressive generative model for 3D meshes that introduces several influential ideas . One of the proposed ideas , that we also build on , is to employ pointer networks [ 32 ] to refer to previously generated objects ( i.e. , linking vertices to form polygonal mesh faces ) . Unlike PolyGen , we work with heterogeneous primitives and constraints , and with edges that constrain the primitive parameters , motivating our constraint optimization step . CAD Datasets and CAD generation . Recently , several notable datasets have emerged for CAD models : ABC [ 13 ] , Fusion360 [ 37 ] , and SketchGraphs [ 27 ] . Only the latter two include constraints and only SketchGraphs introduces a framework for generative modeling . There are several papers that focus on the boundary ( surface ) representation for CAD model generation without constraints such as UV-Net [ 9 ] and BRepNet [ 14 ] . A related topic is to recover ( parse ) a CAD-like representation from unstructured input data , e.g . [ 28 , 6 , 30 , 18 , 33 , 35 , 11 , 3 , 29 , 40 , 7 , 10 ] . Concurrent work . Multiple papers , namely Computer-Aided Design as Language [ 8 ] , Engineering Sketch Generation for Computer-Aided Design [ 38 ] , DeepCAD [ 39 ] , have appeared on arXiv very recently , and should be considered to be concurrently works . 3 Overview . CAD sketches . A CAD sketch can be defined by a graph S = ( P , C ) , whereP is a set of parametric primitives , such as points , lines , and circular arcs and C is a set of constraints between the primitives , such as coincidence , parallelism , or orthogonality . See Figure 1 for an example . Some of the primitives are only used as construction aids and are not part of the final 3D CAD model ( shown dashed in Figure 1 ) . They can be used to construct more complex constraints ; in Figure 1 , for example , the center of the circle is aligned with the midpoint of the yellow line , which serves as a construction aid . Both primitives and constraints in the graph are heterogeneous : different primitives have different numbers and types of parameters and different constraints may reference a different number of primitives . Primitives P ∈ P are defined by a type and a tuple of parameters P = ( τ , κ , ρ ) , where τ is the type of primitive , κ is a binary variable which indicates if the primitive is a construction aid , and ρ is a tuple of parameters with a length dependent on the primitive type . A point , for example , has parameters ρ = ( x , y ) . See Figure 2 for a complete list of primitives and their parameter type . Constraints C ∈ C are defined by a type and a tuple of target primitives C = ( ν , γ ) , where ν is the constraint type and γ is a tuple of references to primitives with a length dependent on the constraint type . Constraints can target either the entire primitive , or a specific part of the primitive , such as the center of a circle , or the start/end point of a line . In a coincidence constraint , for example , γ = ( λ1 , µ1 , λ2 , µ2 ) refers to two primitives Pλ1 and Pλ2 , while µ1 , µ2 indicate which part of each primitive the constraint targets , or if it targets the entire primitive . CAD sketch generation with Transformers . We show how a generative model based on the transformer architecture [ 31 ] can be used to generate CAD sketches defined by these graphs . Transformers have proven very successful as generative models in a wide range of domains , including geometric layouts [ 21 , 36 ] , but require converting data samples into sequences of tokens . The choice of sequence representation is a main factor influencing the performance of the transformer and has to be designed carefully . The main challenge in our setting is then to find a conversion of our heterogeneous graphs into a suitable sequence of tokens . Will , will Will will Will Will 's will ? possesive name proper name nounverbproper name aux . verb proper name sentence VP VP NP VP VP : verb phrase NP : noun phrase To this end , we design a language to describe CAD sketches that is endowed with a simple syntax . The syntax imposes constraints on the tokens that can be generated at any given part of the sequence and can help interpreting a sequence of tokens . An extreme example is the famous natural language sentence `` Will , will Will will Will Will ’ s will ? `` , which is a valid sentence that is easier to interpret given its syntax tree , as shown in the inset . In natural language processing , syntax is complex and hard to infer automatically from a given sentence , so generative models usually only infer it implicitly in a data-driven approach . On the other end of the spectrum of syntactic complexity are geometric problems such as mesh generation [ 21 ] , where the syntax consists of repeating triples of vertex coordinates or triangle indices that can easily be given explicitly , for example as indices from 1 to 3 that denote which element of the triple a given token represents . In our case , the grammar is more complex due to the heterogeneous nature of our sketch graphs , but can still be stated explicitly . We show that giving the syntax of a sequence as additional input to the transformer helps sequence interpretation and increases the performance of our generative model for CAD sketches . We describe our language for CAD sketches in Section 4 . In this language , primitives are described first , followed by constraints . We train two separate generative models , one model for primitives that we describe in Section 5.1 , and a second model for constraints that we describe in Section 5.2 . 4 A Language for CAD Sketches . We define a formal language for CAD sketches , where each valid sentence is a sequence of tokens Q = ( q1 , q2 , . . . ) that specifies a CAD sketch . A grammar for our language is defined in Figure 2 , with production rules for primitives on the left and for constraints on the right . Terminal symbols for primitives include { Λ , Ω , τ , κ , x , y , u , v , a , b } and for constraints { Λ , ν , λ , µ , Ω } . Each terminal symbol denotes a variable that holds the numerical value of one token qi in the sequence Q . The symbols Λ and Ω are special constants ; Λ marks the start of a new primitive or constraint , while Ω marks the end of the primitive or constraint sequence . τ , ν , κ , λ , and µ were defined in Section 3 and denote the primitive type , constraint type , construction indicator , primitive reference , and part reference type , respectively . The remaining terminal symbols denote specific parameters of primitives , please refer to the supplementary material for a full description . Syntax trees . A derivation of a given sequence in our grammar can be represented with a syntax tree , where the leafs are the terminal symbols that appear in the sequence , and their ancestors are non-terminal symbols . An example of a sequence for a CAD sketch and its syntax tree are shown in Figure 1 . The syntax tree provides additional information about a token sequence that we can use to 1 ) interpret a given token sequence in order to convert it into a sketch , 2 ) enforce syntactic rules during generation to ensure generated sequences are well-formed , and 3 ) help our generative model interpret previously generated tokens , thereby improving its performance . Given a syntax tree T , we create two additional sequences Q3 and Q4 . These sequences contain the ancestors of each token from two specific levels of the syntax tree . Qx contains the ancestors of each token at depth x : Qx = ( axT ( q1 ) , a x T ( q2 ) , . . . ) , where a x T ( q ) is a function that returns the ancestor of q at level x of the syntax tree T , or a filler token if q does not have such an ancestor . Level 3 of the syntax tree contains non-terminal symbols corresponding to primitive or constraint types , such as point , line , or coincident , while level 4 contains parameter types , such as location and direction . The two sequences Q3 and Q4 are used alongside Q as additional input to our generative model . Parsing sketches . To parse a sketch into a sequence of tokens that follows our grammar , we iterate through primitives first and then constraints . For each , we create a corresponding sequence of tokens using derivations in our grammar , choosing production rules based on the type of primitive/constraint and filling in the terminal symbols with the parameters of the primitive/constraint . Concatenating the resulting per-primitive and per-constraint sequences separated by tokens Λ gives us the full sequence Q for the sketch . During the primitive and constraint derivations , we also store the parent and grandparent non-terminal symbols of each token , giving us sequences Q3 and Q4 . The primitives and constraints can be sorted by a variety of criteria . In our experiments , we use the order in which the primitives were drawn by the designer of the sketch [ 27 ] . In this order , the most constrained primitives typically occur earlier in the sequence . The constraints are arranged based on prevalence in the dataset , constraints that are used more frequently in the dataset occur earlier .

Summary: This paper proposes a sequential language to convert the CAD sketches to syntax trees. With the help of proposed grammar, each sketch is represented as a sequence of primitives (e.g. lines, arcs) and constraints (e.g. collinear, parallel, tangent). The authors propose a generative model based on the transformer architecture to generate primitives linked by constraints which could be optimized by a constraint solver, provided by Onshape. The proposed method is evaluated on two tasks: a) generating sketches from scratch, and b) auto-constraining sketches.