SketchGen: Generating Constrained CAD Sketches

1 Introduction . Computer Aided Design ( CAD ) tools are popular for modeling in industrial design and engineering . The employed representations are popular due to their compactness , precision , simplicity , and the fact that they are directly ‘ understood ’ by many fabrication procedures . CAD models are essentially a collection of primitives ( e.g. , lines , arcs ) that are held together by a set of relations ( e.g. , collinear , parallel , tangent ) . Complex shapes are then formed by a sequence of CAD operations , forming a base shape in 2D that is subsequently lifted to 3D using extrusion operations . Thus , at the core , most CAD models are coplanar constrained sketches , i.e. , a sequence of planar curves linked by constraints . For example , CAD models in some of the largest open-sourced datasets like Fusion360 [ 37 ] and SketchGraphs [ 27 ] are thus structured , or sketches drawn in algebraic systems like Cinderella [ 1 ] . Such sequences are not only useful for shape creation , but also facilitate easy editing and design exploration , as relations are preserved when individual elements are edited . CAD representations are heterogeneous . First , different instructions have their unique sets of parameters , with different parameter counts , data types , and semantics . Second , constraint specifications involve links ( i.e. , pointers ) to existing primitives or parts of primitives . Further , the same final shape 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can be equivalently realized with different sets of primitives and constraints . Thus , CAD models neither come in regular structures ( e.g. , image grids ) nor can they be easily encoded using a fixed length encoding . The heterogeneous nature of CAD models makes it difficult to adopt existing deep learning frameworks to train generative models . One easy-to-code solution is to simply ignore the constraints and rasterize the primitives into an image . While such a representation is easy to train , the output is either in the image domain , or needs to be converted to primitives via some form of differential rendering . In either case , the constraints and the regularization parametric primitives provide is lost , and the conversion to primitives is likely to be unreliable . A slightly better option is to add padding to the primitive and constraint representations , and then treat them as fixed-length encodings for each primitive/constraint . However this only hides the difficulty , as the content of the representation is still heterogeneous , the generator now additionally has to learn the length of the padding that needs to be added , and the increased length is inefficient and becomes unworkable for complex CAD models . We introduce SketchGen as a generative framework for learning a distribution of constrained sketches . Our main observation is that the key to working with a heterogeneous representation is a wellstructured language . We develop a language for sketches that is endowed with a simple syntax , where each sketch is represented as a sequence of tokens . We explicitly use the syntax tree of a sequence as additional guidance for our generative model . Our architecture makes use of transformers to capture long range dependencies and outputs both primitives along with their constraints . We can aggressively quantize/approximate the continuous primitive parameters in our sequences as long as the inter-primitive constraints can be correctly generated . A sampled graph can then be converted , if desired , to a physical sketch by solving a constrained optimization using traditional solvers , removing errors that were introduced by the quantization . We evaluate SketchGen on one of the largest publicly-available constrained sketch datasets SketchGraph [ 27 ] . We compare with a set of existing and contemporary works , and report noticeable improvement in the quality of the generated sketches . We also present ablation studies to evaluate the efficacy of our various design choices . 2 Related Work . Vector graphics generation . Vector graphics , unlike images , are presented in domain-specific languages ( e.g. , SVG ) and are not in a format easily utilized by standard deep learning setups . Recently , various approaches have been proposed by linking images and vectors using differentiable renderer [ 19 , 26 ] or supervised with vector sequences ( e.g. , deepSVG [ 4 ] , SVG-VAE [ 20 ] , and Cloud2Curve [ 5 ] ) . DeepSVG , mostly closely related to our work , proposes a hierarchical nonautoregressive model for generation , building on command , coordinate , and index embeddings . Structured model generation . Geometric layouts are often represented as graphs encoding relationships between geometric entities . Motivated by the success in image synthesis , several authors attempted to build on generative adversarial networks [ 17 , 22 ] or variational autoencoder [ 15 ] . Recently , autoregressive models like the transformer architecture [ 31 ] emerged as an important tool for generative modeling of layouts , e.g . [ 34 , 23 , 36 ] . Closely related to our work are DeepSVG [ 4 ] and Polygen [ 21 ] . These solutions are also related to modeling with variable length commands , but they simplify the representation so that commands are padded to a fixed length . Polygen is an auto-regressive generative model for 3D meshes that introduces several influential ideas . One of the proposed ideas , that we also build on , is to employ pointer networks [ 32 ] to refer to previously generated objects ( i.e. , linking vertices to form polygonal mesh faces ) . Unlike PolyGen , we work with heterogeneous primitives and constraints , and with edges that constrain the primitive parameters , motivating our constraint optimization step . CAD Datasets and CAD generation . Recently , several notable datasets have emerged for CAD models : ABC [ 13 ] , Fusion360 [ 37 ] , and SketchGraphs [ 27 ] . Only the latter two include constraints and only SketchGraphs introduces a framework for generative modeling . There are several papers that focus on the boundary ( surface ) representation for CAD model generation without constraints such as UV-Net [ 9 ] and BRepNet [ 14 ] . A related topic is to recover ( parse ) a CAD-like representation from unstructured input data , e.g . [ 28 , 6 , 30 , 18 , 33 , 35 , 11 , 3 , 29 , 40 , 7 , 10 ] . Concurrent work . Multiple papers , namely Computer-Aided Design as Language [ 8 ] , Engineering Sketch Generation for Computer-Aided Design [ 38 ] , DeepCAD [ 39 ] , have appeared on arXiv very recently , and should be considered to be concurrently works . 3 Overview . CAD sketches . A CAD sketch can be defined by a graph S = ( P , C ) , whereP is a set of parametric primitives , such as points , lines , and circular arcs and C is a set of constraints between the primitives , such as coincidence , parallelism , or orthogonality . See Figure 1 for an example . Some of the primitives are only used as construction aids and are not part of the final 3D CAD model ( shown dashed in Figure 1 ) . They can be used to construct more complex constraints ; in Figure 1 , for example , the center of the circle is aligned with the midpoint of the yellow line , which serves as a construction aid . Both primitives and constraints in the graph are heterogeneous : different primitives have different numbers and types of parameters and different constraints may reference a different number of primitives . Primitives P ∈ P are defined by a type and a tuple of parameters P = ( τ , κ , ρ ) , where τ is the type of primitive , κ is a binary variable which indicates if the primitive is a construction aid , and ρ is a tuple of parameters with a length dependent on the primitive type . A point , for example , has parameters ρ = ( x , y ) . See Figure 2 for a complete list of primitives and their parameter type . Constraints C ∈ C are defined by a type and a tuple of target primitives C = ( ν , γ ) , where ν is the constraint type and γ is a tuple of references to primitives with a length dependent on the constraint type . Constraints can target either the entire primitive , or a specific part of the primitive , such as the center of a circle , or the start/end point of a line . In a coincidence constraint , for example , γ = ( λ1 , µ1 , λ2 , µ2 ) refers to two primitives Pλ1 and Pλ2 , while µ1 , µ2 indicate which part of each primitive the constraint targets , or if it targets the entire primitive . CAD sketch generation with Transformers . We show how a generative model based on the transformer architecture [ 31 ] can be used to generate CAD sketches defined by these graphs . Transformers have proven very successful as generative models in a wide range of domains , including geometric layouts [ 21 , 36 ] , but require converting data samples into sequences of tokens . The choice of sequence representation is a main factor influencing the performance of the transformer and has to be designed carefully . The main challenge in our setting is then to find a conversion of our heterogeneous graphs into a suitable sequence of tokens . Will , will Will will Will Will 's will ? possesive name proper name nounverbproper name aux . verb proper name sentence VP VP NP VP VP : verb phrase NP : noun phrase To this end , we design a language to describe CAD sketches that is endowed with a simple syntax . The syntax imposes constraints on the tokens that can be generated at any given part of the sequence and can help interpreting a sequence of tokens . An extreme example is the famous natural language sentence `` Will , will Will will Will Will ’ s will ? `` , which is a valid sentence that is easier to interpret given its syntax tree , as shown in the inset . In natural language processing , syntax is complex and hard to infer automatically from a given sentence , so generative models usually only infer it implicitly in a data-driven approach . On the other end of the spectrum of syntactic complexity are geometric problems such as mesh generation [ 21 ] , where the syntax consists of repeating triples of vertex coordinates or triangle indices that can easily be given explicitly , for example as indices from 1 to 3 that denote which element of the triple a given token represents . In our case , the grammar is more complex due to the heterogeneous nature of our sketch graphs , but can still be stated explicitly . We show that giving the syntax of a sequence as additional input to the transformer helps sequence interpretation and increases the performance of our generative model for CAD sketches . We describe our language for CAD sketches in Section 4 . In this language , primitives are described first , followed by constraints . We train two separate generative models , one model for primitives that we describe in Section 5.1 , and a second model for constraints that we describe in Section 5.2 . 4 A Language for CAD Sketches . We define a formal language for CAD sketches , where each valid sentence is a sequence of tokens Q = ( q1 , q2 , . . . ) that specifies a CAD sketch . A grammar for our language is defined in Figure 2 , with production rules for primitives on the left and for constraints on the right . Terminal symbols for primitives include { Λ , Ω , τ , κ , x , y , u , v , a , b } and for constraints { Λ , ν , λ , µ , Ω } . Each terminal symbol denotes a variable that holds the numerical value of one token qi in the sequence Q . The symbols Λ and Ω are special constants ; Λ marks the start of a new primitive or constraint , while Ω marks the end of the primitive or constraint sequence . τ , ν , κ , λ , and µ were defined in Section 3 and denote the primitive type , constraint type , construction indicator , primitive reference , and part reference type , respectively . The remaining terminal symbols denote specific parameters of primitives , please refer to the supplementary material for a full description . Syntax trees . A derivation of a given sequence in our grammar can be represented with a syntax tree , where the leafs are the terminal symbols that appear in the sequence , and their ancestors are non-terminal symbols . An example of a sequence for a CAD sketch and its syntax tree are shown in Figure 1 . The syntax tree provides additional information about a token sequence that we can use to 1 ) interpret a given token sequence in order to convert it into a sketch , 2 ) enforce syntactic rules during generation to ensure generated sequences are well-formed , and 3 ) help our generative model interpret previously generated tokens , thereby improving its performance . Given a syntax tree T , we create two additional sequences Q3 and Q4 . These sequences contain the ancestors of each token from two specific levels of the syntax tree . Qx contains the ancestors of each token at depth x : Qx = ( axT ( q1 ) , a x T ( q2 ) , . . . ) , where a x T ( q ) is a function that returns the ancestor of q at level x of the syntax tree T , or a filler token if q does not have such an ancestor . Level 3 of the syntax tree contains non-terminal symbols corresponding to primitive or constraint types , such as point , line , or coincident , while level 4 contains parameter types , such as location and direction . The two sequences Q3 and Q4 are used alongside Q as additional input to our generative model . Parsing sketches . To parse a sketch into a sequence of tokens that follows our grammar , we iterate through primitives first and then constraints . For each , we create a corresponding sequence of tokens using derivations in our grammar , choosing production rules based on the type of primitive/constraint and filling in the terminal symbols with the parameters of the primitive/constraint . Concatenating the resulting per-primitive and per-constraint sequences separated by tokens Λ gives us the full sequence Q for the sketch . During the primitive and constraint derivations , we also store the parent and grandparent non-terminal symbols of each token , giving us sequences Q3 and Q4 . The primitives and constraints can be sorted by a variety of criteria . In our experiments , we use the order in which the primitives were drawn by the designer of the sketch [ 27 ] . In this order , the most constrained primitives typically occur earlier in the sequence . The constraints are arranged based on prevalence in the dataset , constraints that are used more frequently in the dataset occur earlier .

Summary: The paper proposes a generative model for CAD sketches. The task of modeling CAD format is challenging due to the heterogeneity of the primitives and the constraints. In other words, the CAD representation lacks a common interface. To deal with this problem, the paper presents a domain-specific language enabling tokenizing a CAD sketch. Being able to represent sketch primitives and constraints allows using sequence models such as Transformers. The task of modeling CAD sketches is decomposed into the modeling of primitives and the constraints given the set of primitives. By using the predicted constraints, a post-processing optimization can be used to correct the alignment and positioning issues with the predicted primitives.