Vitruvion: A Generative Model of Parametric CAD Sketches

1 INTRODUCTION . Parametric computer-aided design ( CAD ) tools are at the core of the design process in mechanical engineering , aerospace , architecture , and many other disciplines . These tools enable the specification of two- and three-dimensional structures in a way that empowers the user to explore parameterized variations on their designs , while also providing a structured representation for downstream tasks such as simulation and manufacturing . In the context of CAD , parametric refers to the dominant paradigm for professionals , in which the software tool essentially provides a graphical interface to the specification of a constraint program which can then be solved to provide a geometric configuration . This implicit programmatic workflow means that modifications to the design—alteration of dimensions , angles , etc.—propagate across the construction , even if it is an assembly with many hundreds or thousands of components . At an operational level , parametric CAD starts with the specification of two-dimensional geometric representations , universally referred to as “ sketches ” in the CAD community ( Shah , 1998 ; Camba et al. , 2016 ; Choi et al. , 2002 ) . A sketch consists of a collection of geometric primitives ( e.g. , lines , circles ) , along with a set of constraints that relate these primitives to one another . Examples of sketch constraints include parallelism , tangency , coincidence , and rotational symmetry . Constraints are central to the parametric CAD process as they reflect abstract design intent on the part of the user , making it possible for numerical modifications to coherently propagate into other parts of the design . Figure 2 , which demonstrates a capability made possible by our approach , illustrates how constraints are used to express design intent : the specific dimensions of the final model ( shown on the right ) can be modified while preserving the geometric relationships between the primitives . These constraints are the basis of our view of parametric CAD as the specification of a program ; indeed the actual file formats themselves are indistinguishable from conventional computer code . This design paradigm , while powerful , is often a challenging and tedious process . An engineer may execute similar sets of operations many dozens of times per design . Furthermore , general motifs may be repeated across related parts , resulting in duplicated effort . Learning to accurately predict such patterns has the potential to mitigate the inefficiencies involved in repetitive manual design . In addition , engineers often begin visualizing a design by roughly drawing it by hand ( Fig . 1 ) . The automatic and reliable conversion of such hand drawings , or similarly noisy inputs such as 3D scans , to parametric , editable models remains a highly sought feature . In this work , we introduce Vitruvion , a generative model trained to synthesize coherent CAD sketches by autoregressively sampling geometric primitives and constraints ( Fig . 3 ) . The model employs self-attention ( Vaswani et al. , 2017 ) to flexibly propagate information about the current state of a sketch to a next-step prediction module . This next-step prediction scheme is iterated until the selection of a stop token signals a completed sample . The resultant geometric constraint graph is then handed to a solver that identifies the final configuration of the sketch primitives . By generating sketches via constraint graphs , we allow for automatic edit propagation in standard CAD software . Constraint supervision for the model is provided by the SketchGraphs dataset Seff et al . ( 2020 ) , which pairs millions of real-world sketches with their ground truth geometric constraint graphs . In addition to evaluating the model via log-likelihood and distributional statistics , we demonstrate the model ’ s ability to aid in three application scenarios when conditioned on specific kinds of context . In autocomplete , we first prime the model with an incomplete sketch ( e.g. , with several primitives missing ) and query for plausible completion ( s ) . In autoconstrain , the model is conditioned on a set of primitives ( subject to position noise ) , and attempts to infer the intended constraints . We also explore image-conditional synthesis , where the model is first exposed to a raster image of a handdrawn sketch . In this case , the model is tasked with inferring both the primitives and constraints corresponding to the sketch depicted in the image . Overall , we find the proposed model is capable of synthesizing realistic CAD sketches and exhibits potential to aid the mechanical design workflow . 2 RELATED WORK . When discussing related work , is important to note that in computer-aided design , the word “ sketch ” is a term of art that refers to this combination of primitives and constraints . In particular , it should not be confused with the colloquial use of the word “ sketch ” as has sometimes appeared in the machine learning literature , as in Ha & Eck ( 2018 ) discussed below . CAD sketch generation The SketchGraphs dataset ( Seff et al. , 2020 ) was recently introduced to target the underlying relational structure and construction operations found in mechanical design software . Specifically , the dataset provides ground truth geometric constraint graphs , and thus the original design steps , for millions of real-world CAD sketches created in Onshape.1 Initial models are trained in Seff et al . ( 2020 ) for constraint graph generation using message-passing networks , but without any learned attributes for the positions of the primitives ; the sketch configuration is determined via constraints alone , limiting the sophistication of output samples . Following SketchGraphs , several works concurrent to ours explore generative modeling of CAD sketches utilizing transformer-based architectures ( Willis et al. , 2021 ; Ganin et al. , 2021 ; Para et al. , 2021 ; Wu et al. , 2021 ) . In Willis et al . ( 2021 ) , modeling is limited to primitives , without any learned specification of geometric constraints ; Vitruvion , however , enables edit propagation in standard CAD software by outputting constraints . Similar to our approach , both primitives and constraints are modeled in Ganin et al . ( 2021 ) ; Para et al . ( 2021 ) . In contrast to our work , these do not condition on hand-drawn sketches nor consider noisy inputs when training on the constraint inference task ( which proves crucial for generalization in Section 4.4 ) . Unlike the above works that solely target 2D sketches ( including ours ) , Wu et al . ( 2021 ) attempts to model sequences of both 2D and 3D CAD commands , but , similarly to ( Willis et al. , 2021 ) , does not output sketch constraints . Geometric program synthesis CAD sketch generation may be viewed via the broader lens of geometric program synthesis . This comprises a practical subset of program synthesis generally concerned with inferring a set of instructions ( a program ) for reconstructing geometry from some input specification . Recent examples include inferring TikZ drawing commands from images ( Ellis et al. , 2018 ) and constructive solid geometry programs from voxel representations ( Sharma et al. , 2018 ) . Related to these efforts is the highly-sought ability to accurately reverse engineer a mechanical part given underspecified and noisy observations of it ( e.g. , scans , photos , or drawings ) . One of the conditional generation settings we explore in this work is training a version of our model that generates parametric sketches when given a raster hand drawing . Note that previous work on geometric program synthesis have not generally considered constraints , which are critical to the CAD application . Vector graphics generation Generative models have been successfully applied to vector graphics in recent years . Sketch-RNN ( Ha & Eck , 2018 ) is trained on sequences of human drawing strokes to learn to produce vector drawings of generic categories ( dog , bus , etc. ) . Note the common meaning of the word sketch differs slightly from its usage as a term of art in mechanical design ( as we use it in this work ) . DeepSVG ( Carlier et al. , 2020 ) models SVG icons , demonstrating flexible editing with latent space interpolation . Less recent are tracing programs , which can convert raster drawings to vector graphics and have been widely available for decades ( e.g. , Selinger ( 2003 ) ) . None of this previous work considers constraints as a mechanism for specifying design intent as performed in 1https : //www.onshape.com CAD . The image-conditional version of Vitruvion also takes as input a raster image ( hand-drawn ) and can output vector graphics . Unlike tracing , which attempts to faithfully reproduce an input image as closely as possible , our model attempts to infer the sequence of geometric primitives and constraints intended by the user , learning to ignore noise in the input . Graph-structured modeling This work is also related to modeling of graph structures , via our use of geometric constraint graphs . Graphs constitute a natural representation of relational structure across a variety of domains , including chemistry , social networks , and robotics . In recent years , message-passing networks ( Gilmer et al. , 2017 ; Duvenaud et al. , 2015 ; Kipf & Welling , 2017 ) , building off of Scarselli et al . ( 2009 ) , have been extensively applied to generative modeling of graph-structured data , such as molecular graphs ( Jin et al. , 2018 ; Liu et al. , 2018 ) . These networks propagate information along edges , learning node- or graph-level representations . While message passing between adjacent nodes can be a useful inductive bias , it can be an impediment to effective learning when communication between distant nodes is necessary , e.g. , in sparse graphs ( Kurin et al. , 2021 ) . The constraint graphs in our work are sparse , as the constraints ( edges ) grow at most linearly in the number of primitives ( nodes ) . Recent work leverages the flexible self-attention mechanism of transformers ( Vaswani et al. , 2017 ) to generate graph-structured data without the limitations of fixed , neighbor-to-neighbor communication paths , such as for mesh generation ( Nash et al. , 2020 ) . Our work similarly bypasses edge-specific message passing , using self-attention to train a generative model of geometric constraint graphs representing sketches . 3 METHOD . We aim to model a distribution over parametric CAD sketches . Each sketch is comprised of both a sequence of primitives , P , and a sequence of constraints , C. We note that these are sequences , as opposed to unordered sets , due to the ground truth orderings included in the data format ( Seff et al. , 2020 ) . Primitives and constraints are represented similarly in that both have a categorical type ( e.g. , Line , Arc , Coincidence ) as well as a set of additional parameters which determine their placement/behavior in a sketch . Each constraint includes reference parameters indicated precisely which primitive ( s ) ( or components thereof ) must adhere to it . Rather than serving as static geometry , CAD sketches are intended to be dynamic—automatically updating in response to altered design parameters . Likewise , equipping our model with the ability to explicitly constrain generated primitives serves to ensure the preservation of various geometric relationships during downstream editing . We divide the sketch generation task into three subtasks : primitive generation , constraint generation , and constraint solving . In our approach , the first two tasks utilize learned models , while the last step may be conducted by any standard geometric constraint solver.2 We find that independently trained models for primitives and constraints allows for a simpler implementation . This setup is similar to that of Nash et al . ( 2020 ) , where distinct vertex and face models are trained for mesh generation . The overall sketch generation proceeds as : pθ ( P , C | ctx ) = pθ ( C | P ) pθ ( P | ctx ) S = solve ( P , C ) ( 1 ) where : • P is a sequence of primitives , including parameters specifying their positions ; • C is a sequence of constraints , imposing mandatory relationships between primitives ; • S is the resulting sketch , formed by invoking a solve routine on the ( P , C ) pair ; and • ctx is an optional context for conditioning , such as an image or prefix ( primer ) . This factorization assumes C is conditionally independent of the context given P . For example , in an image-conditional setting , access to the raster representation of a sketch is assumed to be superfluous for the constraint model given accurate identification of the portrayed primitives and their respective positions in the sketch . Unconditional samples may be generated by providing a null context .

Summary: This paper presents a model capable of generating CAD sketches with constraints. Such sketches are typically formed as a sequence of 2D primitives (points, lines, arcs, circles) related by geometric constraints (parallelism, orthogonality, coincidence, etc). The model generates these sequences in an auto-regressive fashion, one primitive at a time, and then one constraint at a time. The model is trained on a large dataset of CAD sketches [Seff et al. 2020] that contains the necessary ordered sequences of CAD primitives and constraints.  The model is implemented as two Transformers, one to generate the sequence of primitive, and one to generate the sequence of constraints conditioned on the primitives. For the second part, a PointerNetwork mechanism is used to identify the primitives referenced by each constraint.   The model is used either to generate random CAD sketches that exhibit similar statistics as the training data (as measured by the number of primitives and the number of constraints used), or to auto-complete a partial CAD sketch by predicting missing primitives and constraints. In addition, an image-conditioned version of the model is also described, which can take as input an approximate bitmap drawing and turn it into a constrained CAD sketch.