Discovering Design Concepts for CAD Sketches
Authors: Yuezhi Yang
Hao Pan
Venue: NeurIPS 2022 Accept (2022)
Keywords: CAD sketch graph
program library induction
neural-symbolic learning
auto-completion

Abstract: Sketch design concepts are recurring patterns found in parametric CAD sketches. Though rarely explicitly formalized by the CAD designers, these concepts are implicitly used in design for modularity and regularity. In this paper, we propose a learning based approach that discovers the modular concepts by induction over raw sketches. We propose the dual implicit-explicit representation of concept structures that allows implicit detection and explicit generation, and the separation of structure generation and parameter instantiation for parameterized concept generation, to learn modular concepts by end-to-end training. We demonstrate the design concept learning on a large scale CAD sketch dataset and show its applications for design intent interpretation and auto-completion.

Discovering Design Concepts for CAD Sketches
Yuezhi Yang' The University of Hong Kong Microsoft Research Asia yzyangâcs. hku.hh.
Hao Pan Microsoft Research Asia haopan@miccossof..cco com
Abstract
Sketch design concepts are recurring patterns found in parametric CAD sketches. Though rarely explicitly formalized by the CAD designers, these concepts are implicitly used in design for modularity and regularity. In this paper, we propose a learning based approach that discovers the modular concepts by induction over raw sketches. We propose the dual implicit-explicit representation of concept structures that allows implicit detection and explicit generation, and the separation of structure generation and parameter instantiation for parameterized concept generation, to learn modular concepts by end-to-end training. We demonstrate the design concept learning on a large scale CAD sketch dataset and show its applications for design intent interpretation and auto-completion.
Introduction
Parametric CAD modeling is a standard paradigm for mechanical CAD design nowadays. In parametric modeling, CAD sketches are fundamental 2D shapes used for various 3D construction operations. As shown in Fig. D a CAD sketch made of primitive geometric elements (e.g. lines, ares, points) which are constrained by different relationships (e.g. coincident, parallel, tangent); the sketch graph of primitive elements and constraints captures design intents, and allows adaptation and reuse of designed parts by changing parameters and updating all related elements automatically []] Designers are therefore tasked with the meticulous design of such sketch graphs, so that the inherent high-level design intents are easy to interpret and disentangle. To this end, meta-structures (Fig. [ which we call sketch concepts this paper, capture repetitive design patterns and regulate the design process with more efficient intent construction and communication [9 [2). Concretely, each sketch concept is structure that encapsulates specific primitive elements and their compositional constraints, and the interactions of its internal elements with outside only go through the interface of the concept.
How discover these modular concepts automatically from raw sketch graphs? In this paper, we cast this task as program library induction problem by formulating a domain specific language (DSL) for program, and sketch concepts are modular
sketch generation, where sketch graph is formalized functions that abstract primitive elements and compose the program (Fig. D. Discovering sketch concepts thus becomes the induction of library functions from sketch programs. While previous works address the general library induction problem via expensive combinatorial search 20 5. [7].
we present a simple end-to-end deep learning solution for sketch concepts. Specifically, we bridge the implicit and explicit representations of sketch concepts, and separate concept structure generation from parameter instantiation, so that a powerful deep network can detect and generate sketch concepts, by training with the inductive objective of reconstructing sketch with modular concepts.
We conduct experiments on large-scale sketch datasets [7]. they provide modular interpretation of design sketches. The network can also be trained on incomplete input sketches and learn to auto-complete them. Comparisons with state-of-ther -oeth--rrtt approaches that
The learned sketch concepts show that
Work done during internship Microsoft Research Asia.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
8.4g T
S.da [T]
Tangent
linger
Coine
Coinc.
Lare
Parallel Dustrne
Ciom:
Coine
Ditweg
Inarg
Coinc:
Parllll
Huiion
Are
Coinc.
Parpona
Coine
Coine.
Coins.
( my
In Mgg
(Lin
Parcerd
Colte, Caine
Line
Distance
Trrgon
Taagen
Cote:
8.by To
(b) Learned program that restructures the sketch:
To ->a0g of)- (fo:Are, t TTnngg ta, Ca:Cnine, R= (fs (to, og), (2(1,, (6) (t(aa of))
Ti -Mtag af). (f)-(tost ta..hi.t,ttiiine to. Perppeddd Ea:Parallel, Drssannnece, Hoooiioo, (= fa), tolfo: f2), fo(t,, (2), (7)1, ta). (t(1 (2): fo(t,, ta), (o0t, (in(t)) of()))
8 -+o,ff Ta: R= (to((tta.of fa.o)), t1 (ta. fa.oi)))
Figure 1: Concept learning from sketch graphs. (a) In black are the raw sketch and its constraint graph, with nodes showing primitives and edges depicting constraints. Colored are the restructured sketch and its modular constraint graph, where each module box represents a concept; primitives and constraint edges are colored according the modular concepts.
(b) The restructured sketch graph in our DSL program representation (List I.. where the whole sketch S is compactly constructed with three instances of two learned L' types. We simplify notation super/sub- scruubb--sstsssss for readability.
solve sketch graph generation through autoregressive models show that the modular sketch concepts learned by our approach enable more accurate and interpretable completion results.
To summarize, we make the following contributions in this paper:
We formulate the task of discovering modular CAD sketch concepts program library induction for a declarative DSL modeling sketch graphs.
We propose supervied deep learning framework that discovers modular libraries for the DSL with simple end-to-end training.
We show the framework learns from large-scale datasets sketch concepts capturing intuitive and reusable components, and enables structured sketch interpretation and auto-completion.
2
Related work
Concept discovery for CAD sketch It well acknowledged in the CAD design community that design intents are inherent to and implicitly encoded by the combinations of geometric primitives and constraints [ 10]. However, there is generally no easy approach to discover the intents and make them explicit, albeit through manual design of meta-templates guided by expert knowledge (9] 10] We propose automatic approach to discover such intents, by formulating the intents modular structures with coff-contained references, and learning them through self-supervised inductive training with simple objectives large raw sketch dataset. Therefore, we provide automatic approach for discovering combinatorially complex structures through end-to-end neural network learning.
Generative models for CAD sketch A series of recent works 6 24
13
18 25
use autoregressive
models 22 generate CAD sketches and constraints modeled through pointer networks (23. These works focus on learning from large datasets [17] to generate plausible layouts of geometric primitives and their constraints, which can then fine-tuned with constraint solver for more regular sketches. Different from these works, our aim is to discover modular structures (i.e. sketch concepts) from the concrete sketches. Therefore, our framework provides higher-level interpretation of raw sketches and more transparent auto-completion than these works (cf. Sec. 6).
Program library induction for CAD modeling Program library induction has been studied in the shape modeling domain [7]. General program synthesis assisted by deep learning is a research 19 (5) The library induction task specifically involves
topic with increasing popularity (2
combinatorial search, as has been handled by neural guided search 220 [S] or by pure stochastic sampling 7. We instead present an end-to-end learning algorithm for sketch concept induction. In particular, based on key observations about sketch concepts, we present implicit-explici dual representations of concept library functions, and separate the concept structure generation from parameter instantiation, to enable supervipe training with induction objectives.

---

List 1: A domain-specific language formulating CAD sketch concepts
// Basic data types
Length, Angle, Coord, Ref
// L primitive types
Line + Cstart_x: Catort_y: Cend x; Gend_y ; Coord
Circle ceenter_æy Ccenter_y Coord, fradius Length
// L constraint types Coincident + A2 Ref).(} Parallel Distance > A(11,
Length}
ILL composite types
T + X([ax (e T3 E L RTy ([H))U[ax])
// Sketch decomposition
+
[t] T. e L ((!)))
3 CAD sketch concept formulation
To capture the notion of sketch concepts precisely, formulate domain specific language (DSL)
(syntax given in List [ an exhaustive list of data types given in the supplementary). In the DSL, we first define the basic data types, including length, angle, coordinate, and the reference type, where reference binds to another reference or a primitive for modeling the constraint relationships. Second, define the L collection of primitive and constraint types given in raw sketches. In particular, we regard the constraints as functions whose arguments are the references to bind with primitives, e.g. a coincident constraint style (one may refer to 14 for introductory lambda calculus formality). Some constraints have parameters other than mere references, which are treated as variables inside, e.g. parallel distance in List Third, we define the sketch concepts as II.² types composed of T. types. To be specific, composite type T! E T__1 a function with arguments [ou] and members 29.5 T I.", which are connected through a composition operator R = (p(4)lp. E [t?]][[a]]> that specifies how each pair of primitive elements binds together. For example, a coincident constraint take line primitive q as its first argument and bind to an argument ag of the composite type as its second argument, i.e. p(q, 02.) E Ryy; on the other hand, an argument 0g may bind to a primitive q. which is specified by ar (g) E Ry. Finally, an input sketch S is restructured
X(11. F2 : Ref). where a function is represented in the lambda calculus
M211,22).(1 may
collection of composite types t/ T E L' as well as their connections specified by corresponding composition operator Rs. Rs records how different concepts bind through their arguments, which further transfers to IL typed elements inside the concepts and translates into the raw constraint relationships of the sketch graph. Fig. (b) shows an example DSL program encoding sketches and concepts.
Given the explicit formulation of CAD sketches through a DSL. the discovery sketch concepts becomes the task of learning program libraries L by induction on many sketch samples. Therefore, our task resembles shape program synthesis that aims at building modular programs for generating shapes 5 7 and differs from works that use autoregressive language models to generate CAD sketch programs one token at a time 6
[ 18 In Sec. 6.2] we show that the structured learning of CAD sketches enables more robust auto cotto-coolttionn than unstructured language modeling.
The search of structured concepts is clearly combinatorial problem with exponential complexity, which is intractable unless we can exploit the inherent patterns in large-scale sketch datasets. However, to enable deep learning based detection and search of structured concepts, we need to bridge the implicit deep representations and the explicit and interpretable structures, which we build through the following two key observations:
A concept has dual representations: implicit and explicit. The implicit representation as embeddings in latent spaces is compatible with deep learning, while the explicit representa- tion provides structures on which desired properties (e.g. modularity) can be imposed.
18 have skipped such constraints, we preserve them but omit generating the "Wilee other works 3
parameter values that can be reliably deduced from primitives. See discussions in the supplementary.

---

q
generation
Parum
Rs
instance
Encoder
Decoder
assemble
91
inpul sketch S=11
lib L
T
(a) detection module
(b) generation module
(C) loss computation
Figure 2: Framework illustration. (a) The detection module is transformer network that detects from the sketch sequence [t]] implicitly encoded concepts [q:] and their composition qıt (b) Each quantized against the concept library L' to obtain prototype q', which is expanded by structure network into an explicit structure T and further instantiated by parameter network into t'. (c) The
collection of [4]] are assembled by composition operator Rg generated from qu obtain the final generated sketch graph, which is compared with the input sketch for loss computation.
A concept is parameterized structure. A concept is composite type with fixed modular structure for interpretability, but the structure is always instantiated by assigning parameters to its component primitives when the concept is found in sketch.
3.1 Method overview
According to the two observations, we design an end-to-end sketch concept learning framework by self-supervised induction on sketch graphs. As shown Fig. 2 the framework has two main steps before loss computation: a detection step that generates implicit representations of concepts making up the input sketch, and explicit generation step that expands the implicit concepts into concrete structures which self-supervision targets like reconstruction and modularity are applied.
Building state-of- of-te-of-the-art detection architecture (2). the detection module D takes sketch S as input and detects the modular concepts within it, i.e. {q:} D(S, {q.}), where the concepts are represented implicitly latent codes {q:}, and {q.} are learnable set of concept instance queries. Notably, we apply vector quantization to the latent codes and obtain {qf = minpel' ||p - (1/12), which ensures that each concept is selected from the common collection of learnable concepts IL! used for restructuring all sketches.
The explicit generation module is separated into two sub-steps, structure generation and parameter instantiation, which ensures that the modular concept structures are explicit and reused throughout different sketch instances. Specifically, the structure network takes each quantized concept code q and generates its explicit form T! in terms of primitives and constraints of LL types along with the composition operator Ryy. Subsequently, the parameter network instantiates the concept structure by assigning parameter values each component of Ti conditioned qi and input sketch, to obtain d7
The composition operator Rg for combining {4} } is generated from special latent code qæ trans- formed by D from a learnable token In appended to {q.}.
The entire model is trained end-to-end by reconstruction and modularity objectives. In particular, we design loss functions that measure differences between the generated and groundtruth sketch graphs, in terms of both per-element attributes and pairwise references. Given our explicit modeling of encapsulated structures of the learned concepts, we can further enhance the modularity of the generation by introducing bias loss that encourages in-concept references.
End-to-end sketch concept induction
4.1 Implicit concept detection
Sketch encoding A raw sketch S can be serialized into sequence of L primitives and constraints.
Previous works have adopted slightly different schemes to encode sequence [3 [8
25)
In this paper, we build the previous works and take simple strategy akin to [ 25 for input sketch encoding. Specifically, we split each T.O typed instance 1 into several tokens: type, parameter, specific embedding module. For
and list of references. For each of the token category,

---

example, parameters as scalars are quantized into finite bins before being embedded as vectors (see supplementary for the quantization details), and since there are at most five parameters for each primitive, we pack all parameter embeddings into a single code. On the other hand, each constraint reference as a primitive index is directly embedded as a code. Therefore, each token of T." typed instance is encoded as
eyo. enCtype (19) enC (1".x) + [enCparam (C..)))).eee)))ee .e)eeecr (19 ((xx)]
(1)
where ... iterates over the split tokens (i.e., type, parameters, references). the type embedding is shared for all tokens of the instance, the position embedding counts the token index in the whole split-tokenized sequence of S, and parameter or reference embeddings are applied where applicable.
Concept detection We build the detection network as an encoder-decoder transformer following 2] The transformer encoder operates on the sketch encoded sequence (cres) and produces the contextualized sequence ( through layers self-attention and feea-forward. The transformer decoder takes a learnable set of concept queries [q.] of size kery plus a special query OR for composition generation, and applies interleaved self-attention, cross-attention to [e[g] and feed-
forward layers to obtain the implicit concept codes [q:] and qu. The concept codes are further quantized into [q]] by selecting concept prototypes from a library L implicitly encoding L', before being expanded into explicit forms.
4.2 Explicit concept structure generation
Concept structure expansion Given library code q' L' representing type T E L' through an MLP we expand its explicit structure as a collection of codes [t]] representing the type instances
[49] and a matrix representing the composition Ry. of [(]] and arguments (ef. List
We fix the maximum number of L type instances to k (12 by default), and split the arguments into two groups, inward arguments and outward arguments, each of maximum number kang (2 by default). Each type code tp is decoded into discrete probabilities over L with an additional probability for null type é to indicate the emptiness of this element (cf. Sec. (.11. by decrype (-) as the inverse of enC;ype() (.) Sec. 4. An inward argument only points to a primitive inside the concept structure and originates from constraint outside,
concent A Rg concept
primitive
constraint
and conversely an outward argument only points to primitives outside and originates from constraint inside the concept (see inset for illustration); the split into two groups cases composition computation, as discussed below.
inward arg
outward arg
The composition operator Rty is implemented as assignment matrix Ryy of shape (2kro (okzo-tkarg hirgg) x (kro +harg), where each row corresponds to constraint reference or inward argument, and each column to a primitive or outward argument. The two-fold coefficient of constraint references comes from that any constraint we considered in the dataset [17] has most two arguments. Each row is a discrete probability distribution such that E, RT [i,5] 1, with the maximum entry signifying that the i-th constraintouttward argument refers to the j-th primitive/inward argument. We compute RT by first mapping the concept code q' to a matrix of logits in the shape of RTI, and then applying softmax transform for each row. Notably, we avoid the meaningless loops of an element referring back itself, and inward arguments referring to outward arguments, by masking the diagonal blocks R [2i:2i+2, i], [k() and the argument block Ry [2kza;, kyo:] by setting their logits to -o0.
Cross-conceps composition Aside from references inside a concept, references across concepts are generated to complete the whole sketch graph. We achieve cross cross-coneppl references by argument passing (see inset above for illustration). In particular, we implement the consect composition operator Rs as an assignment matrix R,s of shape Eegry-Kary x through an MLP. Similar to the in in-concept composition matrix, each row of the cross-concept matrix is discrete distribution such that E, [i, 1, with the maximum entry signifying that the (i mod )-t outward argument of the Li/k kargl Ith concept instance refers to the (j mod Kary)-th inward argument of the [ h//krgl -th concept instance.
directly mapped from qæ
The complete cross-concepe reference is therefore the product of three transport matrices:
Rore / [t2, (t] = R2 [,2,,, k10:] x Rs (i- (i+1)) j-karg +1)-krrg) x RRy 12k sk,o),
5

---

(2)
where Rores t.] is a block assignment matrix of shape 2k xkgo Intuitively, Ro ( t)] specifies how constraints inside ! refers to primitives of l/ throughout all possible paths crossing the arguments of concepts.
Collectively, we denote the complete reference matrix for all pairs of generated L elements as R of shape (xo x (1o)) which includes in in-concept and cross-concept references.
4.3 Concept instantiation by parameter generation
Instantiating a concept structure requires assigning parameters to the components where applicable. Therefore, as shown in Fig. 2 the parameter generation network takes a concept structure T and its implicit instance encoding q as input, and produces the specific parameters for each IL typed instances inside the concept. In addition, the parameters of generated instances directly related to the input parameters of the raw sketch S, we find it improves convergence and accuracy by allowing the parameter network to attend to the input tokens.
We implement the parameter network as transformer decoder in a similar way as 12. The instance code is first expanded to 2 tokens by small MLP, which are summed with [t_] token-win to obtain the query codes. The parameter network then transforms the query codes through interleaved layers of self-attention, cross-attention to the contextualized input sequence [e/a], and feed-forward, before finally mapped to explicit parameters the form of probabilities over quantized bins, through a decoding layer perparam that is inverse of enCperam (-) in Sec. 4.1
5 Induction objectives
Without any given labels of concepts, we use the following objectives to supervise the inductive network training: sketch reconstruction, concept quantization, and modularity enhancement.
5.1 Reconstruction loss
As discussed Sec. an input sketch is restructured into a set of sketch concepts which are expanded into a graph of primitives and constraints; the generated sketch graph S is compared with the input sketch S for reconstruction supervision.
The comparison of generated and target graphs requires one-to-one correspondence between elements of the two graphs, on which the graph differences can be measured. However, it is nontrivial to find such a matching, because not only are there variable numbers of elements in the two graphs, but also both elements and references between elements must be taken into account for matching. To this end, we build cost matrix that measures the difference for each pair of generated and target elements, in terms of their attributes and references, and apply linear assignment matching the cost matrix [1] to establish the optimal correspondence between two graphs.
Cost matrix construction To compare each pair of generated element and target element, we measure their type differences, and further use type casting to interpret the generated element as the target type, so that their parameters can be compared. To account for reference differences between the two elements, we compare the reference arrows by the differences of their pointed primitives.
Specifically, given the target graph S of kage elements and the generated graph S of kg elements, build the matrix C of shape kegt x (Kary in two steps. First, for pair of target element p and generated element q, we compare their type and parameter differences by cross-entropy. We denote the cost matrix in this stage as Cury. as it accounts for the element-wise unary distances between two graphs. Second, measure the binary distances of references, for each target constraint element p and its r-th referenced primitive p.r, its distance from the generated references of element g is computed (also illustrated by inset figure):
generations
a r R[2g+r,;]
p.r
targets
cost matrix
greent primitives
orange: consiraints
Binary cost between
target constraint p and generated element
Cirylp, Σ
ΣR R(2g + r.5] x Curylpr,
1E00.1,ee$

---

where R[2g+r,,] r.j (] is the probability of taking j as its r-th reference, as predicted by the composition operation (Sec. 4.2). Intuitively. the binary cost a summation of unary costs weighted by predicted reference probabilities, where the unary costs measure how different a generated pointed primitive is from the target pointed primitive. The complete cost matrix is = Wury Cury + Whry Cory. with Mary 50, Wbry 1; we give a larger weight to the unary costs because meaningful binary costs depend on reliable unary costs in the first place, as evident in Eq. (3).
Matching and reconstruction loss Given the cost matrix C, we apply linear assignment to obtain matching S {} between 5 and S. Note that the number of elements of these two graphs can different, but we have chosen kigyy og such that the generated elements always cover the target elements. Therefore, (q) assigns matched generation qEMCS to target PES, but assigns the rest unmatched generations M' S the empty target Φ i.e. (M') 6. The loss terms for matched generations are simply the corresponding cost terms (q). g). 4 E M; for unmatched generations, supervise its type to be the empty type & and neglect its parameters or references. We denote the average loss of all generated terms as Lrecon.
Besides matching cost, we also use an additional reference loss to encourage the generated references to be sharp (i.e., R being closer to binary). This loss complements the binary costs mentioned above by making that even if the generated primitives similar, generated constraint only refers to one primitive sharply. We define the sharp reference loss as
Lsharp
Σ PES..T
log R(20 (p) + r, (r,r))
JSel
where p iterates over the target constraints Se, o (.) is the inverse mapping from target element to generation, and we skip term if p.r does not exist for constraints with one reference.
5.2
Concept quantization loss
Following
21
16
we optimize the concept code quantization against library L' with:
Log
[[sg(q:) d'll + Bl|q: sg(g()),,
(5)
Kary (e(aar
where sg(.) is the stop gradient operation. For training stability, we follow [16] and replace the first term with EMA updates of q' L'. Furthermore, improve spare code usage by reviving unused code L periodically [16] (please refer to supplementary for details).
5.3 Modularity enhancement loss
We look for modular LL' concepts that have rich and meaningful encapsulated structures, rather than arbitrary groups of L elements that rely on cross-group group references to recover the graph structures. This modularity can be enhanced by limiting the use of arguments for sketch concepts. Instead of allocating very few arguments as hard constraint, we introduce soft bias loss to encourage the restrictive use of arguments, which may still cover cases when more arguments are needed for accurate reconstruction. To be specific, we penalize the accumulated probability of elements pointing to outward arguments:
Lbias
Σ
Σ
R 120 (p)) + +r, rr kr,o),
SSal
PES (R(oo
where again p iterates over the target constraints Sc, o -1() is the inverse mapping from target element to generation, and i + kyo slices the reference probabilities to outward arguments.
5.4
Total loss
The training objective sums up losses of reconstruction, concept quantization and modularity bias:
Crotat Crecnn + WsharpCeharp
+ WogCvg + Whine Chiasr
(7)
where we empirically use weights Wyecon 1, Wsharp = 20, Woy 1, Whias 25 throughout all experiments unless otherwise specified in the ablation studies.
7

---

Results
adopt the SketchGraphs Dataset and implementation Following previous works 6 L3 18
dataset [77] which contains millions roal-world CAD sketches for training and evaluation. We filter the data by removing trivially simple sketches and duplicates, and limit the sketch complexity such that the number of primitives and constraints is within [20, 50]. As a result, we obtain around million sketches and randomly split them into 950k for training and 50k for testing. We defer network details to the supplementary and open-source code and data to facilitate future researct
Evaluation metrics We evaluate the generated sketches in terms of reconstruction accuracy and sketch concept modularity, which are the two major objectives our task. measure reconstruction accuracy by the F-scores of generated primitives and constraints, where F-score is simply the harmonic mean precision and recall. A generated primitive considered a correct match with ground-truth if its type and parameters are correct, where for the scalar parameters quantization levels. A constraint is correct if and only its type, parameter and references match ground-truth, i.e., the generated q is correct w.r.t target iff has the same type and parameters with p and the primitives q.r and p.r are correctly matched. Modularity is measured by the percentage of constraints with references entirely within the encapsulating concepts, among all correct constraints.
allow threshold of 10% of
Prabd
Criy
Dome Halzen
Cono
Haice
Colic:
Line
Core
Eipan
Tingen
Coler
Lis
Coree
Figure 3: Design intent parsing. Left: input sketches and sketches restructured with concepts. Right: raw constraint graphs and modular constraint graphs. Primitives and constraints in the restructured sketches and graphs are colored according to concepts.
Cuze
Coloe
Puplill
Ferrand
Cuts:
Figure Instances of concepts. Left: learned rectangle concepts with subtly different structures. Right: four sketches containing instances of these two concepts.
(a)
(b)
Figure 5: Auto completion. Each example shows the input partial sketch (black) and groundtruth completion (red), result of the autoregressive baseline, and our result (colored by concepts).
PRR code and data: http:p///gggtt....om/yye//eeeeee

---

Adangrost
0.54
Oure
OMr
Config Primitive Constraint Modular($})
0.00
Cery 0.993
0.290
34.2
0.35
Esharp
0.983
0.104
100
0.39
Cisas 0.992
0.743
13.3
0.42
0.25
Ours
0.994
0.766
50.8
0.50
0.60
0.70
0.90
0.50
0.00
0.20
5.50
0.30
(a) primitive
(b) constraint
Table I: Loss ablation. F-scores are reported for primitives and constraints.
Figure 6: Auto completion comparison. Plotted
are F-scores at different ratios of partial input.
6.1 Design intent interpretation
By training our model on the raw sketches with self-supervised induction losses, we obtain a result library of sketch concepts and model for design intent parsing that interprets given sketch into modular concepts and their combination. Indeed, we find the automatically discovered concepts capture natural design intents and modular structures. For example, through the restructured sketches and constraint graphs in Figs. 3 we find that our network decomposes sketches into modular structures like rectangles, line-arcs and parallel lines that align symmetrically, even though no such prior knowledge applied during training except for concept modularity. Fig. shows that a given concept can be used repetitively in different sketches. and structures with subtle differences in constraint relations can be detected and distinguished into different concepts of the library. Note that these subtle structural differences difficult to detect. We refer to the supplementary for more examples of design intent parsing and instantiation of learned concepts, well as quantitative analysis of the learned library.
subsumed in the input sketch graph, which makes them more
6.2 Auto completion
Auto-complction is critical feature of CAD modeling software for assisting designers. Given partial sketch of primitives and their constraints, auto-completion aims at complementing them with the rest primitives and constraints to form regular and well-structured designs. Therefore, our concept detection and generation approach would naturally enhance the auto-completion task with better regularity. For training and evaluation, following previous work [18), synthesize the partial input by removing a suffix of random length (up to 50%) from the sketch sequence, along with constraints that refer to the removed primitives, and make the model Iearn to generate the full sketches.
State-of aftthe--rtt methods and constraint generation models, both of which operate in an autoregressive fashion, with the constraint model conditioned on and referring (by pointers (23) to the generated primitives. Since these works use diverse sketch encodings and have publicly released code at submission time, for fair comparison, we implement the autoregressive baseline with our sketch encoding (Sec. 4.1).
24 formulate auto-completion through a combination of primitive
Fig. O compares our method with autoregressive baseline under various primitive mask ratios: our method has superior primitive and constraint accuracy than the autoregressive baseline at almost all mask ratios. This difference confirms that since our method completes sketches concept-by-cone instead of wemmiiiv--b--rrmittee
more meaningful structures are likely to be generated. Our model also gains advantage by taking primitives and constraints together input and generating primitives and constraints simultaneously, while in comparison the autoregressive baseline separates generation in two steps (primitives followed by constraints). Indeed, in practice CAD designers rarely finish all primitives first before supplementing the constraints, but rather apply constraints on partial primitives immediately whenever they form design intent. Fig shows how our approach interprets the partial inputs and completes with modular concepts (see supplementary for more examples); in comparison, the autoregressive baseline does not provide such interpretable or regular completions.
6.3 Ablation study
To evaluate the impact of different loss terms of the induction objective (Sec. 5), we train several models in the absence of these losses respectively on the auto-encoding task. The results are shown in Table. T We see that removing the binary costs Cary from reconstruction loss results in significant drop of constraint reconstruction, showing its necessity for constraint reference learning. Remov- ing sharp reference loss Labarp similarly fails constraint reference learning, although modularity

---

enhancement bias loss makes all constraint references inside concepts. Removing the modularity enhancement bias loss Liiss only results slight drop in reconstruction quality but significant drop in modularity, since without it cross-concept reference through arguments is more likely and therefore modularity suffers. We provide more ablation tests on hyper parameters like the numbers of concept queries kiry and arguments karg in the supplementary.
7
Conclusion
CAD sketch concepts are meta-structures containing primitives and constraints that define modular sub-graphs and capture design intents. By formulating the sketch concepts as program libraries of DSL, we present end-to-end approach for discovering CAD sketch concepts through library induction learning. Key to our approach are the implicit-explis representation of concepts and the separated structure generation and parameter instantiation for concept generation, which together enable the end-to-end training under self-supervised induction objectives. By training on large-scale sketch dataset, our approach enables the discovery of repetitive and modular concepts from raw sketches, and more structured and interpretable auto-completion than baseline autoregressive models.
Limitations and future work Design intents can be hierarchical [10], meaning that higher order meta- mnett-structures can be built out of lower order ones. In this sense, our framework only addresses the first order library induction, and should be extended for higher order library learning; toward this goal, we believe a progressive approach like [5] can be used with our framework as the one-step induction. In addition, similar strategies of end-to-end induction learning can be applied to constraint graphs involving 3D CAD operations or even more general programs in other domains, as long as they have similar declarative and parametric structures sketch graphs.
References
[1] Jorge D Camba, Manuel Contero, and Pedro Company. Parametric cad modeling: An analysis of strategies for design reusability. Computer-Aided Design, 74, 2016.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[3] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphies programs from hand-drawn images. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[4] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama, Write, execute, assess: Program synthesis with a repl. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[5] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the ACM SIGPLAN International Conference on Programming Language Design and Implementation. PLDI 2021, 2021.
[6] Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer-aided design as language. In Advances in Neural Information Processing Systems, volume 34, 2021.
[7] R. Kenny Jones, David Charatan, Paul Guerrero, Niloy Mitra, and Daniel Ritchie. Shapemod: Macro operation discovery for 3d shape programs. ACM Transactions on Graphics (Siggraph), 40(4), 2021.
[8] Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83-97, 1955.
[9] Sofia Kyratzi and Philip Azariadis. A constraint-based framework to recognize design intent during sketching in parametric environments. Computer-Aided Design and Applications, 18(3), 2021.
10

---

[10] Sofia Kyratzi and Philip Azariadis. Integrated design intent of 3d parametric models. Computer- Aided Design, 146, 2022.
[11] James Munkres. Algorithmss for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1):32-38, 1957.
[12] Jeffrey Otey, Pedro Company, Manuel Contero, and Jorge D. Camba. Revisiting the design intent concept in the context mechanical cad education. Computer-Aided Design and Applications, 15(1), 2018
[13] Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas J Guibas, and Peter Wonka. Sketchgen: Generating constrained cad sketches. In Advances in Neural Information Processing Systems, volume 34, 2021.
[14] Benjamin C Pierce. Types and programming languages. languagg MIT press, 2002.
[15] PTC Inc. Onshape, 2022. URL htttps::/www.onhhettttt
en/ Accessed Apr. 20, 2022.
[16] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelitty images with vq-vae-2. In Advances in Neural Information Processing Systems, volume 32, 2019.
[17] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. SketchGraphs: A large-scale dataset for modeling relational geometry in computer-aides design. In ICML 2020 Workshop on Object-Oriented Learning, 2020.
[18] Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A generative model of parametric CAD sketches. In International Conference on Learning Representutions, 2022.
[19] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations, 2019.
[20] Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning program synthesis. In International Conference Neural Information Processing Systems, 2018.
[21] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- fion Processing Systems, volume 30, 2017.
[23] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, volume 28, 2015.
[24] Karl D.D. Willis, Pradeep Kumar Jayaraman, Joseph G. Lambourne, Hang Chu, and Yewen Pu. Engineering sketch generation for computer-aided design. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2021.
[25] Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer- aided design models. In IEEE International Conference on Computer Vision (ICCV), 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Sec. 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
11

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We present deep learning algorithm rather than theorems in this paper.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We provided
the url to open-source repository Sec. 6
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We did not report error bars due to lack of computational resources for running all comparisons and ablations multiple times consistently. But we note that the contrasts between different methods are very obvious, and we did run different methods in non-uniform repetitions and have found the same conclusions.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] Please see Sec. 6
(b) Did you mention the license of the assets? [Yes] The citation [17] clearly states that the SketchGraphs dataset we use has an MIT license.
(c) Did you include any new assets either in the supplemental material or as URL.2 [Yes] We have provided the url to open-source repository Sec.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [Yes] The MIT license of [7 permits research purpose usage as done in this paper.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The CAD sketch graph data used in this paper is not personally related.
5. If you used crowdsoureing conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "zgLl2uUZxKO", "reviewer": "Reviewer_vFn4", "summary": "**Summary:**\n\nThe paper presents aims to present a method for decomposing a CAD sketch into a set of of _modular-concepts_ that are learned from a large corpus of CAD sketches. The modular concepts are compostions of lower-level primitives connected via constraints.\n\nIn order to do this, the authors design a DSL for expressing these modular-concepts, inspired by Lambda-Calculus. Then they use sequence encoder + vector quantized compression as a bottleneck to force the network to compress the sketch into a bunch of _concepts_. A sequence decoder takes concepts from  the concept library and instantiates them with parameters. The whole pipeline is trained end-to-end under a (permutation-equivariant) reconstruction loss, with permutation equivariance guaranteed by graph matching.", "questions": "- On P16, Fig. 8, subfig 4 from the top. We see the salmon colored concept which is basically one _Equal_ constraint. This seems sketchy. There are many other concepts that basically amount to a concept containing a single constraint. My first question is how many concepts were learned for this qualitative result, and could you please present a histogram of how many primitives/constraints each concept contains when training with different number of allowable concepts. It seems weird that the network would basically call an $\\mathbb{L}^0$ type an $\\mathbb{L}^1$. Does it have something to do with the data? I would like an explanation.\n- The second question relates to the same subfigure - why do we have overlapping concepts - is this a drawing artifact or something significant?\n- In List 1, what exactly is $b_{dash}$? I can assume it means whether to draw the line as dashed, but it would be much better to have an explanation.\n- You mention type casting in P6, L214  but how exactly does that help you compare different types? Would be nice to have an example. Like if a prmitive is a line and the target is a circle, how exactly is the loss computed?\n- In P5, L198, \"decoding layer $dec_{param}(·)$ that is inverse of $enc_{param}(·)$ in Sec.4.1.\". What exactly does inverse mean here? Do you convert logits to actual values by argmaxing and querying the unquantized value? Do you take a weighted sum of the logits? Please explain in more detail.\n- Could you explain the 100% modularity in Table 1, when removing the sharpness loss? What exactly does it mean? As it is just a bunch of primitives ( constraint F1 score is too low), how exactly does the concept become 100% modular?\n- (*Speculative*) This question more curiosity but would it be possible to run the method on the dataset from [1]? This dataset is a lot more complex than SketchGraphs, and it would be interesting to see the concepts that emerge from this more complex dataset.\n\n**References:**\n\n[1] https://github.com/deepmind/deepmind-research/tree/master/cadl", "limitations": "**Negative Social Impact**\n\nI foresee no negative social impact from this work unless our Future Robot Overlords are unhappy with the concepts generated running this method.\n\n**Technical Limitations**\n\nI would like to delineate between the limitations of this paper versus that of the method.\n\n_Paper:_\n\n- The paper severely lacks good comparisons to other methods. While this is mainly because of a lack of code, it is still a limitation.\n- The paper lacks comprehensive ablations - while the authors do a good job of ablating their contributions and how much they affect the modularity of their _concepts_, practioners might also want ablations relating to how many layers they need, and how to tune the VQ and how to rank concepts and so on. This does not hold me back from accepting the paper, but the paper might be improved if the authors add to the paper their negative results as well - eg that having too many codes in the VQ led to ..., or that too many layers never converged. I think adding such practical advice to the paper will make it more useful.\n\n_Method:_\n\n - The limitations of this method would be having to retrain for every desired level of modularity.\n - Unless I am very wrong, I guess that the mathching part of the algorithm is somewhat unstable (The authors can correct me here). This is based on my experiences with DETR, and requires designing the cost matrices with considerable care.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper has a  very original idea - it uses previously existing building blocks like Transformers, Vector Quantization and Hungarian Matching for set-structured data - but in a creative way. I commend the authors on this.\n- I found the paper very logically structured with a good introduction to the problem the authors are trying to set-up and solve. It clearly highlights how it is different from previous work [1,2], which relies on reconstruction from lower-level primitives (line, point, arc...).\nI am not an expert in compilers/grammars/linguistics so I am not sure about the quality of the references in that domain, but they seem sufficient to me as an introduction to the state of the field, although they seem to a bit old judging from the years of the papers cited [3,4,5]. I cannot recite off the top of my head papers the authors might need to cite in order to get a better view of the current (~2022) landscape, but perhaps other reviewers could chime in here.\n- The qualitative results are plenty and seem to support the paper well in terms of the diversity and quality  of the discovered concepts. \n- The authors use the supplementary material very well and I think there are enough details to reimplement the paper, but perhaps not enough to reproduce the results.\n- The qualitative figures are top-notch at showcasing what the method does.\n\n**Weaknesses:**\n- While I commended the authors on the general writing, some writing is very stilted. For starters, the authors do not use equation numbers in all places which makes it harder to reference equations.\n- Continuing with equations, the notation seems a bit lacking - using $\\mathbf{q}$, $\\hat{\\mathbf{q}}$,${\\mathbf{q'}}$ makes it a bit harder to read the figures as well as the equations. Perhaps some cleaning up would be nice - as an example p-> primitives, e-> (contextualized) embeddings,  and you can retain q and q' for unquantized and quantized versions of the concepts respectively. \n- Again continuing with equations, you mention indices implicitly often with \"where the [index] iterates over ...\". This often makes the sums harder to parse. It might be better to list out the indices explicitly. As an example, on L136, it is not really clear until ones reads the supplementary, what exactly $t.x$ runs over. \n- There are not a lot of comparisons but that is mostly fine as there is not a lot of code from competitors.\n- The authors do not discuss the weaknesses very well - to me a major weakness would be that a) the number of concepts per model is fixed b) the number of primitives per model is fixed and that to change both one needs to retrain the model. Additionally, the authors should mention explicitly that they do not handle constraints with parameters and engage in a preprocessing step. This is not a big deal as others simply drop them.  But I believe this should be mentioned explicitly. Currently it is a throwaway line in the supplementary.\n- The authors do not describe their autoregressive baseline that well - how is the encoding performed - like the current submission or like SketchGen[1] or Vitruvion[2]? Please try and explain it in greater detail.\n\n**Minor Weaknesses:**\n- The paper would benefit by explictly stating that the constraints are described by a lambda calculus and adding a citation to an introductory book on the subject. Right now it is unclear as to why constraints have a $\\lambda$-dot notation.\n\n\n\nOverall, the paper is very original, has high quality, is reasonably clear and I hope is significant in the field.\n\n**References:** \n\n[1] Para W, Bhat S, Guerrero P, Kelly T, Mitra N, Guibas LJ, Wonka P. Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems. 2021 \n\n[2] Seff, A., Zhou, W., Richardson, N., & Adams, R.P. (2021). Vitruvion: A Generative Model of Parametric CAD Sketches. ArXiv, abs/2109.14124.\n\n[3] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.\n\n[4] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a repl. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019\n\n[5]Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini:Lifelong learning as program synthesis. In International Conference on Neural Information Processing Systems, 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AWefkvOr9Zw", "reviewer": "Reviewer_VMk7", "summary": "The authors proposed a method to decompose CAD sketches into sub modular concepts, for design intent parsing. Specifically, the task is treated as a problem of program library induction. The goal is to discover sketch concepts (modular structures) from raw sketches in the format of sketch graphs. DSL is formulated to enable a concise way to represent sketch concepts. There are two key factors to make concepts searching trainable, I.e., the dual representation and the parameterized structure for concepts. In particular, authors utilize a transformer-based object detection framework for implicitly detecting and encoding concepts. Then, a generation module is to explicitly generate shapes corresponding to the found concepts. Finally, a self-supervised reconstruction loss coupled with other two losses (concept quantization loss and modularity enhancement loss) are severed as objectives.\n\nExperiments are conducted on a subset of SketchGraphs dataset. Reconstruction accuracy and sketch concept modularity are used for evaluation. The proposed method also is capable of auto-completion for CAD modeling.", "questions": "(1) I am curious if the discovered sketch concepts are actually some advanced primitive shapes used by other generative models for CAD sketches, such as [6] [13][17]? \n\n(2) The input sketch is $L^0$ typed instance, i.e., CAD sketches. Could the proposed method possibly to be applied to free-hand sketches?\n\n(3) How to learn the set of concept queries $[\\bar{q}_i]$? \n\n(4) How library $L^1$ is defined? And what is the impact of the library size which is 1000 currently?\n\n(5) Is the output ( a set of sketch concepts) unique for an input sketch? Since different sets of basic shapes can compose the same raw sketch as well, and multiple answers might be all correct. Is that really safe to say the discovered sketch concepts are well-aligned to groundtruth design intents with a high reconstruction score and sketch concept modularity?\n\n(6) Is there a scalability issue as I saw most of the cases shown in this paper produce three sketch concepts for a single sketch input?", "limitations": "I doubt there is a scalability issue when the library size increases (1000 for the current setting), since transformer-based framework is used. As we know, the number of tokens has a significant impact on the computational cost.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- To parse raw sketch graphs into sketch concepts to reveal design intents is meaningful. This work proposed a method to explicitly discover sketch concepts, which seemed different from shape primitives used by previous works. This work is more like to segment sketches into semantic parts (sketch concepts).\n\n- The proposed method can be trained in a self-supervised manner, which facilities easier network training.\n\n- The paper is organized and well-writing, I really enjoy the paper reading.\n\nWeaknesses\n\n- The total loss is a bit complex, with multiple losses combined. And each wights are set empirically, which potentially requires efforts to fine-tune especially when the dataset is changed. This would lead prohibition in practical usage.\n\n- The experiments are somehow limited. It would be better to provide more baseline methods for comparison if possible. In addition, the experimental analysis is limited too, especially about the design intent interpretation. So it is unclear how the discovered sketch concepts support design intent parsing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9KyYZKpahHF", "reviewer": "Reviewer_5G2f", "summary": "This paper proposes a novel learning based approach to discover the modular concepts (i.e., modular structure) from raw CAD sketches. To tackle the problem, the authors first define a domain specific language (DSL) such that modular concepts can be represented in a network-friendly manner. \n\nA Trasnformer-based detection module takes in a CAD sketch sequence and outputs a a set of latent embeddings, which are further decoded to parameterized modular concepts by a generation module. The whole model is trained in an end-to-end self-supervised manner, using reconstruction loss plus regularization terms.\n\nThe authors perform experiments on a large scale CAD sketch dataset and mainly demonstrates its applications for design intent interpretation (i.e., parse modular concepts from a raw CAD sketch) and auto-completion (i.e., complete a partial CAD sketch).\n", "questions": "- Line 73-77, the description for $L^1$ composite type is confusing, probably because it's a abstract concept.    I can understand it conceptually but hard to match the mathematic formulation. If possible, I would suggest explicitly writing out the $L^1$ composite types in Figure 1 example such that readers can understand this concept more precisely.\n\n- Line 134-135, \"each constraint reference as a primitive index is directly embedded as a code\", can you be more specific? How does a primitive index be embeded as a code? Is it via positional encoding?\n- Line 184-185, \"Collectively...R of shape $(2k_{qry}\\cdot k_{L^0})\\times(k_{qry}\\cdot k_{L^0})$\". How is the complete reference matrix R generated? From Line 168 and Line 175, in-concept and cross-concept reference matrices $(R_{T^1}, R_S)$ are generated separately. So is R just composed of $R_{T^1}$ and $R_S$?\n\n- Figure 5(a), what is the dash line (on the left)? \n- Sec. 5.2, what's the motivation for using a vector quantization loss? In other words, what's the benefit of building a discrete concept code space? Additionally, There is no ablation study on this loss term. \n", "limitations": "I think limitations are well discussed and I do not foresee any potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- From the motivation side, I agree that CAD design intents, which are implicitly encoded in the CAD sketches, are best reflected in mid-level recurring patterns. It's an intellectually interesting problem whether one can discover these concepts from raw sketches. As perhaps a first attempt, this paper formulate this problem in a principled way by defining a domain specific language. It brings new knowledge to the field of ML-aided CAD design.\n- The main technical contribution, from my point of view, is the concept representation that is based on the defined DSL. The proposed framework is built upon such representation and there are many interesting technical designs: 1) a DETR-like detection module in the context of CAD sketches; 2) the separation of topological structure and parameterization; 3) formulation of loss functions based on reference matrices.\n\nWeakness:\n\n- The design intent interpretation gives good results on some examples (Fig.3) but seem to fail in many cases (Fig. 9). For example, for those symmetric CAD sketches (e.g., Fig. 9 first row first column, sixth row first column, sixth row four column), the parsed decompositions do not respect the symmetry, even though they are relatively simple sketches. The question then is how often the method fails? It would be helpful to have a user study to see how much percentage of the results agree with human interpretation, but I understand the difficulty to do that.\n- Auto completion is indeed an crucial feature in CAD design and I agree that the modular concepts should be helpful if they are used properly. Conceptually, a CAD auto completion method should be able to give multiple plausible suggestions for the user. However, this method is determinstic --- it only generates one completion results given the partial input. It doesn't seem easy to extend this work to allow multiple possible outputs, which limits its practical potentials.\n- The presentation for the defined DSL could be improved. I would suggest explicitly writing out all the types for the Fig. 1 example such that readers can align the abstract concepts with graphical illustrations.\n\nOverall, I think that the paper formulates an intellectually interesting problem in a princepled way and tries to solve it with many unique technical designs. Though the results are not good enough, making it still far from practical usage, I think they are acceptable for a first attempt in this particular problem. In addition, I do think its technical contributions are novel and beneficial to the community. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7YEnpZ_wdKL", "reviewer": "Reviewer_Jko6", "summary": "This paper contributes an end-to-end multi-module architecture that learns design concepts from CAD sketches, a specific type of \"sketches\" that are composed with well-formed primitives and constraints to be used for further CAD modelling. The author(s) define a DSL to encode CAD sketches, and construct implicit and explicit representations of deep-learning based methods to detect design concepts and generate sketch graphs. The implicit and explicit representations respectively correspond to the two primary modules in the architecture, which first transforms raw sketches into learnt discrete latent representations (i.e., implicit), and subsequently a generation module takes these implicit representations and generates the final graph in an explicitly defined format.\n\nThis model is trained end-to-end on self-supervised objectives contributed by the author(s). The author(s) evaluated the models to have learnt reasonable design concepts and achieved greater auto-completing accuracy over a representative autoregressive baseline, on an established large-scale dataset of CAD sketches.\n", "questions": "Most of my questions are already embedded in the Weaknesses section above. To reiterate in more directly and clearly:\n\n- Is this implicit / explicit representation framework entirely novel at the highest level, or are there other instantiations that already exist in other domains? (i.e., Is your contribution only applying this framework to CAD sketches, or did you contribute this framework in general?)\n- Did a designer and/or domain expert review the learnt concepts formally or informally? If so, did they find them to be useful?\n- Would you further clarify how the \"removed primitives\" were represented in the auto-completion task in the proposed method?\n- How might this framework be applied to discover higher order concepts?", "limitations": "I think it is generally reasonable to consider the targeted task of CAD sketch modelling to not have significantly negative societal impacts as the author(s) have outlined. However, one might further consider the minor impact when these models are deployed in real CAD use-cases when the user might become over-reliant on trusting the model as an absolute source-of-truth. This can have varying degrees of negative consequences depending on the final application of the CAD models, and can become harmful such as when the generated CAD model is structurally unsafe. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Novel and well-designed architecture that both utilises latest advances in deep-learning and generates human-interpretable sketch graphs\n- Detailed and well-defined DSL for encoding and interpreting CAD sketches\n- Showed promising results in balancing reconstruction accuracy of graphs and ensuring modularity of outputs\n- Achieved better performance on a significant auto-completing task over a representative baseline\n\nWeaknesses:\n- The greatest weakness of the current paper is the lack of explicit evaluation on the learnt design concepts (which is the primary claimed goal that the model should achieve). While I appreciate that the authors have shown extensive qualitative examples in the paper and supplementary materials, I am curious to see if a designer and/or domain expert find these learnt concepts to be useful as some of them are relatively trivial (e.g., rectangles).\n- When comparatively evaluating autoregressive baselines and the proposed method on the auto-completing task, how were the \"removed primitives\" represented respectively? If this is done in the proposed method by replacing existing primitives with an additional 'mask' class while keeping other aspects of inputs unchanged, it might be slightly unfair to the autoregressive baseline. This is because the information of the overall length or some further details about the sketch to be auto-completed might be leaked through the masked input to the proposed method. \n- The proposed method tackles a very specific task in an expert domain. However, I believe there could be greater applications for the proposed framework of implicit / explicit representation, which makes this a more minor weakness.\n- Related to the above point on the lack of generality: the current description on the exact scope of novelty in the paper is not clear. Is this implicit / explicit representation framework entirely novel at the highest level, or are there instantiations of it that already exist in other domains? I also recommend the author(s) to list the specific contribution(s) at the end of the introduction.\n- The current method only works at the first order, which the author(s) acknowledged. I wonder if they can further discuss how might the framework be applied (potentially recursively) to discover higher order concepts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept