Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: **Summary:**

The paper presents aims to present a method for decomposing a CAD sketch into a set of of _modular-concepts_ that are learned from a large corpus of CAD sketches. The modular concepts are compostions of lower-level primitives connected via constraints.

In order to do this, the authors design a DSL for expressing these modular-concepts, inspired by Lambda-Calculus. Then they use sequence encoder + vector quantized compression as a bottleneck to force the network to compress the sketch into a bunch of _concepts_. A sequence decoder takes concepts from  the concept library and instantiates them with parameters. The whole pipeline is trained end-to-end under a (permutation-equivariant) reconstruction loss, with permutation equivariance guaranteed by graph matching.
strengths_and_weaknesses: **Strengths:**

- The paper has a  very original idea - it uses previously existing building blocks like Transformers, Vector Quantization and Hungarian Matching for set-structured data - but in a creative way. I commend the authors on this.
- I found the paper very logically structured with a good introduction to the problem the authors are trying to set-up and solve. It clearly highlights how it is different from previous work [1,2], which relies on reconstruction from lower-level primitives (line, point, arc...).
I am not an expert in compilers/grammars/linguistics so I am not sure about the quality of the references in that domain, but they seem sufficient to me as an introduction to the state of the field, although they seem to a bit old judging from the years of the papers cited [3,4,5]. I cannot recite off the top of my head papers the authors might need to cite in order to get a better view of the current (~2022) landscape, but perhaps other reviewers could chime in here.
- The qualitative results are plenty and seem to support the paper well in terms of the diversity and quality  of the discovered concepts. 
- The authors use the supplementary material very well and I think there are enough details to reimplement the paper, but perhaps not enough to reproduce the results.
- The qualitative figures are top-notch at showcasing what the method does.

**Weaknesses:**
- While I commended the authors on the general writing, some writing is very stilted. For starters, the authors do not use equation numbers in all places which makes it harder to reference equations.
- Continuing with equations, the notation seems a bit lacking - using $\mathbf{q}$, $\hat{\mathbf{q}}$,${\mathbf{q'}}$ makes it a bit harder to read the figures as well as the equations. Perhaps some cleaning up would be nice - as an example p-> primitives, e-> (contextualized) embeddings,  and you can retain q and q' for unquantized and quantized versions of the concepts respectively. 
- Again continuing with equations, you mention indices implicitly often with "where the [index] iterates over ...". This often makes the sums harder to parse. It might be better to list out the indices explicitly. As an example, on L136, it is not really clear until ones reads the supplementary, what exactly $t.x$ runs over. 
- There are not a lot of comparisons but that is mostly fine as there is not a lot of code from competitors.
- The authors do not discuss the weaknesses very well - to me a major weakness would be that a) the number of concepts per model is fixed b) the number of primitives per model is fixed and that to change both one needs to retrain the model. Additionally, the authors should mention explicitly that they do not handle constraints with parameters and engage in a preprocessing step. This is not a big deal as others simply drop them.  But I believe this should be mentioned explicitly. Currently it is a throwaway line in the supplementary.
- The authors do not describe their autoregressive baseline that well - how is the encoding performed - like the current submission or like SketchGen[1] or Vitruvion[2]? Please try and explain it in greater detail.

**Minor Weaknesses:**
- The paper would benefit by explictly stating that the constraints are described by a lambda calculus and adding a citation to an introductory book on the subject. Right now it is unclear as to why constraints have a $\lambda$-dot notation.



Overall, the paper is very original, has high quality, is reasonably clear and I hope is significant in the field.

**References:** 

[1] Para W, Bhat S, Guerrero P, Kelly T, Mitra N, Guibas LJ, Wonka P. Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems. 2021 

[2] Seff, A., Zhou, W., Richardson, N., & Adams, R.P. (2021). Vitruvion: A Generative Model of Parametric CAD Sketches. ArXiv, abs/2109.14124.

[3] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

[4] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a repl. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

[5]Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini:Lifelong learning as program synthesis. In International Conference on Neural Information Processing Systems, 2018.
questions: - On P16, Fig. 8, subfig 4 from the top. We see the salmon colored concept which is basically one _Equal_ constraint. This seems sketchy. There are many other concepts that basically amount to a concept containing a single constraint. My first question is how many concepts were learned for this qualitative result, and could you please present a histogram of how many primitives/constraints each concept contains when training with different number of allowable concepts. It seems weird that the network would basically call an $\mathbb{L}^0$ type an $\mathbb{L}^1$. Does it have something to do with the data? I would like an explanation.
- The second question relates to the same subfigure - why do we have overlapping concepts - is this a drawing artifact or something significant?
- In List 1, what exactly is $b_{dash}$? I can assume it means whether to draw the line as dashed, but it would be much better to have an explanation.
- You mention type casting in P6, L214  but how exactly does that help you compare different types? Would be nice to have an example. Like if a prmitive is a line and the target is a circle, how exactly is the loss computed?
- In P5, L198, "decoding layer $dec_{param}(·)$ that is inverse of $enc_{param}(·)$ in Sec.4.1.". What exactly does inverse mean here? Do you convert logits to actual values by argmaxing and querying the unquantized value? Do you take a weighted sum of the logits? Please explain in more detail.
- Could you explain the 100% modularity in Table 1, when removing the sharpness loss? What exactly does it mean? As it is just a bunch of primitives ( constraint F1 score is too low), how exactly does the concept become 100% modular?
- (*Speculative*) This question more curiosity but would it be possible to run the method on the dataset from [1]? This dataset is a lot more complex than SketchGraphs, and it would be interesting to see the concepts that emerge from this more complex dataset.

**References:**

[1] https://github.com/deepmind/deepmind-research/tree/master/cadl
limitations: **Negative Social Impact**

I foresee no negative social impact from this work unless our Future Robot Overlords are unhappy with the concepts generated running this method.

**Technical Limitations**

I would like to delineate between the limitations of this paper versus that of the method.

_Paper:_

- The paper severely lacks good comparisons to other methods. While this is mainly because of a lack of code, it is still a limitation.
- The paper lacks comprehensive ablations - while the authors do a good job of ablating their contributions and how much they affect the modularity of their _concepts_, practioners might also want ablations relating to how many layers they need, and how to tune the VQ and how to rank concepts and so on. This does not hold me back from accepting the paper, but the paper might be improved if the authors add to the paper their negative results as well - eg that having too many codes in the VQ led to ..., or that too many layers never converged. I think adding such practical advice to the paper will make it more useful.

_Method:_

 - The limitations of this method would be having to retrain for every desired level of modularity.
 - Unless I am very wrong, I guess that the mathching part of the algorithm is somewhat unstable (The authors can correct me here). This is based on my experiences with DETR, and requires designing the cost matrices with considerable care.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper contributes an end-to-end multi-module architecture that learns design concepts from CAD sketches, a specific type of "sketches" that are composed with well-formed primitives and constraints to be used for further CAD modelling. The author(s) define a DSL to encode CAD sketches, and construct implicit and explicit representations of deep-learning based methods to detect design concepts and generate sketch graphs. The implicit and explicit representations respectively correspond to the two primary modules in the architecture, which first transforms raw sketches into learnt discrete latent representations (i.e., implicit), and subsequently a generation module takes these implicit representations and generates the final graph in an explicitly defined format.

This model is trained end-to-end on self-supervised objectives contributed by the author(s). The author(s) evaluated the models to have learnt reasonable design concepts and achieved greater auto-completing accuracy over a representative autoregressive baseline, on an established large-scale dataset of CAD sketches.

strengths_and_weaknesses: Strengths:
- Novel and well-designed architecture that both utilises latest advances in deep-learning and generates human-interpretable sketch graphs
- Detailed and well-defined DSL for encoding and interpreting CAD sketches
- Showed promising results in balancing reconstruction accuracy of graphs and ensuring modularity of outputs
- Achieved better performance on a significant auto-completing task over a representative baseline

Weaknesses:
- The greatest weakness of the current paper is the lack of explicit evaluation on the learnt design concepts (which is the primary claimed goal that the model should achieve). While I appreciate that the authors have shown extensive qualitative examples in the paper and supplementary materials, I am curious to see if a designer and/or domain expert find these learnt concepts to be useful as some of them are relatively trivial (e.g., rectangles).
- When comparatively evaluating autoregressive baselines and the proposed method on the auto-completing task, how were the "removed primitives" represented respectively? If this is done in the proposed method by replacing existing primitives with an additional 'mask' class while keeping other aspects of inputs unchanged, it might be slightly unfair to the autoregressive baseline. This is because the information of the overall length or some further details about the sketch to be auto-completed might be leaked through the masked input to the proposed method. 
- The proposed method tackles a very specific task in an expert domain. However, I believe there could be greater applications for the proposed framework of implicit / explicit representation, which makes this a more minor weakness.
- Related to the above point on the lack of generality: the current description on the exact scope of novelty in the paper is not clear. Is this implicit / explicit representation framework entirely novel at the highest level, or are there instantiations of it that already exist in other domains? I also recommend the author(s) to list the specific contribution(s) at the end of the introduction.
- The current method only works at the first order, which the author(s) acknowledged. I wonder if they can further discuss how might the framework be applied (potentially recursively) to discover higher order concepts.
questions: Most of my questions are already embedded in the Weaknesses section above. To reiterate in more directly and clearly:

- Is this implicit / explicit representation framework entirely novel at the highest level, or are there other instantiations that already exist in other domains? (i.e., Is your contribution only applying this framework to CAD sketches, or did you contribute this framework in general?)
- Did a designer and/or domain expert review the learnt concepts formally or informally? If so, did they find them to be useful?
- Would you further clarify how the "removed primitives" were represented in the auto-completion task in the proposed method?
- How might this framework be applied to discover higher order concepts?
limitations: I think it is generally reasonable to consider the targeted task of CAD sketch modelling to not have significantly negative societal impacts as the author(s) have outlined. However, one might further consider the minor impact when these models are deployed in real CAD use-cases when the user might become over-reliant on trusting the model as an absolute source-of-truth. This can have varying degrees of negative consequences depending on the final application of the CAD models, and can become harmful such as when the generated CAD model is structurally unsafe.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a method to decompose CAD sketches into sub modular concepts, for design intent parsing. Specifically, the task is treated as a problem of program library induction. The goal is to discover sketch concepts (modular structures) from raw sketches in the format of sketch graphs. DSL is formulated to enable a concise way to represent sketch concepts. There are two key factors to make concepts searching trainable, I.e., the dual representation and the parameterized structure for concepts. In particular, authors utilize a transformer-based object detection framework for implicitly detecting and encoding concepts. Then, a generation module is to explicitly generate shapes corresponding to the found concepts. Finally, a self-supervised reconstruction loss coupled with other two losses (concept quantization loss and modularity enhancement loss) are severed as objectives.

Experiments are conducted on a subset of SketchGraphs dataset. Reconstruction accuracy and sketch concept modularity are used for evaluation. The proposed method also is capable of auto-completion for CAD modeling.
strengths_and_weaknesses: Strengths

- To parse raw sketch graphs into sketch concepts to reveal design intents is meaningful. This work proposed a method to explicitly discover sketch concepts, which seemed different from shape primitives used by previous works. This work is more like to segment sketches into semantic parts (sketch concepts).

- The proposed method can be trained in a self-supervised manner, which facilities easier network training.

- The paper is organized and well-writing, I really enjoy the paper reading.

Weaknesses

- The total loss is a bit complex, with multiple losses combined. And each wights are set empirically, which potentially requires efforts to fine-tune especially when the dataset is changed. This would lead prohibition in practical usage.

- The experiments are somehow limited. It would be better to provide more baseline methods for comparison if possible. In addition, the experimental analysis is limited too, especially about the design intent interpretation. So it is unclear how the discovered sketch concepts support design intent parsing.
questions: (1) I am curious if the discovered sketch concepts are actually some advanced primitive shapes used by other generative models for CAD sketches, such as [6] [13][17]? 

(2) The input sketch is $L^0$ typed instance, i.e., CAD sketches. Could the proposed method possibly to be applied to free-hand sketches?

(3) How to learn the set of concept queries $[\bar{q}_i]$? 

(4) How library $L^1$ is defined? And what is the impact of the library size which is 1000 currently?

(5) Is the output ( a set of sketch concepts) unique for an input sketch? Since different sets of basic shapes can compose the same raw sketch as well, and multiple answers might be all correct. Is that really safe to say the discovered sketch concepts are well-aligned to groundtruth design intents with a high reconstruction score and sketch concept modularity?

(6) Is there a scalability issue as I saw most of the cases shown in this paper produce three sketch concepts for a single sketch input?
limitations: I doubt there is a scalability issue when the library size increases (1000 for the current setting), since transformer-based framework is used. As we know, the number of tokens has a significant impact on the computational cost.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel learning based approach to discover the modular concepts (i.e., modular structure) from raw CAD sketches. To tackle the problem, the authors first define a domain specific language (DSL) such that modular concepts can be represented in a network-friendly manner. 

A Trasnformer-based detection module takes in a CAD sketch sequence and outputs a a set of latent embeddings, which are further decoded to parameterized modular concepts by a generation module. The whole model is trained in an end-to-end self-supervised manner, using reconstruction loss plus regularization terms.

The authors perform experiments on a large scale CAD sketch dataset and mainly demonstrates its applications for design intent interpretation (i.e., parse modular concepts from a raw CAD sketch) and auto-completion (i.e., complete a partial CAD sketch).

strengths_and_weaknesses: Strengths:

- From the motivation side, I agree that CAD design intents, which are implicitly encoded in the CAD sketches, are best reflected in mid-level recurring patterns. It's an intellectually interesting problem whether one can discover these concepts from raw sketches. As perhaps a first attempt, this paper formulate this problem in a principled way by defining a domain specific language. It brings new knowledge to the field of ML-aided CAD design.
- The main technical contribution, from my point of view, is the concept representation that is based on the defined DSL. The proposed framework is built upon such representation and there are many interesting technical designs: 1) a DETR-like detection module in the context of CAD sketches; 2) the separation of topological structure and parameterization; 3) formulation of loss functions based on reference matrices.

Weakness:

- The design intent interpretation gives good results on some examples (Fig.3) but seem to fail in many cases (Fig. 9). For example, for those symmetric CAD sketches (e.g., Fig. 9 first row first column, sixth row first column, sixth row four column), the parsed decompositions do not respect the symmetry, even though they are relatively simple sketches. The question then is how often the method fails? It would be helpful to have a user study to see how much percentage of the results agree with human interpretation, but I understand the difficulty to do that.
- Auto completion is indeed an crucial feature in CAD design and I agree that the modular concepts should be helpful if they are used properly. Conceptually, a CAD auto completion method should be able to give multiple plausible suggestions for the user. However, this method is determinstic --- it only generates one completion results given the partial input. It doesn't seem easy to extend this work to allow multiple possible outputs, which limits its practical potentials.
- The presentation for the defined DSL could be improved. I would suggest explicitly writing out all the types for the Fig. 1 example such that readers can align the abstract concepts with graphical illustrations.

Overall, I think that the paper formulates an intellectually interesting problem in a princepled way and tries to solve it with many unique technical designs. Though the results are not good enough, making it still far from practical usage, I think they are acceptable for a first attempt in this particular problem. In addition, I do think its technical contributions are novel and beneficial to the community. 

questions: - Line 73-77, the description for $L^1$ composite type is confusing, probably because it's a abstract concept.    I can understand it conceptually but hard to match the mathematic formulation. If possible, I would suggest explicitly writing out the $L^1$ composite types in Figure 1 example such that readers can understand this concept more precisely.

- Line 134-135, "each constraint reference as a primitive index is directly embedded as a code", can you be more specific? How does a primitive index be embeded as a code? Is it via positional encoding?
- Line 184-185, "Collectively...R of shape $(2k_{qry}\cdot k_{L^0})\times(k_{qry}\cdot k_{L^0})$". How is the complete reference matrix R generated? From Line 168 and Line 175, in-concept and cross-concept reference matrices $(R_{T^1}, R_S)$ are generated separately. So is R just composed of $R_{T^1}$ and $R_S$?

- Figure 5(a), what is the dash line (on the left)? 
- Sec. 5.2, what's the motivation for using a vector quantization loss? In other words, what's the benefit of building a discrete concept code space? Additionally, There is no ablation study on this loss term. 

limitations: I think limitations are well discussed and I do not foresee any potential negative societal impact.
review_title: null

Meta-review: metareview: As summarized by reviewer 5G2f, this paper proposes a novel learning-based approach to discover the modular concepts (i.e., modular structure) from raw CAD sketches. To tackle the problem, the authors first define a domain specific language (DSL) such that modular concepts can be represented in a network-friendly manner.

A Transformer-based detection module takes in a CAD sketch sequence and outputs a a set of latent embeddings, which are further decoded to parameterized modular concepts by a generation module. The whole model is trained in an end-to-end self-supervised manner, using reconstruction loss plus regularization terms.

The authors perform experiments on a large scale CAD sketch dataset and mainly demonstrate its applications for design intent interpretation (i.e., parse modular concepts from a raw CAD sketch) and auto-completion (i.e., complete a partial CAD sketch).

All reviewers recognize the novelty and contribution of this work, and the reviewer-author discussion was quite fruitful as many points, ranging from designer/user interaction, comparison to baseline methods, and issues with the library size are discussed and addressed. With such clear contribution and applicability to the CAD domain, I highly recommend the acceptance of this work.

confidence: Certain
Decision: Accept