Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a sequential language to convert the CAD sketches to syntax trees. With
the help of proposed grammar, each sketch is represented as a sequence of primitives (e.g.
lines, arcs) and constraints (e.g. collinear, parallel, tangent). The authors propose a generative
model based on the transformer architecture to generate primitives linked by constraints which
could be optimized by a constraint solver, provided by Onshape. The proposed method is
evaluated on two tasks: a) generating sketches from scratch, and b) auto-constraining
sketches.
main_review: *Strengths*:   
The paper is well-written and the logic of the paper is smooth. Figure 1 and Figure 3 clearly
state the proposed sequential for representing the sketch and workflow of the generative
model. The sequential language for representing the sketches as syntax trees is a novel way to
solve the heterogeneous nature between different sketch graphs. The result shows the statistical
consistency between the data sketches and the generated sketches via the proposed generative
model. The generative constraints help obviously correcting the quantization error.

*Weakness*:
1. Would it be better to include other generative models for sketches as baselines make this paper
stronger, e.g. Sketch-RNN. In this paper, the baseline is not sufficient.
2. For your ablation study, I think adding the sequence Q3 and/or Q4 does not make a very
significant difference. This makes me question the effectiveness of using this information.
3. I think it is necessary to compare with some non-transformer-based methods to show the
effectiveness of using this architecture in the paper.
limitations_and_societal_impact: While the proposed sequential language for representing the sketches as syntax trees is
helpful for solving the heterogeneous nature between sketch graphs, and the proposed
generative model shows good quantitative results, I think the experiments in this paper are not
sufficient to show the superiority of the proposed method on CAD sketch generation. The
ablation study also does not show a very significant effect of different components of their
methods.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new CAD sketches generation method, which designs a new language for CAD primitives and constraints and proposes a transformer-based CAD sketch generator. Results on sketch generation and auto-constraining sketches show the effectiveness of the proposed method.

main_review: Strengths:

1. The paper decomposes CAD sketched into primitives and constraints and designs a new language for CAD sketches. The grammar is carefully designed.

2. The generative model achieves successful sketch generation from scratch and generating constraints for primitives.

3. The quantitative evaluation and ablation studies are sufficient. Ablation studies validate additional syntax tree information is helpful for both primitive and constraint generation.

4. The paper is well-written, and the figures are clear and helpful for understanding.

Weaknesses:

1. The primitive and constraint types are quite limited and can only generate relatively simple sketches.

2. The design of the generative model is similar to PolyGen[9], primitive generator corresponds to vertex model and constraint generator corresponds to face model. This limits the novelty of the model.

Considering the complex CAD sketch language design and limited novelty of the generative model, I regard this paper as marginally above the acceptance threshold.

***
Update:

I thank the authors for their response. The response addressed my concerns about the primitive and constraints types, and the difference between the proposed method and PolyGen.

After reading the other reviewers' comments and author rebuttals, I remain my original rating. The submission solves an interesting and challenging problem, and proposes a solid method to solve it. I recommend acceptance.
limitations_and_societal_impact: The authors discussed the limitations and failure cases in the paper.

There are not much negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a generative model for CAD sketches. The task of modeling CAD format is challenging due to the heterogeneity of the primitives and the constraints. In other words, the CAD representation lacks a common interface. To deal with this problem, the paper presents a domain-specific language enabling tokenizing a CAD sketch. Being able to represent sketch primitives and constraints allows using sequence models such as Transformers.
The task of modeling CAD sketches is decomposed into the modeling of primitives and the constraints given the set of primitives. By using the predicted constraints, a post-processing optimization can be used to correct the alignment and positioning issues with the predicted primitives.

main_review: **Originality**
The paper proposes a domain-specific language and a task decomposition to model the primitives and the constraints separately. Both approaches have been used in different domains and shown to be effective. The paper successfully combines them for CAD sketches. The presented model, training objectives, and evaluations are not novel, but reasonable choices for this task. Modeling primitives and the relations among them is a common pattern followed in “layout modeling” tasks. In the related work section, the authors make connections to relevant works. Though, the long citation list (Ln. 86) is not useful as the “CAD-like representations” statement is too vague to summarize the listed works. [1, 2, 3] should also be included.

**Quality**
- The paper follows a symbolic approach in representing the CAD sketches rather than images. It is well motivated. The proposed grammar and the language disambiguate the tokens (i.e., primitive and constraint parameters) and provide semantic information to the model. 

- Due to the quantization of the 2D space, the primitive model often makes misaligned predictions. The results are arguably good as the model requires a post-processing optimization to correct the placement of the primitives with respect to the constraints. The constraint model, on the other hand, performs well in determining the constraints between the primitives (98% constraint prediction accuracy). This enables running the optimization with a high success rate. 

- The paper presents results on the quantized data only. Training the primitive model on real-valued data could be an interesting baseline. It could alleviate the issues with positioning due to the quantization. For example, in [2], the Transformer model makes real-valued predictions for stroke positions. 

- Both for the primitives and the constraints, the paper follows a sequential approach. Is the ordering of the shapes or constraints carry useful information (see [2])? Running the same model with a set assumption would be a useful ablation. 

- How are the $Q^3$ and $Q^4$ sequences processed by the model? Consider the example illustrated in Fig. 3. The “line” label starts at index 2 and continues until index 9. Are the line embeddings (i.e., $\xi_{q_i^3}^3$) the same across these indices?  

- The paper provides an interesting model design in the introduction section (Ln. 44-49). Having it among the baseline models would strengthen the story of the paper. Having the syntax labels as part of the padded representation (see my previous comment regarding $Q^3$ and $Q^4$), I do not see a reason for why this baseline fails. 

- Ablations: In Table 3, the model is evaluated with and without $Q^3$ and $Q^4$ input sequences. I do not think the difference is significant. What is the difference between NLLs of 79.94 and 79.96? 
- Qualitative comparison: The paper does not present qualitative results for the baseline SG-sketch. 
- NLL metric for the generated sketches: How is it calculated for the sketches generated from scratch? 
- The paper does not provide a measure for the positioning errors. Decomposing it into prediction (i.e., incorrect bin) and quantization errors (i.e., correct bin prediction) could be a useful metric. 

[1] Ellis et al., Learning to Infer Graphics Programs from Hand-Drawn Images 

[2] Aksan et al., CoSE: Compositional Stroke Embeddings

[3] Wang et al., Deep Convolutional Priors for Indoor Scene Synthesis

Note: I tried to open the manuscript both in Adobe Reader and Chrome browser. There was no gray background in the figures for me. I interpreted the results with failures as the ones with a gray background.

**Clarity**
The paper is well organized. I find the grammar notation cluttered, and hence it is not straightforward to follow section 4. I took several passes. A toy example (like the one in Fig. 1) with numbers along with the symbols could be useful. 

**Significance**
It is an interesting and rather new problem for the community. The proposed approach is reasonable, yet it is not evaluated thoroughly due to the lack of self-comparisons. As I mentioned under the **Quality** section, more evidence could be provided to support the design choices. The provided ablations are not very informative. 

-------------------
**Update:**
I thank the authors for their response. After reading the other reviews and the rebuttal, I decided not to chance my rating. The paper presents an interesting problem and a technically solid approach. The submission could be stronger by supporting some of the design choices or the claims with experimental evidence. 
limitations_and_societal_impact: The limitations of the work are addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a deep generative model for widely used computer-aided design (CAD) sketches, which are typically composed of parametric primitives (e.g. line, arc) and geometric constraints (e.g. coincidence, parallelism). The problem is challenging because of the complexity of the sketch graph and the heterogeneous nature of the primitives and constraints. To this end, the authors carefully develop a language for CAD sketches to represent each sketch as a sequence of tokens, and then make use of Transformers network to model the data distribution in an autoregressive manner. Evaluation is carried out on unconditional generation and sketch auto-constraining. The generated CAD sketch can be further optimized via an external solver to meet the generated constraints.

main_review: The novelty of this work is clear to me, as a first attempt on generative modeling for engineering CAD sketches (together with some concurrent works), it leverages the similarity between CAD sketches and natural language processing and develops a language to properly represent both parametric primitives and geometric constraints. 

The technical part looks sound and there are some interesting design choices, such as the three levels of syntax tree (line 157-158) and non-uniform quantization (line 179-180). Also, the post-processing with an external constraint solver further improves the results,  which is not considered in prior works.

My main concerns are as follows:
1. Although learning a generative model for CAD sketches of engineering standard is interesting, random generation from scratch seems to have less practical importance. I can hardly think of a situation where random CAD sketch generation would be useful. On the other hand, auto-constraining is a more compelling use case, but is less touch upon in the paper. Besides, generation on conditional inputs (e.g. image, partial sketch) is another way to go. It will be better if the author could show some results on that.
2. For the comparison part, it would be better to compare with more previous sketch generation works (e.g. DeepSVG), as the SG-sketch method is not a very strong baseline. I understand that DeepSVG does not model the constraints, but it's still possible to purely compare the sketch quality (using their proposed metrics). 
3. All the metrics are in the parametric space. Since the CAD language finally results in a visual 2D sketch, it would be better to include some measurements in the image space, e.g. the image reconstruction error.
4. Another minor point is that the test set seems small, 86K out of 2.4M (3.6%). It might result in much variance during different training runs.

Some confusing details:
1. In line 168-169, what exactly does "the frequency of constraints" refer to? Is it the frequency in the whole dataset or the frequency in the single sketch.
2. How is the token "sub" represented in the constraint generator? By definition, it should indicate some part of a primitive. I'm a bit confused how exactly it is represented as the pointer seems to refer only an entire primitive.


limitations_and_societal_impact: Suggestions as above. Mostly, I hope the comparison to DeepSVG can be included.
review_title: null

Meta-review: The paper contributes a generative model for CAD sketches. In particular, the work proposes a language to address the heterogeneity in sketch primitives and constraints, which allows a CAD sketch to be tokenized such that it can be handled using sequence models (such as Transformers) followed by a constraint optimizer. Overall, the reviewers are positive about the paper. The reviewers think the work is well motivated and the paper is clearly written. While several reviewers think the experiments can be strengthened with more comparison, the reviewers generally agree the work is novel and makes reasonable modeling choices, which are valuable to the field. The reviewers raised a set of questions for clarification, which were mostly addressed by the authors in the rebuttal. The authors should further address these points in the revision.

 
Decision: Accept (Poster)