Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a learning-based model to generate 2D sketches that are widely used in 3D CAD applications. It considers 2D sketches consisting of entities (lines, curves, etc.) and constraints (mirror, coincide, perpendicular, etc.), a representation that is commonly used in modern CAD software. By design, 2D sketches are structured objects. To handle these structured data, this paper presents a linear representation using Protocol Buffers, which flattens the structured data into a sequence of triplets. Through this representation, the paper establishes similarities between natural language modeling and 2D sketches and proposes to use Transformer to train a generative model that captures the distribution of sketches in a large dataset. The paper reports quantitative and qualitative performance of the generative model.
main_review: This paper is dealing with a practical problem that has wide industrial applications. The high-level idea behind the proposed technical method (bridging 2D sketches and languages) makes sense to me. The quantitative and qualitative evaluation seems OK, although I think a few more experiments could be added to make the paper stronger (see below). Overall, I think this paper has contributed some technical ideas that this community can benefit from. However, I still have a few questions and would like to make my final decision after reading the authors’ answers:
- A high-level comment on the motivation and application of the technical method: the paper attempts to make an analogy between 2D sketches and natural language modeling. They are indeed similar in some way but also have differences. In particular, 2D sketches in mechanical designs have a low tolerance of inaccurate or imprecise designs, while imprecise natural languages are quite acceptable in many applications. As an example, 2D sketches of a pair of a nut and a bolt must have compatible radii, but a generative model like this paper might generate designs with roughly the same radii which, while looking similar, won’t work at all in practice. To what extent can we extend this Transformer idea to generate precise 2D sketches that are really usable in real-world mechanical design applications?
- The paper mentioned that long sequences of data are a challenge, motivating the usage of PB messages. From what I understand, the proposed data representation still duplicates geometric information, e.g., line 4-5 and line 11-12 in Table 1 store the same point (x, y) twice. Why not separate topology and geometry information like in a boundary representation (B-rep), or did I misunderstand your data representation? What is fundamentally novel in your Protocol Buffer representation if I compare it with B-rep?
- The paper mentioned “from coffee mugs to sports cars” twice in the main paper and its supplemental material, yet the 2D sketches shown in the paper are not even close to the complexity of a sports car. In particular, line 513-515 seems to imply some of your sketches are from sports cars. If you have impressive results from these sophisticated models, please add them to the main paper, otherwise, I would suggest toning down this description and state what examples your 2D sketches can handle in a more factual way.
- One thing that makes 2D sketches really useful in CAD applications is that they are parametric designs: the (entity, constraint) combo not only defines a particular design itself but also provides a family of similar designs if you perturb the continuous parameters (e.g., length of a line, angle of an arc) in these entities. In practice, it is surprisingly not uncommon for these perturbations to result in invalid sketches that break some constraints if constraints are not defined properly. I would love to see some small experiments on the 2D sketches generated by this paper, i.e., mildly perturbing the entity parameters in the resultant 2D sketches, and see how often they lead to invalid sketches that break the constraints. I understand this may be out of the scope of this paper, but I think having such an experiment would be helpful for us to fully understand how well these generated 2D sketches can be used for parametric designs.

limitations_and_societal_impact: Please see my review above for limitations. I do not see any major issues with negative societal impact from this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel method to represent structured CAD sketch objects via Protocol Buffers
and demonstrate its generalization ability across the heterogeneous nature of sketches. They
also proposed a transformer-based generative model for generating the CAD sketches and evaluate
its effectiveness on carefully collected and preprocessed data which consists of over 4.7M
sketches. They show the capability of the generative model for both unconditional and conditional
generation tasks.
main_review: *Strengths*:
1. In general, the motivation of the paper is quite strong. There are few learning-based methods
for generating CAD sketches.
2. The proposed data representation costs less space than the classic CAD sketch format by
removing unnecessary information.
3. The proposed triplet and byte representation overcome the necessity of converting the raw
sketch message into a text format that contains both the structure and content of the data.
4. The quantitative (Table 2) and qualitative (Figure 3) experiment results demonstrate the better
performance of the triple representations than byte ones.
5. The predicted constraints by the model are reasonable for making a closed-shape CAD sketch.
6. The proposed representation methods lead to a high generation quality for both conditional
and unconditional generation tasks.

*Weakness*:
1. The metric for evaluating the performance of the proposed method is not sufficient. How about the 
fidelity and diversity of the generation with respect to the training dataset? Those should be quantitatively evaluated also.
2. The paper lack comparison with some existing generative models for sketch generation, such as Sketch-RNN.
3. I think it is necessary to compare the proposed representation with the naïve text format.
4. Line 141 has a typo. No Figure 1b.
limitations_and_societal_impact: This paper shows strong motivation for automated CAD sketches generation. The proposed
sequential data representation takes the advantage of using the transformer-based architecture
for the sketch generation. Yet, this paper lacks some important evaluation metrics and
baseline comparisons, which is the main reason for my rating not being higher. 
The experiment results show the high generation quality of the CAD
sketches generation for both conditional, and unconditional tasks.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides a large dataset of structured 2D sketches and proposes transformer-based models capable of both unconditional and conditional generations of such sketches. The authors present a novel representation for structured sketches based on Protocol Buffers, encoding both objects and the constraint between them, and further devise a custom tokenization scheme to adapt it for the training and inference using a transformer decoder architecture, showing advantages compared to a simple byte tokenization. Their model can successfully generate valid structured sketches from scratch, and moreover infer valid objects and constraints from an input image (as a condition).
main_review: The paper is mostly well-written and easy to follow. The proposed coding of structured 2D sketches is novel and seems useful, particularly for an effective correction of predicted objects based on constraints. The dataset, which the authors plan to make public, could be useful to other applications due to its quality and structured labeling, and the experiments, though not extensive, show that their proposed models work as expected. I have a few concerns and questions that I will elaborate below:

1. Run-time and train-time of the models are missing, and this makes comparisons between models and an overall efficiency/practicality evaluation hard. For example, how far is the generation (unconditional and conditional) from real-time performance?

2.  Comparison to similar models is missing, that is, how does the model compare in quality and speed with other vector image generation methods (conditional and unconditional)? (an extension of table 4 in appendix H).

3. Whenever reporting average, it is good practice to report standard deviation as well, to provide a sense of confidence.

4. It is important to highlight/discuss the types of failures (wrong shapes and wrong constraints) and the rates with which each happen, to give a better picture of the models’ capabilities.  Also, I think discussion of model failure, such as the compression failure discussed in appendix G, are very valuable to the more broad transformer research, and are a good idea to include and elaborate on in the main text.

Side note: wrong syntax in Eq.1.
limitations_and_societal_impact: Ok
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a deep learning model of automatically generating computer-aided design (CAD) sketches, which are typically composed of entities (e.g. line and arc) and constraints (e.g. tangent and mirror). The authors devise a method to describe CAD sketches as serialized Protocol Buffers, and further represent them as sequences of triplet tokens. Then they combine the devised data representation with language modeling techniques (Transformer + PointerNet) to capture the data distribution in an auto-regressive manner. The proposed method is evaluated mainly on unconditional generation and various design choices are ablated. Additional generation results are shown conditioned on binary images.
main_review: The novelty of this work is clear to me. As one of very first attempts on generative modeling for engineering CAD sketches (there are some concurrent works), it devise a method to represent CAD sketch as a sequence of triplet tokens and combine this representation with general language modeling technique to capture both the geometric primitives and constraints. The proposed special embedding and pointer strategy for its triplet representation is also interesting. 

The results in general look good and I like the colored visualization. Besides unconditional generation, one of the potential application, image-conditional generation is also thoroughly evaluated. 

My concerns and suggestions are as follows:
1. It does not compare to any prior works. I understand it's the very first work for CAD sketch generation, but there are prior works (e.g. DeepSVG) that generates vector graphics, which are also parametric sketches but without constraints. The authors can just compare results in the image domain. The same applies to the image-conditioned generation.
2. The test set is too small. A test set of 50K is only 1% compared to a training set of 4.6M (line 239-241). This could result in much variance. 90%-10%-10% is the general practice. I hope the author to use a larger test set in the revision. If not, it's better to explain why. 
3. Consider to cite concurrent works on CAD generation: 
    - DeepCAD: A Deep Generative Network for Computer-Aided Design Models
    - SketchGen: Generating Constrained CAD Sketches
4. What exactly is external sketch solver (line 288)? I guess it's some sort of constraint optimizer. It's better to give some detailed descriptions (including in the supplement is OK).

Lastly, will the dataset be released? A large dataset is always beneficial to the community.

limitations_and_societal_impact: Suggestions are as above.
review_title: null

Meta-review: This paper presents a new approach to generating constrained CAD sketches. The key challenge in this problem is generating constraints that relate all the different strokes in a sketch. At a high level, the main idea in this paper is to treat this as a language generation problem, since the sketches with their constraints can be represented in a language. The key new idea for this paper is that instead of learning directly in the language of CAD sketches, they first encode the strings to a very compact and dense representation, which allows the network to learn more efficiently. Versions of this idea have been used in other settings but the idea is new in this context. 

Additionally, the paper introduces a new dataset for this problem which should be of significant value to the community.

Decision: Accept (Poster)