Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a generator for constrained CAD sketches that can also be used to convert hand-drawn line drawings to constrained CAD sketches or to auto-complete partial sketches. Two transformer models are used, the first model generates CAD primitives, optionally conditioned on a line drawing image or on a partial CAD sketch, and the second model generates constraints conditioned on the generated primitives. The authors show good qualitative results and quantitatively improved CAD generation performance over two simple baselines.
main_review: The SketchGraphs dataset has resulted in quite a few concurrent CAD sketch generation methods that roughly follow a similar approach, with a few variations. The proposed method differs from prior work SketchGraphs and Willis et al. in its generation approach, and its ability to generate explicit constraints, respectively, and from both prior and concurrent work (SketchGen, DeepCAD and Ganin et al.) in its ability to successfully infer a CAD sketches from hand-drawn images. I would consider this to be a good contribution, but unfortunately the authors do not compare their results to any of these methods, making it hard to judge the relative quality of the generated CAD sketches. Comparing the qualitative results manually across papers, does seem to show at least a similar level of quality as the concurrent work, and better quality than prior work, so I lean towards accept, conditioned on the authors including at least qualitative comparisons in their revision, and ideally quantitative comparisons as well.

Details (in order of importance):
- A comparison to prior work (SketchGraphs and ideally also Willis et al.) is needed. While both have a different generation approach than the proposed method, it should be possible to compare the constraint NLL and the full sketch NLL to SketchGraphs (as shown in Para et al. and Willis et al.), the primitive NLL to Willis et al., and the distributional statistics of generated sketches (possibly augmented with a few additional statistics to batter capture primitive and constraint quality) to both SketchGraphs and Willis et al. The qualitative results look quite good, so I would expect the quantitative results to be favorable as well. A comparison to concurrent work would be welcome, but is not necessary.
- The following concurrent work is missing and should be discussed briefly in the related work:
DeepCAD: A Deep Generative Network for Computer-Aided Design Models, Wu et al., ICCV 2021
- The exposition is generally very clear, I am just missing two pieces of information that I can't find mentioned explicitly:
--- Are types of primitives and constraints represented as separate 'type' tokens? If not, how are they represented? And if they are separate tokens, how are they represented in the constraint model, where the outputs are pointers rather than one-hot vectors?
--- In the standalone primitive model, each primitive is represented by multiple tokens, but constraints only refer to either full primitives or sub-primitives. How is a reference to a full primitive obtained if only token embeddings are available? The authors only mention briefly: "From the tuple of representations for each primitive, we extract reference embeddings for both the primitive as a whole (e.g., a line segment) and each of its sub-primitives (e.g., a line endpoint) that may be involved in constraints.", but the authors don't mention how this is done.
- For the primer-conditional generation, it might be good to discuss briefly if the primitives of an incomplete sketch need to be given in a similar order as during training (the typical order a designer would create the primitives in).
- Willis at al. and Ganin et al. are missing the venue in the bibliography.
summary_of_the_review: The proposed methods contributes a new generation approach and explicit generation of both primitives and constraints over prior methods; and reconstruction of CAD sketches from hand-drawn images over both prior and concurrent work. The authors also show good qualitative results. However, as a big minus, the result are not compared to any related work. On the balance though, I am still leaning slightly towards acceptance, since the contributions are good, and the qualitative results are convincing enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new deep generative model for 2D CAD sketches. The transformer-based model is able to generate both the primitives as well as the primitive constraints that make up a sketch, and its output can be then imported into standard CAD software. In addition to unconstrained sketch generation, the authors demonstrate experiments where they use the model to automatically complete partial sketches as well as produce sketches that resemble a rough human-drawn sketch that is given as input.
main_review: The task of using deep learning to generated structure representations of graphics useful in CAD applications is an interesting and open area of research that has potential for practical applications. The method proposed in this paper utilizes a reasonable approach based on state-of-the-art architectures and techniques. The paper is well-written, and the shown results look good given the dataset that is used for training.

My main concern about this paper has to do with its technical novelty. While it is true that the SketchGraphs dataset is fairly recent, and there have not been many papers published that utilize it, it is not completely obvious what are the high-level technical takeaways that can be useful for building on the proposed method. All of the components are certainly reasonable and demonstrate impressive engineering but seem like fairly standard applications of existing architectures. On the other hand, while the results are nice, I don't think they are ready to be put into a production-level system, in part due to the relative simplicity of the generated sketches. I would encourage the authors to clarify what is the broad novel insight for building learning-based systems that produce vectorized or CAD-style data and/or to provide some validation that the method can be practically helpful in a real-world CAD pipeline.

The other main issue that I see is with regards to comparison with related work. Some of the related work that's noted to be concurrent is in fact prior work. In particular, "Engineering Sketch Generation for Computer-Aided Design" [Willis et al. 2021] appeared at CVPR 2021 and, as the authors note, proposes a similar transformer-based CAD sketch generative model. It would be important to include a comparison to this work or at least a more detailed discussion. The authors mention that the main distinction is that Willis et al. do not produce the constraint graph as part of their output. This is true, but the evaluation of the learned constraint graphs is fairly limited---it is essentially only demonstrated in one small figure. Willis et al. show how the standard auto-constrain feature in AutoCAD can be used to infer constraints for sketches produced by their model. Do the learned constrain graphs here address certain cases that cannot just be handled by this sort of simple post-processing?

A more minor issue is that it would be very helpful to include a diagram showing the network architecture---while the text is clearly written, it is a bit hard to keep track of all the moving parts.
summary_of_the_review: This paper is well-written and tackles an interesting problem in a sensible way. However, due to limited novel technical insight and comparison to past work, I think it requires some improvement before it is ready to be published.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work builds on the SketchGraphs dataset that contains CAD sketches, consisting of primitives and constraints. This dataset opened many opportunities to tackle research problems that could not be easily be tackled before. One important question is how to define a generative model that can unconditionally sample from the distribution. Another important question is how to sample conditioned on some input, e.g. a hand drawn sketch. This paper looks at both of these questions.

main_review: It is somehow natural that after a release of an exciting new dataset multiple groups are trying to target core questions that can be tackled due to its release. There is already important work in this area, i.e. Ganin et al. (2021) and Para et al. (2021), that has substantial overlap with this submission. According to the rules of ICLR, I believe this previous work should count as independent publication. The authors also position their work as concurrently developed. 

Overall, a suitable engineering choice is to model sketches as sequences of tokens and to employ multiple transformers for sequence modeling. A strong inspiration for this (as well as the other papers) was PolyGen from Nash 2020 which is appropriately cited multiple times in the paper. Working with transformers to model sequences is difficult work and it typically takes quite a bit of skill and work to achieve good results, such as the ones shown in the paper. Working with such large datasets is also difficult in general.

The paper is generally very well written (language wise) and nicely illustrated. My main request for improvement is to add details in the writing. I would be interested to know more details of
*) the architecture and training
*) the data formats for sequence modeling (input as well as output)
There is a promise to release code, which should help clarify some details in the future. However, I would strongly request that these details are included in the paper. For example, the paper proposes to use references to primitives and sub-primitives. It is unclear how exactly this is done. The tokenization in 3.1 and 3.2 could be explained better with examples and the precise number of bits and precise number of choices that are being encoded. It seems that Ganin et al. and Para et al. have more content devloted to clarify details, but this submission does not have a corresponding appendix with details (e.g. appendix A and B in Ganin et al.). For example xCenter is an ID token, but what are all the other ones? What are the lookup table options for ID, Value, and Position (mainly ID and value are unclear). How many lookup tables (quantization tables) are there? Are they context dependent? There are multiple choices how exactly to do the encoding.

6-bit quantization appears a bit low and I wonder if that is really a suitable choice to achieve very good results. Maybe there could be an ablation study. Intuitively, I really doubt that 6-bit is enough to encode details. I do not see how the constraints can fix that large loss of information.

The paper does provide a list of reasonable results, but at the same time there could be more comparisons and ablation studies to put the work into context. The results feel maybe a bit less extensive than the other two papers. It would also be informative if the paper could add more comparison to the previous papers to be helpful to future researchers in the field (even though they should not be required to do so). Running other code is probably tricky, but writing a table or a list comparing some of the design decisions in the text should be feasible.

The work does a good job conditioning on images, a result that is not present in Ganin et al. and Para et al. and a unique feature of this work.
summary_of_the_review: Assuming the authors are willing to add (a lot) more details, I would suggest that this paper should be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a model capable of generating CAD sketches with constraints. Such sketches are typically formed as a sequence of 2D primitives (points, lines, arcs, circles) related by geometric constraints (parallelism, orthogonality, coincidence, etc). The model generates these sequences in an auto-regressive fashion, one primitive at a time, and then one constraint at a time. The model is trained on a large dataset of CAD sketches [Seff et al. 2020] that contains the necessary ordered sequences of CAD primitives and constraints.

The model is implemented as two Transformers, one to generate the sequence of primitive, and one to generate the sequence of constraints conditioned on the primitives. For the second part, a PointerNetwork mechanism is used to identify the primitives referenced by each constraint. 

The model is used either to generate random CAD sketches that exhibit similar statistics as the training data (as measured by the number of primitives and the number of constraints used), or to auto-complete a partial CAD sketch by predicting missing primitives and constraints. In addition, an image-conditioned version of the model is also described, which can take as input an approximate bitmap drawing and turn it into a constrained CAD sketch.
main_review: **Strengths** 
- The paper is well written, easy to follow. Section 2 provides a clear positioning of the work against prior efforts on similar problems. 
- The method itself is well motivated, the proposed architecture makes sense and is described in sufficient details. 
- The qualitative and quantitative evaluations suggest that the model performs well on the task.

**Weaknesses**
- I would have appreciated additional visual results on the task of inferring constraints over existing sketches (Fig.8) and on the task of generating CAD sketches from hand-drawn sketches (Fig.6). While the few results shown are very encouraging, I would have liked a discussion of typical failure cases (since I suspect some failure on such ambiguous tasks).
- My main concern about this paper is that it appears very similar to [Ganin et al. 2021, Para et al. 2021], which will both appear at Neurips. Checking the ICLR reviewer instructions, I see that these papers are indeed considered concurrent (*if a paper was published (i.e., at a peer-reviewed venue) on or after June 5, 2021, authors are not required to compare their own work to that paper.*), which I interpret as not being a reason for rejecting this submission. 
Still, from a reader perspective, I would have liked a detailed discussion of [Ganin et al. 2021, Para et al. 2021] to understand if they are indeed very similar (both target the same problem, both rely on a Transformer+PointerNetwork architecture), or if the proposed model introduces significant differences. The submission mentions that these two concurrent papers do not attempt to train/test on hand-drawn / noisy input, but this seems more of a detail in the way the method is used than a fundamental difference in the actual model.

summary_of_the_review: If the similarity to the concurrent work of Ganin et al. 2021 and Para et al. 2021 should not be accounted for, then I am in favor of acceptance because the described model is sound and effective. But I remain uncomfortable accepting a paper that shares so many similarities with papers that have already been reviewed and accepted, even if shortly before the deadline. I suspect that the authors have been inspired by these concurrent works, maybe in the development of their method, its description, its evaluation, which is why I have a hard time ignoring them in my evaluation.

To be clear, if [Ganin et al. 2021] and [Para et al. 2021] are to be ignored, then the technical novelty is of level 4, but if they should be considered, then the technical novelty drops to 1 or 2 and I would argue for rejection.

--- Additional questions ---

Since the constraint model is trained separately from the primitive model, is it trained with ground-truth primitive sequences from Seff et al., or from the (approximate) sequences predicted by the primitive model? Section 3.2 mentions that the constraint model must account for potentially imperfect generation from the primitive model, but I didn’t fully understand how this is achieved. Is it only achieved by perturbing the primitive parameters with Gaussian noise? What if the predicted primitive type is wrong, wouldn’t such error degrade all subsequent predictions?

I appreciate the practical details provided in Section 3.1 and 3.2, and in Appendix C. Yet, I would have liked a diagram of the two Transformers to better understand the flow of information, and to illustrate how the sequences are fed to the networks. This would be particularly useful for the constraint prediction network, as the mechanism to cross-attend the primitive sequence and to point to its elements remains a bit vague to me.

I didn’t fully understand the tokenization in Section 3.1. Since the ID token specifes the parameter type, why not also use it to specify the primitive type rather than allocate part of the value token for this purpose?
review_title: null

Meta-review: The paper describes an approach for automatically generating CAD sketches, including both the primitives that describe the drawing, as well as the constraints that describe relationships between the primitives that need to be maintained even if the primitives are changed. This is an important problem that is starting to receive a lot of attention from the literature. 

Overall, the paper is very well executed and the results are quite compelling. 

There were some concerns about the relationship with the work by Willis et al. and other papers that were published around the time when this paper was submitted. There is still some novelty in this paper relative to those works as argued in appendix H, but it would have been really good to have a more quantitative comparison. However, the authors pointed out that this work was concurrent as opposed to prior work as per the ICLR reviewer guidelines. 

Overall, given the quality of this paper and the guidance given in the ICLR reviewer guide, most reviewers agree with the meta-reviewer that this paper should be accepted (the lowest reviewer still indicated it is above the acceptance threshold). However, there is some discomfort around not having an explicit comparison with very closely related work that ultimately was published before this paper.
Decision: Accept (Poster)