CAD-Editor: Text-based CAD Editing through Adapting Large Language Models with Synthetic Data

Authors: Yu Yuan, Shizhao Sun, Qi Liu, Jiang Bian
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Computer Aided Design, Generative Models, Text-based Editing, Large Language Models

Abstract:
Computer Aided Design (CAD) is indispensable across various industries. 
\emph{Text-based CAD editing}, which automatically modifies CAD models following textual instructions, is important yet not extensively studied. 
Existing work explores design variation generation, which randomly alters specific parts of a CAD model, offering no control over the final appearance.
This work introduces \emph{CAD-Editor} for text-based editing.
We leverage Large Language Models (LLMs) as the backbone to take the concatenation of textual instruction and original CAD sequence as input and predict the edited CAD sequence, where the sequence representation of a CAD model is designed for easier processing by LLMs.
Moreover, we propose fine-tuning LLMs by using a synthetic dataset followed by a selective dataset.
The synthetic data is produced by leveraging powerful existing models, including design variation generation models for producing paired CAD models and multi-modal models for capturing textual differences between these pairs.
The selective data is created by choosing top examples from outputs of the initially fine-tuned LLMs based on human feedback or metrics.
In this way, a large-scale synthetic dataset offers basic capability while a selective dataset that is less noisy and better aligned with human intentions boosts performance further.
Extensive experiments demonstrate the advantage of CAD-Editor both quantitatively and qualitatively.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730372857268

thread_type:
official_review

<thread_object>
creator: reviewer_czqp

<content>
summary:
The paper proposes a CAD editing dataset. The CAD models are collected and fed into design variation generation models to create design variations. A large language model (LLM) is used to generate multi-level captions, which are then compressed into sentences with a maximum of 10 words. To enhance model performance, the paper suggests filtering some synthetic data using D-CLIP or human evaluation. The model is fine-tuned on LLAMA 3 and achieves the best performance compared to other methods.

strengths:
The paper proposes an interesting dataset focused on instruction-based CAD editing, a task that holds significant value for industrial applications.

The paper is well-written and easy to follow.

It introduces an innovative model-based data selection method that enhances the quality of the dataset.

weaknesses:
I am curious about the D-CLIP evaluation. It is unclear whether the adapted D-CLIP model was pretrained on any specific datasets. Was it fine-tuned on the proposed dataset? If not, is there a potential domain gap between the proposed dataset and the datasets used to train D-CLIP? This domain gap could potentially influence the effectiveness of the data selection process.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536218187

thread_type:
official_review

<thread_object>
creator: reviewer_f1vw

<content>
summary:
The paper introduces CAD-Editor, a text-driven CAD editing method based on large language models (LLMs). To generate high-quality training data, the article adopts a two-stage training strategy. The first stage generates a large-scale synthetic dataset, and the second stage constructs a selective dataset. Experimental results show that CAD-Editor outperforms other methods on multiple evaluation metrics, including GPT-4o.

strengths:
- Innovatively transforms CAD editing tasks into sequence-to-sequence problems, leveraging large language models (LLMs) to achieve text-driven CAD model editing.
- Adopts a multi-stage training strategy, significantly improving model performance by combining synthetic datasets and selective datasets.

weaknesses:
- I am deeply concerned about the generalization ability of CAD-Editor. On one hand, the choice of the base dataset is crucial for the model's generalization ability, and CAD-Editor is only optimized on the DeepCAD dataset. On the other hand, the generation of the synthetic dataset relies on existing CAD generation models (such as SkexGen and Hnc-CAD), both of which are also trained on DeepCAD. Therefore, the performance of CAD-Editor on the DeepCAD benchmark raises suspicions of overfitting. That is, if the fine-tuning dataset mainly contains CAD models from a specific industry (such as the automotive industry), the trained model may not handle CAD models from other industries (such as aerospace or construction) well.
- The reliance on GPT-4o. The proposed method requires multiple prompts to GPT-4o to obtain generated data. The article does not discuss the token cost of GPT-4o and potential performance bottlenecks.

questions:
- The article mainly focuses on models with 8B parameters, and based on the qualitative analysis results provided, there is still a significant gap between the current model's effectiveness and being able to provide productivity. I am curious whether CAD-Editor can be more effective when scaled to larger models or more diverse datasets.
- How does CAD-Editor perform when dealing with vague or ambiguous text instructions? Does the model have the ability to identify and correct erroneous editing instructions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696162989

thread_type:
official_review

<thread_object>
creator: reviewer_bbso

<content>
summary:
The paper introduces CAD-Editor, a novel approach for text-based CAD editing that leverages LLMs to interpret and apply textual instructions to CAD models. CAD-Editor is designed as a sequence-to-sequence generation model, enabling users to edit CAD models via natural language inputs. The paper outlines a method involving two phases of fine-tuning on synthetic and selective datasets to optimize performance. Experimental results demonstrate CAD-Editor’s improvements in accuracy, alignment with textual instructions, and CAD model validity over other models, including SkexGen and Hnc-CAD. This work also establishes a benchmark for text-based CAD editing.

strengths:
- The paper is well-written and it's easy to follow. 
- As the authors have also mentioned, CAD generation and editing has gotten relatively little research attention although they play a huge role in different industries.

weaknesses:
- I believe this paper is a better fit for a computer vision conference for being more of an applied paper investigating CAD editing. 
- I think the authors should provide a couple of examples of D-CLIP scores and their corresponding text/image pairs to show the effectiveness of their proposed metric.

questions:
- In line 69, what do the authors mean by "without control over the final appearance"? Aren't they text to CAD models? 
- In line 259, how do you automatically exclude instructions with more than three edits? 
- Have you tried fine-tuning any of the baseline text-to-cad models instead of LLama3-8b-instruct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733141005465

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response by Authors

comment:
We would like to thank all reviewers for providing constructive feedback that help us improve the paper. We look forward to your further feedback.

We are encouraged that reviewers think our paper:

1. **Novel Task Introduction**: We introduce a new task called text-based CAD editing. Reviewer CzQp noted that it "holds significant value for industrial applications", while Reviewer Bbso highlighted that it addresses a research area with "relatively little research attention, despite its importance".

2. **Innovative Problem Formulation**: We formulate text-based editing as a seq2seq problem and employ pre-trained LLMs as our backbone. Reviewer F1Vw described this approach as "innovative transformation of CAD editing tasks into sequence-to-sequence problems, utilizing LLMs for text-driven CAD model editing."

3. **Innovative Training Strategy and superiority of CAD-Editor**: We propose CAD-Editor, the first model specifically for text-based CAD editing. Our multi-stage training strategy, combining synthetic and selective datasets, significantly enhances performance. Reviewer F1Vw praised the "multi-stage training strategy, significantly improving model performance by combining synthetic and selective datasets", and Reviewer CzQp highlighted our "innovative model-based data selection method that enhances dataset quality."

4. **New Benchmark Dataset**: We introduce the first benchmark for text-based CAD editing utilizing design variation model and MLLMs. Our proposed Multi-level Captioning method enhances captioning accuracy. Reviewer CzQp noted it as "an interesting dataset focused on instruction-based CAD editing".

5. **Clarity**: Both Reviewer Bbso and Reviewer CzQp appreciated that the paper is "well-written and easy to follow." 

In our rebuttal, we addressed all raised concerns and misunderstandings:

1. **Misunderstandings and Explanations:** 
   - For alignment with ICLR raised by Reviewer Bbso, we explained that our work is highly relevant to major machine learning conferences like ICLR, ICML, NeurIPS, aligning well with their focus and precedent publications. 
   - Regarding whether fine-tuning existing text-to-CAD models was considered raised by Reviewer Bbso, we clarified that there were no existing text-to-CAD models available for fine-tuning, as our work is the first to apply text control to CAD models. 
   - Addressing concerns about GPT-4o's token costs and bottlenecks raised by Reviewer F1Vw, we noted the manageable token usage and our Multi-level Captionning method which improve performance, and suggested alternatives like LLaVA-OneVision for those with budget constraints. 
   - Following Reviewer F1Vw's suggestion, we provided a comprehensive discussion with a recent work Text2CAD, which was published on September 25, 2024 and received by NeurIPS 2024 as a spotlight.

2. **Additional Experiments:**
   - For CAD-Editor's potential overfitting to the DeepCAD dataset raised by Reviewer F1Vw, we clarify that its substantial size and broad scope across industries makes overfitting less likely. Moreover, empirical evidence supports its ability to generalize, as demonstrated in related works. Furthermore, we conducted additional experiments on the Fusion 360 dataset, showcasing CAD-Editor’s robust generalization capabilities (details in Appendix E).
   - Regarding whether CAD-Editor can be more effective when scaled to larger models raised by Reviewer F1Vw, our preliminary results (see Appendix F) show that fine-tuning the 70B model achieves a faster loss reduction compared to the 8B model.
   - For the ability of our CAD-Editor to deal with vague and erroneous instructions, we have shown in Figure 7 that CAD-Editor handles vague or ambiguous text instructions well. Moreover, we added additional experiments to show CAD-Editor can identify and correct errors in instructions (see Figure 14, Appendix C).
   - We addressed Reviewer Bbso's request by adding D-CLIP score examples in Appendix D and responded to Reviewer CzQp's concerns about D-CLIP by clarifying that since CLIP was not trained on the DeepCAD or CAD-Editor-Dataset, overfitting during evaluation is not a concern. We highlighted CLIP’s training on a vast dataset of 400 million images which included CAD-related content, underscoring its robust generalization abilities.

Please see our reviewer-specific feedback for more information.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces CAD-Editor, an approach leveraging LLMs for text-based CAD editing by framing it as a sequence-to-sequence problem. The methodology includes a multi-stage training strategy with synthetic and selective datasets and introduces a tailored benchmark dataset.  The paper has notable strengths. Text-based CAD editing addresses a significant area with practical industrial implications, and the innovative use of LLMs to interpret and apply textual instructions to CAD models is a promising step forward, likely to inspire further research. Additionally, the paper is well-written and effectively communicates complex ideas.

However, concerns remain that prevent acceptance in its current form. A key issue is the model's ability to generalize beyond the specific datasets used for training and evaluation. The reliance on these datasets raises questions about the applicability of the approach to CAD models from other domains. Furthermore, the computational demands and dependence on large-scale models like GPT-4 pose challenges regarding the practicality and scalability of the method in real-world settings. While the authors have attempted to address some of these issues, they remain significant and highlight areas that require further refinement and validation.

While the paper presents innovative ideas with significant potential, it requires further development to address key concerns. Enhancing generalizability, tackling scalability challenges, and providing more robust validation of the approach are necessary to strengthen its contributions. As such, this paper does not meet the requirements for acceptance in its current form, but I encourage the authors to refine their work for future consideration.