FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models

Authors: Zhanwei Zhang, Shizhao Sun, Wenxiao Wang, Deng Cai, Jiang Bian
Primary area: generative models
Keywords: cad generation, large language model

Abstract:
Recently, there is a growing interest in creating computer-aided design (CAD) models based on user intent, known as controllable CAD generation. Existing work offers limited controllability and needs separate models for different types of control, reducing efficiency and practicality. To achieve controllable generation across all CAD construction hierarchies, such as sketch-extrusion, extrusion, sketch, face, loop and curve, we propose FlexCAD, a unified model by fine-tuning large language models (LLMs). First, to enhance comprehension by LLMs, we represent a CAD model as a structured text by abstracting each hierarchy as a sequence of text tokens. Second, to address various controllable generation tasks in a unified model, we introduce a hierarchy-aware masking strategy. Specifically, during training, we mask a hierarchy-aware field in the CAD text with a mask token. This field, composed of a sequence of tokens, can be set flexibly to represent various hierarchies. Subsequently, we ask LLMs to predict this masked field. During inference, the user intent is converted into a CAD text with a mask token replacing the part the user wants to modify, which is then fed into FlexCAD to generate new CAD models. 
Comprehensive experiments on public dataset demonstrate the effectiveness of FlexCAD in both generation quality and controllability.

Decision: accept (poster)
Reviewer scores: 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729331837638

thread_type:
official_review

<thread_object>
creator: reviewer_4ahh

<content>
summary:
This paper presents FlexCAD, a unified model for controllable CAD generation, enhancing the ability to generate CAD models based on user intent. Unlike existing methods requiring separate models for different controls, FlexCAD enables controllable generation across multiple CAD construction hierarchies (e.g., sketch, extrusion, face). The authors fine-tune large language models (LLMs) and represent CAD models as structured text, abstracting each hierarchy into a sequence of text tokens for better comprehension. A hierarchy-aware masking strategy is introduced, masking specific fields during training to predict the masked parts based on user input during inference. In general, this paper is well organized.

strengths:
+ Novel use of LLMs for CAD design
+ The authors propose a unified framework based on fine-tuning LLMs to bridge LLMs and CAD models, including dividing CAD models into multiple hierarchies and representing CAD models with sequential tokens and 
+ The authors introduce a hierarchy-aware masking strategy to achieve controllable generation based on the mask-and-prediction method
+ Experimental results show the proposed method achieves good results and verify the effectiveness of each component

weaknesses:
- The motivation of using LLMs for CAD generation is not very clear. The authors are recommended to clarify why LLMs are the good choice to do this task. 

- The ablation study in Tab 3 is clear to make a comparison among different foundation models, pre-training, and fine-tuning. However, could LLaMA-3 do CAD generation in a zero-shot manner? How helpful is the proposed fine-tuning method compared with the zero-shot setting?

- The evaluation for prompt-result alignment is unclear, e.g., intent understanding and instruction following. 

- Some experimental details may be missing, e.g., the instructions used to prompt GPT-4o to generate CAD models.

questions:
- Does the proposed method inherit some advantages of LLMs, such as instruction following and in-context learning? 

- The proposed method assumes users skilled at the defined hierarchies so that they can transform their intent into mask-based instructions during inference. Can users express their intent using natural language and then the transformation task is given to LLMs?

- Does the fine-tuned model strictly follow the grammar of CAD structured text? If not, is there any failure case which may not be parsed into a valid CAD model? How about the percentage? 

- Compared with prior work, is there any drawback to using LLMs? How about the efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590563740

thread_type:
official_review

<thread_object>
creator: reviewer_daws

<content>
summary:
This paper proposes a unified CAD generation model that leverages large language models with a hierarchy-aware masking strategy, achieving fine-grained controllability across multiple CAD construction hierarchies and outperforming existing methods in generation quality and user-intent alignment. 
In detail, FlexCAD represents CAD models as structured text, transforming CAD construction hierarchies into sequences of text tokens, which improves comprehension by LLMs. During training, the hierarchy-aware masking replaces specific fields in CAD text with a mask token, enabling the model to predict missing parts based on user intent.
The model demonstrates significant performance improvements over existing methods like SkexGen and Hnc-CAD.

strengths:
1. The proposed FlexCAD model effectively enables controllable CAD generation across all hierarchies, addressing a significant gap in CAD modeling by unifying control within a single framework. It achieves superior performance on the DeepCAD dataset compared to existing baselines, highlighting the method's robustness and effectiveness.
2. The paper is well-organized and clearly written, making the methodology and contributions easy to understand and follow.
3. The authors provide extensive ablation studies to analyze the impact of LLM pre-training and hierarchy-aware tokenization. These experiments demonstrate that fine-tuning pre-trained LLMs with LoRA yields the best results, underscoring the advantages of leveraging pre-trained models for CAD representation—a promising insight for future research in CAD and design automation.

weaknesses:
1. The evaluation is limited to the DeepCAD dataset, which restricts the generalizability of the results. Testing FlexCAD on additional CAD datasets would better demonstrate its robustness and adaptability across diverse design scenarios.
2. The paper lacks references to relevant recent work, such as “OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Design” and “CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches,” which address related challenges in CAD generation and infilling. Including these would strengthen the context and positioning of the work.

questions:
1. During training, what is the average mask ratio per training sample, and is there any observed relationship between performance and the mask ratio?
2. Given that a single masked input can produce multiple possible outputs, as shown in Figure 5, which output is selected for the final evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605253466

thread_type:
official_review

<thread_object>
creator: reviewer_zstf

<content>
summary:
This paper proposed, flexCAD, a unified and versatile model for controllable CAD generation across all hierarchies using LLMs. FlexCAD addresses various limitations in existing CAD generation models. Previous models typically lack comprehensive control across CAD hierarchies and require separate models for different tasks. The proposed model leverages a structured text representation of CAD models and a hierarchy-aware masking strategy to control the generation across various levels (e.g., sketch, face, loop, extrusion). Experiments show that FlexCAD outperforms existing models on various metrics. 

Contributions: 
- FlexCAD is the first to leverage LLMs for controllable CAD generation. 
- FlexCAD greatly improves generation quality and controllability, showing its effectiveness on the sketch-level and extrusion-level controllable generation tasks.

strengths:
- FlexCAD improves generation quality and controllability in some metrics and demonstrates its effectiveness on the sketch-level and extrusion-level controllable generation tasks.

weaknesses:
- Novelty is limited. The training methods being used are very standard tuning methods for LLM. 
- FlexCAD is the first to leverage LLMs for controllable CAD generation (based on the author's statement). Previous baseline "SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks" and "Hierarchical Neural Coding for Controllable CAD Model Generation (ICML 2023)" already use transformer architecture. 
- It would be stronger if FlexCAD can show improvement with larger margin given FlexCAD is not the best among all metrics. 
- No discussion on ethics and potential risk of this research.

questions:
- Can the authors provide some analysis on why full model fine-tuning performs worse than LoRA on some metrics? 
- For FlexCAD, does it use few-shot or zero-shot at inference time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002100213

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all Reviewers

comment:
Dear Reviewers,

We sincerely thank you for your valuable comments.  
We are encouraged that you found our method **novel** [4Ahh], **bridging LLMs and CAD models** [4Ahh], 
offering **a promising insight for future CAD design** [DaWS]. 
We are also pleased that you found our approach to be effective [zSTF, DaWS, 4Ahh], 
improving generation quality and controllability [zSTF], well-organized [DaWS], clearly written [DaWS], and easy to follow [DaWS].

In the following, we will do our best to address the misunderstandings. 
Additionally, our manuscript has been revised to add Fig. 18 (a five-shot in-context learning prompt example for GPT-4o) and the Ethics Statement section, 
based on the reviewers’ insightful comments, making our research more robust and accessible.

As the deadline for Author-Reviewer discussion approaches, 
we eagerly look forward to your responses. 
Please let us know if there are any additional clarifications or experiments we could provide. 
We would be happy to discuss further if any concerns remain.
Thanks again for your time!

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes FlexCAD, a unified model for controllable CAD generation across construction hierarchies using fine-tuned LLMs. By representing CAD models as structured text and employing a hierarchy-aware masking strategy, it bridges CAD's complex numerical data with LLMs' textual capabilities, enabling unified generation without separate models.  

A notable strength of this paper is its pioneering application of LLMs to CAD generation. While fine-tuning LLMs is well-established, applying it to CAD—characterized by complex geometric data and hierarchies—demonstrates that LLMs can effectively handle structured CAD text, enabling new interactions and control beyond traditional methods. The authors demonstrate promising improvements in generation quality and controllability as an early attempt to apply LLMs to CAD generation.  

While the technical novelty in model architecture may be limited, applying LLMs to CAD generation marks a significant advancement in a specialized field. Representing CAD models as structured text for LLMs is a meaningful contribution, making complex CAD data accessible to language models despite limited improvements in spatial understanding and data representation. This paper adapts LLMs for CAD generation, opening new avenues in design automation and encouraging further exploration.