Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Authors: Ruiyu Wang
Yu Yuan
Shizhao Sun
Jiang Bian
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Computer-aided Design
Text-to-CAD Generation
Primary area: applications->computer_vision

Abstract: Creating Computer-Aided Design (CAD) models requires significant expertise and effort. Text-to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in streamlining this process. 
Recent studies have utilized ground-truth parametric sequences, known as sequential signals, as supervision to achieve this goal.
However, CAD models are inherently multimodal, comprising parametric sequences and corresponding rendered visual objects.
Besides, the rendering process from parametric sequences to visual objects is many-to-one.
Therefore, both sequential and visual signals are critical for effective training.
In this work, we introduce CADFusion, a framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL) stage and the visual feedback (VF) stage. 
In the SL stage, we train LLMs using ground-truth parametric sequences, enabling the generation of logically coherent parametric sequences. 
In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated.
These two stages alternate throughout the training, ensuring balanced learning and preserving benefits of both signals.
Experiments demonstrate that CADFusion significantly improves performance, both qualitatively and quantitatively.

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Ruiyu Wang
Yu Yuan 21
Shizhao Sun
Jiang Bian
Abstract
1. Introduction
Creating Computer-Aided Design (CAD) models requires considerable expertise and effort. Text- to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in stream- lining this process. Recent studies have utilized ground-truth parametric sequences, known as se- quential signals, as supervision to achieve this goal. However, CAD models are inherently multi- modal, comprising parametric sequences and cor- responding rendered visual objects. Besides, the rendering process from parametric sequences to visual objects is many-to-one. Therefore, both se- quential and visual signals are critical for effective training. In this work, we introduce CADFusion, framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL.) stage and the visual feedback (VF) stage. the SL stage, we train LLMs using ground-trute para- metric sequences, enabling the generation of log- ically coherent parametric sequences. In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated. These two stages alternate throughout the training, ensur- ing balanced learning and preserving benefits of both signals. Experiments demonstrate that CAD- Fusion improves performance, both qualitatively and quantitatively. Code is available at https: / //github.
Computer- Ainduter-Aided Design (CAD) is indispensable for 3D creation across industrial sectors (Deng et al., 2023). It rep- resents 3D models through sequence of operations known as a parametric sequence, which combines lines, arcs, and circles create 2D sketches and then extrude them form 3D models. CAD models inherently multimodal, as they are constructed using parametric sequences for precise edit- ing and manufacturing, while also being rendered as visual objects for practical use, referred to as multimodal charac- teristic (Figure ((b)c)). Moreover, the process of rendering parametric sequences into visual objects exhibits a many- to-one mapping, where different parametric sequences can result in identical visual objects, referred to many-lo-one rendering characteristic (Figure ((d)).
Creating CAD models demands considerable expertise and numerous iterations, making it complex and time- consuming. Text-to-CAD (Figure (aa))))), which refers to the automatic generation of parametric sequences from tex- tual descriptions, is critical for streamlining this creation pro- It allows designers and engineers to quickly prototype and iterate designs by describing their intent in natural lan- guage, reducing the time spent on manually creating CAD models from scratch. Additionally, it makes the creation process more accessible to individuals without extensive training, enabling wider participation.
While important, Text-to-CAD has received limited atten- tion. Most studies do not utilize text to control CAD gener- ation. Instead, they explore generating CAD designs from random noise (Wu et al., 2021; Xu et al., 2022), by randomly altering components of existing CAD designs (Xu et al., 2022; 2023; Zhang et al., 2025), or from point cloud (Khan et al., 2024a). A few studies make preliminary attempts at Text-to-CAD (Khan al., 2024b; Li al., 2024b). They train Transformer-baee framework with ground-truth para- metric sequences as supervision, termed sequential signal.
Work done during the authors' internship Microsoft Re- search Asia. Open-source research project starts at March 2024. Ruiyu Wang: <rwang @cogg@ess.ooooo toronto.euus Yu Yuan: <yy- happier@ mail.usc..et .ut.c.ddu.ex> Jiang Bian: <jiabia@ microsoft.com>. 'University of Toronto. Universiity of Science and Technology of China. "Microsoft Research Asia. Correspondence to: Shizhao Sun <shizsu @ microsooft.cmm..
However, due to multimodal and many-to-on rendering characteristic of CAD models (Figure ((b)())d)),, both the sequential signal and visual signal are crucial for training Text-to-CAl CDDC model. The sequential signal, derived from ground-truth parametric sequences, provides critical infor-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Text-to-CAr Generation Through Infusing Visual Feedback in Large Language Models
(a) Input Prompt
(b) CAD Design Sequence
(c) CAD Visual Object
A rectangular prism with a total of five squere holes, One centrally located and four surrounding
I5ne 26, cource and 1ine 52.7 courve_andh 1ime 6D, d6 courve,nnd andd 11se, 30, s0j mi Lime 23,50 a5 courve and <low ende life, scurve end Lire, 23
Generates
ande dinn S3.8 13/0 scures onde Line, 54,9 scurve onder L2,9,53,9
end Lime
<ioog
25, 22, 22, 1,0, A,
32,48 eencrute_eeci
Design
(d) The Many-to-one Relationship
1immy
Cloos enio 1ine vourive <2001 enjy
scurve exto
Denggs
eouree 0153
<loos ands
Vake Dougs
cakatch add CRAATO CiTy
restmute ancs Gharmate AND
Design
LERK
101 <3n09 mnto NExtrade TOAP
<ateton_.nd a2d, rextrastt_snd snnd
Figure 1. (a) and (b): Illustration of Text-to-CAD, which converts a textual description into CAD parametric sequences. (b) and (c): Illustration of multimodal characteristics. CAD models created using parametric sequences and rendered visual objects for practical use. (d): Illustration of many-to-one rendering characteristics. Diffcrent parametric sequenees can produce identical visual objects.
mation about sequence structure and parametric operations. Without it, learning to generate logically coherent paramet- ric sequences becomes challenging, as there is no direct supervision for sequence structure and parametric opera- tions. The visual signal, obtained from rendered visual objects, indicates how CAD models are perceived and evalu- ated in practical applications. Without it, learning efficiency is compromised, as the goal of Text-to-CAD is for the ren- dered visual objects of the generated parametric sequences to match ground-truth visual objects. First, sequential signal learning typically depends on auto-regressive generation, which emphasizes the local continuity between tokens but may not fully capture the global appearance of the CAD model. Second, given the many-to-one rendering character- istic, multiple parametric sequences can produce the same visual object. Training solely on parametric sequences may cause the model to give more emphasis to those present in the training set, overlooking other valid ones that could achieve the same visual outcome.
mization (DPO) (Rafailov et al., 2024). Specifically, pref- erences are assigned to the rendered visual objects, and the LLMs are optimized to increase the likelihood of parametric sequences that produce preferred visual objects while de- creasing the likelihood of those that yield less preferred ones. This approach enables effective training of LLMs, even with non- dindiffeeeentible rendering pathway. Second, collecting reliable preference data is costly and labor-intensive. To address this, we introduce an automated pipeline that uti- lizes large vision-language models (LVMs) to efficiently score the rendered visual objects. Finally, to ensure bal- anced learning and retain the contributions of both signals, we alternate between the sequential learning stage and the visual feedback stage throughout training.
We summarize our main contributions as follows:
We propose to leverage both the sequential signal and visual signal to train a Text-to-CAD model.
For the sequential signal, we use LLMs as the back- bone and fine-tune it on ground-truth parametric se- quences. For the visual signal, adopt direct prefer- ence optimization to bypass non-differentiable render- ing and introduce LVM-based scoring pipeline for efficient preference data collection. To balance both signals, we alternate between the sequential learning and the visual feedback stage.
To this end, we propose CADFusion, a framework that combines sequential and visual signals train Text-to- CAD model. It uses Large Language Models (LLMs) as its backbone and alternates between two stages: the sequential learning stage and the visual feedback stage. In the sequen- tial learning stage, LLMs are fine-tuned using ground- truth parametric sequences. Unlike prior works (Khan et al., 2024b; Li et al., 2024b) that train Transformer-basse mod- els from scratch, we take advantage of pre-trained LLMss, which leverages their inherent natural language understand- ing and foundational knowledge of CAD design (Makatura et al., 2023) acquired during the extensive pre-training. In the visual feedback stage, feedback derived from rendered visual objects integrated into the L.Mss. This stage ad- dresses two critical challenges. First, the rendering process that converts parametric sequences into visual objects is non-differentiable, making backpropagation through this pathway infeasible. To overcome this, we frame the problem as preference learning task and adopt direct preference opti-
We contribute two datasets for Text-to-CAD one with the sequential signal and another with the visual signal.
We present qualitative and quantitative experiments to showcase CADFusion's 's superior ability.
2. Related Works
CAD Generation. CAD generation takes user requirements as input and generates CAD models as output.
On the input side, user requirements can be expressed in diverse ways. Wu et al. (2021) uses random noise as input to generate CAD models randomly. Zhang et al. (2025),
2

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Xu al. (2022) and Xu et al. (2023) modify specific parts of the existing CAD models to generate new ones. Khan et al. (2024a) and Ma et al. (2024) take point cloud input to produce corresponding CAD models, while Seff et al. (2022) uses hand sketches. In contrast, our work focuses on textual descriptions as input, leverages LLM as backbone on stringified representations, and does not adapt codebook encoders such as VQ- (QE to generate initial representations. Recent studies (Khan al., 2024b; Li et al., 2024b) explore text-based input for CAD generation. Khan et al. (2024b) proposes a data annotation pipeline for synthesizing training data and transformer-based autoregressive network. Li et al. (2024b) designs an encoder-decoder framework with a cascading contrastive strategy and CT-Mix to align text with parametric sequences. Unlike these studies, which rely solely on sequential signals, our work combines sequential and visual signals for improved performance.
text-to-image models (Liang et al., 2024) and text-to-video models (Wu al., 2024). As human annotation in RLHF is costly and not easily scalable, reinforcement learning on Al feedback (RLAIF) (Liu et al., 2023; Zhang et al., 2024; Lee et al., 2024), which leverages machine learning models to annotate data, has been proposed as a more affordable alter- native to RLHE. Since RLHF/RLAIF pipeline are complex, direct preference optimization (DPO) (Rafailov et al., 2024), which directly optimize model to adhere to human prefer- ences, has been proposed to avoid explicit reward modeling or reinforcement learning. In this work, we adopt DPO to address the challenge of non-differentiable rendering when learning from visual signals, as it offers a simpler yet ef- fective solution compared to RLHF Besides, inspired by RLAIF, we propose an automatic scoring pipeline for CAD models using LVMs. The generated scores are used to con- struct preference data, enabling efficient learning without reliance on costly human annotations.
On the output side, CAD models can be represented in vari- ous formats, including Constructive Solid Geometry (CSG), Boundary Representation (B-Rep) and Sketch-and-Exn -nttExt (SE). CSG constructs 3D models by combining basic primi- tives such as cubes, cylinders, and spheres, through Boolean operations and subtractions (Du et al., 2018; Kania al., 2020; Yu et al., 2022; 2024). B- B-Rep represents 3D models using geometric elements such as vertices, edges, and faces (Jayaraman et al., 2023; Wang et al., 2022; Xu et al., 2024). SE begins with 2D sketches composed of lines, arcs, and circles, which are then extruded to form 3D models (Willis et al., 2021; Wu et al., 2021). In this work, we adopt SE as it preserves the design history of CAD models, making them more intuitive to edit.
3. Method
3.1. Approach Overview
Let textual description be denoted as x. a CAD parametric sequence as y, and a rendered visual object as o. The render- ing process from a parametric sequence y to visual object is represented as r(-), such that o = r(y). Text-to-CAD involves learning a function f() that transforms the textual description æ into the CAD parametric sequence y. i.e., y = f(z). The goal is for the rendered visual object o of the generated parametric sequence y. i.e., o = r(y) = (()()),, to match the user's desired visual object (Figure 1).
Large Language Models (LLMs). LLMs have recently achieved remarkable success (Touvron et al., 2023; Brown et al., 2020; OpenAI, 2024; Bubeck et al., 2023; Zhao et al., 2023). Supervised fine-tuning (SFT) is widely used to im- prove performance, while reinforcement learning (RL) is often employed to align LLM output with human prefer- ence (Brown et al., 2020; Hong et al., 2024; Kaufmann et al., 2024). Our work leverages SFT and RL! but intro- duces two key differences. First, we utilize SFT and RL to learn from different signals (i.e., sequential and visual signals) whereas existing work focuses on a single signal (i.e., sequential signals). Second, we alternate between SFT and RL stages to preserve contributions from both signals, strategy not commonly employed in prior work.
CADFusion introduces a framework that combines sequen- tial and visual signal for training Text-to-CAD model (Figure 2). It leverages Large Language Models (LLMs) as the backbone and alternates between two stages: the se- quential learning (SL) stage and the visual feedback (VF) stage, We denote the model after the i-th round of sequential learning as (-) and after the i-th round of visual feed- back as (-). In the sequential learning stage, CADFusion trains LLMs to learn sequence structures and parametric operations from ground- -truth parametric sequences, guiding LLMs to generate logically coherent parametric sequences (Section 3.2). In the visual feedback stage, CADFusion trains LLMs to understand how the rendered visual object will be perceived and evaluated. By rewarding parametric sequences that render into visually preferred objects and penalizing those that do not, this stage encourages LLMs to generate parametric sequences capable of producing the desired visual object (Section 3.3). These two stages are alternated throughout training, ensuring balanced learning and preserving contributions of both signals (Section 3.4).
Reinforcement Learning with Human Feedback (RLHF). RLHF has been widely applied to align model output with human preference across various domains, including L.Mss (Brown al., 2020; Radford al., 2021; Meta, 2024),
'We adopt DPO (Rafailov et al., 2024) in practice and refer to it as RL here for simplicity, as it implicitly optimizes the same objective as traditional RLHF despite not being typical RL.
3

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
(b) The Sequential Leaming (SL.
(b) The Visual Feedback (VF)
Rejected CAD Probubility
Pretemed CAD
Profobiiity
Ground muth Sequence
CB Less
Predicted CAD Tokens
Preferred CAD object
Rejected CAD abject
Text prompts
Pre-trained Backbone LLM
Text prompts
Pre-trained Backbone LLM
(c) tteraniie training procedure
Pre-trained Backhone LLM
Initial Sequential Learning
Visual Feedback
Sequential Learning
Visual Feedback
Sequential Learning
N times
Ours
Figure 2. Overview of CADFusion. (a): The sequential learning stage trains LLMs using ground-trutt CAD parametric sequences. (b): The visual feedback stage rewards CAD parametric sequences that render into preferred visual objects and penalizes those that do not. (c): The two stages are alternated to preserve contributions of both signals.
3.2. Sequential Learning Stage
While existing studies (Khan et al., 2024b; Li et al., 2024b) also consider sequential signals, CADFusion introduces three distinctions: 1) it uses an LLM backbone to leverage pre-trained knowledge, unlike prior work that trains Trans- formers from scratch; 2) it represents CAD sequences as text tokens, processed with the LLM's tokenizer, whereas others use custom tokenizers; 3) its training data undergoes human annotation, while prior work relies solely on syn- thesized data. These enhancements enable it to outperform existing approaches, even without the visual feedback stage.
Text-to-CAD requires model capable of understanding tex- tual descriptions and generating CAD parametric sequences that adhere valid sequence formats and employ meaning- ful parametric operations. We adopt the following strategies to efficiently achieve these capabilities.
1) Model architecture. We use LLMs as the backbone, lever- aging their strong natural language understanding and basic CAD design knowledge (Makatura al., 2023).
2) CAD Paramelric Sequence Format. We adopt the for- mat proposed by Zhang et al. (2025) (Figure 1(b)), which represents CAD parametric sequences as text tokens rather than binary representations or numerical attributes (Xu et al., 2022; 2023; Wu et al., 2021). This text-based format simpli- fies processing and interpretation by LLMs.
3.3. Visual Feedback Stage
The goal of Text-to- CAD is to ensure the rendered visual object from the generated parametric sequence matches the desired visual object. Relying solely on sequential signals compromises training efficiency (see Section 1). To address this, we incorporate visual feedback into the model already trained on sequential signals (i.e., (x)))
3) Dataset. Existing CAD datasets (Wu et al., 2021) include CAD parametric sequences but lack paired tex- tual descriptions, To address this, construct dataset Dss= = {z, HM ('SL' for sequential learning) containing paired text & and CAD parametric sequences y. We first prompt a LVM to generate draft captions for rendered CAD model images and then refine these drafts through human annotation to ensure accuracy and conciseness.
Learning Visual Feedback through DPO. straightfor- ward way incorporate visual feedback is through super- vised learning, which minimizes the loss between the ren- dered visual object from the generated parametric sequence ô = r(f())) ) and the ground -oound visual object o. How- ever, since the rendering process r(-) is non-differentiable, this loss cannot be backpropagated to the model f(-). To address this, we reformulate the task as a reward maxi- mization problem, where visual feedback serves as the re- ward, enabling optimization without requiring differen- tiable rendering process. Since conventional RL is computa- tionally expensive, we adopt direct preference optimization (DPO) (Rafailov et al., 2024), a simpler and more efficient approach that implicitly performs reward maximization.
4) Training. We fine-tune the pre- mre-trained LLMs by mini- mizing the discrepancy between the generated parametric sequence ğ får (z) and the ground-true -ruth parametric se- quence y using cross entropy loss, denoted as LSt:
1 T
Ls.. -E (m.a)) Daz
Σ
log logp(ū vel (x) (1)
where T is the sequence length and p(-) is the predicted probability of the 1-th token by the model (x).
Specifically, we construct a preference dataset DVF (1,, ow or)] where on and or are rendered from the para-

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
(a) Sample sequence responses & Reader Images
(b) Assigning visual scores to CAD objects
(c) Form preterence data
from eligible pairs
Multi-Aspec: Grading Criteria
Input Command
Backhone after Secuantial Learning
<seepcceqq> <seqpcseeq>
CAD
18/10
7/18
700
LVM
3/18 9/10
Illustration of preference data construction. (a): Sample CAD parametric sequences and render them into visual objects. (b): Score the visual ohjects using LVMs with multi-aspect grading criteria. (c): Construct preference data based on L.M-generated scores.
Figure 3.
metric sequences Yu and Ur. representing preferred and less preferred visual objects, respectively. We then optimize the model to increase the likelihood of parametric sequences that produce preferred visual objects (yw), while decreasing the likelihood of those that yield less preferred ones (y:):
Shape Quality
II. Shape Quantity
Iti. Item Distribution
Trtanghular shape
woth
wit creclax holes
dreular
%
Figure 4. An illustrative example of the multi-aspect evaluation criteria used in LVM scoring. Note that the illustrations simpli- fied to conceptually represent each criterion.
LvF = - (e..
(2)
plù = Vwl Blog Blog p(ù = (x) Pre yd (m)
logar (Blog log
Pr = Yau (.)
3.4. Alternate Training
where p(-) is the predicted probability of a parametric se- quence under the current model (fvr(z)), (-e(()) the prob- ability under the reference model from the last round of sequential learning (&LL (z)) and B is scaling factor.
Each stage of the training process seceess-eeeee sequential learning and visual feedback has specialized focus. Excessive train- ing in one stage can lead to the degradation of skills acquired in the other. For example, we empirically observe that ex- tended training with visual feedback can impair the model's ability to generate well-formatted parametric sequences, a skill developed during sequential learning. Conversely, prolonged training with sequential signals can weaken the model's capacity produce parametric sequences that ren- der visually natural objects, a capability enhanced during the visual feedback stage. To mitigate this, we introduce an alternate training strategy (Figure 2(c)). The process begins with the sequential learning stage, ensuring the model ac- quires the ability to generate logically coherent parametric sequences. Subsequently, the training is divided into smaller blocks. Within each block, the model first learns from the visual signal, followed by the sequential signal, balancing the two objectives effectively.
Constructing Preference Data with LVM Scoring. Col- lecting preference data is both costly and labor-intensive The iterative use of the visual feedback stage in our frame- work (Section 3.4) further highlights the need for a quick and efficient approach for obtaining preference data. To address this, we propose leveraging the strong visual under- standing capabilities of LVMs score visual objects and construct preference data. Figure 3 outlines the pipeline. First, the textual description x is input into the finetuned model after sequential learning (fši. to generate multiple parametric sequences, which are then rendered into visual objects (e.g.. CAD images in our implementation). Next, the rendered CAD images, along with an instruction detail- ing the evaluation criteria, are input into LVM to obtain scores. Finally, the CAD image with the higher score is regarded as the preferred one (i.e., ow), while the one with the lower score is deemed as the less preferred one (i.e., oi).
4. Experiments
4.1. Setups
Specifically, inspired by recent work (Liang et al., 2024) on evaluating text-to-image generation across rich aspects, we incorporate multiple evaluation criteria into the LVM instruction. As shown in Figure 4, these criteria assess both the appearance of CAD designs and their alignment with textual descriptions: 1) shape quality evaluates the regularity, naturalness, and realism of the design; 2) shape quantity checks whether the number of components matches the description; and 3) distribution ensures components are arranged naturally, avoiding collisions or excessive spacing.
Datasets. For the dataset used in the sequential learn- ing stage, we use DeepCAD dataset (Wu et al., 2021) as the source for CAD parametric sequences (specifically the version processed by Xu et al. (2022)). We construct dataset compromising 20k pairs of textual instructions and CAD parametric sequence using the techniques intro- duced in Section 3.2 and Appendix B.3. For the prefer- ence data used in the visual feedback stage, we employ llava-onevi i is oo--qwe2277 (Li et al., 2024a) to
5

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
construct it using the method introduced in Section 3.3. For each iteration of the visual feedback, we generate approxi- mately 1,500 preference pairs, by using 1,000 text prompts as input, sampling 5 times per prompt, and filtering out in- valid or low-quality samples. For the test set, we construct it by splitting the dataset used in sequential learning into train, validation, and test sets with a 90:5:5 ratio.
dence between model predictions and input instructions. To this end, we employ GPT-4o with a dedicated evaluation prompt. Further details are provided in Appendix D.I. Fi- nally, we conduct human assessments to rank generations from different baselines, denoted as Avg. Rank. Details on this evaluation can be found in Appendix D.2.
4.2. Main Results
Implementation Details. LLaMA-3 3- 8b Instruct is used as the LLM backbone, with maximum token length of 1024. For efficient fine-tuning, we adopt Low-Rank Adapta- tion (LoRA) (Hu et al., 2022) with hyperparameters r = 32 and a = 32. The initial sequential learning stage lasts for 40 epochs with a learning rate of 1 x 10-4, using the A optimizer. Following this, we run 5 iterations of alternat- ing visual feedback and sequential learning stages. The visual feedback stage lasts for 5 epochs on the preference data, while the sequential learning stage lasts for epoch using the same dataset as the initial sequential learning stage. Training is conducted on four NVIDIA A6000-48GB SMX GPUs using PyTorch Distributed Data Parallel (DDP).
Quantitative Evaluation. Table summarizes the quantita- tive results comparing CADFusion with baseline methods (see Appendix D.4 for more details). Compared to GPT 4o, CADFusion outperforms it across all metrics. This suggests that while the general model may have acquired some CAD knowledge during pre-training, explicitly optimizing for Text-to-Can to-D as in our approach, is crucial for improv- ing performance, Compared to Tex12CA CADFusion achieves comparable or better performance on all metrics, with particular strengths in metrics reflecting the visual qual- ity such as LVM score and Avg. Rank. This highlights the effectiveness of incorporating visual signals in our approach, as opposed to Tex12CAD, which relies solely on sequential signals. This outcome also aligns with Khan et al. (2024b)'s s limitation statement that Tex12CA is limited to generating only rectangular and cylindrical shapes. When faced with complex geometries, struggles to perform effectively.
Baselines. We consider two types of baselines. The first is a specialized model for Text-to-CAD (Khan al., 2024b; Li et al., 2024b). We use Khan et al. (2024b) as our base- line, as Li et al. (2024b) is not open-sourced and we were unable to reproduce it ourselves. The second baseline is a general model that acquires some CAD knowledge during pre-training. We use the most powerful model, GPT-4o, as our baseline. Specifically, we apply few-shot learning, providing examples as input for GPT-4o.
Qualitative Evaluation. Figure 5 compares the re- sults among the ground truth, our method, GPT-4o, and Text2CAD on the test set. GPT-4o frequently fails to pro- duce renderable results across most test cases, which aligns with its high invalidity ratio (IR) reported in Table 1. While it occasionally generates valid shapes, its outputs are of- ten misaligned with the input prompts. Text2CAD generate well-formed shapes without irregular edges or corners. How- ever, it often produces oversimplified shapes and, for more complex prompts, tends to generate multiple cubes or pan- els instead of accurately capturing the intended structure. This aligns with its low invalidity ratio (IR) but poor visual scores. such as LVM score and Avg. Rank, in Table 1. CADFusion provides the most precise response to input in- structions and achieves the highest similarity to the ground truth. It successfully captures complex shapes, including rectangles, hexagons, and nested structures, such a hexag- onal hole within a cylinder. Additionally, it exhibits a strong understanding of language cues, accurately interpreting nu- merical and qualitative descriptors like "long" or "T-shape". Additional qualitative results, as well as our model's abil- ity to generate multiple varied outputs, are presented and discussed in Appendix D.6 and D.7.
Metrics. Our evaluation focuses on assessing the alignment of generated CAD models with input instructions and the overall quality of the generated CAD models. We employ the metrics at both the sequential level and visual level. First, to evaluate the correspondence between the ground-truth and generated parametric sequences, we use Fl scores fol- lowing Khan et al. (2024b). Specifically, we compute Fl score for primitives (averaged over lines, ares, and circles for brevity) and extrusions, denoted as F1-Sketch and F1- Extrusion. Second, to assess the quality of the generated CAD models, we compare the ground-trum truth and generated point clouds. We adopt Chamfer Distance (CD) from Khan et al. (2024b) and additional metrics from Xu et al. (2022), including Coverage (COV), which quantifies the percent- age of real data covered by generated samples using CD; Minimum Matching Distance (MMD), which evaluates the closest match between generated samples and real data; and Jensen-Shannon Divergence (JSD), which measures distri- bution similarity. Additionally, we compute the Invalidity Ratio (IR), which quantifies the percentage of generated parametric sequences that fail to render into valid visual objects. Furthermore, we introduce an LVM-based met- ric, denoted as LVM Score, to assess the visual correspon-
"Our comparison with Tex12CAD is not entirely aligned and is in favor of it. Performing poorly on prompts we provided, we have present the results of Tex12CAD tested with their original prompts. We detail this problem in Appendix D.4.
6

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
FIf
Method
CD4
COV
MMD
JSD 4
IR
LVM Score
Avg. Rank
Sketch
Extrusion
GPT-4o
82.96
85.72
68.50
72.40
6.60
37.93
74.26
5.13
3.22
Texf2CAD
63.94
92.13
30.23
3.37
2.01
2.97
CADFusion
85.22
92.79
19.89
90.40
3.49
17.11
6.20
8.96
1.86
Table 1. Quantative results- Test results on Fl scores including Sketch (primitive, averaged) and Extrusion, Chamfer Distance (CD). Coverage (COV), Minimum Matching Distance (MMD), Jensen-Shannon Divergence (JSD), Invalidity Ratio (IR), the LVM Score the average rank from human evaluation (Avg. Rank). An upward arrow (t) indicates that higher values are better, while downward arrow (4) signifies that lower values are preferred. Since Tex12CAD does not release COV, MMD, and ISD, we were unable compute them ourselves due differenees in setup, these values unavailable.
Prompts
cylinder hole smaller wall thas"
(2) shape is rectingolar semicylinéncal ouiou center fomning Usdtsped channe!"
Ground
Truch
Ours
GPT-4o
Text2CAD
stape cylndrcaa lsnghiwise, resembling half-pipe."
"The 3D shape squar bexagonal plate."
(3) "The 3D shape is teardrop like piece with two ciroalar holts. use the browsee and and small aour the ixtrower and.""
shape uquree top.
"The comprued of four veriical cylinders, muuhhty usevenly distibuted the four
"The shape trapeonid
excedding top
(9) "Theee idensical zectangular shees placed verically, aranged peralia) and evenly speced"
(10) "The shape is hollow cylininccal band with vertical secoa removed, resemibling an incomplate ring"
(11) "The 3D shape hollow iiangular prism. The walls are the have smalle thickness"
two pozallel siim pipes.
ree-dmmmnttonal
(14) "The thre-dieraioall shape a cylinder"
(15) "Th 3D shape recingall "esst(ssbou))
hillspinke
reconnolle peam
(FF) A The innangile plvtr Tair comeer en mintied and timee arcularhole zarte duurater each
(18) "The 3D slape usssob of smaal Bia redangulaa miblle the side rectinplarpioos
(19) "The ssape hesagonal prem The hollow cener toens an open besaganal
(20) "The 3D doope is redannglir cebost with rounded edges conves..
Figure 5. Qualitative results. The input prompt is shown at the top of each subsection. Images are arranged from left to right in the following order: ground truth, CADFusion, GPT-4o, and Text2CAD. Outputs that cannot be rendered are marked with a red cross. CADFusion outperforms all baselines in understanding instructions and generating CAD objects that are both sequentially and visually high quality. GPT-4o frequently produces invalid samples and pays little attention to shape details. TextCCAD generates well-formed basic shapes with regular appearance but struggles accurately follow input instructions and represent complex geometries.
4.3. Ablation Studies
back, we conduct an ablation study on CADFusion using only sequential learning, denoted CADFusionsL. The first row of Table 2 presents its LVM score and invalidity ratio. Compared to our approach, denoted as CADFusiong, vFSLaS) in Table 2, while CADFusionsL. improves the invalidity ra- tio by 1.36%, it results in a considerable decrease in the
We conduct ablation studies on the effectiveness of the vi- sual feedback stage, the impact of the alternate training, and the choice between human and LVM annotation for data.
Visual Feedback. To assess the importance of visual feed-
7

---

Text-to-CAl Generation Through Infusing Visual Feedback in Large Language Models
Method
LVM Score f
IR +
that it fails to effectively balance the contributions of both sequential signals and visual signals.
CADFusionsI.
7.69
4.84
CADFusion
6.56
6.00
We also compare model variants that use different numbers of iterations of visual feedback and sequential learning. In Table 2, for each CADFusiongL VESL(() variant, the number in parentheses indicates the number of alternative training rounds performed. The results for iterations 1, 3. and are reported, showing a gradual increase in LVM scores along with stable invalidity ratio. This further validates the effectiveness of our approach.
CADFusionsI-VF
5.94
88.87
CADFusionsL-VEun
6.21
3.46
CADFusionss..SSL....m
8.28
17.03
CADFusionsL.vASLL
8.76
4.42
CADFusionsi.vTs..0
8.89
4.21
CADFusionsI.vFSL.S
8.96
6.20
Table LVM scores and invalidity ratios across different CADFu- sion variants. The suffix SL indicates that the model is trained with the initial Sequential Learning stage, while VF denotes the Visual Feedback stage without additional Sequential Learning. VFLL represents Visual Feedback with alternating Sequential Learning. The tag w/ HA signifies that the data is preprocessed with human annotation, whereas w/o o HA denotes the absence of human an- notation. Numbers in parentheses indicate the number of VFSI, rounds performed. RPO refers to the model using Regularized Pref- erence Optimization (RPO) (Liu et al., 2024) to stabilize DPO.
Data Annotation. We examine the impact of our choice of data annotation. In the sequential learning stage, the dataset constructed by first using LVMs to generate initial captions, followed by human annotators refining them. To evaluate the effect of this decision, we conduct an experi- ment in which our method is trained on data without human annotation, denoted as CADFusions The second row of Table 2 presents the results. It shows worse LVM score and IR compared to the version using data with human amno- tations (CADFusionsL), highlighting the necessity of human annotation in the sequential learning stage.
LVM score. This underscores the crucial role of the visual feedback stage: by leveraging visual preference data, our framework effectively enhances the visual quality of the generated CAD models. Additionally, CADFusionsL. out- performs the baseline method, Tex22CD, which also relies solely on sequential signals. Note that this advantage is achieved using 20k data, while Tex12CAD uses 150k data. This demonstrates the effectiveness of the techniques em- ployed in our sequential learning stage, including leveraging LLMs as the backbone, representing CAD parametric se- quences as textual tokens, and utilizing human annotations (Section 3.2).
In the visual feedback stage, LVMs used to score CAD models and generate preference data. This design choice is driven by the high cost of human annotation and the challenge of managing human annotators to en- sure consistent scoring. To evaluate the effect of this de- cision, we conduct an experiment where the visual feed- back stage of our method trained on numan-scored prefer- ence pairs, denoted CADFusionsL VFSL Compared to the LVM-scored version (i.e., CADFusionsL- achieves worse LVM score and IR. This aligns with our intuition that, while human annotation may be more ac- curate, managing annotators for consistent scoring is diffi- cult. Furthermore, using LVM-scored preference data al- lows CADFusiong..vrssuu to scale across more rounds of visual feedback (e.g.. CADFusionsL leading im- proved performance. Achieving this with human annotation would be challenging and expensive.
it
Alternate Training. In Section 3.4. propose an alter- nate training strategy to retain the benefits of both sequen- tial learning and visual feedback stage. We compare this approach with three variations: 1) visual feedback only (CADFusionsL-vp)) 2) visual feedback with an additional Negative Log Likelihood loss CCADDusionsL-Veer to regu- larize and stabilize DPO (Liu al., 2024), and 3) iterative visual-sequential training (our method).
5. Limitation
Table 2 presents the results, with our approach denoted as CADFusionsL-vesLss The high invalidity ratio of CADFusion indicates that it struggles to generate ren- derable sequences, suggesting that extended training with visual signals can impair the model's ability to generate well- formatted parametric sequences. Besides, CADFusionsL.VE receives low rating from the LVM judge, revealing that training with visual feedback along provides limited bene- which incorporates the additional loss, while it achieves low invalidity ratio, its visual quality, as assessed by the LVM judge, is even lower than the SL-- setup (i.e., CADFusionsL). This indicates
CADFusion's 's results are overall promising. However, there are limitations that could be addressed in future work. First, modern LVMs suffer from performance drop when handling multiple images input. Currently, LVM with a single-view image to ensure both accurate im- age understanding and prompt following. This limitation prevents us from achieving a more effective Visual Feed- back pipeline and evaluator. Second, CADFusion struggles to generate very complex shapes that require spatial and commonsense reasoning, such as the shapes of letters and words (see Appendix D.8).
only provide
fit. Regarding CADFusionsL-vee
8

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
6. Conclusion
M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
We propose CADFusion for Text-to-CAD, the first approach to incorporate visual feedback from rendered CAD objects into the training pipeline. CADFusion uses LLMs as back- bone and alternates between the sequential learning stage and the visual feedback stage. We conduct extensive exper- iments to demonstrate the superiority of CADFusion and validate the effectiveness of the design choices. In the future, we plan to further improve the preference data construction pipeline to enhance performance, and collect more CAD data with more complex geometric shapes investigate CADFusion's 's performance more challenging cases.
Bubeck, S., Chandrasckaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg, S., Nori, H., Palangi, H., Ribeiro, M. T.,, and Zhang, Y. Sparks of artificial general intelligence: Early experiments with gpt-4, March 2023. URL https: //arxiv. www. abs//bb//220
Deng, Y., Chen, J., and Olechowski, A. What Sets Proficient and Expert Users Apart? Results of Computer-Aides Design Experiment. Journal of Mechanical Design, 146 (1):011401, 10 2023. ISSN 1050-0472. doi: 10.1115/ 1.4063360. URL httph::/p////. //tp:::///di..o /tw.//iooooooooooo http:///dooooog////// org/10 1115/1. 4063360.
Acknowledgement
We thank Weijian Ma for insightful conversation and dis- cussion. We also appreciate the informative comments and suggestions provided by the anonymous reviewers.
Du, T., Inala, J. P., Pu, Y., Spielberg, A., Schulz, A., Rus, D., Solar-Lezama, A., and Matusik, W. Inverseosg: au- tomatic conversion of 3d models to csg trees. ACM Trans. Graph.. 37(6), December 2018. ISSN 0730- 0301. doi: 10.1 10.11455/7722227.2255006 URL https: 10. 1 / 327113//77222..373770000 3275006.
Impact Statement
This paper presents work aimed at improving Text-to-CAD generation through the use of L.L.-bbseed frameworks and the incorporation of visual feedback. Our work has the potential to enhance the CAD design process, offering the benefits of automation and efficiency while reducing re- liance intensive training and specialized expertise. This could make CAD design more accessible, particularly in industries where skilled designers are in short supply or where rapid prototyping is essential.
Hong, Z., Yuan, Z., Chen, H., Zhang, Q. Huang, F,, and Huang, X. Knowledge-to-SO2 Enhancing SQL gen- eration with data expert LLM. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp. 10997- 11008, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/204, andings-acl.663 URL. https:://// a org org/2020 2224/222.202 findings acl. 653/.
Ethics Statement
http://aaaaaatoooogooeoo anthology.
In this work, we have invited crowd workers to give textual descriptions to CAD models. We conducted this work in ac- cordance with ethical guidelines to ensure that participants were treated fairly, respectfully, and safely throughout the process. We took steps to protect the privacy of crowd work- ers by not collecting personally identifiable information. The data annotated by the crowd workers was used only for research purpose related improving CAD generating techniques.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openrevies eww net forumorrmm//m din.zddeeeeeeee.....
Jayaraman, P. K., Lambourne, J. G., Desai, N., Willis, K., Sanghi, A., and Morris, N. J. W. Solidgen: An au- toregressive model for direct b-rep synthesis. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL https: wwwpneeeee..eeeeeeeo//p/e. nnt/ forum? id= (d-2R2CDgADRo. Featured Certification.
References
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss A., Krueger, G.,
Kania, K., Zieba, M., and Kajdanowiez, T. UCSG-NET- unsupervised discovering of constructive solid geometry tree. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, L., and Amodei, D. Language models few-shot learners. In Larochelle, H., Ranzato,

---

Text-toCAAD Generation Through Infusing Visual Feedback in Large Language Models
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. A survey of reinforcement learning from human /
153. URL https:://cclaatttt www.hhhho.ooo.../222....33 emnlp-main.153.
feedback, 2024. URL ptttps://axxi...o 14125.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss implicitly an adversarial regularizer. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp. 198663-138699 Curran Associates, Inc., 2024.
Khan, M. S., Dupont, E., Ali, S. A., Cherenkova, K., Kacem, A., and Aouada, D. Cad-signet: Cad language inference from point clouds using layer-wise sketch instance guided attention. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 4713-4722, June 2024a.
Khan, M. S., Sinha, S., Uddin, S. T., Stricker, D., Ali, S. A., and Afzal, M. Z. Text2cad: Generating sequential CAD designs from beginner-to-cxper level text prompts. In Advances in Neural Information Processing Systems, vol- 37, pp. 7552-7579, Curran Associates, Inc., 2024b.
Ma, W., Chen, S., Lou, Y., Li, X., and Zhou, X. Draw step by step: Reconstructing cad construction sequences from point clouds via multimodal diffusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27154-27163, June 2024.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF: scaling rein- forcement learning from human feedback with AI feed- back. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpsttttt/... /tappp:://eeeee http///epeeeeeeeeeeeeeeee.....er net/ htt.//oommnddduydd2222
Makatura, L., Foshey, M., Wang, B., HähnLein, F., Ma, P., Deng, B., Tjandrasuwita, M., Spielherg, A., Owens, C. E., Chen, P. Y., et al. How can large language models help humans in design and manufacturing? arXiv preprint arXiv:2307. 14377, 2023.
Meta. The Ilama herd of models, 2024. URL https:
/ //arxiv.org// abs 2407 21783.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 2024a.
OpenAL. Gpt-4 technical report, 2024. URL https:// arxiv.org httgg/bbs/23000 08774.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 8763. PMLR, 2021. URL http::/proceeeiiii mir www.ppressvlll... http:///11///aafodd2222 www.oradfordaaaaaaa html.
Li, X.. Song, Y., Lou, Y., and Zhou, X. CAD translator: An effective drive for text 3d parametric computer-aided design generative modeling. In ACM Multimedia 2024, 2024b. URL https: :ttttt::://peeee htwpperrevieeeee eew. wwt. forum? id-DN3722rnLd.
Liang Y., He, J., Li, G., Li, P., Klimovskiy, A., Car- olan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., Ke. J., Dvijotham, K. D., Collins, K. M., Luo, Y., Li, Y., Kohlhoff, K. J., Ramachandran, D., and Naval- pakkam, V. Rich human feedback for text-to-image generation. In JEFF/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 19401-19411, IEEE, 2024. doi: 10.1009CVVPR2233..22000022 doi oro...rg/.....
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, D., and Finn, C. Direct preference optimization: Your language model is secretly reward model, 2024. URL https:
/taddb///333..2 2.30. 8290.
URL 2024. 01835..
Seff, A., Zhou, W., Richardson, N., and Adams, R. P. Vitru- vion: A generative model of parametric CAD sketches. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. Open 2022. URL wtt.hhtps///ooeeee httttss:/opeerevierr net
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. G-eval: NLG evaluation using gpt-4 with better human alignment. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2511-2522 Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10. 8653/v 1/2.l1.222222202 2e0003..mmlpmmmmm
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and
10

---

Text-toCAAD Generation Through Infusing Visual Feedback in Large Language Models
Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL httts:://rxii..... abs/2302, 13971.
Zhang, J., Hou, Z., Lv, X., Cao, S., Hou, Z., Niu, Y.,
Hou, I.., Dong, Y., Feng, L.,, and Li, J. Longreward: Improving long-context large language models with ai feedback, 2024. URL https:
httpss//axxiv..rrre http:////axvvvo./g.o/.e
Wang. K., Zheng, J., and Zhou, Z. Neural face identification in a wireframe projection of manifold object. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1622-1631, June 2022.
Zhang. Z.,, Sun, S., Wang, W., Cai, D., and Bian, J. Flexcad: Unified and versatile controllable CAD generation with fine-tuned large language models. In The Thirteenth Inter- national Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025. URL https: http::h/tttpteeeeeeeeee httpp//pppweeeeeeeeeeeeeee http://eee/ooooo/ooeeeoee/////o forum? id=20eiiv3Yyh.
Willis, K. D. D., Jayaraman, P. K., Lambourne, J. G., Chu, H., and Pu, Engineering sketch generation for computer-aided design. In 2021 IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition Work- shops (CVPRW), pp. 211-,114, 2021. doi: 10.1109) CVPRW53098.208 202.00098..22.000000 .00239.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J.. Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. Survey of Large Language Models, May 2023. URL http: 18223. arXiv:2303.1222
Wu, R., Xiao, C., and Zheng, C. Deepead: A deep gen- erative network for computer-aided design models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 6772-6782, October 2021.
wwx.v.orgaabs////oooo.........st [cs].
Wu, X., Huang, S., Wang, G., Xiong. J., and Wei, F. Boost- ing text-to-video generative model with milms feedback. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Pa- quet, U.. Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 199444-139499 Curran Associates, Inc., 2024.
Xu, X., Willis, D., Lambourne, J. G., Cheng, C.-Y,, Jayaraman, K., and Furukawa, Y. Skexgen: Autore- gressive generation of cad construction sequences with disentangled codebooks. In International Conference Machine Learning, pp. 14698-247722 PMLR, 2022.
Xu, X., Jayaraman, P. K., Lambourne, J. G., Willis, K. D., and Furukawa, Y. Hierarchical neural coding for control- lable cad model generation. International Conference Machine Learning, pp. 38443-38444 2023.
Xu, X., Lambourne, J., Jayaraman, P., Wang, Z., Willis, K., and Furukawa, Y. Brepgen: A b-rep generative diffusion model with structured latent geometry. ACM Transactions on Graphies (TOG). 43(4):1-14, 2024.
Yu, F., Chen, Z. Li, M., Sanghi, A., Shayani, H., Mahdavi- Amiri, A., and Zhang, H. Capri-net: Learning compact cad shapes with adaptive primitive assembly. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11768-11778, June 2022.
Yu, F., Chen, Q., Tanveer, M., Mahdavi Amiri, A., and Zhang, H. D2CSG: Unsupervised learning of compact csg trees with dual complements and dropouts. Advances in Neural Information Processing Systems, 36, 2024.

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Data Preprocessing
VLM caption
Human revisson
oriee
SkexGen
B: 3
Flatten to string
2,2 courvn-ents
Synthesized
le beuee
Hi Sas, 180,
rabelct ands eaxt readinn
Toxt-to CAD data
Figure 6. An overview of the data preprocessing steps. The original dataset is transformed into captions that serve as textual inputs, while the corresponding stringified CAD representations used as ground truth references.
A. User Guidelines for Prompting
A good prompt follows a structured description: (1) shape overview, (2) shape details, and (3) shape applications. Given the varying shape complexity, we encourage but do not enforce describing item (2) and (3). Below is an example caption retrieved from Figure 8, Row 2, Item 3, demonstrating this approach:
[Shape Overview] The 3D shape consists of large, flat rectangular slab with evenly spaced, identical cylindrical protrusions extending vertically from its surface. [Shape Details (Opticnal)] The slab provides stable base with significant length and width compared to its thin height, while the cylinders are relatively short and have small diameters. [Shape Applications (Optional)] The overall design is synnetrical and balanced, potentially serving as mounting base or connector.
Listing I. User guidelines for prompting.
B. Additional Dataset Construction Detail
B.1. Converting Raw Data into Strings
CADFusion's String Format Our representation adopts the Sketch-and-Extee Modeling (SEM) format, wherein a CAD instance is composed of sketches and extrusions. Each sketch structured into multiple faces, and each face comprises multiple loops. Within each loop, geometric primitives such as lines, ares, and circles are parameterized follows:
Line: Represented by line identifier and one coordinate.
Are: Defined by an are identifier and two coordinates.9
Circle: Represented by circle identifier and four coordinates.
Each extrusion is represented as a sequence formatted as BWITTTRRRRPRRSOO, where the components are defined as follows:
B: The boolean operation, selected from add, cut, intersect.
'While lines and ares generally require 2 and 3 coordinates for representation, respectively, this work leverages simplified representation where the endpoints of lines and ares determined by the first point of the subsequent curve. If the loop closed at the current curve, endpoint determined by first curve the lnop.
12

---

Text-to-CAl Generation Through Infusing Visual Feedback in Large Language Models
CAD Regresentzaton SEM format
Flaitened CAD Sequence
Textual Description Retileeed via LVM Capcicning
(eeuagt 01,
(The 3D shyp is squie plate with lange cencal sylindicaal hale faces, jelgs,, and 5 vertoos. The plate is relatively this compared to its side jength likely used mounting strcctraal cemenor with the central hole for shaft or pipe and the cocner holes for bolts for serews..
COORD_ EXT PAD
forr smalle cytinchical holes each
121
31
31
[18) 681
481-
122
22-
621
155
443
181
line,9,9 end- line,5a5 end- line,
9.53 courve. sloop ends
[99, 283-
1-1-
190
481
183
18.
83
21
(r0,e,14..8,14..11..28 14,11, ocurve end> <loop end> 14, A8, courve and> <loop end> end and> circle, 51,1,, couvee and> <lopp and- circla,48,51,,88.4 A8,,4,,1,,,,,4,,4 44,45 oourve and -oooppoppppee ands
11-
9.
511
Circle,
"Me.ses" [errart
circle,31,44,333 31,,3,1,4,,1110
Fretual Descnition Afler Uumen
Annetution
The sp clope
(385. 600 504D
5405
2.2. 1579, 723,
<lece wwdl <skeichh und-
Jeeerui gybbutica phoogh- h-ost four craaler didiiiaal holee
L187. 2028, 2098,
1984, 756, 163, 546,
cextrude_erge end>>
The plan relatiedy compurt pide segph iiesty andd rounting la structaed reaferonnecs wits cautral bulk for slafl prptp and camaa lodes lez ho
(2x, 12, 22,
Figure 7. An overview of multiple CAD representations and their corresponding captions. Left: A CAD representation in the raw SEM format alongside its stringified sequence, with values highlighted in different colors based on the padding used for decoding. Right: Captions generated by the LVM and refined by human annotation. Phrases removed during human fine tuning are marked in red, while those added by humans are marked in green. All representations and captions correspond to the same CAD figure, which is displayed in the bottom-right corner.
V: The displacements of the top and bottom planes from the reference plane.
T: The 3D translation vector.
R: The 3D rotation, represented quaternion or equivalent.
S: The scaling factor.
O: The center of scaling.
Converting Source Data to CADFusion's Format The original representation derived from the SkexGen dataset (Xu al., 2022). Each CAD instance includes several components: sketch commands, sketch coordinates, and extrusion commands, which are stored in the se_cmd, se_xy, and se_ext entries, respectively. The lengths of these entries correspond the number of sketch-extrusion pairs within the complete CAD shape. To convert these entries into strings, we iteratively describe the sketches and extrusions in our format, ensuring that the resulting sequence reflects the chronological design order of the CAD process.
In iteration i, we select the i-th item from the se se__y__________seeeeeee_ee_eees_ se_ ___dd, and se eeeeettttteeeeeeeeeeeeete_e__tee entries. For each digit in the se_ se_emd array, we perform operations based on the command value as follows:
Command value = 5: Create a cirele; use the first 4 items in se se_xyy___eeeeeeeeeee as XY coordinates and append the <ourve_ent _ndd> token. Skip 5 positions in the se_xy array.
Command value = 4: Create an are; use the first items in se _e_xy__ as XY coordinates and append the Scurve_ends token. Skip 3 positions in the se_xy array.
Command value- = 3: Create line; use the first item se_xy as an XY coordinate and append the <curve_end>
token. Skip 2 positions in the se_xy array.
Command value- = Mark the end of the loop by appending the <loop_end> token. Skip position in the se_xy array.
Command value 1: Mark the end of the face by appending the <face_ token. Skip position in the se_xy array.
Command value = 0: Mark the end of the sketch by appending the caketch seketch__edd> token. Skip position in the se_xy array.
13

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Extrusions are represented by the ID array se_ext. The operation identifier translated into word, and the remaining values are flattened. To distinguish coordinates from special tokens, all coordinates initially padded; they subsequently unpadded based on the original padding values. Figure illustrates the conversion process from the SkexGen representation format to our stringified sequence.
B.2. Generating Textual Instructions
Textual instructions generated in two steps: first, by applying LVM captioning single-view images of CAD models; second, through human refinmmen of the generated captions to ensure clarity and accuracy.
Given a sequence representation, the CAD instance is rendered into an image, and captions are generated using GPT-4o. This step is designed to extract geometric properties, including the number shapes, their dimensions, spatial arrangements, and other relevant details. The prompt used for this step is provided Listing 1.
"Prompt "Propose a series questions about first question should ask for a detailed description and others should focus on the specifis fic geometric properties, number, size proportions and positional relationship and other details.", "Prompt2": "Based the dialogue, please give final description of the 3D shape. No more than 70 words."
shape and give the answers. The
Listing Prompts that used for making captions. The first prompt is used to generate question-answe pairs, and the second prompt collects and summarizes the informations in the first prompt yield the final caption.
The W-ggeerrtted captions are further refined by human annotators to produce fine-grained captions that can serve as precise textual instructions. The human annotators follow these guidelines during the editing process:
Ensuring Correspondence:
The description must accurately reflect the figure without any discrepancies.
Ensuring Succinctness: The description should be concise as possible while maintaining clarity and completeness.
Permission for Removal: Figures that are excessively complex or challenging to describe may be excluded from the dataset. In practice, the annotators are permitted to mark the revised descriptions of such instances as "null".
Figure illustrates an example of how an image is captioned by the LVM and subsequently refined by human annotators.
B.3. Dataset Construction
The dataset construction process is illustrated in Figure 6. Starting with a CAD representation from the original dataset, we generate a paired textual instruction and stringified CAD representation. The textual instruction is created through the captioning process detailed Section B.2, while the ground truth reference obtained by converting the CAD formatting described in Appendix B.I.
C. Additional Training Detail
C.1. Sequential Learning
We fine-tune a LLaMA- 3-8b 8b Instruct by 40 epochs on 4 NVIDIA A6000-48GB SMX GPUs with a LoRA with rank
32. Further details regarding the fine-tuning process are provided in the Experiment Section of the main paper. The specific prompt used for the learning is as follows:
"Below is description of shape: An
[description) Generate a Computer-Aides Design (CAD) command sequence of the 3D shape An"
Listing Prompt used sequential learning. description refers the actual textual commands samples
14

---

Text-to-CAl Generation Through Infusing Visual Feedback in Large Language Models
Rectangular Panel
"Tutt teteee surte couppiitie
swor.
wi oig sugeel dhis du 3. dipere un puoee pereeee coeerepe noa
"Te
eoegl de denoer da
al lee oe lape bi den boie
"Te de andur my la muugie Lak monk dD de compaar
vialis
prur
wate oper
meses claning rade did
parsadly unring saveiing
Rod Objects
"Ix leps log Ande rocepll aociid inno
vee des welend mndile cmserna
bo own ou dogs autine waandd
supti ont we couett Th berg Bo
cape regon in umplatt ceelidyy shied bageica asidd serr dda
"Tee the chilk with
The
you vere prodecel to oo to
Circalar Dijjects
"Bu Oher ceplectall du mil hat
Expooiioo non lilea
The Seevica and Podeleccn
yuuup unaing crmanii
detern ad liet cnshanal Enege carn peneldyy
The Sasu fe Eatrad abolied withoe 4gc34 Sacea Ohicll verglyy coaing
'C' cign de del
deston laber aores Thrma
Lage anah Ta verai aant
Figure Additional qualitative results, Part The results are grouped by categories such panels and circular objects. In each sub-figure, the left image shows the figure rendered from the ground truth, while the right image displays the generation by CADFusion. The corresponding textual instructions are provided at bottom.
C.2. Visual Feedback
The visual feedback is collected outlined in the main paper. The is utilized to generate visual descriptions. For each input sequence produced by the post-SL CADFusion, the corresponding rendered figure is evaluated by the model, which assigns a score ranging from to 10. The prompt used for this scoring process is detailed in Listing 3:
model
"You are a harsh grader for new CAD designers' works. The following is text description of CAD fi gure that they desi designed and an image of CAD instance. An Description: (description] \n Comment on this work for \n
1. If the overall shape remains correct; An
2. If the number of components are correct, especially the circular holes; An natural, i.e. they not clustered
3. If the distribution of the cemponents together or collide with each other. \n
After that, give a score out of 10. Do not comment on issuos such as texture, smoothness and colors."
Listing 4, Prompt used by LLaVA-OV OV for scoring an input figure. description refers to the textual of the sample.
The DPO procedure conducted on 4 NVIDIA A6000-48GB SMX GPUs with LoRA with rank 32. The training involves
15

---

Text-to-CAl Generation Through Infusing Visual Feedback in Large Language Models
Plaral Items
The mn the ooe shond emmsareee ndaa Tansseee
mere un dede andden aeheroe
viauu
entel.
Blociular Olbjects
TDuip andiiie
Einu bes Apped puil laglle digend"
dwne nos perte diger acriing unbel
the Mene""
Complex Shapes
The Root dencrite serneeglee odter
beeer cone snd wheer seee seed
adteeo monoeec roaele Copppceeene
Others
petopderrinn On deree
"3t Aie F1SDeBC poat Ela laga appaedd
cneegle thee serti far Meataal
Figure Additional qualitative results, The results are grouped by categories such as multiple distinct items and complex shapes. In each sub-figure. the left image shows the figure rendered from the ground truth, while the right image displays the generation by CADFusion. The corresponding textual instructions are provided the bottom.
five iterative DPO/SFT rounds, which require approximately 2.5 days to complete.
D. Additional Experimental Results
D.1. LVM Evaluation Setups
As mentioned in the main paper, used CPT-4o model as the LVM evaluator. is selected over LLaMA-OV because attempt to prevent the impact of Al bias that makes it prefer its own generation 4, The prompt we used for LVM evaluation is detailed in Listing 4:
The following is text description of CAD figure and image of CAD instance. Measure the figure corresponds to the given description, and give score the scale of 10. Do not conment on issues such as texture, smoothnoss and colors An description: [description/ An
Listing 5. Prompt used by GPT-4o for evaluation.
D.2. Human Evaluation Setups
We generate a quadruple of outputs for each test set instruction. Each quadruple presents four rendered generations from CADFusionsL, CADFusion, GPT-4o and Text2CAD, respectively. The generations are tested by their correspondence between CAD shapes and instructions. Six human judges (with college-level higher education records) are asked to rank the generations with the first place being the best model. The ranks are then collected and averaged to the scores we
*We acknowledge this choice may introduce bias the GPT-4o results. However, we have decided proceed with two reasons: 1) our primary focus on comparing Tex12CAD with our model, and 2) the GPT-based generations during our experiment showed big margin in L.M scores compared to other methods, the impact of this bias is minimal. Due the lack overlapping, we obtain approximately unique samples.
16

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
presented as human evaluation.
D.3. GPT Baselines
The prompt we used for the GPT-4o baseline is detailed in Listing
Helow is descr intion of 3D shape:
(description)
Generate a Computer-Aides en (CAD) command sequence the 30 shape. The command sequence involves sketches such as lines, arcs, and circles, each marked by the endpoints, and extrusions that make the sketch into 3D volumes.
Here are seme examples:
1. <few shot example>
2. <few shot example>
3. <few shot example>
4. <few shot example>
11
5. <few shot example>
12
6. <few shot example>
13
7. <few shot example>
8. <few shot example>
15
16
Now it's your turn. Remind that this your description: dlescription No explanation is needed. Only return your final sequence, and in one line.
17
Listing 6. Prompt used by GPT-4o for baseline comparison.
Alongside the current 8-shot version, we also tested a 3-shot GPT-4o model to reduce computation costs. However, the 3-shot model resulted in approximately 92% invalidity ratio, and the 8% of renderable outputs were barely recognizable in relation the prompt. Given these issues, have decided to use the 8-shot version as our baseline for GPT.
D.4. Additional Statements on Text2CAD Results
In the quantitative experiments, our setups are not fully aligned with those of Text2CAD, This discrepancy arises because when we used our test set prompts input, we observed performance degradation and significant gap between our computed results and those reported by the authors.
We discovered that the discrepancy stems from the model's sensitivity to the level of detail in the prompt. Text2CAD performs well only with expert-level prompts, which contain step-by-step sketching guidelines. Our prompts, however, do not include this level of detail To ensure consistency, in Table 1, we report Tex12CAD's performance based on their expert prompts when computing the metrics they introduced. Specifically, for each item in the test set, CADFusion and GPT-4o's s predictions were generated using our prompts, while Text2CAD's predictions were generated using the expert prompt for the same item from their prompt base.
This approach aligns the results we reproduced with the reported scores from the original paper. To present comprehensive and accurate study, we also report Text2CAD's results using our prompts and intermediate-Ievel prompts in Table 3. The last two rows, CADFusion and Tex12CAD-ourproome
are aligned as the same prompt is used.
By changing the prompt from the expert-level prompt in their database to an intermediate- prompt, we observe a similar performance drop. This indicates that our prompting method does not degrade Tex12CA'ss performance. Instead, an limitation stemmed from Text2CAD itself. Our model, using simplified prompt, outperforms Text2CAD experr.. Given that the expert-level prompt from Tex12CAD is too long and specific be feasible in the real designing process, believe that our quantitative advantage over it is non-trivial.
WWe concerned that the impact of detailed prompts containing step-by-step instructions and point coordinates is limited, they may not be feasible in real-life scenarios.
17

---

Text-toCAAD Generation Through Infusing Visual Feedback in Large Language Models
FIf
Method
CD4
Line
Arc
Circle
Extrusion
Text2CAD-intermeeditte
66.65
4.85
47.62
93.56
146.15
Text2CAD-expert
79.59
42.79
69.45
92.13
30.23
Text2CAD-our-pompp
54.42
0.92
18.42
75.37
235.91
CADFusion
83.71
81.99
89.97
92.79
19.89
Table Results of our model and different Tex12CAD prompts on metrics Proposed by Khan et al. (2024b). The suffix indicates the prompt type used for testing. Tex2CAD-ours and CADFusion are the most aligned pairs, while Texe2CAD-expet and CADFusion are the ones reported in the main paper.
Method
LVM Score t
IR $
CADFusionst.
7.69
4.84
Method
Avg. Rank
CADFusionsL. RA-18A
6,56
6.00
GPT-4o -8shot
3.22
CADFusions HAw170k
6.60
9.04
Text2CAD
2.97
CADFusion-SPP only
2.03
Table LVM scores and invalidity ratios across different CADFu- sion variants. All three models are trained using only the initial Sequential Learning stage. The suffix w/o HA indicates that the variant does not use human-annokee annotated data, while the number de- notes the size of the training set.
CADFusion
1.86
Table 5. Human Evaluation Results. Human annotators ranked the generations of different methods based on their quality, with a lower rank indicating higher human preference.
D.5. Additional Quantative Results
We report additional quantitative results in this section.
On Dataset Size. The dataset used in our experiments is subset of SkexGen (Xu et al., 2022). Since human annotation is not scalable, we evaluate the trade-off between scalability and data quality. One such evaluation, discussed in the Ablation Study (Section 4.3), demonstrates that, given the same number of training samples, data quality outweighs dataset scalability in terms of model performance.
Additionally, we investigate whether increasing dataset size can mitigate quality limitations by conducting an experiment on the full SkexGen-based Text-to-CAD dataset 170k samples). The results, presented in Table 4, indicate that increasing dataset size does not significantly improve the visual quality of model generations. While slight performance gain is observed with additional training samples, the improvement is marginal, and none of the w/o HA variants outperform the human-annotated counterpart.
On Human Evaluation. We conducted human evaluations across four models: GPT-40, Tex12CAD, CADFusion, and CADFusion trained only with the Sequential Learning stage. However, only the first three models are reported in Table 1. The complete results of human evaluation are presented in Table 5. As indicated by the evaluation, the two CADFusion variants are preferred over the baselines, with the version incorporating Visual Feedback receiving higher rankings from human judges. This highlights the effectiveness of visual feedback in improving model performance.
D.6. Additional Qualitative Results
In this section, we present additional qualitative results. Figures and display these results, organized by CAD shape properties such as panels and circular objects. These examples demonstrate that our model can efficiently handle variety of CAD shapes with distinct instructions, such as holes and frames. Furthermore, the model performs well in generating multiple identical objects, as shown in the first row of Figure 9, and can effectively generate more complex shapes, such as stars and V-shapes.
D.7. Text to Multiple CAD Figures
During inference, we set the temperature f = 0.3, top_p pp 0.9, and top_k = 50 to enable non-deterministic generation. This configuration allows us to produce varied CAD figures that meet the instructed requirements, with slight differences between them. As result, users can select the design that best aligns with their specific needs. Examples of such outputs are shown Figure 10. These results demonstrate that while adhering to the provided instructions, CADFusion is capable
18

---

Text-to-CAl Generation Through Infusing Visual Feedback in Large Language Models
elorpoed inprosslll caseat seaated lregh the leagts the sectanggia pron, the with equal io the with reclingrar prise, the height lwva then beighs of recuogplar prion. The recussu are syrroetically dintiooee.
urndengale praas
top and bonsn oxthons, two conert
The Thetrredinesiza stape is semictrous the segnans, criller to dape. open ard its tackress vanes diffncer loc locaties and minlly chaisteczed by lack tuo. dnakeese
The 30 di fuL innooor poa ronded ocee, feanting bege cestol eaterpilr cdtst and laarevelly ypu el cinallr leles ner ouch come: syseetiiall Hosy decigned as movertee pass srbesckrr allwwing for srer stadeener sta teter aocess thrughh the cestal oncor.
Its slape of stepped prist, consiating of frre uncaaggli pir prans of equal with beight sacind on up of each scher: The inmets decrsso from lonten to topy wit the lett sties of prims algret
Figure 10. An overview of the generation of CAD instances with slight variations from single prompt. In each sub-figure the top-left image shows the ground truth generation, while the remaining three represent CADFusion's outputs, which exhibit variations in thickness, width, and cutout size. The prompt is displayed at the top of each sub-figure.
fimalid Sample: deceding too much information pepesiiotall
Invalid Sample: characters with CAD shapes
lecangalir Lugerin
X
X
Discrepant Sample: too any Items
Discrepant Sample: complex Integration of multiple shapes
The image feauns vne steeiaal ane calass roraaned nanion sunams Eacks code apai wiy keogh, nakine cooguants and aranad ruch thachar facu slaa with dhe Image sam. The casse adbils sccordde suarn, spreetty os velectonn und there ar vaytis sae osheees fasta vidh dhcts dhaboo chorcing.
The ID dage causie of ballow cylidde with thouad consiced pappeiiaaitty buyy as aasl messnder prirs The prin punds verialyy spwanss hax the buttorid criedtzil aur Theasipe blanndl symasatial coaltaeee tnockukak consccor crizaak leikee.
Figure 11. Invalid and discrepant samples. CADFusion generates invalid samples when the instructions are too complex involve word shape knowledge, and produces discrepant outcomes when there are many distinct items to generate or when complex merges are required to form the final CAD instance.
of generating diversified outputs. The variations primarily affect attributes such as thickness, width, and the size holes and cutouts, while maintaining the overall shape. This flexibility offers users a broader range of choices, thereby reducing the amount of additional work required when integrating such Text-to-CAD systems into industrial applications.
19

---

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
D.8. Failure Cases
We identify two types of failures in our work: sequences that are not renderable and shapes that are rendered but misaligned with the intended design. We refer to the former Invalid Samples and the latter Discrepant Samples. Examples of both types of failures shown in Figure 11.
In our analysis, samples are often invalid when the input instruction is too complex, meaning there are too many elements to be drawn. The case shown in the top-left corner of Figure 11 involves more than 20 loops and 50 curves in the ground truth. Additionally, CADFusion struggles to map CAD shapes to characters such as letters, resulting in failures when attempting to construct shapes that spell words or names.
Discrepancies between the rendered shapes and the intended design occur when the input instruction involves too many distinct items. While CADFusion demonstrates advanced capabilities in understanding numerical values compared to other models, handling more than 8 separate items remains a challenging task. In such cases, CADFusion may either miscalculate the number of items to draw or generate incorrect shapes, as shown in the bottom-left corner of Figure 11. Furthermore, integrating multiple items into complex shapes another frequent challenge for CADFusion.
20


Reviews:
[{"review_id": "Y04aV2Bzif", "reviewer": "Reviewer_3nTM", "summary": "Authors build upon existing CAD data representation and introduce a novel visual feedback into text2cad. A dpo algorithm is used together with LVM-based scoring to improve the text2cad pipeline. Two new datasets are also proposed (text-cad pair dataset and preference dataset).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your feedback. We appreciate the opportunity to clarify your concerns.\n\n## SL(no VF) & SL-VF(pro)\nWe are unsure what SL-VF(pro) refers to. We presume you are mentioning SF-VF(rpo). If so, we would like to point out that this is **not our main method**, but **an ablation** using existing techniques to regularize and stabilize DPO (Line 410). We included it to highlight the effectiveness of our *iterative visual-sequential training* (line 413-426). \n\nIn Table 2, our main method is **SL-VFSL(5) in the last row**, where a considerable improvement on VLM score can be observed.\n\n## Evaluation Criteria\nWe regret to point this out, but there is a **factual misunderstanding**  in the statement *\"Evaluations for whether or not the generation truly follows the text description is not obvious.\"* \n\n1. In our human evaluation, the textual description are shown alongside the generation results, and annotators were asked to evaluate the alignment between the CAD model and text (Line 808). Thus, the human rankings directly reflect instruction following.\n\n2. Second, our VLM score also supports this evaluation by design (Line 317). Reviewer `X4me` calls this \"a reasonable score\", which supports the soundness of our metrics.\n\nWe would appreciate it if you could reconsider this claim after reviewing the referenced lines.\n\n## Technical Contribution\nWe would like to highlight that the motivation behind introducing DPO is itself a contribution. Observing **limitations of sequence learning**, we proposed using visual signals to adjust model preference: an approach has never been previously explored. Moreover, as the rendering process is not differentiable, directly encoding visual signals and backpropagating them is infeasible. Consequently, we propose DPO as a solution.\n\nFurthermore, integrating DPO into our setup is **not a trivial application** of existing techniques. Table 2 (Rows 3, 4) shows that directly applying DPO or standard stabilizing algorithms (RPO) does not yield strong results. We hypothesize that this is due to the **cross-modality** nature of our task. A significant portion of our work focused on overcoming challenges though SL-VFSL training.\n\nAdditionally, we would like to highlight how other reviewers acknowledge our contributions. Reviewer `jpmw` calls our handling of the non-differentiable rendering process \"clever and appropriate,\" and Reviewer `X4me` refers to our approach as \"interesting\" and a \"nice way\" of tackling the problem.\n\n## Simple Shapes in Figures\nFigure 8, 9 includes CAD generations with more complex shapes and textual descriptions specifying CAD structure and topology details. We hope these results satisfy you. \n\nThe reason Figure 5 contains 'simple' shapes is that we had to choose samples from the intersection of valid generations across models. We are happy to move some instances in Figure 7, 8 to the main manuscript at your wish.\n\n## Data Size and Captioning\n1. While we acknowledge that 20k human-annotated samples could be further expanded, we choose this setting for some reasons:\n- First, we annotated data and trained models iteratively. At 20k samples, our method already outperformed baselines (Table 1, main experiment).\n- Second, the cost of annotating 20k samples is affordable for research groups of any scale. By keeping the dataset size reasonable, we aim to ensure the reproducibility of our results and provide meaningful insights to the broader community.\n\n2. VLM itself is reliable. The CADFusion variant trained on only VLM data with sequential signals (Table 4, row 2) outperforms baselines (Table 1, row 1, 2), demonstrating improvement.\n\n## COV/MMD/JSD Scores\nThe Text2CAD paper does not report these metrics. We attempted to compute them using our evaluation code but encountered challenges when applying it to Text2CAD's code and results. However, we have reported other metrics that effectively reflect the our method's efficacy.\n\n## Comparing to DeepCAD/SkexGen/HNC-CAD\nWe did not compare against them as none take text as input or claim text-based capabilities. Adapting their frameworks for textual input requires modifications that constitute a new research project rather than a baseline.\n\nAlthough this requirement is doubtful, we implemented a text-guided SkexGen variant by integrating a text encoder with SkexGen’s decoder for Text-to-CAD generation. Training on the same setups, our results are:\n\n| Method    |  COV  | MMD  |  JSD  |  IR   |\n|-----------|-------|------|-------|-------|\n| SkexGen   | 72.39 | 3.60 | 11.56 | 20.90 |\n| CADFusion | 90.40 | 3.49 | 17.11 | 6.20  |\n\nCADFusion outperforms SkexGen in multiple metrics, reinforcing its effectiveness.\n\n## Collisions\nThe term \"collisions\" refers to part intersections within a CAD model. VLM does not *explicitly* prevent collisions but evaluates their presence. This scoring mechanism benefits the subsequent visual-feedback stage, where we empirically observed an improvement in collision avoidance.", "claims_and_evidence": "From table 2, the evaluation scores between SL (no visual feedback) and SL-VF(pro) is pretty close, I thought with the multi-aspect preference scores the valid ratio would greatly improve. But results do not seem to suggest so.", "methods_and_evaluation_criteria": "Evaluations for  whether or not the generation truly follows the text description is not obvious. LVM is not tuned to assess CAD model quality and the score might not be very reliable. The human ranking score is judging which output is the best out of four different models, but even the best model could still fail to adhere to some text constraints and this score will not reflect this.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Same as above, there is a lack of metric for benchmarking how well the text description controls the CAD generation.", "relation_to_broader_scientific_literature": "Text2CAD is generally an interesting topic in Computer-aided design. This has been less explored than other fields (image, video) due to lack of high-quality data. The paper is moving towards the right direction and demonstrates some early promising results to automate the design process.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "While I like the text2cad task, the technique contribution in this paper is limited. The CAD data representation is from previous work as well as the base dataset (DeepCAD) and the use of DPO. New contributions are limited to combining DPO with visual score feedback and  using VLM to annotate the DeepCAD models. But there are a lot of CAD structure and topology information not available from just a rendered image. So I am not sure if a VLM can capture all the information from image modality. The refined human annotation in figure 7 looks better, but going through the text annotations in figure 5, I find most of them are biased towards coarse shape of the CAD model (cylinder, rectangular, holes)…. This limits the text-cad pairs to very simple shapes (at least judging from the provided figures).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1) 20k data is quite small for training a generative model. I wonder why authors did not annotate the full DeepCAD/SkexGen dataset. If human annotation is not scalable (as written in supplementary) then does it mean the VLM approach proposed in the paper can not reliablely generate high-quality annotations?\n\n2) Why is there no cov/mmd/jsd scores for the other baseline models. Also why authors did not compare to DeepCAD/SkexGen/hnc-cad? Seems like data representation is the only difference here and it is not hard to retrain those models on the provided text annotations (all are open source).\n\n3) I might have missed this but how exactly does VLM avoids collisions (figure 4)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5J91ocJ6BR", "reviewer": "Reviewer_jo2e", "summary": "This paper proposes a text-to-CAD model that leverages LLMs to generate CAD commend sequences as sequential signals. The authors use a pre-trained LLM as the backbone and perform SFT on CAD parametric sequences. They further use DPO to perform RL-based fine-tuning and introduce an LVM-based scoring pipeline to construct preference data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe sincerely appreciate your thoughtful comments and the opportunity to clarify the concerns raised.\n\n## How Well Captions Describe CAD Models and Model Accomodates to Real Users\n\nDuring development, we considered similar concerns and would like to share our findings:\n* VLM-generated captions are effective for moderately detailed descriptions, such as structure, topology, and key component relationships. However, they struggle with precise numerical specifications, such as exact dimensions (e.g., \"13.5 × 4.3 × 2\"). Since designing is an iterative process, moderately detailed descriptions still provide substantial guidance to CAD modeling.\n\n* We provided guidelines to human users and empirically find that they quickly learned to interact with our model by following them. We provide more detail of the guideline and an example at the end of this response. \n\n## Handling Numerical Parameters\nCurrently, users should manually adjust numerical parameters as a post-processing step. In a pratical designing scenario, they normally create rough designs first and refine the numerical details iteratively. We believe our model provides a sufficient response automating the former step, and optimistically anticipate future work that supports the latter, aiming to fully automate the design process.\n\n## More discussion for HNC-CAD\nWe have discussed HNC-CAD in the introduction and related works as an unconditional CAD generation method, and cited it in our paper. Following your suggestion, we will add more discussion about HNC-CAD. Specifically,\n1. HNC-CAD is also an autoregressive model. However, it trains a transformer **VQ-VAE** on **codebook** representations on **sequential** signals, while ours leverages a pretrained **LLM** (decoder only) on **stringified** representations on both **sequential and visual** signals.\n2. The auto-completion task in HNC generates a random completion. It can be used jointly with the text-based CAD generation method. By providing a CAD sequence prefix and text guidence to the framework, we achieve a text-guided completion.\n\n## User's Ability to Modify Generations with Additional Prompts\nWe would put it as a promising future direction called text-based editing. It relies on different model capabilities from text-based generation. For instance, model may need to learn how to localize the place for edition based on additional prompts and make changes to that specific location. This requires not only text understanding but also CAD understanding.\n\n## Generation Diversity Regarding the Same Prompt\nWe did include exploration on this part! Please refer to Figure 10 in Page 18. It corresponds to Appendix C.7.\n\n[1] Khan et al., Text2cad: Generating sequential cad models from beginner-to-expert level text prompts, NeurIPS 2024.\n\n[2] Xu et al., SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks, ICML 2022.\n\n## (Appendix) User Guidelines for Prompting\nA good prompt follows a structured description: (1) *shape overview*, (2) *shape details*, and (3) *shape applications*. Given the varying shape complexity, we encourage but do not enforce describing item (2) and (3). Below is an example caption retrieved from Figure 8, Row 2, Item 3, demonstrating this approach:\n\n```\n[Shape Overview] The 3D shape consists of a large, flat rectangular slab with two evenly spaced, identical cylindrical protrusions extending vertically from its surface. [Shape Details (Optional)] The slab provides a stable base with significant length and width compared to its thin height, while the cylinders are relatively short and have small diameters. [Shape Applications (Optional)]The overall design is symmetrical and balanced, potentially serving as a mounting base or connector.\n```\n\nWe believe that such structured descriptions are clearly formatted and easy for users to interpret. If the reviewer finds it necessary, we are happy to include detailed instructions in the camera-ready version of the paper or the code repository upon release.", "claims_and_evidence": "1. The authors claim that they use DPO as the RL process to optimize the training.\n2. An LVM is used to generate and rank the preference dataset.\n\nThe ablation study supports the claims.", "methods_and_evaluation_criteria": "The authors evaluated the method using F1 score, CD, COV, MMD, JSD, IR, LVM score and average rank. The former ones are common in the CAD generation tasks. LVM score and average rank are also reasonable as the authors use Large vision-language models to generate the preference dataset.", "theoretical_claims": "Equation 1 is a commonly used cross entropy loss, and equation 2 is the loss from Direct Preference Optimization. No other theoretical claims are made in this paper.", "experimental_designs_or_analyses": "The ablation studies support the claims of the modules.\n\nA suggestion is that parital CAD completion could be a good application and experiment like in HNC-CAD", "relation_to_broader_scientific_literature": "This work incorporates powerful pre-trained LLMs into CAD generation. Could be a good start to the community.", "essential_references_not_discussed": "HNC-CAD is able to perform partial CAD completion, which is similar to auto-regressive generation in this work and could be discussed.\n\n[1] Xu, Xiang, et al. \"Hierarchical neural coding for controllable cad model generation.\" arXiv preprint arXiv:2307.00149 (2023).", "other_strengths_and_weaknesses": "Weaknesses\n1. I still have concerns that the VLM-generated captions (even with human revision) could describe the CAD models well. Even if the training results are good, there may be some kind of overfitting to the dataset. It may still be hard for human users to generate such prompts that fit the VLM caption styles and also difficult for human users to correctly describe the CAD models purely in text prompts.\n\n2. The captions are scale-invariant, so how do the users adjust the numerical parameters to scale different parts of the CAD models? Is it integrated to the LLMs are the users have to manually adjust them as a postprocessing step?", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Are the users able to modify the previously generated results with additional prompts?\n\n2. What about the generation diversity regarding to the same text prompt?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wbc2rimTpC", "reviewer": "Reviewer_jpmw", "summary": "This paper introduces CADFusion, a framework for Text-to-CAD generation that leverages LLMs and incorporates visual feedback to improve the quality and accuracy of generated CAD models. The core contribution is a two-stage training procedure that alternates between Sequential Learning (SL) and Visual Feedback (VF) stages. In the SL stage, an LLM (LLAMA-3-8b-Instruct) is fine-tuned on a dataset of paired textual descriptions and CAD parametric sequences (represented as text tokens), using a standard cross-entropy loss. This stage aims to teach the model to generate syntactically correct and logically coherent CAD commands. The VF stage then refines the model by incorporating visual information. Multiple CAD sequences are generated and rendered into visual representations (images). A VLM is employed to automatically score these rendered images based on multi-aspect criteria (shape quality, quantity, and distribution), creating preference data. DPO is then used to train the LLM to favor sequences that produce higher-scoring (visually preferable) CAD models, thus circumventing the non-differentiability of the rendering process. The authors propose an alternating training strategy, switching between SL and VF stages to preserve the benefits of both.\n\nThe paper presents both quantitative and qualitative results comparing CADFusion to two baselines: GPT-4o (with few-shot learning) and Text2CAD (Khan et al., 2024b). The authors utilize a variety of metrics, including F1 scores for sequence accuracy, Chamfer Distance, Coverage, MMD, JSD, and Invalidity Ratio for visual quality, and a VLM-based score to assess visual-textual correspondence. Human evaluations are also conducted. The main finding is that CADFusion outperforms both baselines on most metrics, particularly those related to visual quality. Ablation studies demonstrate the importance of the visual feedback stage and the alternating training strategy. The authors also contribute two datasets: one for sequential learning (text-sequence pairs) and another for visual feedback (preference pairs). The paper acknowledges limitations regarding the generation of very complex shapes.\n\nIn essence, the paper's main contributions include the combination of sequential and visual feedback training for Text-to-CAD, the use of DPO and LVM scoring to integrate visual information, and the demonstration of improved performance compared to existing methods.\n\n## Update after Rebuttal\nI thank the authors for their rebuttal. They have convincingly addressed several key concerns. They acknowledged the need for more precise claims and better highlighting of limitations regarding complex models, promising revisions. Their explanation of the VLM scoring process, including steps taken to ensure stability, was mostly clear. They also clarified the Text2CAD comparison, confirming they have results using identical prompts and will include these in the main paper. The provided clarifications and planned updates sufficiently address my main concerns, leading me to raise my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThank you for your thoughtful comments and for recognizing our contributions, particularly in the design approach and experimental setup. We appreciate the opportunity to clarify the points you raised.\n\n## Claim Could be a Bit More Precise\nWe realize this problem and apologize for that. We will tone down relevant parts such as \"significant performance improvements\" in the updated version. The limitation about complex models (demonstrated in figure 11) will also be brought to the main body.\n\n## Could Different VLMs Give Different Scores?\nFirst, VLM's scores depend on their base capabilities. We will add relevant discussions in the appendix. The followings are some discoveries we have made:\n\n1. LLaVA (non One-Vison) cannot give correct scores. It tends to give very high scores (e.g., 9 or 10) or very low scores (e.g., 2 or 3) randomly.\n2. How LLaVA-One-Vision (LLaVA-OV) scores rendered CAD objects is quite similar to how GPT-4o does.\n\nSecond, the scores are partially sensitive to the formation of the evaluation prompts. Specifically, models randomly over-focus on aspects such as texture and colors of the rendered CAD objects, and score some rendered CAD objects lower than others based on that factors, which leads to inconsistency. Consequently, we instruct the VLMs not to focus on these aspects (details can be found in Line 752 and 800). After this modification, the scoring seems stable with prompts.\n\n## Comparisons with Text2CAD\nWe have the results where Text2CAD uses the same prompts as ours (i.e. the **solid** one). It is the comparison between row 3 and 4 in Table 3. The result we selected for presentation in Table 1 is the setup **favors Text2CAD the most**, which does not affect the fairness of our comparison. We will add the results with the same prompts to the main sections to make the comparison more comprehensive.", "claims_and_evidence": "The paper puts forward several claims regarding the effectiveness of CADFusion, and for the most part, these are backed up by the experiments. For instance, the results in Table 1, along with the visual examples in Figure 5, make a good case that CADFusion generally outperforms the baselines, GPT-4o and Text2CAD, across a range of metrics. The ablation study in Table 2 also adds weight to this. Similarly, the benefits of the alternating training strategy are reasonably demonstrated by comparing different training variations within the ablation study. \n\nHowever, there are a few places where the claims could be a bit more precise. The paper often uses phrases like \"significantly improves performance,\" but without statistical significance tests, it's hard to know for sure if the improvements are truly significant or due to chance. Also, while the paper shows improvements, it's implicitly suggesting that CADFusion works well for all types of CAD models. This isn't entirely supported by the evidence. The examples are mostly simpler shapes, and the authors admit in the limitation section that the method struggles with more complex designs. This limitation should be brought forward more prominently, rather than implying broad applicability.\n\nAnother point concerns the VLM Score. Could different VLMs give different scores? Is the score sensitive to how the evaluation prompt is phrased? Finally, the comparisons with Text2CAD is not that solid, because they are using different prompts, as shown in Table 3.\n\nIn short, the core claims about the advantages of visual feedback and alternating training seem solid. But the claims about the extent of improvement and the general applicability need some toning down and more careful qualification, given the limited evaluation of complex models.", "methods_and_evaluation_criteria": "Overall, the proposed methods and evaluation criteria seem suited to the problem of Text-to-CAD generation. The core idea of combining sequential learning with visual feedback makes sense. CAD models have this dual nature – they need to be syntactically correct (valid sequences of commands) and visually accurate (matching the intended design). The two-stage approach, using an LLM for sequence generation and then refining it with visual feedback, directly addresses this. The use of DPO to handle the non-differentiable rendering process is a clever and appropriate technical solution. It's a good way to get around the problem of backpropagating through the rendering step. Similarly, using a VLM to automatically generate preference data seems a practical approach. Using F1-scores, Chamfer Distance, Coverage, MMD, JSD, and Invalidity Ratio for evaluation covers different aspects of visual quality and validity.\n\nHowever, as mentioned before, an analysis of the VLM score's properties would strengthen the evaluation. Also, current metrics don't directly address aspects like manufacturability or design practicality. Furthermore, a more complete evaluation of the complex CAD models is needed.\n\nIn short, the methods and evaluation criteria make sense for the problem. The main areas for potential improvement are a more thorough analysis of the VLM score, and ideally, the inclusion of some evaluation related to design practicality and, most importantly, complex CAD models.", "theoretical_claims": "This paper is primarily empirical and does not present theoretical claims or proofs.", "experimental_designs_or_analyses": "They're mostly solid. The comparisons with GPT-4o and Text2CAD, using a mix of sequential and visual metrics, seem reasonable. The ablation studies in Table 2 also give us a good sense of how much each part (the visual feedback and the alternating training) actually contributes. The process of constructing datasets is clear.", "relation_to_broader_scientific_literature": "The paper does a reasonable job of positioning itself within the existing literature. The connections to broader work on LLMs, RLHF, and DPO are appropriately established.", "essential_references_not_discussed": "References seem adequate.", "other_strengths_and_weaknesses": "The strengths and weaknesses of the paper have been covered in detail in the previous sections. Please refer to the \"Claims and Evidence,\" \"Methods and Evaluation Criteria,\" and \"Experimental Designs and Analyses\" sections for a discussion of these points.", "other_comments_or_suggestions": "There are a few minor typos and grammatical issues, e.g,  Line 066: \"texutal\" -> \"textual\".\n\nWhile the paper acknowledges limitations, a slightly more expansive discussion of future work would be valuable.", "questions_for_authors": "Please refer to the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IU5xvTvTTt", "reviewer": "Reviewer_X4me", "summary": "The authors use DPO/Rinforcement Learning to fine-tune a LLM to produce CAD instructions from a prompt, so tht the rendered objects are ranked VLM. They fined tune Llamma and use prewarming to make sure it can generate CAD sewuences froma prompt before alternating betwen DPO and direct sequential learning steps. I beleive this is \"On Policy DPO\". The result is a model that scores higher than GPT-4o and one other method, according to  automatic and user-study based metrics on the quality of CAD files generated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThank you for the valuable comments. We are pleased to know that you are happy with our evaluation details including the qualitative results and user study. We hope the following clarifications address your concerns. \n\n## More References\nThanks for pointing out these relevant works. Below is our discussion to them, and we will add them to our updated manuscript.\n\n1. CADLLM [1]: It is a work on *unconditional* CAD generation. We referenced similar works such as DeepCAD [3] and SkexGen [4]. Our work is different to them by focusing on a *conditional* task that deals with text-guided CAD generation.\n\n2. Vitruvion [2]: It focuses on generating 2D sketches from hand drawings. Our work, in contrast, does not focus on *CAD reconstructions* that takes visual inputs, but more abstract text inputs instead. We cited similar works to Vitruvion but on 3D reconstructions [5] [6].\n\n## Baseline Methods\nAs all reviewers acknowledge, the text-to-CAD task is relatively new - we have highlighted this in the introduction. As a result, there are only a few open-source baseline methods that are directly comparable to our approach. We sincerely thank you for your understanding on this matter. \n\nYou are welcomed to have a look at our response to reviewer `3nTM`, in which we reported a comparison between our method with an adaption of Skexgen for text-based generation. We hope this confirms that our framework demonstrates superior capability among all existing works.\n\n## Details of User Study\nOur user evaluation was conducted as a *blind* study. Participants were asked to **rank** the rendered CAD objects (without knowing their origin) based on 1. their **alignment** to the given textual descriptions, and 2. the **quality** of the CAD models. The users had college-level (or higher) education but we did not restrict their majors. Since our evaluation focuses solely on the visual appearance of the generated models, we believe this is appropriate.\n\nWe have included an description of our human evaluation setup in Appendix C.2. However, we realize that this section may not be entirely clear and could be difficult to follow. We will update and clarify this section in the camera-ready version to provide more detailed instructions and improve overall readability.\n\n## DPO: Static vs. On-policy\nYes, our DPO implementation is not entirely \"static\". By splitting DPOs into smaller iterations and making it iterate with the sequential learning, the model being optimized is much closer to the policy model generating the sample pairs. It is a step towards an \"on-policy\" DPO.\n\n[1] Seff et al., Vitruvion: A Generative Model of Parametric CAD Sketches, ICLR 2022.\n\n[2] Wu et al., CAD-LLM: Large Language Model for CAD Generation, NeurIPS 2023 Workshop. \n\n[3] Wu et al., DeepCAD: A Deep Generative Network for Computer-Aided Design Models, ICCV 2021.\n\n[4] Xu et al., SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks, ICML 2022.\n\n[5] Ma et al., Draw Step by Step: Reconstructing CAD Construction Sequences from Point Clouds via Multimodal Diffusion. CVPR 2024.\n\n[6] Khan et al., CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention. CVPR 2024.", "claims_and_evidence": "The paper lists 4 contributions - butthey dont seem tightly linked to the evaluation. \nWorking backwards from their evaluation, they are/shouldbe claiming:\n1. Improved ability to handle simple prompts \n2. Improved ability handle (more) complex shapes -- mainly shown in qualitative eval. \n3. Higher accuracy / quality according to automatic benchmark and human scores\n4. Human feedback in training improves quality\n5. DPO (On policy) is effective for text-2-CAD to incorporate the appearance of the object. \n\nThese things are all evaluated at different levels, each with acceptable evaluation methods.", "methods_and_evaluation_criteria": "The methods and evaluatin criteria make sense. \nThe authors use pre-existing benchmarks for the most part, introducing LVM score as a new metric. \nThis is a reasonable score as far as I can tell, and it is paired with more established ones. \n\nImportantly, they include a user study. \nThe study had 6 people (I dont know who) rank 50 samples comparing 2 versions of CADFusion to the competing approachs (GPT-4o, Text2CAD).  One could complain that this is not enough, or that we do not know their qualifications (do they know anything about CAD?) or id they have potential bias -- was it a blind study where they dont know which is which?  A rubric or instructions to the participants would be nice. However I am pleased that the did a user study as I think it is important for this work.", "theoretical_claims": "This is not a thoery paper -- there are not proofs or theortetical claims. They present a framework and use empirical evaluation with a small user study.\n\nI would like to see discussion of static vs on-policy iterative DPO.", "experimental_designs_or_analyses": "As mentioned - I would like to know more about the user study. \nI ampleased with the amount of qualitative results shown, and the metrics used. \nThere do not seem to be too-many baseliens to compare against in the acadmic literator (3D not 2D CAD, from text), and the wuthors were not able to implement one method, so they effectively compated only to Khan's work. I could not find a more appropriate reference either, although there is \"zoo.dev\" (commercial) and a text_to_cad github repo (no publication).  I think that what they have done is likely sufficient.", "relation_to_broader_scientific_literature": "I think there is a need for more editable structured representaitons from geenrative models and this is a step inthat direction. The visual feedback/DPO appeoach seems novel.", "essential_references_not_discussed": "I think they should mention CAD-LLM and Vitruvian.", "other_strengths_and_weaknesses": "Strength: \n- Interesting approach, nice way to handle non differentiabel rendering\n- Good results qualitiatively and quant. \n\nWeakness:\n- I immediately found some refs that _seem_ relavant and I am concerned that they werent mentioned. \n- Only comparing to one real other method (GPT-4o makes two, but it is not aimed at CAD)\n- User study was small / underexplained\n\nNeutral/Unsure:\n- Relying on LLVM vs human feedback to evaluate", "other_comments_or_suggestions": "L153 bad reference\nL217 col2 - misplaced eqn number\nnot sure footnote 5 is needed.", "questions_for_authors": "How does your work relate to the CAD-LLM or Vitruvian work? \nCan you explain your user-study a bit more? Was it blind? What were the users qualifications? What were their instructions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received one Accept, two Weak Accepts, and one Weak Reject. Three reviewers were positive overall, highlighting the novelty of combining sequential learning with visual feedback using DPO, the introduction of VLM-based scoring, and strong empirical results. The one negative review raised concerns about the dataset and annotation and baseline comparisons. The authors provided a detailed rebuttal, which the area chair found convincing. Reviewers are overall positive and the decision is an accept.
Decision: Accept (poster)