Generating CAD Code with Vision-Language Models for 3D Designs

## Abstract

Abstract Generative AI has transformed the fields of Design and Manufacturing by providing efficient and automated methods for generating and modifying 3D objects. One approach involves using Large Language Models (LLMs) to generate Computer-Aided Design (CAD) scripting code, which can then be executed to render a 3D object; however, the resulting 3D object may not meet the specified requirements. Testing the correctness of CAD generated code is challenging due to the complexity and structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are not feasible in code. In this paper, we introduce CADCodeVerify, a novel approach to iteratively verify and improve 3D objects generated from CAD code. Our approach works by producing ameliorative feedback by prompting a Vision-Language Model (VLM) to generate and answer a set of validation questions to verify the generated object and prompt the VLM to correct deviations. To evaluate CADCodeVerify, we introduce, CADPrompt , the first benchmark for CAD code generation, consisting of 200 200 200 200 natural language prompts paired with expert-annotated scripting code for 3D objects to benchmark progress. Our findings show that CADCodeVerify improves VLM performance by providing visual feedback, enhancing the structure of the 3D objects, and increasing the success rate of the compiled program. When applied to GPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a 5.0% improvement in success rate compared to prior work.

## 1 Introduction

Generative AI, such as Large Language Models (LLMs) and Vision-Language Models (VLMs), offers a unique opportunity to enhance productivity, reduce costs, and increase efficiency within Design and Manufacturing sectors  (Kumar et al., ). These industries are critical contributors to the global economy and responsible for creating products and infrastructure. Recent work has shown that Generative AI can provide solutions towards generating, evaluating, and correcting designs of 3D objects  (Nelson et al., ; Kodnongbua et al., ; Makatura et al., ). However, while these solutions improve efficiency for designers and engineers, they lack efficient feedback mechanisms or refinement loops to automatically correct any inaccuracies in the initially generated 3D object.

Our research explores the potential of using VLMs to generate and refine Computer-Aided Designs (CAD), the leading method in industrial design for creating, modifying, analyzing, and optimizing 3D objects (Sarcar et al., ). Utilizing CAD software such as FreeCAD (Riegel et al., ) and AutoCAD (Yarwood, ) to design products typically requires significant training and domain expertise.
CAD software typically includes underlying scripting languages which allow a user to build parametric 3D objects using code, rather than through the user interface. Leveraging the code generating faculties of Generative AI allows us to circumvent the challenges of utilizing CAD software by directly generating the underlying scripting code representing a 3D object.

We define the process of generating and refining 3D objects using LLMs or VLMs, via CAD scripting code, as CAD code generation. Designs produced by off-the-shelf VLMs or LLMs frequently deviate functionally or structurally from stakeholder specifications, because Generative AI models tend to hallucinate on complex, out-of-distribution tasks. Developing refinement methods to correct inaccuracies in a generated design is a critical next-step in this research. Current state-of-the-art refinement methods are contingent on human-in-the-loop domain expertise  (Makatura et al., ; Nelson et al., ). Obtaining a domain expert is not always feasible and can be time consuming. Automated refinement mechanisms are necessary to improve on the initially imperfect designs and close the feedback loop. Augmenting CAD code generation methods with fully-automated feedback mechanisms will empower non-expert end users to generate their own designs, and reduce the onerous, iterative design process.

In this work, we introduce CADCodeVerify, a fully automated method for refining CAD code generation. CADCodeVerify eliminates the need for human involvement by generating and answering validation questions based on user requirements and providing feedback to refine 3D object code.
This feedback is employed to iteratively refine the design.
To evaluate CADCodeVerify, we introduce a novel benchmark, CADPrompt, designed specifically for assessing the performance of Generative AI (e.g., LLMs and VLMs) in CAD code generation and refinement. CADPrompt contains $200$ 3D objects annotated with natural language prompts and expert-written Python code. This multimodal benchmark enables researchers to evaluate aspects of CAD code generation such as object quality and the syntactical correctness of the generated code. Our experimental results demonstrate that CADCodeVerify not only enhances the quality of generated 3D objects but also improves the performance of LLMs and VLMs by increasing the success rate of compiled programs.

Our work presents three key contributions:

- 1.
We propose CADCodeVerify, a novel CAD code refinement method that enables a VLM to visually inspect generated objects and provide corrective feedback, through a question-generation and answering process, to resolve any deviations from the user’s specifications.
- 2.
With our benchmark, CADPrompt, we provide the first quantitative evaluation of CAD code generation capabilities across widely used VLMs, including GPT-4, Gemini 1.5 Pro, and CodeLlama.
- 3.
We demonstrate that CADCodeVerify sets a new state-of-the-art in refining 3D designs via CAD scripting, achieving a $7.30\%$ reduction in Point Cloud distance and a $5.5\%$ increase in successful object generation when applied to GPT-4, the leading VLM for CAD code generation.

Figure: Figure 1: Our approach enables VLMs to automatically generate and refine 3D objects via a CAD scripting code (i.e., CADQuery) in three steps; (1) Code Generation, where the VLM generates CAD scripting code from a language prompt, (2) Code Execution, where the code generated by the model is rendered as a 3D object through a compiler, and (3) Code Refinement, wherein the language model engages in a self-initiated question-answering process to validate the generated object, with respect to the initial prompt, to generate actionable feedback to refine the code.
Refer to caption: extracted/5905939/fig/Overview.png

## 2 Related Work

We review two integral categories of prior work: LLMs for code generation and recent advances in automated design and manufacturing.

### 2.1 LLMs for Code Generation

LLMs have demonstrated impressive results in a variety of code generation applications. The most popular application is the text-to-code task, wherein users prompt LLMs with a description of the code, which facilitates the LLM to generate the code (Rozière et al., ; Svyatkovskiy et al., ; Brown et al., ; Poesia et al., ).
Another application is the text-to-SQL task, in which a database and question are provided to the LLMs, and then the LLMs generate the SQL query (Rajkumar et al., ).
We build upon these contributions to explore how LLMs can assist non-experts and designers to generate 3D objects via CAD code.

Prior research has demonstrated that LLMs can notably improve their generated code through feedback pertaining to the quality of the generated code  (Chen et al., ). One approach involves LLMs providing their own feedback, wherein the LLMs generate explanatory rationales for the initially generated code. These rationales are then used as feedback, along with the initial code, to refine the output (Chen et al., ). Another approach involves receiving feedback from external tools, such as static code analysis (Alrashedy et al., ), or a Python Interpreter (Madaan et al., ). Contrary to other works, self-correction  (Welleck et al., ), involves training two models: a generator and a corrector. The generator model produces the initial output, which is then refined by the corrector model. In CAD code refinement, one line of work introduced a basic feedback mechanism  (Yuan et al., ), using VLMs to refine 3D object code based on a script and an image. However, this method still struggles with feedback efficiency and refinement detail. In this work, we propose a novel approach to generate visual feedback that enhances the shapes, surfaces, and dimensions of 3D objects through code refinement.

### 2.2 Language for Automated Design and Manufacturing

Learning a shared embedding space between language and 3D objects is essential to integrate language modeling into design and manufacturing.
Recent work has developed foundational multimodal transformers in the image-text space (Radford et al., ; Li et al., ; Liu et al., ), which can be leveraged to perform a variety of tasks.
These approaches have inspired similar techniques to integrating 3D objects into this a shared embedding space (Zeng et al., ; Xue et al., ; Yu et al., ; Guo et al., ), thereby facilitating a range of multimodal tasks such as dialog, classification and evaluation with 3D data (Hong et al., ; Xu et al., ).
While these approaches are capable of learning a shared embedding space between language and 3D objects, they are incapable of generating 3D objects.

Generative modeling of 3D data has historically relied on probabilistic models for 3D space, such as Generative Adversarial Networks (GANs) (Goodfellow et al., ), Variational AutoEncoders (VAEs) (Kingma & Welling, ), and Diffusion Models (Ho et al., ). While GANs and VAEs (Wu et al., ; Achlioptas et al., ; Gadelha et al., ; Yang et al., ; Yan et al., ) have historically been popular approaches, diffusion models have recently emerged as the state-of-the-art for probabilistic 3D modeling (Zeng et al., ; Koo et al., ). Diffusion models facilitate controllable object generation through language-guided shape generation and completion albeit their applicability in producing manufacturable designs remains limited  (Li et al., ; Kodnongbua et al., ). These models typically output point clouds or voxels, neither of which are readily adaptable to manufacturing, due to their non-parametric nature. Thus, methodologies producing parametric outputs suitable for manufacturing are needed.

Recent studies have revealed the capability of VLMs to produce designs via parametric CAD code (Makatura et al., ; Nelson et al., ); however, these works find that GPT-4 requires a significant amount of human feedback to produce code that accurately reflects the initial specifications.
In this paper, we address this limitation by developing CADCodeVerify to alleviate the need for a human expert by allowing a VLM to automatically generate and adapt a 3D object.

## 3 CAD Code Generation

Our goal is to enable VLMs to accurately generate 3D objects from non-expert language specifications. As shown in Figure 1, we take a three-step approach: (1) code generation (§3.1), (2) code execution (§3.2), and (3) code refinement via CADCodeVerify (§3.3).

### 3.1 Code Generation

We prompt the VLM to generate initial CAD scripting code, $y_{0}$, based on a natural language description of the 3D object, $x$, along with task specifications. For the few-shot experiment, we include a set of few-shot demonstrations, $E_{g}$, in the prompt (see Appendix B.4). We employ CADQuery, a Python-based parametric CAD language equipped with a compiler that processes code, to output parametric 3D objects, i.e., render.

Thus, CAD code generation can be broadly formulated as per Equation 1.

$$ $y_{0}\sim P_{LM}(y_{0}|x,E_{g}).$ (1) $$

### 3.2 Code Execution

The generated CAD code, $y_{0}$, is next executed by the CADQuery compiler, $\psi$, to produce the initial CAD design, $d_{0}$, in the Standard Triangle Language (STL) format,(^1^11Standard Triangle Language is also referred to as “Standard Tessellation Language”. The STL (STereoLithography) file format is an openly documented format for describing the surface of an object as a triangular mesh, that is, as a representation of a 3-dimensional surface in triangular facets.) where 0 0 indicates the initial version of both the generated code and CAD design. We denote this process as $d_{0}\sim\psi(y_{0})$. Occasionally, the model produces code which does not compile, typically caused by syntactical issues in the Python code.
We apply insights from prior work on code repair (Chen et al., ), to CAD code generation.
To resolve syntactical issues in the code, we leverage the CADQuery compiler error as feedback, $F_{err}$, for the VLM (see Appendix Figure 8, for the prompt employed in code repair).
This process is repeated until a 3D-object, $d_{k}$, is successfully rendered by $\psi(y_{k})$, or the maximum number of iterations $N$ is reached as per Equation 2.

$$ $y_{k}\sim\{P_{LM}(y_{k}|x,y_{k-1},F_{err})\}_{k=1}^{N}.$ (2) $$

### 3.3 Code Refinement via CADCodeVerify

To correct any discrepencies in the generated design, we construct feedback, $F_{ref}$, to further refine $y_{k}$ as described by Equation 3, where $M$ is the number of refinement iterations,(^2^22In all our experiments, we set the number of refinements to 2, as we did not observe any improvement after the second refinement, which is also consistent with prior work on refinement for code generation (Madaan et al., ; Chen et al., )). We optionally include a set of four reference images of the generated object, taken from different angles (0, 90, 180, and 270 degrees), $I_{ref}=\{I_{l}^{0},I_{l}^{90},I_{l}^{180},I_{l}^{270}\}$.

$$ $y_{M}\sim\{P_{LM}(y_{l+1}|x,y_{l},{F_{ref}},I_{ref})\}_{l=N}^{N+M}.$ (3) $$

The key intuition behind our approach, CADCodeVerify, is to develop a feedback mechanism that leverages a vision-language model to produce visual feedback, which helps improve the generated 3D object without needing human expertise as in prior work (Makatura et al., ) or any external tools (e.g., geometric solvers — see §5.1). CADCodeVerify consists of two sub-steps:

#### (1) Question-Answer Generation:

CADCodeVerify generates a set of binary Yes/No verification questions, $\mathrm{Q}=\{q_{1},q_{2},\dots,q_{n}\}$, based on the language description, $x$, and a set of few-shot example questions, $E_{q}$, i.e., $\mathrm{Q}\sim P_{LM}(\mathrm{Q}|x,E_{q})$.
CADCodeVerify generates between two to five questions per example (see prompt in Figure 7 in the Appendix). To generate answers, CADCodeVerify uses images of the generated 3D object ${I_{ref}}$, the generated questions $Q$, and the language description $x$, where CADCodeVerify answers each of the validation questions as described in Equation 4.

$$ $A\sim P_{LM}(A|x,\mathrm{Q},\mathrm{I_{ref}}).$ (4) $$

CADCodeVerify generates the answers, $\mathrm{A}=(a_{1},a_{2},\dots,a_{n})$, using the Chain-of-Thought format (Wei et al., ), where each answer is accompanied by corresponding reasoning (see prompt example in Figure 9 in the Appendix). Additionally, CADCodeVerify is instructed to respond with "Unclear" when it determines that sufficient information is lacking to accurately answer the question.

#### (2) Feedback Generation:

The question-answer sets are next used to generate ameliorative feedback for refining the 3D object as per Equation 5.

$$ $F_{ref}\sim P_{LM}(F_{ref}|\mathrm{Q},\mathrm{A}).$ (5) $$

The feedback, $F_{ref}$, is then utilized for the code refinement, as described Equation (3). During the feedback generation step, we exclude any questions for which the answer is “Yes” to enable the model to focus on resolving any outstanding issues. If the answer to all questions is “Yes,” then we assume that no further refinement is needed. The full feedback generation prompt can be found in Figure 10 in the Appendix. Two examples of the complete interactions with GPT-4 for one step of CADCodeVerify have been included in the Appendix (see Figures 5 and 6).

Figure: Figure 2: Examples of 3D objects generated by GPT4 after utilizing each feedback method for code refinement. Column 1 shows the ground truth object. The second column shows the language prompt passed into the vision-language model. Columns three, four, and five show the generated object after the initial generation, first-refine, and second-refine phases. In each refine phase, we show the object generated by each feedback method used for code refinement. The X icon denotes situations wherein GPT4 generated code did not compile.
Refer to caption: extracted/5905939/fig/examples.png

**Table 1: Statistics of the CADPrompt dataset, including the vertex and face counts of ground truth 3D objects, the lengths of language descriptions, as well as Python code. We split the dataset into simple and complex categories (§4.3).**
|  | Structural Complexity of 3D Object | Natural Language Descriptions | Python Code |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Category | Vertices | Faces | Words | Sentences | Lines of Code | Tokens |  |  |  |  |  |  |  |  |  |  |  |  |
| min | max | avg. | min | max | avg. | min | max | avg. | min | max | avg. | min | max | avg. | min | max | avg. |  |
| Simple | 6 | 44 | 16.7 | 8 | 84 | 30.0 | 9 | 188 | 51.0 | 1 | 11 | 3.4 | 6 | 46 | 16.9 | 18 | 117 | 46.3 |
| Complex | 52 | 540 | 161.7 | 100 | 1092 | 323.2 | 9 | 152 | 49.5 | 1 | 10 | 3.3 | 6 | 40 | 16.3 | 18 | 104 | 43.6 |
| All | 6 | 540 | 88.8 | 8 | 1092 | 175.7 | 9 | 188 | 50.13 | 1 | 11 | 3.4 | 6 | 46 | 16.6 | 18 | 117 | 45.0 |

## 4 CADPrompt dataset

For evaluating CADCodeVerify on CAD code generation, we introduce a new benchmark CADPrompt, which consists of $200$ 3D objects, represented both in images and in the Standard Triangle Language (STL) formats. We hired annotators to write natural language prompts for creating each of the objects step-by-step (see examples in Figure 2), and we also hired a CAD design expert to write Python code.

### 4.1 Prompt Creation

To construct CADPrompt, we first selected $200$ 3D objects from a dataset of modular CAD objects used in the prior work  (Wu et al., ). We then manually annotated each of the $200$ objects with natural language prompts. If a 3D object was abstract or difficult to describe, two annotators would write separate prompts, followed by a third independent annotator who selected the better option.
This dual annotation process was conducted for $65$ out of the $200$ examples.
Finally, an independent reviewer, who was not involved in the original annotation process, verified and further refined each of the $200$ language prompts to ensure accuracy and grammatical correctness.
We chose to exclude the precise numeric parameters (e.g., height of an object) in the prompt as average users may not have access to such knowledge or lack the expertise necessary to specify accurate parametric designs in the real-world use cases. Table 1 provides the statistics of the CADPrompt dataset.

### 4.2 Code Annotation

To annotate CADPrompt with Python code, we hired a CAD design expert to develop Python scripts that generate the 3D objects described in §4.1. The expert was provided thelanguage description of the 3D object, the 3D object in STL format, and the geometric properties generated by the geometric solver (see Figure 4). We then use Blender (Flavell, ) to verify the 3D objects produced by the Python code with the ground truth, as well as evaluate its geometric properties (see Figure 14). In cases where discrepancies were found, the Python code was returned to the CAD expert for further refinement. Figures 12 and 13 provide examples of Python code from CADPrompt.

### 4.3 Data Stratification

We stratify CADPrompt examples according to compilation difficulty and object complexity to gain insights into model performance (§6).

#### Object complexity:

The complexity of a 3D object is defined by the total number of faces and vertices it contains. The ground truth of the 3D object is represented in mesh file formats, which consist of a collection of vertices, edges, and faces $(x,y,z)$ used to create the 3D object in the STL format. A 3D object with a large number of faces and vertices is considered more complex, while an object with fewer faces and vertices is regarded as simple (see Table 1). We split our dataset by complexity, classifying each 3D object as either (i) Simple or (ii) Complex based on its complexity in relation to the median. Simple objects are defined as those with less complexity than the median while complex objects are greater than the median.

#### Compilation difficulty:

The difficulty of the 3D object is measured by the total amount of compiled code for each object generated during the initial generation step across all experiments. We compare the compiled code resulting from zero-shot and few-shot prompting techniques across all language models utilized in this paper: GPT-4, Genimi, and CodeLlama. We split the data based on the total number of compilable outputs from the experiments (ranging from 0 to 6). Those 3D objects with fewer than four compilable outputs are classified as (i) Easy, whereas 3D objects with $4$ or more compilable outputs are classified as (ii) Hard (see Appendix Figure 11).

## 5 Experimental Setup

### 5.1 Baselines

We compare our approach, CADCodeVerify, with two baselines for feedback generation: (i) 3D-Premise, and (ii) Geometric solver feedback.

#### 3D Premise ( Yuan et al. , 2024 ) :

3D-Premise facilitates code refinement by providing GPT-4 with the image of a generated object, paired with its original description, and prompted GPT-4 to correct any discrepancies between the original and generated objects. To create a baseline against which to we provide the VLM with an image of the generated object and we utilize the same prompts from the original work. Note that this baseline is not applicable for approaches without multimodal capabilities, such as CodeLlama.

#### Geometric solver feedback (this work):

We develop a novel baseline that utilizes FreeCAD, an open-source geometric solver, to provide information regarding the geometric dimensions and structure of the generated 3D object, $d_{g}$. However, our proposed baseline, which calculates geometric properties feedback using the geometric solver, requires access to the ground truth 3D objects, $d_{gt}$ in STL format.
Specifically, the feedback, $F_{\mathcal{GS}}$, corresponds to numerical parameters across thirteen categories, $S$, each describing a unique geometric dimension or component of the object, e.g., width, height, number of faces, number of vertices, volume. Formally, this feedback process is shown in Equation 6 and 7. We compute $F_{\mathcal{GS}}$ for both the generated design, $d_{g}$, and ground truth, $d_{gt}$, to provide feedback, we can describe the geometric solver as follows:

$$ $F_{\mathcal{GS}}(d_{g})=\{(s,\mathcal{GS}(s,d_{g})),\forall s\in S\}$ (6) $$

$$ $F_{r}=F_{\mathcal{GS}}(d_{g})\oplus F_{\mathcal{GS}}(d_{gt})$ (7) $$

The feedback from the geometric solver is a numerical value across thirteen categories of geometric information. An example of this feedback, $F_{\mathcal{GS}}$, is shown in Appendix (see Figure 4).This feedback method serves as an upper limit for CAD code refinement, as it conveys the precise geometric differences between the generated 3D object and the ground truth.

We evaluate the performance of each baseline in both zero-shot and few-shot settings. In the few-shot setting, we provide five demonstration examples, with each example consisting of a 3D object description followed by the corresponding code. We use the OpenAI for GPT-4, Replicate API for CodeLlama, and Google’s GenerativeAI API for Gemini 1.5 Pro respectively.(^3^33Experiments were conducted from 06 JAN to 15 FEB, 2024, and May 15 to August 15, 2024.)

### 5.2 Evaluation Metrics

This paper aims to evaluate how accurately VLMs generate and refine 3D objects through CAD code. To evaluate the quality of the generated 3D object compared to the ground truth, we utilize three evaluation metrics: (i) Point Cloud distance, (ii) Hausdorff distance, and (iii) Intersection over the Ground Truth (IoGT). We apply the Iterative Closest Point (ICP) algorithm to rotate and translate the generated 3D object to choose the best possible alignment with the ground truth object (Besl & McKay, ). Finally, we normalize each point cloud such that it fits within a unit cube, as per prior work (Zheng et al., ). In some cases, the generated CAD code does not compile, which prevents the rendering of a valid 3D object. We report the percentage of successfully compiled code as the “success rate.” The formulas for the evaluation metrics are described as follows:

#### Point Cloud distance:

Point Cloud distance $D(P,Q)$ is shown in Equation 8, where $d_{p,q}=\|p-q\|_{2}$.

$$ $D(P,Q)=\frac{1}{2|P|}\sum_{p\in P}\min_{q\in Q}d_{p,q}+\frac{1}{2|Q|}\sum_{p \in Q}\min_{p\in P}d_{p,q}$ (8) $$

#### Hausdorff distance:

Hausdorff distance, $H(P,Q)$, is given by Equation 9, where $\sup$ and $\inf$ represent the supremum and infimum operators, respectively (Beauchemin et al., ).

$$ $H(P,Q)=\max\{\sup_{p\in P}\inf_{q\in Q}d_{p,q},\sup_{q\in Q}\inf_{p\in P}d_{q, p}\}$ (9) $$

#### Intersection over the Ground Truth (IoGT):

This metric used to evaluate how well the generated 3D object $P$ aligns with the ground truth $Q$. As shown in Equation 10, it calculates the area of intersection between $P$ and $Q$ to the area of $Q$.

$$ $IoGT=\frac{|P\cap Q|}{|Q|}$ (10) $$

Point Cloud distance and Hausdorff distance are based on the geometric properties (e.g., height, width, and volume) of 3D objects, while the IoGT measures the overlap of the bounding boxes between the generated and ground truth 3D objects. When computing these distances, we assign the maximum distance of two points within a unit cube, i.e., $\sqrt{3}$, and an IoGT value of zero to any data point where the code fails to compile. This approach maximally penalizes unsuccessful 3D object generations, ensuring consistent and fair evaluation criteria across all the experiments.

**Table 2: We present our results using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and success rates. The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.**
| Model | Feedback Mechanism | IoGT $\uparrow$ | Point Cloud dist. $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| GPT-4: Zero-shot | Generated | 0.935 (0.043) | 0.153 (0.146) | 0.484 (0.405) | 92.0% |
| 3D-Premise | 0.939 (0.034) | 0.150 (0.143) | 0.440 (0.372) | 91.5% |  |
| CADCodeVerify (Ours) | 0.941 (0.034) | 0.132 (0.137) | 0.455 (0.354) | 94.0% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.943 (0.037) | 0.102 (0.159) | 0.378 (0.434) | 91.5% |
| GPT-4: Few-shot | Generated | 0.939 (0.030) | 0.155 (0.140) | 0.494 (0.368) | 96.0% |
| 3D-Premise | 0.942 (0.033) | 0.137 (0.155) | 0.446 (0.396) | 91.0% |  |
| CADCodeVerify (Ours) | 0.944 (0.028) | 0.127 (0.135) | 0.419 (0.356) | 96.5% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.944 (0.031) | 0.103 (0.152) | 0.399 (0.433) | 95.5% |
| Gemini: Zero-shot | Generated | 0.905 (0.088) | 0.159 (0.180) | 0.531 (0.451) | 85.0% |
| 3D-Premise | 0.911 (0.079) | 0.150 (0.180) | 0.496 (0.431) | 83.5% |  |
| CADCodeVerify (Ours) | 0.914 (0.082) | 0.138 (0.165) | 0.497 (0.384) | 84.5% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.917 (0.070) | 0.113 (0.188) | 0.416 (0.458) | 83.5% |
| Gemini: Few-shot | Generated | 0.933 (0.061) | 0.171 (0.174) | 0.521 (0.426) | 85.0% |
| 3D-Premise | 0.939 (0.070) | 0.169 (0.184) | 0.521 (0.516) | 81.5% |  |
| CADCodeVerify (Ours) | 0.939 (0.052) | 0.147 (0.160) | 0.492 (0.358) | 85.0% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.944 (0.060) | 0.104 (0.146) | 0.386 (0.470) | 84.5% |
| CodeLlama: Zero-shot | Generated | 0.920 (0.943) | 0.237 (1.591) | 0.731 (1.270) | 64.5% |
| CADCodeVerify (Ours) | 0.930 (0.955) | 0.211 (1.599) | 0.641 (1.309) | 70.0% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.888 (0.941) | 0.280 (1.595) | 0.823 (1.258) | 55.5% |
| CodeLlama: Few-shot | Generated | 0.927 (0.949) | 0.224 (1.597) | 0.657 (1.294) | 67.0% |
| CADCodeVerify (Ours) | 0.935 (0.957) | 0.185 (1.620) | 0.582 (1.366) | 73.5% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.906 (0.949) | 0.239 (1.606) | 0.727 (1.301) | 60.5% |

## 6 Experiment Results

We utilize CADPrompt to quantitatively evaluate the capabilities of CADCodeVerify approach across various VLMs. The following are our key findings:

GPT-4 exhibits the highest capacity to generate compile-able code on CADPrompt.
We report our primary results in Table 2, comparing the distance and success-rate for three LLMs and three refinement methods.
With regards to “success-rate,” GPT-4 exhibits superior performance $96.5\%$ on the CADPrompt benchmark compared to Gemini $85\%$ and CodeLlama $73.5\%$.
While prior work has provided valuable qualitative insights into GPT-4’s competency to producing compile-able code (Makatura et al., ), our analysis, via CADPrompt, provides a concrete assessment of each LLMs CAD code generation capacity.

CADCodeVerify shows superior performance on more challenging data.
Figure 3 presents the success rates for each feedback mechanism at generation and refinement stages, using the difficulty and complexity splits of CADPrompt. For "Easy" data, the performance across all baselines is relatively comparable. However, results on the "Hard" data underscore the effectiveness of CADCodeVerify. It is the only refinement approach that improves the success rate of object generation, achieving approximately a 9% increase at the Refine-1 stage. In contrast, 3D-Premise struggles to provide effective feedback for "Hard" data, resulting in a 20% drop in success rate at Refine-1, bringing it down to 62%. Real-world data in design and manufacturing is likely to align more closely with the "Complex" and "Hard" data splits. Our results demonstrate the value of CADCodeVerify in providing feedback that can be applied to real-world product design.

Figure: Figure 3: Success rate comparison across different feedback mechanisms, categorized by difficulty and complexity. Each figure corresponds to a specific subset of the CADPrompt dataset, covering both generated and refined objects. The plots for the “Hard” and “Complex” splits indicate that the feedback approach from 3D-Premise significantly reduces GPT-4’s success rate in object generation.
Refer to caption: x1.png

CADCodeVerify generates model-agnostic feedback to improve 3D object generation.
Table 2 also reports the distance measure from the ground truth at the generate and refine stages.
Since CodeLlama does not possess multimodal capabilities, we utilize GPT-4 to perform CADCodeVerify to generate refinement feedback.
Our results show that CADCodeVerify improves the quality of the generated object across all three LLMs/VLMs as measured by IoGT, Point Cloud distance and Hausdorff distance.
CADCodeVerify also improves both GPT-4 and CodeLlama’s “success-rate” in producing compile-able CAD code.
This result emphasizes the model-agnostic nature of the multimodal “assess and reason” analysis induced by our CADCodeVerify refinement process, to universally produce actionable feedback for correcting object generation errors.

**Table 3: The main results and ablation analysis for the GPT-4 few-shot setting on a subset of 100 3D objects. Results are reported using the median and interquartile range (IQR).**
| Ablation Study | IoGT $\uparrow$ | PLC distances $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- |
| Generated | 0.909 (0.062) | 0.156 (0.150) | 0.491 (0.348) | 96.5% |
| Refine without images | 0.914 (0.058) | 0.153 (0.123) | 0.451 (0.321) | 96.0% |
| Zero-shot QA generation | 0.919 (0.049) | 0.141 (0.112) | 0.471 (0.280) | 98.0% |
| CADCodeVerify (Ours) | 0.919 (0.045) | 0.126 (0.122) | 0.444 (0.308) | 97.5% |

CADCodeVerify outperforms 3D-Premise across VLMs.
In Table 2, we quantitatively compare our CADCodeVerify approach to a refinement approach proposed in prior work, 3D-Premise.
Since 3D-Premise involves directly uploading an image of the generated object to the VLM, this approach cannot be applied to the unimodal CodeLlama model.
With respect to GPT-4 and Gemini, CADCodeVerify is measured to more accurately refine objects, compared to 3D-Premise for objects in the CADPrompt dataset ($>7\%$ reduction in point cloud distance for GPT-4 and Gemini) .
This result shows that CADCodeVerify is the new state-of-the-art for CAD code refinement.
Next, we compare our proposed refinement approach to an upper benchmark which utilizes a geometric solver to provide parametric feedback regarding the specific, geometric differences with regards to the target object, as per § 5.
Our results show that through CADCodeVerify, we can achieve comparable performance without needing to access the target object, which is unlikely to be available in real-world use-cases.

Ablation Analysis. We perform an ablation analysis to test the effectiveness of each independent component of CADCodeVerify (See Table 2). We conduct this ablation with GPT-4, with a few-shot prompt, on 100 randomly selected examples from CADPrompt. First, we ablate the inclusion of few-shot example questions in the question-generation component (see Equation 3). The results shows that zero-shot QA generation increases the distance between the ground truth and refined 3D objects from $0.126$ to $0.141$ in Point Cloud distance and from $0.444$ to $0.471$ in Hausdorff distance.
Our second ablation measures the impact of the reference images, $I_{ref}$, during the code refinement step of Equation 3. Removing the reference images worsens the Point Cloud distance and Hausdorff distances from $0.126$, $0.444$ to $0.153$, $0.451$ respectively, highlighting the importance of the reference images.

## 7 Limitations

It is essential to recognize that Point Cloud distance and Hausdorff distance are noisy metrics, measuring only the spatial similarity between two 3D objects. While they provides an broad estimate of similarity, there is a need for a more granular metric that can distinguish the structural differences between objects. For instance, a desk and a desk with small gaps between the legs and the surface might have a small distance measure, but these gaps represent critical logical or structural issues that should be captured in the evaluation. In future work, we aim to explore evaluation methods for CAD code generation that incorporate logical design principles to assess object similarity more accurately.

The quality of the generated objects and the likelihood of successfully compiling code are highly dependent on the user’s initial prompt. Due to the flexibility of natural language, the interpretation of 3D objects can vary, resulting in multiple valid but functionally different descriptions for the same object. For example, one could describe a desk functionally as "Draw a desk with four legs," or more geometrically as "Draw an object with a flat rectangular top supported by four long rectangular prisms at each corner." In our approach, we incorporated review procedures to improve data quality, though it was not feasible to exhaustively explore all possible annotation methods. Future research could further investigate this prompt sensitivity and explore strategies for identifying the most effective prompts for specific VLMs.

## 8 Conclusion

In this work, we formally define a nascent task, CAD code generation, wherein VLMs are utilized to generate code for 3D parametric models.
We compiled a novel CADPrompt dataset, consisting of $200$ 3D objects paired with corresponding language descriptions and Python code. Next, we propose a novel approach for code refinement within CAD code generation, called CADCodeVerify, wherein we enable a VLM to validate and correct its generate object to fix any errors in the generation.
We compare CADCodeVerify to two relevant approaches for code refinement for CAD code generation to highlight the strengths and weaknesses of each code refinement method. Our approach marks a significant improvement over previous CAD code generation methods, which primarily relied on manual human feedback through language interactions with VLMs.

## 9 Acknowledgment

This work was supported by the National Science Foundation under grant number CMMI-2229260.

## Appendix A Ethics Discussion

As our approach utilizes VLMs, we need to be cognizant of the potential pitfalls of utilizing these approaches.
VLMs are capable of hallucinations to produce context that is unrelated or unhelpful to the desired context.
These hallucinations may also include harmful propagation of the inherent stereotypes within the datasets utilized to train these models. While these concerns are important to note for any VLM-based approach, the potential downstream impact on users via our approach is minimal. Since our approach is employed to produce 3D objects, there is limited harm which may be incurred by any potential hallucinations via our approach. However, a malicious user may choose to leverage our method to generate a 3D object of a weapon or harmful item such as guns, knives, etc. While there are many steps between the process of generating a design and procuring the item, we encourage readers to exercise caution in identifying and reporting any misuse of this approach.

## Appendix B Additional Methodology Details

### B.1 Why CADQuery?

CADQuery is one of several open-source CAD scripting languages, alongside tools like FreeCAD and OpenSCAD. Previous work has primarily used OpenSCAD for CAD code generation. However, we chose to use CADQuery as our parametric CAD programming language instead of OpenSCAD, which has been the dominant language for generating 3D objects. Our decision to switch to CADQuery was based on two key reasons: (1) CADQuery is built in Python, making it more suitable for LLM-generated code, given the vast amount of Python code available online. (2) CADQuery’s "design-intent" approach allows it to generate more concise code for complex objects compared to OpenSCAD.

### B.2 Geometric Solver Verbalization

As an intermediate step, we utilize an LLM to verbalize this feedback, offering detailed insights into how the generated design $d_{g}$ differs from the ground truth design $d_{gt}$. Verbalization has been demonstrated to enhance the model’s understanding and response accuracy (Madaan et al., ).

### B.3 Experimental Parameters

We performed the experiments using GPT-4 ("gpt-v4") via the OpenAI API and Gemini ("gemini-1.5-flash-latest") through the Google API, with the temperature set to 0 for code generation and refinement. In cases where the generated code had bugs or failed to compile, we resubmitted both the code and the compiler error message to the model, adjusting the temperature to 1. For CodeLlama B70, we utilized the Replicate API(^4^44https://replicate.com/), setting the temperature to 0.8 for code generation, refinement, and bug fixing. Other hyperparameters, such as top_k = 10, top_p = 0.9, and repeat_penalty = 1.1, were kept at their default values. The total cost for running the experiments was approximately $1000: $450 for GPT-4, $5 for Gemini, and $150 for CodeLlama.

To compute distance measures, we converted the generated STL files into point clouds rendered with 1000 points. We used the Open3D and Pandas libraries to calculate the Point Cloud Distance.

### B.4 Few-Shot Prompt

We design a few-shot prompt to enable VLMs to adeptly perform CAD code generation.
The prompt, $p\sim y_{1}\oplus y_{2}\ldots\oplus y_{k},$ where the $y$ is CAD code and is comprised of a set of $k$ examples. We can formulate the few shot learning as $P\sim\{(y_{i})\}_{i=1}^{k}$. Each example, $(y_{i})$ is sourced from a memory of code samples, $D$, collected from the CADQuery documentation with a total of 40 examples. We include a text-description of the object or additional comments describing the code, in each code snippet, if it is available in the documentation.

### B.5 Geometric solver feedback

Figure: Figure 4: This figure presents an example of the feedback generated by the geometric solver, which calculates the geometric properties between the generated and ground truth 3D objects, requiring access to the ground truth.
Refer to caption: extracted/5905939/fig/geometric_solver_feedback.png

## Appendix C All the results

**Table 4: We present our results for GPT-4 using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and success rates. Table 2 reports the Best Refine results. The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.**
| Model | Feedback Mechanism | Iterations | IoGT $\uparrow$ | Point Cloud dist. $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-4: Zero-shot | Generated | – | 0.935 (0.043) | 0.153 (0.146) | 0.484 (0.405) | 92.0% |
| 3D-Premise | Refine_1 | 0.934 (0.035) | 0.164 (0.154) | 0.478 (0.383) | 91.5% |  |
| Refine_2 | 0.935 (0.037) | 0.170 (0.171) | 0.493 (0.436) | 89.5% |  |  |
| Best Refine | 0.939 (0.034) | 0.150 (0.143) | 0.440 (0.372) | 91.5% |  |  |
| CADCodeVerify (Ours) | Refine_1 | 0.936 (0.032) | 0.146 (0.134) | 0.491 (0.362) | 94.0% |  |
| Refine_2 | 0.936 (0.039) | 0.159 (0.151) | 0.497 (0.389) | 93.5% |  |  |
| Best Refine | 0.941 (0.034) | 0.132 (0.137) | 0.455 (0.354) | 94.0% |  |  |
| Geometric solver* | Refine_1 | 0.939 (0.039) | 0.119 (0.170) | 0.427 (0.470) | 91.5% |  |
| Refine_2 | 0.940 (0.047) | 0.125 (0.172) | 0.439 (0.457) | 91.0% |  |  |
|  | Best Refine | 0.943 (0.037) | 0.102 (0.159) | 0.378 (0.434) | 91.5% |  |
| GPT-4: Few-shot | Generated | – | 0.939 (0.030) | 0.155 (0.140) | 0.494 (0.368) | 96.0% |
| 3D-Premise | Refine_1 | 0.937 (0.032) | 0.156 (0.176) | 0.486 (0.424) | 91.0% |  |
| Refine_2 | 0.939 (0.039) | 0.154 (0.192) | 0.467 (0.414) | 88.5% |  |  |
| Best Refine | 0.942 (0.033) | 0.137 (0.155) | 0.446 (0.396) | 91.0% |  |  |
| CADCodeVerify (Ours) | Refine_1 | 0.941 (0.030) | 0.147 (0.148) | 0.470 (0.378) | 96.5% |  |
| Refine_2 | 0.941 (0.028) | 0.137 (0.139) | 0.460 (0.365) | 95.5% |  |  |
| Best Refine | 0.944 (0.028) | 0.127 (0.135) | 0.419 (0.356) | 96.5% |  |  |
| Geometric solver* | Refine_1 | 0.938 (0.037) | 0.120 (0.162) | 0.429 (0.436) | 95.5% |  |
| Refine_2 | 0.941 (0.043) | 0.110 (0.162) | 0.432 (0.448) | 94.5% |  |  |
|  | Best Refine | 0.944 (0.031) | 0.103 (0.152) | 0.399 (0.433) | 95.5% |  |

**Table 5: We present our results for Gemini using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and success rates. Table 2 reports the Best Refine results. The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.**
| Model | Feedback Mechanism | Iterations | IoGT $\uparrow$ | Point Cloud dist. $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Gemini: Zero-shot | Generated | – | 0.905 (0.088) | 0.159 (0.180) | 0.531 (0.451) | 85.0% |
| 3D-Premise | Refine_1 | 0.903 (0.083) | 0.167 (0.192) | 0.541 (0.431) | 83.5% |  |
| Refine_2 | 0.906 (0.082) | 0.163 (0.190) | 0.548 (0.440) | 83.5% |  |  |
| Best Refine | 0.911 (0.079) | 0.150 (0.180) | 0.496 (0.431) | 83.5% |  |  |
| CADCodeVerify (Ours) | Refine_1 | 0.909 (0.091) | 0.162 (0.188) | 0.527 (0.479) | 84.5% |  |
| Refine_2 | 0.906 (0.092) | 0.152 (0.170) | 0.529 (0.392) | 84.5% |  |  |
| Best Refine | 0.914 (0.082) | 0.138 (0.165) | 0.497 (0.384) | 84.5% |  |  |
| Geometric solver* | Refine_1 | 0.907 (0.083) | 0.146 (0.220) | 0.479 (0.523) | 83.5% |  |
| Refine_2 | 0.910 (0.085) | 0.139 (0.203) | 0.468 (0.499) | 82.5% |  |  |
|  | Best Refine | 0.917 (0.070) | 0.113 (0.188) | 0.416 (0.458) | 83.5% |  |
| Gemini: Few-shot | Generated | – | 0.933 (0.061) | 0.171 (0.174) | 0.521 (0.426) | 85.0% |
| 3D-Premise | Refine_1 | 0.934 (0.067) | 0.180 (0.193) | 0.555 (0.472) | 81.5% |  |
| Refine_2 | 0.935 (0.063) | 0.182 (0.237) | 0.576 (0.480) | 81.0% |  |  |
| Best Refine | 0.939 (0.070) | 0.169 (0.184) | 0.521 (0.516) | 81.5% |  |  |
| CADCodeVerify (Ours) | Refine_1 | 0.935 (0.060) | 0.166 (0.173) | 0.541 (0.408) | 85.0% |  |
| Refine_2 | 0.932 (0.062) | 0.178 (0.187) | 0.543 (0.413) | 83.0% |  |  |
| Best Refine | 0.939 (0.052) | 0.147 (0.160) | 0.492 (0.358) | 85.0% |  |  |
| Geometric solver* | Refine_1 | 0.938 (0.067) | 0.124 (0.193) | 0.433 (0.452) | 84.5% |  |
| Refine_2 | 0.937 (0.064) | 0.124 (0.208) | 0.432 (0.537) | 82.5% |  |  |
|  | Best Refine | 0.944 (0.060) | 0.104 (0.146) | 0.386 (0.470) | 84.5% |  |

**Table 6: We present our results for CodeLLama using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and success rates. Table 2 reports the Best Refine results. The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.**
| Model | Feedback Mechanism | Iterations | IoGT $\uparrow$ | Point Cloud dist. $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Zero-shot CodeLlama | Generated | – | 0.92 (0.943) | 0.237 (1.591) | 0.731 (1.270) | 64.5% |
| CADCodeVerify (Ours) | Refine_1 | 0.928 (0.949) | 0.223 (1.590) | 0.685 (1.281) | 70.0% |  |
| Refine_2 | 0.000 (0.939) | 1.730 (1.553) | 1.730 (1.168) | 47.0% |  |  |
| Best Refine | 0.930 (0.955) | 0.211 (1.599) | 0.641 (1.309) | 70.0% |  |  |
| Geometric solver | Refine_1 | 0.888 (0.939) | 0.286 (1.590) | 0.794 (1.250) | 55.5% |  |
| Refine_2 | 0.000 (0.915) | 1.730 (1.477) | 1.730 (0.946) | 30.5% |  |  |
|  | Best Refine | 0.888 (0.941) | 0.280 (1.595) | 0.823 (1.258) | 55.5% |  |
| Few-shot CodeLlama | Generated | – | 0.927 (0.949) | 0.224 (1.597) | 0.657 (1.294) | 67.0% |
| CADCodeVerify (Ours) | Refine_1 | 0.928 (0.950) | 0.212 (1.608) | 0.630 (1.324) | 73.5% |  |
| Refine_2 | 0.924 (0.946) | 0.260 (1.591) | 0.714 (1.290) | 65.0% |  |  |
| Best Refine | 0.935 (0.957) | 0.185 (1.620) | 0.582 (1.366) | 73.5% |  |  |
| Geometric solver | Refine_1 | 0.903 (0.948) | 0.250 (1.604) | 0.765 (1.298) | 60.5% |  |
| Refine_2 | 0.000 (0.889) | 1.730 (1.474) | 1.730 (1.018) | 31.0% |  |  |
|  | Best Refine | 0.906 (0.949) | 0.239 (1.606) | 0.727 (1.301) | 60.5% |  |

**Table 7: We present our results for the GPT-4 few-shot setting, stratified by object complexity (simple and complex), as discussed in §4.3. The results are reported using the median and interquartile range (IQR). The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.**
| Complexity | Feedback Mechanism | IoGT $\uparrow$ | PLC distances $\downarrow$ | Hausdorff dist. $\downarrow$ | Success Rate $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Simple | Generated | 0.944 (0.024) | 0.145 (0.102) | 0.470 (0.309) | 99.0% |
| 3D-Premise | 0.948 (0.024) | 0.120 (0.136) | 0.396 (0.292) | 96.5% |  |
| CADCodeVerify (Ours) | 0.950 (0.025) | 0.114 (0.124) | 0.390 (0.288) | 100.0% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.947 (0.026) | 0.097 (0.116) | 0.400 (0.414) | 98.5% |
| Complex | Generated | 0.929 (0.048) | 0.183 (0.202) | 0.522 (0.481) | 98.0% |
| 3D-Premise | 0.933 (0.057) | 0.169 (0.221) | 0.470 (0.445) | 95.0% |  |
| CADCodeVerify (Ours) | 0.941 (0.036) | 0.141 (0.190) | 0.465 (0.414) | 97.5% |  |
| \cdashline2-6[2pt/2pt] | Geometric solver* | 0.934 (0.048) | 0.115 (0.174) | 0.408 (0.488) | 98.0% |

Figure: Figure 5: This figure provides an example (1) of one step of CADCodeVerify, via GPT-4V, for a prompt from CADPrompt
Refer to caption: extracted/5905939/fig/example1-appendix.png

Figure: Figure 6: This figure provides an example (2) of one step of CADCodeVerify, via GPT-4V, for a prompt from CADPrompt
Refer to caption: extracted/5905939/fig/example2-appendix.png

Figure: Figure 7: In the CADCodeVerify approach, we utilize this prompt with two examples to generate the verification questions.

Figure: Figure 8: If the generated CAD code fails to compile due to a compiler error, we pass both the error and the generated CAD code to LLMs for correction.

Figure: Figure 9: In the CADCodeVerify approach, we use this prompt along with images to generate the answer and reasoning for each question.

Figure: Figure 10: In the CADCodeVerify approach, we use this prompt to generate feedback from the generated question-answer set.

Figure: Figure 11: We conducted six experiments across three LLMs—GPT-4, Gemini, and CodeLlama—and in both zero-shot and few-shot settings for CADPrompt, which contains $200$ examples. Some of these experiments did not generate compiled code that produced valid 3D objects for some of the examples. We calculated the total compiled code for each example across all experiments, where Max = 6 means the example was generated in all experiments, and Min = 0 indicates that the example was not generated by any experiment. We then sorted them based on difficulty and split the dataset into "Easy" and "Hard" categories as discussed in §4.3
Refer to caption: x2.png

Figure: (a) The natural language descriptions of the 3D object.
Refer to caption: extracted/5905939/fig/00998714_image.png

Figure: (a) The natural language descriptions of the 3D object.
Refer to caption: extracted/5905939/fig/00998698_image.png

Figure: (a) Lamp
Refer to caption: extracted/5905939/fig/Comparison_1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - Q1: I appreciate the framework’s contribution, but could you provide a bit more on how this specifically differs from or builds upon prior work (3D-Premise)? Is it an improvement or an entire new approach?

- Q2: Can you tell us more about the quality and diversity of the CADPrompt dataset? I’m curious about how it’s tailored to the goals of your approach. A bit more on why it’s a good benchmark—perhaps a breakdown of object types, complexity levels, or any specific challenges it presents?

- Q3: The experimental results look interesting, but it would be good to understand more in details where the approach performs well and where it might have limitations. Do you have any insight to share about this?
soundness: 3
strengths: - Novelty and Originality: The integration of question/answering based VLM to refine the object quality. 

- Evaluation: The increase of success rate of compilable outputs at the end of the refinement shows that the method can improve the generation of CAD code from language prompt.

- Soundness: The technical approach is sound and could be applied to a lot of CAD or CAV tasks. The approach is a specification refinement method using generative AI. From an initial specification, an initial object plus query are generated. Then, feedback from query satisfaction are used to improve the object. A dataset is provided and metrics are determined to rank the quality of the generated object. This has applications beyond CAD specifically.
summary: The paper entitled “GENERATING CAD CODE WITH VISION-LANGUAGE MODELS FOR 3D DESIGNS” proposes an approach to generate and verify CAD 3D objects from natural language. The proposed approach introduces a verification step called CADCodeVerify. CADCodeVerify use Vison Language Machine (VLM) to verify CAD objects against generated visual questions. The hypothesis is that the question serves as visual properties the object should satisfy to meet the structural specification. The key contribution of this approach is the use of a refinement step. From the language prompt that describes the object in natural language, a first CAD description code (CADQuery) is generated. The visual questions are also generated from this language prompt input. The object (CADQuery) is verified against visual properties questions using a VLM that generates feedback for each questions. This feedback is used to refined the CADQuery code.

To summarise, the contributions of this work is two fold: 1) an automated CAD generation approach that does not necessitate humain interaction; 2) a dataset called CADPrompt that comprises 200 CAD objects. Expermental results shows when implemented using GPT-4 LLM, we observe improvements in object generation accuracy and an increased success rate compared to other LLMs.
weaknesses: - Dataset Scope: CADPrompt could have been better introduce, showing the most complex objects in terms of complexity and difficulty metrics. 

- Complexity Metric for Objects: This paper measures object complexity by counting vertices and faces. However, using metrics like bounding boxes or decomposed bounding volumes could better reflect structural complexity. Vertices mainly define shape details, not true complexity—a simple shape like a cube can have many polygons, while complex shapes like an aircraft wing might need fewer. In physics engines, object complexity is often defined by the volume structure the object requires. Lower polygons can shape more complex structures.

- Difficulty Metric for Objects: The current method of measuring complexity through word and line count for the natural language descriptions and Python code may not fully capture their complexity. A semantic-based complexity metric, where each instruction has a complexity value, would better reflect how nuanced or contextually challenging the description and code are, which could impact model performance.

- Clarity and Readability: Certain sections are overly technical and hard to understand without referencing the appendix. Simplifying these sections would improve accessibility for readers. Terminological inconsistencies between LLM used for language to code and code with feedback to code and VLM for image and questions to answers also introduce potential confusion, which could be resolved for clarity. Certain sections are overly technical and hard to understand without referencing the appendix. Simplifying these sections could enhance accessibility for readers. For example, consider moving equations 1-7, which aren't essential for understanding the main contributions, to the appendix. Conversely, evaluation metric equations (8-10), such as those for point cloud distances, Hausdorff distance, and IoGT, should remain in the main text. Additionally, bringing valuable visuals currently in the appendix (Figures 5, 11, 12, and 14) into the main text would ease the understanding and improve work illustration. The paper would also benefit from a stronger focus on scientific insights and hypothesis testing over too much detailled descriptions. For instance, providing context around why a fixed number of five verification questions is used, without adapting to object complexity, would clarify the rationale. Similarly, explaining why only two refinement iterations are applied, regardless of complexity, would better illustrate the scientific intuitions driving these choices. Furthermore, details on how AI context like 7, 9, and 10, have been engineered such as crafted based on expert knowledge or through authors tuning would be valuable. Shifting focus to the "why" rather than the "what" would help readers grasp the paper challenges.

- Positioning: the paper’s positioning within the state of the art could be made clearer, with more explicit distinctions from existing work. The paper integrates the related work 3D-premise in the experiments, but didn't explicitly state the approach difference. The feedback is based on question answering rather than the initial description of the object.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Would providing images of the generated object from multiple viewpoints improve 3D-Premise’s performance? What about using a chain of thought prompting (for e.g. explicitly ask the model to reason whether each criterion in the initial task are satisfied)? This will be similar to CADCodeVerify, but doing 1 VLM call instead of doing 3 VLM calls in the verify step  to reduce the inference time. 

2. In Figure 3, does 3D-Premise also repair code based on compile error messages?

3. How do you handle ambiguity in inputs in ground truth solutions? For example, if the NL input does not specify a particular size of hole, there could be many ground truth solutions which might impact the cloud-distance based metrics. 

4. Since Point Cloud distance and Hausdorff distance are noisy metrics, how does the generated CAD program compare on the geometric metrics obtained using the geometric solver?

Minor: 
- Consider renaming success rates to compile rates or something similar to convey that is only compilation success rate and not overall task success rate. 

- For figures 5,6 (running examples), it will be useful to also see the repair steps.
soundness: 2
strengths: -	The paper comes with a new benchmark suite (CADPrompt) which is a well-curated, crowd-sourced benchmark with annotations and quality checks, which is a valuable resource for assessing CAD code generation and refinement methods.
-	CADCodeVerify uses an interesting novel idea to eliminate the need for human involvement by generating validation questions and answering them using VLMs.
-	The geometric solver-based baseline is very interesting and gives an upper estimate of the self-refinement process; it would be interesting to explore if this solver could also be used as a metric for evaluation.
summary: The paper introduces CADCodeVerify, a approach to iteratively verify and improve 3D objects generated from CAD code using Vision-Language Models (VLMs).  The method involves generating CAD scripting code from natural language prompts, executing the code to render a 3D object, and refining the code based on visual feedback through a question generation and answering process, to correct deviations from the initial specifications.  The approach is evaluated using CADPrompt, a benchmark dataset of 200 3D objects with corresponding natural language prompts and expert-annotated code.  The approach is evaluated on GPT-4, Gemini 1.5 Pro and CodeLLama models.
weaknesses: -	The fundamental differences between CADCodeVerify and 3D-Premise are unclear. For example, it’s not specified whether 3D-Premise uses execution-error-based repair. Also, both approaches seem to use totally different prompts, so it is not clear if it is just a matter of better prompting or something fundamental (such as the question-answer based method)
-	The paper would be stronger if the approach was also evaluated on the 3D premise dataset
-	Some other details are missing (see below questions) regarding ambiguity in NL input and chain-of-thought based extension to 3D-Premise.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Is the Geometric Solver output given to an LLM during the feedback phase? I assume so given how it varies depending on which LLM is used in Table 2. Does it get to do two rounds of feedback like the CADCodeVerify model does?
- "Refine 1", "Refine 2", and "Generated" should be defined somewhere, right now the first place they appear is in Figure 2 headings (which are not searchable text in the PDF) and then when discussing results, but they aren't clearly defined.
    - Is "Generated" the step right after Section 3.1 finishes or right after Section 3.2 finishes – i.e. is "Generated" before or after the compiler-error based verification step?
    - Are "Refine-1" and "Refine-2" the two rounds of refinement (Section 3.3) or is the first one the code repair from Section 3.2?
- Why does "Refine-1" go *down* so much in Figure 3 for 3D-Premise? This is very surprising to me and would be helpful to clarify. Refinement ought to help, and it did help them in that prior work. While it's not as strong of an effect, refinement also seems to hurt all the methods a bit in some of the other plots (like the Complex plot) – why would rounds of refinement hurt instead of help?
    - These graphs are showing the compilation rate. It'd be helpful to see changes in the other metrics from Table 2 that have to do with correctness instead of whether or not the code compiles – those seem key to understanding whether feedback is helping.

Minor fixes:
- The CADQuery compiler $\phi$ is deterministic, right? So the "$\sim$" should be an "=" on line 177
- The description of the "number of compilable outputs" discussed in 305-310 was confusing to me until I saw the caption of Figure 11 in the appendix. It would be helpful to move part of this description to the main paper – it wasn't clear to me that this meant the number of experiments where the code compiled successfully. And to clarify – are those 6 experiments the six large rows in Table 2 (that vary between language model used and not between method used)?
- In discussing the geometric solver the paper notes "This feedback method serves as an upper limit for CAD code refinement, as it conveys the precise geometric differences between the generated 3D object and the ground truth." While I think this is a useful baseline, it's not literally an upper limit for performance from code refinement feedback – as shown for example by how it's occasionally outperformed in the experiments. It's a great baseline to have, I just would suggest against using the term "upper limit" for something that's not actually an upper bound.
soundness: 2
strengths: - The contribution of a language-to-CAD dataset is useful for the community, and hiring CAD experts to write the CAD code for it means it's likely high quality.
- The comparison is run across a range of LLMs including open source ones (CodeLlama) 
- They generally surpass the 3D-Premise baseline and score fairly close to the geometric solver baseline that gets to cheat and use the ground truth CAD model in calculating its feedback. They score particularly well on the more difficult problems, and benefit more from feedback than 3D-Premise. They show improvement across three different similarity metrics in addition to compilation success rate.
summary: This work contributes a new method for VLM-based CAD code synthesis, and a new dataset for evaluating CAD programming models. It builds on past work on code repair for CAD synthesis – the core idea is that instead of just providing the current render during repair (as prior work 3D-Premise did), the LLM also comes up with *verification questions* based on the task description (not using the current render) which it then answers (using the current render). These Q/A pairs are summarized into feedback for what needs changing, which is provided to the repair model alongside the current render to prompt code repair.

The dataset, CADPrompt, was constructed by selecting objects from a dataset used in prior work and annotating them with language prompts along with Python CAD code. They evaluate on a range of models (GPT4, Gemini, CodeLLama) and compare to several baselines (no refinement, 3D-Premise, and Geometric Solver). The last of these baselines, Geometric Solver, is also a contribution – it's a baseline that gets to "cheat" and compare the ground truth model to the current rendered model in terms of a range of geometric feaures. Their methods generally outperforms the baselines.
weaknesses: - The term "success rate" is a bit of a misleading name for something that means "compilation rate" or "compilation success rate" – it gives me the impression that the model succeeded at solving the task, not that it generated an arbitrary piece of code that compiled. For example in the abstract "increasing the success rate of the compiled program" or in the intro "5.5% increase in successful object generation" all gave me this impression until I dug into it.
    - Also, the abstract says 5.0% and the intro says 5.5% – which is correct?
- Does 3D-Premise get to do the Code Execution step described in 3.2, where if compilation fails it retries with compiler feedback up to N times? If not, it seems important to do this comparison to disentangle how much of the benefit is coming from compiler feedback versus the Q/A contribution.
- The improvements are modest but still reasonable – I would defer to a reviewer more familiar with these metrics on this. Looking at GPT-4 Few-shot in Table 2 (which looks like overall the best model), the IoGT metric difference is .942 vs .944, the Point Cloud distance is .127 vs .137, and the Hausdorff distance is .446 vs .419. These are not huge differences, but still a notable step towards the Geometric Solver performance.
- It's confusing that "Refine 1" causes 3D-Premise to decrease in performance in Figure 3 – see full note in "Questions" section, but clarification on this would be helpful to understanding the results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. In Fig 1, within the CADCodeVerify block, I see step 3.c generating feedback which then produces a new object. My question is: what exactly is the input to the VLLM? Is it the feedback, that then generates the code that compiles to produce the new object, or is it the new object itself? I suspect it is the former, but the figure indicates it to be the latter.
2. In the evaluation, do you consider naive or basic prompting? For example, just a zero-shot or few-shot prompt to an LLM that produces code? Is that what the "Generated" field in Table 2 is?
3. Are there any insights as to why CADCodeVerify performs better than baselines? For instance, what is it in the proposed technique that allows it to outperform 3D-premise? And why is VLLM feedback more useful than geometric feedback, even when the latter needs the presence of ground truths? One would imagine a technique needing GT labels for providing feedback would be more accurate or equivalent to a technique operating without the labels.
4. Maybe I missed this in the text, but in Fig 3, what is Refine 1 and Refine 2?
5. Are there any limitations as to the complexity and difficulty of the benchmark tasks? I understand they vary in complexity and difficulty, but at the same time CADCodeVerify does quite well, even reaching around 90% accuracy in the hardest case. Is there a class of problems this benchmark is not testing, or are the hard problems simply not hard enough?
6. How representative is the benchmark of the objects actually being drawn in the industry today? One major challenge with code generation has always been that it has never been representative of IRL code. Do we see the same challenges here?
7. What dataset was 3D-Premise originally evaluated on, and why have you not evaluated CADCodeVerify over that? What is different about your dataset that sets it apart from the one 3D-Premise used?
soundness: 4
strengths: This paper is written really well. As someone who doesn't work much with CAD code, it was immediately clear what the limitations were and how they intended to address them. The technique itself is novel for the application and is well described with appropriate examples. I also appreciate the effort taken for creating the benchmark tasks over which the systems were evaluated.
summary: This paper proposes a technique called CADCodeVerify that uses feedback generated from VLMs to refine CAD code generated by LLMs in an iterative manner. This is a novel technique proposed to address a reasonably difficult and seemingly important problem. The paper evaluates this technique against an existing baseline called 3D-Premise and also with a baseline using feedback from a geometric solver. CADCodeVerify shows better performance against both baselines.
weaknesses: One main weakness I can see is the lack of baselines. However, this is not a major issue to me since I don't know of other systems that target this specific problem. However, I can think of other techniques like chain of thought prompting, or techniques using some form of reasoning, against which we can compare CADCodeVerify. Right now, the only true baseline seems to be 3D-Premise, and another baseline that uses an alternate form of feedback generation, though I could be mistaken here.

Meta-review: In this paper, the authors introduce CADCodeVerify, a novel approach that uses VLMs for iterative verification and refinement of CAD code generation. To evaluate CADCodeVerify, the authors also propose CADPrompt, a benchmark for CAD code generation consisting of 200 natural language prompts paired with expert-annotated scripting code for 3D objects. Under this benchmark, the experiments include comparisons with existing methods as well as with human experts, demonstrating that the proposed method is superior.

However, there are a limited number of technical contributions. The main contribution seems to be the novel mechanism of using off-the-shelf LLMs and VLMs to automatically generate high-quality CAD code. Though the experiments show superior results, the small scale of the experiments makes it difficult to conclude that the pipeline is universally applicable. In addition, the method's reliance on natural language input may not account for precise numerical specifications, which is another limitation. Therefore, I recommend accepting the paper but as a poster.

Decision: Accept (Poster)