Generating CAD Code with Vision-Language Models for 3D Designs
Authors: Kamel Alrashedy
Pradyumna Tambwekar
Zulfiqar Haider Zaidi
Megan Langwasser
Wei Xu
Matthew Gombolay
Venue: ICLR 2025 Poster (2025)
Keywords: Code Generation
Self-refinement
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: we develop a Self-Verification approach, wherein LLMs generate 3D objects through scripting code, and then verify whether the generated 3D objects meet the requirements.

Abstract: Generative AI has transformed the fields of Design and Manufacturing by providing
efficient and automated methods for generating and modifying 3D objects. One
approach involves using Large Language Models (LLMs) to generate Computer-
Aided Design (CAD) scripting code, which can then be executed to render a 3D
object; however, the resulting 3D object may not meet the specified requirements.
Testing the correctness of CAD generated code is challenging due to the complexity
and structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are not
feasible in code. In this paper, we introduce CADCodeVerify, a novel approach to
iteratively verify and improve 3D objects generated from CAD code. Our approach
works by producing ameliorative feedback by prompting a Vision-Language Model
(VLM) to generate and answer a set of validation questions to verify the generated
object and prompt the VLM to correct deviations. To evaluate CADCodeVerify, we
introduce, CADPrompt, the first benchmark for CAD code generation, consisting of
200 natural language prompts paired with expert-annotated scripting code for 3D
objects to benchmark progress. Our findings show that CADCodeVerify improves
VLM performance by providing visual feedback, enhancing the structure of the 3D
objects, and increasing the success rate of the compiled program. When applied to
GPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a
5.0% improvement in success rate compared to prior work.

Published as a conference paper at ICLR 2025
GENERATING CAD CODE WITH VISION-LANGUAGE MODELS FOR 3D DESIGNS
Kamel Alrashedy; Pradyumna Tambwekar", Zulfiqar Zaidi, Megan Langwasser, Wei Xu, Matthew Gombolay Georgia Institute of Technology, GA, USA
(kalrashedy3, ptambwekar3, 3 zzaidi8, a mlangwasser 3 @ @atechh.edu (wei xu, mat mhtthew.wee wwwwww.oooolooooom a } @ ec.gaatehhedu
ABSTRACT
Generative AI has revolutionized the fields of Design and Manufacturing by provid- ing efficient and automated methods for generating and modifying 3D objects. One approach involves using Large Language Models (LLMs) to generate Computer- Aided Design (CAD) scripting code, which can then be executed to render a 3D object; however, the resulting 3D object may not meet the specified requirements. Testing the correctness of D-D-generated code is challenging due to the structural intricacies of 3D objects that are not discernable in code. In this paper, introduce CADCodeVerrin a novel approach to iteratively verify and improve the design out- put of 3D objects generated from CAD code. Our approach provides ameliorative feedback by prompting a Vision Language Model (VLM) to generate and answer of validation questions verify the generated object and prompt the VLM to correct deviations. To evaluate CADCode Veriff, we introduce, CADPrompt, the first benchmark for CAD code generation, consisting of 200 natural language prompts paired with expert- anpert-annottted scripting code for 3D objects to benchmark progress. Our findings show that CADCode Verify improves VLM performance by providing visual feedback by enhancing the structure of the 3D objects and increasing the compile rate the compiled program. When applied to GPT-4, CADCode Verify achieved 7.30% reduction in Point Cloud distance and a 5.5% improvement in compile rate compared to prior work. Code and data are available at athttss://gthhut www./aamwe7///////mmccoo httpa//....oooe.emmee//////cc... Generttttiooono
INTRODUCTION
Generative AI, such as Large Language Models (LLMs) offers a unique opportunity to enhance productivity, reduce costs, and increase efficiency within Design and Manufacturing sectors (Kumar et al. 2023). These industries are critical contributors to the global economy and responsible for creating products and infrastructure. Recent research has demonstrated that Generative can support the generation, evaluation, and correction of 3D object designs (Nelson et al.) 2023 Kodnongbua cct al. 2023; Makatura et al. 2023). However, while these solutions improve efficiency for designers and engincers, they often lack effective feedback mechanisms or refinement loops to automatically address inaccuracies in the initially generated 3D object.
Our research explores the potential of using VLMs to generate and refine Computer-Aided De- signs (CAD), the CAD a leading approach in industrial design for creating, modifying, analyzing. and optimizing 3D objects (Sarcar et al. 2008). Designing products with CAD software such as FreeCAD (Riegel et al. 2016) and Auto CAD (Yarwood 2013) generally requires substantial training and domain expertise. CAD software often includes scripting languages that allow users build parametric 3D objects using code, rather than relying solely on the user interface. Leveraging the
code-generating capabilities of Generative AI enables users bypass the complexities of traditional CAD software by directly generating the underlying scripting code for representing 3D objects.
We define the process of generating and refining 3D objects using LLMs or VLMs, through CAD scripting code, as CAD code generation. Designs produced by off-the-shelf VLMS or LLMs often
"These authors contributed equally this work

---

size Published as a conference paper at ICLR 2025
(3.a) Question Generation
Language Pronpt (x)
obfect's larga
Python using CADQuery
rectangle its height about 2/3rd its width?
to generate CAD model for
object. Sketch large rectangle whose height is about 2/3rd its width, Now from reciangle, should rectangles
Ground Truth
Language Pronpt
there cutout froe the rectangle ractangle?
the of the top coincident
(3) Code Refinement
edge the rectangle?
VLM
CADCodeVeriyy
(3.b) Question Answering
Yes,
2. Yes, Reason:
[3] Code
Reason:
Generation
Generated Object
CaDquery Code (p)
irpat oae aay 2006
Generated Feedback
wath 23:2
cq
(3.c) Feedback
Based on the answers provided, the only correction needed object adjust position cutoul....
Generation
(eatt Www.aaaaee" light
Refined Ooject
Our approach enables VLMs to automatically generate and refine 3D objects through a CAD scripting code (e.g., CADQuery) in three steps; (1) Code Generation, where the VLM generates CAD scripting code from a language prompt, (2) Code Execution, where the code generated by the model is rendered as 3D object through a compiler, and (3) Code Refinement, wherein the language model engages in a self-initiated guestion-answerinn process to validate the generated object, with respect the initial prompt, to generate actionable feedback refine the code.
Figure I:
deviate functionally or structurally from stakeholder specifications these models often hallueinate on complex, out-of-distributtin tasks. Developing refinement methods to correct inaccuracies in generated design is a critical next step in this research. Current state-of-the-art refinement methods are contingent on human in-the-loop expertise (Makatura et al. 2023 Nelson et al. 2023) which can be time- and prohibitive. Instead, we propose to develop automated feedback mechanisms for CAD code generation to reduce the barrier to entry for design and expedite lengthy design processes.
In this work, we introduce CADCode Verify, an automated method for refining CAD code generation. MADCodeVeriff eliminates the need for human involvement by generating and answering validation questions based on user requirements, offering feedback refine 3D object code. This feedback is used to iteratively refine the design. To evaluate the performance of CADCode Verify, we introduce novel benchmark, CADPrompt contains 200 3D objects annotated with natural language prompts and expert- written Python code. This multimodal benchmark enables researchers to evaluate aspects of CAD code generation, including object quality and the syntactical correctness of the generated code. Our experimental results demonstrate that CADCode Verify not only enhances the quality of generated 3D objects but also improves the performance of LLMSs and VLMs by increasing the compile rate of compiled programs. Our work presents three key contributions:
1. We propose CADCode Verify, novel CAD code refinement method that enables VLM to visually inspect generated objects and provide corrective feedback, through a question- generation and answering process, resolve any deviations from the user's specifications.
2. With our benchmark, CADPrompt, we provide the first quantitative evaluation of CAD code generation across GPT-4, Gemini 1.5 Pro, and Codel.lama.
3. We demonstrate that CADCode ing, achieving 7.30% reduction in Point Cloud distance and a 5.5% increase in successful object generation using to GPT-4, the leading VLM for CAD code generation.
new sfate-of-the-art 3D design via CAD script-
2 RELATED WORK
2.1 LLMS FOR CODE GENERATION
LLMs have demonstrated impressive results in a variety of code generation applications. The most popular application is the text-to-code task, wherein users prompt L.LMs with code deseription,
2

---

Published as a conference paper at ICLR 2025
enabling the LLM to generate the code (Rozière et al. 2023: Svyatkovskiy et al. 2020 Brown et al. 2020; Poesia et al. 2022). Another application is the text-to-SQU task, where a database and a question provided to the LLMs, which then generate the corresponding SQL query (Rajkumar ett al. 2022). Building on these advances, we explore how LLMS can support non-experts and designers in generating 3D objects via CAD code.
Prior research has demonstrated that LLMs can significantly improve the quality their generated code by incorporating feedback (Chen et al. 2023a). One such approach involves LLMs providing their own feedback by generating explanatory rationales for the initially generated code. These rationales are then used as feedback, along with the initial code, refine the output (Chen al.
2023b).
Another approach relies on feedback from external tools, such as static code analysis (A yArrashedy et al. 2024). or a Python Interpreter (Madaan et al. 2023).
In contrast, the self-correction method (Welleck et al. 2022) trains two separate models: generator, which creates the initial output, and a corrector, which refines the generated code. In CAD code refinement, one line of research introduces basic feedback mechanism (Yuan et al. 2024). wherein VLMs refine 3D object code based both on a script and an image. However, this method still faces challenges in terms of feedback efficiency and refinement precision. In this work, we propose a novel approach to generating visual feedback that improves the shapes, surfaces, and dimensions of 3D objects through more effective code refinement.
2.2 LANGUAGE FOR AUTOMATED DESIGN AND MANUFACTURING
Learning a shared embedding space between language and 3D objects is essential for integrating language into design and manufacturing. Recent advancements have introduced foundational multi- modal transformers for image-text data (Radford et al. 2021 Li et al. 2023a; Liu et al. (203)) which
can used to perform variety of tasks. These techniques have inspired similar approaches for incorporating 3D objects into shared embedding spaces (Zeng al.) [023] Xue et al. 2023] Yu et al.)
2022| Guo et al. 2023), enabling a wide range of multimodal tasks such as dialogue, classification,
and evaluation involving 3D data (Hong et al. 2023 Xu et al. 2023 While successfully learning shared-embedding space, these approaches are not capable of generating 3D objects.
Generative modeling of 3D data has traditionally relied on probabilistic models for 3D space. These include: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). Variational AutoEncoders (VAEs) (Kingma & Welling (013). and Diffusion Models (Ho et al.) [0200_ While et al. 2018 Gadelha et al. 2018 Yang et al. (019 Yan et al. 2024) have been popular approaches, diffusion models have recently emerged as the state-of- the-art for probabilistic 3D modeling (Zeng et al.) 2022)) Koo et al. 2002). Diffusion models enable controllable object generation through language-gune guided shape generation and completion. However, their application in producing manufacturable designs remains limited (Li et al. 20236; Kodnongbual al. (2023). These models typically generate point clouds or voxels, which are non parameerii and not easily adapted for manufacturing. As such, there is a need for methods that produce parametric outputs suitable for manufacturing.
GANs and VAES (Wu et. | [015]
Recent research has shown that VLMs can generate designs using parametric CAD code (Makatura al., 2023) Nelson et al.) 2023). However, these approaches still require significant human feedback to produce code ensure the generated codes meets the user's specifications. In this paper, we address this limitation by developing CADCode which autonomously generates and refines 3D objects.
CAD CODE GENERATION
As shown in Figure ! we employ three-step process: (1) code generation ($.I), (2) code execution (.22). and (3) code refinement via VerfodeVerify for CAD code generation( 3.3
3.1 CODE GENERATION
We prompt the VLM to generate initial CAD script code, yo, based on a natural language description of the 3D object, x, along with task specifications. For the few-shot experiment, set of few-shot demonstrations, Eg. in the prompt (see Appendix [.4].. We utilize CADQuery, a Python-based parametric CAD language with built-in compiler that interprets code and renders
incorporate
3

---

Published as a conference paper at ICLR 2025
parametric 3D object. This CAD code generation can be broadly formulated as per Eq.
Yo PLar (oo Feg)
3.2 CODE EXECUTION
The generated CAD code, Yo. is subsequently executed by the CADQuery compiler, v), to produce the initial CAD design, dg. in the Standard Triangle Language (STL) format where indicates the initial version of both the generated code and CAD design. We denote this process do (yo)). Occasionally, the models generate code that fails to compile, usually due to syntax errors in the Python code. We leverage insights from previous work on code repair (Chen al. (023b) and adapt similar techniques for CAD code generation. To resolve syntactical issues in the code, we leverage the CADQuery compiler error as feedback, Ferr. for the VLM (see Appendix Figure [2] for the prompt employed in code repair). This process is repeated until 3D object, d3. is successfully rendered by (3 or the maximum number of iterations, N, reached as per Eq. 2
yk {PLM (yk æ. 1k- Ferr) )
(2)
3.3 CODE REFINEMENT VIA CADCODEVERIFY
To address discrepancies in the generated design, we use a feedback loop to further refine the CAD code, Fref. to further refine y/k as described by Eq. 3 where M is the number of refinement iterations. Optionally, we include set of four reference images of the generated object, captured from different angles (0, 90, 180, and 270 degrees), L = {4p 19D 1 180 17700.
yM ~ {PLM x yt, Fres. Iref) N N
The key innovation of CADCode Verify is its feedback loop, which uses VLM to generate visual feedback automatically, without human intervention or the need for external tools like geometric solvers (Makatura et al. 2023) (see $5.1) CADCode Verify is two-step process consisting of question-answer generation and feedback generation.
(1) Question-Answer Generation: CADCode Verify first generates of binary "Yes/No" verifi- cation questions, Q = (?1, 12......9000 example questions, Egp i.e., Q PLas (QJær, E,). CADCode Veriff generates between two five questions per example (see Figure Mi the Appendix). To answer these questions, CADCodeVerify uses the reference images of the generated 3D object, Inef. the generated questions, Q, and the language description x. CADCode Verify then answers each question as described in Eq.
On }. based on the language description, I, and a set of few-shot
A PLM (2.((A(ax) [A|r, Q, Teer).
CADCode Verify generates the answers, A = (a1 a2, a,), using Chain-of-Thoowe (Wei et al. 2022), where each answer is accompanied by supporting reasoning (see Figure 13 in the Appendix). Furthermore, CADCode is directed to respond with "Unclear" if it determines that there is insufficient information to answer the question.
(2) Feedback Generation: The question-ansee answer pairs are then used to generate ameliorative feedback to further refine the 3D object as per Eq.
Frof ~ PT.M (Fres Q A).
(5)
feedback, Frez, is then applied to refine the code, described Eq. During this step, omit any questions for which the answer is "Yes" to allow the model to focus on addressing any unresolved issues. If all questions
answered "Yes" we assume further refinement is necessary. The full
feedback generation process is provided in Figure 14in the Appendix. Additionally, two examples of full interactions with GPT-4 for single iteration of CADCode Verify are included in the Appendix (
Figures 9 and 10).
'Standard Triangle Language is also referred to as "Standard Tessellation Language". The STL (STere- ol itthorraphh) file format is openly documented format for describing the surface of object a triangular mesh, that representation of 3-dimensional surface in triangular facets.
4

---

Published as a conference paper at ICLR 2025
Ground Truth
Natural Language Descriptions (Prompt)
Generated
Feedback
Refine-1
Refine-2
Write Python code using CADQuery to generale model
object. Siketch large rectangle whose height is about 2'3rd its width. Now create a square cutout
3D-Premise
rectangle, should 1/3rd the rectangles size. This square cutout is placed such center of the top edge of the first rectangle. This square cutout extends downward to about center of the rectangle. Now extrude the remaining shape to get final 3D object.
CADCodeVerily (Ours)
(Simple, Hard)
top edge
Geometric Solver
Write Python code using CADQuery to create a 3D object that resembles square baking First, draw a sketch of square with rounded comners and then extrude this sketch. Next, draw another sketch of a slightly smaller square with rounded edges, centered the same point of the original square, the surface first object. Negatively extrude this object by the same amount the first extrusion to create hole in original object.
3D-Premise
CADCodeVerify (Ours)
(Complex, Hard):
Geometric Solver
Figure 2: Examples of 3D objects generated by GPT4 after utilizing each feedback method for code refinement from our CADPrompt dataset.
Table Statisties of the CADPrompt dataset, including the vertex and face counts of ground truth objects, well as the lengths of language descriptions and Python code ($4.))
Structural Complexity of 3D Object
Natural Langunge Descciptions
Python Code
Total Datapoints
Category
Vertices mas
Faces mas
Words mar
Sentences max
Lines of Code
Tokkns max
min
ang.
min
avg.
min
min
avg.
min
mas
min
Simple
106
25.6
213
47.3
13.4
1.0
18
8.6
18
48
26.5
17
Moderate
344
78.6
684
155.0
9.
87
50.5
5
2.2
21
12.6
22
60
35.2
59
Complex
8
540
93.1
12
1092
185.8
13
104
53.9
7
3.6
32
16.9
25
82
47.3
87
Very Complex
12
531
108.0
20
1070
216.4
13
188
63.5
11
4.5
46
21.3
18
117
53.6
57
All
540
$8.8
1092
175.7
9
188
50.13
11
34
46
16.6
18
117
45.0
200
CoDPrompt DATASET
For evaluating CADCodeVerify on CAD code generation, we introduce a new benchmark the CADPrompt, which consists of 200 3D objects, represented both in images and STL. Each object in CADPrompt is annotated with ground truth code, and a natural language prompt.
4.1 PROMPT CREATION
To construct CADPrompi dataset, we first selected 200 3D objects from collection of modular CAD objects from previous work (Wu et al. 202T). Each object was manually annotated with a natural language prompt. For difficult-o-ddecribe objects, two annotators independently provided prompts. An independent third annotator then selected the more suitable prompt. Finally, a fourth independent reviewer, not involved in the original annotation, verified and refined each of the 200 prompts to ensure accuracy and grammatical correctness. Table provides the statistics of CADPrompt.
4.2 CODE ANNOTATION
We recruited CAD Design expert to annotate CADPrompt with ground truth code for each object. The CAD design expert was provided with the language description of the 3D object, the object in STL format, and its geometric properties generated by the geometric solver (Figure We then used Blender (Flavell 2011) to validate the 3D objects produced by the expert's 's Python code against the ground truth and evaluate their geometric properties (Figure I8). In cases where discrepancies were identified, the Python code was returned to the CAD expert for additional refinement. Figures 16 17 provide examples of Python code from CADPrompt.
5

---

Published as a conference paper at ICLR 2025
4.3 DATA STRATIFICATION
We stratify CADPrompl examples by mesh complexity, geometric complexity and compilation difficulty gain insights into model performance ($6).
Mesh complexity: We define "mesh complexity" as the total number of faces and vertices of an object. The ground truth of the 3D object is represented in mesh file formats, which consist of collection of vertices, edges, and faces (æ, y, used to create the 3D object in the STL format. Objects with more faces and vertices are classified as more complex, while those with fewer faces and vertices are considered simple (Table We split the dataset into two groups based on the median complexity: (i) Simple (those with fewer faces and vertices than the median) and (ii) Complex objects (those with more).
Compilation difficulty: We define "compilation difficulty" to be measure of how difficult it is for of three language models (i.e., GPT-4, Gemini, and CodeLlama) to generate code for given 3D object across two prompting methods (i.e., zero- and few-shot prompting), for total of six attempts to generate compilable code. 3D objects were labeled then as either (i) Easy (at least four of six methods generated compilable code) and (ii) Hard (otherwise). See Appendix Figure 15
Geometric complexity: We enlisted a CAD design expert to evaluate the "geometric complexity" of each object in CAD- Prompt. Each object was assigned one of the following levels: (i) Simple: the object is basic, with few features. It may consist of one geometric shape; (ii) Moderate: the object has a moder- ate amount of detail, with few distinct features components; (iii) Complex: the object has many interconnected parts, fine details, or intricate shapes; and (iv) Very Complex: the object is highly intricate, with many components, detailed textures, or complex shapes. It may have a large number of fine details, interlocking parts, or unique geometric features (Figure
Simple
Compphon Roooo
complex
Figure 3: Examples of 3D objects from the CADPrompt dataset.
5 EXPERIMENTAL SETUP
5.1 BASELINES
We compare our approach, CADCodeVerify, with two baseline methods for feedback generation: (i) 3D-Premise, and (ii) Geometric solver feedback.
3D Premise (Yuan et al. 2024): 3D-Premise facilitates code refinement by providing GPT-4 with both the image of generated object and its original description. GPT-4 is then prompted correct any discrepancies between the original description and generated objects. To establish a baseline for comparison, we supply the VLM with an image the generated object and use the same prompts as used in the original work. Note that this baseline is not applicable for approaches lacking multimodal capabilities, such as CodeLlama.
Geometric solver feedback (this work): We develop novel baseline that leverages FreeCAD, an open-source geometric solver, to provide information about the geometric dimensions and structure of the generated 3D object, dg. However, our proposed bascline, which computes geometric properties feedback using the geometric solver, requires access to the ground truth 3D objects, dyt in STL format. Specifically, the feedback, Fas, consists of numerical values across thirteen categories, S, each representing a unique geometric dimension or component of the object, such as width, height, number of faces, number of vertices, and volume. Formally, this feedback process is shown in Eq. and/] We compute Føs for both the generated design, dg. and ground truth, dgs. In summary, we can describe the role of the geometric solver as follows:
Fos(dg) = ((s, (S dg)), Vs
(6)
F, (d)(dg) @ (dg(
(7)

---

Published as a conference paper at ICLR 2025
The feedback from the geometric solver consists of numerical values across thirteen categories of geometric information. An example of this feedback, Fgs. is shown in Appendix (see Figure This feedback method acts as upper bound for CAD code refinement, as it conveys the exact geometric differences between the generated 3D object and the ground truth.
5.2 EVALUATION METRICS
This evaluates how accurately VLMs generate and refine objects through CAD code. We use three evaluation metrics to compare the quality of the generated 3D object against the ground truth: (i) Point Cloud distance, (ii) Hausdorff distance, and (ii) Intersection over the Ground Truth (IoGT). We apply the Iterative Closest Point (ICP) algorithm to rotate and translate the generated 3D object for optimal alignment with the ground truth object (Besl & McKay 1992). Finally, each point cloud is normalized fit within unit cube, per prior work (Zheng al. (023).. We also report the percentage of successfully compiled code as the "compile rate". The formulas for the evaluation metrics are described as follows:
Point Cloud distance:
Point Cloud distance D(P, Q) is shown in Eq. where dp.q = 1lp - 4112-
D Q)
min d pEp 9Eq
min dy.q
(8)
P
2/Q1
PEP
PEQ
Hausdorff distance: Hausdorff distance, H(P, Q), is given by Eq. where sup and inf represent
the supremum and infimum operators, respectively (Beauchemin al. 1998).
(PP, Q) max inf dp.a, sup inf dg,z)
(9)
PEP PEP
Intersection over the Ground Truth (loGT): This metric used evaluate how closely the generated 3D object aligns with the ground truth Q. As shown in Eq. 10 it calculates the ratio of the intersection area between P and Q to the area of Q.
IoGT PnQ [Q]
(10)
Point Cloud distance and Hausdorff distance are based on the geometric properties (c.g., height, width, and volume) of 3D objects, while the IoGT measures the overlap of the bounding boxes between the generated and ground truth 3D objects. If any generation fails to compile, we set the distance (Point Cloud and Hausdroff) to v (the largest possible distance between two points in a unit cube) and an loGT value of zero (the worse possible loGT value between two 3D objects) maximally and uniformly penalize unsuccessful 3D object generations.
6 EXPERIMENT RESULTS
We utilize CADPrompt to quantitatively evaluate the capabilities of CADCode Verify approach across various VLMs. the rest of this section, we refer to the three stages of the CAD code generation process as "Generate" "Refine-1" " and "Refine-2" "Generated" refers to the object generated by the LLM after the Oode-Execution Step. "Refine- refers the object after the first step of refinement, and Refine-2 refers to the object after the second step of refinement. Our key findings are as follows:
GPT-4 demonstrates the highest capacity generate compilable code CADPrompt. We present our primary results in 2 comparing the distance and success-rate across three L.MSs and three refinement methods. Regarding compile rate, GPT-4 shows superior performance at 96.5% on the CADPrompt benchmark, compared to Gemini 85% and CodeLlama 73.5%. While previous studies have provided valuable qualitative insights into GPT-4's ability to produce compilable code (Makatura al.) 2023), our analysis using CADPrompt offers concrete evaluation of each LLM's CAD code generation capability.
CADCode Verify demonstrates superior performance more challenging data. Figure shows the compile rates for each feedback mechanism at the generation and refinement stages, using the difficulty and complexity splits of CADPrompt. For "Easy" data, the performance across all

---

Published as a conference paper at ICLR 2025
Table This table reports the median (IQR) benchmarking results for baselines across metrics. The symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.
Model
Feedback Mechanism
IoGT
Point Cloud dist.
Hausdorff dist.
Compile Rate
Generated
0.935 (0.043)
0.153 (0.146)
0,484 (0.405)
92.0%
GPT-4 Zeto-shot
3D Premise
0.939 (0.034)
0.150 (0.143)
0.440 (0.372)
91.5%
CADCodeVerify (Ours)
0,941 (0.034)
0.132 (0.137)
0.455 (0.354)
94.0%
Geometric solver?
0.943 (0.037)
0.102 (0:759)
0,378 (0.433)
"1.3%
Generated
0.939 (0.030)
0.155 (0.140)
0.494 (0.368)
96.0%
GPT-4: Few-shot
3D-Premise
0.942 (0.033)
0.137 (0.155)
0.446
91.0%
CADCode (Ours)
0.944 (0.028)
0.127 (0.135)
0.419 (0.356
96.5%
Geometric solver
0.944 (0.031)
0.103 (0.152)
0.399 (0.433)
95.5%
Generated
0.905 (0.088)
0.159 (0.180)
0.531 (0.451)
85.0%
Gemini: Zero-shot
3D-Premise
0.911 (0.079)
0,150 (0.180)
0.496 (0.4311)
83.5%
CADCode Verify (Ours)
0.914 (0.082)
0.138 (0.165)
0,497 (0.384)
84.5%
Geometric solver*
0.917 (0.070)
0.113 (0.188)
0.416 (0:458)
83.5%
Generated
0.933 (0.061)
0.171 (0.174)
0,521 (0,426)
85.0%
Gemini: Few-shnt
3D-Premise
0.939 (0.070)
0.169 (0.184)
0.521 (0.516)
81.5%
CADCode Verify (Ours)
0.939
0.147 (0.160)
0.492
85.0%
Geometric solver"
0.944 (000606)
0.103' (0C148)
0.386 (0:3770)
"K435%"
Generated
0.920 (0.943)
0.237 (1.591)
0.731 (1.270)
64.5%
CodeLlama: Zero-shot
CADCode Verify (Ours)
(0.955)
0.211 (1.599)
0.641 (1.309)
70.0%
Geometric solver?
0.888 (0.941)
0.2800 5535)
008823 (1.288
33.33%
Generated
0.927 (0.949)
0.224 (1.597)
0.657 (1.294)
67.0%
CodeLlama: Few-shot
CADCodeVerify (Ours)
0.935 (0.957)
0.185 (1.620)
0.582 (1.366)
73.5%
Geometric solver*
0.906 (0.949)
0.239 (1.606)
0.727 (1.301)
60.5%
100
100
100
100
Comple Rate 1961
70
70
70
70
60
60
60
60
Generated Reliee 1 Refine
Generated Refine
Refine 2
Generated Refine 1 Refine
Generated Refine
Refine 2
Difficulty: Easy
Difficulty: Hard
Complexity: Simple
Mesh Complexity: Complex
3D-Premise
CADCodeVerity (Ours)
Geometric solver
Figure 4: Compile rate comparison across different feedback mechanisms, categorized by difficulty and mesh complexity. Each figure corresponds to a specific subset of the CADPrompt dataset, covering both generated and refined objects. The plots for the "Hard" and "Complex" splits indicate that the feedback approach from 3D-Premise reduces GPT-4's compile rate in object generation.
baselines is relatively similar. However, results on the "Hard" data underscore the effectiveness of Verify, which is the only refinement approach that enhances the compile rate of object generation, achieving approximately a 9% increase at the Refine-1 stage. In contrast, 3D-Premise struggles provide effective feedback for "Hard" data, resulting in 20% drop compile rate at 1efine-1, bringing it down to 62%. Real-world data in design and manufacturing is likely to align more closely with the "Complex" and "Hard" data splits, underscoring the value of CADCode. Our findings underscore the value of CADCode Veriff in delivering feedback that can be applied to real-world product design.
CADCode generates model-agnostic feedback to improve 3D object generation. Table presents the distance measure from the ground truth at the generation and refinement stages. Since CodeLlama lacks multimodal capabilities, we employ GPT-4 execute CADCode refinement feedback. Our results indicate that CADCode Verify improves the quality of the generated object across all three LLMs/VLMs as measured by IoGT, Point Cloud distance and Hausdorff distance. CADCode Verify also increases both GPT-4 and CodeLlama's s "compile rate" in producing compilable CAD code. This result highlights the model-agnostic nature of the multimodal "assess
and generate
8

---

Published as a conference paper at ICLR 2025
and reason" analysis induced by our CADCode Verify refinement process, to universally provides actionable feedback for correcting object generation errors.
Table 3: Median (IQR) results from our ablation study conducted on randomly selected subset of 100 samples.
Ablation Study
loGT f
distances :
Hausdorff dist. v
Compile
f
Generated
0.909 (0.062)
0.156 (0.150)
0.491 (0.348)
96.5%
Refine without images
0.914 (0.058)
0.153 (0.123)
0.451 (0.321)
96.0%
Zero-shot QA generation
0.919 (0.049)
0,141 (0.112)
0.471 (0.280)
98.0%
CADCodeVerify (Ours)
0.919 (0.045)
0.126 (0.122)
0.444 (0.308)
97.5%
CADCodeVerify outperforms 3D-Premise across VLMs. Table 2 we quantitatively compare our CADCode Verify approach to refinement method proposed in prior work, 3D-Premise. Since 3D-Premise requires directly uploading an image of the generated object to the VLM, this approach cannot be applied to the unimodal CodeLlama model. For GPT-4 and Gemini, CADCode Veriff is shown to refine objects more accurately than 3D-Premise for objects in the CADPrompt dataset (> 7% reduction in point cloud distance for GPT-4 and Gemini). This finding establishes CADCode Verify as the new state-of-the-art for CAD code refinement. Next, we compare our proposed refinement approach to upper benchmark that uses a geometric solver to provide parametric feedback on specific geometric differences relative to the target object, as per $ $ Our results show that through CADCode Verify,, we can achieve comparable performance without requiring access to the target object, which is typically unavailable in real-world applications.
Ablation analysis. We conduct an ablation to evaluate the effectiveness of each independent component of CADCode VeDCodeVerify (see T This analysis is performed using GPT-4 with a few-shot prompt on 100 randomly selected examples from CADPrompt. First, we test the effect of removing the few-shot example questions in the question-generatiin that zero-shot QA generation increases the distance between the ground truth and refined 3D objects from 0.126 to 0.141 in Point Cloud distance and from 0.444 to 0.471 1 in Hausdorff distance. Our second ablation measures the impact of the reference images, Ire/s during the code refinement step of Eq. 3 Excluding the reference images worsens the Point Cloud and Hausdorff distances from 0.626 and 0.444 to 0.153 and 0.451, respectively, emphasizing the importance of the reference images.
component (see 3). The results indicate
Table 4: Performance Comparison of CADCodeVerify and Human-in -ie-iiin-ttt-tttt (HITL) approaches on a subset of 50 examples from the GPT-4 few-shot setting.
Model
Feedback Mechanism
1oGT
Point Cloud dist.
Hausdorff dist.
Compile Rate
Generated
0.930 (0.043)
0.156 (0.138)
0.495 (0.287)
98.5%
GPT-4: Few-shot
CADCodeVerify
0.948 (0.036)
0.137 (0.136)
0.445 (0.302)
98.5%
Human-in-the-Ile
0.944 (0.0322
0.120 (0.140)
0.397 (0.354)
99.0%
Gold-Standard human feedback outperforms CADCode Verify slightly. This experiment evaluates the performance of CADCode Verify compared to human-in-the-loop approach, where instead of using CADCodeVeriie for code refinement, we provide gold standard language feedback which includes the exact changes that need to be made the object. We performed this experiment on randomly selected subset of 50 examples using the GPT-4 few-shot setting. The experiment involves two steps: (1) a human participant selects the image with the best viewing angle of the 3D object from four options (see Eq. 3. and (2) the human provides written feedback. The results indicate that the human-in-thee -ooe approach led to a slight improvement in Point Cloud distance and Hausdorff' distance, with performance increasing from 0.137 and 0.445 to 0.120 and 0.397, respectively (see Table 4). The result of the human- in-tun----the--oop contextualizes the performance of CADCodeVerify with respect to method which injects dxpain-expertise provide the best possible feedback. While CADCodeVeriity makes significant strides in refining 3D objects, without any requiring domain expertise, some improvements are still needed to reach the level of human expert.

---

Published as a conference paper at ICLR 2025
Natural Language Descriptions (Prompt)
Generated
Ground truth
Write Python code using CADQuery to create 30 model. This object comprised of two marginally extruded rectangles that connected at their edges like the intersection between the floor. Both rectangles extruded rectangles are parpendicular to each other and are connected by the elongated edgo.
Correct answer?
same length, width, height. The two
the 3D model consist of two rectangles? Reasoning: From the images provided, is clear that the object consists of two distinct rectangulor shapes.
Yes
X
Qz. Are the two rectangles connected at their edges? Answer: Yes
Reasoning: images that the rectangles connected at their edges, forming a joint they meet.
both rectangles have the length, width, and height? Answer: No
Reasoning: The rectangles appear to have different widths and possibly different lengths, as observed from the images where the proportions of the rectangles differ.
the rectangles perpendicular Reasoning: The images show that the rectangles are orlented at 90-dagree angle to each other, indicating that they are perpendicular.
other?
Qs. Are the rectangles connected by an elongated edge? Answer:
Reasoning: The connection between the rectangles does not appear be along an elangated edge but rather at single point shorter edge, the images
rectangles
side.
Figure 5: An example of subset of questions and answers generated by CADCodeVerify.
Accuracy of generated answers. In the Question-Answering phase, AADCodeVerify is prompted to respond with "Yes," "No" or "Unclear," in situations when there is insufficient information (See Figure To evaluate the accuracy of these answers, we randomly selected subset of 50 examples from the GPT-4 few-shot setting, then manually validated the answers for both refinement stages. The results indicate that CADCode I and 68.2% for Refine 2 (see Table 9 To reduce hallucinations from the LLMs, we instructed it to respond with "Unclear" whenever it lacked confidence in its answers. In future work, we aim to explore how LLMs can interpret 3D objects and investigate methods to teach LLMs to self self-verify their generated responses.
provides correct answers with an accuracy of 64.6% for Refine
LIMITATIONS
It is essential to recognize that Point Cloud distance and Hausdorff distance are noisy metrics, measuring only the spatial similarity between two 3D objects. While they provide a broad estimate of similarity, a more granular metric needed to capture structural differences between objects. For instance, desk and a desk with small gaps between the legs and surface might have a low distance measure, but these gaps represent critical logical or structural issues that should be captured in the evaluation. In future work, we plan to investigate evaluation methods for CAD code generation that incorporate logical design principles.
The quality of generated objects and the likelihood of successful code compilation are influenced by the user's initial prompt. Due to the flexibility of natural language, the interpretation of 3D objects can vary, resulting in multiple valid but functionally different descriptions for the same object. For example, one could describe a desk could be described functionally as "Draw a desk with four legs." or more geometrically as "Draw an object with flat rectangular top supported by four long rectangular prisms at each corner". In our approach, we incorporated review procedures to improve data quality, though it was feasible to exhaustively explore all possible annotation methods. Future research could delve deeper into this prompt sensitivity and explore strategies for identifying the most effective prompts for specific VLMs.
8 CONCLUSION
In this work, we formally define a novel task, CAD code generation, wherein VLMs are employed to generate code for 3D parametric models. We compiled a novel CADPrompt dataset, comprising 200 3D objects paired with corresponding language descriptions and Python code. Next, we introduce novel approach for code refinement within CAD code generation, called CADCode which enables a VLM to validate and correct its generated object to address any errors in the output. We compare M.DCodeVeriffy to two other relevant approaches for code refinement in CAD code generation, highlighting the strengths and limitations of each method. Our approach represents substantial improvement over previous CAD code generation methods, which primarily relied on manual human feedback through language interactions with VLMSs.
10

---

Published as a conference paper at ICLR 2025
9 ACKNOWLEDGMENT
This work was supported by the National Science Foundation under grant number CMM1-2229260.. We extend our gratitude Jamie Adams, a CAD design expert, for annotating our dataset using Python code.
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. International conference on machine learning, pp- 40-49. PMLR, 2018.
Kamel Alrashedy, Vincent J. Hellendoorn, and Alessandro Orso. Learning defect prediction from unrealistic data. Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering, 2024.
Mario Beauchemin, Keith P. Thomson, and Geoffrey Edwards. On the hausdorff distance used for the evaluation of segmentation results. Canadian Journal of Remote Sensing, 24:3-8, 1998. URL
https://api. www.ntiiscoolllattt.eeeeiaataaae htt.cccchhha..ooooooooooossoss/e
13813367
Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pp. 586 186-606. Spie, 1992.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam Girish Sastry, Amanda Askell, Sandhini Agarwa, 1 Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Clemens Winter Jeffrey Wu, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020. doi: 877-1901.
Hailin Chen, Amrita Saha, Steven Hoi, and Shafiq Joty. Personalised distillation: Empowering open-sourced Ilms with adaptive learning for code generation. arXiv preprint arXiv:2310.18626. 2023a.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. 2023b.
Lance Flavell. Beginning blender: open source 3d modeling, animation, and game design. Apress, 2011.
Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 103-118, 2018.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma. Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with molti-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.0066 00615, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307. 12981, 2023.
Diederik Kingma and Max Welling. arXiv:1312.611 2013.
Auto-encoding variational bayes.
arXiv preprint
11

---

Published as a conference paper at ICLR 2025
Milin Kodnongbua, Benjamin T Jones, Maaz Bin Safeer Ahmad, Vladimir G Kim, and Adriana Schulz. Reparamead: Zero-shot cad program re- parameterizztion for interactive manipulation. 2023.
Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. Salad: Part-level latent diffu- sion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14441-14451, 2023.
Sachin Kumar, Gopi, N Harikeerthana, Munish Kumar Gupta, Vidit Gaur, Grzegorz M Krolczyk, and ChuanSong Wu. Machine learning techniques in additive manufacturing: state of the art review on design, processes and production control. Journal of Intelligent Manufacturing, 34(1): 21-55, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12642-12651, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304 08445, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
Liane Makatura, Michael Foshey, Bohan Wang. Felix HähnLein, Pingchuan Ma, Bolei Deng, Megan Tjandrasuwita, Andrew Spielberg, Crystal Elaine Owens, Peter Yichen Chen, et al. How can large language models help humans in design and manufacturing? arXiv preprint -xXiv:3307.1337. 2023.
Matt D Nelson, Brady L Goenner, and Bruce K Gale. Utilizing chatgpt to assist cad design for microfluidic devices. Lab on Chip, 23(17):3778-378, 2023.
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint ar arXiv:2201.11233 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763. PMLR, 2021.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204. 00498. 2022.
Juergen Riegel, Werner Mayer, and Yorik van Havre. Freecad. Freecadspec2002. pdf. 2016.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer. Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. 2023.
MMM Sarcar, K Mallikarjuna Rao, and K Lalit Narayan. Computer aided design and manufacturing. PHI Learning Pyt. Ltd., 2008
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1433-1443, 2020.
12

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. th-ouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24885 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning self-correct. arXiv preprint arXiv:221 1.00053, 2022.
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer- aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6772-6782, October 2021.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 179-1189, 2023.
Xiaoliang Yan, Reed Williams, Elena Arvanitis, and Shreyes Melkote. Deep dearning-based semantic segmentation of machinable volumes for cyber manufacturing service. Journal of Manufacturing Systems, 72:16-25, 2024. ISSN 0278-6125. 25. doi: https://doi.org/10.. 1016/j :osy...111111m1..m22.1111... 2023.1 URL
https://ww/ www.sscciinnede...ooom wttp://tc/ccccm/ccc////c//////ae http://eeearrriicee////s/e/e/eee ottpe//eeooocooooooooooooooooo// 1523002225
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEF/CVF international conference computer vision, pp. 4541-4550, 2019.
Alf Yarwood. Introduction to AutoCAD 2004. Routledge, 2013.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19313-19322, 2022.
Zeqing Yuan, Haoxuan Lan, Qiang Zou, and Junbo Zhao. 3d-premise: Can large language models generate 3d shapes with sharp features and parametric control? arXiv preprint arXiv:2401.0643 2024.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06977, 2022.
Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit- Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive Language-image- pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15244-15253, 2023.
Shen Zheng, Jinqian Pan, Changjie Lu, and Gaurav Gupta. Pointnorm: Dual normalization is all you need for point cloud analysis. In 2023 International Joint Conference on Neural Networks (IJCNN, pp. 1-8. IEEE, 2023.
A ETHICS DISCUSSION
As our approach utilizes VLMs, need to be cognizant of the potential pitfalls of utilizing these approaches. VLMs are capable of hallucinations to produce context that is unrelated or unhelpful to the desired context. These hallucinations may also include harmful propagation of the inherent
13

---

Published as a conference paper at ICLR 2025
stereotypes within the datasets utilized to train these models. While these concerns important to note for any VLM- baM-daaeddae approach, the potential downstream impact on users via our approach is minimal. Since approach is employed to produce 3D objects, there is limited harm which may be incurred by any potential hallucinations via our approach. However, a malicious user may choose to leverage method to generate 3D object of weapon harmful item such guns, knives, etc. While there are many steps between the process of generating design and procuring the item, we encourage readers to exercise caution in identifying and reporting any misuse of this approach.
B ADDITIONAL METHODOLOGY DETAILS
B.1 WHY CADQUERY?
CADQuery is one of several open-source CAD scripting languages, alongside tools like FreeCAD and OpenSCAD. Previous work has primarily used OpenSCAD for CAD code generation. However, we chose use CADQuery as our parametric CAD programming language instead of OpenSCAD, which has been the dominant language for generating 3D objects. Our decision to switch to CADQuery was based on two key reasons: (1) CADQuery built in Python, making more suitable for L.M-ggeneraedd code, given the vast amount of Python code available online. (2) CADQuery's "design-intent" approach allows it to generate more concise code for complex objects compared to OpenSCAD.
B.2 GEOMETRIC SOLVER VERBALIZATION
As intermediate step, utilize an LLM to verbalize this feedback, offering detailed insights into how the generated design d, differs from the ground truth design dge. Verbalization has been demonstrated to enhance the model's understanding and response accuracy (Madaan et al.) 2023).
B.3 EXPERIMENTAL PARAMETERS
We performed the experiments using GPT-4 ("gpt-v4") via the OpenAI API and Gemini ("gemini- -s-llssh-latess") through the Google API, with the temperature set to 0 for code generation and refinement. In cases where the generated code had bugs or failed to compile, we resubmitted both the code and the compiler етror message to the model, adjusting the temperature to 1. For CodeLlama B70, we utilized the Replicate A setting the temperature to 0.8 for code generation, refinement, and bug fixing. Other hyperparameters, such as top_k 10, top_p 0.9, and repeat_penalty = 1.1, were kept at their default values. The total cost for running the experiments was approximately $1000: $450 for GPT-4, $5 for Gemini, and $150 for Lada Experiments JAN to 15 FEB, 2024, and May 15 to August 15, 2024. In all our experiments, we set the number of refinements 2, improvement
conducted from 06
observed beyond the second refinement. This setting
is consistent with prior work on refinement for code generation (Madaan et al. 2023 Chen et al. 2023a).
To compute distance measures, we converted the generated STL files into point clouds rendered with 1000 points. We used the Open3D and Pandas libraries to calculate the Point Cloud Distance.
B.4 FEW-SHOT PROMPT
We design a few-shot prompt enable VLMs to adeptly perform CAD code generation. The prompt, p y1 ( ./... @ yk, where the is CAD code and is comprised of a set of k examples. We can formulate the few shot learning as of code samples, D, collected from the CADQuery documentation with total of 40 examples. We include a text-description of the object or additional comments describing the code, in each code snippet, if it is available in the documentation.
Each example, (y:) is sourced from a memory
hhups//treplicaee.co/
14

---

Published as a conference paper at ICLR 2025
B.5 GEOMETRIC SOLVER FEEDBACK
An example of the geometric solver's feedback for both generated and ground truth 3D objects is shown in Figure
"Ground_Truth"
Number_of_Faces" 140,
Linier_of__nqqueeeee 210, "Number_of_Verticee"" 72,
"Kidth_am": 1.5, "Height_mm": 0.44,
"Depth_am"; 1.5,
"Bounding_Box_Gentee"" "Vector (0.0, - 20044551256724403 0.0)",
"Aspect_Ratio_X'"" 3.4,
"Aspect_Ratio_X22: 1.0,
"Aspect_Ratio_yV2 0.29,
13
"Volume_cubic_mm" 0.78,
14
5.59
15
},
16
"Generated": {
17
ofaeer_____Faa"s"e 509,
18
cfubber_o_uniqucc...
750,
19
NNumber_of_ 252,
20
"Kidth_am": 200.0,
21
"Height_mm": 199.94,
22
"Depth_am"; 5.0,
23
"Bounding_Box_Genter" "Vector (0.0, 0.0, 2.5)",
24
"Aspect_Ratio_xX": 1.0,
25
"Aspect_Ratio_X2": 40.0,
26
"Aspect_Ratio_YZ". 39.99,
27
"Volume_cubic_nt 157014.44,
28
"Surfacc_Aree_qqurre_m""" 65947.11
29
30
Figure 6: This figure presents example of the feedback generated by the geometric solver, which calculates the geometric properties between the generated and ground truth 3D objects, requiring access the ground truth.
C QUALITATIVE ANALYSIS
We conduct a qualitative analysis on the outputs of CADCodeVerify on a randomly selected set of 50 examples to provide further insights for salient questions: (1) What types of feedback does CADCode Verify generate? (2) What kinds of errors are present in the generated 3D objects?
C.I TYPES OF FEEDBACK GENERATED BY CADCODEVERIFY.
The feedback produced by CADCode Verify is designed to enhance LLMs in refining the generated 3D objects. To understand the nature of the feedback generated by CADCodeVerify
conducted
a qualitative analysis and manually categorized the feedback into three main types: (i) Structural Feedback: the feedback is correct the structure of the object (e.g., "make cylindrical or adjust corner shape); (ii) Dimensional Feedback: the feedback is an instructions related to size and scale of objects (e.g., increase height and reduce width); and (iii) Positional Feedback: the feedback focuses on the alignment of the objects (e.g., center object and align with base). illustrated in Figure [ the percentage Structural Feedback for generated objects starts 52.0% in Refine and decreases to 38.0% in Refine 2, demonstrating CADCode 3D objects. Meanwhile, Dimensional Feedback increases from 20% to 26%. likely due to some previously resolved structural errors being recategorized as dimensional issues. In future work, aim to explore the impact of feedback types and investigate the extent to which LLMs can refine objects based on the specific type of feedback provided.
ability to correct structural errors in the generated
15

---

Published as a conference paper at ICLR 2025
Refine
Refine
4.0%
4,0%
16.0%
24.0%
38.0%
52.0%
8. 0%0%
20.0%
Structural Feedback
No Feedback
Dimensional
Failure:
Positional
Figure 7: Analysis of the types of feedback produced by CADCode Verify.
Distances Generated and Refined 3D Objects for Each Error Type
Distribution Types Generated Objects
0.25
Generated 3D Objects Refined 30 Objects
12.0%
18.3%
Point Cloud
0.35
Structural Configuration Error
Correct
0.00
Structural
Logical Error Spatial Precision Error
Spatial Error
Configuration
Logical Error
Error
Error Types Generated 3D objects
Figure 8: Analysis of errors in the generated and refined 3D objects by CADCode
C.2 ERRORS ANALYSIS
We conduct in-depth analysis categorize the types of errors present in the generated 3D objects and evaluate how much CADCode Verify improves these objects in terms of Point Cloud distance. Following the approach in (Yuan al.) 2024), identified five types of errors: (i) Structural Configuration Error: errors where the structure of the 3D object is incorrectly arranged; (ii) Spatial Precision Error: minor error related to spatial parameters (e.g., height, width, and volume); (iii)
Logical Error: implausible configurations of 3D objects that do not resemble real-world contexts; (iv) Correct: objects without errors; and (v) Failure Rate: objects that failed generate due compile error. To identify these errors, three annotators independently categorized them, with the final annotation determined by majority vote. Figure S
illustrates the findings: pic chart shows that the largest proportion of errors 48% are due to Structural Configuration Errors, followed by Logical Errors at 18%. Additionally, a bar graph compares the Point Cloud distance for generated and refined 3D objects across each error type, highlighting the improvements achieved by CADCode Verify.
D ALL THE RESULTS
16

---

Published as a conference paper at ICLR 2025
Table 5: We present our results for GPT-4 using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and compile rates. Table 2 reports the Best Refine results. The * symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.
Model
Feedback Mechanism
Iterations
IoGT
Point Cloud dist.
Hausdorff dist.
Compile Rate
Generated
0935
(0:146)
Refine
0.34 (0.035)
0.164
0.47% (0 3833
91.5%
3D PDePrmm
Retine 2
0.935 (0.037)
0,120 (0.171)
0.493 (0.436)
89.5%
Best Refine
0.939 (0.024)
0.150 (0.143)
0.480 (0.372)
91.5%
GPT-4: Zero-shot
Refine I
0.93 (0.032)
0.146 (0.
0.491 (0 (63662)
94.0%
CADCodeVerify (Ours)
Refine
0.936 (0.02)
0.159
0.497 (0.389)
(0.024)
(0.354)
Refine
0.39 (0.039)
0.119 (0.170)
0.427 (0-470)
91.5%
Geometric salver
Refine 2
0.91 (0.047)
0.125 (0.172)
01439 (0.457)
91 11.0%
Best Refine
0.943 (0.037)
0.102 (0.159)
0.378 (0.434)
91,55
Gonerated
0.939 (0.020)
0.155 (0.140)
0.494 (0 (0.368)
96.0%
Refine
0.937
0.1%
0.486 4344
Refine
(0.029)
(0.192)
(0.414)
GPT-4: Fewesbot
Best Refine
0.942 (0.033)
0.137 (0.155)
0.446 (0.396)
91.0%
Refine I
0.94 (0.030)
0.147 (0.148)
0.470 (0. (73578
96.5%
CADCodeNerity (Ours)
Refine 2
0.941 (0.028)
0.157 (0.139)
01460 (0 3655
95.55%
Best Refine
0.444 (0.028)
(0.356)
0.938
0.120
(0:126)
Goometric solver*
Reline 2
0.91 (0.013)
0.110 162)
0/432 (0-448)
94.5%
Best Refine
0.944 (0.031))
0.103 (0.152)
0.399 (0.433)
95.5%
Table 6: We present our results for Gemini using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and compile rates. Table 2 reports the Best Refine results. The symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.
Model
Feedhack Mechanism
Iterations
loCT
Point Clond dist.
Hausdorff
Complle Rate
Generated
(0.088)
0.159 (0.180)
(0.451)
Retine_
0,93 (0.083)
0.157 (0.192)
0.54] (0.431)
83.5%
3D-Premise
Refine 2
0.906 (0.08))
0.163 (0.190)
0.545 (0.440)
83.5%
Gemini: Zero-shot
Best Refine
0.911 (0.079)
0.150 (0.180)
0.496 (0.431)
83.5%
Reline
0,909 c0.091)
0.162 (0.188)
0.527 (0.479)
84.5%
CArCCiee (Ouns)
Retine
0.152
5%
(0.165)
(0.083)
Geametric salver?
Refine 2
0.910
0.189 (0.203)
0.408 (0.499)
82.5%
Best Refine
0.917 (0.070)
0.113 (0.188)
0,436 (0.458)
83.5%
Generated
0,33 (0.061)
0.171 (0.174)
0.521 (0.426)
85 (79
Refine_1
0.944
0.180
0.555 (0.472)
81.5%
3D-Presnise
Gemini: Few-shot
(0.070)
0.169 (0.184)
(0.516)
81.55
Refine_
0.935 (0.050)
0.166 (0.173)
0.541 (0.408)
85 0%.0%
CADCodeVenis (Ouns)
Retine 2
0.902 (0.052)
0.178 (0.187)
0.543 (0.413)
83 0%%%
Best Reline
0.939 (0.052)
0.147 (0. (66)
0.492 (0.358)
85 $5.0%%
Refine_
0.938 (0.067)
0.124 (0.193)
0.433 (0.452)
Geumetric solver
0.124
83.5%
(0.060)
0.104 (0.146)
(0.470)
17

---

Published as a conference paper at ICLR 2025
Table 7: We present our results for CodeLLLama using the median Point Cloud distance, Hausdorff distance, and Intersection over Ground Truth (IoGT), along with the interquartile range (IQR) and compile rates. Table 2 reports the Best Refine results. The symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.
Model
Feedback Mechanisus
Iterations
InGI
Point Cloud dist.
Hausdorff dist.
Complie Rate
Generated
0.92 (0,943)
0.237 (1.591)
0.731 (1.270)
64.5%
Refise_1
0.925 (0:349)
0.223 (1.59))
0.685 (1.281)
70.0%%
Zero-shox CodeLiana
CADCudeVerify (Ours)
1.730 (1.553)
Best
(0:95)
0,211 (1.599)
0.641 (1.309)
70.0%6
Refine
(0.930)
0.286 (1,590)
0.794 (1.20)
55.5%
Geometric solver
Retine 2
(0.915))
(1.47))
1.730 (0.946)
30.516
Best Refine
0.888 (0:941)
0.280 (1.595)
0.823 (1.258)
55.5%
Generated
0.927 (0.949)
0.224 (1,597)
0.657 (1.294)
67.0%
Refine
0.928 (0: (0.950)
0.212 (1.608
0.630 (.434)
73.5%
Rew-shot Codel lame
CADCodeVerily (Ouns
Retine
0.924 (0.946)
0.260 (1.591)
0.714 (1.290)
65.0%
Best Retine
(0:957)
0.185 (1.620)
0.582 (1.366)
Retine_
(0.915)
0.250 (1.601)
0.765 (1.296)
Geometric soliver
Retine 0.000 (0.889)
1.230 (1.474)
(1.011)
31.0%5
Best Keline
0.906 (0.949)
0.239 (1.606)
(1.300)
60.516
Table 8: We present our results for the GPT-4 few-shot setting, stratified by object complexity (simple and complex), as discussed in 44.3 The results are reported using the median and interquartile range (IQR). The # symbol indicates that the geometric solver accesses the ground truth to compute the geometric differences between the ground truth and the generated 3D object.
Complexity
Feedhack Mechanism
loGT f
PLC distances L
Hausdorff dist. 1 Compile Rate
Generated
0.941 (0.017)
0.168 (0.174)
0.373 (0.323)
100.0%
Simple
3D-Premise
0.943 (0.018)
0.114 (0.112)
0.310 (0.298)
100.0%
CADCodeVerify (Ours)
0.944 (0.026)
0.1 144)
0.329 (0.291)
100.0%
Geometric solver
0.953 (0.021)
0.035 (0.075)
0.081 (0.336)
100.0%
Generated
0.936 (0.052)
0.146 (0.109)
0.465 (0.392)
97.4%
Moderate Complex
ID-Premise
0.942 (0.047)
0,102 (0.107)
0.341 (0.331)
100.0%
CADCodeVenily (Ours)
0.943 (0.035)
0.114
0.416 (0.332)
Goometric solver
0.937 (0:049)
0.085 (0.082)
0,311 (0.319)
97.4%
Generated
(0.024)
0.157 (0.184)
0.504 (0.387)
Complex
3D-Premise
0.939 (0.029)
0,154 228)
0.469 (0.427)
87 87.4%
CADCodeVerily (Ours)
0.942 (0.022)
0.412 (0.297)
"Goometric Solver
0.3486 (0:027)
0.110 (0.1159
003429 (0:44)))))))
94.3%
Generated
(0.038)
0.165 (40)
0.541 (0.273)
Very Complex
3D-Premise
0.946 (0.030)
0.156 (0.198)
0.498 (0.478)
87.7%
CADCodeVerify Verify (Ours)
0.954 (0.039)
0.144 (0.123)
0.475 (0.359)
94.7%
Geometric solver*
0.943 (0.047)
0.114 (0.152)
0.465 (0.330)
94.7%
Table 9: Accuracy of answers generated by CADCodeVerify on subset of 50 examples in the GPT-4
few-shot setting.
Answers
Refine
Refine 2
Description
Total answers
219
176
Total number of responses evaluated
Correct answers
64.6%
68.2%
We evaluate each answer as either "Yes" or "No" and calculate the percentage of these responses, including incorrect and "Unclear" answers. For Refine I: Out of 214 answers labeled "Yes" or "No," only were incorrect, resulting an accuracy rate 91%. For Retine 2: Out of 176 answers labeled as "Yes" or "No," 22 were incorrect, yielding an accuracy rate of 87.5%.
Incorrect answers
8.8%
12.5%
The percentage of incorrect answers where the CADCodeVerify response was "Yes" "No," but the answer incorrect.
Answers with "Unclear"
26.6%
19.3%
minimize hallucinations from the LLM, we instructed it to re- spond with "Unclear" when it lacked confidence in its answer. Any response labeled as "Unclear" was sent back to the LLM during the refinement phase for further evaluation.
18

---

Published as a conference paper at ICLR 2025
Python using caaquery cylinerical object with first, create large, with 1/8th
cylinder. Next, treate diareter as that of the cylinder, it should be at the center of the cylinder going all the way through. I finelly, create three even smaller holes, in triargular formation, around the first hole in the cylinder.
Question
Question Generation Pronpt
the object cylindrical
the object cylinder's diasster?
wey through
cylinder?
Are there additional arranged triangular forration around central hole?
5. Are these three holes smaller than the central holc?
Question Answering
Question Answoring Pronpt
"Answor:"*
**Reasoning images dbject with circular top view consistent height, which are characteristiie cylindrical shape.
**Ansver:"* Unclear
"Reasoning: ** while the obfect does have contral hole, the esact proportions the hole reletive the cylinder' is dianeter alone cannot be determined froe without specific
"Ressoninn:"""
images, botton vieas, indicating passes completely through the cylinder
fron
**Amsarr** No
*"Ressoning:*** ** Kone of the teages show any additional holes besides the central hole the surface of the cylinder.
"*Answer:" Since holes present, applicable.
additional
Feedback Generation
Prospt
Based the answers provided, following corrections need to be mode to the 30 ooject:
1. The central hole's diarwter needs to be adjusted to be 1/8th of the cylinder's 's diaeteer. This currently uncicar and needs to be mode sore apsarent.
Three additional holes need to be added around forsation. central hole in eissing. these additions] should
currently
present, change needs imploxented.
Figure 9: This figure provides an example (1) of one step of CADCodeVerify, via GPT-4V, for prompt from CADPrompt
19

---

Published as a conference paper at ICLR 2025
Write Python code using cloquery to create a 30 rectangular object, with a smaleer rectangular slat vithin it. First, create sketch of rectargle and slightly extrude it, lext,, create a sketch of seoller rectargle, around half the size of the original sketch, at the center the original sketch, and negatively extrude Negatively
sealler than rectangular object.
within
Quastion Generation
Question Generation Prompt
1. Is the object rectangular in shape?
2. Does the object have a smaller rectangular slot
kithin it?
3. Is the slot located at the center of the object?
Is the depth of the slot less than the depth of the
original object?
Is the size of the slot approxinatall half the size the original object?
Question Answering
Question Answering Prompt
Ansswrr:
**Reasoni ig: The object appears have hexagonal. shape when viewed from the top, as seen in the isages. The sides are not perpondicular to each other, which is a characteristic of rectangular shapes.
*Answrr a* Unclear
**Remsoning: ng: ** from the provided images, there is visible
Indentation that clearly solid object without any
identifled
openings.
"Answer: Unclear
**Baasoning:* Since
unclear whethar slot kithin the object from provided images, it is also unclear whether such sist, if it exists, is located at the center of the object.
*tnnwer e* Unclear
"Reasoninng e As there is no vistble slot in the Images, it is not possible to detersine the depth slot in relation the depth of the object.
Unclear
visible
images, the conparison
object.
Feedback
Generation Proept
answers provided, folloxing object:
corrections need
1. Change the shape of the object fros hexagonal to rectanguiar. Ensure that all sides are persendicular to each other to achieve a rectangular shape.
2. Incorporate rectangular slot within the object.
This slot should be clearly vistble.
a. Position the slot the center of the dbject.
Ensure that depth the slot less then depth
object. be approximately half
original object.
Figure 10: This figure provides an example (2) of one step of CADCode Verify, via GPT-4V, for prompt from CADPrompt
20

---

Published as a conference paper at ICLR 2025
Prompt for generating questions.
You will be given description of how human-desigue would describe the design of a 3D object. Your job is to provide between 2-5 (Yes or No) questions that can use to verify how similar the generated object is to the description generated by human. The questions should be framed such that answering "No" implies that there is a change that needs to be made the object regarding the verification question. Here are some important points note for this task;
(1) Do make up questions you cannot generate questions based on the description provided.
(2) Ensure that your questions only reference entities mentioned within the description.
(3) Try not reference orientation the components of the 3D object. Your generated questions should not ask whether component on the "right" or "left" side as this orientation is relative.
will give you two examples with a language description followed by the appropriate verification questions. Please reference these examples while generating your verification questions.
### Example ###
Description:
Extrude a cylindrical plate with rectangular hole in the middle of it.
Generated Questions:
1. the object cylindrical shape?
2. Does the object have rectangular hole in the center?
3. Is the object extruded in one dimension?
### Example 2 ###
Description:
Design a 3D object that resembles a cone. First draw a sketch of a square and extrude to create the base of the cone. Next, draw sketch of circle centered at the center of the square base. Extrude this sketch vertically into conical shape, such that the diameter of the circle decreases the height increases. Finally cutout the tip of the cone, such that the tip of the cone is now rectangular in shape.
Generated Questions:
Does the object resemble cone?
2- Is the base of the object square-shaped?
Is the circular base of the cone centered at the same point as the center of the square base?
4- Is the tip of the cone rectangular?
5- Does the diameter of the cone decrease as the height increases?
Figure 11: In the CADCode Verify approach, we utilize this prompt with two examples to generate the verification questions.
21

---

Published as a conference paper at ICLR 2025
Prompt for code repair.
You will be provided with piece of Python code and compiler error message, and then your task will be to fix the bugs and rewrite the code.
#### Compiler error messages:
line 20: citiicees..illeeeuutttt_t__eee
raise ValorE
ValucError: Cannot find solid on the stack or in the parent chain
#### Python Code:
import cadquery as eq
# Dimensions
plate length 100
plate_ width = 50
plate_thicknen thickness 3
cutout_sizz sizeetsszeeeeee 15
rrnbuusradiuu 5
# Create the base plate plate (cq. Workplane Y"X"") rect(plate length, plate width) extrude(platee_ (ateeeekktttthcceessss)
# Create the cutout sketch with rounded corners cutout_ sketch (cq. Workplane("XX"" mnoveTo(cutout_size// (itouutsiize2)) rect(cutout size, cutout_sizz forConstruction-TTre) pertices()fillle(ctttoo) (rddius)
# Cutouts two corners cutout! pplate.workplang
placeSketch(cutoutt (cctooussstttthhh extrude(-plate (hiikknssshhhhkssss
cutout2 (plate.workplann workplanc()
work _enetteeenhh_hhh - cutout_size) size)
placeSketch(cutoor,, _ketch)
(rrrude(platt__tiickesss))
# Combine the base plate and cutouts final object plate.cut(cutoouu. cut(cutout2)
# Export the result to an STL. file
# This line caused the error. ".val(" is removed final final_object.exppoo oboeecc.eppoorSsll"""" (atricc.ecppoSs(("Geeeeeeeedddl
Figure 12: If the generated CAD code fails to compile due to a compiler error, we pass both the error and the generated CAD code LLMs for correction.
22

---

Published as a conference paper at ICLR 2025
Prompt for generating answer.
Your job is to answer this of questions with respect to the object have shared with you. I will be providing 4 images of the object from different orientations so that you can get a complete picture of the 3D object. Here are some important points to note regarding your task:
(1) Remember that these images are all the same object from different angles.
(2) The answer to each of these questions should always be one of three options which are "Yes" or "No" or "Unclear."
(3) Your answer should be "Unclear" in situations where you unsure of the answer do not have enough information to answer the question.
Make sure provide reasoning supporting all your answers.
# Your answer should follow the same format as below:
1. *uuestion?**
*
Reasoning::*
2. **Qestton?***
** *
Reasoning::*
Figure 13: In the CADCodeVerify approach, use this prompt along with images generate the answer and reasoning for each question.
Prompt for generating feedback.
These were the answers to the questions I asked to validate generated 3D object. Can you utilize the answers these questions to generate actionable feedback that will help the model to correct the mistakes in the 3D object? Your job is to summarize these answers into practical corrections that need be made the 3D object. Please note the following while generating your feedback:
(I) The corrections should be such that the answers to all questions provided will become yes upon applying the suggested corrections.
(2) Your corrections should not change the object such that any of the answers that are already, "Yes" become "No."
(3) You only want to change the object such that the answers which are "No" or "Unclear" become "Yes." " The summary should be specific and only few sentences long.
(4) Your corrections should not be regarding the quality or orientation of the images.
(5) Your feedback should not attempt to fix issues in the scale. DO NOT ask for the addition
of additional scale or reference objects.
(6) Do not ask for details regarding the size or dimensions of the object.
(7) Your corrections should be constructed such that human designer can use your feedback
to update the object such that all questions have "Yes" as the answer.
Figure 14: In the CADCodeVerify approach, we use this prompt to generate feedback from the generated question-answer set.
23

---

Published as a conference paper at ICLR 2025
Total number of compiled code for each object across experiments
Easy
Hard
Each example (generated code) sorted descending order by difficulty.
Figure 15: We conducted six experiments across three LLMs-GPTE Gemini, and aodeLlama-and both zero-shot and few-shot settings for CADPrompt, which contains 200 examples. Some of these experiments did not generate compiled code that produced valid 3D objects for some of the examples. We calculated the total compiled code for each example across all experiments, where Max 6 means the example was generated in all experiments, and Min 0 indicates that the example was not generated by any experiment. We then sorted them based on difficulty and split the dataset into "Easy" and "Hard" categories discussed in 44.3
24

---

Published as a conference paper at ICLR 2025
Write Python code using CADQuery to create a triangular 3D object. First, draw sketch of an equilateral triangle, pointing downwards. Next, cutout a semicirele from the bottom corner of the triangle. The diameter of this semicircular cutout should be approximately 2/3rd of the length of each side of the triangle. Finally, extrude this sketch to create 3D object.
(a) The natural language descriptions of 3D object.
(b) 3D object
import cadquery cq from typing import List, Tuple
lengch:float 8.929516
fldat:float 0.54
height: Float = 8.3
top_lemgth:float 1.5
fop_x: (length top_length)/2
poinss::List(tullllllltttt fleat]l (0, 8). (length, 6).
(top_x topgtength, width). (top_x, width).
Norkplane ( eq Norkplane ("X2") .enter (-length/222 wiidhh/2) polyyiieeppintssscclos
part:eq Norkplane (heighh) translate ((0, height/2 width 12))
radius (length/2) 48.014
ellipse cq. Workplane ("X2"), "xxk"".ciire" circci circie exxtu ((0, height/2.))) cut (ellipse, wra.pipss...aattteeeee.e... (ranslste width 12- 22-radius 18)))
extrude (height) translace
<q. www.rters.ss...o expor (part, Bode__Ground_Trus Truth.stl')
(c) Python Code
Figure 16: An example from the CADPrompt dataset, showing (a) the prompt, (b) the corresponding 3D object, and (c) the auman-annotated Python code used to generate the 3D object.
25

---

Published as a conference paper at ICLR 2025
Write Python code using CADQuery create inverted desk, with hexagonal legs. First, draw a rectangular sketch, and cutout four small, right-angle triangles from each corner the rectangle create octagonal surface. Next, extrude this sketch by small amount. Draw two large, regular hexagons, placed symmetrically on opposite ends of the octagonal surface. The hexagons should be the same size and should align perfectly with each end of the surface. Finally, extrude these hexagons outwards to give the appearance of two hexagonal columns protruding from horizontal surface.
The natural language descriptions of the 3D object.
(b) 3D object
inport cadquery as
2.333313
spacer = 155128
sides 6
height 0.75
hexes
ca. Sketch .push ssidhhsppceer -0.61863))
14
15 hex. base =
Sketch()
[ (I(., (riidssaeee -0.618963)3)
15
-regular Polygon(diameeter sices) .wires
1
-ull
13 hexagans ca, Werkplane(("XY') placesketch (hexes) (hatgd)))aaggat)
25
base (e.Kerkplare""""")
-placesketch(her base)
32
31
cq. Morkplaree""""") .union (hexaggas))) (base)
37
part part ratata((a, (3, (8,0,8) (0, translate ((80. 339552, -8, 3438322,
(part. Grouhd__rruth.st]
(c) Python Code
Figure 17: An example from the CADPrompt dataset, showing (a) the prompt, (b) the corresponding 3D object, and (c) the human-annotated Python code used to generate the 3D object.
26

---

Published as a conference paper at ICLR 2025
Lamp
(b) Rectangle cut semi Circle
(c) Shelf
(d) Bobbin
Figure 18: Comparison of the ground truth 3D object with the 3D) object generated by Python code
written by a human CAD design expert.
27


Reviews:
[{"review_id": "dhe9mMCJ6e", "reviewer": "Reviewer_jRaW", "summary": "The paper entitled “GENERATING CAD CODE WITH VISION-LANGUAGE MODELS FOR 3D DESIGNS” proposes an approach to generate and verify CAD 3D objects from natural language. The proposed approach introduces a verification step called CADCodeVerify. CADCodeVerify use Vison Language Machine (VLM) to verify CAD objects against generated visual questions. The hypothesis is that the question serves as visual properties the object should satisfy to meet the structural specification. The key contribution of this approach is the use of a refinement step. From the language prompt that describes the object in natural language, a first CAD description code (CADQuery) is generated. The visual questions are also generated from this language prompt input. The object (CADQuery) is verified against visual properties questions using a VLM that generates feedback for each questions. This feedback is used to refined the CADQuery code.\n\nTo summarise, the contributions of this work is two fold: 1) an automated CAD generation approach that does not necessitate humain interaction; 2) a dataset called CADPrompt that comprises 200 CAD objects. Expermental results shows when implemented using GPT-4 LLM, we observe improvements in object generation accuracy and an increased success rate compared to other LLMs.", "questions": "- Q1: I appreciate the framework’s contribution, but could you provide a bit more on how this specifically differs from or builds upon prior work (3D-Premise)? Is it an improvement or an entire new approach?\n\n- Q2: Can you tell us more about the quality and diversity of the CADPrompt dataset? I’m curious about how it’s tailored to the goals of your approach. A bit more on why it’s a good benchmark—perhaps a breakdown of object types, complexity levels, or any specific challenges it presents?\n\n- Q3: The experimental results look interesting, but it would be good to understand more in details where the approach performs well and where it might have limitations. Do you have any insight to share about this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novelty and Originality: The integration of question/answering based VLM to refine the object quality. \n\n- Evaluation: The increase of success rate of compilable outputs at the end of the refinement shows that the method can improve the generation of CAD code from language prompt.\n\n- Soundness: The technical approach is sound and could be applied to a lot of CAD or CAV tasks. The approach is a specification refinement method using generative AI. From an initial specification, an initial object plus query are generated. Then, feedback from query satisfaction are used to improve the object. A dataset is provided and metrics are determined to rank the quality of the generated object. This has applications beyond CAD specifically.", "weaknesses": "- Dataset Scope: CADPrompt could have been better introduce, showing the most complex objects in terms of complexity and difficulty metrics. \n\n- Complexity Metric for Objects: This paper measures object complexity by counting vertices and faces. However, using metrics like bounding boxes or decomposed bounding volumes could better reflect structural complexity. Vertices mainly define shape details, not true complexity—a simple shape like a cube can have many polygons, while complex shapes like an aircraft wing might need fewer. In physics engines, object complexity is often defined by the volume structure the object requires. Lower polygons can shape more complex structures.\n\n- Difficulty Metric for Objects: The current method of measuring complexity through word and line count for the natural language descriptions and Python code may not fully capture their complexity. A semantic-based complexity metric, where each instruction has a complexity value, would better reflect how nuanced or contextually challenging the description and code are, which could impact model performance.\n\n- Clarity and Readability: Certain sections are overly technical and hard to understand without referencing the appendix. Simplifying these sections would improve accessibility for readers. Terminological inconsistencies between LLM used for language to code and code with feedback to code and VLM for image and questions to answers also introduce potential confusion, which could be resolved for clarity. Certain sections are overly technical and hard to understand without referencing the appendix. Simplifying these sections could enhance accessibility for readers. For example, consider moving equations 1-7, which aren't essential for understanding the main contributions, to the appendix. Conversely, evaluation metric equations (8-10), such as those for point cloud distances, Hausdorff distance, and IoGT, should remain in the main text. Additionally, bringing valuable visuals currently in the appendix (Figures 5, 11, 12, and 14) into the main text would ease the understanding and improve work illustration. The paper would also benefit from a stronger focus on scientific insights and hypothesis testing over too much detailled descriptions. For instance, providing context around why a fixed number of five verification questions is used, without adapting to object complexity, would clarify the rationale. Similarly, explaining why only two refinement iterations are applied, regardless of complexity, would better illustrate the scientific intuitions driving these choices. Furthermore, details on how AI context like 7, 9, and 10, have been engineered such as crafted based on expert knowledge or through authors tuning would be valuable. Shifting focus to the \"why\" rather than the \"what\" would help readers grasp the paper challenges.\n\n- Positioning: the paper’s positioning within the state of the art could be made clearer, with more explicit distinctions from existing work. The paper integrates the related work 3D-premise in the experiments, but didn't explicitly state the approach difference. The feedback is based on question answering rather than the initial description of the object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUMNFYjcGW", "reviewer": "Reviewer_Ghys", "summary": "The paper introduces CADCodeVerify, a approach to iteratively verify and improve 3D objects generated from CAD code using Vision-Language Models (VLMs).  The method involves generating CAD scripting code from natural language prompts, executing the code to render a 3D object, and refining the code based on visual feedback through a question generation and answering process, to correct deviations from the initial specifications.  The approach is evaluated using CADPrompt, a benchmark dataset of 200 3D objects with corresponding natural language prompts and expert-annotated code.  The approach is evaluated on GPT-4, Gemini 1.5 Pro and CodeLLama models.", "questions": "1. Would providing images of the generated object from multiple viewpoints improve 3D-Premise’s performance? What about using a chain of thought prompting (for e.g. explicitly ask the model to reason whether each criterion in the initial task are satisfied)? This will be similar to CADCodeVerify, but doing 1 VLM call instead of doing 3 VLM calls in the verify step  to reduce the inference time. \n\n2. In Figure 3, does 3D-Premise also repair code based on compile error messages?\n\n3. How do you handle ambiguity in inputs in ground truth solutions? For example, if the NL input does not specify a particular size of hole, there could be many ground truth solutions which might impact the cloud-distance based metrics. \n\n4. Since Point Cloud distance and Hausdorff distance are noisy metrics, how does the generated CAD program compare on the geometric metrics obtained using the geometric solver?\n\nMinor: \n- Consider renaming success rates to compile rates or something similar to convey that is only compilation success rate and not overall task success rate. \n\n- For figures 5,6 (running examples), it will be useful to also see the repair steps.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper comes with a new benchmark suite (CADPrompt) which is a well-curated, crowd-sourced benchmark with annotations and quality checks, which is a valuable resource for assessing CAD code generation and refinement methods.\n-\tCADCodeVerify uses an interesting novel idea to eliminate the need for human involvement by generating validation questions and answering them using VLMs.\n-\tThe geometric solver-based baseline is very interesting and gives an upper estimate of the self-refinement process; it would be interesting to explore if this solver could also be used as a metric for evaluation.", "weaknesses": "-\tThe fundamental differences between CADCodeVerify and 3D-Premise are unclear. For example, it’s not specified whether 3D-Premise uses execution-error-based repair. Also, both approaches seem to use totally different prompts, so it is not clear if it is just a matter of better prompting or something fundamental (such as the question-answer based method)\n-\tThe paper would be stronger if the approach was also evaluated on the 3D premise dataset\n-\tSome other details are missing (see below questions) regarding ambiguity in NL input and chain-of-thought based extension to 3D-Premise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGrLioN0g8", "reviewer": "Reviewer_8EJ1", "summary": "This work contributes a new method for VLM-based CAD code synthesis, and a new dataset for evaluating CAD programming models. It builds on past work on code repair for CAD synthesis – the core idea is that instead of just providing the current render during repair (as prior work 3D-Premise did), the LLM also comes up with *verification questions* based on the task description (not using the current render) which it then answers (using the current render). These Q/A pairs are summarized into feedback for what needs changing, which is provided to the repair model alongside the current render to prompt code repair.\n\nThe dataset, CADPrompt, was constructed by selecting objects from a dataset used in prior work and annotating them with language prompts along with Python CAD code. They evaluate on a range of models (GPT4, Gemini, CodeLLama) and compare to several baselines (no refinement, 3D-Premise, and Geometric Solver). The last of these baselines, Geometric Solver, is also a contribution – it's a baseline that gets to \"cheat\" and compare the ground truth model to the current rendered model in terms of a range of geometric feaures. Their methods generally outperforms the baselines.", "questions": "- Is the Geometric Solver output given to an LLM during the feedback phase? I assume so given how it varies depending on which LLM is used in Table 2. Does it get to do two rounds of feedback like the CADCodeVerify model does?\n- \"Refine 1\", \"Refine 2\", and \"Generated\" should be defined somewhere, right now the first place they appear is in Figure 2 headings (which are not searchable text in the PDF) and then when discussing results, but they aren't clearly defined.\n    - Is \"Generated\" the step right after Section 3.1 finishes or right after Section 3.2 finishes – i.e. is \"Generated\" before or after the compiler-error based verification step?\n    - Are \"Refine-1\" and \"Refine-2\" the two rounds of refinement (Section 3.3) or is the first one the code repair from Section 3.2?\n- Why does \"Refine-1\" go *down* so much in Figure 3 for 3D-Premise? This is very surprising to me and would be helpful to clarify. Refinement ought to help, and it did help them in that prior work. While it's not as strong of an effect, refinement also seems to hurt all the methods a bit in some of the other plots (like the Complex plot) – why would rounds of refinement hurt instead of help?\n    - These graphs are showing the compilation rate. It'd be helpful to see changes in the other metrics from Table 2 that have to do with correctness instead of whether or not the code compiles – those seem key to understanding whether feedback is helping.\n\nMinor fixes:\n- The CADQuery compiler $\\phi$ is deterministic, right? So the \"$\\sim$\" should be an \"=\" on line 177\n- The description of the \"number of compilable outputs\" discussed in 305-310 was confusing to me until I saw the caption of Figure 11 in the appendix. It would be helpful to move part of this description to the main paper – it wasn't clear to me that this meant the number of experiments where the code compiled successfully. And to clarify – are those 6 experiments the six large rows in Table 2 (that vary between language model used and not between method used)?\n- In discussing the geometric solver the paper notes \"This feedback method serves as an upper limit for CAD code refinement, as it conveys the precise geometric differences between the generated 3D object and the ground truth.\" While I think this is a useful baseline, it's not literally an upper limit for performance from code refinement feedback – as shown for example by how it's occasionally outperformed in the experiments. It's a great baseline to have, I just would suggest against using the term \"upper limit\" for something that's not actually an upper bound.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The contribution of a language-to-CAD dataset is useful for the community, and hiring CAD experts to write the CAD code for it means it's likely high quality.\n- The comparison is run across a range of LLMs including open source ones (CodeLlama) \n- They generally surpass the 3D-Premise baseline and score fairly close to the geometric solver baseline that gets to cheat and use the ground truth CAD model in calculating its feedback. They score particularly well on the more difficult problems, and benefit more from feedback than 3D-Premise. They show improvement across three different similarity metrics in addition to compilation success rate.", "weaknesses": "- The term \"success rate\" is a bit of a misleading name for something that means \"compilation rate\" or \"compilation success rate\" – it gives me the impression that the model succeeded at solving the task, not that it generated an arbitrary piece of code that compiled. For example in the abstract \"increasing the success rate of the compiled program\" or in the intro \"5.5% increase in successful object generation\" all gave me this impression until I dug into it.\n    - Also, the abstract says 5.0% and the intro says 5.5% – which is correct?\n- Does 3D-Premise get to do the Code Execution step described in 3.2, where if compilation fails it retries with compiler feedback up to N times? If not, it seems important to do this comparison to disentangle how much of the benefit is coming from compiler feedback versus the Q/A contribution.\n- The improvements are modest but still reasonable – I would defer to a reviewer more familiar with these metrics on this. Looking at GPT-4 Few-shot in Table 2 (which looks like overall the best model), the IoGT metric difference is .942 vs .944, the Point Cloud distance is .127 vs .137, and the Hausdorff distance is .446 vs .419. These are not huge differences, but still a notable step towards the Geometric Solver performance.\n- It's confusing that \"Refine 1\" causes 3D-Premise to decrease in performance in Figure 3 – see full note in \"Questions\" section, but clarification on this would be helpful to understanding the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VIdS4fp2xD", "reviewer": "Reviewer_9oaM", "summary": "This paper proposes a technique called CADCodeVerify that uses feedback generated from VLMs to refine CAD code generated by LLMs in an iterative manner. This is a novel technique proposed to address a reasonably difficult and seemingly important problem. The paper evaluates this technique against an existing baseline called 3D-Premise and also with a baseline using feedback from a geometric solver. CADCodeVerify shows better performance against both baselines.", "questions": "1. In Fig 1, within the CADCodeVerify block, I see step 3.c generating feedback which then produces a new object. My question is: what exactly is the input to the VLLM? Is it the feedback, that then generates the code that compiles to produce the new object, or is it the new object itself? I suspect it is the former, but the figure indicates it to be the latter.\n2. In the evaluation, do you consider naive or basic prompting? For example, just a zero-shot or few-shot prompt to an LLM that produces code? Is that what the \"Generated\" field in Table 2 is?\n3. Are there any insights as to why CADCodeVerify performs better than baselines? For instance, what is it in the proposed technique that allows it to outperform 3D-premise? And why is VLLM feedback more useful than geometric feedback, even when the latter needs the presence of ground truths? One would imagine a technique needing GT labels for providing feedback would be more accurate or equivalent to a technique operating without the labels.\n4. Maybe I missed this in the text, but in Fig 3, what is Refine 1 and Refine 2?\n5. Are there any limitations as to the complexity and difficulty of the benchmark tasks? I understand they vary in complexity and difficulty, but at the same time CADCodeVerify does quite well, even reaching around 90% accuracy in the hardest case. Is there a class of problems this benchmark is not testing, or are the hard problems simply not hard enough?\n6. How representative is the benchmark of the objects actually being drawn in the industry today? One major challenge with code generation has always been that it has never been representative of IRL code. Do we see the same challenges here?\n7. What dataset was 3D-Premise originally evaluated on, and why have you not evaluated CADCodeVerify over that? What is different about your dataset that sets it apart from the one 3D-Premise used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is written really well. As someone who doesn't work much with CAD code, it was immediately clear what the limitations were and how they intended to address them. The technique itself is novel for the application and is well described with appropriate examples. I also appreciate the effort taken for creating the benchmark tasks over which the systems were evaluated.", "weaknesses": "One main weakness I can see is the lack of baselines. However, this is not a major issue to me since I don't know of other systems that target this specific problem. However, I can think of other techniques like chain of thought prompting, or techniques using some form of reasoning, against which we can compare CADCodeVerify. Right now, the only true baseline seems to be 3D-Premise, and another baseline that uses an alternate form of feedback generation, though I could be mistaken here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)