Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts
Authors: Mohammad Sadil Khan
Sankalp Sinha
Sheikh Talha Uddin
Didier Stricker
Sk Aziz Ali
Muhammad Zeshan Afzal
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Text-to-CAD
Natural Language Instructions
Large Language Models
Parametric Computer-Aided-Design (CAD)
CAD language
Transformer
Primary area: machine_vision

Abstract: Prototyping complex computer-aided design (CAD) models in modern softwares can be very time-consuming. This is due to the lack of intelligent systems that can quickly generate simpler intermediate parts. We propose Text2CAD, the first AI framework for generating text-to-parametric CAD models using designer-friendly instructions for all skill levels. Furthermore, we introduce a data annotation pipeline for generating text prompts based on natural language instructions for the DeepCAD dataset using Mistral and LLaVA-NeXT. The dataset contains $\sim170$K models and $\sim660$K text annotations, from abstract CAD descriptions (e.g., _generate two concentric cylinders_) to detailed specifications (e.g., _draw two circles with center_ $(x,y)$ and _radius_ $r_{1}$, $r_{2}$, \textit{and extrude along the normal by} $d$...). Within the Text2CAD framework, we propose an end-to-end transformer-based auto-regressive network to generate parametric CAD models from input texts. We evaluate the performance of our model through a mixture of metrics, including visual quality, parametric precision, and geometrical accuracy. Our proposed framework shows great potential in AI-aided design applications. Project page is available at https://sadilkhan.github.io/text2cad-project/.

Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts
Mohammad Sadil Khan 112233 1..233
Sankalp Sinha +1,,3
Talha Uddin Sheikh 1,2,3
Didier Stricker1.2.3
Sk Aziz AlI
Muhammad Zeshan Afzal
DFKI
2RPTU Kaiserslautern-Lane
MindGarage
*BITS Pilani, Hyderabad
Abstract
Prototyping complex computer-aided design (CAD) models in modern softwares can be very time-consuming. This is due to the lack of intelligent systems that quickly generate simpler intermediate parts. We propose Text2CAD, the CAD models using designer-
first Al framework for generating text-to-parament friendly instructions for all skill levels. Furthermore, introduce data annotation pipeline for generating text prompts based on natural language instructions for the DeepCAD dataset using Mistral and VLaVA-NeXT. The dataset contains 170K models and ~ 660K text annotations, from abstract CAD descriptions (e.g., generate two concentric cylinders) to detailed specifications (e.g., draw two circles with center {z. y) and radius T2, and extrude along the normal by d.... Within the Tex12CAD framework, we propose an end-to-end caanformer-basee regressive network to generate parametric CAD models from input texts. We evaluate the performance of our model through mixture of metrics, including visual quality, parametric precision, and geometrical accuracy. Our proposed framework shows great potential Al-aided design applications. Project page is available https//sadilkkhan wwt.aaddiiinaniiiih.....aee www.ggt.t.tt.tteeeeeeeee wwtpeeettcadprroppppieeeee
auto-
Introduction
Computer-Ain Aiipueer-Aieed Design (CAD) plays a crucial role in in- dustrial design and additive manufacturing (AM), revolu- tionizing the way products are prototyped [7]. This type of prototyping requires feature- based part modeling [7]. pre-
A ring shape is created by dkawding no concentle choles on the XY respectvaly, extrading along the 7-asis à uni to faom 30 model with hallow center.
cision measurements [40], and creative part editing [60] at different design stages [40, 60]. While CAD software saves the final model as a boundary representation (B-Rep) [22], the design process often involves chain of 2D sketches (e.g., circles, lines, splines) and 3D operations (e.g., extru- sion, loft, fillet) [56, 58, 59, 19J. This representation allows the designers to control the design history and iteratively refine the final models.
bd A rectangular shape
simple rectenguAr longth 0.00 unit and height oni
Despite their capabilities. modern CAD tools lack the AI- assisted design integration [37]. In Figure 1, we illustrate how an intelligent system capable of generating parametric CAD models from textual descriptions can be utilized to as- semble a complex 3D model. Although tools like FreeCAD [1]. SolidWorks [46], and Para-Solid [45] offer 3D CAD models from catalogs like McMaster-Can [2] for the reuse
Figure 1: Designers can efficiently gen- erate parametric CAD models from text prompts. The prompts can vary from abstract shape descriptions to detailed parametric instructions.
Equal Contributions. "Correppondigg author (mohammad.khan@dkk.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of existing CAD models, no such system currently exists that can generate parametric CAD models from textual design descriptions. One primary challenge for developing such a system is defining suitable textual descriptions for parametric CAD generation, making it difficult to create deep learning methods that accurately convert these deseriptions into precise CAD models.
To address this gap, in this paper we propose Tex12CAD as the first Al framework for generating parametric CAD models represented by construction sequences (i.e., parameters for 2D sketches and extrusions) from design-related text prompts. We faced two primary challenges in fulfilling this goal: (1) the unavailability of the dataset and (2) a network to map the texts into CAD construction sequences. Towards this end, we introduce a data annotation pipeline to generate a dataset containing textual descriptions of the CAD models in DeepCAD [56] dataset. We leverage the open-source Large Language Models (LLMs) [16] and Vision Language Models [27, 26] for this task. Our annotated text prompts are multi-level in nature ranging from highly abstract (e.g., a long rectangular shape, a thin S-shaped object) to more specific with detailed parametric descriptions (e.g., first draw rectangle from (I1, y1) and (22, y2)... then extrude the sketch along z-axis.). These prompts are designed for users of all skill levels and can contain arithmetic logic and numerical expressions as part of the design details. Within this framework, we introduce Text2CAD Transformer [49], a network for generating CAD construction language from text prompt in an ruto-segressive fashion.
conditional
Currently, there are works on text-to-3D generation [25, 34, 51, 32, 12] that have shown significant advancements in creating 3D scenes and shapes from textual descriptions. But existing text-to-3D methods are not applicable for generating CAD models from text descriptions as the final output of these models is neither parametric nor human-editable in nature. Very recently web API from 200 developers [4] has introduced CAD generation app using text prompt from users and programmable scripting language (as KittCADDaanuaage )) for designers to edit and modify. However, the generated CAD models are obtained in the form of solid-body, and not decomposed to its intermediate sketch- and-extrusion steps as proposed in our Tex12CAD, On the other hand, Given raw numerical data of any parametric CAD model, current state-of-the-arr large language models (LLMs), such as pre-trained Mistral-50bl [16] or GPT-4 [35] and open source Llama [47] may only derive procedural scripting codes for other APIs, such as FreeCAD [38] or OpenSCAD [30], to generate model. However, in contrast to our Tex12CAD, such LLM augmented CAD generation approach will not be designer-friendly, not suitable for beginner-level designers, will not automate the development process in easy ways, and will restrict the re-usability of the scripts in case of complex shapes. Alternatively, Using offe--of-the-rt vision language models, such LL [27, 26), GPT-4V [61], as an alternative for deducing CAD construction sequences performs poorly because of two main (I) no training datasets are available that provide natural language-baset design instructions as annotations for raw CAD construction sequences and (2) most VLMs are trained on categorical description/caption datasets of 3D objects (e.g., LLaVA-NeXT [26] predicts 'two concentric hollow cylinders 'as toilet paper). We remove the above limitations in our Text2CAD by creating new large-scale annotations for DeepCAD [56] dataset using responses from LLMs and VLMs to train our multi-modal model. Our contributions can be summarized as follows:
We propose Tex12CAD as the first Al framework for generating parametric 3D CAD models using textual descriptions.
We introduce data annotation pipeline that leverages both LLMs and VLMs to generate a dataset that contains text prompts with varying level of complexities and parametric details.
We propose an end-to-end transformer-based autoregressive architecture for generating CAD design history from input text prompts.
Our experimental analysis demonstrates superior performance over the two-stage baseline method adapted for the task at hand.
The rest of the sections are organized as follows: Section 2 reviews the related work in CAD domains. Section 3 outlines our data annotation pipeline. Section 4 details our proposed Tex2CCD transformer architecture. Section 5 presents our experimental results. Section discusses the limitations of our current framework, and Section 7 concludes the paper.
'In this paper, the phrases 'CAD construction language: ", "CAD design history 'and 'CAD construction sequence 'are used interchangeably.
hhttps: / //gi thub. com/KiittyCAD/zzoeliing

---

2 Related Work
Datasets and Generative models for CAD: Current datasets and generative models for CAD are limited and often not suited for developing knowledge-baser CAD applications. Some datasets focus solely on 2D sketch design [43, 10, 44], and other popular datasets like ABC [20], Fusion360 Gallery [55]. Thingi10K [62], and CC3D [6, 9] provide 31) meshes, BRep (boundary representation), and other geometry or topology related annotations that are suitable for 3D modeling. DeepCAD [56] dataset, a subset of ABC, and Fusion360 [55] provide CAD construction sequences in the form of sketch and extrusion to deduce design history. However, CAD models may consist of numerous other types of operations beside extrusion, and such construction sequences with other CAD operations are not available in the current datasets. Finally, there is no dataset available that provides textual design descriptions as annotations to create a conversational Al system for CAD modeling.
Sketch/CAD language modeling (56, 58, 19, 10] filters out unnecessary metadata from lengthy raw design files and represent them as desired sequence of input/output tokens. For instance, Ganin et al. [10] represents design files as messages in Protocol Buffer [48] format. Hierarchical Neural Coding (HNC) method [58] represents the desired design sequence in tree structure of 2D sketch loops. 2D bounding boxes over all loops as profile, and 3D bounding boxes over all profiles as solid. CAD-SIGNet [19] represents CAD construction language as a sequence composed of 2D sketch and extrusion parameters. In Text2CAD method, we map the raw design history obtained from DeepCAD metadata into textual descriptions.
Current supervised learning methods that fall under rqquence-tossquere
CAD Construction Language using Transformers: Transformer- [49] network architecture is the preferred choice for many deep learning-based applications related to CAD modeling [56], 3D scan-to-CAD reverse engineering [19, 23], representation learning [18] and others [39]. CAD as a language [10] describe how 2D sketches can be transformed into design language by sequencing tokens of 2D parametric curves as message passing units. Mixture of Transformer [49] and Pointer Networks [50] decode the sketch parameters in auto-regressive fashion.
Formalizing constrained 2D sketches, i.e., collection of curves (e.g., line, are, circle and splines) with dimensional and geometric constraints (e.g., ro-incidence, perpendicular, co-linearity), as a language for CAD modeling has been studied over last few years [36, 10, 33, 54, 24]. However, the first proposal of developing CAD language interface was suggested decades ago in [41]. Among the recent works in this direction, SketchGen [36] represents 2D sketches as a sequence of the tokens for curves and constraints. The decoder-only transformer model in [36] predicts optimal sketches through nucleus sampling [13] of token embedding vectors, focusing on replicating drawing processes of CAD designers. Polygen [33] method also employs Transformer model [49] to generate detailed 3D polygonal meshes by learning joint distribution on vertices and faces of a CAD. As an extension of [33], TurtleGen [54] also propose decoder-only transformer model to learn joint distribution of vertices and edges together that form sketches and represented as graphs in CAD models.
3D CAD modeling steps as language is not directly formulated by any tfate-of-the-arr multi- modal CAD learning methods [56, 29, 19, 59, 9, 31, 58, 24J. Khan al. [19] propose a novel parameters directly from 3D point clouds as input whereas DeepCAD [56]. SkexGen [59], HNC [58] and MultiCAD [29] adopts a two-stage strategy to generate the output. MultiCAD [29] adopt multi- contrastive learning to associate geometry features with features of CAD construction sequences whereas CAD-SIGNet [19] requires an extra step user feedback to vote for one of the many generated sketches at current step to predict the next one. Unlike previous approaches, our proposed Tex12CAD transformer is the first auto-regressive network that generates CAD construction sequences directly from textual descriptions.
auto-regressive generation of beeich-and-extruse
3 Tex12CAD Data Annotation
The diagram in Figure outlines the process of generating textual annotations for DeepCAD dataset [56] using Large Language Models (LLMs) [17, 35, 47] and Vision Language Models (VLMs) [27, 26]. These annotations describe the corresponding CAD construction workflow in human interpretable format. To enrich the DeepCAD [56] dataset with textual annotations, we implement a tto-stage description generation pipeline using the capabilities of both LLMs and VLMs. The two stages are - (1) generating abstract shape descriptions using VLM, and (2) extracting multi-level textual instructions from L.LM based on the shape descriptions and design details provided in the datasct. An example text prompt for the CAD model shown in top-left of the Figure 2: 'The CAD
3

---

VLM Procepe
(WST) ih Comps Aced Design ICAD mokel You
aun verior CAO engioee losee olieet sulee whewn yow
langage foor
aoout CAB nocil
CAB you cimpll desilpricn wiss the sins iptiiiiiin the follwwng gags
Maltiliee heage
Dispclo
suanplac
CAD modes
VDr Ptrnprape
Mesli dlject
MeiralMesades
Abide the lowinng rddes
LLataiteat
Seceisse
Do vorda lik "shae' "Shadwo' Tour spatnt "reea""
Shaps interuta
"Slatic "image "back" "gay'" CAD model" aiberac "curge"
puapir puiden? "graar
tay boctere
"pAtS" ('pert T
Nataral incrrctiion
spstem"
"Fransiation Vectar" (8.197,0.107 (.977.
"Bube OR
Scropl Nil Prongt
You une siner cWO ve on bndert prosid aus of harging
Minimal ISON
Masp 174
cince 1
6.111
Dvey CAO cone tee vultcbe ioeer ollice useerily inarsedee
Lid vile par lii the prociseed coed scling CAtrrol
po0
Focke?
tetinctiee (M sivoraationn hrarpe
prowad und sa wang for myro Bel
"ectnude depth
5487)
"deech acalh:3 6420
Eopert CAD manurrrr
Seal Dow 20
TotnBByyysauurreeeeer 3-4
Sen's Twrek
Leve-1
Leve22
(esar)
Figure 2: Text2CAD Data Annotation Pipeline: Our data annotation pipeline generates multi-level text prompts describing the construction workflow of a CAD model with varying complexities. We use two-stage method (Stage 1) Shape description generation using VLM (Stage 2) Multi-Level textual annotation generation using L.M.
model consists of a cylindrical object with a flat top and bottom connected by a curved surface and slightly tapered towards the bottom. This object is created by first setting up coordinate system, then sketching two concentric circles and drawing a closed loop with lines and an arc on a shared plane. The sketch is then extruded along the normal direction to form a solid body. The resulting part has a height of approximately 0.0487 units" " In this example, the phrase in the violet color is generated by VLM. An LLM uses this description along with the CAD construction information to generate the prompt.
The initial step of our annotation generation pipeline involves generating abstract object-level descriptions of the CAD models using LLaVA-NeXT [26] model. The objective in this step is to accurately capture the structural descriptions of the 3D shape. such cylinder", or "a hexagon with a cylinder on top". " We generate shape descriptions for both the final CAD model and its intermediate parts. We first produce multi-view images from predetermined camera angles for each individual parts and the final CAD model. These images are then utilized in predefined prompt (refer to the top-right of Figure 2) for the VA- NeXT [26] model to generate simplified shape descriptions of all individual parts as well as the complete final shape.
Shape Descriptions using VLM:
as "a ring-like structure",
In this stage, multiple textual annotations correspond- ing to different design details of CAD model are generated using Mixtral-50B [17] through a series of steps (refer to the middle-columm Figure 2). The DecpCAD [56] dataset contains CAD construc- tion sequences in ISON format. We first preprocess the raw CAD construction sequences using 'Minimal Metadata Generator 'which replaces random, meaningless keys with more meaningful terms (e.g.. "c.""arr___ "Ioop_1"). This step aims to reduce the hallucinations [14] by Mixtral-50B [17]. The minimal metadata is further augmented with the shape descriptions for each parts and the final model generated by the VLM. The output of this process is condensed representation of the shapes and their relational attributes within the CAD design (see bottom-left in Figure 2). With the minimal metadata at hand, we then craft prompt (refer to the bottom-right in Figure 2) to generate detailed natural language instructions (NLI) ensuring a minimal loss of information from the minimal metadata. Afterward, the NLI responses are refined by LLM using k-shot [5] 'Multi-Level Natural Language Instruction Generation Prompt" to generate multi-level instructions of different specificity and details. We categorize these levels as:
Multi-level
Design Instructions using LLM:
Abstract level (L.0): Abstract Shape Descriptions of the final CAD model extracted using VLM in the first stage.

---

Beginner level (LI); Simplified Description Aimed at laypersons or preliminary design stages, this level provides simplified account of the design steps, eschewing complex measurements and jargon.
Intermediate level (L2): Generalized Geometric Description This level abstracts some of the details, providing a generalized deseription that balances comprehensibility with technical accuracy.
Expert level (L3); Detailed Geometric Description with Relative Values Here, the instruc- tions include precise geometric descriptions and relative measurements, catering users who require an in-depth understanding or are performing the CAD modeling task.
Our annotations consist of the generated multi-level instructions at the final stage. We generate these annotations over the course of 10 days. It's worth noting that one can directly generate the multi-level instructions from the minimal metadata without creating the detailed natural language instructions in the second stage. We observe that this strategy increases the LLM's 's tendency for hallucinations [14] and it generates more inaccurate multi-level instructions. Instead our method follows chain-of chain-of-thouge thouhh prompting strategy as outlined [52] which greatly reduces such bottleneck. More details on our annotation pipeline are provided in Section 10 and 11 of the supplementary material.
Text2CAD Transformer
The Tex12CAD transformer architecture, as shown in Figure 3, is designed to transform natural language descriptions into 3D CAD models by deducing all its intermediate design steps autoregres- sively. Given an input text prompt T E IR,,,, where p is the number of words in the text, our model learns the probability distribution, P(C[T) defined as
No P(C = II P(c.ema-. t=1
(1)
where C is the output CAD) sequence, No is the number of tokens in C and 0 is the learnable model parameter. We represent C sequence of sketch and extrusion tokens as proposed in [19]. Each to- ken ez E C is 2D token that either denotes (1) 2D-coordinate of the primitives in sketch, (2) one of the extrusion parameters (euler angles/translation vector/cxtrusion distances/boolean operation/sketch scale) (3) one of the end tokens Following [56, 19]. We quantize the 2D coordinates as well as the continuous extrusion parameters in 8 bits resulting in 256 class labels for each token. An example CAD sequence representation is provided in Figure: 3 (in blue table). For more details, please refer to the supplementary section 9.
sequence/end sequence).
Now we elaborate on the various components of the architecture, detailing the processes involved in converting text CAD representations. Let the input text prompt at timestep / T E RR,, and the input CAD subsequence C1:-11 E TR N
The initial step in the Tex12CAD network involves encoding the textual description provided by the user. This description can range from highly abstract, beginner-friendly instructions to detailed, expert-level commands. To handle this diversity, we used pre-trained BERT (Bidirectional Encoder Representations from Transformers) [8] model, denoted BERT pre-trained- The input text T' E R. is tokenized and passed through the BERT [8] model to generate contextual embedding:
Pretrained Bert Encoder:
T BERT preRTare--aineed (T')
Here, T E IR Xd represents the sequence of token embedding vectors that capture the semantic meaning of the input text, where N, is the number of tokens and dp is the dimension of the embedding.
adaptive layer consisting of transformer encoder layer, refines the output T of the BERT [8] encoder to better suit the CAD domain aligning with the specific vocabulary and structural requirements of CAD instructions. The adaptive layer outputs the embedding Talapt
Adaptive Layer.
R N,X,, using
T adipp (daptiveLaayer(()
CAD Sequence Embedder: Each token in the input CAD subsequence C1a-1 is initially rep- resented as a one-hot vector with a dimension of 256, resulting in a one-hot representation, C C 1gN--12x256 For the sake of simplicity, we represent C [C C.y-al
5

---

Reginner CAO Designer
Pegerts Lived inttucttons
Friendyy tedrusttion
for ceesn oyaine
sible biw or sieuet slgust eight and and tasss soojons leep furade alreeh
Toe jrobon rigert serel
the proker ruget semd
to dhach maburty
V.
ecuilo speestee to froe ricteed fyy A.5
T
trained
addapt
Slotch Tolona
VHA
Crase Attestion
mTV
MLP
Pootional
Raconumuced Model
Porumeters
Freeen
Eetrationn
CAD
Takene
Tolont
Figure 3: Network architecture: Tex12CAD Transformer takes as input text prompt T and CAD subsequence C1a-1 of length 1. The text embedding Tadopt is extracted from T using pretrained BeRT Encoder ( ([]) followed by a trainable Adaptive layer. The resulting embedding T Tadagt and the CAD sequence embedding F is passed through L decoder blocks to generate the full CAD sequence in auto-regressive way.
where Cit-1- C1:-1 obtained using Eq. 4
IR NR-1111288 X256 The initial CAD sequence embedding F E R- xd
F = Ci. W C W 1 +P
(4)
where W W
t-1 E R. xx are learnable weights and E R. 1xd is the positional encoding.
Layer-wise Cross Attention. We use standard transformer decoder [49] with layer-wise cross- attention mechanism between the CAD and the text embedding within the decoder blocks. The layerwise cross-attention mechanism facilitates the integration of contextual text features with the CAD embedding, allowing the model to focus on relevant parts the text during CAD construction. Each decoder block takes as input CAD embedding F and text embedding nndapt, where F the output of the previous decoder block (for the first decoder block, the input CAD embedding is F At first, the CAD embedding F E RR N. xd is generated from t-1 using
F MHA(F
where MHA is the multi-head self-attention [49] operation. Afterwards, We downsample Tadup to generate TV R Nxxd using
adaps
T adapi = adaae W
(6)
adupt
1R xd is the learnable projection matrix. The cross-attention mechanism involves query (Q), key (K), and value (V) generation using
where W adapt
Q = F F-Ft-iQ K T adapt W
adapt Wy
(7)
The Here, Wq dddd WK E dddx, and Wv E gdxd, are learned projection matrices.
cross-attention output E xdx is computed as:
QK
A Softmax
V
(8)
where d, is the dimensionality of the key vectors. The cross attention mechanism enables the model to dynamically adjust the importance of different parts of the text relative to the CAD sequence. Afterwards, the output embedding of the decoder block / generated using
F +- LaverNorm(FFN(LLyeenNO
Dropout (oooout(A))))
(9)

---

Expert Lovet Prinet
DUEEK Stnde
100. M000
The CAC ros drn entiarms
Dosie thr wes cra mark, in wiludtii Wegerter
Cla. cyindria sbact
dhaccing
commic arden or chand
covelll
io korn aode basy, The sauding
becromeen le deiten hern Lame' wtthr comtrr tiems 17M4
sepa zh oppactee carator, dectre acaar passrata bir 3u souzr RoS.37
Extrusion Details
Nrmm fowkr
bobyy from
Simple details
The frv par in rellwinng de ane lergn 18 3755 friw skintt det
Shape description
par codeing reanttass by ociicc elpeci siiht sanees along is tengiht
Figure 4: Parametric CAD model generation by Tex12CAD transformer using different text prompts. Our text prompts follow a certain structure highlighting the different design aspects of CAD construc- tion workflow (shown in different colors). Abstract (LO) and Beginner (L1) level prompts contain shape descriptions (teal color) whereas Intermediate (L2) and Expert (L3) level prompts are more parametric and contain design details for sketch and extrusion (yellow and red).
.where FFN is the feed foward network [49] and LayerNorm is the Layer Normalization [49]. The complete Transformer decoder block is repeated I. times, allowing for deeper integration and refinement of the text and CAD tokens. The final CAD embedding F/L1 E NeN
xd is passed to an
MLP to generate the output CAD sequence. We use Cross-Entrope loss during training.
5 Experiment
Dataset. We use the DeepCAD [56] dataset which contains approximately ~ 150k training CAD sequences and 8k test and validation sequences in kketch-and-extrude format. Following, [56, 19], the sketches and the final CAD models are normalized within a unit bounding box. For each sample in the dataset, four design prompts ranging from abstract expert levels (LO, L1, L,,, L3) are generated using our data annotation pipeline resulting in ~ 600k training samples, and ~ 32k test and validation samples.
Implementation Details. Text2CAD transformer consists of L, = 8 decoder blocks with 8 self- attention heads. The learning rate is 0.001 with AdamW [28] optimizer. Dropout is 0.1. Maximum number of word tokens, Np is fixed as 512 and CAD tokens N₂ as 272. The dimension d, for the pre-trained Bert encoder [8] embedding T as well as T adopp is 1024. The CAD sequence embedding d is 256. Following [19]. the first two decoder blocks do not use any cross-attention operation between the text embedding and the CAD sequence embedding. The Tex12CAD transformer has been trained with leacher-forcing [53] strategy for 160 epochs using 1 Nvidia A100-80GB GPU for 2 days. During inference, top-1 sampling has been used to autoregressively generate the CAD sequences from an input text.
Since there are existing methods for generating parametric CAD sequences from text prompts, we use DeepCAD [56] and our Text2CAD w/o o AL. (i.e., without Adaptive Layer) variant as our baselines. To adjust DeepCAD [56] for performing CAD generation from text inputs, the Adaptive Layer [56] (see Section 4) is trained to map the pre-trained BERT [8] embedding T into the ground truth latent vector z. During inference, the predicted latent vector 2 then passed the pre-trained DeepCAD [56] decoder to generate the CAD sequences. For Text2CAD w/o AL, the pre-trained BERT [8] embedding T is directly passed the transformer decoder.
Baseline.
Experimental Setup. For setting up our evaluation protocol, the selection of the input text prompts and desired outcomes of the CAD models depicted in Figure 4. Our textual annotations follow certain structure. In the abstract (LO) and beginner (LI) level prompts, the shape descriptions are more prevalent (in teal color in Figure 4). The intermediate level prompts augment simple sketch (in yellow color) and extrusion (in red color) details with the shape descriptions. Expert-level prompts include more precise details for each of the design aspects previously highlighted. In all our experiments discussed below, we use these four levels of prompts following the aforementioned formats. However, we have conducted another experiment, where we interpolate between the abstract and expert prompts to generate multiple new prompts and observe the performance of our model on these prompts. Due to the space restriction, we provide the results of this experiment in the supplementary Section 12.
5.1 Evaluation
Our goal is to evaluate how well the generated CAD sequences align with their respective input text prompts. We concentrate on two primary aspects: (1) examining the parametric correspondence be- tween the predicted CAD sequences and the input text prompts, and (2) conducting visual inspections
7

---

CN oRN taon thu owen S07, AOT ond'
Sirpeet centang with voums
bot dar And ae CWWW
glonia Denace rest ehin sevrnc
Input Text
For Rich0r SEs. plox and Burtain aby laga
Srader wis aircte revethigg acaloregom
JOCTIN Paye STRoONiteE DENDNLL Fowagss IEvELv
cane colver 31977 Lot Grma Noget 6 6.75 regieoc
moanyorr Rorui E/EITiC 17770 ueit..
DeepCAD
Text2CAD
sxord econdiodeeto caacaco mode length matching the oppeseno wheel desciition,
oxcandreace ostar strusture ochhoodd pronent
secoed model interoemected
cocad
recandnessal of partice mstang.a pi vicledgiens
sacasd better syleehal otpara and rittanuaa algnedie
desmpption whe restongiar on,
Evaluation
mail gaanettic decall best descripiinn
somseddd boie
with tanted drorption
cenntil cyiidiica holk
Figure 5: Qualitative results of the reconstructed CAD models of DeepCAD [56] and Tex12CAD on DeepCAD [56] dataset. From top to bottom Input Texts, Reconstructed CAD models using DeepCAD [56] and Text2CAD respectively and GPT-4V [35] Evaluation.
of the reconstructed 3D CAD models. Currently, there is standardized benchmark for text-to-3D models that can be directly applied this task. Existing methods for text-to-3D [51, 15] utilize pre-trained CLIP [42] models to measure the alignment between reconstructed 3D models and their text prompts. However, CLIP [42] scores may not adequately evaluate the geometric [57] and the parametric aspects of CAD designs. Therefore, inspired by [57, 11], employ three evaluation metrics to thoroughly measure the parametric and geometric alignment between the generated CAD designs and the text prompts.
A. CAD Sequence Evaluation: In this evaluation strategy, we comprehensively assess the parametric correspondence between the generated CAD sequences with the input text. We use the groud truth CAD sequence for this purpose. We only use this evaluation strategy for expert-level (L3) prompts. Since, expert-level prompts, being highly detailed with parametric details, exhibit higher one-to-one correspondence with the ground truth CAD construction sequences compared to other levels.
To measure the correspondence between the ground truth and the predicted sequence, the strategy outlined in [19] followed. We evaluate the F1 scores the primitives and extrusions by aligning the predicted loops with the ground truth loops within the same sketch using the Hungarian matching algorithm [21] (An example is provided in supplementary Figure 10). The geometric alignment between the ground truth and reconstructed CAD models is measured using the chamfer distance (CD). The Invalidity Ratio (IR) is calculated measure the proportion invalid CAD sequences sequencess that can not generate any CAD model).
Table I: Quantitative evaluation between DeepCAD [56] and our method (AL is Adaptive Layer). The scores are evaluated only for Expert Level (L3) prompts. The results include F1 scores for primitives and extrusions well mean and median CD and IR. CD is multiplied by 103.
FIf
Median
Mean
Model
IR4
Line
Arc
Circle
Extrusion
CD}
CD1
DeepCAD
76.78
20.04
65.14
88.72
32.82
97.93
10.00
Tex12CAD w/o AL
78.88
27.18
71.44
93.28
0.82
35.91
2.69
Tex12CAD
81.13
36.03
74.25
93.31
0.37
26.41
0.93
Table summarizes the quantitative results between the baseline methods and our final Tex12CAD transformer. Compared to the baselines (rows 1-2), our model (row 3) achieves higher F1 scores for all the primitives and extrusion, with the most notable being an 80% improvement in the F1
score for ares. The results indicate a better correspondence between the expert-level prompts and the predicted CAD sequences. Notably, our model significantly outperforms the baseline DeepCAD [56] in terms of median CD and invalidity ratio by factor of ~ 88.7 and 10.75 respectively. The higher CD despite having relatively high scores for DeepCAD [56] indicates that even though DeepCAD [56] can recognize the primitives and the number of extrusions from the input texts, it

---

recannguoo prizis with cylinrical hale center- lines the reciungle... Scale 0.155 units length 0. Extrode .. 0.25 unit Fort 1. from jomed together
bracke with noves on appeste side... ardi and the secoed shoue concats lines helpht... 0.128 unis.
of
A fox, diessor object with four lis
recioggulor drucin along leggii, uped securigg obyech togetter
hesapon shope oraalor shope extrudd along the normal direction hegnt... 0.225 wnnzz
Input Text
DeepCAD
Text2CAD
The more accurate: it has four perimeter holes.
second matches the description: rectangular metal bracket shape, four wigned holes present.
The second model a hesagonal promm central cyllndrical holk, matching the prompt' description shope
The sacond model more accurate: matches the bracker's shape hole
The saconsi model more accurate: has rectangular prism with cylindrical ho convistett the prompt
Evaluation
Figure 6: Additional Qualitative results of the reconstructed CAD models of DeepCAD [56] and Text2CAD DeepCAD [56] dataset. From top bottom Input Texts, Reconstructed CAD models using DeepCAD [56] and Tex12CAD respectively and GPT-4V [35] Evaluation.
fails to correctly parameterize those primitives the extrusions. Compared DeepCAD [56], the Text2CAD transformer predicts more accurate sketches and extrusion parameters thanks to its layerwise cross-attention mechanism. In Figure and 6, some qualitative results are shown.
Table (rows and 3) shows the results between Text2CAD w/o AL. and our final model. The incorporation the Adaptive layer in the Tex12CAD transformer improves the scores for primitives such lines, ares, and circles. Notably, there is 2.85% increase of F1 scores for lines, 32.56% for ares, and 3. 3.93% for circles. Moreover, the improvement is particularly striking in terms of the IR, which sees remarkable reduction by factor of ~ 2.9.
Table 2: GPT-4 evaluation of the CAD models generated from 1000 prompts per level and User studies of 100 samples per level. In both evaluations, overall Text2CAD is a favored choice over DeepCAD [56].
GPT-4 Evaluation (%)
User Study-based Evluation (%)
Model
Abstract Level
Beginner Level (L.)
Intermediate Level (L.2)
Expert Level (1.3)
Abstract Level (1.0)
Beginner Level (L1)
Intermediate Level (1.2)
Expert Level (L3)
(LO)
Undecided
0.80
0.3
0.70
DeepCAD
47.40
51.15
40.20
36.06
50.95
48.73
44.94
41.14
Tex2CAD
48.35
B. GPT-4V Evaluation: To perform the visual inspections of the 3D CAD models generated from abstract (LO), beginner (L1), intermediate (L2), expert (L3) level prompts, GPT-4V [35] has been used. We follow the protocol outlined [57] and generate meta prompt that consists of multi-view images of the reconstructed CAD models from both DeepCAD [56] and our model as well as the input text prompts. Following this, GPT-4V [35] provides verdict on which model predictions accurately reflect the text descriptions in terms of shape and geometry. But if the two models are very similar in shape and match the input text accurately, then it outputs 'Undecided'.
We randomly select 1,000 samples for each level from the test dataset and generate verdict per sample using GPT-4V [35]. Table (left) presents the final results. These results indicate that overall Text2CAD outperforms baseline DeepCAD [56]. Additionally, we observe that the performance gap between Tex12CAD and DeepCAD [56] minimal at the abstract (LO) and beginner (L.) levels, despite Tex12CAD losing by 2.8% at the beginner level. However, the complexity and parametric details the prompts increase, the gap widens significantly. Text2CAD outperforms DeepCAD [56] at the intermediate (I.2) and expert (L.3) levels, leading by as much as 18.6% and 27.18% respectively. This indicates that Tex12CAD more inclusive of the highly detailed design prompts. In supplementary Figure we have provided some examples.

---

a circuler object with gyindrrcl hole the center
shape with points
stor shope consisonng trangles jeined together
a ring
three-dimensioonll geometric shope compused interconnected triongles, forming a symonetrical star-ikke structure.
round item hoving gylindrical shope
2
Figure 7: Visual examples of 3D CAD model generation using varied prompts. (1) Three different prompts yielding the same ring- llke model, some without explicitly mentioning 'ring'. (2) Three diverse prompts resulting in same star-shaped model, each emphasizing different star characteristics.
C. User Study-based Evaluation: We conduct user study with 100 randomly selected examples per level to evaluate the preference between method and DeepCAD [56]. Five CAD designers with varying experience levels participate in the evaluation. Each participant shown multi-view images of the reconstructed 3D CÁD models from both methods side by side, along with their corresponding input texts. They are then asked to determine which CAD model is more geometrically accurate and easier edit to achieve the desired outcome. Each participant evaluates 20 samples per level. The final result is provided in Table 2 (right). To our surprise, the result follows a similar pattern as GPT-4V [35] evaluation with minimal performance gap in the abstract (LO) and beginner (L.1) levels and a wider gap for more detailed intermediate (L2) and expert (L3) prompts.
5.2 Prompt Diversity
The diversity the generated textual prompts the Tex12CAD dataset depends on both the variety of CAD models available in the DeepCAD [56] dataset, as well as the performance of Mistral [16] and VAL [26]. enhance prompt variety, we have focused more on generating shape descriptions from VLaNeee [26] rather than only object names. For example, in annotation "a ring" can be sometimes described as "a circular object with a cylindrical hole in the center". This approach enables our transformer model to learn generate the same CAD models using different styles of textual prompts. In Figure 7, we show two examples where the network generated same CAD models from various types of prompts.
6 Limitation
Despite the promising results of Tex12CAD, several limitations exist. Firstly, LLaVA-NeXX [26] sensitive to the perspective distortions in milti-view images, which affects the generated shape descriptions and final I.L.-generin ters in texts is still an open research problem in NLP. Tex12CAD transformer architecture, we used BERT [8] tokenizer which might tokenize some numerical parameters as an UNK token. Thirdly, the lack of standardized benchmarks for evaluating text-toCCAD generation poses challenges assessing model performance comprehensively. Furthermore, the DeepCAD [56] dataset is imbalanced, pre- dominantly featuring rectangular and cylindrical shapes, which limits the model's robustness towards more complex shapes. Some failure cases are described in supplementary Section 13.
prompts. Secondly, effective tokenization of numerical parame-
7 Conclusion
In this paper, we introduce Tex12CAD, the first AI framework designed to generate parametric CAD models from text prompts suitable for users of all skill levels. Our contributions include two-stage data annotation pipeline using Mistral-50B [16] and LLaVA-NeXT [26] and novel end-to-end trainable Text2CAD Transformer architecture that effectively transforms natural language instructions into sequential CAD models. Through a comprehensive evaluation, including GPT-4V [35] assessments and user studies by CAD designers, we demonstrate that Text2CAD outperforms adapted two-stage baseline, especially as the complexity and detail of the prompts increase. Future work will focus on addressing the current limitations, such as reducing annotation inaccuracies and improving dataset diversity, to further enhance the robustness and applicability of Tex12CAD.
10

---

8 Acknowledgement
This work was in parts supported by the EU Horizon Europe Framework under grant agreement 101 (LUMINOUS).
References
[1] Freecad. http:://www.rreecdkkee orgg
org/ Accessed: <Date>,
[2] Mcmaster. http://www.cmmtttr..coo//////// I
[3] Open3d.
org/. 19
[4] Zoo.dev. https://zoo.ddw/ 2
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. 4
[6] Kseniya Cherenkova, Djamila Aouada, and Gleb Gusev. Pvdeconv: Point-voxel deconvolution for autoencoding cad construction in 3d. pages 2741-2744 10 2020. 3
[7] John G Cherng, Xin- Yu Shao, Yubao Chen, and Peter R Sferro. Feature-based part modeling and process planning for rapid response manufacturing. Computers & industrial engineering, 34(2):515 530, 1998.
[8] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. 5, 6, 7, 10
[9] Elona Dupont, Kseniya Cherenkova, Anis Kacem, Sk Aziz Ali, Ilya Aryhannikov, Gleb Gusev, and Djamila Aouada. Cadops-net: Jointly learning cad operation types and steps from boundary- representations. In 2022 International Conference on 3D Vision (3DV), 2022. 3
[10] Y. Ganin, S. Bartunov, Y. Li, E. Keller, and S. Saliceti. Computer-aided design as language. Advances in Neural Information Processing Systems, 34, 2021. 3
[11] Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. bench: Benchmarking current progress in text-to-3d generation, 2023.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:68440-881,, 2020.
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. 3
[14] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qiang- long Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ArXiv,
Abs/2311.05233 2023. 4, 5, 16
[15] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, P. Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 857-866, 2021. 8
[16] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023. 2, 10, 15, 16

---

[17] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04M0
2024. 3, 4
[18] Minseop Jung, Minseong Kim, and Jibum Kim. Contrastcad: Contrastive learning- based representation learning for computer-aided design models. 2024. 3
[19] Mohammad Sadil Khan, Elona Dupont, Sk Aziz Ali, Kseniya Cherenkova, Anis Kacem, and Djamila Aouada. Cad-signet: Cad language inference from point clouds using layer-wise sketch instance guided attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3, 7, 8, 15, 18
[20] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. Abc: A big cad model dataset for geometric deep learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9601-9611, 2019. 3
[21] Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, (11-2):83-97, March 1955. 8
[22] Joseph G Lambourne, Karl DD Willis, Pradeep Kumar Jayaraman, Aditya Sanghi, Peter Meltzer, and Hooman Shayani. Brepnet: A topological message passing system for solid models. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 12772-12782, 2021.
[23] Joseph George Lambourne, Karl Willis, Pradeep Kumar Jayaraman, Longfei Zhang, Aditya Sanghi, and Kamal Rahimi Malekshan. Reconstructing editable prismatic cad from rounded voxel models. In SIGGRAPH Asia 2022 Conference Papers, pages 2022. 3
[24] Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J Mitra. Free2cad: Parsing freehand drawings into cad commands. ACM Transactions Graphics (TOG), 41(4):1-16, 2022. 3
[25] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 2, 3,4, 4, 10, 15
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023. 2.3
(28) Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[29] Weijian Ma, Minyang Xu, Xueyang Li, and Xiangdong Zhou. Multicad: Contrastive representa- tion learning for multi-modal 3d computer-aided design models. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, CIKM '23, page 1766-1776, New York, NY, USA, 2023. Association for Computing Machinery. 3
[30] Felipe Machado, Norberto Malpica, and Susana Borromeo. Parametric cad modeling for open source scientific hardware: Comparing openscad and freecad python scripts. Plos one, 14(12):e0225795. 2019. 2
[31] Dimitrios Maliis, Ali Sk Aziz, Elona Dupont, Kseniya Cherenkova, Ahmet Serdar Karadeniz, Mohammad Sadil Khan, Anis Kacem, Gleb Gusev, and Djamila Aouada. Sharp challenge 2023: Solving cad history and parameters recovery from point clouds and 3d scans. overview, datasets, metrics, and baselines. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1786-1795, 2023. 3
12

---

[32] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[33] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning, pages 7220-7229. PMLR, 2020. 3
[34] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts, 2022. 2
[35] Josh OpenAl, Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 4pt-4 technical report. arXiv preprint arXiv:2303.0877:
2023. 2, 3, 8, 9, 10, 16, 19, 20, 21
[36] Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas J Guibas, and Peter Wonka. Sketchgen: Generating constrained cad sketches. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5077-5088, Curran Associates, Inc., 2021.
[37] Bonsa Regassa Hunde and Abraham Debebe Woldeyohannes. Future prospects of computer- aided design (cad) review from the perspective of artificial intelligence (ai), extended reality, and 3d printing. Results in Engineering, 14:100478 2022.
[38] Juergen Riegel, Werner Mayer, and Yorik van Havre. Freecad. Freecadspec2002, pdf. 2016. 2
[39] Daniel Ritchie, Paul Guerrero, R Kenny Jones, Niloy J Mitra, Adriana Schulz, Karl DD Willis, and Jiajun Wu. Neurosymbolic models for computer graphics. In Computer Graphics Forum, volume 42, pages 545-568. Wiley Online Library, 2023. 3
[40] David Robertson and Thomas J Allen. Cad system and engineering performance. IEEE Transactions on Engineering Management, 40(3):274-288 1993.
[41] Tariq Samad. A natural language interface for computer-aided design. Springer Science & Business Media, 1986. 3
[42] Aditya Sanghi, Rao Fu, Vivian Liu, Karl DD Willis, Hooman Shayani, Amir Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18339-18348, 2023. 8
[43] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P Adams. Sketchgraphs: A large-scale dataset for modeling relational geometry in computer-aide aided design. arXiv preprint arXiv:200.008506, 2020.
[44] Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A generative model of parametric cad sketches. In International Conference on Learning Representations, 2021. 3
[45] Siemens. ParaSolid. https://wwz www.aases..comppooccttccccceeeea
[46] Solidworks. 3D CAD Design Software. 02-June-2022. 1
Online: accessed
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023. 2,
[48] Kenton Varda. Google protocol buffers: Google's data interchange format. Technical report, 2008.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 17, page 6000-6010. Red Hook, NY, USA, 2017. Curran Associates Inc. 2, 3, 6,7
13

---

[50] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural information processing systems, 28, 2015.
[51] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In seirth-seventh Conference on Neural Information Processing Systems, 2023. 2, 8
[52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[53] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, (2):2770-280) 1989. 7
[54] Karl DD Willis, Pradeep Kumar Jayaraman, Joseph G Lambourne, Hang Chu, and Yewen Pu. Engineering sketch generation for computer-aided design. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2105-2114, 2021. 3
[55] Karl DD Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, Joseph G Lambourne, Armando Solar-Lezama, and Wojciech Matusik. Fusion 360 gallery: A dataset and environment for programmatic cad construction from human design sequences. ACM Transactions Graphics (TOG), 10(4):1224, 20
[56] Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer- aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6772-6782, October 2021. 1, 3, 4, 5, 7, 8, 9, 10, 15, 16, 19
[57] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) a human-aligned evaluator for text-to-3d generation. In CVPR, 2024. 8,9
[58] Xiang Xu, Pradeep Kumar Jayaraman, Joseph G Lambourne, Karl DD Willis, and Yasutaka Furukawa. Hierarchical neural coding for controllable cad model generation. arXiv preprint 2377v:2307.00149 2023. 3,3
[59] Xiang Xu, Karl DD Willis, Joseph G Lambourne, Chin- Yi Cheng, Pradeep Kumar Jayaraman, and Yasutaka Furukawa. Skexgen: Autoregressive generation of cad construction sequences with disentangled codebooks. In International Conference Machine Learning (ICML), pages 24698-24724, 2022. 1.3
[60] Yasuhiro Yamamoto and Kumiyo Nakakoji. Interaction design of tools for fostering creativity in the early stages of information design. International Journal of Human- Computer Studies, 63(4):513-55) 5355, 2005. Computer support for creativity. 1
[61] Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold, Gpt-4v(ision) as a generalist evaluator for vision-language tasks. ArXiv, abs/2311.01366 2023.
[62] Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10, 000 3d-printing models. ArXiv, abs/1 605.04797 2016.
14

---

CAD Sequence Representation
Table 3 shows all the tokens used in CAD sequence representation. We use the representation as proposed by Khan [19] which uses a sketch-and-ets and-extrude format. Each 2D sketch consists of multiple faces and each face consists of multiple loops and every loop either contains line and a or circle. Loops are always closed same start and end coordinate). We parameterize the curves
in the following way
Line: Start and End coordinate.
Are: Start, Mid and End coordinate.
Circle: Center and top-most coordinate.
Finally, represent sketch using sequence of 2D coordinates only with specific end tokens for the end of curve, loop, face and sketch. Each extrusion sequence consists of the 10 parameters followed by end of extrusion token. These
Euler Angles: parameters (0, o, 7) determining the orientation of the sketch plane.
Translation Vector: parameters (Tz, Typ Ta) that describe the translation of the sketch plane.
Sketch Scale: 1 parameter (α) for scaling the 2D sketches.
Extrude distances: parameters (d+ .d containing the extrusion distances towards and opposite of the normal of the sketch plane.
Boolean Operation: 1 parameter (ß) determining the extrusion operation. There are
extrusion operation in DeepCAD [56] dataset namely solid body, cut, join and intersection.
Except the boolean operation and all the end tokens, all the 2D sketch parameters as well as the extrusion parameters are quantized in 8 bits.
Table 3: CAD sequence representation used our experiment.
Sequence Type
Token
Token
Token
Description
Type pad
Value
Representation
(0, 0)
Padding Token
cls
(1,0)
Start Token
end
(1,0)
End Token
6
(2,0)
End Sketch
es
(3,0)
End Facc
61
(4,0)
End Loop
Cc
(5,0)
End Curve
(pus Py)
[11.266]]
(Pa. P.)
Coordinates
Extrusion Distance Towards
d+
[11..260]
(d+ .))
Sketch Planc Normal
Extrusion Distance Opposite
d
[11 266]
(d (d-,0)
Sketch Plane Normal
Tx
[1..266]]
Tay
Extrusion Sequence
Ty
11..266
Tys
Sketch Plane Origin
T2
11..266]
[T,,
0
11..266
(0,0)
Φ
11.2666
(Φ,0)
Sketch Plane Orientation
7
11.2666
(7,0)
0
11..2601
(0,0)
Sketch Scaling Factor Boolean (New, Cut, Join, Intersect) End Extrude
3
77, 8, 10)
(3,0)
Ee
(6,0)
10 Implementation Details for Data Annotation
As mentioned in Section 3, we use LLaVA-NeXT [26] for the VLM oriented task and Mistral- 50B [16] for L.LM tasks. We use Nvidia A100-40GB GPU run LLaVA-NeXT [26] and 4 Nvidia A100-80GB GPUs to run Mistral-50B [16].
15

---

Random Keys removed
Redundant Information is removed
"ontales:: MM4DC S (
haae": "Fatrude It,
Shape information from LLaVANext
"ype": "Extrudefessure"" pronilest:[(
"profile" 1,07
Stetch:: #1QgWG/WWrgalluyy)
Tinal shapet t 1A cylinricd ial object flat and cottom, and slightyy tapered middle section," "parts:: 1"s
rextent two": (
distancee {
Typet: "ModelParameter""
Restructuring
"extrusion'? E
Trole" "Sgairst
"extrude depth towards, normatt 1.145,
"name: Tnone",
"extrude depthh www.pppeeeeoomaafft.
000
Skketh scale's 0.75.
"ype": "ppe::"oisanncinnnneetttttt
"aperation" ont Neeeaadyyfacuuneeeeaoooo
(Othur endty relaned omiited livvtty
*coordinate, system":
aperatiinn: Neeeookffsttrreppraalii
"Euler Anglesft [0.0, 0.0)
"scat, Extent":
"Transiation Vector": c.o.p.pp
"type, "wwilleflannsseeeeeiitttrrr
*sketch":
"yxtent_bype"= tDne-naasssasstueeeeenneenrrr
"tace_""
"COuWc/sWhoolpply""
"oop_11
"transform":
"circle11s 154
"origin: (yy:0.0, k0:000 12":0.0).
"Center"s p0.35, 0.375]
"eiss: 'x" (7:00)
"Racius": "Racius"..5555pD 0.57539.
2. y-:00 1x11.0 30000)
"description::
2. sc8 (y:00, 51000 '25:10333
"shape": cylndrical abject
"Sketch" "Sketch""ttch",
bottom, and slightly tapered middle section."
"name":
"lengtht: 0. 7.749999333 90778833317711
"proflest: 4
"widtt' 7499999331144 99673377888
"center. "cnnter_point: 50.00, 1200000)
"heightt 1481144444401177 )3)
"properties::
"referenre_planee:..
"proportiest:
Minimal JSON
"bounding box"
"max, point" (y 0.09 048514433344
**" 0.0510044511455555000
2.025-00000000000000
DeepCAD JSON
Figure 8: An example of Minimal metadata JSON (right) generated from DeepCAD [56] JSON (left). During the minimal metadata generation, random keys (e.g.. "FF444C(1990000x55555 or redundant design information (e.g../'type": "ModelParametee"" "role"; "AgainstDistance" /) is removed.
11 Additional Details on Data Annotation Pipeline
Since LLMs are prone to hallucinations [14], we employ several strategies to mitigate this issue. Firstly, we observe that when we directly pass the DeepCAD [56] jsons to Mistral [16] instead of the minimal metadata to generate the detailed natural language instructions, the model often uses the random keys provided the dictionary refer the corresponding curves the sketches. Additionally, the json contains redundant information that is not necessary in the final CAD con- struction. To overcome this, generate minimal metadata restructuring the dictionary into more human-read readable format. In Figure 8, we provide an example DeepCAD [56] Json (left) and a minimal data(right)
12 Additional Experimental Details on Interpolated prompts
In this section, we provide details on our model's performance on text prompts that contain different structure than the training samples. To generate these prompts, we pass all four text prompts abstaabt beginner, intermediate and expert) of CAD model to GPT-4V [35] and ask to generate 20 more samples by interpolating between the all the levels. Figure 12 and Figure 13 shows visual results of two examples. The results indicate that Text2CAD can effectively handle varying level of prompt structures, generating accurate similar CAD models in shape compared to the ground truth. It retains the overall geometry for prompts with highly abstract shape descriptions
16

---

Invalidity
Discrepancy
rechongulor pricen base extruded width 17.09 mmn (0.1709 length (0.4933 m).
simple, symmetrical metallic object with curved, looped shope, resemblnng stylized letter 's' twisted
The CAD mode! includes shope with curved edge. slontlor to spatuia podale
rectongulor prism (0.0..00)
Input Text
8
(0.001,0.6) fj line sketch scale 0.25
Create the part. This part is then joined with the previous
extrude 0.75
Text2CAD
INVALID
INVALID
Figure 9: Failure cases for Text2CAD Transformer. Invalid Samples (left): The network fails to generate any valid CAD model. Discrepancy Cases (right): The generated CAD model does not match the input text prompts.
(fistl and second row in Figure 12 and Figure 13). As the parametric details increase in the text prompts, generates the precise CAD model as the ground truth (third and fourth row in Figure 12 and Figure 13)
13 Discussion Failure Cases
In this section, we describe two types of failure cases for Text2CAD Transformer. In Figure 9, we have shown examples of both type of cases.
Invalidity: In this scenario, the model fails to generate any CAD model from the text prompts. As reported in Table 1, this occurs in approximately 1% of the test samples. In these cases, the model predicts invalid sketch or extrusion parameters, such as the same start and end points for lines or arcs, values for extrusion depth on both sides.
Discrepancy refers to situations where the generated model does not precisely match the shape described in the text prompts. This is more prevalent in our model than invalidity and harder to quantify. We notice that this occurs when prompts are more focused on object name (e.g..saatul,, paddle) rather than parametric descriptions. We argue that this issue comes from noisy VLM annotations. As mentioned in the Section 6, the perspective distortions in multi-view images affect the accuracy of shape object name recognition. These noisy descriptions propagate into other prompt levels using the annotation pipeline.
Discrepancy:
17

---

Ground Truth Sketches
Predicted Sketches
Loop Indox
Loop Matching
Loop Matching
Sketch
For Sketch 2
Primitive Marthing
Primitive Musching
Mankk
(00)
Piimtive
Privitvv Munching
Hangorian Marching
Porriive Matching
Cost Matro
Pattched RemitieeRRe Paire
BPoll
0100 wij oi Wol Minl ofl ollle
Line
Girlle
6
0.6
8
2
0.38
Precision
0.06
0.50
Figure 10: Fl score calculation for CAD sequence evaluation as proposed in [19].
18

---

Intermediate
wring drow considing of light the specyled coortinore gytem, Lines to jorm borton ttyer with inights ord 1278, 0.575 and 0.5 unit....
pectongoior
siles brocher created by seming cooraloate system, skatching shopes poralee places. The first shape is dirte on one plang, and the second shope consicts lines ond orci on thee other plane...
bracket with
gysem with ongles of
translation recoor of 64,
DeepCAD
DespCAD
Prompt Goommttic Detaila Evaluation
Frompt
Ceometric Details Evelvation
Beginner Prompt
Prompt (LO)
model
resuting shape using specyfed system hos a neight cpprox. 2.375 unts.
madel features o sylndiiia abject syinaiical cente.
DeepCAD
DespCAD
Pronep
Algermet Evaluntion Geomeerk Decall Dvaluation
Geometiic Decalls Dralustion
Wedk: Zexand Hadal
Figure 11: GPT-4V Evaluation Strategy: Four prompts (one per level) are randomly sampled from the test set. These prompts are used to reconstruct CAD models from the predicted CAD sequences using both DeepCAD [56] and the proposed Tex12CAD. Nine multi-view images of these models are rendered using Open3D [3] and stacked in 3 grid, which are used for the GPT-4V [35] evaluation. GPT-4V [35] analyzes their alignment with the initial text prompt and their geometric details and provides a final verdict for this comparison. As shown in the evaluation, our model performs better when input text prompts contain more parametric details.
19

---

Groudhrtth CAD model
Esimusion Height: 0.07
X
Redllsk
A C4D roed dush you crrutar
Aeca CAD concheeda sppccieenniingg for untomb qfindd ofinct callole
Bea Level (L.) CAD Dasinnecs Prsnntttttt......... a0d Output Modal
mandy aO the priteeer foouiing raisd
opiddical
The modil wct boke diribedd arundd the ped mame..
heblic eaetly diaihandd sound edge.
The CAG 8:0 pmores dis uscdd in wilh of
Cwnredeeeeeaaammmmmmmmmmmm ant. tins Alt aosd and
cyloiical
paled quiniicd hake
auddadd uerly svend cale vags
Intermediazz Level (L22) CóD www.oeoaaaaaaacooooooo and Cuc.uuu Model
CAD rode a caraud sipcci pertneter The
cincatr dis rilri with loat ewity spacud tako wyy are dic aa sysaem excuding de
ciradi with
detiing new cocilira ryear, coaing discc and randoriing
atskinn lring contiius spate ahod mop hime
pours is a aradee
cytedical
eordinggio 30
Expert Level (L3) CAD
Rmspooe's www.moottscc..... Cusput Model
stamiaaionn vacke elgi oascs angins stech for mondieess josrs exm1 and
DE cona dusd wi kot toiol deimngua mandraak updkrrs
Cenesau rucel
man pocy with eoce si
coont ian oyanes one clnechig tta and
skoldh by kutoria of 6.75 hurarom inn to siuter ramaal
mote 13 km,, 12g Pash hale hara doowenr of 20 cadkon ontes Sea the vate dumbhry tatoo stn ms inntion
ai model The CAO iidel consarss of ralsed in four eaartyy spaced oj sarcaal steps CAt assenily way steeei etaet with height oppereer atey Ihs
nuruuding ualerd
wiithhe ongles translation vocter derteed Gaoude sianch aking be namal vacke by 2025 unios in appoake divccion o uniii la pruna finat rode: CAl mucil follooin
Figure 12: Visual results of Tex12CAD on interpolated text prompts generated by GPT-4V [35]. From top bottom, the geometric details in the text prompts increase.
20

---

Grouttoath CAD model
0.375
Length: 1.75
Extrnsionn Height 0225
2i PL Taiy
Radaus
Recdinee Lecal (L3) C.AAD
receensker
dissalar thoug certer
Insincal FacuRAa Jolg 6.3 mpi edtts proces
slacching
Decinner's Promtat
cytechod bodore Bu contor
stopec the 30V
AO uhiddig
auch recarglamibe
rectungtt cerar. Exaak ahaddi wool tob:
nodiric ooken ooe
erectarafe on par.
XCY pins crchi
center cdh angge
Intemmatttie Level (L-2)
and OLTNI (OO0S!
Cel cocditen pedken WPP angia
Croda conEInAs BydE ongiee Soach unsdinghe A
Salle uccasii spamurng Dile D8 and intraaiinn
Ctedl COud KSEST Calr sunkarr
GAD Designars
engina
Shalch an paka weth ciece in cunter and
dicennrr 1S5 0.0)
JuY 0.0 1275, 0U,, (075 3755 (VO Add cacc
shan wili corden callae. allapoo
(D775 0.ty 13 0375) and DUTT
dutie suaks
Add cach canar aitors
DACAN Aeddiis
Exeert Loved 1.30 CAD
Orea the pat of
6.6. rnsenaation vense Poo oo, Edt Cauas fick ofe dkarch usin knae hov 2
owne:
Sulle
creit Cel AINO DARdiNAd puOoT rectangge keY
anglis dfD.
sredang
Crease onrer at (uars, a Lardi and matk etousss Saae the akarch 6.75 noenal disction by as dota "he dineraiona langth with
clide bes ofte
(SAL DE 75 IUS, 1U.% E3TTL
122 OZ, MuS OL3TS and C3r2) Adi cedeecee 17.77, sarnllaeder
akrzh
skinniy Extaa sketn
anta including the sxtution dipal
Figure 13: Visual results Tex12CAD on interpolated text prompts generated by GPT-4V [35]. From top to bottom, the geometric details in the text prompts increase.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section 5 (Experiment), we evaluated our method using various evaluation strategies to justify our contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We have provided limitation section of our method in the main paper. Please refer to Section (Limitation).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: We do not have any theoretical result. Our contribution is focused on novel application in CAD domains.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the experimental setup in Section 5 (Experiment). We have also provided the LLM and VLM prompts that we used in the Figure 2 and 11.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Currently we have not published code and dataset. As mentioned in the abstract, we will publish both of them soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..
wwwasssssolliiieeo onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppuiiiieeeeeeeepppppppp wctpublic//aeccccccccccccccc ggwii/gg///////eeeeee /Cae//ooeubbiiicc cetsSuummiiioo.. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the data-splits, hyperparameter details, training and inference setup in Section 5 (Experiment).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Given that there is no standardized benchmark for our task (text-to-CAD) at hand, it's not applicable for our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 5 (Experiment) and Section 10 (Implementation Details Data Annotation Pipeline).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: //neurip/// ct/ccc//p//c wtcppblccc/aaaececccceccceeaeeat wtwteettethiiieeeeeeeeeeeeeeeeee
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes have followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have provided the positive impact of our framework in Section (Introduc- tion). We are not yet aware of any negative societal impacts as of now.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will implement thorough verification processes to identify and mitigate any potential misuse of our data before releasing the annotations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited every dataset and other supporting architecturr/framewwrkk to the best of our knowledge.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new assets are generated via our data annotation pipeline. (Please refer to Section 3 (Data Annotation Pipeline).
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "j2hXirTzsd", "reviewer": "Reviewer_ovUL", "summary": "This paper proposes the first text2cad framework. This includes a CAD model annotation pipeline using VLM + LLM. And an autoregressive Transformer for generating sketch and extrude. Authors extended DeepCAD dataset with different level of text description using their method and demonstrate promising results for text-based CAD generation.", "questions": "What is the quality of the annotated text? Can authors provide some way to evaluate it? \n\nCan the cross-attention model used in the paper be modified for classifier-free training? \n\nCould the annotation be very different for the same model? How would the author address this discrepency?", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Most CAD dataset like DeepCAD and ABC do not contain text description. The challenge lies in annotation CAD mechanical parts by non-expert user. This grealy hinders the advancement of text2cad. The method proposed in this paper use the advantage of VLM and LLM to automatically annotate different level of expert textural description for DeepCAD. This is a big improvement and solves a challenging task. Authors further demonstrate promising text2cad results by training an autoregressive Transformer to generate the sketch and extrude cad parameters via cross-attention. Overall I think the motivation is strong, the annotation method is novel and beneficial for the whole community, and the initial results are very promising.", "weaknesses": "Since this is the first work for large-scale text2cad generation, there wasn’t too many baseline to compare to, this is understandable and should not be taken as a major weakness. On the other land, I think the paper could possibly improve by providing an analysis of the annotation quality. A simple test could be given one model and several text annotations, can gpt-4v selects the correct annotation corresponding to that model. This would test if the annotated text contains enough critical information for identifying the correct model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's careful consideration of our work and the thoughtful responses. Below are our responses to the specific points you raised.\n\n# Response to Weaknesses and Questions\n\n**(Q1) Reviewer:** *What is the quality of the annotated text? Can authors provide some way to evaluate it?*\n\nWe thank the reviewer for suggesting the use of GPT-4V to assess the quality of our annotations. Below, we describe the experimental setup and discuss the results.\n\n**Experimental Setup**: We provided GPT-4V with four text prompts—three incorrect and one correct—along with the corresponding minimal JSON and multi-view stacked images of the CAD model. We then asked GPT-4V to select the text prompt that best matches the minimal JSON, utilizing both its understanding of the minimal JSON and the multi-view stacked images.\n\n**Experimental Results**: We conducted this experiment using Expert level (L3) text prompts, which contain the highest level of parametric details and numerical values, ensuring a *one-to-one* correspondence between the CAD design history and the text prompts. The matching accuracy was 99.72%, indicating that the annotated texts at the Expert level (L3) provide sufficient critical information for precise 3D CAD model reconstruction. However, we found that the GPT-4V results for Abstract (L0), Beginner (L1), and Intermediate (L2) level prompts are unreliable. This unreliability arises because these prompts are designed to progressively lose parametric information to better suit different user levels. Consequently, the same text prompt can lead to different CAD models (*one-to-many*), or different text prompts can lead to the same CAD model (*many-to-one*). This variability makes matching the correct text prompts to the minimal JSON or the multi-view images a one-to-many or many-to-one problem, which is inherently ill-posed even for humans.\n\nIt is worth noting that the merit of our proposed data pipeline remains significant as it is model-agnostic. Our pipeline leverages open-sourced LLM and VLM, and during the course of this work, we utilized the best available models, namely LLaVA-NeXT and Mistral-7x8B. However, the landscape of available open-source models has evolved rapidly, with some achieving GPT-4 level performance. Therefore, replacing the current models with the latest LLM and VLM can provide annotation quality approaching the GPT-4 level. We consider this an important extension for future work.\n\n**(Q2) Reviewer:** *Can the cross-attention model used in the paper be modified for classifier-free training?*\n\nCurrently, we do not have a definitive answer to this question as we have not yet explored modifications of the cross-attention model for classifier-free training. However, we recognize this as a potential direction for future research. We would be glad to discuss this further.\n\n**(Q3) Reviewer:** *Could the annotation be very different for the same model? How would the author address this discrepancy?*\n\nOur data annotation pipeline includes different annotations (because of *top-p* sampling from VLM and LLM) for identical CAD models to enhance data diversity and capture the multiple ways users might describe these identical models. This diversity in the training samples helps our transformer model to learn to generate similar CAD models from different types of prompts. For example, in our dataset a CAD model of a pipe can be described as both \"*a pipe*\" and \"*a thin cylindrical shape with a hole.*\" **These variations reflect the real-world scenario where different users describe the same object differently based on context or personal preference**. In the attached pdf, Figure 3 shows two examples of the same CAD models being produced from three different text prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rv17tcJOGH", "reviewer": "Reviewer_hkcu", "summary": "This paper proposes Text2CAD, the first approach for generating parametric CAD models from different levels of designer-friendly language instructions. The critical challenge of generating CAD models from text is the lack of high-quality paired data. The paper's main contribution lies in a well-designed LLM-assisted data annotation pipeline, which augments the existing DeepCAD dataset with rich text descriptions at four skill levels. Based on the paired text-CAD data, the paper proposes an autoregressive transformer to generate CAD construction sequences from text inputs. Text2CAD outperforms an adapted DeepCAD baseline and produces promising qualitative results.", "questions": "In general, this paper presents an interesting and promising attempt to generate CAD models from language descriptions at various skill levels. I don't have major concerns about the data curation process and model design. For experiments, more qualitative results (including failure cases) should be included.", "limitations": "The potential social impact is discussed in Section 1 of the paper, and the limitations are properly discussed in Section 6 of the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generating CAD models from different levels of textual description is an interesting direction. Existing CAD generation methods have limited flexibility on the condition format, and this paper makes an exciting attempt that can bring practical impact. \n\n2. The data annotation pipeline is creative and reasonable. The authors leverage LLaVa and Mistral with well-designed prompts to generate different levels of text descriptions for DeepCAD data. The data quality looks good in the visualized examples.\n\n3. The Text2CAD transformer is properly designed to generate CAD construction sequence in an autoregressive manner conditioned on the textual descriptions.\n\n4. A baseline is built up by adapting DeepCAD and training with the curated Text2CAD data. The proposed Text2CAD transformer model consistently outperforms DeepCAD across quantitative and qualitative comparisons.", "weaknesses": "1. The main paper and the appendix provide only a small number of qualitative examples. More examples should be provided to better demonstrate Text2CAD's generation capacity. \n\n2.  Some typical failure examples should be presented and discussed, which could provide more insights for future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's careful consideration of our work and the thoughtful responses. Below are our responses to the specific points you raised.\n\n# Response to Weaknesses and Questions\n\n**(W1) Reviewer:** *The main paper and the appendix provide only a small number of qualitative examples. More examples should be provided to better demonstrate Text2CAD's generation capacity.*\n\nWe will provide more qualitative samples in the appendix of the final version. We have added some qualitative samples in Figure 1 in the attached pdf.\n\n**(W2) Reviewer:** *Some typical failure examples should be presented and discussed, which could provide more insights for future works.*\n\nWe have identified two types of failure cases for our model:\n\n**1. Invalidity**: In this scenario, the model fails to generate any CAD model from the text prompts. As reported in Table 1 of the main paper, this occurs in approximately 1% of the test samples. In these cases, the model predicts invalid sketch or extrusion parameters, such as the *same start and end points for lines or arcs*, or *zero values for extrusion depth on both sides.*\n\n**2. Discrepancy**: Discrepancy refers to situations where the generated model does not precisely match the shape described in the text prompts. This is relatively more prevalent in our model than invalidity and is harder to quantify. We notice that this occurs when prompts are more focused on rare object names (e.g., spatula, paddle) in the dataset rather than parametric descriptions. To showcase this, we have provided some samples in Figure 2 of the attached pdf. We will add these samples in the appendix of our final version.\n\nFor future work, we plan to address these challenges through:\n- Developing more robust parameter prediction methods by **imposing syntax-based criterion during training phase** to reduce invalidity cases.\n- **Generate more text prompts using interpolated prompt generation** (The method is shown in Section 11 and Figure 9 and 10 - Appendix) for rare objects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WezREwVdRC", "reviewer": "Reviewer_HTpR", "summary": "The paper proposes Text2CAD, a framework designed to expedite the prototyping of complex computer-aided design (CAD) models. The proposes method uses designer-friendly text instructions to generate parametric CAD models, making it accessible for all skill levels. To facilitate this, a data annotation pipeline is introduced, which generates text prompts based on natural language instructions for the DeepCAD dataset using Mistral and LLaVA-NeXT. The Text2CAD employs an end-to-end transformer-based autoregressive network to generate parametric CAD models from input texts. Authors have evaluated the performance of their method through a combination of metrics. s.", "questions": "Please check the weaknesses section for the questions.", "limitations": "Authors have provided the limitations statement.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Authors have done nice work to simplify the annotations in natural language\n- The use of VLM and LLM in generating prompts for CAD images is an interesting direction.\n- The results with the proposed method are decent.", "weaknesses": "- The authors assert that the proposed method Text2CAD is the first AI framework for CAD image generation. However Text2CAD is not the first AI method for generating CAD images. A previous work **SketchGen [1]** exists which was the first to generate CAD images. Therefore, authors should include this work and enlist how different is their work compared to SketchGen. Surprisingly SketchGen also generates CAD sketches auto-regressively, which makes the novelty of proposed approach weaker.\n- While there are 4 levels of language prompts used in the method. However, there is no separate analysis on how each type of prompt is helping the final generation. There is an ablation on alignment module, however, there is no analysis on the use of 4 prompts. This is important to validate the role of each prompt in aiding the generating process. I am just wondering why we need 4 levels of prompt abstractions. In general, the baselines used to compare the results are limited.\n- While the layerwise cross attention is not a novel component in the proposed method, did the authors try to explore alternative methods for information fusion?\n\n\nAuthors are encouraged to provide solid explanations to the above queries.\n\nReferences:\n[1] Wamiq Para et al. \"Sketchgen: Generating constrained cad sketches\". NeurIPS 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's careful consideration of our work and the thoughtful responses. Below are our responses to the specific points you raised.\n# Response to Weaknesses and Questions\n**(W1) Reviewer:** *The authors assert that the proposed method Text2CAD is the first AI framework for CAD image generation. However...SketchGen also generates CAD sketches auto-regressively...*\n\nWe appreciate the reviewer's comments and welcome the opportunity to clarify the fundamental contributions of our method. Our approach, Text2CAD produces parameterized 3D CAD models directly from textual descriptions where SketchGen[1] only generates 2D sketches. This capability supports our claim as pioneers in the field of parametric CAD model generation from text. Additionally, the 3D CAD models created by our method can be further edited using commercial CAD software. Below, we outline the key distinctions and novel contributions of Text2CAD compared to SketchGen[1].\n\n1. **Scope and Focus - *Generative vs Modality-Translation***\n   - SketchGen is a generative network focused on generating 2D-constrained CAD sketches. It tackles a *generative-modeling* problem for 2D sketches.\n   - Text2CAD is the first AI framework to generate parametric 3D CAD models in *a sketch-extrude* format conditioned upon textual descriptions. It tackles a *modality-translation* problem.\n\n2. **Input Modality - *Unconditional vs text***\n   - SketchGen generates 2D sketches from scratch (unconditional generation) due to its generative nature, but its real-world applicability is limited due to the lack of user-guided sketch generation.\n   - Text2CAD leverages text prompts to guide the generation of 3D CAD models, enabling more practical applications in CAD design workflows.\n\n3. **Output Modality - *2D vs 3D***\n   - SketchGen produces only 2D sketches with constraints. Since extrusion parameters are not predicted, the final models are not actual 3D CAD models. These can be only visualized as an image.\n   - Text2CAD generates actual 3D CAD models in *sketch-extrude* format which can be visualized as boundary representations (Brep) or meshes and be further edited in CAD software.\n\n4. **Novelty of Text2CAD**\n   - Text2CAD's novelty lies in its ability to interpret text descriptions and generate parametric 3D CAD models, which involves a different set of challenges compared to generating 2D-constrained sketches from scratch.\n\nWhile both works utilize autoregressive generation, their scope, input, output, and applications are fundamentally different. We appreciate the reviewer's suggestion to include this work. **We already included SketchGen[1] in Line 120 (Section 2 Related work) of the main paper** and we will update the explanation on how Text2CAD differs in the final version. We are happy to discuss more and clarify any further questions on this matter.\n\n**(W2.1) Reviewer:** *While there are 4 levels of language prompts used in the method. However...why we need 4 levels of prompt abstractions.*\n\nBelow, we explain the role and importance of each of the four levels of prompts within our framework.\n\n1. **Single Input Prompt for Training and Inference:** Figure 3 in the main paper may imply that all four prompts are used simultaneously to generate the final CAD model. However, our model uses only a single text prompt of any level as input during both training and inference. In Figure 3, the four prompts are only for visualization purposes. In reality, these are four different training instances. Each training instance uses one prompt level at a time, resulting in ~600k training samples, as mentioned in Line 250-252 (Section 5) of the main paper. We will update the Figure 3 caption in the final version to clarify this.\n\n2. **Purpose of Four Prompt Levels:** The four levels of language prompts—abstract, beginner, intermediate, and expert—were developed during our data annotation process to accommodate users of all skill levels. Unlike the text prompts in existing text-to-3D methods, which are typically more object-centric, text-to-CAD methods must handle a range of prompts, from simple object-centric descriptions to detailed parametric instructions (Line 48-53 of the main paper).\n\n3. **Analysis of prompts:** Since each text prompt, regardless of its level, can independently generate the final CAD model, there is no context of *helping the final generation*. However, in Table 2 of the main paper, we have provided the performance of our model on each level separately using both GPT-4V and Human evaluation.\n\nWe hope this clarifies the reviewer's question. **In summary, our model can generate a 3D CAD model from an abstract shape description (e.g, generate a star) or a detailed parametric one. It doesn't require all four level prompts**. We are happy to discuss more on this matter.\n\n**(W2.2) Reviewer:** *In general, the baselines used to compare the results are limited.*\n\nWe acknowledge the limited baselines in our paper. We agree with **Reviewer ovUL**'s understanding of the challenges in establishing baselines - *Since this is the first work for large-scale text2cad generation, there wasn’t too many baseline to compare to, this is understandable and should not be taken as a major weakness* (ovUL).\n\n**(W3) Reviewer:** *While the layerwise cross attention is not a novel component in the proposed method, did the authors try to explore alternative methods for information fusion?*\n\nWe acknowledge that layerwise cross-attention mechanism is a well-established technique. In our current work, we have primarily focused on demonstrating the feasibility and effectiveness of the Text2CAD transformer using this approach. However, we recognize the potential for improvement in this area. In future research, we plan to explore alternative fusion mechanisms more specifically suitable for text-to-CAD domain for integrating textual and geometric information.\n\n\n[1] Wamiq Para et al. \"Sketchgen: Generating constrained cad sketches\". NeurIPS 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLKeiDocoe", "reviewer": "Reviewer_mcF8", "summary": "This paper investigates an interesting task (Text2CAD) in CAD automated applications, which achieves generating parametric CAD models from text prompts. Specifically, the authors first introduce a data annotation pipeline to generate suitable text prompts for CAD models in public DeepCAD dataset (including about 170K CAD models) via conducting LLM models such as Mistral and LLaVA-NeXT. The generated text prompts mainly consist of abstract CAD descriptions and detailed specifications. Second, the authors further design a transformer-based auto-regressive network to learn the gap between generated text prompts and CAD models. Finally, the effectiveness of proposed method is validated under a mixture of metrics to show its potential in AI-aided CAD design.", "questions": "Q1: Given the design of network architecture, it seems that both text prompts and CAD sequences need to be used as inputs for forwarding. But in the inference stage, text prompts as only inputs for generating CAD models according to Figure 5. The reviewer is curious about is this just not showing the CAD tokens in Figure 5 or CAD tokens is completely unused during the inference stage? This is crucial for text2CAD generation problem, as it only makes sense when only text prompt is used as input.\n\nQ2: For those numerical numbers within CAD commands, it is very difficult to encode accurately, which is still an open problem in the NLP. Would author kindly further give more details about how to encode them and maintain their validity after decoding? This is important to demonstrate the proposed framework is able to generate CAD models precisely from textual prompts. \n\nQ3: The reviewer puts other questions in Weaknesses and Limitations.", "limitations": "L1: This paper does not discuss the diversity and bias of the generated textual prompts, is it possible to generate the same/different CAD models using similar/different style of textual prompts? Discussing this issue would be better to show how the framework performs.\n\nL2:  Given the experiments are only conducted on DeepCAD dataset, it would be better to show the extend results on other datasets to demonstrate its generalized capability.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: The paper is well organized and easy to understand.\n\nS2: The visualization of data annotation pipeline is clear to view.\n\nS3: The structure of the paper is clear and the logic is rigorous.", "weaknesses": "W1: Some typos need to be corrected (“Computer-Aided Design (CAD) play”).\n\nW2: Insufficient quantitative experiments are conducted, which currently appears to be less impactful.\n\nW3: The figures in the manuscript could be enhanced. For example, it would be better to use more colors to show the generated CAD model.\n\nW4: Lack the training computational costs and the inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's careful consideration of our work and the thoughtful responses. Below are our responses to the specific points you raised.\n\n# Response to Weaknesses\n\n**(W1) Reviewer:** *Some typos need to be corrected ...*\n\nWe have corrected the typo.\n\n**(W2) Reviewer:** *Insufficient quantitative experiments are conducted, which currently appears to be less impactful.*\n\nWe acknowledge that this is a limitation in the text-to-3D domain and remains an area of active research. Given the lack of standardized benchmarks in text-to-3D domain that can be directly applied to text-to-CAD tasks, we assess the quality of generated CAD sequences against ground truth, using a multi-faceted evaluation approach. This includes Sequence-Level, GPT-4 [1], and Human evaluations to provide a comprehensive assessment of our model's performance.\n\n[1] Wu et al. GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation, CVPR 2024\n\n**(W3) Reviewer:** *The figures in the manuscript could be enhanced...*\n\nIn the final version, we will update the colors for better visibility.\n\n**(W4) Reviewer:** *Lack the training computational costs and the inference time.*\n\n**1. Training Computation Cost:**\n- **GPU**:  1 A100-80GB Nvidia GPU\n- **Training Time**: 2 days.\n- **Trainable Parameters**: ~23M\n\n**2. Average Inference Time per Sample:** ~ 0.3s (RTX 4090)\n\n# Response to Questions\n\n**(Q1) Reviewer:** *Given the design of network architecture, it seems that both text prompts and CAD sequences need to be used as inputs for forwarding...*\n\n**Training:** Yes, we train the model using both the *CAD sequence* and *text prompt*. We use a teacher forcing strategy which is a common practice to train an autoregressive network as it converges faster. Since our model architecture is autoregressive, it learns to predict the next token in the *CAD sequence* given the textual contexts.\n\n**Inference:** However, during inference, our model only requires *text prompts* as input. Since the architecture is autoregressive, the model doesn't generate the complete CAD sequence in one forward pass. For example, the model takes as input: $(C_{1:1}, T_{adapt})$, where $C_{1:1}$ is the *start token* and $T_{adapt}$ is the adapted BERT encoding of the text prompts. The model then outputs: $(C_{2:2})$, where $C_{2:2}$ is the second token of the output sequence. In the next iteration, it takes as input: $(C_{1:2}, T_{adapt})$, and outputs: $(C_{3:3})$, the third token of the output sequence. This process continues until the end of sequence token is generated. Hence, during inference only the *start token* and the user's *text prompt* are required to generate the complete *CAD sequence*. This complete CAD sequence can then be rendered into a 3D CAD model (which is shown in Figure 5).\n\nWe concur with the reviewer that *it only makes sense when only text prompts used as input.* Accordingly, our model generates the final CAD sequence exclusively from *text prompts*, using a text-conditioned autoregressive approach.\n\n**(Q2) Reviewer:** *For those numerical numbers within CAD commands, it is very difficult to encode accurately, which is still an open problem in the NLP...*\n\nWe acknowledge the reviewer's point about the challenges in encoding numerical values in NLP. To address this, our approach avoids direct arithmetic operations on numbers extracted from text. Instead, we discretize the numerical values in our ground truth CAD sequences. Our model is then trained to predict these discrete values directly from the text prompts (e.g., \"$0.0208$\" → $142$).\n\nAs all text prompts are encoded using the BERT encoder, it is possible that BERT may encode some of the numerical values (e.g., \"0.0208\", \"-0.1512\") into the *UNK* token. To mitigate this issue, we use the Adaptive layer and downsampling layer. These components fine-tune the text features of BERT to better align with continuous values, specific vocabulary, and structural requirements of CAD instructions. The Layerwise Cross-Attention mechanism in the decoder learns the mapping between quantized and continuous values.\n\nThe effectiveness of our approach is conclusively demonstrated in Table 1. Our model achieves low Chamfer Distance (CD) for expert-level prompts (L3), which contain the highest number of continuous values and complex geometric descriptions. The lower CD indicate that the reconstructed CAD model is very similar to the ground truth CAD model. This is further supported by our ablation study, where directly using BERT encoding without our specialized layers results in higher CD. These results indicate that our model successfully encodes numerical values and maintains their validity after decoding.\n\n# Response to Limitations\n\n**(L1) Reviewer:** *This paper does not discuss the diversity and bias of the generated textual prompts, is it possible to generate the same/different CAD models using similar/different ... prompts?*\n\nThe diversity of our generated textual prompts is influenced by dataset variety and the performance of Mistral and LLaVA-Next. We note in Section 6 of the main paper that the DeepCAD dataset, the only large-scale dataset with a full design history, predominantly features simpler rectangular, and cylindrical shapes. To enhance prompt diversity, we've shifted focus from generating mere object names to detailed shape descriptions using VLM. For instance, *a ring* might be described as *a circular object with a cylindrical hole in the center*. This approach allows for the generation of identical CAD models using varied textual styles. In the attached pdf, Figure 3 shows examples of the same CAD models being produced from diverse text prompts. \n\n**(L2) Reviewer:** *Given the experiments are only conducted on DeepCAD dataset...*\n\nTo showcase our cross-dataset performance, we will conduct experiments on the Fusion360 [1] dataset where both DeepCAD and Text2CAD are trained on the DeepCAD dataset. We will add it to the main matter in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their thorough evaluation and insightful comments. We are delighted that the reviewers recognize the significance of our work.

1. Text2CAD is acknowledged as the *first framework* (**ovUL**) for generating parametric 3D CAD models *from different levels of designer-friendly language instructions* (**hkcu**).
2. Our work is seen as having high potential for *practical impact* in the CAD industry and design processes (**hkcu, ovUL**).
3. Our *novel data annotation pipeline using VLMs and LLMs* is praised for its creativity and effectiveness in generating high-quality paired text-CAD data (**ovUL, hkcu, HTpR**). *This is a big improvement and solves a challenging task* (**ovUL**). Also this *is an interesting direction* (**HTpR**).
4. The autoregressive Text2CAD transformer model is *properly designed* (**hkcu**) for CAD construction sequence generation from text prompts. *The initial results are very promising* (**ovUL**) and the *model consistently outperforms DeepCAD across quantitative and qualitative comparisons*. (**hkcu**)
5. *The paper is well organized and easy to understand* and *the structure of the paper is clear and the logic is rigorous* (**mcF8**).


Our work makes a significant contribution to the field of AI-assisted CAD modeling and opens up a new research domain in text-to-CAD modeling. We appreciate the reviewers' constructive feedback and have addressed each point in the individual responses below. The comments and subsequent revisions have strengthened our paper considerably. We hope we have answered all the questions properly within the given time-frame and are open to further discussion if additional questions arise. Please find attached the pdf for the figures.

Meta-review: The paper introduces Text2CAD, a framework for generating parametric CAD models from text prompts. The method combines a novel data annotation pipeline and an autoregressive transformer-based model. Reviewers appreciated the approach novelty and its potential for real-world impact in CAD design. However, concerns were raised about limited baselines, insufficient qualitative examples, and the absence of detailed analysis on prompt effectiveness. The authors addressed these concerns with additional experiments and clarifications during the rebuttal, which were well-received by the reviewers. Consequently, all reviewers vote for acceptance.

The AC considered the reviewers’ feedback, the authors’ responses, and the overall quality of the work. Given the novelty, practical relevance, and satisfactory rebuttal, the AC recommends an "accept" decision for the paper. Authors are reminded to include all suggested changes in the final version.
Decision: Accept (spotlight)