GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors
Authors: Md Ferdous Alam
Faez Ahmed
Venue: Accepted by TMLR ()

Abstract: The creation of manufacturable and editable 3D shapes through Computer-Aided Design (CAD) remains a highly manual and time-consuming task, hampered by the complex topology of boundary representations of 3D solids and unintuitive design tools. While most work in the 3D shape generation literature focuses on representations like meshes, voxels, or point clouds, practical engineering applications demand the modifiability and manufacturability of CAD models and the ability for multi-modal conditional CAD model generation. This paper introduces GenCAD, a generative model that employs autoregressive transformers with a contrastive learning framework and latent diffusion models to transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-art methods in terms of the unconditional and conditional generations of CAD models. Additionally, the contrastive learning framework of GenCAD facilitates the retrieval of CAD models using image queries from large CAD databases, which is a critical challenge within the CAD community. Our results provide a significant step forward in highlighting the potential of generative models to expedite the entire design-to-production pipeline and
seamlessly integrate different design modalities.

Published in Transactions on Machine Learning Research (04/2025)
GenCAD: Image-Conditioned Computer-Aided Design Gen- eration with cansformer-Based Contrastive Representation and Diffusion Priors
Md Ferdous Alam
mfalam Amill edu
Department of Mechanical Engineering Massachusetts Institute of Technology
Faez Ahmed
faezumit. edu
Department of Mechanical Engineering Massuchusells Institute of Technology
Reviewed on OpenReview: /ttps://openreeet net/ forum? id= e817c1w526
Abstract
The creation of manufacturable and editable 3D shapes through Computer-Aiden Aided Design (CAD) remains highly manual and time-consuming lask, hampered by the complex topol- ogy of boundary representations of 3D solids and unintuitive design tools. While most work in the 3D shape generation literature focuses on representations like moshes, voxcls, or point clouds, practical engineering applications demand the modifiability and manufactura, bility of CAD models and the ability for multi-moda conditional CAD model generation. This paper introduces GenCAD, generative model that employs autoregressive transform- ers with a contrastive learning framework and latent diffusion models transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representa- tions. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-arl methods in terms of the unconditional and conditional generations of CAD models. Additionally, the contrastive learning framework of GenCAD facilitates the retrieval of CAD models using image queries from large CAD databases, which is a critical challenge within the CAD community. Our results provide a significant step forward in highlighting the potential of generative models to expedite the entire design-to-produe to-produntion pipeline and seamlessly integrate different design modalities.
1 INTRODUCTION
Generating 3D shapes is a fundamental part of the engineering design process that morphs the design from creative intuition to digital 3D shapes. Professional engineers use modern Computer-Aided Design (CAD) models to digitally represent 3D shapes for applications in automotive, aerospace, manufacturing, medical devices, consumer products, and architectural designs. Engineering design of such 3D solid modeling is a complex soquential process that requires human expertise and intuition. Since the invention of CAD, most CAD modeling tasks have been manual and typically go through long iterative process finalize design with desired requirements. Additionally, modern CAD software is highly non-intuitive for humans and comes with steep learning curve for most practitioners (Piegl 2005). Based on the motivations of recent literature, we argue that some portion of the 3D CAD modeling process can be automated using lcarning- based approaches, which can offer tremendous acceleration in the design and manufacturing pipeline. An important first step in this direction is to build generative models that can provide valid and real-world 3D CAD models based on user intuitions. As most of the generative model architectures are proposed for image or text generation, carefully designed framework needed for 3D CAD modeling in engineering computational design tasks.

---

Published in Transactions on Machine Learning Research (04/2025)
000
O00 O
Gnometry Kernel
sketch
GenCAD
Image
CAD program
000
ODD O
Figure 1: GenCAD demonstrates generative CAD models conditioned on images or sketches, where the CAD model generated sequentially using a language-like representation of CAD operations
Generalive models, built upon large neural network architectures, have recently shown impressive perfor- mance in image and text generation tasks (Ramesh et al., 2021, Razavi et al. 2019 Saharia et al. 2022; Karras et al. 2019; Rombach et al.) 2022). To align the outputs of these generative models with user in- tentions, they are often conditioned on additional image or text input. Inspired by the success of text and image modalities, many recent studies have focused on building generative models for 3D representations
such as meshes (Geuzaine & Remacce 2009 Feng et al., 2019), voxels (Wang et al., 2017 Alam et al., 2025
point clouds (Qi et al., 2017 Qin et al. 2019),, and implicit representations (Jun & Nichol 2023). These
applications do not include CAD modeling tasks that require edilable and monufacluuuble representation of 3D solids. One of the fundamental challenges in building generative models for CAD is the underlying data structure of the 3D solid model. Since the 1980s, the industry standard in Computer-Aided Design (CAD) has been the use of boundary representations (B-rep) (Weiler) 1986) that encodes geometry information in the parametric representation of surfaces, odges, and vertices. Although B-rep is more complex than other types of 3D shape representations, offers several benefits. For example, the boundary representation of the and memory-intensive such as meshes, voxels, or point clouds while also being easier to render, unlike their counterparts, such as implicit representations. Additionally, the existing off-the-shelf geometry kernels can scamlessly convert parametric CAD command sequence or a CAD progran into a B-rep, which makes this representation the de facto industry standard. However, due to the complex topological relationship between geometric entities, B-rep data structure not directly suitable as input for neural network architectures, and an intermediate network-friendl representation must be utilized. Few recent studies have developed CAD datasets that are neural network friendly, which makes it feasible to build powerful generative models for directly generating B-reps or CAD models (ooch et al., 2019 Wu et al. 2021 Willis et al. 2021 2022). Most of these approaches utilize one of the following: graph-based representation of 3D solid models (Jayaraman et al. 2021 Lambourne et al. 2021 sequen- tial representation of CAD modeling operations [Xu et al.] 2022
3D solid model is not resolution-dependeee
Wu el al. 2021), or a specialized variant
of the B-rep such as indexed list data structure Jayaraman et al. 2022).
Learning to generate the CAD modeling sequence can provide more usefulness than directly learning generate a B-rep. We argue that direct B-rep generation is less attractive as it does not encode the underlying design history. The sequence of solid modeling operations, CAD program, is critical to modern CAD software and offers more flexible and interpretable representation than the direct B-rep. More specifically, B-rep model can be thought of as a sequence of parametric CAD commands that can create a 3D solid shape using an off-the-shelf geometry kernel. Although this representation of 3D CAD straightforward and scalable, most available literature focuses on the unconditional generation of CAD programs, which is not aligned with any user input. We argue that these unconditional generative models lack the true potential of generative CAD models for useful 3D solid modeling tasks. To align the generative CAD models with user intention, we propose a new type of neural network-based architecture that is conditioned CAD images. The idea of inage-to-CAD conversion or image-lo-technical drawing conversion is of particular
A sequence of parameterized CAD madeling operations is denated CAD program
2

---

Published in Transactions on Machine Learning Research (04/2025)
importance to the design community (Dori & Tombre 1995 Dori & Wenyin 1999 Nagasamy & Langrana, 1990 Gümeli et al. 2022 Majumdar & Seethalakshmy 1997). Although several attempts have been made in the past, most of the approaches use heuristics and are application-speciii,,, which makes them difficult to generalize. Our approach is largely motivated by the recent success of text-to-image and imago-to-mesh
literature (Alliegro al. 2023 Tyszkiewiz et al., 2023 Xu el al. 2024a).
Here we focus on the problem of learning to generate CAD programs in terms of parametric command sequences based on CAD images. We propose new generative model architecture, generative CAD (Gen- CAD), that can sequentially generate entire CAD program aligned with input CAD-imap Figure 2 Our main idea is to learn the joint distribution of the latent representation of the CAD command sequences and the CAD images or sketches using a contrastive learning-basee approach. Next, we develop a conditional latent diffusion model as a prior network that can generate CAD latent conditioned on the imput image latent. Finally, we present a transformer-basee the input CAD latent. Note that the final output of GenCAD is not merely 3D solid model but a CAD program; which is an entire sequence of parameterized CAD commands. This is critical because the CAD command sequence can be converted to B-rep models other convenient representations such as mesh, point clouds, or voxels using any off-tho-shelf geometry kernel. This trivial conversion allows GenCAD to be directly useful in generating conceptual 3D engineering design which has the potential to automate many complex 3D solid modeling tasks such as the reverse engineering of CAD models from image or sketch inputs [Toopson et al. [999). Another major benefit of GenCAD is its ability to successfully perform image- based CAD program retrieval tasks, major challenge in the engineering design community. We show that our framework can utilize the learned joint representation of CAD images and CAD command sequences to retrieve CAD programs using image input. Our contributions can be summarized as follows:
shown in
decoder model that can create the CAD program from
We develop a transformer-based autoregressive model for representation learning of CAD sequences and show that our autoregressive model is more accurate in reconstructing CAD sequences than state-of-the- art models
We propose and develop GenCAD, an image and sketch conditional generative model for CAD, and show that conditional generation of CAD outperforms unconditional models in terms of diversity, fidelity, and statistical distance
We show that contrastive learning enables image-based CAD retrieval, which is more than 15x accurate in retrieval when compared to image-to to- to-image search
2 RELATED WORK
2.1
CAD as a language modeling problem
The recent success of deep neural networks for language modeling tasks (Brown et al., 2020 Vaswani et al. has allowed many domains to formulate language-like problems for specific applications. Although learning design intent and developing intelligent CAD systems have been topics of interest since the 1990s
2017
[Ault 1999 Ohsuga 1989), treating CAD as a language modeling task is a somewhat recent approach. Ganin ct al. 2021 showed that CAD can be converted to a language modeling task by considering protocol buffers (PB) for describing CAD sketch structures. The authors used PB to describe various sketch entities and constraints. Extending this idea to 3D is non-trivial due to the requirement of specific protocol buffers for 3D CAD commands. Similarly, another study by Para et al. (2021) focused on developing language- problem formulation for CAD sketches that can handle various primitives and constraints. This approach is also limited to 2D sketches, and the scalability of the approach to 3D CAD is also non-trivial. Wu et al. (2021) also locused on converting CAD as a language modeling problem. This approach utilizes the transformer architecture for sequential 3D CAD generation and develops a simple vocabulary for CAD commands. This approach capable of both sketch and 3D operation but only limited number of operations. Note that the generative model proposed by Wu al. (2021) is entirely unconditional and does not allow any user input during generation.
3

---

Published in Transactions on Machine Learning Research (04/2025)
CI
C2
C2
02
Transformer Encoder
Transformer Encoder
Transformer Decoder
Image Encoder
CCAD
FN
ex
ZCAD
Zing
Step 1: Command Sequence Reconstruction (CSR) model
Step 2: Contrastive CAD-Image Pre-training (CCIP) model
image
C1
Image Encoder
Transformer Decoder
C2
Bnd o
Diffusion Prior
Geometry Kernel
Zing
ZCAD
ZCAD
CN
B-rep
Step 3: CAD Diffusion Prior (CDP) model
Step 4: CAD Decoder model
The proposed framework, GenCAD, consists of four steps: 1) transformer- based encoder-decoder architecture is trained autoregressively to learn the latent representation of the vectorized CAD commands, 2) contrastive-learnin based model is used to learn the joint representation of the latent space of CAD command sequence and CAD image, 3) image conditional CAD generation can be achieved by sampling CAD latents from the diffusion model conditioned on image latents, and 4) using the trained transformer decoder to predict the CAD commands autoregressively. Note that * denotes a frozen model that is not updated during training.
Figure GenCAD:
2.2 Datasets for CAD
Unlike other representations of 3D shapes such as meshes, point clouds, voxels, implicits, there is lack of large-scale datasets for CAD. There are only a few publicly available CAD datasets, among which the ABC dataset is the largest, containing 1 million CAD models obtained from online public repositories (Koch) et al. 2019). This dataset contains B-reps but does not provide any design histories or labels. Later the CAD dataset was developed by cleaning the ABC dataset (Wu et al. 2021). The DeepCAD dataset provides design history by parsing each CAD design from the Onshape public repository. In addition this, the Fusion 360 dataset provides both design history, assembly, and face segmentation although the
dataset is quite small for learning generalizable models (Willis et al., 2021 2022 Lambourne al. 2021 Additionally, the MFCAD and MFCAD++ datasets provide machining feature recognition data in terms of B-reps Cao et al.
2020
Colligan et al. 2022)
2.3 Generative models for CAD
In addition to learning specific tasks directly from B-rep data, recent studies have also developed generative
models by directly synthesizing from B-rep data structure (Xu et al. 2022 Jayaraman al. 2022
Xu al.
2024b). Recent generative approaches mostly focus utilizing transformer- based architecture for generating
unconditional sketches (Xu et al. 2022 Para et al. 2021) 3D CAD (Xu et al. 2022 Wu et al. 2021 Xu
ct al. 2024b; Jayaraman et al.] 2022). In contrast to directly synthesizing the B-rep, the CAD generation
process (CAD program) is of crucial importance due to its potential in the automation of many design lasks.
Most importantly, the parameterized CAD operations or commands contain valuable design history. [Wu] ct al. (202)) developed DcepCAD, a transformer-base latent generative adversarial network (I-GAN) model, that can generate unconditional CAD. Most of the generative CAD approaches only consider unconditional generation specific data structures such as graph representation from boundary representation of CAD
4

---

Published in Transactions on Machine Learning Research (04/2025)
Line
Line
Line
Arc
Circle
Extrude
Geometry Kernel
C1
C2
C CN
Figure 3: CAD as a language modeling problem using geometry kernel: Our main idea is based on the fact that real-world CAD design is sequential and a learning-base baaning-based approach should capture the correlation in the sequential design from a large-scale dataset. This sequential approach converts the CAD problem into a language modeling problem where vector representation of CAD commands, C; E R are fed into off-the-shelf geometry modeling kernel to sequentially create the 3D solid geometry such Boundary-ropresennatin (B-rep). In this toy example, the sequence is follows: line line line - circle + extrusion.
models. To align CAD model generation with user intention, a conditional input, such as an image, is unlded-unlike the models developed in previous studies. To the best of our knowledge, there has been no generative model reported in the literature that can provide an entire CAD program from an image input.
3 METHOD
In contrast to the studies discussed in section 2 our main motivation for this work to develop a learning- based approach that is scalable and generate conditional 3D CAD. More specifically, we interested in creating CAD programs that are conditioned on images. In the following, describe our main contribution, four-step framework, for generating image-conditional CAD models.
3.1 From natural language to CAD language
To represent the CAD program in a neural network-friendee representation, we opt for a natural language-like representation of a CAD program. The key idea is to represent a CAD model as a sequence of parameterized CAD commands that can be fed into a standard geometry kernel to create the 3D solid model. The goal of the parametcrization is to allow the CAD command to have fixed-dimensional vector reprosentation. Each of the CAD commands is analogous to tokenization in natural language processing (NLP) tasks. Similarly, the fixed dimensional vector representation of the CAD command can be thought of as the embedding of token in NLP tasks. A major difference between the tokenization of words and the tokenization of CAD commands that the CAD commands are parameterized where the parameters ensure geometric consistency. Fach CAD command represents the type of CAD operation that being used and the associated parameters required to perform this CAD operation. An example of this mechanism shown in Figure B where the vectorized CAD commands are fed into geometry kernel to create the 3D solid geometry sequentially. Note that the type of the CAD operation is discrete and is similar to NLP tokens, but the parameter values are continuous.
Here we closely follow the CAD command representation criteria proposed by Wu et al. (2021). Specifically, each CAD command, c; E R!7, can be represented as c, = (t,, P,) where t, E R represents the command type and P; R represents the parameters of each command. Note that cach CAD command has different number of parameters but they can be converted fixed dimensional vector by concatenating additional parameters and masking the values for unused parameters. Additionally, the parameters are carefully chosen
5

---

Published in Transactions on Machine Learning Research (04/2025)
so that they are sufficient for the geometry kernel to create the geometric entity. The types of CAD tokens used in this study are described in the following.
Special tokens: We define two special tokens (SOL) and (EOS) to represent the start of a loop of a sketch and end of sequence respectively. We draw motivation from the NLP literature which also defines special tokens to represent the start of a sentence or end of a sentence. These special tokens do not require any parameters and only indicate the state of the CAD geometry.
We define three tokens to represent sketch; Line token, Circle token, and Arc token. In geometry kernel, a line can be generated using only the end point coordinate, (æ, v), if the start point is known. As we are generating a CAD sketch sequentially, each sketch command provides us with the starting position for the next command. A circle can be generated based on the following three parameters; the center of the circle (z, y) and the radius of the circle, r. Similarly, an arc can be represented by the four parameters: end point of the arc; y); sweep angle, a and directional flag of the arc, which represents either a clockwise or counter-clockwise arc.
Sketch token:
One of the most complex yet widely used operations in CAD is the extrusion command. Representing extrusion requires keeping track of a sketch plane, extrude direction, and distance. To represent the current sketch plane we use these six parameters: three for the orientation of the current sketch plane (0, o, 7) and three for the origin of the current sketch plane (Pa.Py. P=). Additionally, a scale parameter is used to represent the scale of the sketch profile 8. Extrude distance can be represented by two parameters e2) for each side of the sketch. Finally, to sequentially generate the solid, we need either create a new solid body join, cut, and intersect the extruded body from the existing solid body. So, we use boolean paramcter to indicate the type of operation of the body based the extrude command. Finally, we use another parameter to represent whether it is a one-sided or two-sided extrude. As a result, the extrude command is represented using a total of 10 parameters.
Extrusion token:
3.2 Dataset
Here use the DeepCAD dataset (u et al. 2021 to train and evaluate all of our models. This dataset is created based on the ABC dataset which is obtained from the publicly available human CAD designs from Onshape Inc. (Oushape 2007). Note that the ABC dataset only contains the B-rep data and does not provide any design history. The DeepCAD dataset is created by parsing the design history of each CAD using the Onshape API. As CAD designs are complex and can be difficult to learn for sophisticated 3D shapes, this dataset is limited to sketch and extrude operations to make more approachable for neural neaeer models. Thus, this dataset does not contain other CAD operations, such edge operations fillets/chamfeet revolve, and mirror. The sketch operation is also limited to lines, circles, and arcs. After filtering out the ABC dataset based on these operations, the DeepCAD datasct contains 178, 238 CAD designs and is the largest CAD dataset with design history. Not all of these designs can render a 3D solid which makes it difficult to extract CAD images for our purpose. We filter this dataset based on 3D solid creation using an off-the-shelf geometry kernel, Open Cascade, and end up with a total dataset of 168, 674 CAD models with command sequences. Among these, we use 152, 530 for training, 8515 for validation, and 7629 for testing for all the models reported in this study. For CAD images, we create five different versions of each CAD model by changing the scale of the solid model in the x, y, and 2 axes. Each image is grayscale and has dimensions of 1 x 448 x 448. Note that some of the scaling operations do not create a valid CAD and we omit those images. We end up with 845, 105 images obtained from the CAD dataset. Similar to the CAD images, we also create a CAD sketch dataset by taking the canny edge of these images and adding Gaussian blur. The sketch dataset consists of 845, 105 CAD sketches.
3.3 GenCAD: conditional generation of CAD programs
Using the CAD language modeling formulation in section 3.1. we propose Generative CAD (GenCAD), a four-step framework for conditional CAD program generation. First, we develop an autoregressive trans- former encodor-dccoder model, the Command Sequence Reconstruction (CSR) model, using CAD command sequences as input learn the latent representation of the final 3D solid geometry model. Second, we develop Contrastive CAD-Image Pretraining (CCIP) model with ResNet-based image encoder and con-
6

---

Published in Transactions on Machine Learning Research (04/2025)
CAD latent
noised latent
ZCAD
Forward diffusion process
Z
Head
Nx
Linear
Activation
BatchNorm
Dropout
Lineal
Dropout
Activation
Linear
BatchNorm
Projection
image latent
ZCAD
z
Zing
Denoising model
Figure 4: CAD Diffusion Prior (CDP):
The diffusion prior takes CAD latent as well as optional image
latent input. The denoising model of the diffusion prior is based on MLP-RReNNct architccture proposod by Gorishniy et al. (2021)
trastive loss to jointly learn the latent representation of the CAD image with the latent representation of the CAD command sequences from step Third, we develop a CAD diffusion prior (CDP) model that can generate CAD latents, and optionally take image latents, from step 2, input. Finally, the CAD latent samples generated from the CDP model are fed into a transformer-based decoder model that can generate the entire CAD command sequence autoregressively. To make our approach scalable for large-scale datasets and models, we utilize the pre-trained CAD encoder from step during the training of the CCIP model instead of training from scratch. We also use the pre-trained CAD decoder model from step for decoding the CAD latent. from the CDP model. Once, we have the sequence of CAD commands, we can use any off-the-shed geometry kernel to obtain the final 3D solid model. By combining these four steps, we show that our approach can generate valid CADs conditioned image inputs which is shown in Figure 2
3.3.1 Command Sequence Reconstruction (CSR): Latent representation learning for CAD programs
The first step of our method is to learn underlying latent representation of the CAD program the command sequences. This is analogous to the pre-training of modern large language models (LLM) for downstream applications. We draw motivation from the recent success of large-scale generative pre-trained models (GPT) (Brown et al. 2020 Radford et al. 2019) and propose model for lcarning the CAD commands. The main idea is to predict the next CAD command at each timestep of the sequence. This autoregressive training done by observing the previous CAD commands within the sequence and masking all the future commands at each timestep. We use standard transformer encoder and decoder network (Vaswani al. 2017) with causal masking for both the CAD sequence encoder and the decoder. As the command parameters contain discrete and continuous values, opt for learning the distribution in the latent space instead of the parameter space.
transformer-based autoregressive
As mentioned in section 3.1 the parameters of each CAD command can be a combination of discrete and continuous values. To unify these types of parameters, we quantize their values into 256 levels and express them using 8-bit integers similar to Wu et al. (2021). Next, we organize each command vector onto a continuous space by adding an embedding layer before the encoder network. The output of the embedding layer is a di-dimensi vector which is fed into the transformer encoder. We use d_ 256 for all experiments reported in this study. The transformer encoder produces latent representation of the CAD commands, ECAD,: E &d where 1 < t < N and N is the padded sequence length. As our goal is to train generative model using the learned representation atinn, we create a single latent vector, ZCAD E Rd:, for each sequence by average pooling the sequence latent vectors from the CAD encoder output. Constant embeddings are Icarned during training to project this latent vector into again sequence latent vectors before feeding into the decoder network. Next, the latent vector, ZCAD, goes through the transformer decoder and then another embedding layer to project the continuous values into the CAD command parameters. The embedding layer

---

Published in Transactions on Machine Learning Research (04/2025)
is combination of both the CAD command embedding and the positional encoding of the sequence. Here we use the sinusoidal positional encoding from the original transformer implementation (Vaswani et al. 2017
3.3.2 Contrastive CAD-Image Pre-training (CCIP): Joint representation learning of the CAD program and CAD-image
As our goal is to develop an image-conditional CAD generative model, the second step of our framework focuses on learning the joint distribution of the latent space of the CAD images and their corresponding CAD command sequences. For this purpose, we utilize the trained CAD encoder model from Step to create CAD latents To generate image latents, Zin E TRd; we ResNet-18 based architecture the image encoder. We have experimented with larger ResNet architectures and vision transformers (ViT) but found R8 to be sulficient for our datasel (see Appendix Table 3 for ablation). First, we preprocess the input images by resizing them into 256 x 256 pixels, center cropping and then normalizing with (0055, 0.5). Then the images are passed through four layers with increasing dimensionality: 64, 128, 256 and 512. Each of these layers contains two blocks of convolutional layers. We use several dropout layers in each encoder block of the image encoder model. The output of the image encoder is 512 8 8 which projected onto the de de-dimensional space using linear layer. During training, we keep the CAD encoder frozen. Note that the training dataset contains several scaled versions of the same CAD model. This image augmentation helps the model to learn the underlying representation of similar types of CAD geometries.
?
3.3.3 CAD Diffusion Prior (CDP): Conditional generative model for CAD
One of the most critical components of our framework is the diffusion prior model that generates CAD latents conditioned on image latents. In essence, the diffusion prior model is a conditional latent denoising diffusion probabilistic model (Ho et al. 2020). The forward diffusion process standard where the CAD latent, ECAD, destroyed with Gaussian noise sequentially to create the noised CAD latent, z. During the denoising process, we concatenate the noised CAD latent, z with image latent, Zimage and use a projection layer before foeding into the denoising model. For the denoising model, use a ResNet-MLPP architecture similar to Gorishniy et al. (202T) where several blocks of ResNet-MLP blocks are used with a normalization and linear head as shown in Figure 4 For completeness, we also develop deterministic prior in addition to the diffusion prior. The deterministic prior is straightforward and deterministically predicts ZCAD from Zimage. For this study, we use a simple ResNet-MLP architecture as our deterministic prior model. Note that the CDP model is a generative model while the deterministic prior model only maps Zimage to ZCAD deterministically. The complete architecture of the CDP model is shown in Figure
3.3.4 CAD Decoder model: Generating CAD sequences from latent representations
Finally, we need decoder model to convert the generated CAD latents from step 3 into CAD command sequences. To this end, propose to utilize the pre-trained decoder part of the CSR model from Step I instead of training a decoder model from scratch. The use of pre-trained decoder model is particularly useful for scaling our approach to large-scale datasets. During inference time, we would generate ZCAD from the diffusion prior model and feod it to the frozon CAD decoder model to generate the CAD command sequences .N, Note that the <SOL> token prepended to this output to create the complete CAD sequence.
4 EXPERIMENTS
4.1 Training details
Command Sequence Reconstruction (CSR) model: For both the encoder and the decoder models, we use the standard transformer encoder and decoder architecture, respectively. Each transformer model consists of four self-attention layers and eight attention heads. The output of the final encoder layer goes
2CAD latents denate the embedding vectors obtained from encoding the CAD command sequences, whereas image latents represent the image embeddings obtained from encoding the images
8

---

Published in Transactions on Machine Learning Research (04/2025)
through another tonh layer to create the latent vectors, ZCAD- In total, there are 6.72 million trainable parameters in our model. To train the autoencoder model, we use the following loss:
N: C
No
N (p pij),
Σ
(lit +
Σ 1=1 ==
(1)
where No is the number of CAD commands, Np is the number of paramcters in each CAD command, t, is the command type and Pyj is the parameters in each command. Recall that, in this study Ne 6 and N₂ = 16 due to the modeling of the CAD sequence in section 3.1 Here we use cross-entropy loss as ((::) in equation | with a weight parameter 3 that regulates the relative weight of the loss the command type and their parameters, respectively. The loss combines the CAD operation type loss and the corresponding parameter value loss.
Contrastive CAD-Image Pre-training (CCIP) model: For the CCIP model, we follow the standard contrastive learning procedure and use the normalized temperature-scalle
cross entropy loss (Chen et al. 2020). The main idea is to maximize the similarity between a CAD image, I E R 11x448x448 and its cor- responding CAD command sequence {c in a given batch of B training data pairs (I, [c,)). Given a batch of example pairs, we have 2B data points in total for the contrastive prediction. By considering the positive pair as the only positive data sample, we consider the rest of the 2 1) samples as negative data pairs. If the similarity between the data pairs is expressed cosine similarity, sim(u, v) u //||/////]] then the contrastive loss can be defined as the following
Zimage,,)
Lij = - -log 28 k=1 [|*t exp(sim(ZcAD, Zimage,k)/T)
(2)
where [[=11 E 10 is the indicator function and is the temperature parameter. The final contrastive loss is then calculated as -= 2B ΣB [f(2k: - 1, 2k) + E(2k, 2k - 1)].
For our study, we experiment with three types of image encoders: ResNet-18, esNet-35 (e et al., 2016 and ViT (Dosovitskiy et al.] 2020), while keeping the CAD encoder frozen. For the ResNet- 18 based CCIP model, there are 28.22 million trainable paramcters in total.
CAD Diffusion Prior (CDP) model: Traditionally diffusion models are used for image generation tasks and thus require a 2D U-net architecture for the denoising process. In our case, the diffusion model is trained on the vector representation of the latent space. We experiment with two different denoising models to replace the U-net architecture for this purpose: an MLP-based architecture and an MLP with residual connections, Nes We find the ResNet-MLP model be more effective for the denoising model and use the model for all results reported in this paper.
4.2 Training
The CSR model trained using learning rate of 1 10-3 and batch size of 512. We use the Adam optimizer with scheduler to warm up the learning rate from zero to 1 x 10-3 for the first 2000 steps. The CSR model is trained for 1000 epochs. The CCIP is trained using the Adam optimizer with decoupled weight decay [Lohchilov & Hutter (017)) with the RedueeLROnnlatenn schoduler and an initial lcarning rale of x 10-3, The CCIP model is trained for 300 epoch. For the CDP model, both the diffusion and de-non de-noising process use 500 timesteps in our model. The CDP model is trained for million timesteps with a fixed learning rate of 10-5 where the gradient accumulated every steps. A maximum gradient norm of 1.0 also used for training. All models are trained on an 80GB NVIDIA A100 GPU, Additional training details be found in the Appendix.
9

---

Published in Transactions on Machine Learning Research (04/2025)
4.3
Evaluation and baselines
4.3.1
Command Sequence Reconstruction (CSR) model:
As our transformer-based CSR model lakes the sequence of CAD commands as input, we directly compare the CSR. model against the DeepCAD autoencoder (Wu et al. 2021) to show how autoregressive training helps in obtaining more robust latent space. We mainly evaluate the reconstruction performance of cach CAD command sequence in terms of three metrics as proposed by Wu et al. 2021): the mean accuracy of the CAD command generation Memd; the mean accuracy of the CAD parameter value generation Mparam: the mean chamfer distance, JCD, and the ratio of invalid shapes IR for the generated CAD against the ground truth. The mean CAD command generation accuracy and the mcan command paramcter generation accuracy for a set of reconstructed CAD models, 9, are defined as the following,
19
N I[t, = fd).
1
Hemd
9
N
(3)
|9]
N Pij Pa.l <n T[, =.]],
Ppaaam
Dis1 I14
where I[-] is the indicator function, N: is the total number of CAD commands in the sequence, t; and f; are ground truth and predicted CAD commands respectively, Paj and p.j are the ground truth and predicted jth parameter value of the ith CAD command respectively. As mentioned in section 3.3.1 the parameter values are quantized into 8-bit integers.
As we cannol directly caleulate the chamfer distance from the B-rep data, we convert each 3D solid into a point cloud using fixed number of points, 2000 in this case. Once we have the point clouds of both the reconstructed, G and the ground truth S shapes, can calculate the chamner distance, CD. Finally, calculate the mean chamfer distance as the following, μCD T Exeg CD(k, We also evaluate these metrics with respect to the sequence length of the CAD commands. Intuitively, higher-length sequences should be more challenging to reconstruct due to the complexity of the CAD commands.
4.3.2 Contrastive CAD-Image Pre-training (CCIP) model:
Due to the lack of existing literature on using learning-based approach for image conditional CAD program generation, we evaluate the performance of the proposed CCIP model in terms of image-based CAD retrieval lasks. This is an important problem in the engineering design domain due to the wide applications of CAD retrieval from large CAD databases. The retrieval task is formulated as the following: given a CAD- image, I, the model should identify the CAD program, {c;}, that aligns with this image from a set of ny CAD programs. Drawing motivation from image- based retrieval literature (Koh et al. 2024), we sample several batches of CAD examples, п5 = 10, 1 128, 1024, 2048, from the test dataset. Among these ny examples, we randomly select one sample and utilize its CAD-image as the retrieval input to the CCIP model. Next, find the CCIP similarity between the n, n, examples and pick the CAD example corresponding to the highest cosine similarity. As a baseline, we compare our methods against the image-to-image search of CAD models. For this purpose, we utilize a ResNet-18 18 model with pre-trained ImageNet weights.
4.3.3 CAD Diffusion Prior (CDP) model:
To evaluate the image conditional generation, we compare method against unconditional generation. For this purpose, we train an unconditional latent diffusion model from scratch baseline generative model. In addition to the unconditional model, we also compare the CDP model against other four basclines from literature the latent. GAN (I-GAN) model for CAD program generation Wu et. al. (2021). the SkexGen model for B-rep generation (Xu et al., 2022), the BrepGen model from B-rep generation {Xu et al. 2024b). and the ContrastCAD model for CAD program generation (Jung et al. 2024). For direct comparison with the basclines, we report the metrics proposed in AAhhioptas et al.) (2018) for point-cloud generative models. Specifically, create two sets of point clouds: one for ground truth shapes and one for the generated
10

---

Published in Transactions on Machine Learning Research (04/2025)
shapes S. Next, calculate the following: Coverage (COV) which mcasures the diversity of the shapes in G, Minimum matching distance (MMD) which measures the fidelity of generated shapes, and Jensen-Shannon Divergence (JSD) which calculates the statistical distance metric between the two distributions. Details of these metrics can be found in the Appendix.
Finally, we evaluate the image-to-CAn conversion by calculating the FID score of the generated CAD programs. We create CAD programs from image inputs for the entire test data and extract their latent vectors ground truth. Next, create the CAD latents using conditional, unconditional, and deterministic MLP- Prior models. If the ground truth embeddings and the generated embeddings have the normal distributions N(us, Is) and N (ug. Eg) respectively then the FID score can be calculated as, FID = llus og ll + tr (Es + Eg 2(Es ) Inluitively, the FID score calculates how aligned the generated CAD programs are when compared to the test CAD-images.
5 RESULTS
5.1 CAD sequence reconstruction
We directly compare our autoregressive CSR model with the DeepCAD transformer autoencoder model. As shown in Table [ our model outperforms the DeepCAD transformer autoencoder in every metric: mean command accuracy Hend, mean parameter accuracy ppaaam: mean chamfer distance μCD, and invalid ratio IR.
Table 1: Shape encoding performance of the proposed autoregressive encoder-decoder model. Arrows indicate whether higher or lower is better. Bold numbers represent the overall best performance.
Method
Pend (1)
Mparam (1)
(CD(4)
IR(L)
DeepCAD 99.36
97.59
0.783
3.44
GenCAD
99.51
97.78
0.762
3.32
GerCAD
DeeDCAD
GenCAD
DeeCAD
GeaCAD
DepCAD
1.0
1.00
Rend
RCD
0.95
0.00
31
45
39
0.8
3
17
31
45
50
17
91
58
sequetce length
sequeeice length:
sequence length
Figure 5: Performance of GenCAD-antoencode corresponding to the length of CAD command sequences
Most importantly, our model is more accurate for higher lengths of CAD command sequences as shown in We anticipate that the high accuracy for longer sequence lengths is due to the autoregressive encoding of the latent, space. By utilizing autoregressive training, we allow the model to encode more information about the CAD command sequence. Some qualitative examples of our model are shown in Figure Our model can reconstruct the CAD programs more accurately for relatively complex shapes when compared to DeepCAD.
Figure 5
5.2
Image-based CAD program retrieval
While there are methods for image- based CAD-retrievals for specific datasets such as ROCA Gümeli et al. (2022) and DiffCAD (Gao et al.) (202)), there are no existing imago-based CAD program retricval methods. For this reason, we compare our model against an image-to-image similarity metric. Using pre-trained ResNet-18 architecture, extract. the image latents and find the cosine similarity between the ground
11

---

Published in Transactions on Machine Learning Research (04/2025)
Ground truth
DeepCAD
GenCAD
Figure 6: 3D CAD shape encoding performance: We show the qualitative examples against DeepCAD which also uses transformer-based model for CAD command reconstruction but does not use autoregressive training. Our method gives more accurate CAD command prediction for a test dataset that is close to the ground truth.
Input image
143
107
105
102
.097
096
032
092
087
080v
input image
-115
108
.107
000.00
p=.101
095
p-
004
--00
089v
Input image
P..114
p=.108
p=.10
13M
p=.097
p-.055 09.
.292
p
input image
pm .118
p0.108
103
101101
p- 10-
p- 100
p0.095
.293
pm.099
p -90
Figure 7: Image-based CAD retrieval: Qualitative performance of GenCAD-CCIP model is shown in terms of image-based based CAD retrieval task for randomly sampled batches of 10 examples from the test dataset. The input CAD-image or CAD-sketch is shown on the left most column and the retricved CAD programs are shown in the rest of the columns. We also provide the probability assigned by the CCIP model for each retrieval. The multimodal latent space helps the accurate retrieval of CAD programs from input CAD images by assigning higher probabilities similar CAD programs and lower probabilities the rest of the CAD programs.
truth and the randomly sampled CAD images. We also provide random guesses as a simple baseline for comparison. As shown in Table 2 our model outperforms other baselines by large margin. Note that our model is most accurate for small batches of CAD programs, and the accuracy drops when the number of CAD programs increases. Our model is also almost 61% accurate in retrieving CAD programs based provided input images from relatively large collection of CAD programs, i.c., 2048. Overall, the CCIP model is more Ihan 15x accurate in retrieving the correct CAD program when compared to image-to-image search. Some qualitative examples of retrieval results are shown in Figure 7 and Figure g As the CAD programs are randomly sampled within the batch, diverse number of shapes are present which makes the retrieval challenging. is easy see that our model provides higher probability values to the correct CAD program based the image input. Additionally, similar shapes assigned relatively higher probabilities
12

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: image-based and sketch- based CAD retrieval performance of GenCAD. пi = 10, 128, 1024 and 2048 are repeated 1000, 10, and 3 times respectively. Arrows indicate whether higher is better or lower better. Bold numbers represent the overall best performance.
Method
RB=10(1)
RB=12s(1)
RB=1021
RB=2018(1)
Random
10.06 0.19
N/A
N/A
N/A
ResNet-18 (pro-trained)
77.70=
19.26 + 0.00
5.21 + 0.45
3.91 + 0.64
GenCAD-iman
98.49 + 3.93
91.41 + 0.6
70.28 + 0.79
60.77 + 0.99
GenCAD sketch
98.36 = 4.03
87.5 = 3.49
70.67 1.03
60.77 = 0.75
compared to shapes that are different. This is more clearly seen in Figure where retrieve the top 5 CAD programs from collection of 2048 CAD programs based on the provided imput image. Our model extracts shapes that are similar to the input image and also provides diversity within the shape.
image
retrieved CAD programs
sketch
retrieved CAD programs
1=1
k=2
k=3
k=4
k=S
k=2
k=3
k=4
k-5
Figure 8: Image-based and sketch-based CAD retrieval from large CAD database: Qualitative performance of GenCAD-CCIP model shown terms of image-based retrieval of CAD programs from large database. Here, we retrieve top-k, where k CAD programs from the entire test dataset of CAD programs based input images. The first column shows the input image sketch and the rest of the columns show retrieved CAD based the image.
5.3 Unconditional CAD program generation
Figure 9: GenCAD (unconditional) 3D CAD generation.
As mentioned in section 4.3 we compare our unconditional CAD generation model against the DeepCAD latent GAN (I-GAN) model, the SkexGen model, the Brepgen model and the Cast model. Below we provide the performance in terms of coverage (COV), Maximum Mean Diserepancy (MMD), and Jensen Shannon Divergence (JSD). Our unconditional model outperforms both the DecpCAD 1-GAN model and the SkexGen model in terms of both COV and MMD metrics as shown in Table 3 Our model also provides comparative performance to Contrast CAD and Brepgen. few qualitative examples of generated 3D shapes
13

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: 3D CAD (B-rep) shape generation performance. Arrows indicate whether higher is better or lower is better. Bold numbers represent the overall best performance.
Method
type
COV (t)
MMD (4)
JSD (4)
DeepCAD (Wu al.
2021
unconditional
78.13
1.45
3.76
SkexGen [Xu et al.
2022)
unconditional
78.17
1.55
4.89
Brepgen Xu ct al.
2024b
unconditional
73.10
1.05
1.22
maattCCCCA + RRE (Jung et al.
2024
unconditional
78.93
1.44
3.67
GenCAD
unconditional
78.27
1.44
3.94
GonCAD-image
conditional
81.37
1.38
3.49
GenCAD-sketch
conditional
82.59
1.33
3.53
using the latent diffusion model are shown in Figure Note that our model can generate complex CAD programs resulting in realistic 3D shapes.
5.4 Image conditional CAD program generation
sketch
O
CAD
080 D
image
CAD
Figure 10: GenCAD (Conditional) 3D CAD generation: Qualitative examples of sketch-to-CAA and image-to-CAD generation capability of GenCAD using the conditional latent diffusion prior model.
In Table 8 show that our conditional diffusion generative model outperforms DeepCAD I-GAN, SkexGen, Contrast.CAD, and also our unconditional diffusion model in terms of COV, MMD, and JSD metrics. Note that we compare our model against the most capable version of Contrast which uses data augmentation. Additionally, our conditional model also significantly outperforms Brepgen terms of the COV metric. Interestingly, Brepgen performs slightly better the other two metrics. Note that COV represents the diversity of the generated shapes, while MMD represents the fidelity the shapes and JSD represents the statistical distribution of the (est set and the generated set. Thus, Brepgen performs better in terms of MMD and JSD by sacrificing the diversity of the shapes. We argue that GenCAD maintains careful balance between diversity, fidelity, and statistical distance.
Finally, to quantify the alignment of the generated samples compared to the input images we compare the FID score between the distribution of CAD latents between a generated dataset and the test data set. As DeepCAD the only available model that generates CAD program, we dircctly compare our results against DeepCAD model. As an additional baseline, we also compare our results against the deterministic-prin model from 4.3.3 Our results indicate that the image-conditional diffusion prior model provides the lowest
14

---

Published in Transactions on Machine Learning Research (04/2025)
image
generated CADS
image
generated CADS
Figure 11: Diversity of generated CAD: Several samples, conditioned on image or sketch, drawn from the latent conditional model of GenCAD.
GenCAD
0,99
(deterministic-prio)
DeepCAD
0.87
(unconditional)
Ce
0.32
(sketch conditional)
GenCAD
0.22
(unconditional)
CenCAD (image conditional)
1.0
FID score
Figure 12: FID score of the generated datasct and the corresponding test dataset (lower is better).
FID score when compared to the sketch-conditional diffusion prior model, I-GAN, deterministic-prior, and unconditional diffusion model as shown in Figure Note that the deterministic prior has the highest FID score and the unconditional diffusson model outperforms the I-GAN model. This result shows that the diffusion prior model can generate CAD programs that close the ground truth distribution. While the I-GAN model can generate unconditional CAD programs, these CAD models are not well aligned with the test dataset and the latent diffusion model is in-between the prior model and the J-GAN model. Finally, we anticipate that the FID score for the CAD-sketch model is higher due to its capability in generating more diverse shapes than the image-conditional model which is also supported by the COV metric in table 3 We also provide qualitative examples of our model in Figure 10
Finally, show the diversity of our diffusion prior model in Figure 11 where sample multiple CAD programs for the same image input. Our model can generate similar CAD programs while providing useful variation in the parameter space.
6 LIMITATIONS AND FUTURE WORK
Although GenCAD shows impressive image-to-CAD performance, there are some limitations to the current version of this model. The CAD programs used in this study are comparatively simpler than industrial design tasks. The limited CAD vocabulary used in this study needs to be extended by including more sophisticated CAD tokens such as revolve operation, edge operation (e.g., fillets/che theets/chaffrs) and other sketch operations. Additionally, GenCAD cannot guarantee the generation of valid CAD similar to most of the generative CAD models reported in other studies. Drawing molivation from recent. NLP lilerature, a step forward can be the addition feedback from a CAD verification tool such as a geometry kernel. Building such a framework is non-trivial and will have the potential to create user-specified CAD programs. Finally, the images used in this study are mostly isometric and noise-free CAD-images. Future studies should focus on generating CAD programs from real-world object images with noisy backgrounds and non-isometric views.
15

---

Published in Transactions on Machine Learning Research (04/2025)
CAD Program
Onshape
Onshape (After edit)
Figure 13: GenCAD generates the entire CAD program sequentially, which allows user-editability of the 3D model in commercial CAD software. Here we show visualization of the original generated CAD program, the editable CAD in a commercial CAD software, e.g. Onshape, and the result of editing a feature.
Finally, to show the real-world applicability of GenCAD, we show its integration with a commercial CAD software ((Onshape) 2007)) Figure 13 which enables user editability the generated shape.
7 CONCLUSION
CAD program generation is challenging task for AI models that have the potential to automate industrial design pipelines. While unconditional generation of CAD programs has been studied, a more important problem to generate grounded CAD programs basod on user intention. We provided GenCAD, the first generative model that can generate entire CAD program for complex 3D shape conditioned on image sketch inputs. We argue that the image-aligned CAD program generation has the potential to be directly utilized for design task acceleration. We also show the application of GenCAD in image-based CAD program retrieval tasks. In summary, GenCAD solves two long-standing challenges in the 3D CAD community; it can retrieve a CAD program from large databases using an image and also generate the CAD program based on the image. The current limitation of GenCAD is its dataset which includes relatively simpler CAD commands. In future studies, we aim create a more sophisticated and real-world CAD datasel that can enhance the capabilities of GenCAD.
References
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40 49. PMLR, 2018.
Md Ferdous Alam, Yi Wang, Chin-Y Yi Cheng, and Jieliang Luo. Representation learning for sequential volumetric design tasks. Journal of Mechanical Design 147(5):051704, 2025.
Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, and Matthias Nießner. Polydiff: Generating 3d polyg- onal meshes with diffusion models. arXiv preprint arXiw:2312.11411, 2023.
Holly K Ault. Using geometric constraints to capture design intent. Journal for Geometry and Graphics, 3 (1):39-45, 1999.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1990 2020.
Weijuan Cao, Trevor Robinson, Yang Hua, Flavien Boussuge, Andrew Colligan, and Wanbin Pan. Graph representation of 3d cad models for machining feature recognition with deep learning. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 84003, pp. V11ATI A003. American Society of Mechanical Engineers, 2020.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMT.R, 2020.
Andrew R Colligan, Trevor T Robinson, Declan C Nolan, Yang Hua, and Weijuan Cao. Hicrarchical cadnet:
Learning from b-reps for machining feature recognition. Compuler-Aided Design, 147:103226, 2022.
Dov Dori and Karl Tombre. From enginecring drawings to 3d cad models: are we ready now? Computer- Aided Design, 27 (7(4))243 254, 1995.
Dov Dori and Liu Wenyin. Automated cad conversion with the machine drawing understanding system: concepts, algorithms, and performance. JEFE Transactions Systems, Man, and Cyhernelics-part A: systems and humans, 29(4):411 416, 1999.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXie:2010. 11929, 2020.
Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8279 8286, 2019.
Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer-aided design language. Advances in Neural Information Processing Systems, 34:5885 5897, 2021.
Daoyi Gao, Dávid Rozenberszki, Stefan Leutenegger, and Angela Dai. Diffcad: Weakly-supervised proba- bilistic cad model retrieval and alignment from an rgb image. ACM Transactions on Graphics (TOG), 43 (4):1 15, 2024.
Christophe Geuzaine and Jean-François Remacle. Gmsh: A 3-d finite element mesh generator with built- in pre-and post-processing facilities. International journal for numerical melhods in engineering, 79(11): 2009.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932 18943, 2021.
Can Gümeli, Angela Dai, and Matthias Nießner. Roca: Robust cad model retrieval and alignment from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4022-4031, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-777 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Pradeop Kumar Jayaraman, Aditya Sanghi, Joseph G Lambourne, Karl DD Willis, Thomas Davies, Hooman Shayani, and Nigel Morris. Uv-net: Learning from boundary representations. In Proceedings of the CCFF/CVF conference on compuler vision and pallern recognition, pp. 11703- 11712, 2021.
Pradeep Kumar Jayaraman, Joseph G Lambourne, Nishkrit Desai, Karl DD Willis, Aditya Sanghi, and Nigel JW Morris. Solidgen: An autoregressive model for direct b-rep synthesis. arXiv preprint arXiv:2203.138 13944, 2022.
arXiv preprint Minscop Jung, Minscong Kim, and Jibum Kim. Contrastcad: Contrastive learning- based representation
Heewoo Jun and Alex Nichol. Shap-c: Generating conditional 3d implicit functions. arXiv:2305.02//0 2023.
learning for computer-idded design models. arXiv preprint N4o:2444.016455 2024.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVI conference on computer vision and pollern recognilion, pp- 4401 4410, 2019.
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Danicle Panozzo. Abc: A big cad model datasct for geometric deep learning, In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 9601 9611, 2019.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Newral Information Processing Systems, 36, 2024.
Joseph G Lambourne, Karl DD Willis, Pradeep Kumar Jayaraman, Aditya Sanghi, Peter Meltzer, and Hooman Shayani. Brepnet: A topological message passing system for solid models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12773-12782, 2021.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXie preprint 1rXi::1711.01001 2017.
Tharna Majumdar and AG Seethalakshny. A cad model based system for object recognition. Journal of Intelligent and Robotic Systems, 18:351 365, 1997.
Vijay Nagasamy and Noshir A Tangrana. Engineering drawing processing and vectorization system. Com- puter Vision, Graphics, and Image Processing, 49(3):379-397, 1990.
Setsuo Ohsuga. Toward intelligent cad systems. Computer-aided design, 21(5):315 307,3377 1989.
Onshape. NVIDIA CUDA Compute Undfied Device Architecture Programming Guide. NVIDIA Corporation, 2007.
Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas Guibas, and Peter Wonka, Sketchgen: Generating constrained cad sketches. Advances in Newral Information Processing Systems, 34: 5077-5088, 2021.
Les A Piegl. Ten challenges in computer-aided design. Computer-aided design, 37(4):461 470, 2005.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652 660, 2017.
Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with va-vae-2. Advances in neurol information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition,, pp. 10684 10695, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trensactions on Pattern Analysis and Machine Intelligence, 45(4):4713-472) 2022.
18

---

Published in Transactions on Machine Learning Research (04/2025)
William B Thompson, Jonathan C Owen, HJ de St Germain, Stevan R. Stark, and Thomas C Henderson. Feature-based reverse engineering of mechanical parts. IEEE Transactions on robolics and oulomalion, 15(1):57-66 66, 1999.
Michal J Tyszkiewiez, Pascal Fua, and Eduard Trulls. Geceo: Goometrcally-conddiione point diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2128-2138. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Peng-Shun Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun- Yu Sun, and Xin Tong. O-enn: Octree based convolutional neural networks for 3d shape analysis. ACM Transactions On Graphies (TOG), 36(4):1-11, 2017.
Kevin J Weiler. Topological structures for geometric modeling (Boundary representation, manifold, radial edge structure). Rensselaer Polytechnic Institute, 1986.
Karl DD Willis, Yewen Pu, Jicliang Luo, Hang Chu, Tao Du, Joseph G Lambourne, Armando Solar-Lezama, and Wojciech Matusik. Fusion 360 gallery: A dataset and enviromment for programmalic cad construction from human design sequences, ACM Transactions on Graphics (TOG), 40(4):1-24 2021.
Karl DD Willis, Pradeep Kumar Jayaraman, Hang Chu, Yunsheng Tian, Yifei Li, Daniele Grandi, Aditya Sanghi, Linh Tran, Joseph G Lambourne, Armando Solar-Lezama, et al. Joinable: Learning bottom-up assembly of parametric cad joints. In Proceedingg of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15849-15860 2022.
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepead: deep generative network for computer-aided design models. In Proceedings of the IFFE/CVE International Conference Computer Vision, pp. 2021.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh gencration from single image with sparso-view large reconstruction models. arXiv preprint arXiv:2404.01199 2024a.
Xiang Xu, Karl DD Willis, Joseph G Lambourne, Chin-Yi Yi Cheng, Pradeop Kumar Jayaraman, and Yasutaka Furukawa. Skexgen: Autoregressive generation of cad construction sequences with disentangled codebooks. arXiv preprint ar 2ix: 04632, 2022.
Xiang Xu, Joseph G Lambourne, Pradeep Kumar Jayaraman, Zhengqing Wang, Karl DD Willis, and Ya- sutaka Furukawa. Brepgen: b-rep generative diffusion model with structured latent geometry. arXiv preprint ar Xiv: 15563, 2024b.
19


Reviews:
[{"review_id": "0hs3vLP6CM", "reviewer": "Reviewer_YRo2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. GenCAD propose the first image-conditioned CAD generation model. Compared to its previous unconditional version, DeepCAD, it enables more flexible CAD generation that can be controled by users\n2. GenCAD use contrastive leaning between images and CAD models, which facilitates CAD model retrieval\n\nWeakness:\n1. compared to current image-to-3d model, which use NeRF, gaussian splatting or mesh to represent the shape, CAD lacks capability to represent complex objects that have complicate topology and textures. The results shown in the paper only contains simple workpieces, it would be better to show its effectiveness on more complex furniture datasets, like [1].\n2. The architecture of CSR model is confusing. CSR model is an auto-encoder model to encode and reconstruct the input CAD model, while autoregressive model is a generation model for generating the next token given previous ones. The standard operation of CSR model is to encode CAD sequence from c_1 to c_N and then decode it from c_1 to c_N, while the current CSR is to decode it from c_2 to c_N+1. Furthremore, because transformer is an N-in-N-out architecture, it's also confusing of using trapezoid rather than rectangular shape to represent encoder and decoder in Figure 2. The author should illustrate the motivation and advantage of such design in CSR model and provide the comparison between standard (variational) auto-encoder model with current model.\n3. The architecture design for CDP is also wired. Compared to current design, it would be better to use DiT architecture, which use transformer architecture and regard condition as a injection latent injected in the model via cross attention. It would be better to provide comparison experiment between current design and DiT.\n\n\n[1] BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry\n[2] Scalable Diffusion Models with Transformers", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose a framework, dubbed GenCAD, for generating CAD models given a single image, which utlize contrastive representation and diffusion model, therefore enabling single-image-to-CAD and CAD model retrieval. GenCAD contains the following contributions:\n1. a transformer-based model for CAD sequences encoding and reconstruction\n2. an image-conditioned diffusion model for CAD generation\n3. contranstive learning between image and CAD, which facilitate fast and accurate CAD retrieval", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. As illustrated in weakness 1, provide more complex object generation results.\n2. As illustrated in weakness 2, add the comparison experiment with  (variational) auto-encoder and correct the figure 2.\n3. As illustrated in weakness 3, add comparison experiment with DiT.", "broader_impact_concerns": "No ethical concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3hJhtby0uG", "reviewer": "Reviewer_EWNT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- Aligning the latent space of image and CAD model enable accurate retrieval compared with image-to-image search. \n- It appears to work better for obvious reasons. (Obvious reasons are fine; it makes sense.)\n##  Weaknesses:\n- The use of latent representations for CAD programs may introduces a significant limitation. Specifically, I think this approach may not scale effectively to more complex CAD models or larger dataset(compared with Autoregresive models.). It would be beneficial for the authors to discuss this limitation and its potential impact on the applicability of the method in the paper.\n- lacking necessary Ablation Studies to prove the design of model(Crucial).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary \nThis paper introduces the GenCAD generative model, which utilizes an autoregressive transformer with a contrastive learning framework and a latent diffusion model to convert image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. Results show that GenCAD outperforms existing state-of-the-art methods in generation of CAD models. Additionally, GenCAD's contrastive learning framework notably enhances the retrieval of CAD models through image queries.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see weaknesses parts.", "broader_impact_concerns": "None,", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fcWJD2ei3I", "reviewer": "Reviewer_5nfB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths they claimed:\n- They enable both conditional and unconditional generation of CAD models.\n- The performance is clearly superior to that of DeepCAD and SkexGen.\n\n\nWeaknesses:\n1. Incremental contributions; novelty is not enough; the experiments are not yet convincing.\n2. Retrieval experiments are incomplete. \n   - The authors stated, \"as there are no existing image-based CAD retrieval methods, we compare our model against an image-to-image similarity metric.\" This assertion is incorrect. Numerous image-based CAD retrieval methods exist. For instance, DiffCAD and ROCA can even handle CAD retrieval using real image inputs. It would be more appropriate to compare and conduct experiments on both synthetic and real image inputs.\n   - References:\n     - DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image\n     - ROCA: Robust CAD Model Retrieval and Alignment from a Single Image\n\n3. SolidGen performed similar tasks; however, it does not generate CAD programs. It would be valuable to see whether producing CAD programs is clearly more effective and better than direct B-rep synthesis. Your paper did not present any evidence.\n\n4. Some claims are perhaps overstated. I would recommend that the authors carefully revise their wording and include more comparisons and discussions in the related work section. For example:\n   - In Section 7, the authors stated, \"We provided a first-of-its-kind AI model, GenCAD, that can generate CAD programs for complex 3D shapes both unconditionally and also based on image inputs.\" However, the term \"first-of-its-kind\" is likely overstated, as another work called \"SolidGen\" also supports both unconditioned generation and conditioned generation from image inputs. This work was published in TMLR-2023. \n\n5. Some experimental setups are not very practical. In Sections 5.2 and 5.4, how can we acquire such an input image in a practical scenario? I believe it would be challenging to use this in a real-world situation, as images captured by a camera do not resemble those shown.\n\n6. Please consider comparing the latest methods on CAD generation in the appropriate experimental sections, including but not limited to:\n   - \"SolidGen: An Autoregressive Model for Direct B-rep Synthesis\": This paper proposes using an auto-regressive transformer to generate CADs. It also supports unconditioned generation and image-conditioned generation.\n   - \"ContrastCAD: Contrastive Learning-based Representation Learning for Computer-Aided Design Models\": It also utilizes contrastive learning techniques to effectively capture information and can handle long construction sequences as well.\n \n7. They mentioned in Section 2.1 that DeepCAD has a limited number of sketches and 3D operations; however, GenCAD does not address this point, and I am unsure why they mention this limitation.\n\n8. Typos (please revise your paper carefully again):\n   - The second paragraph of Section 3.1:\n     - The notation $\\mathbf{R}^{17}$ should be changed to $\\mathbb{R}^{17}$\n     - The notation $\\mathbf{t}_i \\in \\mathbb{R}$ should be changed to $t_i \\in \\mathbb{R}$\n   - Section 5.2:\n     - You typed \"image-tp-image\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "They propose a solution called GenCAD to generate/retrieve CAD programs/models from image inputs. \n\nThey first use a transformer-based encoder-decoder architecture to auto-regressively learn the latent features for generating CAD programs that represent the CAD model in B-rep. They then utilize contrastive learning to learn a joint latent space between CADs and images. Additionally, they train a diffusion model to generate CAD latents from image inputs. Finally, CAD programs can be decoded using the transformer's decoder.\n\nI believe their main focus is on enhancing a previous work, DeepCAD, which converts CAD into a language model. The authors' enhancements include:\n1. Conditional generation from images.\n2. Improvements in diversity, fidelity, and accuracy.\n\nAs for the technical contributions they claimed in the paper (see the end of Section 1):\n1. A transformer-based autoregressive model: SolidGen also proposed using an autoregressive transformer for conditioned/unconditioned generation. Therefore, I find this point less convincing.\n2. A conditional CAD generative model from image inputs: This contribution is not particularly novel as SolidGen has already implemented it.\n3. Contrastive learning: This contribution appears trivial, as ContrastCAD has undertaken a similar approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weak points. Requested changes:\n1. More comparisons are needed in retrieval experiments, *e.g.*, DiffCAD and ROCA.\n2. Compare it against SolidGen and ContrastCAD.\n3. Revise any typos and inappropriate wordings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper works on learning to generate CADs, including conditioning on sketch or image inputs. Technically, it proposes GenCAD that is based on autoregressive transformers with contrastive learning, and the parametric CAD command sequences are generated by using latent diffusion. Contrastive learning also enables the retrieval of CAD models using image queries from large CAD databases.

Reviewer 5nfB suggests to improve the completeness of retrieval experiments, present more fair comparisons with SolidGen and ContrastCAD, and make the experiment settings more practical. Reviewer EWNT pointed out that the paper is short of ablation studies and the proposed method may not deal with complex CADs. The authors have carefully responded to these comments, with necessary revisions that improve the paper quality greatly. Reviewer YRo2 initially raised concerns on architecture of CSR and how the proposed architecture design for CDP compares against DiT based one; for the former one, the authors have taken the suggestion and improved Fig. 2, and for the latter one, the authors explained why DiT cannot be directly applied to this context. AE has read the paper and partially agrees with the authors.

However, AE still suggests that the authors follow the reviewer's suggestion to compare with DiT, e.g., by NOT average-pooling the sequence of latent vectors generated by CSR encoder.
Decision: Accept (poster)