CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
Authors: name: Jingwei Xu, name: Chenyu Wang, name: Zibo Zhao, name: Wen Liu, name: Yi Ma, name: Shenghua Gao

This paper aims to design a unified Computer-Aided Design (CAD) generation
system that can easily generate CAD models based on the user's inputs in the
form of textual description, images, point clouds, or even a combination of
them. Towards this goal, we introduce the CAD-MLLM, the first system capable of
generating parametric CAD models conditioned on the multimodal input.
Specifically, within the CAD-MLLM framework, we leverage the command sequences
of CAD models and then employ advanced large language models (LLMs) to align
the feature space across these diverse multi-modalities data and CAD models'
vectorized representations. To facilitate the model training, we design a
comprehensive data construction and annotation pipeline that equips each CAD
model with corresponding multimodal data. Our resulting dataset, named
Omni-CAD, is the first multimodal CAD dataset that contains textual
description, multi-view images, points, and command sequence for each CAD
model. It contains approximately 450K instances and their CAD construction
sequences. To thoroughly evaluate the quality of our generated CAD models, we
go beyond current evaluation metrics that focus on reconstruction quality by
introducing additional metrics that assess topology quality and surface
enclosure extent. Extensive experimental results demonstrate that CAD-MLLM
significantly outperforms existing conditional generative methods and remains
highly robust to noises and missing points. The project page and more
visualizations can be found at: https://cad-mllm.github.io/