cadrille: Multi-modal CAD Reconstruction with Online Reinforcement
  Learning
Authors: name: Maksim Kolodiazhnyi, name: Denis Tarasov, name: Dmitrii Zhemchuzhnikov, name: Alexander Nikulin, name: Ilya Zisman, name: Anna Vorontsova, name: Anton Konushin, name: Vladislav Kurenkov, name: Danila Rukhovich

Computer-Aided Design (CAD) plays a central role in engineering and
manufacturing, making it possible to create precise and editable 3D models.
Using a variety of sensor or user-provided data as inputs for CAD
reconstruction can democratize access to design applications. However, existing
methods typically focus on a single input modality, such as point clouds,
images, or text, which limits their generalizability and robustness. Leveraging
recent advances in vision-language models (VLM), we propose a multi-modal CAD
reconstruction model that simultaneously processes all three input modalities.
Inspired by large language model (LLM) training paradigms, we adopt a two-stage
pipeline: supervised fine-tuning (SFT) on large-scale procedurally generated
data, followed by reinforcement learning (RL) fine-tuning using online
feedback, obtained programatically. Furthermore, we are the first to explore RL
fine-tuning of LLMs for CAD tasks demonstrating that online RL algorithms such
as Group Relative Preference Optimization (GRPO) outperform offline
alternatives. In the DeepCAD benchmark, our SFT model outperforms existing
single-modal approaches in all three input modalities simultaneously. More
importantly, after RL fine-tuning, cadrille sets new state-of-the-art on three
challenging datasets, including a real-world one.