ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
Authors: Junliang Ye
Zhengyi Wang
Ruowen Zhao
Shenghao Xie
Jun Zhu
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM
MLLM
3D generation
gpt-4o
Editing
Primary area: applications

Abstract: Recently, the powerful text-to-image capabilities of GPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni—a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, we perform instruction-based fine-tuning of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset, equipping it with native 3D understanding and generation capabilities. Our work represents an effective step toward extending multimodal large language models with fundamental 3D intelligence, paving the way for future advances in 3D-native AI.

ShapeLLM-Omnn A Native Multimodal LLM for 3D Generation and Understanding
Junliang 3. Zhengyi Wang 1.3+ Ruowen Zhao'
Shenghao Xie
Jun Zhu 1,31
Tsinghua University?
Peking University² ShengShu
setts://githhn cow Com//JAMESYLLLSSaaLLLM 0mni/
Abstract
Recently, the powerful text-to-image capabilities of GPT-4o have led to growing appreciation for native multimodal large language models. However, its multi- modal capabilities remain confined images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni-44 native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D qucttr-qanttied variational autoencoder (VQVAE), which maps 3D objects into discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct large-scale continuous training dataset named 3D) 31D-Alpaca, encompass- ing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, we perform instruction bbasd fine-tuning of the 2.wen-2.5vvl77--innte
model on the 3D-Alpaca dataset, equipping it with native 3D understanding and generation capabilities. Our work represents an effective step toward extending multimodal large language models with fundamental 3D intelligence, paving the way for future advances in 3D-native Al.
Introduction
Large language models have made significant achievements, including text-only language models
(LLMs) Achiam al. [2023]. Liu al. [2024a] Bai et al.
[2033, Touvron al. [2033]
Multimodal
Large Language Models (MLLMs) that can understand images Hurst et al. [024a]]
GLM et al.
(2024) Team [002), video Guo al. [2025] Cheng al.
[2024]. Maaz et al.
2023
al.
[202bb and 3D Wang et al. [2024b], Siddiqui et al. [2024a] Chen et al. [2023a 2025b content.
These models employ similar transformer architectures, using dedicated encoders to model each modality independently, thereby integrating images, video, and 3D modalities into existing L.Mss.
Recently, ChatGPTT-4o/ Hurstt al. 2024a has demonstrated remarkable performance. By natively incorporating image generation and understanding into the large language model (LLM) architecture, it enables more fine-grained and precise control through human instructions. However, its multimodal capabilities remain confined to images and text, limiting its potential in more complex spatial domains.
In this work, propose unified approach to integrate 3D generation and understanding into pre-trained multimodal large language model (MLLM). Enhancing LLMs with native 3D capabilities erucial for downstream applications such 3D content creation, robotics, digital twins, and immersive virtual environments.
Our method adopts fully next-token prediction paradigm, which ensures natural compatibility with joint training and large-scale scalability. We leverage a VQVAE to encode 3D meshes into compact
Equal contribution
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

3D mesh
Continusus Image feature
This 3D mesh "Beige Teddy Bear
Discrete Voxel Tokens
Discrete Text Tokens
3D VQVAE Decoder
Text Detokenizer
ShapeLLM-Omni
Vision Encoder
Text Tokenizer
Text Tokenizer
3D VQVAE Encoder
Generate 3D mesh using the image gave you.
Provide a brief analysis of the 3D asset shown
3D Understanding Figure 1: ShapeLLM-Omin inherits Qwen2.5-vI's strong multimodal capabilities and additionally supports text-to-3D, image-to-3D, 3D captioning, and 3D editing using text instruction.
Image
3D Generation
here:
3D mesh
discrete tokens, enabling a unified representation. These tokens are utilized for both understanding and generating 3D meshes, following a format analogous to language modeling.
To enable LLMs with 3D ability, we construct a comprehensive training dataset using 3D shapes from mixture of 3D datasets Deitke al. (203a b). Collins al. [2022] Chang et al.) (2015) We
construct interleaved 710k text/image-3D pairs to enable the model for basic 3D understanding ability and text/image 3D generation ability.
Furthermore, to enable interactive 3D mesh editing, we introduce a novel dataset of 62k paired 3D meshes and corresponding text-based editing instructions. This facilitates fine-grained manipulation of 3D assets through natural language, making real-time editing more intuitive and controllable.
After that, we train an LLM on the corpus. We resume from Qwen-2.5 -25en-2.5-1 2.--VLL-nnstrutt-- Bai et al. [2025] utilize the effective its large-scale pre-training on text and images. Our model demon- strates a wide range of capabilities, including: (1) generating 3D content from language instructions; (2) generating objects from image inputs; (3) interactively editing 3D assets using natural language;
(4) understanding and interpreting 3D meshes for semantic and geometric reasoning.
In all, our contributions are:
We propose novel framework for unified 3D object generation and understanding based on fully autoregressive next-token prediction paradigm.
We present the 3D-Alpaca dataset for training large language models (LLMs) with 3D capabilities. Comprising 3.46 billion tokens, covers three core tasks: 3D generation, 3D understanding, and 3D editing.
Our experimental results provide strong empirical evidence supporting the effectiveness of the proposed method.
Related Work
2.1
3D Mesh Generation
The remarkable achievement of 2D diffusion models Ho et al. [2020), Rombach et al. [2022] has facilitated the exploration of 3D generative models. Early 3D generation methods Poole et al. 2022

---

Wang et al.
2023e 3 Chen et al.
2023b] Lin et al.
2023 Raj et al. 2023 Li et al
[202bb] Sun
al. 2023], Chen et al. 2024h] Wang et al. [022]] Tang al. [2023] Yi et a.. [024 often rely on
SDS base- optimization to distill 3D content due to the limited 3DD data, but encounter challenges
such as long optimization time and Janus problem. Subsequent works such as Wang and Shi [2023)
Shhi et al. [2023b) . Wang et al. [203cc] [Liu et al. [2025a]. Ye et al. [2024b), Qiu et al. [2024]. Chen
cct al. [2024a] enhance semantic consistency across different views during multi-view image synthesis.
To minimize generation time, more recent approaches Long et al. [2024]. Zhao et al. (2024), Liu
ct al.
2023d cl Shi et al. [2023a] , Weng et al. [2023]. Liu et al. [2023b], Wu et al. 2024a| Chen
al.
2024:]. Voleti et al. 2024 Ye et al. 2024a] s Liu et al. [2024b) adopt a two-stage pipeline that
:
integrates multi-view image prediction with 3D reconstruction to produce 3D models. LRM Hong
ct al. [2023a] and other works Tang et al. (2024a]] Wei et al. (2024). Ziwen et al. [2024], ; Li et al.
[2023a] Xu et al. [2023]. J. Wang et al. [023a]] Siddiquu et al. [024bb] /Zangg et al. 2024a bj. iZo
et al. [2024] Xu et al. [2024a] Nawrot et al. (202]) J. Wang et al. [2024c] build
feed-forward
reconstruction model and predict 3D structures within seconds. Additionally, native 3D diffusion
models Zhao et al. [003]] Wang et al. [023b]] Wu et (2024b), Yang et al. [024]] Huang et al.
2022bb Yang et al. [2024). Zhang et al. [2024c] Xiang et al. [2024). Chen et al. [2024f]. Li et al.
2024a Wu et al. 2024b), , Ye et al. [2025] encode 3D objects into a VAE latent and adapt latent diffusion model on the resulting representations for comprehensive 3D understanding. Nevertheless,
the above methods treat 3D objects as numerical fields Mildenhal et al. (202T), , Kerbl et [023] and
extract meshes using Marching Cubes Lorensen and Cline 1998). which are not easily represented
as discrete tokens.
2.2 Autoregressive 3D Generation
Inspired by the success of auto- rego-segressive models in language and image synthesis, some pioncering works (iikkquu et al. [2024a]. Chen et al. [2024d]. Weng et al. [2024a] have explored their use in 3D shape generation. They adopt VQVAE Van Den Oord al. 2017] to compress 3D shapes into latent spaces, which are subsequently quantized into discrete tokens for learning via an auto-regressive
transformer. Instead of employing VQVAE, other studies Chen al. [222e]] 2025a Liu al.
blc), Yang et al.] (2025), Weng et al.] (2024b). Tang et al. [0024b] Hao et al. [2024). Zhao al. 2025 have proposed specialized mesh tokenization techniques that transform mesh vertices and faces into compact discrete token sequences, while preserving the original complex geometric details. These approaches enable the auto-regressive model to effectively generate meshes in face-by-face manner. Building on 3D auto-regressive models, LLaMA- Wang et al. [2024b] explores the integration of natural language instructions with mesh generation and understanding, enabling interactive 3D content creation through unified framework. However, it treats the 3D OBJ mesh file as text for language model to process, which overlooks the inherent topological structures of 3D data.
2.3 Unified Models for Multimodal Understanding and Generation
Extending large language models (LLMs) to process, generate, and comprehend multiple modali- ties-such as vision and language- wwtthi a unified framework has become a major research frontier.
Previous studies Bai et al. [2023]. Chen et al. [2024g]. |laayrac et al.] [002) have advanced this direction by equipping LLMs with visual understanding capabilities for multimodal tasks. Concur-
rently, other works Team (2024). Liu et al. [2024c), Wang et al. [2024a] Xie et al. [2024), Zhou
al. 2024 have proposed the integration of image and text generation through specialized visual tokenizers. More recently, CharGPT-40 has further propelled this progress, achieving state-of-the-art
performance in both visual comprehension and image synthesis. Beyond 2D modalities. a growing body of research Hong et al. [2023b). Xu et al. (2024b). Qi et al. [2024a] Xue et al. [2023). Huang al. [2025a] Chen et al. [024b). Huang et al. 2024)). Kang et al. 2025]. Chen et al. [202cc]. Wang et al. 2023d has extended LLMS to 3D content understanding, primarily through point cloud representations. However, point clouds often lack fine- -raiindiind geometric detail and are challenging to acquire in real-world settings, limiting their applicability for interactive generation. Despite these advancements, there remains a notable gap: very few models are capable of jointly processing and generating text, images, and 3D data in an integrated manner. To bridge this gap, we introduce a 3D VQVAF module that encodes 3D shapes into discrete representations, enabling autoregressive models to perform unified multimodal understanding and generation across text, images, and 3D content.
3

---

Voxel- Voxel-Encode Encoder 3D-U
Voxel -ecodder 3D- 3D-Unet
codebook
quantization
Zq
Figure 2: The pipeline of VQVAF, which can compress voxels into discrete tokens.
3 Method
Table I: Modality comparison. In contrast to the task-specific model architectures of SAR3D
and Trellis, ShapeLLM--mi achieves cross-modal alignment by jointly modeling text and 3D
representations in a shared latent space, enabling unified understanding and generation capabilities.
Input Modality
Output Modality
Text
Image
3D
Unified model
Text
Image
3D
SAR3D Chen et al.
2025b
Trellis Xiang
2024
PointLLMXX Xu et al.
2024b
4
LLaMA-Mesh Wang et al.
2024b
7
ChatGPTT44f Hurst al. 2024b
/
Qwen-2.5vl Bai et al. 2025
4
/
ShapeL.LM--mn (ours)
4
3.1 Overview
Figure I provides overview of our native Multimodal LLM framework, which can handle mixed sequences of text, images, and 3D data and produce corresponding text or 3D outputs. We begin by converting 3D assets into discrete tokens using 3D VQVAE (See. (.3), which allows to leverage the same transformer architecture for both 3D and text token sequences. Subsequently, assemble comprehensive 3D supervised fine-tuning dataset, 3D-Alpaca (Sec. 3.4). covering text-to-3D generation, image-to-3D generation, 3D captioning, and 3D editing.
3.2 Architecture
As shown in Figure [ we represent both text and 3D data as sequences of discrete tokens, enabling fully autoregressive multimodal generation. This design allows for flexible input and output across modalities in any order. While adopt token-based representations for both text and 3D modalities, we use continuous features for images. This is because images are only involved in understanding tasks, whereas 3D data supports both understanding and generation. Such unified modeling approach- based on early Compared prior work in the 3D domain Table I our model is the first unified auto-regressive framework that supports text-to-3D, image-to-3D. 3D understanding, and 3D editing a single system. It also marks the first attempt at ChatGPT-44-stylle model tailored for 3D tasks.
better modality integration within the language model.
3.3
3D VQVAE
In this section, we introduce our 3D representationn-vxeee explain why we chose voxels, and how we compress voxels into discrete tokens using 3D VQVAE. Finally, we describe how to reconstruct high-quality 3D meshes from voxels.
Toxel-Based Representation 3D assets can be represented in various ways-such as voxels, vecset Zhang et al. [2023), Face- Vertex representation Wang et al. [2024b), Point Clouds Xu et al. 2024b), or Gaussian splats Kerbl al. 20033 In this work, we adopt low-resolution voxels as

---

Prompt
Ours
Trellis
S
3Dlopia-XL
storyed building
A iguania with shirt yellow
A cartoon robolic while loy
Green coloured toy van with rectangular curved surface yellow front lights and seats with grey coloured whests
Figure 3: Our 3D-Alpaca dataset comprises 3D generation, understanding, and editing components, providing a comprehensive foundation for training and evaluating 3D large language models.
our 3D representation. We now explain the rationale behind this choice. First, we do not adopt the Face-Vertex representation because it quantizes mesh geometry into discrete spatial tokens, resulting in excessively long token sequences that hinder the training efficiency of unified models. Second, we do not use CosSets-based representations. On one hand, VecSets encode highly informative and continuous geometric features, making it challenging to train a complete 3D VQ- VQ-VAE for their encoding. On the other hand, VecSets are inherently implicit, whereas voxels provide explicit and structured spatial representations that are more suitable for 3D editing tasks requiring direct geometric manipulation. In contrast, voxels strike favorable balance between compactness and expressiveness: they compress complex 3D information into a much smaller latent space, facilitating efficient training, asset's essential shape and skeletal structure, thereby providing
while effectively preserving sufficient geometric cues for language models. Moreover, open-source reconstruction models can be readily leveraged to convert coarse-resolution voxels into high-quality, detail-rich meshes.
Model Architecture We adopt a 64 voxel grid resolution, as voxels at this resolution strike the optimal balance for modeling 3D skeletons, preserving essential structural details while avoiding excessive redundancy Xiang et al. [2024). Although voxel representations are compact, even modeling single 3D object with 64 voxel grid still requires 64 tokens far beyond what a large language model can handle. Therefore, we further compress voxels using a 3D VQVAE Xiang et al. [024]] first, we encode the 64 grid into a 16 latent grid: then we serialize it into 4096 tokens. However, 4096 tokens remain too long. Inspired by Team [2024), which represents images as 1024 tokens, we concatenate every four neighboring tokens along the channel
transforming the original 4096 tokens with 8 channels into 1024 tokens with 32 channels. Finally, we employ an 81 8192-entry codebook to compress the voxels into 1024 discrete tokens. In all, we represent single 3D object using 1024 discrete tokens, for both generation and understanding.
Shape Reconstruction Although we employ voxel-based representations for 3D shape generation, practical deployment often necessitates converting voxels into meshes for downstream applications. To address this, we adopt the approach proposed by Xiang et al. Xiang et al. [2024), which utilizes Rectified Flow model to refine and complete voxel information, enabling high-quality mesh reconstruction. By first generating 3D shapes in the voxel domain and then converting them into meshes using this method, our framework achieves a balance between precision and efficiency. This hybrid representation allows large language models to exert fine- iinegrrinned control over 3D content generation while avoiding the computational burden associated with high-resolution geometry.
3.4 3D-Alpaca Dataset Construction
Although a wealth of datasets has been developed for the supervised fine-tuning of multimodal large-language models, dialogue data within the 3D LLM |ong et al. (2023b). Chen et al. (2025b). Xu et al. (024bb domain remains relatively searce. To bridge this gap, we introduce 3D-alpaca, comprchensive datasct encompassing tasks in 3D content generation, comprehension, and editing.
5

---

3D Generation and Understanding Dataset We select a high-quality subset of approximately 712k 3D assets from Trellis Xiang et al. (2024] and internal collection. For the image collection, each asset is rendered into a 2D image, and random offset is applied to the frontal view to create the input. Moreover, these rendered images also underpin the construction of the editing dataset in the Sec. 3.4 To generate the text collection and enable early fusion across all three modalities, we render four orthogonal frews-fronk, back, left, and right-of each asset. These multi-view images are then input into the base model 2.5--....-V---i
Bai et al. [2025] to generate descriptive captions. The resulting captions are utilized both as prompts for text- 3D generation and as ground-truth targets for 3D-to-text captioning tasks.
3D Edited Dataset We aim to build 3D asset-editing dataset composed of paired 3D assets, where each pair is linked to a specific editing instruction. Despite recent advances in 3D content creation, the field still lacks a model capable of performing consistent edits on 3D assets. In light of the promising performance of current anage-editing models, therefore adopt an image-mediated pipeline: first rendering each 3D asset into images and applying an image-editing model, then reconstructing the edited images back into 3D assets via an image-to-3D generation method. Based the multimodal alignment demonstrated and with the aim of equipping the model with CharGPT-4o-Ievel editing capabilities, follow six-step pipeline.
(1) Category: We reference the data distribution of Objaverse- Xl. DDiike et al. [023a] and manually selected the 100 most representative and frequent object categories, such cars, tables, cabinets, human figures, etc.
(2) Asset Classification:
Using ChatGPTT-4,, we classify the 3D assets in our dataset into fine-grained subcategories, with the frontal view renderings of each asset as input. From the 3D asset dataset, we filtered 31 lk assets belonging to the predefined 100 major categories.
(3) Editing- Prompt Definition: We provide the category names CharGPT-40 and instruct it to generate 20 feasible editing-prompts for each category. The instruction given to ChatGPT-44 is: "For each given category name, suggest potential image editing operations that could be applied objects of that category." Next, we manually review each generated editing prompt and retain only those that meet both our technical feasibility and visual engagement criteria, resulting in 371 unique editing prompts (e.g: "Replace the chair's 's backrest with a mesh frame").
(4) Asset Sampling & Annotation: Due to time and resource constraints, we build compact, high- quality dataset of editing prompts rather than applying every possible editing prompt to each asset. Specifically, we allocate 200 assets to each editing prompt.
(5) raditing-mage Pair Collection: For each sampled asset, we provide ChatGPT-40 with its frontal render plus the chosen editing-prompe and ChatGPT-40 produces the corresponding edited image, yielding lmage-level editing pairs. After filtering out erroneous cases, we end up with 70k valid editing samples.
(6) 3D reconstruction: Finally, we employ Trellis Xiang et al. [024] to convert the curated images into 3D assets, resulting in 3D pairs before/after editing.
Dialogue Data Construction We define 25 dialogue templates per task (c.g.. "Generate a 3D asset of promptimages"") and encode all 3D assets into discrete token sequences with our pre-trained 3D VQVAE (Sec. 3.3). For each 3D-edit instance, we randomly select templates from pool of 25; for all other instances, we randomly assign one template each. By merging the tokens with these templates, we create a training corpus of 2.5 million 3D dialogues.
General Conversation To ensure the model's general conversational capability, we adopt Ultra- Chat Ding et al. [2023] as our text-only dataset, with its data distribution shown in the Table 2 For additional details, please refer to the Appendix.
Putting these together After data processing and construction, we finally arrive at the 3D-Alpaca dataset. As shown in the Table the dataset includes four types of tasks: image-to-3D text-to-3D, 3D-to-caption, and 3D-editing. Together, these four subsets form a total of 2.56 million samples, comprising 3.46 billion tokens. To ensure the large language model retains its original reasoning and dialogue capabilities, we additionally include the UltraChat (aing et al. [023) dataset, a high-quality, large-scale multi-turn dialogue corpus.

---

Table 2: Corpus Data Proportions An overview of token and item counts in the training corpus, covering two datasets: the 3DD-Alacca dataset, which includes four task types- Text-to-3 to-3D, Image-to- 3D, 3--to-Captinn, and 3D-Editing and the text-only Ultrahaat dataset Ding et al. (2023)
Text- To-3D
Image- To-3D
Image-To-3D
3D-to-Caption
3D-Edit
3D-AII
Text-Only
Token count
0.77B
1.01B
0.77B
0.91B
3.46B
2.16B
Item count
712k
712k
712k
420k
2.56M
1.47M
4
Experiments
Image
Ours
Sar3D
Trellis
CRM
3Dtopia-XL
Figure 4: Comparisons with other baselines on the image-to-3D task. Our results demonstrate more complete geometry and high-fidelity textures compared to baselines, enabling photorealistic image-to-3i generation.
4.1 Implementation Details
For training our 3D VQVAE, adopt 3D U-Net VAE architecture introduced in Trellis Xiang et al. (2024). Our training follows a two-stage strategy: In Stage 1, we freeze the VAE's pre-trained parameters and train only the codebook. In Stage 2, unfreeze the VAE and jointly fine-tune it with the codebook. Concretely, each stage runs for 1000 steps on 48 NVIDIA H100 GPUs with batch size of 25, while the learning rate decays from 5 x 10 10-3 to x 10 For the training of ShapeLLM-Ommn we use Qwen-2.5-V --n-n-..------sstuutteeee--- with image-understaane understanding capability, our backbone. Specifically, we extend its base architecture by adding the 8192 3D VQVAE codebook. To preserve its original image- unnagg--nddrstnnddng skills, we freeze the parameters of Qwen2.5-vi visual encoder. While training, the learning rate decays from 5 x 10-5 to 5 x 10-6, with per-GPU batch size of 2 and gradient accumulation over steps. The model trained for 15 epochs on 48 NVIDIA H100 GPUs.
Bai et al. 2025 J. a multimodal large language model (MLLM)
4.2 Quantitative comparisons
Language and Conversational Abilities Table presents quantitative results evaluating lan- guage abilities. The table provides a comparison with models: I.LaM-MMshh Wang al. (024b). Chameleon Team 22044) and Qwen2.5-vl Bai et al. 2025 The metrics include SIQA Sap et al.

---

2019), PIQA Bisk et al. (020], MMLU Hendrrckss et al. [2020], and GSM8K Cobbe et al. [2021] Fine-tuned on 3D-Alpaca for both 3D mesh generation and comprehension. our ShapeLLM -mmn maintains language understanding and reasoning performance on par with baseline models. The result demonstrates that ShapeI.L.M-Om effectively extends the MILLM's capabilities 3D content generation while preserving its native language capabilities.
Table 3: Language capabilities comparison. We provide comparison with models: LLaMA- Mesh Wang et al. [2024b], Chameleon [Team] (2024), and Qwen2.5-vl Bai et al. (2025) The metrics include SIQA Sap et al. [2019]. PIQA Bisk et al. 2020). MMLU Hendrycks et al. (2020).
and GSM8K Cobbe et al. [202]] Fine-tuned on 3D-Alpaca for both 3D mesh generation and
comprehension, our ShapeLLM-Omin maintains language understanding and reasoning performance. The table highlights the optimal values bold and the suboptimal values with underlining.
Metric
Qwen2.5-vl-7B
ShapeL.LM-Omni-77
Chameleon-P5 7B
ALaMA-Mesh-aa
MMLU
66.9
63.9
59.4
57.4
PIQA
81.0
78.6
79.6
78.9
GSM8K
42.9
55.1
66.9
33.1
SIQA
40.7
41.0
57
40.4
3D VQVAE Reconstruction Evaluation To assess the reconstruction quality our 3D VQVAE model, we randomly select 1000 samples from the test set and feed them into the model. We then calculate several metrics between original and reconstructed voxel grids, including IoU, Recall, Precision, F1 and Chamfer Distance. These results, summarized in the table 4 indicate that 3D VQVAE model preserves geometric structure with high fidelity, providing a reliable reconstruction basis for following generation tasks.
Table 4: Quantitative Evaluation of 3D VQVAE reconstruction performance. We report IoU, Recall, Precision, F1, and Chamfer Distance between original and reconstructed voxel grids. The results demonstrate that our 3D VQVAE effectively preserves geometric structure with high fidelity.
IOU
Average Recall
Average F1
Average Precision
Chamfer Distance
3D VQVAE 0.9168
0.9357
0.9450
0.9549
0.0214
3D Generation We compare our methods on both text-to-3D and image-to-3D generation tasks against CRM CRM|Wang al. [024cc.. SAR3D Chen et al. (0255) 3DTopia-XI.(Cherr et al. [2024f] and TRELLIS Xiang et al. 2224 When evaluating the generation performance of MhapeeLM-Omee
we set the model's 's top-k parameter equal to the size of the 3D vocabulary (81 (8192), with top-p=0.7 and temperature=0.7. Regarding the dialogue templates, the image-to- 3D template is formulated as: "Create a 3D asset using the following image: <image>", while the text-to-3D template is expressed as: "Please generate a 3D mesh based on the prompt provided: <prompt>" " Quantitative evaluations are conducted using image and text prompts sampled from the Toys4K [Stojanov et al.) [202T) test dataset, with the results summarized in Table 5 To assess the overall quality of the generated 3D outputs, following Xiang et al. [2024) we compute Frechet Distance (FD) Heusel et al. [2017] and
Kernel Distance (KD) Binkowski et al. [2018) using Inception-Vy V3 (Szegeyy et al. [2016] features. Additionally, we report the CLIP score Radford et al. [202T) to measure the semantic alignment
between the generated outputs and their input prompts. As shown in the Table 5 our generation results outperform all baseline methods except for Trellis.
3D Understanding Following the evaluation settings provided by PointLLM Xu et al. (2024)) we test the same metrics on the benchmark dataset used by PointLLM. We adopt the same curated test set to assess the 3D-to-caption task. The dialogue prompt structured as: "<mesh>. Caption this 3D model in detail.". As shown in Table our ShapeL.LM-Ommn demonstrates strong 3D understanding capabilities, with performance second only to P., which is specifically tailored for single-task 3D understanding.
8

---

Table 5: Comparison of methods Text-to-D and Image-to-3D tasks. We scale KD by (x (x10²).
Text-to-D
Image-to-3D
Method
CLIPT
FDincop
KDincop
CLIP
FDincey
KDincep
CRM
76.1
14.7
0.12
3DTopia-XI.
76.5
49.5
1.63
SAR3D
23.9
27.2
0.28
84.70
20.6
0.17
Trellis
30.8
18.3
0.19
85.0
8.31
0.07
ShapeLLM--mml (ours)
26.7
25.9
0.25
84.5
12.2
0.09
Table 6: 3D object captioning results Xu et al. [2024b] on Objaverse Deitke et al. [002aa] As can be seen from the table, model achieves better performance 3D understanding/catton tasks. "*" indicate PointLLM was prompted for shorter captions with no more than 20 words.
Model
B-1
R-L
METEOR
S-BERT
S-CSE
InstructBLIP. 13B Wenliang et al.
2023
4.65
8.85
13.23
45.90
48.86
LLaVA-13B Liu et
2023a
4.02
8.15
12.58
46.37
45.90
GPT4Point Qi et al.
2024b
8.45
10.11
13.13
40.31
42.88
Shapel.LM Q et
2024a
17.88
19.24
17.96
48.52
49.98
3D-LLM Hong et al.
2023b
16.91
19.48
19.73
44.48
43.68
PoinLLLM-13B Xu et al.
2022bb
3.38
7.23
12.26
47.91
49.12
PointLLM-131 Xu et a.
2024b
17.09
20.99
16.45
50.15
50.83
ShapeLLMM-mmi (ours)
18.51
21.37
19.89
49.34
50.72
Prompt
Ours
Trellis
Sar3D
3Dtopia-XL
storyed building
A iguana with shirt yellow
A cartoon robotic white toy
Green coloured toy van rectangular surface yellow front light and red seats with grey coloured wheels
Figure 5: Comparisons with other baselines on text-to-3d task. Our method achieves better text alignment, with 3D shapes accurately reflecting input descriptions.
4.3 Qualitative comparisons
To evaluate the effectiveness of image-conditioned generation, compare against baselines including SAR3D, TRELLIS, CRM, and 3 As illustrated in Figure the baselines exhibit limitations capturing fine-grained visual features, suffering from geometric distortions and texture misalignments. In contrast, our method generates high-quality 3D meshes that preserve both geometry and appearance details. Moreover, our generation quality matches that of TRELLIS, our base model and performance upper bound, due to the integration of a well-trained 3D VQVAE and carefully constructed image-to-3D dataset for LLM one-tuning. For text-to-3D tasks, Figure Saaure//preeennt. qualitative comparisons among baselines. The input prompts are randomly generated by ChatGPT-40 to cover diverse range of objects. Since 3Dtopia-XI. does not support text-to-3D tasks, we use ChatGPT-40 to generate reference images from the prompts. These images evident that our method achieves precise alignment with the text prompts and excels at generating intricate, coherent details.
3D Generation
are then used as input for image-to-3D generation.

---

3D Editing As shown in Figure 6 ShapeLLM M-Omni Omni can edit 3D assets according to user user-provided instructions while maintaining good identity consistency.
Before
+'open the cabinet doors"
+"add lid on top"
"add wings"
"grow a tail"
After
Figure 6: Some cases of 3D editing result from our method. Our method enables the editing of 3D assets based on textual instructions while preserving their original identity and visual consistency,
4.4 Compared with Trellis
Table 7: Comparison of Trellis and ShapeL.LM-Omni in text-to-3D generation performance after task-specific fine-tuning.
Model
CLIP1
FDincep +
KDincep +
Trellis
30.8
18.3
0.19
ShapeLLM- Ommni
26.7
25.9
0.25
ShapeLLM Mammmi (Overfiting) 30.1
18.9
0.21
Our results are slightly inferior to Trellis due to two main factors.
Trellis employs separate models for text-to-3D and image-to-3D generation, whereas our ShapeLLM-Omni unifies six teaks-text-to-311 and image-to-3D generation, 3D understanding. 3D editing, image understanding, and text reasoning within single model that also supports interactive conversation. This all-in-one design introduces optimization trade-offs that can affect generation quality. To verify this, we fine-tuned our model specifically for text-to-3D generation using the pre-trained weights, removing redundant prompts and freezing non-mesh textual embeddings. With a learning rate of 1e-5, context size of 1536, batch size of 4, gradient accumulation of 2, and epochs of training. the fine-tuned model lost its general text capabilities but ,as shown in the Table [ achieved text-to-3D results comparable to drelis-demoonstratiin the inherent difficulty of balancing multiple tasks within a unified framework.
2) Trellis is built on Rectified Flow (diffusion) architecture, while our model adopts discrete autoregressive design. Diffusion and flow-based models currently hold an inherent advantage in visual generation quality, and surpassing them with autoregressive architectures remains an open research challenge. Nonetheless, our focus lies not in pushing the absolute performance of autoregressive models, but in enabling unified 3D generation and understanding under this paradigm-as innovative and promising direction as autoregressive visual generation continues to advance.
Conclusion
In this work, introduce ShapeLLM-Omni, novel framework that advances both 3D generation and understanding through 3D VQVAE. By constructing a comprehensive 3D-Alpaca dataset, we provide data foundation support future research native 3D- 3D-modality large language models.
Limitation Constrained by limited resources, we possess only 70k 3D-editing pairs-far far too few to achieve ChatGPT-4o-leve results in 3D editing. Due to limited computing resources, our Shapel.LM- Omni only has 7B parameters. As a result, our performance hasn't yet reached the level of a true "3D version of ChatGPT-4o".
10

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenshmiidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jean- Saan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:237 35:23716-23736 16-23336, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song. Kai Dang, Peng Wang. Shijie Wang. Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.13923, 2025.
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.0144 2018.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432- 7439, 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rict 3d model repository. arXiv preprint arXiv:1512.0300 2015.
Guangyan Chen, Mciling Wang, Yi Yang, Kai Yu, Li Yuan, and Yufeng Yue. Pointgpt: Auto-regressively generative pre-training from point clouds. Advances Neural Information Processing Systems, 36:29667- 29679, 2023a.
Luxi Chen, Zhengyi Wang. Chongxuan Li, Tingting Gao, Hang Su, and Jun Zhu, Microdreamer: Zero-shot 3d generation in 20 seconds by score- based iterative reconstruction. arXiv e-prints, pages
2024a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv: 13873, 2023b.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d 3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26428- 26438, June 2024b.
Sijin Chen, Xin Chen, Anqi Pang, Xianfang Zeng, Wei Cheng, Yijun Fu, Fukun Yin, Billzb Wang, Jingyi Yu, Gang Yu, et al. Meshxl: Neural coordinate field for generative 3d foundation models. Advances in Neural Information Processing Systems, 37:97 17:97141--7766, 2025a.
Yilun Chen, Shuai Yang. Haifeng Huang, Tai Wang. Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-l1m with referent tokens, 2024c. https:: wrggabs///00000 10370
Yiwen Chen, Tong Hc, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, et Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406. 10163, 2024d.
Yiwen Chen, Yikai Wang. Yihao Luo, Zhengyi Wang, Zilong Chen, Jun Zhu, Chi Zhang, and Guosheng Lin. Meshanything thing v2: Artist-created mesh generation with adjacent mesh tokenization. arXiv preprint arXiv:2408.0l 200.:0255.00055 2024e.
Yongwei Chen, Yushi Lan, Shangehen Zhou, Tengfei Wang, and Xingang Pan. Sar3d: Autoregressive 3d object generation and understanding via multi-scale 3d vqvae. In CVPR, 2025b.
Zhaoxi Chen, Jiaxiang Tang, Yuhao Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie, Tong Wu, Shunsuke Saito, et al. 3dtopia-xl: Scaling high-quality 3d asset generation via primitive diffusion. arXiv preprint arXiv:2409.1 12957, 2024f.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic risual-linguise tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 28185-24199 2024g.
11

---

Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pages 21401-21412, 2024h.
Zilong Chen, Yikai Wang. Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06755 2024i.
Zesen Cheng, Sicong Leng. Hang Zhang. Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476 2024.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL hutps://arxiv. org/abs/21 110. 14168, 9, 2021.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11116-211332 2022.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe 10m+ 3d objects. Advances in Neural Information Processing Syxtems, 36:35799-35813, 2023a.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaversexxl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36:35799 35813, 2023b.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprins a305v:2305. 14233, 2023.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang. Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family large language models from glm- arXiv preprint arXiv:2406. 12793, 2024.
glm-130b 130b to glm-4 all tools.
Yanan Guo, Wenhui Dong. Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingho Wang, Yang Du, Xianing Chen, and Bo Zheng. Fila-video: tempool compression for fine- fine-grained long video understanding. arXiv preprint xrXiv22504.2038 2025.
Zekun Hao, David W Romero, Tsung- Yi Lin, and Ming-Yu Yu Liu. Meshtron: High-fidelity, artist-like 3d mesh generation at scale. arXiv preprini 241.02241,.09548, 2024.
Zexin He and Tengfei Wang. Openirm: Open-source large reconstruction models. https://githa com/ Prrtopia/OpenLRM 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash cquilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan, Lam: Large reconstruction model for single image to 3d. arXiv preprint N.NNXi3311.4440 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-1lm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 20482 20494, 2023b.
Haifeng Huang. Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng. Yang Zhao, Jiangmiao Pang. and Zhou Zhao. Chat-scene Bridging 3d scene and large language models with object identifiers, 2024, URL https:::: / arxiv. org/ 08168
12

---

Haifeng Huang. Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang. Jiangmiao Pang, and Zhou Zhao. Roboground: Robotic manipulation with grounded vision-language priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22540-22550, June 2025a.
Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M Rehg. and Varun Jampani. Spar3d: Stable point-aware reconstruction of 3d objects from single images. arXiv preprint arXiv:2501 046689, 2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt--oo system card. arXiv preprint arXiv:24
2024a.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024b.
Weitai Kang. Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large language model via robust instruction tuning, 2025, URL https:t//arxit .rgg wwwp///..../../. 00255
Berhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatring for real-time radiance field rendering. ACM Trans. Graph,, 42(4):139-1, 2023.
Jiahao Li, Hao Tan, Kai Zhang. Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instamt3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint 23 14, 2023a.
Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors 2d diffusion for consistent text-to-3d. arxiv:2310.02556, 2023b.
Weiyu Li. Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long. Craftsman: figcliiiciitty mesh generation with 3d native generation and interactive geometry refiner. arXiv preprint arXiv:2405. 14979, 2024a.
Yanwei Li. Chengyao Wang. and Jiaya Jia. Llama-vid: An image worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024b.
Chen-Hsuae Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2023.
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Yc, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with vidco diffusion model, 2024b. https
7/arxiv. www.//////////eeeeeeee 2408 16767
Fangfu Liu, Junliang Ye. Yikai Wang, Hanyang Wang, Zhengyi Wang. Jun Zhu, and Yueqi Duan. Dreamreward- x: Boosting high-quality 3d generation with human preference alignment. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025a.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-lengle length video and language with blockwise ringattention. arXiv preprint arXiv:2402.0820 08268, 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing syxtems, 36:34892-34916, 2023a.
Jian Liu, Chunshi Wang, Song Guo, Haohan Weng, Zhen Zhou, Zhiqi Li, Jiaao Yu, Yiling Zhu, Jing Xu, Biwen Lei, et al. Quadgpt: Native quadrilateral mesh generation with autoregressive models. arXiv preprint arXiv:2509.2 12200 2025b.
Jian Liu, Haohan Weng, Biwen Lei, Xianghui Yang, Zibo Zhao, Zhuo Chen, Song Guo, Tao Han, and Chunchao Guo. Freemesh: Boosting mesh generation with coordinates merging. arXiv preprint 135553350 2025c.
Jian Liu, Jing Xu, Song Guo, Jing Li, Jingfeng Guo, Jiaao Yu, Haohan Weng, Biwen Lei, Xianghui Yang, Zhuo Chen, et al. Mesh-rftt Enhancing mesh generation via fine-grained reinforcement fine-tuning. arXiv preprint arXiv:2505.1670 2025d.
Minghua Liu, Chao Xu. Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without pha-shape optimization. Advances in Neural Information Processing Systems, (6.22226-22246,, 2023b.
13

---

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov. Sergey Zakharov, and Carl Vondrick. Zero-I-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9299, 2023c.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consisteee images from single-view image. arXiv preprint arXiv:239.005533 2023d.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingje Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7980, 2024.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347-353, 1998.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint urXiv:2306.05424 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99-106, 2021.
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Eukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski, Hierarchical transformers are more efficient language models. arXiv preprint arXiv:2110.13771 2021.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Zekun Qi, Runpei Dong, Shaochen Zhang. Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d ohject understanding for embodied interaction. Europaan Conference on Computer Vision, pages 214 214-238. Springer, 2024a.
Zhangyang Qi. Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for laint-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26427, June 2024b.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9914 9925, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8876.. PmLR, 2021.
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfrr Aberman, Michael Rubinstein. Jonathan Barron, al. Dreambooth3d: Subject-driven text-to-3d generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2349-2359, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv: 1904.0978. 2019.
Ruoxi Shi. Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.151 1110, 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. /2, 2023b.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießnnr. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19615-19623 2024a.
14

---

Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya. Yanir Kleiman, Emilien Garreau. Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, and David Novotny. Meta 3d assetgen: Text-to-mesh generation with high-quality geometry, texture. and pbr materials. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 9532 9564, Curran Asso- ciates, Inc., 2024b. URL https: p/trceed//proccedingg... neurips. pepuurps.ccppppeeee wwtpapper.tiieeeeceee htt.://llesppp//22//// file// Coooffr/88b77000a////f4464666333
Conference pdf
Stefan Stojanov, Anh Thai, and James M Rehg. Using shape to categorize: Low-shot learning with explicit shape bias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1798-1808, 2021.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yehin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.1683 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 28 2818-2826, 2016.
Jiaxiang Tang. Jiawei Ren. Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv: 16653, 2023.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arX55:2402.0005, 2024a.
Jiaxiang Tang, Zhaoshuo Li, Zekun Hao, Xian Liu, Gang Zeng, Ming- Yu Liu, and Qinsheng Zhang. Edgerunner: Auto-regressive auto-encoder for artistic mesh generation. arXiv preprint arXiv:2409. /81 2024b.
Chameleon Team. Chameleon: Mixed-modal early foundation models. arXiv preprint 240:: 2024. doi: 00448550/a211.2405.000
URL http:/ttww...ooeoocooooooeeeeeea chw.eeeaaaaareeecceelooeeoeooooe
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pages 439-457, Springer, 2024.
Haochen Wang, Xiaodan Du, Jiahao Li. Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.00200 2023.
Peng Wang, Hao Tan, Sai Bi. Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lmm Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:231 120024, 2023a.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen. et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 4563-4573 2023b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang. Zhen Li. Qiying Yu, et al. Emu3: Next-token prediction all you need. arXiv preprint WVXiv:2409.18866 2024a.
Xinzhou Wang, Yikai Wang, Junliang Ye, Zhengyi Wang, Fuchun Sun, Pengkun Liu, Ling Wang, Kai Sun, Xin- tong Wang, and Bin Hc. Animatabledrcamer: Text-guided non-rigid 3d model generation and reconstruction with canonical score distillation. arXiv preprint arXiv:2312.087955 2023c.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao, Chat-3d: orw./oo//////ae..e ab/////b..///../////////// 28066 language model for universal dialogue of 3d scenes, 2023d. URL https/tts:///rrrt
tuning large
15

---

Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Advances in Neural Information Processing Systems (NewrIPS), 2023e.
Zhengyi Wang, Jonathan Lorraine, Yikai Wang. Hang Su, Jun Zhu, Sanja Fidler, and Xiaohui Zeng. Llama-mesh: Unifying 3d mesh generation with language models. arXiv preprint arXwv2441.009555, 2024b.
Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu, Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.055 05034, 2024c.
Xinyue Wei, Kai Zhang. Sai Bi. Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshirm: Large reconstruction model for high-quality meshes. arXiv preprint 1rXiv:2404.12385, 2024.
Haohan Weng, Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, CL Chen, and Lei Zhang. Consistent123: Improve consistency for image 3d object synthesis. arXiv preprint arXiv:2310.08092 2023.
Haohan Weng, Yikai Wang, Tong Zhang, CL Chen, and Jun Zhu. Pivotmesh: Generie 3d mesh generation via pivot vertices guidance. arXiv preprint arXiv:2405. /68900 2024a.
Haohan Weng, Zibo Zhao, Biwen Lei, Xianghui Yang, Jian Liu, Zeqiang Lai. Zhuo Chen. Yuhong Liu, Jie Jiang. Chunchao Guo, et al. Scaling mesh generation via compressive tokenization. arXiv preprint arXiv:2411.0002. 2024b.
Wenliang, L Junnan, Dongxu, T Anthony Meng Huat, Z Junqi. W Weisheng. L. Boyang, F Pascale, and H Steven. Instructblip: Towards general-purpose vision-language models with instruction tuning [c]. Advances in Neural Information Processing Systems, 36, 2023.
Kailu Wu, Fangfu Liu, Zhihan Cai, Runjic Yan, Hanyang Wang, Yating Hu, Yueqi Duan, and Kaisheng Ma. Unique3d: High-quality and efficient 3d mesh generation from a single image. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Shuang Wu, Youtian Lin. Feihu Zhang. Yifei Zeng. Jingxi Xu, Philip Tomr Xun Cao, and Yao Yao. Direct3d: Scalable image-to-34 3d generation via 3d latent diffusion transformer. arXiv preprint arXiv:2405. 14832, 2024b.
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint urXiv:2412.01500 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view view large reconstruction models. arXiv preprint (xXv::440.07719) 2024a.
Runsen Xu. Xiaolong Wang. Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pages 131-147. Springer, 2024b.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311 02222311192221 2023.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 1 1179-1189) 189, 2023.
Xianghui Yang, Huiwen Shi, Bowen Zhang, Fan Yang, Jiacheng Wang. Hongxu Zhao, Xinhai Liu, Xinzhou Wang, Qingxiang Lin, Jiaao Yu, ct al. Hunyuan3d 1.0: A unified framework for text-to-3d and image-to-3d generation. arXiv preprint arXiv:2411.02200 2024.
Yunhan Yang. Yufan Zhou, Yuan Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying- -aanTiin o Liu, Hao Xu, Ding Liang. Pan- Cao. and Xihui Liu. Omnipart: Part-aware 3d generation with semantic decoupling and structural cohesion. arXiv preprint arXiv:2507.06165, 2025.
16

---

Chongjie Ye, Lingteng Qiu, Xiaodong Gu. Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 2024a.
Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han. Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging. arXiv preprint arXiv:2503.22236, 3, 2025.
Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang. Yueqi Duan, and Jun Zhu. Dreamreward: Text-to-3d generation with human preference. In European Conference on Computer Vision, pages 259 276. Springer, 2024b.
Taoran Yi, Jiemin Fang. Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6796 6807, 2024.
Biao Zhang. Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2veeset: A 3d shape reprexentation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG), 42(4):1-16, 2023.
Chubin Zhang. Hongliang Song. Yi Wei, Yu Chen, Jiwen Lu, and Yansong Tang. Geolrm: Geometry aware aware large reconstruction model for high nigh-quality 3d gaussian generation. arXiv preprint arXiv:2406. 15333, 2024a.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for gaussian splatting. European Conference on Computer Vision, pages 1-19. Springer, 2024b.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang. Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: controllable large-scale generative model creating high-quality 3d assets. ACM Transactions on Graphics (TOG), 43(4):1-20, 2024c.
Ruowen Zhao, Zhengyi Wang. Yikai Wang, Zihan Zhou, and Jun Zhu. Flexidreamer: single image-to-23 generation with flexicubes. arXiv preprint 2rX4::2404.00987 2024.
Ruowen Zhao, Junliang Yc, Zhengyi Wang, Guangee Liu, Yiwen Chen, Yikai Wang, and Jun Zhu. Deepmesh: Auto-regressive artist-mesh creation with reinforcement learning. arXiv preprint arXiv:2503. 15265, 2025.
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape- image-tett aligned latent representation. Advances in neural information processing systems, (6.73969-73982, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma. Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
Chen Ziwen, Hao Tan, Kai Zhang. Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, and Zexiang Xu. Long- Long- sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint 17xv:22400.12788 2024.
Zi-Xin 7ou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers, In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 10324-10335. 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction concisely capture the paper's key contribu- tions and scope without overstatement.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper includes dedicated "Limitations" section at the end to explicitly address the constraints of the study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: Yes, for each theoretical result, the paper provides the complete set of underly- ing assumptions along with rigorous and correct proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Confirmed. All replication critical details (data construction ,training parame- ters) exhaustively specified.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data and code access permissions not yet available. The code will be released upon the paper's s formal acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Soblic//guides/Coeeeuubiiss www.eeeeuuuuisssoooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppppccggua.eeaeaeccpuee
Saaeee//CoSSSubisss..S.. www.sssoiicgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper fully presents the training and testing details the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, we evaluated sufficiently large number of test cases, including both quantitative metrics and visual assessments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, these details are fully specified in the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp::///eeuriie
Answer: [Yes]
Justification: Confirmed. The study strictly adheres to the NeurIPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have included a discussion of socictal impacts (both positive and negative) in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our model does not carry such risks, therefore no safeguards are described.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all assets (code, data, models) used in this paper are properly credited to their original creators/owners, with explicit declarations of licensing terms and full compliance with usage requirements.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Confirmed. Complete documentation will be provided with all new assets. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: No, this paper does not involve any crowdsourcing experiments or human subject research, therefore such documentation is applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: no human subjects or sensitive data involved
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: We employed LLMs solely for writing assistance, text editing, and formatting purposes, which did not affect the core methodology scientific validity of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurij) httrre//eearrrsscccCoofeeereeeee for what should or should not be described.
24

---

A More Experiments
A.1 More Implementation details
Text/Image Condition
Sparse Flow Transformer
Texture latent
Voxel-to-Mesh Decoder
Voxel
3D Mesh
With Texture
Figure About how generate 3d mesh from voxel. This image illustrates the process of reconstructing a textured mesh from voxel inputs using a texture transformer Xiang et al. [2024] and mesh decoder.
upper part of Figure 7 we first utilize a texture transformer, named Sparse-Flow Transformer Xiang et al. [2024] to extract texture latents from the voxel representation. These latents are then fed into a voxel-to-mesh decoder, which generates a mesh with associated texture information. Interestingly, we observe that the geometry of the output mesh is entirely determined by the input voxel representation, regardless of the presence of texture information.
Decoding Voxel into 3D Mesh As illustrated
More Details about Training The model is trained on 48 H100 GPUs for 60k iterations. We conduct full parameter fine-tuning. We use the Adam optimizer, with learning rate of 1e-5, warm-up of 400 steps with cosine scheduling, and a global batch size of 192. The total training time around 5 days.
A.2 More details about 3D-Alpaca
3D Editing Prompt List As shown in Table 13 Table we present 70 out of the 100 categories from the 3D editing dataset, along with their corresponding editing prompts.
3D Editing Data As shown in Figure T0 The figure illustrates that 3D editing data pairs support effective modifications while preserving subject consistency between the original and edited versions.
we present several examples from our 3D editing dataset.
A.3 More Qualitative comparisons
In Figure []] Figure 12 and Figure 13 we showcase additional Image-to-3) generation results. To
maintain consistency with the training setup, all input images are resized to 5 512x512 resolution with white background. This preprocessing step is crucial, as our base model, Qwen VL Bai et al. (2025). encodes images into token sequences whose length depends
the input resolution. Additional Text-to-3D generation examples are presented in Figure [44] The visual results clearly demonstrate that model is capable of producing high-fide Sigh-fiielity 3D assets through unified architecture. Furthermore, Figure 9 provides additional 3D-to-caption generation results, and Figure g shows two caption examples from Objaverse Deitke al. [202aa]. The generated captions demonstrate that Shapel.. exhibits robust 3D understanding capabilities.
A.4 More Quantitative comparisons
Image-to-3D To provide a more comprehensive comparison, we quantitatively evaluate the image to 3D generation performance of OpenLRM, LGM, InstantMesh, and Unique3D the same test set used in our paper. We also adopt the same evaluation metrics, including CLIPScore, F Dnneptp and KD incept- As shown in the table 8 our method consistently outperforms all baselines and demonstrates superior 3D generation quality.
Evaluation GPT-Annotated Dataset To quantitatively assess the quality of our GPT-annotated text-to-3D dataset, randomly sample 1000 text-image pairs and evaluate their semantic alignment
25

---

UID InstructBLIP 3D-LLM
0ea33666771453007766777222222222
An appleavatar model model of red apple.
black insect
small, black spider with long tail.
This 3D model of unique apple,
This 3D model depicts realistic, jet set-black
PointLLM
distinctively adorned with a single, vibrant green leaf at the top. An apple with stem leaf. Figure 8: Qualitative results on Objaverse.
insect with pair of striking, golden brown
eyes
ShapeLLM-Omm
spider with multiple legs segmented body
Rendering tedteiggg impout mesh
Caption
robot with rectangular head and four limbs.
futuristic vehicle with multiple sections and protrusions.
lighthouse with cylindrical base and tower.
fantasy creature with wings and pointed head.
Figure Some cases of 3D-to-caption result from our method.
using CL.PScorre and ViLT R-Precision. As shown the Table 9 our dataset achieves high scores across both metrics, indicating strong correspondence between the captions and 3D content.
Evaluation of 3D Editing Dataset To evaluate the alignment in 3D edting dataset, we compute the same metrics between the editing prompts and the rendered front-view of edited objects. The results, summarized in Table TO indicate the editing prompts are well aligned with the resulting modifications.
We conduct additional experiments to evaluate our model on 3D question- answering tasks. As shown in the table below, the best scores for each metric in the Table II highlighted. Our model achieves consistently superior performance across all methods, demonstrating its strong capabilities in 3D QA tasks.
Evaluation of QA
A.5 Ablation
To determine the optimal codebook size for our 3D VQVAE model, we train several variants with different codebook sizes. We randomly sample 1000 meshes from the test dataset, voxelize them, and encode them into discrete tokens using each model. These tokens are then decoded into voxel grids and converted back meshes through voxel-to-mesh decoder. We evaluate reconstruction quality using Chamfer Distance (CD) and Hausdorff Distance (HD). As shown in Table I2 larger codebooks lead better reconstruction performance. However, the improvement levels off beyond codebook size of 8 8192, indicating saturation. We therefore choose 8192 as the final codebook size strike a balance between quality and efficiency.
3D VQVAE
26

---

Table 8: More quantitative comparison of image-to-3D generation performance across OpenLRM, LGM, Mesh, and Unique3D on shared test set, evaluated using CLIPScore, F inoeptss and
K D incept
Model
CLIPT
incep
KDincep
Instantmesh Xu al. 2024a
74.50
22.0
0.25
OpenI_MM He and Wang 2023
72.75
20.1
0.26
U et al. 2024a
77.10
16.8
0.13
LGM Tang al. 2024a
75.20
18.5
0.15
ShapeLLM-Ommi
84.50
12.2
0.09
Table Quantitative evaluation of the GPT-annotated text-to-31) dataset using CLIPScore, ViLT R- R-Precision@5 and ViLT R- R-Precision @ 10.
CLIPScore ViLT
R-Precision R@5
R-Precision R@ 10
Our
29.58
35.3
42.2
Table 10: Quantitative evaluation of the GPT-annotated 3D
Editing dataset using CLIPScore, ViLT
R-Pre @ and ViLT R- R-Precision @ 10.
CLIPScore ViLT
R-Precision R@s
R-Precision R@ 10
Our
27.41
33.6
43.5
Table
Performance comparison on 3D question-answering tasks.
Model
B-1
R-L
METEOR
S-BERT
S-CSE
ShapeLLM Qi et 2024a
17.73
19.91
21.86
51.32
50.95
GPT4Point Qi et 2024b
6.51
8.59
5.80
29.66
32.18
PointLLM- Xu et al. 2024b
17.23
19.70
20.48
52.66
53.21
ShapeLLM-Omni (Ours)
19.66
21.31
22.68
52.51
53.33
Table 12: Ablation study on the codebook Size of 3D VQVAE
Vocabulary Size
Chamfer Distance.
Hausdorff Distance.
4096
0.0102
0.0561
8192
0.0525 0.0094
16384
0.0095
0.0534
27

---

Table 13: Edited Prompt Collection: Part One
ID
Category
Edited prompt
Add cannon to the front, Open the door. Add rool rack, Add rear wing. Lengthen the car body, Shorten the car body, Convert into a convertible, Change wheels to square shape, Bend the roof, Add air vents on the sides, Install spotlight on the roof, Open the hood, Install rear-view camera
Car
2
Tricycle
Add a wheel, Install a small trumpet
Bicycle
Raise the seat, Add a wheel, Install basket
Traffic light
Add an extra light, Install surveillance camera
Add wings, Add jet flames, Add solar panels, Install radar antenna, Shorten fuselage, Bend the tail downward. Bend the tail fins upward, Widen wingspan, Narrow the wingspan, Tilt the whole body, Mount small missiles on wings
5
Spaceship
6
Tank
Rotate cannon to the side, Mount a telescope on the turret top
Raise both hands, Raise left hand, Raise right hand. Hold sword, Enlarge the head, Sit cross cross-legged. Wear backpack, Wear shoulder bag. Change to running pose. Grow puir of wings, Stand on wind-fire wheels. Step rocket launchers, Wear glasses, Wear tall hat, Spread arms, High knee movement, Stand on one leg. Add cape, Hold shield, Grow tail, Twist the waist, Stand skateboard, Change hairstyle
7
Character
bun, Enlarge the ears, Bend the elbows, Wear armor, Kneel both legs, Cross both arms, Add halo above the head Turn feet into wheels, Turn hands
hayonets, Wear an Iron Man helmet, Lengthen arms, Mount mechanical wings on the back, Add antenna to head, Add springs to soles, Mount rocket buoster on the back, Lengthen legs, Turn hands into cannons, Turn hands into claws, Turn arms into chainsaws, Add solar panels to the back, Transform into spider legs
Robot
Put a vase on the table, Change table shape to round, Lay a tablecloth, Spiral- shaped table legs, Add drawer under the tabletop, Jagged edges on the tahlctop, Dig hole in the center, Put cup on the table, Add wheels under table legs, Put fruit plate on the table
Table
Place cushion, Extend the legs. Shorten the legs, Add wheels to the feet, Install footrest, Place seat cushion, Add storage bags the sides, Put a speaker on it. Turn into rocking chair
10
Chair
Add cabinet doors. Open the cabinet doors, Add drawers, Pull drawer, Put table lamp top, Add lock, Add internal shelves, Place potted plant top
Cahinet
Bowl: Change to square, Put egg inside, Add pair of chopsticks
Add a pillow, Change round shape, Add curtains, Place kitten on the bed, bed
12
Bed
Convert into
13
Sofa
Place blanket, Place a teddy bear, Add throw pillow
14
Bowl
Change to square, Put an egg inside, Add a pair of chopsticks
15
Backpack
Transform into jetpack, Transform into rolling backpack
Lengthen the harrel, Add harrels on both sides, Mount scope top, Add magazinc slot the left, Attach bayonet under the muzzle
16
Gun
17
Shoes
Extend the upper part, Thicken the sole, Attach wind-fire wheels Convert to short-sleeva Convert to long- Add scarf
Clothes
19
Hat
Raise the crown, Add wings to the sides, Turn the top into animal ears Change to round frames, Add a head strap, Remove the frames Add a diamond, Remove the diamond
20
Glasses
21
Ring
22
Knife
Extend the blade, Turn into 'Zangetsu" from Bleach
Lengthen the blade, Wrap the blade in flames, Make the blade serrated, Add ring guard the hilt, Embed gems in the blade
23
Sword
Change the spout length, Open the lid, Turn the spout into chainsaw, Add heater at the bottom
24
Teapot
Only upper half remains, Insert rose, Pour tea into the bottle, Replace cap with cork. Tie label around the neck Turn into conical flask, Add handle, Add lid, Insert a straw, Add cup heater Jumping pose, Skating on a skateboard, Add pair of wings. Wear clothes, Wear a bow on the head
25
Bottle
25
Cup
Cat
27
Dog Insect
Hold a bone in mouth, Add a dog leash, Wear clothes, Wear Christmas hat Remove wings, Remove antennae, Add antenna, Add pair of wings
28
Fish
Wear goggles
Block Block-shaped Object
Be stretched
Ball-shaped Object
Change
28

---

Table 14: Edited Prompt Collection: Part Two
Category
Edited prompt
Add chimney roof, Add open door, Change roof to dome, Change door to arch, Add canopy on door, Add garage on the side, Add balcony, Add street lamp next to house, Add fence, Add a mailbox at entrance, Install solar panels on roof
32
House
Shorten height, Add flag top, Add door at base, Add spotight tip Add fence around, Add antenna top, Add spiral staircase outside, Add window in middle, Add vines surface, Keep only lower half, Add observation deck top Grow giant hands, Grow giant flowers on top, Grow stars top, Grow two long legs, Grow large wings on sides, Butterfly perching on tree, Add a door on trunk, Hang lanterns on branches
33
Tower
Tree
35
Flower
Add more petals, Insert into vase, Bee perching on it Put in fruit plate, Peel skin, Insert small umbrella surface Be stretched
36
Fruit
Vegetable
Phone
Turn tri-flld screen, Add stylus edge
39
Computer
Grow wheels
40
TV
Add two antennas, Install base stand
41
Keyboard
Change to round keycaps
Book
Grow arms and legs, Grow wings
Add arched entrance in front, Install antenna on roof, Add chimney on roof, Add external staircase, Add billboard on top. Helicopter parked on roof, Add fence in front, Make building round, Install solar panels on roof, Add flag roof, Change door to revolving door, Add clock on wall, Hang string lights wall Remove column, Change flat roof, Convert to castle top, Add cable support structure
Building
44
Building Structure
pair wings, Wear sunglasses. Wear headphones, Wear hat, Add halo above, Add fence around, Add multiple arms, Change head Medusa, Wear a flower crown, Be wrapped in chains
45
Statue
Change bulb to square, Change lampshade shape, Add more lamp heads, Change lamp head direction, Add hanging chains
Lamp
Replace rectangle door with arch, Add doorbell, Add surveillance camera, Add door lock, Add steps entrance, Open the door, Wrap door with vines, Add peep-
47
Door
Bird: Claw grasping branch. Wings spread, Pecking downward. Lengthen beak, Shorten heak, Wear top hat, Hold branch beak, Wear goggles Wear crown, Wear armor, Wear mask, Hold scepter
48
Sculpture
49
Weapon
Add hook at front, Make blade wavy. Change to double-headed, Be chained Add goggles, Add visor, Change pointed top, Unfold side wings
50
Helmet
Convert suspension bridge, Add pillars, Make multi-level, Add street lights, Add toy cars
51
Bridge
52
Vase
Insert flowers, Place on table, Add handles on sides
53
Mechanical Arm
Replace hand with clamp. Arm rotates
54
Plant
Add fruits, Broken branches, Grow upwards
Shield
Change octagonal, Emhed gem in center, Insert arrow, Wrap vines
56
Chest
flattened, Open lid, Lock with chains
Mount missiles under wings, Retract landing gear, Extend landing gear, Add more engines
Airplane
Add drawbridge entrance, Attach dragon on wall, Connect towers with bridges. Hang flags walls
58
Castle
Mythical Creature
Add saddle, Grow spikes on back, Sleep curled on ground
Pillar
Change to polygonal, Bend to one side, Add grooves to body
61
Tool
Lengthen handle, Replace tool head with bayonet, Bend the handle
62
Lighthouse
Add radar antenna top, Add spiral staircase outside, Add window
Box
Be flattened, Open the lid, Punch hole
Monument
Change top to pointed, Add flag on top, Add steps base
65
Animal
Grow antennae
66
Stairs
Add more steps. Change to spiral stairs. Remove handrails
67
Tent
Extend awning, Change dome-shaped
Street Light
Add signboard on pole, Add camera on pole
69
Trophy
Add lid, Add handles
70
Machine
Add wheels
29

---

Before
Editing Action
After
add storage bags on the sides
add wheels under table legs
change hairstyle to bun
enlarge the ears
Wear a tall hat
mount a scope on top
put a vase on the table
turn the spout into a chainsaw
Grow a pair of wings
Figure 10: Some cases of our 3D-Editing Data
30

---

Input
3D Mesh Output
1
Figure 11: More cases of Image-to-3D result from method.
31

---

Input
3D Mesh Output
Figure 12: More cases of Image-to-3D result from our method.
32

---

Input
3D Mesh Output
Figure 13: More cases of Image-to-3D result from our method.
33

---

Input
3D Mesh Output
A truck
sword with hilt and pointed tip
A small house with a snow- covered roof
pair pants with tied waistband
A large building with covered porch and muttiple doors
handgun with suppressor
handgun with a grip and barrel
guitar pickguard and bridge
A hammer with a handle and head
A building with staircase and balcony
A chair with curved backrest and four legs
female figure in futuristic outfit with high heels
Figure 14: More cases of Text-to-3D result from our method.
34


Reviews:
[{"review_id": "Gia1BBFqIp", "reviewer": "Reviewer_pjk4", "summary": "This work demonstrates an effective approach to integrate 3D generation (text-to-3D and image-to-3D), understanding (3D captioning) and editing abilities into a pre-trained multimodal large language model (MLLM). In essence, they build a 3D foundation model. Furthermore, they introduce a large-scale 3D dataset, 3D-Alpaca, consisting of 2.56M data samples divided into 4 subsets, 1 for each of the following tasks: text-to-3D generation, image-to-3D, 3D captioning and 3D editing. Their approach involves first training a 3D VQVAE to encode voxel-based representations into compact discretized tokens which can be decoded back to voxels. They take Qwen2.5-VL-Instruct-7B as their backbone MLLM which is trained on their 3D-Alpaca dataset to extend its generative and understanding capabilities to 3D assets. The discretized token 3D representation facilitated by the 3D VQVAE enables them to feed these tokens as input to the model (for captioning and/or editing) and also to generate these tokens in an autoregressive next-token prediction paradigm, effectively treating them in a similar manner to discrete textual tokens. They demonstrate the efficacy of their model on all 4 tasks with extensive experiments to compare against previous baselines.", "questions": "1. Is the primary reason for using a voxel-based representation the ease of encoding given that voxels are inherently discrete? However, this comes at a cost such as loss in finer details which is why quite a few approaches have been explored to encode and/or compress continuous representations for any downstream task. Did you look into any of these representations and approaches? In essence, I just want to better understand the thinking that went behind the choice of using voxels.\n2. It would help if some of the weaknesses mentioned above could be addressed.", "limitations": "Not sufficiently discussed.\n1. Voxels have a limited resolution which means that they result in the loss of information and are not adequate to model finer details.\n2. The data creation pipeline is dependent on several external models which means that the quality of the dataset is closely tied to the quality of the models used. While this approach to use existing SOTA models for synthetic dataset creation and/or labeling is becoming more common, specially given the rapid advancements made in some of these models, it is still important to acknowledge the inherent limitation on the quality of the dataset.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is very well-written and was easy to understand.\n2. This work addresses two important problems. There is a lack of 3D foundation models that can both understand and generate 3D assets and a shortage of adequate large-scale 3D datasets.\n3. Use an additional text-based dataset, UltraChat, to ensure the model retains its original reasoning and dialogue capabilities.\n4. Included many visuals to showcase the model's quality.\n5. Simple but effective dataset curation approach.\n6. Simply using discretized 3D tokens as input and output enables the seamless integration of 3D abilities into the model without significant changes to the core architecture. This further enhances the scalability of this approach as it can be applied to most MLLMs and is not strictly restricted to a specific model type.\n\n**Weaknesses:**\n1. It would have been to nice to see some results to evaluate the quality of their 3D VQ-VAE considering that it is a vital part of the methodology. Specifically, want to get an idea of how lossy the compression/encoding is. This can be checked by feeding some 3D assets from an unseen dataset (or even an unseen part of their own dataset but would be better if it is from a different dataset) to the VQ-VAE, getting the token representation, decoding it, and comparing the decoded asset with the original.\n2. The comparisons against other models could have been a little more comprehensive. While it is not possible to compare against all relevant models, it would still have been better to cover some of the more popular models that have proven to be effective. For instance, for the 3D generative task, models such as OpenLRM, LGM, InstantMesh and Unique3D could have been included.\n3. Not much discussion on the limitations of the work and the cons of some of the choices such as the downsides of using a voxel-based representation.", "was_revised": true, "final_justification": "This work presents an effective approach to extend a pre-trained MLLM with 3D generation, understanding, and editing capabilities, supported by the introduction of a large-scale dataset, 3D-Alpaca. The paper is clearly written, well-motivated, and demonstrates strong results across four tasks, with a simple yet scalable design that can integrate with various MLLMs. The rebuttal addressed all of my concerns sufficiently, including additional experiments on the VQ-VAE quality, image-to-3D evaluation, and limitations discussion. Thus, taking everything into consideration, I have raised my rating of this work.", "rebuttal": "We thank the reviewers for the constructive reviews. We provide our feedbacks as follows. \n\n> **Weaknesses1**\n\n**A1:** To assess the reconstruction quality of our 3D VQ-VAE model, we randomly select 1000 samples from the test set and feed them into the model. We then calculate several metrics between original and reconstructed voxel grids, including IoU, Recall, Precision, F1 and Chamfer Distance. These results, summarized in the table below, indicate that our 3D VQ-VAE model preserves geometric structure with high fidelity, providing a reliable reconstruction basis for following generation tasks.\n| **Method**   | **IOU**   | **Average Recall** | **Average F1** | **Average Precision** | **Chamfer Distance** |\n|:------------:|:---------:|:------------------:|:--------------:|:---------------------:|:--------------------:|\n| 3D VQVAE     | 0.9168    | 0.9357             | 0.9450         | 0.9549                | 0.0214               |\n\n> **Weaknesses2**\n\n**A2:** To provide a more comprehensive comparison, we quantitatively evaluate the image to 3D generation performance of OpenLRM, LGM, InstantMesh, and Unique3D on the same test set used in our paper. We also adopt the same evaluation metrics, including CLIPScore, FD_{incept}, and KD_{incept}. As shown in the table below, our method consistently outperforms all baselines and demonstrates superior 3D generation quality. \n| **Method** | **CLIP↑** | **FDincep↓** | **KDincep↓**(×100) |\n|:-------------:|:---------:|:------------:|:----------------------:|\n| Instantmesh   | 74.50     | 22.00        | 0.25                   |\n| OpenLRM       | 72.75     | 20.10        | 0.26                   |\n| Unique3D      | 77.10     | 16.80        | 0.13                   |\n| LGM           | 75.20     | 18.50        | 0.15                   |\n| ShapeLLM-4o   | **84.5**  | **12.2**     | **0.09**               |\n\n> **Weaknesses3**\n\n**A3:** Our current voxel-based representation provides effective structural alignment, enabling efficient tokenization with 3D VQVAE and seamless integration with autoregressive models. However, it does come with certain limitations. Specifically, voxel representations incur high memory and computational costs at higher spatial resolutions. Moreover, voxel-based representations are unable to model textures, and the use of low-resolution voxels limits ShapeLLM-4o's ability to understand 3D assets at a finer level of granularity. In future work, we aim to explore more efficient and finer-grained 3D representation methods.\n\n> **Questions1**\n\n**A4:** Thank you for your valuable suggestion. We will add some discussion on this section in the main text. In fact, we have considered other encoding methods. For meshtron, encoding mesh into discrete tokens through spatial quantization can result in excessively long sequences, which is not conducive to training autoregressive models. For continuous features represented by vecsets, we believe that on the one hand, their information content will be relatively rich, and it may be difficult to train a complete 3D vqvae for encoding. On the other hand, vecsets are implicit features, while voxels are explicit features, and 3D editing may require more explicit features. Considering downstream tasks, prevalent approaches (e.g., Trellis) increasingly adopt coarse-to-fine generation frameworks. Such pipelines typically first generate coarse voxel structures followed by high-resolution mesh refinement. Consequently, leveraging voxel representations enables ShapeLLM-4o to seamlessly integrate with contemporary 3D generation workflows, establishing greater generalizability.\n\n> **Limitations**\n\nTo quantitatively assess the quality of our GPT-annotated text-to-3D dataset, we randomly sample 1000 text–image pairs and evaluate their semantic alignment using CLIPScore and ViLT R-Precision. As shown in the Table below, our dataset achieves high scores across both metrics, indicating strong correspondence between the captions and 3D content. \n\n|            | **CLIPScore** | **ViLT R-Precision R@5** | **ViLT R-Precision R@10** |\n|:----------:|:-------------:|:------------------------:|:-------------------------:|\n| **Ours**   | 29.58     | 35.3                 | 42.2                 |\n\nSimilarly, to evaluate the alignment in 3D edting dataset, we compute the same metrics between the editing prompts and the rendered front-view of edited objects. The results, summarized in Table below, indicate the editing prompts are well aligned with the resulting modifications.\n\n|            | **CLIPScore** | **ViLT R-Precision R@5** | **ViLT R-Precision R@10** |\n|:----------:|:-------------:|:------------------------:|:-------------------------:|\n| Ours       | 27.41         | 33.6                     | 43.5                      |"}, {"review_id": "t3nDWcsHog", "reviewer": "Reviewer_LdZP", "summary": "This paper introduces a new mllm called ShapeLLM-40. It mainly focuses on 3D generation and understanding. Compared with some existing models like GPT-4o, this model can handle 3D data together with text and image, so it can solve some limitations before. ShapeLLM-40 uses a 3D VQVAE to change 3D objects into discrete tokens. In this way, the model can process text, image and 3D data together in one framework. For training, the authors build a new dataset named 3D-Alpaca, which includes tasks like text-to-3D, image-to-3D, 3D caption and 3D editing. From the experiments, ShapeLLM-40 gets better results than baselines such as Trellis, SAR3D and 3DTopia-XL. It shows better text matching and generation quality, and also keeps good language and conversation ability.", "questions": "Please answer my concerns in the Weaknesses.", "limitations": "There's no obvious negative societal impact but for the limitations please refer to the Weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Provide a good baseline to bridge both understanding and generation in 3D domain.\n2. Contribution an valuable dataset to the community for future works.\n3. The motivation to use 3D VQVAE for 3D modality encoding and decoding is clear and sound.\n\nWeaknesses:\n\n1. The 3D capability only focus on object-level, which limits its application in real life or contribution to 3D reasoning community. Currently, 3D scene level understanding is more realistic and challenging. Please consider to evaluate on 3D scene level benchmark for 3D captioning, QA, and Grounding. Related works include Robin3D, Chat-Scene, LL3DA, etc.\n2. When applied to 3D scene-level reasoning, this 3D VQVAE would inevitable increase computational cost. As shown in Robin3D and Chat-Scene, a scene can be represented by 100 to 150 objects which means will scale up 100 to 150 times of the current number of vision tokens. I don’t think this is a practical approach.\n3. The approach for generating 3D Generation and Understanding Dataset, which is actually generating paired-language data, is similar with Grounded 3D-LLM. Therefore, even though I value the engineering effort, there is indeed not much insight or novelty to the research field.\n4. Most data are generated from GPT. There’s no guarantee on its quality since GPT will hallucinate. How to assess its quality would be a problem since data is the main contribution of this paper.\n5. Given the huge training cost, especially 48 NVIDIA H100, and with a more advanced backbone Qwen-2.5-vl, the sub-optimal performance in Tab.4 shows that the proposed method is not very effective or efficient.", "was_revised": true, "final_justification": "During the rebuttal, the authors have solved my concerns on why their method hasn't compare with scene-level method. However, my concern about their novelty on constructing language data hasn't been fully solved. Their proposed data engine highly overlapped with previous works like Grounded 3D-LLM, Chat-Scene, Robin3D, etc. \n\nGiven that, they are the preliminary work on doing both understanding and generation in 3D field. It is hard to confidently identify their contribution regarding novelty.\n\nAbove all, I raise my score to acceptance level but downweight my confidence score.", "rebuttal": "We thank the reviewers for the constructive reviews. We provide our feedbacks as follows. \n\n> **Weaknesses1**\n\n**A1:** We emphasize that 3D object-level generation and understanding tasks are fundamentally different from scene-level ones in terms of objectives, input modalities, and methods. Due to this gap, models designed for object-level tasks often cannot be directly applied to scene-level benchmarks, and vice versa. This distinction is well acknowledged in the community—existing models such as PointLLM, GPT4Point, Trellis, and ShapeLLM are all tailored specifically for object-level tasks. This is not a limitation of the algorithms themselves, but a reflection of the fundamental differences between the two problem settings. Therefore, existing benchmarks designed for 3D scene understanding are not directly applicable to our method and do not offer a fair comparison. \nMoreover, object-level tasks serve as the foundation for 3D generation and understanding, since the ability to accurately generate and interpret 3D objects is critical for applications in robotics, virtual reality, gaming, and other domains. Robust object-level modeling is necessary for scaling to more complex 3D scene-level tasks.  We contend that ​the practical realism and methodological challenges inherent in object-level tasks are commensurate with those of 3D scene-level tasks.​\nDespite the importance, research combining unified autoregressive models with 3D object generation and understanding is few. Our work focuses on object-level tasks, marking an important step toward unifying 3D object generation and understanding within a generative modeling framework. \n\n> **Weaknesses2**\n\n**A2:** Our proposed unified autoregressive framework and VQ-VAE are specifically designed for object-level tasks, with tailored methodologies, data structures, and objectives not applicable to scene-level tasks. Therefore, directly employing existing scene-level 3D benchmarks to evaluate our framework is not only inappropriate but also fails to reflect the scope and advantages of our approach. Just as Trellis cannot be expected to generate complex scenes, our object-level trained model fundamentally lacks the architectural capacity for scene generation and understanding.\n\n> **Weaknesses3**\n\n**A3:** While our data construction is GPT-assisted, as in prior works, our main contribution lies not in proposing a novel data generation pipeline, but in constructing a critical but underexplored dataset. Specifically, Grounded 3D-LLM emphasizes scene-level understanding, whereas our dataset targets object-level editing. To the best of our knowledge, object-centric 3D editing data remains extremely scarce in existing work. We believe our dataset fills this important gap and offers a valuable resource for advancing research in controllable 3D generation and editing.\n\n> **Weaknesses4**\n\n**A4:**  To quantitatively assess the quality of our GPT-annotated text-to-3D dataset, we randomly sample 1000 text–image pairs and evaluate their semantic alignment using CLIPScore and ViLT R-Precision. As shown in the Table 1, our dataset achieves high scores across both metrics, indicating strong correspondence between the captions and 3D content. \n\n|            | **CLIPScore** | **ViLT R-Precision R@5** | **ViLT R-Precision R@10** |\n|:----------:|:-------------:|:------------------------:|:-------------------------:|\n| **Ours**   | 29.58     | 35.3                 | 42.2                 |\n\nSimilarly, to evaluate the alignment in 3D edting dataset, we compute the same metrics between the editing prompts and the rendered front-view of edited objects. The results, summarized in Table 2, indicate the editing prompts are well aligned with the resulting modifications.\n\n|            | **CLIPScore** | **ViLT R-Precision R@5** | **ViLT R-Precision R@10** |\n|:----------:|:-------------:|:------------------------:|:-------------------------:|\n| Ours       | 27.41         | 33.6                     | 43.5                      |\n\n> **Weaknesses5**\n\n**A5:** Our framework jointly addresses multiple tasks—including 3D generation, understanding, and language capabilities—which inherently increases the training complexity and resource requirements. In particular, maintaining strong language capabilities while integrating diverse 3D tasks demands substantial computation. We verified it by conducting a fine-tuning experiment specifically targeting text-to-3D generation, based on the pretrained weights. During fine-tuning, we removed redundant prompt tokens. Additionally, we froze the gradients of all text token embeddings except for the mesh tokens. The learning rate was set to 1e-5, with a context length of 1536, batch size of 4, and a gradient accumulation step of 2. The fine-tuning lasted for 5 epochs. After fine-tuning, the model lost general text processing ability but improved significantly on text-to-3D tasks, achieving better results on our test dataset. This shows the model can gain on a specific task by sacrificing general capabilities.\n\n| **Method**   | **CLIP↑** | **FDincep↓** | **KDincep↓** | \n|:------------:|:-----------:|:------------:|:-----------:|\n| ShapeLLM-4o|   26.7   |   25.9   |   0.25 |   \n| ShapeLLM-4o（Overfitting）|   **30.1**   |   **18.9**   |   **0.21** |"}, {"review_id": "Km4NtlhrTH", "reviewer": "Reviewer_zpad", "summary": "This paper addresses the modality limitation of current unified multimodal large language models (MLLMs), such as GPT-4o, which primarily support only image and text. To overcome this, the authors propose ShapeLLM-4o, a native autoregressive MLLM with the capability to understand and generate 3D objects.\n\nThe model leverages a pretrained VQ-VAE for 3D object representations, using the encoder for object understanding and the decoder for generation. To enable 3D reasoning capabilities, the authors construct a large-scale dataset covering tasks such as understanding, generation, and editing of 3D content. The training is based on the Qwen-2.5-VL-7B-Instruct backbone, pushing the frontier of multimodal LLMs toward the 3D modality.", "questions": "- In constructing the 3D editing dataset, it is stated that the authors use image editing followed by 3D reconstruction. Does this mean the model is prompted with an input image and generates edited images from multiple viewpoints?\n  - How is consistency across multiple views ensured during this image editing process?\n  - What specific 3D reconstruction method is used to obtain the final 3D shape?\n  - A detailed visualization of this pipeline (from image edit to 3D generation) would be very helpful.\n- The paper adopts Trellis’s 3D tokenizer while using a more powerful Transformer backbone (Qwen-2.5-VL), yet the 3D generation quality is significantly worse. However, the paper does not analyze this performance gap in detail.\n- Is the main architectural difference between this work and Trellis simply replacing the flow transformer with a VLM backbone?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper presents the first autoregressive framework that unifies 3D object understanding and generation, demonstrating the feasibility of extending LLMs to the 3D domain.\n- It contributes a large-scale dataset tailored to training unified 3D-capable MLLMs.\n- The study offers an exploratory step toward expanding the modality range of general-purpose MLLMs.\n\n**Weaknesses**\n- The idea of using a shared tokenizer for understanding and generation has already been validated extensively in the vision domain (e.g., EMU3), and is not novel in principle.\n- In 3D generation visualizations, the results show noticeable inconsistencies compared to baselines like Trellis (e.g., in the third row, the number of buttons on the TV is incorrect).\n- In the 3D understanding evaluation, comparisons are made only with PointLLM, which is relatively outdated. More recent baselines should be included for a fair assessment.", "was_revised": true, "final_justification": "- The paper presents the first autoregressive framework that unifies 3D object understanding and generation, demonstrating the feasibility of extending LLMs to the 3D domain.\n- It contributes a large-scale dataset tailored to training unified 3D-capable MLLMs.\n- The study offers an exploratory step toward expanding the modality range of general-purpose MLLMs.", "rebuttal": "We thank the reviewers for the constructive reviews. We provide our feedbacks as follows. \n\n> **Weaknesses1**\n\n**A1:** While prior works such as Emu3 have demonstrated the effectiveness of using a shared tokenizer for both understanding and generation in the 2D vision domain, extending this paradigm to 3D data is far from straightforward. In general, mapping modality-specific features into a unified language space is a powerful and broadly applicable idea—one that spans across modalities from text to vision. However, each modality presents its own challenges. In the case of 3D, the representation is structurally richer and requires maintaining spatial integrity and multi-view consistency, which makes joint modeling significantly more difficult than in 2D. Our contribution lies in successfully adapting this framework to the 3D object domain by employing a new voxel-based VQ-VAE, and empirically validating its effectiveness on both generation and understanding tasks. We appreciate the reviewer’s insight and will incorporate this discussion into the related work section to better contextualize the novelty of our approach.\n\n> **Weaknesses2**\n\n**A2:** We sincerely thank the reviewer for the careful observation. The inconsistency arises primarily form differencies in data preprocessing: our original setting does not normalize the object occupancy within the input image, whereas Trellis resizes the object to occupy a fixed portion. This discrepancy results in an unfair comparison. After aligning with Trellis, our results demonstrate significantly improved visual consistency in the mentioned failure examples. We will include this updated setting and analysis in the final version to ensure a fair and rigorous comparison.\n\n> **Weaknesses3**\n\n**A3:** We conduct additional 3D understanding comparisons with recent and stronger methods, including ShapeLLM and GPT4Point. As shown in the table below, the best and second-best results for each metric are marked in bold and underline, respectively. Our method consistently outperforms these baselines, achieving state-of-the-art performance in 3D understanding tasks.\n\n| **method**       | **BLUE-1↑** | **ROUGE-L↑** | **METEOR↑** | **Sentence-BERT↑** | **SimCSE↑**   |\n|:--------------:|:-----------:|:------------:|:-----------:|:------------------:|:-------------:|\n| ShapeLLM       | 17.88| 19.24        | 17.96| 48.52              | 49.98         |\n| GPT4Point      | 8.45        | 10.11        | 13.13       | 40.31              | 42.88         |\n| PointLLM-13B   | 17.09       | 20.99 | 16.45       | **50.15**          | **50.83**     |\n| ShapeLLM-4o    | **18.51**   | **21.37**    | **19.89**   | 49.34      | 50.72  |\n\n> **Questions1**\n\n**A4:** We thank the reviewer for this valuable suggestion. In our current pipeline, we first edit single-view image using GPT-4o, then perform image to 3D generation using Trellis. We agree that incorporating view-consistent editing and leveraging more large 3D reconstruction models could further improve the editing fidelity and geometric accuracy of the final 3D shapes. We will serve this suggestion as future direction in the conclusion section, and explore it in our future work.\n\n> **Questions2**\n\n**A5:** We have provided a preliminary discussion on the performance gap in Section 4.2, specifically in the third paragraph under the subsection \"Compared with Trellis.\" We believe the performance gap between our model and Trellis can be attributed to two main reasons:\n\n**Reason 1:** Our model integrates six tasks (3D generation from images, 3D generation from text, 3D understanding, 3D editing, image understanding, and text reasoning) into a single model. In contrast, Trellis separates the tasks of 3D generation from images and 3D generation from text into two distinct models, each trained independently. The multi-task integration is inherently more challenging than training a single-task model. To demonstrate this, we conducted a small-scale fine-tuning experiment specifically for text-to-3D generation, using the existing pre-trained weights. During training, we removed extraneous prompt words, retaining only the basic prompt, and froze the gradients of the textual tokens corresponding to embeddings other than the mesh token. The hyperparameters used were: learning rate of 1e-5, context size of 1536, batch size of 4, and gradient accumulation of 2. We continued fine-tuning for 5 epochs. After fine-tuning, the model lost its ability to generate normal text outputs; however, its ability for text-to-3D generation improved significantly, yielding results on the test dataset comparable to those of Trellis. This experiment shows that our model, by sacrificing other capabilities, can achieve further breakthroughs in a single task.\n\n**Reason 2:** We believe a key factor preventing ShapeLLm-4o from surpassing Trellis is the inherent advantage of diffusion/flow models over autoregressive architectures in visual content generation. The diffusion/flow models currently outperform autoregressive models in visual generation tasks. Overcoming this gap, where autoregressive architectures can outperform diffusion/flow models, remains a significant challenge in the academic community.\nHowever, the focus of our work is not to explore the upper bounds of autoregressive architectures for 3D generation but to enable unified generation and understanding in a model. Our goal is for LLMs to successfully leverage autoregressive paradigms to both understand and generate 3D content. This is a significant innovation in the field. Moreover, autoregressive visual generation is a highly active area of research, and the performance of the unified model architecture we propose will only improve as advances in autoregressive visual generation continue.\n\n| **Method**   | **CLIP↑** | **FDincep↓** | **KDincep↓** | \n|:------------:|:-----------:|:------------:|:-----------:|\n| Trellis     |   30.8   |   18.3   |   0.19 |       \n| ShapeLLM-4o|   26.7   |   25.9   |   0.25 |  \n| ShapeLLM-4o（Overfitting）|   30.1   |   18.9   |   0.21 |       \n\n> **Questions3**\n\n**A6:**  The main difference between our work and Trellis is not just swapping the flow transformer with a VLM backbone. There are two key distinctions:\nGenerative Modeling Approach: Trellis uses a flow-based model, while our approach relies on an autoregressive model. These two methods are fundamentally different in how they generate data.\nDiscrete vs. Continuous Representation: Trellis uses continuous latent spaces, whereas our model works with discrete tokens. This affects how the 3D content is represented and generated.\nSo, the architectural difference is much more than just the backbone change — it's about the overall generative modeling approach and representation type."}, {"review_id": "FGgzipuxHF", "reviewer": "Reviewer_Vmok", "summary": "This paper proposes ShapeLLM-4o, a unified 3D-language model. As the first unified multi-task 3D understanding and generation framework, ShapeLLM-4o incorporates multiple 3D-related tasks like text-to-3D, image-to-3D and 3D editing. By introducing a 3D-VQVAE, 3D assets are aligned to the vision and language tokens in a LVLM. To train the unified model, the paper raises a comprehensive 3D-language dataset called 3D-Alpaca. The proposed ShapeLLM-4o achieves competitive results on a wide range of downstream tasks.", "questions": "1. The paper attributes ShapeLLM-4o’s underperformance on 3D generation tasks (compared to Trellis) to its multi-task design, while Trellis uses task-specific models. However, have the authors experimented with single-task fine-tuning of ShapeLLM-4o?\n\n\n2. Why does the paper omit evaluation on 3D visual question answering (VQA), which is a more comprehensive test of the model’s language understanding than captioning? There are previous works that evaluate 3D LVLMs on QA tasks[a], [b], but we did not see it in ShapeLLM-4o.\n\n[a] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction, ECCV24\n\n[b] GPT4Point: A Unified Framework for Point-Language Understanding and Generation, CVPR24", "limitations": "The authors have addressed the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.\tShapeLLM-4o serves as the first unified model that can tackle text-to-3D, image-to-3D, 3D-to-3D (editing) and 3D-to-text tasks in all. By integrating a wide range of practical tasks, it makes a worth noting contribution to 3D understanding and generation.\n\n2.\tThe proposed 3D-Alpaca dataset is a good advancement to the community, in both its data scale and task comprehensiveness. The curation of 3D data is challenging, but the paper smartly utilizes existing expert models to create high-quality data.\n\n3.\tThe paper writing is clear and organized, making the method easy to follow.\n\nWeaknesses\n\n1.\tAlthough supporting multiple tasks, ShapeLLM-4o underperforms Trellis on 3D generation tasks. It is stated in the paper that a possible reason is Trellis trains separate models for different tasks. However, what is the result of ShapeLLM-4o if further fine-tuned on a single task?\n\n2.\tShapeLLM-4o is evaluated on 3D caption tasks, but why not include 3D question-answering? Captioning could not fully reveal the language power of the model. There are previous works that evaluate 3D LVLMs on QA tasks[a], [b], but we did not see it in ShapeLLM-4o.\n\n[a] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction, ECCV24\n\n[b] GPT4Point: A Unified Framework for Point-Language Understanding and Generation, CVPR24\n\n3.\tWhat is the cost of building the 3D editing data in 3D-Alpaca?\n\n4.\tWhat is the number of 3D editing data? In Tab.2 it is 420k, but in the ‘Limitations’ part it is stated as 70k.", "was_revised": true, "final_justification": "My question has been well resolved.", "rebuttal": "We thank the reviewers for the constructive reviews. We provide our feedbacks as follows. \n\n> **Weaknesses1**\n\n**A1:** Based on the pretrained weights, we conducted a small-scale fine-tuning experiment specifically targeting text-to-3D generation. During fine-tuning, we removed redundant prompt tokens. Additionally, we froze the gradients of all text token embeddings except for the mesh tokens. The learning rate was set to 1e-5, with a context length of 1536, batch size of 4, and a gradient accumulation step of 2. The fine-tuning lasted for 5 epochs.\nAfter fine-tuning, the model lost general text processing ability but improved significantly on text-to-3D tasks, achieving results comparable to Trellis on our test dataset. This shows the model can gain on a specific task by sacrificing general capabilities. \nWe think Shapellm-4o did not outperform Trellis because diffusion/flow models currently work better than autoregressive models for visual generation. Beating diffusion/flow with autoregressive models is a major challenge. Our work focuses not on maximizing 3D generation performance but on creating a unified model that can both understand and generate 3D content autoregressively. This is a key innovation. As autoregressive visual generation advances, our unified model’s performance will also improve.\n\n| **Method**   | **CLIP↑** | **FDincep↓** | **KDincep↓** | \n|:------------:|:-----------:|:------------:|:-----------:|\n| Trellis     |   30.8   |   18.3   |   0.19 |       \n| ShapeLLM-4o|   26.7   |   25.9   |   0.25 |  \n| ShapeLLM-4o（Overfitting）|   30.1   |   18.9   |   0.21 |       \n\n> **Weaknesses2**\n\n**A2:** We conduct additional experiments to evaluate our model on 3D question-answering tasks. As shown in the table below, the best scores for each metric in the table are highlighted. Our model achieves consistently superior performance across all methods, demonstrating its strong capabilities in 3D QA tasks.\n\n| **Method**   | **BLUE-1↑** | **ROUGE-L↑** | **METEOR↑** | **Sentence-BERT↑** | **SimCSE↑** |\n|:------------:|:-----------:|:------------:|:-----------:|:-----------------:|:-----------:|\n| ShapeLLM     |   17.73   |   19.91   |  21.86 |       51.32      |     50.95     |\n| GPT4Point    |     6.51     |     8.59      |     5.80     |       29.66      |     32.18     |\n| PointLLM-13B |     17.23    |     19.70     |     20.48    |     **52.66**    |   53.21  |\n| ShapeLLM-4o  |  **19.66**  |   **21.31**   |  **22.68**   |   52.51   |   **53.33**   |\n\n> **Weaknesses3**\n\n**A3:** We used the GPT-Image-1 API for generating and editing the data, selecting the \"low\" quality option. The average cost per image editing pair was around 0.01, with a total expenditure of approximately 750. After constructing the image editing pairs, we used Trellis for 3D reconstruction of the edited images. On average, the time to generate one asset was 5 seconds, and the total GPU time consumed was about 100 hours, which equals approximately 200. Therefore, the total cost for the entire process amounted to around 950.\n\n> **Weaknesses4**\n\n**A4:** We apologize for the confusion. As described in Section 3.4 of the paper, we initially constructed 70K 3D editing text-image pairs. Based on these, we generated a total of 420K multi-turn dialogue samples following specific rules. Specifically, for each 3D editing pair, we created three dialogue types: (1) pure 3D editing, (2) image-to-3D followed by 3D editing, and (3) text-to-3D followed by 3D editing—yielding approximately 210K dialogues. To further enrich the diversity and support multi-task interleaving, we inserted additional pure text instructions into these dialogues, ultimately expanding the dataset to 420K dialogue samples. We will revise the text in the ‘Limitations’ section to clearly reflect that 70K is the number of base 3D editing pairs, while 420K refers to the full dialogue dataset derived from them."}]

Author rebuttal:

Meta-review: This paper presents ShapeLLM-4o, a multimodal LLM for 3D generation and understanding. ShapeLLM-4o supports 3D object generation (text-to-3D and image-to-3D), understanding (3D captioning), and editing within a pre-trained model.  The paper reveals key components including the 3D VQ-VAE and a 3D-Alpaca dataset.

The reviewers acknowledged the novelty and contribution of this paper. After the rebuttal, the final ratings of the four reviewers are all positive: 1 accept and 3 borderline accept.

The AC agrees with the reviewers' ratings.
AC thinks this work advances the development of unified 3D foundation models with sufficient contribution, thus AC suggests accepting this paper with a spotlight.
Decision: Accept (spotlight)