Title: ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

MLLM

3D generation

gpt-4o

Editing
Abstract: Recently, the powerful text-to-image capabilities of GPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni—a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, we perform instruction-based fine-tuning of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset, equipping it with native 3D understanding and generation capabilities. Our work represents an effective step toward extending multimodal large language models with fundamental 3D intelligence, paving the way for future advances in 3D-native AI.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ShapeLLM-4o, a unified 3D-language model. As the first unified multi-task 3D understanding and generation framework, ShapeLLM-4o incorporates multiple 3D-related tasks like text-to-3D, image-to-3D and 3D editing. By introducing a 3D-VQVAE, 3D assets are aligned to the vision and language tokens in a LVLM. To train the unified model, the paper raises a comprehensive 3D-language dataset called 3D-Alpaca. The proposed ShapeLLM-4o achieves competitive results on a wide range of downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tShapeLLM-4o serves as the first unified model that can tackle text-to-3D, image-to-3D, 3D-to-3D (editing) and 3D-to-text tasks in all. By integrating a wide range of practical tasks, it makes a worth noting contribution to 3D understanding and generation.\n\n2.\tThe proposed 3D-Alpaca dataset is a good advancement to the community, in both its data scale and task comprehensiveness. The curation of 3D data is challenging, but the paper smartly utilizes existing expert models to create high-quality data.\n\n3.\tThe paper writing is clear and organized, making the method easy to follow.\n\nWeaknesses\n\n1.\tAlthough supporting multiple tasks, ShapeLLM-4o underperforms Trellis on 3D generation tasks. It is stated in the paper that a possible reason is Trellis trains separate models for different tasks. However, what is the result of ShapeLLM-4o if further fine-tuned on a single task?\n\n2.\tShapeLLM-4o is evaluated on 3D caption tasks, but why not include 3D question-answering? Captioning could not fully reveal the language power of the model. There are previous works that evaluate 3D LVLMs on QA tasks[a], [b], but we did not see it in ShapeLLM-4o.\n\n[a] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction, ECCV24\n\n[b] GPT4Point: A Unified Framework for Point-Language Understanding and Generation, CVPR24\n\n3.\tWhat is the cost of building the 3D editing data in 3D-Alpaca?\n\n4.\tWhat is the number of 3D editing data? In Tab.2 it is 420k, but in the ‘Limitations’ part it is stated as 70k.\n\nQUESTIONS:\n1. The paper attributes ShapeLLM-4o’s underperformance on 3D generation tasks (compared to Trellis) to its multi-task design, while Trellis uses task-specific models. However, have the authors experimented with single-task fine-tuning of ShapeLLM-4o?\n\n\n2. Why does the paper omit evaluation on 3D visual question answering (VQA), which is a more comprehensive test of the model’s language understanding than captioning? There are previous works that evaluate 3D LVLMs on QA tasks[a], [b], but we did not see it in ShapeLLM-4o.\n\n[a] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction, ECCV24\n\n[b] GPT4Point: A Unified Framework for Point-Language Understanding and Generation, CVPR24\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the modality limitation of current unified multimodal large language models (MLLMs), such as GPT-4o, which primarily support only image and text. To overcome this, the authors propose ShapeLLM-4o, a native autoregressive MLLM with the capability to understand and generate 3D objects.\n\nThe model leverages a pretrained VQ-VAE for 3D object representations, using the encoder for object understanding and the decoder for generation. To enable 3D reasoning capabilities, the authors construct a large-scale dataset covering tasks such as understanding, generation, and editing of 3D content. The training is based on the Qwen-2.5-VL-7B-Instruct backbone, pushing the frontier of multimodal LLMs toward the 3D modality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper presents the first autoregressive framework that unifies 3D object understanding and generation, demonstrating the feasibility of extending LLMs to the 3D domain.\n- It contributes a large-scale dataset tailored to training unified 3D-capable MLLMs.\n- The study offers an exploratory step toward expanding the modality range of general-purpose MLLMs.\n\n**Weaknesses**\n- The idea of using a shared tokenizer for understanding and generation has already been validated extensively in the vision domain (e.g., EMU3), and is not novel in principle.\n- In 3D generation visualizations, the results show noticeable inconsistencies compared to baselines like Trellis (e.g., in the third row, the number of buttons on the TV is incorrect).\n- In the 3D understanding evaluation, comparisons are made only with PointLLM, which is relatively outdated. More recent baselines should be included for a fair assessment.\n\nQUESTIONS:\n- In constructing the 3D editing dataset, it is stated that the authors use image editing followed by 3D reconstruction. Does this mean the model is prompted with an input image and generates edited images from multiple viewpoints?\n  - How is consistency across multiple views ensured during this image editing process?\n  - What specific 3D reconstruction method is used to obtain the final 3D shape?\n  - A detailed visualization of this pipeline (from image edit to 3D generation) would be very helpful.\n- The paper adopts Trellis’s 3D tokenizer while using a more powerful Transformer backbone (Qwen-2.5-VL), yet the 3D generation quality is significantly worse. However, the paper does not analyze this performance gap in detail.\n- Is the main architectural difference between this work and Trellis simply replacing the flow transformer with a VLM backbone?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new mllm called ShapeLLM-40. It mainly focuses on 3D generation and understanding. Compared with some existing models like GPT-4o, this model can handle 3D data together with text and image, so it can solve some limitations before. ShapeLLM-40 uses a 3D VQVAE to change 3D objects into discrete tokens. In this way, the model can process text, image and 3D data together in one framework. For training, the authors build a new dataset named 3D-Alpaca, which includes tasks like text-to-3D, image-to-3D, 3D caption and 3D editing. From the experiments, ShapeLLM-40 gets better results than baselines such as Trellis, SAR3D and 3DTopia-XL. It shows better text matching and generation quality, and also keeps good language and conversation ability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Provide a good baseline to bridge both understanding and generation in 3D domain.\n2. Contribution an valuable dataset to the community for future works.\n3. The motivation to use 3D VQVAE for 3D modality encoding and decoding is clear and sound.\n\nWeaknesses:\n\n1. The 3D capability only focus on object-level, which limits its application in real life or contribution to 3D reasoning community. Currently, 3D scene level understanding is more realistic and challenging. Please consider to evaluate on 3D scene level benchmark for 3D captioning, QA, and Grounding. Related works include Robin3D, Chat-Scene, LL3DA, etc.\n2. When applied to 3D scene-level reasoning, this 3D VQVAE would inevitable increase computational cost. As shown in Robin3D and Chat-Scene, a scene can be represented by 100 to 150 objects which means will scale up 100 to 150 times of the current number of vision tokens. I don’t think this is a practical approach.\n3. The approach for generating 3D Generation and Understanding Dataset, which is actually generating paired-language data, is similar with Grounded 3D-LLM. Therefore, even though I value the engineering effort, there is indeed not much insight or novelty to the research field.\n4. Most data are generated from GPT. There’s no guarantee on its quality since GPT will hallucinate. How to assess its quality would be a problem since data is the main contribution of this paper.\n5. Given the huge training cost, especially 48 NVIDIA H100, and with a more advanced backbone Qwen-2.5-vl, the sub-optimal performance in Tab.4 shows that the proposed method is not very effective or efficient.\n\nQUESTIONS:\nPlease answer my concerns in the Weaknesses.\n\nLIMITATIONS:\nThere's no obvious negative societal impact but for the limitations please refer to the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrates an effective approach to integrate 3D generation (text-to-3D and image-to-3D), understanding (3D captioning) and editing abilities into a pre-trained multimodal large language model (MLLM). In essence, they build a 3D foundation model. Furthermore, they introduce a large-scale 3D dataset, 3D-Alpaca, consisting of 2.56M data samples divided into 4 subsets, 1 for each of the following tasks: text-to-3D generation, image-to-3D, 3D captioning and 3D editing. Their approach involves first training a 3D VQVAE to encode voxel-based representations into compact discretized tokens which can be decoded back to voxels. They take Qwen2.5-VL-Instruct-7B as their backbone MLLM which is trained on their 3D-Alpaca dataset to extend its generative and understanding capabilities to 3D assets. The discretized token 3D representation facilitated by the 3D VQVAE enables them to feed these tokens as input to the model (for captioning and/or editing) and also to generate these tokens in an autoregressive next-token prediction paradigm, effectively treating them in a similar manner to discrete textual tokens. They demonstrate the efficacy of their model on all 4 tasks with extensive experiments to compare against previous baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper is very well-written and was easy to understand.\n2. This work addresses two important problems. There is a lack of 3D foundation models that can both understand and generate 3D assets and a shortage of adequate large-scale 3D datasets.\n3. Use an additional text-based dataset, UltraChat, to ensure the model retains its original reasoning and dialogue capabilities.\n4. Included many visuals to showcase the model's quality.\n5. Simple but effective dataset curation approach.\n6. Simply using discretized 3D tokens as input and output enables the seamless integration of 3D abilities into the model without significant changes to the core architecture. This further enhances the scalability of this approach as it can be applied to most MLLMs and is not strictly restricted to a specific model type.\n\n**Weaknesses:**\n1. It would have been to nice to see some results to evaluate the quality of their 3D VQ-VAE considering that it is a vital part of the methodology. Specifically, want to get an idea of how lossy the compression/encoding is. This can be checked by feeding some 3D assets from an unseen dataset (or even an unseen part of their own dataset but would be better if it is from a different dataset) to the VQ-VAE, getting the token representation, decoding it, and comparing the decoded asset with the original.\n2. The comparisons against other models could have been a little more comprehensive. While it is not possible to compare against all relevant models, it would still have been better to cover some of the more popular models that have proven to be effective. For instance, for the 3D generative task, models such as OpenLRM, LGM, InstantMesh and Unique3D could have been included.\n3. Not much discussion on the limitations of the work and the cons of some of the choices such as the downsides of using a voxel-based representation.\n\nQUESTIONS:\n1. Is the primary reason for using a voxel-based representation the ease of encoding given that voxels are inherently discrete? However, this comes at a cost such as loss in finer details which is why quite a few approaches have been explored to encode and/or compress continuous representations for any downstream task. Did you look into any of these representations and approaches? In essence, I just want to better understand the thinking that went behind the choice of using voxels.\n2. It would help if some of the weaknesses mentioned above could be addressed.\n\nLIMITATIONS:\nNot sufficiently discussed.\n1. Voxels have a limited resolution which means that they result in the loss of information and are not adequate to model finer details.\n2. The data creation pipeline is dependent on several external models which means that the quality of the dataset is closely tied to the quality of the models used. While this approach to use existing SOTA models for synthetic dataset creation and/or labeling is becoming more common, specially given the rapid advancements made in some of these models, it is still important to acknowledge the inherent limitation on the quality of the dataset.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}