Title: AutoPartGen: Autoregressive 3D Part Generation and Discovery
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D generation
Abstract: We introduce AutoPartGen, a model that generates objects composed of 3D parts in an autoregressive manner.
This model can take as input an image of an object,  2D masks of the object's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction.
Our approach builds upon 3DShape2VecSet, a recent latent 3D representation with powerful geometric expressiveness.
We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks.
Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while conditioning on previously generated parts and additional inputs, such as 2D images, masks, or 3D objects.
This process continues until the model decides that all parts have been generated, thus determining automatically the type and number of parts.
The resulting parts can be seamlessly assembled into coherent objects or scenes without requiring additional optimization.
We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGen, demonstrating that it achieves state-of-the-art performance in 3D part generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe method generates 3D objects by dividing them into smaller parts and synthesizing each part independently. It supports generation from either an image or a 3D object, with optional part-specific masks. The process follows an autoregressive strategy, where each part is conditioned on all previously generated parts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. New approach for generating 3D assets, by autoregressive generating parts of the object. This approach can provide more information about the generated object about it's decomposition.\n2. Three tasks involving part generation. image-to-parts, object-to-parts, masks-to-parts.\n\nWeaknesses:\n1. The method cannot handle objects without clear parts, such as natural elements like seas or rivers, or highly complex objects like trees with many small leaves.\n2. Applying the same resolution to all parts may result in some parts being overly detailed, while others appear low-resolution, leading to inconsistent quality.\n3. The paper does not address texture generation, which limits the realism and completeness of the generated 3D parts.\n\nQUESTIONS:\n1. Generation Order: How is the order of part generation determined? Can we predict which parts are generated first? During training, how do you decide the sequence of parts?\n2. Canonical Positioning: Is the canonical position of each part defined based on the final assembled object, or can the object shape and part positions be learned independently?\n3. Detail Level of Parts: How can we assess the level of detail in generated parts? For example, in the room scene (Fig. 1 middle), the “chair” part is generated as a whole, but generating a chair object as parts typically involves distinct components like handles and wheels.\n4. Necessity of Part-wise Generation: If the model is capable of generating full objects, what is the motivation for generating parts separately?\n\nLIMITATIONS:\n1. The method cannot handle objects without clear parts, such as natural elements like seas or rivers, or highly complex objects like trees with many small leaves.\n2. The paper does not address texture generation, which limits the realism and completeness of the generated 3D parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an auto-regressive method for 3D shape part generation. The approach dynamically decides which part to be generated and the number of generated parts. Results demonstrate it can achieve composable object generation without too much overlap.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Generating part-level objects is essential for many downstream tasks. \n+ The overall autoregressive generation is reasonable and novel.\n+ The generated parts have less overlap and are of better quality.\n\nWeakness:\n- The paper and its motivation are not well organized. For example, I did not see why supporting different modal inputs is challenging in the design, but this is emphasized in the introduction. Moreover, I lost myself when going through technical details. For example, how to generate \\hat{z}? If each step takes a point as input, this should be reflected in the pipeline, and there should be more descriptions about how the points are sampled. Line 162: Should there be an additional \\hat{P} for the SDF function? It is unclear to me how the diffusion process is used. In Fig. 3, the symbol z^(1) is not defined, and there are inconsistent symbol usages for z in the method part. In line 181, is the equation correct? Is there any way to ensure the \\hat{z} and the generated parts are consistent? \n- The evaluation is not solid enough. If you have ground truth parts, why not evaluate the mIoU for all ablations and comparisons? For the present results, it would be hard for readers to judge the advantage of the method. Moreover, is there any specific reason why there is less overlap for the reported results? \n- I did not see specific designs of AR models for part generations, and the shape generation is mainly based on Shape2VecSet.\n\nQUESTIONS:\nBasically, I encourage the authors to put more effort into improving the writing quality and paper organization. I did not think the paper is clear enough for readers to understand the technical details and reproduce the method. \nThe author mentioned the ambiguous problem for part generation. I wonder how the method addresses them.\n\nLIMITATIONS:\nIt would be good to see discussions about failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe method is about generating 3D objects/scenes composed of many parts. The training of the method contains two stages: 1) part autoencoder; 2) conditioned diffusion model. When doing inference, an object is generated part-by-part autoregressively. Extensive experiments are done to prove the effectiveness of the method, including object- and scene-level generation.\n\nSTRENGTHS AND WEAKNESSES:\nThe writing is clear. The core intuition is well-described in the introduction. It is based on a simple observation of the combinatorial property of 3DShape2VecSet (see below). The outer level of the generation is done in an autoregressive way, i.e., part-by-part. The inner level is done using diffusion models in the latent space of parts.\n\nThe property described in Fig 3 and Sec 3.1 is a very interesting observation. This not only helps the part-aware generation as shown in this paper, but also might be helpful in scaling 3DShape2VecSet to a larger latent space while preserving high-quality details. Actually, I am quite surprised about the property. It would be better if there were some math analysis about the property.\n\nThe main experiments are about object-level generation, including different conditions (images and objects). The experiment setup makes sense and the results are strong. There are also applications on scene-level generation.\n\nQUESTIONS:\nI still do not understand how the parts are ordered. Usually, we need an order for the elements used in autoregressive models, like the natural language order or rasterization order for images. However, from the context, I do not get what the order is here.\n\nThe inference includes a (autoregressive) set of diffusion generation processes. I would expect it would be very slow. It would be better if there were a time analysis of the inference cost.\n\nLIMITATIONS:\nThe method takes really lots of GPUs (128 for the autoencoder and 256 for the diffusion models) to train. This could be a major limitation for most universities (which do not have access to large computing clusters). However, this is not a real drawback. On the contrary, it shows the scaling ability.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a new part-based 3D generator, named AutoPartGen. AutoPartGen leverages the locality property of the vecset-based 3D VAE latent space and trains a flow model to autoregressively generate part latents. The authors conduct several experiments to validate the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The observation that locality, or at least compositionality, is well supported by the vecset representation is insightful. Interestingly, the authors demonstrate that directly concatenating two different latents can be decoded together.

2. The proposed autoregressive generation process is reasonable. Additionally, AutoPartGen re-encodes all previously generated meshes to obtain more compact conditional latents, enhancing efficiency.

3. The presentation is clear and easy to follow.

Weaknesses:

1. AutoPartGen suffers from part order ambiguity. The authors do not clarify how the model defines the autoregressive order. From my perspective, the conditionally masked image $J$ could provide some information on which part to generate. However, the masked image may contain limited information when the part size is small.

2. More ablation experiments are expected, particularly regarding the re-encoding strategy to inject previous information.

3. It is unclear how AutoPartGen generates texture, as the vecset VAE does not encode appearance information.

QUESTIONS:
See the weaknesses, especially the part-order ambiguity.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an autoregressive model for generating compositional 3D objects part by part. To achieve this, the authors first parameterize 3D objects into a latent space using the existing 3DShape2VecSet representation. They then train a latent diffusion model to progressively predict the latent code of each 3D part, conditioned on input evidence and the latents of previously generated parts. While the experimental results demonstrate plausible generation outcomes, the proposed method does not present significant technical innovation. Additionally, the overall writing quality of the paper could be improved.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe method achieves impressive results in generating compositional 3D objects.\n\nWeaknesses:\nThe technical novelty of the paper is limited, as the approach essentially fine-tunes a latent diffusion model. Besides, many technical details are unclear, and the paper lacks necessary ablation studies and relevant citations.\n\nQUESTIONS:\nHere are my specific questions and suggestions:\n\n1. In the Introduction, it would be helpful to provide a visual example to illustrate the statement that \"the concatenation of two codes can be decoded into the union of the corresponding surfaces.\" Moreover, although the importance of concatenation is emphasized, the actual implementation does not use concatenation but instead re-encodes the known parts. What is the purpose of discussing concatenation in the Introduction if it is not used in practice?\n\n2. The paper claims that the method can automatically generate 3D objects part by part, but does not describe a termination strategy. How does the model determine when the scene has been fully generated?\n\n3. In Sec 3.1, the function ‘sample_N x’ would be clearer if written as ‘sample(N, x)’ to indicate its inputs. Additionally, the point set P is sampled from object x, and then ~P is sampled from P. Why not sample M points directly from x? What is the rationale behind this two-step sampling process?\n\n4. What is the architecture of the decoder D? How is it trained? What is the underlying principle that enables it to achieve the results shown in Fig. 3? Please provide a detailed discussion and explanation.\n\n5. In Sec 3.3, the process involves decoding, sampling, merging, and re-encoding latent codes. How does this approach differ from simply concatenating the latents? It seems that encoding a small 3D part and encoding a complex scene with the same encoder may yield different results. Please provide comparative experiments to demonstrate the differences between these two strategies.\n\n6. How are different parts defined in the training data? How do you ensure that each part has a consistent semantic meaning?\n\n7. The writing, especially in the Methods section, is somewhat verbose and difficult to follow. I recommend revising and refining this section to more clearly define key concepts and articulate the core methodology.\n\n8. Missing citations or comparisons:\n- MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation\n- Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly\n- CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image\n- Part123: Part-aware 3D Reconstruction from a Single-view Image\n\n9. Minor Issues and Typos:\n- The function ‘sample_N’ is defined multiple times in Sec 3.3.\n- The pipeline shown in Fig 2 is not referenced or explained in the main text.\n\nLIMITATIONS:\nNo, in the conclusion, the authors need to provide a necessary discussion of the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}