AutoPartGen: Autoregressive 3D Part Generation and Discovery
Authors: Minghao Chen
Jianyuan Wang
Roman Shapovalov
Tom Monnier
Hyunyoung Jung
Dilin Wang
Rakesh Ranjan
Iro Laina
Andrea Vedaldi
Venue: NeurIPS 2025 poster ()
Keywords: 3D generation
Primary area: deep_learning

Abstract: We introduce AutoPartGen, a model that generates objects composed of 3D parts in an autoregressive manner.
This model can take as input an image of an object,  2D masks of the object's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction.
Our approach builds upon 3DShape2VecSet, a recent latent 3D representation with powerful geometric expressiveness.
We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks.
Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while conditioning on previously generated parts and additional inputs, such as 2D images, masks, or 3D objects.
This process continues until the model decides that all parts have been generated, thus determining automatically the type and number of parts.
The resulting parts can be seamlessly assembled into coherent objects or scenes without requiring additional optimization.
We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGen, demonstrating that it achieves state-of-the-art performance in 3D part generation.

AutoPartGen:
Autoregressive 3D Part Generation and Discovery
Minghao Chen 1.2 Jianyuan Wang 1.2
Roman Shapovalov?
Tom Monnier
Hyunyoung Jung
Dilin Wang² Rakesh Ranjan²
Iro Laina'
Andrea Vedaldi 1,2
Visual Geometry Group, University of Oxford
M
silent-chen, wit.gg.tthooooooo Aotoutrarteennn
Input Image
Object
Parts
Input Image
Small Scene
Objects
with solar panels"
"the surface of Mars with red rocky terrain"
: a futuristic habitat dome with solar panels"
Input prompts
City
Buildings
Figure AutoPartGen can be applied, by itself or combination with other models, to the generation of compositional 3D objects, scenes and cities starting from 3D models, images text.
Abstract
We introduce AutoPartGen, a model that generates objects composed of 3D parts in autoregressive manner. This model can take input an image of object, 2D masks of the object's 's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction. Our approach builds upon PDShape2VeccSee recent latent 3D representation with powerful geometric expressiveness. We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks. Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while condi- tioning on previously generated parts and additional inputs, such as 2D images, masks, 3D objects. This process continues until the model decides that all parts have been generated, thus determining automatically the type and number parts. The resulting parts can be seamlessly assembled into coherent objects scenes without requiring additional optimization. We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGenn demonstrating that it achieves sfate-of oftte--f-th-ann performance in 3D part generation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Processing 3D objects, including generating them based on a textual description image, is an important aspect of Spatial Intelligence. Current 3D generators often treat objects or even entire scenes monolithic shells. However, many applications require modeling their compositional structure, decomposing them into well-defined 3D parts to enable reasoning or manipulation at a finer granularity, such applying textures and materials each part separately. More specifically, character in a video game should be decomposable into different parts to support animation or allow the game software swap clothes accessories. Windows and doors in the 3D model of a house need to be separate entities to allow user interaction, such as opening or closing them. Similarly, the design of machine must consist of distinct parts to functional (c.g.. the cogs in clock) or to enable 3D printing or other kinds of CNC manufacturing.
In this paper, we address the problem of generating 3D objects with a compositional structure. We introduce AutoPartGen, a new autoregressive model that can directly generate a 3D object part by part, building As shown in Fig. 1, AutoPartGenn can be applied, either independently or in combination with other models, to generate compositional 3D objects, scenes, or even cities, starting from 3D models, images, or text prompts. AutoPartGen solves three key problems to enable such applications: (i) object-to-parte where it decomposes an existing 3D object into meaningful parts; (ii) image-to-parts. where the model generates 3D parts from an unstructured input image; and (iii) masks-to-part where users can provide 2D part masks guide the generation. In the first two scenarios, AutoPartGen automatically predicts semantically meaningful 3D parts without requiring part annotations. In the third scenario, user-provice user-provided masks offer fine- grainea control over the model partitioning.
powerful latent 3D representation. AutoPartGen is robust, flexible, and scalable.
Our autoregressive approach has two key benefits. First, it models the joint distribution over the object parts, ensuring that they fit together cohesively. Second, it enables the model to generate variable number of parts, which is crucial since the number of parts is not fixed or known a priori.
We build AutoPartGen on recent advances in latent 3D representations and parameterize the 3D) surface æ C R of the object using a latent vector = E R& where M is the latent length and d is the latent dimension. We use the 3DShape2 representation [65, 31, 66] and observe, for the first time, that this representation is inherently compositional. Specifically, we show that the concatenation z = (1)) @ (2)) E 2d of two codes 211) and ((2) decodes into the union = (1) U (2) of the corresponding surfaces (1) and (²))
Based on this insight, we propose generating a sequence of latent codes z!.),..., (K), each decoding into corresponding 3D part (*)), Crucially, the generation of each part is conditioned on an overall latent representation of the target 3D object as well as on all previously generated parts x(1) (k)), This conditioning improves the consistency of the generated parts, meaning that they better fit each other compared to the output of the methods that extract parts independently [5, 60].
As noted in [5], object decomposition is an ambiguous problem. For example, a chair might be decomposed into few high-level parts (e.g., seat, back, legs) or more granular set of components (c.g., individual leg segments, cushion, backrest slats). This choice typically depends on the application or the preferences of the 3D artist creating the asset. We address this ambiguity by making the 3D autoregressive model stochastic, using denoising diffusion to generate the next part vector ((k based on the previously generated parts (11) ..... 22..,22x0 (k-1) and the available evidence (i.c., the full 3D object, an image of the object, or 2D part masks, depending on the application). Importantly, we train a single diffusion model capable of handling all three cases.
We evaluate AutoPartGen against state-of-the art part- 3D generators. Compared to the recent PartGen [5], AutoPartGen is easier to implement and maintain (as it does not require training several multi-view image generators) and more accurate. Compared to HoloPart [60], a method that completes a pre-segmented outer surface of mesh to form 3D parts, AutoPartGen is more accurate and significantly more capable, as can automatically discover parts and reconstruct them from either a 2D image or a "shell" 3D object, optionally guided by 2D masks, not requiring any 3D annotation.

---

2
Related Work
Generating 3D object from a single image, or even just text, faces an obvious challenge: the 3D object contains significantly more information than the image or the text. This is similar to the problem of generating images or videos from text, and it is solved by learning prior, or conditional distribution, from billions of data samples. However, data of this size is unavailable for 3D objects. Authors address this problem by involving 2D image or video generators in the 3D generation process. We distinguish two main camps: multi-view direct and single-view latent 3D generation.
Multi-view 3D Generation. In multi-view 3D generation, one asks the image generator to do most of the lifting, generating several views of the 3D objects, and thus simplifying extracting a 3D object from them. First, this was done using Score Distillation Sampling (SDS) [42], an idea explored extensively in follow-ups like GET3D [13], ProlificDreamer [54]. DreamGaussian [51], Lucid Dreamer [9] which seek to achieve multi-view consistency via iterative (and slow) optimization of radiance field (NeRF [44] or 3DGS [24]). A significant innovation, pioneered by UpFusion [23],
3DiM [55], Zero-I-to-3 [36] and MVDream [45], was to fine-tune the image generator to directly produce multiple consistent views of the object. By making the image generator more 3D aware, 3D reconstruction becomes simpler, as noted in InstantMesh [57], GRM [58], and others [56, 50].
Single-view Latent 3D Generation. The alternative approach is to start from a single image of the object and directly reconstruct the 3D object from it. Because single-view reconstruction is extremely ambiguous, this requires to learn a reconstruction function. This was the path taken, for example, by LRM [19] and others [17, 48]. However, their deterministic reconstruction model cannot cope well with this ambiguity and often produces blurry outputs. Much better results were recently obtained by switching stochastic 3D reconstruction based on latent diffusion. Some of the best single-image 3D reconstructors are based on the 3DShape2 VeccSt [65] latent representation (also similar to Michelangelo's ' [67]). Building
it, CLAY [66], DreamCraft3D [47]. CraftsMan [30], TripoSG [31]. and others [10, 59, 63] are able to generate highly detailed and accurate 3D shapes. We build this representation well and show that it also supports compositionality very effectively.
Composable 3D Generation. Approaches to composable 3D generation typically start by decompos- ing objects into constituent parts. One common strategy represents objects as mixtures of primitives, often without semantic labels. For instance, SIF [16] models object occupancy using mixtures of 3D Gaussians. LDIF [15] represents shapes as a set of local deep implicit functions (DIFs), spatially arranged and blended using template of Gaussian primitives. Methods such as Neural Template [21] and SPAGHETTI [1] achieve decompositions through auto-decoding. SALAD [29] utilizes SPAGHETTI for diffusion-based generation. PartNeRF [52] expands this concept by employ- ing mixtures of NeRFs. NeuForm [32] and DifFacto [39] specifically target part-level controllability. DBW [38] uses textured superquadrics decompose scenes. In contrast, another research direction emphasizes explicitly semantic parts. PartSLIP [35] and PartSLIP++ [71] segment objects into seman- tic components from point clouds using vision- language models. 1art123 [34] adapts techniques from scene-level approaches like Contrastive Lift [2] to reconstruct object parts. PartSDF [49] learns latent codes for parts using an auto-decoder and then uses SALAD for part prediction. Comboverse [8] leverages single-view inpainting model and 3D generator for composable 3D generation with spatial- aware SDS optimization. Deep Prior Assembly [69] reconstructs 3D scenes from a single image in a zero-shot manner by assembling large models. MIDI [20] extend pre-trained image-to-3D genera- tor to multi-instance generator through costly global attention. CAST [62] reconstructs physically consistent 3D scenes from a single RGB image using occlusion -ware diffusion and GPT GPT-guided g-gdddddd physics correction. HoloPart [60], a recent work, starts from the shell of a 3D object and a level segmentation for it and performs 3D amodal part completion.
The work most related to ours PartGen [5]. This squarely sits on the 'multi-view direct' camp (see above). It uses multi-view diffusion models for segmentation and completion of compositional 3D objects from diverse modalities.
3D Segmentation. 3D parts can be obtained by segmenting a 3D object (although the resulting parts will generally be incomplete). Some approaches for semantic 3D segmentation such as [68, 28, 53, 25, 2] used neural fields to 'fuse' 2D semantic features in 3D. Contrastive Lift [2] introduced slow-fast contrastive clustering scheme for 3D instance segmentation. Recent methods such as [26, 64, 43,3) 3] integrate SAM [27] and often CLIP to model multi-scale concepts, where Spng explicitly encodes scale information and N2F2 learns to bind concepts to scales automatically. Neural
3

---

Mosked
Ooject
(
Whole
Object
J:8]
Object
DINO
Encoder
DINO
Encoder
DINO
Encoder
2(1)
x(2)
Self Attn
Cross Attn
Self- Attn
Cross- Cross-Atth Atth
Cross Attn
Cross Attn
Cross Attss-Attn
FFN
FFN
[EOT]
Cross
XL
X
XL
Decoder
Encoder
Encoder
[SOT]
Figure 2: AutoPartGen generates parts autoregressively. At each step, a 3D latent diffusion model generate the next part, conditioned on the previously generated parts 2(I....k) the overall object ž, and, optionally, an image / of the object and an image ((k) of the part. The latent representation uses 3DShape2 Vec and the diffusion model is DIT.
Part Priors [4] used learned priors for test-time decomposition. Additionally, efforts to develop 3D 'foundation' models [70, 7] are enabling zero-shot point cloud segmentation across diverse domains.
Method
Let x C. R3 be a 3D object given by a surface embedded in RR. We assume that the object is compositional, meaning that it can be expressed as the union U (((k) of K disjoint parts (11)) (K) each of which is also surface. Concretely, is usually 3D mesh created by artist, and (*) are the components of the mesh that the artist has manually defined when creating the mesh. These parts are thus defined to facilitate editing the 3D object or for functional purposes, such animation. Generally, the same 3D object can have different and equally valid part decompositions.
Our aim is to learn to generate 3D objects æ and their part decompositions (1)), ,(!),....o (K) We consider three different scenarios. the first scenario (object-to-parts) we are given the 3D object æ, and the goal is to sample possible decomposition of this object into parts. Furthermore, we may potentially have an object which is incomplete with respect to constituent parts - case that may arise if æ is acquired by 3D scanner that cannot look inside the object or synthesized by generator that is not aware of the internal structure of the object, exemplified by [60]. In this case, therefore, the goal is to also complete the parts, thus recovering the complete object æ as byproduct.
In the second scenario (image-to-partee topprtss,,, the starting point is image / Ω RR3 of the object, where Ω C R2a is the (finite) image domain. Inferring the object i from a single image is also known as the image-to-3D problem. Here, the task is to also infer its part decomposition x (,))...,,(()))
In the third scenario (kk) 32 + {0, 1} that indicate the pixels in the image / that belong to each part (*)) These masks be defined manually or, more likely, automatically, utilizing 2D segmentation model. The problem is the same as before, but the 3D parts must match the prescribed masks.
-artts, we are given, in addition to the image I, K 2D part masks
In all these cases, recovering the parts (or the object) is ambignous. These problems are thus stochastic and are solved by learning suitable conditional probability distribu- tions: (1x( ,,...,,.." (K) æ) (object-to-partts, p(x!!),.... (K)) I) (image-to-parts), and (1))(1) (kk T, M1) ,..,, ,..., M(K) tmasks.to-parts). We develop a single model that can handle all three cases.

---

3.1 Latent 3D shape space
Directly defining, modeling, and learning a distribution on 3D surfaces is difficult. We thus introduce latent space, providing finiteddimensionl parametrization of the surfaces.
We utilize the VecSer representation developed by [65]] This representation is based on learn- ing an encoder-decoder pair (F, D). The encoder E takes collection of N object points P ( PNN = sampley æ and maps them to latent vector & = E(P). Here sample v is a function that samples N random points from the surface of the object
that P C æ. The decoder takes the latent vector z and a query 3D point p E I and evaluates the signed distance function (SDF) p SDF (p) æ²) 2). The encoder-decoder pair is thus 'translational' in the sense that it translates one type of representation of the object (the point cloud P) into another (the signed distance function SDF( (z)).
In more detail, the encoder E compresses the point cloud P into sequence (21, ................................ 2ar) of M tokens E R The M < N tokens are obtained by first subsampling the point cloud P into a much smaller set of points P = {pi..... PM] } = sample P C P and then by applying transformer neural network to the points in P to output z. The transformer also attends to the large number of points P efficiently via cross-attention
The network is designed to be permutation equivariant, meaning that the order of the points and tokens is immaterial, explaining the moniker 'VecSet'. The decoder takes a point p E R and the tokens & and outputs the value of the signed distance function SDF(ppax) also utilizing transformer neural network in the form of Perceiver [22].
The intuition behind this representation is that each token vector encodes local region of the surface centered at the point pi. However, the transformer allows tokens to communicate globally, which makes this interpretation somewhat loose. Empirically, we have discovered that locality, ality, is well supported by the representation. As we show in Fig. 3, the tokens can be concatenated to form new latent vector codes into new surface æ (1) (2) that is a good approximation of the union of the two parts, without any retraining.
20) @i 80
Decoder
Decoder
becoder
least composition-
60033
=(1) @ z(2) that de-
Figure 3:
Compositionality of the VecSet space. Concatenation of two latents will re- sult in a spatial combined mesh.
3.2 Latent 3D diffusion
Having established the latent representation for the shapes, the next task is to learn a model that can sample shape given some evidence y. from a conditional probability distribution p(= y) (for example, y could be the image I of the object). This utilizes (latent) diffusion. In brief, we define sequence of progressively more noised versions of the latent vector z as 2, = /atz + /1 - ate, where _O, Z) is Gaussian noise vector and a,, t 0,1..... . T is schedule of noise levels. Following [46, 14], we introduce the flow velocity vtt, 21, <) = (z, /aze)/ / /1 aq The diffusion model è ét, æ1, is trained to predict the flow velocity étt, 21 y) given only the latent vector and the condition y, minimizing the loss L()) = E( (t, x, y) vtt, æ,, e) averaged over a training set of evidence-latent vector pairs (y, 2), a random time step and noise e.
3.3 Autoregressive 3D part generation
The model described in Section 3.1 generates the entire 3D object x (or, more precisely, its latent representation 2) as a whole. Here, we consider the problem generating the object parts instead. Our goal is to learn a single model that can handle all three part generation scenarios: object-to-part,
For this to work, few technical assumptions are required. We assume æ to be the finite disjoint union of closed regular surfaces smoothly embedded in R without self-intersections. This makes the surfaces orientable; then, the surfaces split R² into disjoint regions alternating between the outside and inside of the object. In this way. the signed distance function is well defined. The parts (4) are defined in the same way, and in fact, they are formed by the union of one more of the closed surfaces that comprise so that signed distance function is defined for each part too.
5

---

Table 1: 3D part completion. Reconstruction quality of the parts and whole object. "reproduced.
Parts
Whole Object
Method 3D Mask
IoU 1
F-Scoret
CD2
IoU 1
F-Scoret
CDJ
HoloPart' [60]
/
0.658
0.836
0.065
0.821
0.945
0.018
PartGen [5]
x
0,614
0.812
0.121
0.779
0,921
0.033
AutoPartGen
X
0.665
0.861
0.047
0.832
0.967
0.012
image-to-part and masks-to-part, depending on the inputs provided. An overview of the pipeline is shown in Figure 2.
To generate an undetermined number of parts K, we consider an autoregressive approach, where a single part (8) is generated each time, based on what was generated before. The model can thus be described as a conditional distribution p(2(k) y, (1) x ((==1)) where x*)) the latent representation of the k-thh part and the input y collects the additional evidence available to the model.
The nature of this evidence depends on the reconstruction scenario. In the object-to-part scenario, is simply the 3D object æ. In the image-to- po-gett scenario, is the image I of the object. In the masks-to-part scenario, y/ is the image as well as the masked image J(k) = (A) C I, denoting which parts should be generated next.
Knowledge of the previously generated parts (())...,, is essential as this allows the model to ensure that the next part fits together well with the previously generated ones. Furthermore, in all cases we consider, the evidence y also provides some evidence on the overall shape of the object. As suggested in Fig. 3, we can represent the union of parts by simply concatenating their latent representations. However, for compactness, we found it useful to fuse their codes into one, which we obtain as: z (111 = E (U/=l sampley D( | 20)).. where sampley is function that samples N points from the surface of the object defined by the zero level set of the SDF function ((-1 we call this strategy re-encoding.
We found optional but useful pin down the overall object by adding to the evidence y a code for the object as a whole, which is either given outright (object-to-part, ž = E(sample ② or can be obtained by the model itself (image-to- parr and masks-to-pane p(z T)) by directly providing an unmasked image to our mode.
With all this, we learn conditional generator model
p( (*) x 21.....k-11), v),
where = & for the object-to-part scenario. = I for the image-to-part scenario, and y = (T, (+) for the masks-to-part scenario. The generation process stops when all the input masks have been processed, if available, or when the model outputs a predefined special [EoT] token, representing empty shape. In practice, we represent the [EoT] token using latents whose values are all zeros.
Based on Section 3.2, learning the distribution Eq. (1) amounts to learning velocity field e(t, #, 2, z =111...,k-11, y). During inference, we use classifier-free guidance (CFG) [18] to modulate the strength of the conditioning. In the most general case, the model is conditioned by the overall (partial) object ă, the previously generated parts 2 z(1,....-1)) and masked image pair y = (I, ((0). We modulate the importance of the geometric and visual conditioning as follows:
(t, Zt ž, (1.....111,, .) Wing (b(t, 2, 2, (P.....11)) (K0) é(t, ze 1, 2 b)))
(0(t, #1 1 (1 útt, x,, 0) + a(t, 21, 0),
+ Wigeomm
(2)
where Wing and Wlgcom modulate, respectively, image and geometry conditioning. The different inputs are implemented by first encoding into tokens, which are then cross-attended by a transformer neural network computing the flow velocity. Hence, suppress input simply replace it with dummy tokens. In the same way, we randomly drop some input at training time to allow the model to learn to use any required subset of the inputs.
Discussion Here, we contrast model to prior works and justify its design. The most straight- forward approach to part generation is sample each part (k) independently from 'marginal'

---

Image
Object
Parts
Image
Object
Parts
scenario. Given input image, AutoPartGen recovers compositional 3D object made up of several meaningful and complete parts.
Figure 4:
distribution (t)) However, this model lacks mechanism to tie the parts together and would result in a soup of random, uncoordinated parts. The simplest such mechanism is to provide evidence y for the overall shape of the 3D object. For instance, in the image-to-3D case, y = T could be 2D image of the object, and we may sample parts from the conditional distribution p(x(() I). While constrains the shape and position of the possible parts, these are still quite ambiguous. This explains why PartGen [5] conditions part generation
multi-view image y Iuv of the 3D object æ, and HoloPart [60] starts from a (partial) 3D reconstruction y = a of the object itself.
Even then, the reconstruction context y is likely insufficient because there is no indication of which part should be reconstructed next. We could sample the parts random order, but this would not be very efficient. Furthermore, because the part decomposition is not unique, we would still need to extract a coherent subset of parts from the 'part soup' so obtained.
Prior works address this issue by explicitly telling the 3D reconstruction model which part to extract next. PartGen does so by providing multi-view image JMv of the part, and HoloPart by providing 3D mask MaD of the part, defining distributions p(a=(k) /MV, Juv) and p(æ{()) ât, MaD). respectively. Hence, the problem of generating a coherent collection of parts offloaded to a mechanism external to the 3D reconstructor. On the contrary, our 3D generator/reconstruuto makes this determination by itself, operating autoregressively, one part at a time, without additional models.
4 Experiments
We first give the implementation details AutoPartGenn, including network architectures, training procedures, and datasets. We then demonstrate its performance under various conditions, highlighting its versatility for different applications. Next, we compare our approach with state-of-the-art 3D completion methods and provide ablation studies to analyze key design choices. Finally, we showcase several applications of AutoPartGen.
4.1 Implementation Details
Architecture. Our architecture builds upon the 3DShape2 VecSet [65] framework, with some mod- ifications. Specifically, we increase the input points of the VAE encoder to 32K, and utilize both point coordinates and normals input features to better capture fine-grained geometric details. The diffusion model is implemented as a DiT [41] with a width of 2048 and 24 layers. For image- conditioned generation, we use DINOv2 [40] to encode the input image I and part-masked images (+)) T (+) independently. The resulting features are concatenated along the channel dimension and passed through a small MLP to match the diffusion transformer input. We provide more details in the supplementary material.
Training. We use the AdamW optimizer with a learning rate of le-4 and train the model for 500K iterations on 256 NVIDIA H100 GPUs. Training the full model takes approximately 4 days. More details on hyperparameters and data preprocessing are provided in the supplementary material. During
7

---

3D Object
Object
Parts
3D Object
Object
Parts
Figure 5: Object-to-parts scenario. Given an input 3ID object, AutoPartGen regenerates it composition of meaningful and complete 3D parts.
Input
Mask 1
Mask 2
Input
Mask
Mask 2
Figure 6: Masks-to-pan to-parts scenario. AutoPartGenn reconstructs a compositional 3D object guided by user-pol diser-prviied 2D part masks. Varying these masks yields different decompositions, potentially at different levels of granularity.
training, we randomly drop the image condition, the geometry condition, or both with probabilities of 0.05 each. For CFG, we use Wing = 7 and Wgroom 4 as the default setting.
Training Data. Our training data pipeline draws inspiration from PartGen [5]. but is substantially scaled to encompass approximately 300K assets and 2M individual parts. We start with a collection of 1.8M 3D assets, all licensed that permit Al training. Each asset is stored in gITF/GLB formats, which contains multiple meshes in and embeds hierarchical structure. To manage complexity, if an asset contains more than a predefined maximum of 15 meshes, we iteratively merge meshes from the bottom up, until the mesh count is within this limit. To prepare for training VAE and diffusion models, we compute truncated signed distance for each part in normalized space and also render different views for image-conditioned cases. More details are included the supplementary materials.
4.2 Object, Image and Masks to 3D Parts Generation
We test AutoPartGen with different types of inputs demonstrate its versatility. Specifically, consider: (1) image-to- mnage-to-parts generation from a single input image, where the images are generated by text-to-image (2D) generators; (2) object-to-parts decomposition from full 3D mesh, with meshes sourced from Google Scanned Objects [1 I]; and (3) masks-to-parts generation with user-provided 2D part masks, where the masks taken from PartObjaverse- [61]. Figures 4 6 qualitatively demonstrates that AutoPartGen produces accurate and consistent 3D parts across all these input types.
4.3 Comparison to the State-of-the-Art
Evaluation Protocol. We use PartObjaverse-Ting [61] for evaluation, filtering out very small parts following the protocol of [60]. This dataset comprises objects from diverse categories with manually annotated 3D part segmentations. We use standard metrics to assess the quality of the reconstructed geometry: pneesectiionover-Unnee on 6 voxel grids, and the F-score adopts a distance threshold of 0.02. We report the quality of the reconstruction of individual parts and of the overall object after merging them.
(IoU), Chamfer Distance (CD), and F-score. loU is calculated
We compare AutoPartGen to two recent methods: PartGen [5] and HoloPart [60]. We focus on mask-controlled part generation. Recall that HoloPart takes as input the overall partial 3D object and 3D part segmentations and outputs the complete parts. We adapt both PartGen and AutoPartGen
Results.
8

---

Input Mesh
PartBen
HoloPart
Ours
Figure 7: Visual comparison of different completion methods. Our approach achieves better geometric coherence by considering previously generated parts in context.
(a)
Non-AR
AR
(b)
Wlnng
Wllng
7, yoon
Figure 8: Ablation Study. (a) Without autoregressive generation, parts overlap and intersect. (b) Increasing image guidance Wing encourages the model to follow image mask, while a larger geometry guidance Wgooo biases the generation towards generic part distribution and order in the data.
to solve the same problem. For AutoPartGen, we provide the overall partial 3D mesh and 2D part masks variant of masks-to-parts), For PartGen, we supply four masked views of each part, which are compatible with its input requirements.
The results in Table show that HoloPart outperforms PartGen, likely due to its access to prehensive input information (the partial 3D object and 3D part masks). Nevertheless, AutoPartGen surpasses both baselines across all key metries: IOU, F-score, and Chamfer Distance. This advantage holds true for both part completion and overall object reconstruction, indicating that AutoPartGen generates geometrically precise parts that form well-formed and coherent whole.
5 Ablation Study
We ablate three factors in AutoPartGen: the autoregressive design, the guidance scale, and the re-encoding strategy. We provide qualitative evidence for the first two factors and report quantitative results for all three.
Autoregressive modeling. We evaluate the contribution of the autoregressive design by removing the autoregressive condition 2(1.....--1) in the masks-to-part setting. This is the only setting where removal is possible, since external part guidance specifies which part generate next and when to stop. As shown in Figure 8(a), removing this conditioning causes parts to overlap and intersect. Quantitatively, Table 2 shows that enabling autoregression improves IoU and F-score, and reduces Chamfer Distance (CD), indicating better geometric fidelity and coherence.
Effectiveness of guidance. We analyze the impact of image guidance Wing and geometry guidance Wgeom in masks-to- tassssto-parts generation. Because image and geometry conditions are randomly dropped during training, the model learns a data data-driven prior for part partitioning and ordering when no image condition is present. As shown in Figure 8(b), increasing Wimg aligns parts more closely with input image masks, while higher Wgeom biases generation toward the learned prior of plausible part structures. Quantitatively, Table shows that moderate guidance values (for example, Wgeem 4 and Wing 7) strike the best balance, achieving peak performance in loU, F-score, and Chamfer Distance.
Effect of re-encoding. We further compare three ways to aggregate the VecSet tokens of different parts: (i) Re-encoding, which first decodes different parts into meshes, concatenates them, and re-encodes the concatenated mesh into a fixed-length latent; (ii) Concal, which directly feeds the concatenated latents into the cross-attention layer of the diffusion model; and (iii) Latent fuser, use 6-layer Perceiver-styie module that fuses the tokens into fix length of 512 latent tokens. All models

---

Table 2: Ablation study autoregressive. Autoregressive generation clearly show better results in terms of the part completion and overall object coherence. The models are only trained for 200 epochs. Here CD denotes Chamfer Distance.
Part Completion
Overall F-Score
Autoregressive IoU t F-Score
CD
IoU f
CD
0.574
0.795
0.067
0.783
0.917
0.031 0.022
0.633
0.825
0.052
0.811
0.934
Table 3: Effects of different guidance scales. We report the three metrics the part completion task, where CD denotes the Chamfer Distance.
9
Geometry/Image IoU
F-score
CD
IoU
F-score
CD
loU
F-score
CD
2.5
0.650
0.847
0.051
0.639
0.839
0.053
0.632
0.833
0.052
4
0.657
0.854
0.050
0.665
0.861
0.047
0.648
0.852
0.052
5
0.635
0.841
0.062
0.662
0.857
0.049
0.647
0.851
0.051
Table 4: Effect of re-encoding. All models are trained for 150 epochs with 512 tokens per part under the same setup.
Method
IoU
F-Score t
Chamfer Distance
Re-encoding
0.627
0.815
0.055
Concat
0.611
0.804
0.059
Latent Fuser
0.608
0.802
0.061
trained for 150 epochs with 512 tokens per part under the same setup. As summarized in Table all three methods perform similarly, with re-encoding slightly outperforms the two other strategies in terms of loU, F-score, and CD.
5.1 Applications
3D Scene Generation. Our method's capability for decomposable object generation naturally extends to entire 3D scenes. As illustrated in Figure (middle row), given an isometric view of a small scene, our approach automatically generates individual scene objects such as chairs, clocks, plants, and tables in a decomposable manner. This decomposable nature facilitates flexible manipulation and editing individual scene components.
City Generation. Figure (bottom row) further showcases our method's potential for large-scale outdoor scene generation. Drawing inspiration from SynCity [12], employ text-to-image genera- tor to produce diverse tile images from text prompts. These tiles are subsequently assembled to form coherent cityscapes, enabling scalable and controllable generation of complex urban environments. Further examples and qualitative demonstrations are provided in the supplementary material.
Conclusion
We introduced AutoPartGen, an autoregressive model for compositional 3D part generation. By leveraging latent 3D representations, our method generates coherent object parts sequentially. The same model can handle different input types, including images, 2D masks, and 3D meshes. Au- toPartGen outperforms existing methods in part completion and coherence while simplifying the overall pipeline. Our experiments demonstrate its effectiveness across various tasks and applications, highlighting its potential for scalable and controllable 3D content creation.
10

---

References
[1] Hertz Amir, Perel Or, Giryes Raja. Sorkine-Horunn Olga, and Cohen-Or Daniel. SPAGHETTI: editing implicit shapes through part aware generation. In ACM Transactions Graphics, 2022.
[2] Yash Sanjay Bhalgat, Iro Laina, Joao F. Henriques, Andrea Vedaldi, and Andrew Zisserman. Contrastive Lift: 3D object instance segmentation by slow-fast contrastive fusion. In Proceed- ings of Advances in Neural Information Processing Systems (NeurlPS), 2023.
[3] Yash Sanjay Bhalgat, Iro Laina, Joao F. Henriques, Andrew Zisserman, and Andrea Vedaldi. N2F2: Hierarchical scene understanding with nested neural feature fields. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[4] Aleksei Bokhovkin and Angela Dai. Neural part priors: Learning to optimize part-based object completion in rgb-d scans. In Proc. CVPR, 2023.
[5] Minghao Chen, Roman Shapovalov, Iro Laina, Jianyuan Wang Tom Monnier, David Novotny, and Andrea Vedaldi. PartGen: Part-level 3d generation and reconstruction with multi-view diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[6] Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng. and Ping Tan. Dora: Sampling and benchmarking for 3D shape variational auto-encoders. arXiv, 2412. 2024.
[7] Yilun Chen, Shuai Yang. Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-lIm with referent tokens. arXiv preprint arXiv: 10370, 2024.
[8] Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Compositional 3d assets creation using spatially-aware diffusion guidance. In Proc. ECCV, 2024.
[9] Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Lucid- Dreamer: Domain-free generation of 3d gaussian splatting scenes. In arXiv, 2023.
[10] Ken Deng, Yuanchen Guo, Jingxiang Sun, Zixin Zou, Yangguang Li, Xin Cai, Yampei Cao, Yebin Liu, and Ding Liang. DetailGen3D: generative 3D geometry enhancement via data-dependent flow. arXiv, 2. 168200, 2024.
[11] Laura Downs, Anthony Francis. Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B. McHugh, and Vincent Vanhoucke. Google Scanned Objects: A high-quality dataset of 3D scanned household items. In Proc. ICRA, 2022.
[12] Paul Engstler, Aleksandar Shtedritski, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. SynCity: training-free generation of 3d worlds. arXiv, 2503.16420, 16420, 2025.
[13] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. GET3D: A generative model of high quality 3D textured shapes learned from images. arXives, abs/2209. 1133, 2022.
[14] Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin, 2024.
(15) Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas A. Funkhouser. Local deep implicit functions for 3D shape. In Proc. CVPR, 2020
[16] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proc. CVPR, 2019.
[17] Junlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models from video diffusion models. In European Conference on Computer Vision, pages 333-350, Springer, 2024.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proc. NeurIPS, 2021.

---

[19] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. In Proc. ICLR, 2024.
[20] Zehuan Huang. Yuan-Chen Guo, Xingqiao An, Yunhan Yang. Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. Midi: Multi-instance diffusion for single image to 3d scene generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025.
[21] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi- Chi-Wing Fu. Neural template: Topology-aware reconstruction and disentangled generation of 3d meshes. In Proc. CVPR, 2022.
[22] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In Proc. ICLR, 2022.
[23] Bharath Raj Nagoor Kani, Hsin-Ying Ying Lee. Sergey Tulyakov, and Shubham Tulsiani. UpFusion: novel view diffusion from unposed sparse view observations. arXiv, 2024.
[24] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for real-time radiance field rendering. Proc. SIGGRAPH, 42(4), 2023.
[25] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: language embedded radiance fields. In Proc. ICCV, 2023.
[26] Chung Min Kim, Mingxuan Wu, Justin KerT, Ken Goldberg, Matthew Tancik, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. arXiv.cs, abs/2401.099499 2024.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proc. CVPR, 2023.
[28] Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. In Proc. NeurIPS, 2022.
[29] Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. SALAD: part-level latent diffusion for 3D shape generation and manipulation. In Proc. ICCV, 2023.
[30] Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long. CraftsMan: high-fidelity mesh generation with 3d native generation and interactive geometry refiner. arXiv, 14979, 2024.
[31] Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, and Yan-Pei Cao. TripoSG: high-fidelity 3D shape synthesis using large-scale rectified flow models. arXiv, 2502.06608, 2025.
[32] Connor Lin, Niloy Mitra, Gordon Wetzstein, Leonidas J. Guibas, and Paul Guerrero. NeuForm: adaptive overfitting for neural shape editing. In Proc. NeurIPS, 2022.
[33] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. arXiv.es, abs/2305.00s .08911, 2023.
[34] Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, and Wenping Wang. Part123: Part-aware 3d reconstruction from single-view image. arXiv, 16888, 2024.
[35] Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. PartSLIP: low-shot part segmentation for 3D point clouds via pretrained image-language models. In Proc. CVPR, 2023.
[36] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3D object. In Proc. ICCV, 2023.
12

---

[37] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDremmer: Generating multiview-consiseee images from single-view image. arXiv, 2300.03353, 2023.
[38] Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei Efros, and Mathieu Aubry. Differentiable blocks world: Qualitative 3d decomposition by rendering primitives. In Proc. NeurIPS, 2023.
[39] George Kiyohiro Nakayama, Mikaela Angelina Uy, Jiahui Huang, Shi-Min Hu, Ke Li, and Leonidas Guibas. DiffFacto: controllable part-based 3D point cloud generation with cross diffusion. In Proc. ICCV, 2023.
[40] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Ya- Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings 2023.
of the IEEE/CVF international conference on computer vision, pages
[42] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In Proc. ICLR, 2023.
[43] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. LangSplat: 3D language Gaussian splatting. In Proc. CVPR, 2024.
[44] Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. NeRF for outdoor scene relighting. In Proc. ECCV, 2021.
[45] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. In Proc. ICLR, 2024.
[46] Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. ICLR, 2021.
[47] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D generation with bootstrapped diffusion prior. arXiv.cs, abs/2310.168188, 2023.
[48] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter Image: Ultra-fast single-view 3D reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[49] Nicolas Talabot, Olivier Clerc, Arda Cinar Demirtas, Doruk Oner, and Pascal Fua. PartSDF: part- based implicit neural representation for composite 3d shape parametrization and optimization. arXiv, 2502. 12985, 2025.
[50] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view Gaussian model for high-resolution 3D content creation. arXiv, 2402.05054, 2024.
[51] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. DreamGaussian: Generative gaussian splatting for efficient 3D content creation. arXiv, 2309.16653 2023.
[52] Konstantinos Tertikas, Despoina Paschalidou, Boxiao Pan, Jeong Joon Park, Mikaela Angelina Uy. loannis Z. Emiris, Yannis Avrithis, and Leonidas J. Guibas. PartNeRF: Generating part- aware editable 3D shapes without 3D supervision. arXiv.cs, abs/2303.095555 2023.
[53] Vadim Tschernezki, Tro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In Proceedings of the International Conference 3D Vision (3DV), 2022.
13

---

[54] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifieDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. arXiv.cs, abs/2305. 16213, 2023.
[55] Daniel Watson, William Chan, Ricardo Martin- Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi. and Mohammad Norouzi. Novel view synthesis with diffusion models. In Proc. ICLR, 2023.
[56] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. MeshLRM: large reconstruction model for high-quality mesh. arXiv, 2404.12388 2024.
[57] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantMesh: efficient 3D mesh generation from a single image with sparse-view large reconstruc- tion models. arXiv, 1404.07191, 2024.
[58] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng. Yujun Shen, and Gordon Wetzstein. GRM: Large gaussian reconstruction model for efficient 3D reconstruction and generation. arXiv, 2403.14621, 2024.
[59] Jiayu Yang, Taizhang Shang. Weixuan Sun, Xibin Song, Ziang Chen, Senbo Wang, Shenzhou Chen, Weizhe Liu, Hongdong Li, and Pan Ji. Pandora3D: A comprehensive framework for high-quality 3D shape and texture generation. arXiv, 2502.14247, 2025.
[60] Yunhan Yang, Yuan-Cher Guo, Yukun Huang, Zi-Xin Zou, Zhipeng Yu, Yangguang Li, Yan-Pei Cao, and Xihui Liu. HoloPart: generative 3d part amodal segmentation. arXiv, 2504.07943, 2025.
[61] Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y Lam, Yan-Pei Cao, and Xihui Liu. Sampart3d: Segment any part in 3d objects. arXiv preprint arXiv:2411.07184. 2024.
[62] Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Qixuan Zhang, Lan Xu, Wei Yang. Jiayuan Gu, and Jingyi Yu. CAST: component-aligned 3D scene reconstruction from RGB image. arXiv, 2502. 12894, 2025.
[63] Chongjie Ye, Yushuang Wa, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han. Hi3DGen: High-fidelity 3D geometry generation from images via normal bridging. arXiv, 2025.
[64] Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu, Ruqi Huang, and Lu Fang. Omniseg3d: Omniversal 3d segmentation via hierarchical contrastive learning. In Proc. CVPR, 2024.
[65] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3eSSate2VecSet: A 3D shape representation for neural fields and generative diffusion models. In ACM Transactions on Graphics, 2023.
[66] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. CLAY: A controllable large-scale generative model for creating high-quality 3D assets. arXiv, 2024.
[67] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang. Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3D shape generation based on shape-image-tex aligned latent representation. In Proc. NeurIPS, 2023.
[68] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In Pruc. ICCV, 2021.
[69] Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Zero-shot scene reconstruction from single images with deep prior assembly. Advances in Neural Information Processing Systems, 2024.
[70] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3D: Exploring unified 3D representation at scale. In Proc. ICLR, 2024.
[71] Yuchen Zhou, Jiayuan Gu, Xuanlin Li, Minghua Liu, Yunhao Fang, and Hao Su. PIrtSLIP++: enhancing low-shot 3d part segmentation via multi-view instance segmentation and maximum likelihood estimation. arXiv, 2312.03015, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As mentioned in the abstract and introduction, we propose model Au- toPartGen that can generate objects composed of 3D parts in an autoregressive manner. Quantitative and qualitative results indicate that AutoPartGen is efficient and accurate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the Limitations in detail the Supplementary Materials.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Our work does not include any theoretical claims or results that require formal proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide enough information to reproduce our results. We provide more details in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will strive to release the code and model weights open source. However, our data license does not allow us release the data for training the model. Other authors should be able to obtain a similar dataset using resources like Objaverse.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Deelli//gudde/CooeeSuee otssSSmmmssioooocccoooooooo...ii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.. wwtpccppcpgguupiieeccceeecpoceee gww.////e/////eg /aaiee///oooodmaiaiaaa
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide such implementation details in the Experiment section and the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We are unable randomize experiments due to the cost of retraining the models for each run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide these details in the Experiment section and the Supplementary Material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http://eeerppp.eee https//cccpcctttcceeeeeeeeaaaed
Answer: [Yes]
Justification:
We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide discussion of Broader impacts in the Supplementary Material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Our model is unlikely to present any significant direct risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use dataset comprising artist-created 3D meshes and commercially licensed from them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy btttp::/neeuips... 2tt.r/ppscc//ooofeeeeee////2255 for what should or should not be described.
21

---

Supplementary Material
This supplementary material provides additional details and results to complement the main paper. includes the following sections:
Implementation Details:
Detailed descriptions of model architectures, training and infer- ence procedures, and evaluation protocols.
Additional Comparisons with PartGen: Extended evaluation against PartGen.
3D Scene Generation: Additional qualitative examples for scene generation.
City Generation: Details of the city generation pipeline and additional visual results.
Failure Case: Visualization of failure cases in AutoPartGien
Limitations and Broader Impact: Discussion of limitations and potential societal implica- tions of AutoParGenn
A Implementation Details
A.1 Training
Our model consists of two primary components: a 3D Variational Autoencoder (VAE) and diffusion model.
3D Variational Autoencoder. We adopt the 3D representation from 3DShape2 VecSet, extending it to larger model capacity compared to the original VAE [65]. Our VAE architecture comprises an 8-layer encoder with a dimension of 768 and a 16-layer decoder with a dimension of 1024. The model trained on approximately 1.7M 3D assets, with data augmentation techniques including point cloud rotations, as suggested by Dora [6]. We employ a signed distance function (SDF) representation for smoother isosurface extraction. During training, we supervise the VAE using a combination of surface normal loss, Eikonal loss, and KL divergence regularization, weighted by 10, 0.1, and 0.001, respectively, following TripoSG [31]. To learn a single signed distance field, we calculate combination of L1 and MSE loss on total of 24,576 points per shape: 8192 each from surface points, near-surface points, and randomly in the volume. We randomly vary the number of input tokens between (512, 2048) during training. The model is optimized using AdamW with learning rate of 1e-4, linearly warmed up from 1e-5 over the first 3 epochs. We use batch size of 1536 and set the weight decay to 0.01. Training is conducted 128 NVIDIA H100 GPUs for 150 epochs.
Pretraining and fine-tuning. For diffusion training, first pretrain general Image-to-31 model on the same 1.7M assets. The diffusion backbone follows DiT [41], with 24 transformer layers and hidden dimension 2048. We train with fixed token length of 512 for 300 epochs, learning rate 1e-4, and batch size 10 per GPU on 128 GPUs. We then fine-tune the model to additionally condition on masked image and geometry tokens the part dataset in autoregressive manner for approximately 300k steps. The image condition is encoded with DINO-v2 [40], and the geometry token is encoded with our trained 3D VAE. To reduce computation, geometry tokens are used only in the first 12 transformer layers. We apply condition dropping with probability 0,05 independently for the different combination of inputs simultaneously. Fine-tuning uses Adam W with weight decay 0.01 and batch size 6 per GPU on 128 GPUs. Subsequently, we increase the token length to 2048 and continue training for an additional 100k steps on 256 GPUs with batch size per GPU. We adopt the DDIM scheduler [46] with 1000 steps, use v-prediction, and a zero signal-to-noise ratio [33].
Ordering of parts. In the masks-to-parts setting, the order of the input masks specifies the order in which parts are generated. Both unmasked and masked images are provided so the model can learn spatial relationships between each part and the whole object in 3D space. In practice, the model can accurately infer very small parts (less than 0.1% of the object volume) from very small masks, and at inference time, users can interactively adjust the image guidance scale to control how strongly masks and images influence generation. the automatic setting, a predefined order is used during training and followed at inference. Training assets are defined in canonical space, and parts are sorted lexicographically by their axis-aligned bounding boxes: bottom to top (Z.), then left to right (X), then front to back (Y), following Blender's 's ZXY axis convention. Concretely, the minimum Z
22

---

values are compared first; if they are similar, the minimum X values are compared, and if still similar, the minimum Y values are used..
A.2 Inference
In all scenarios, we use 50 denoising steps during inference. For the object-to-parts setting, geometry guidance is set to 10, while image guidance disabled (set to 0). For both the image-to-parts and masks-to-par tortttts settings, we first perform image-to-3D reconstruction to obtain the overall object shape. When user-provides masks are available, we apply the default guidance setting, with image guidance set to and geometry guidance set to 4.
A.3 Evaluation
The most relevant baseline AutoPartGen is PartGen. We compare the methods under the object-to-parts setting, without incorporating any user inputs. For this comparison, we use objects from the Google Scanned Objects (GSO) dataset [1]].
When users provide masks to guide part partitioning, we compare our method with recent approaches, including HoloPart and PartGen. For evaluation, use the PartObjaverse- Tiny dataset [61]. To exclude negligible parts, we filter out objects containing segments that occupy only a small fraction of the total object volume in [60].
B Additional Comparison with PartGen
To further highlight the improvements over PartGen, we provide a qualitative comparison in Figure 9. As shown in the figure, AutoPartGen produces sharper and more detailed meshes, highlighted by the red circle. Additionally, the autoregressive generation in AutoPartGen avoids over-generation issues seen in PartGen, which arise from its lack of explicit modeling of the joint distribution of different parts. This is key capability addressed by AutoPartGen.
C 3D Scene Generation
Small scene generation is a natural extension of our method. Specifically, we begin by providing prompts such as "an isometric view of an office" or "an isometric view of small bedroom" to an off-the-shelf 2D text-to-image generator to produce corresponding images. We first generate the overall shape of the small scene. Subsequently, we apply AutoPartGen again to decompose the scene into distinct components such as "chair", "table" and so on. More qualitative examples are provided in Figure 10.
D City Generation
As shown in Figure I of the main paper, we demonstrate the ability of AutoPartGen to generate 3D cities by integrating it into the SynCity [37] pipeline, replacing its original 3D generator. Specifically, the pipeline begins with text prompts generated by a large language model, which are then used to guide 2D image generator in creating isometric views of individual tiles, taking into account the context of neighboring tiles. These generated images are then passed to AutoPartGen to produce compositional 3D tiles.
We present additional examples in Figure 11. As illustrated, AutoPartGen can be seamlessly integrated into the city generation pipeline, generating different cities, such 'medieval town' "solarpunk city".. For further details on prompt generation and 2D image synthesis, please refer to the SynCity paper.
23

---

PartGen
AutoPartGen
Input Object
Object
Parts
Object
Parts
Figure 9: Comparison between AutoPartGen and PartGen. AutoPartGen produces more accurate geometry, as highlighted in the red circle. Additionally, its autoregressive generation prevents the overlapping parts observed in PartGen, shown in the yellow circle.
E
Failure Cases
We present a failure case in Figure 12. As shown in the figure, when the object contains identical parts, the model attempts to predict multiple parts together, even when only is masked. We conjecture that this behavior comes from the bias in the training data, where some artists may have grouped identical parts into a single mesh. Additionally, as discussed in the ablation study, users may adjust the guidance scale to enforce stronger adherence the image, which can amplify this effect. Due to the model's 's au- toregressive nature, each prediction depends on previously generated parts. In the example shown in Figure 12, where both window instances are generated together, when the mask for the second window given as input, the model's prediction will be (nearly) empty, since the model has already generated that content. Hence, despite this potential failure mode, the model produces coherent results, maintaining consistency the overall shape.
Input
Masked Image
Part
Object
Failure Case. When there are identical parts, the model sometimes will try to predict the parts together even if masks are given.
Figure 12:
24

---

Input Image
Small
Parts
Input Image
Small
Parts
Input Image
Small
Parts
Input Image
Small Scene
Parts
Input Image
Small
Parts
Figure 10: 3D scene generation. AutoPartGen generates 3D scenes while decomposing them into their constituent elements. The input images are generated by 2D text-to-image generator.
25

---

arching
gracefully over gentle
Ireely stream winding past movey banks and clusters of wildfoowers
fortifted stucture with clock tower banners fluttering
Input prompts
Medieval town
Buildings
cute garden
greenhouse with some flowers
fews benches and lamp
Input prompts
Town
Buildings
futuristie solarpunk research buiicing
futuristic solarpunk community garden
futuristic solarpunk apartment building with rooftop garden
Input prompts
Salorpunk city
Buildings
Figure 11: City Generation. We showcase AutoPartGen on larger scenes by integrating it within Syncity [12]. From top to bottom, the images depict a medieval town, a cozy town, and a solarpunk city.
F Limitations and Boarder Impact
Limitations. While AutoPartGen demonstrates strong performance across all three scenarios, it also has some limitations that point to potential directions for future improvement. First, the current model only generate bounded scenes, as it inherits the spatial constraints from the underlying VAE latent space. Extending the framework to support unbounded world generation, where scenes can grow or evolve without predefined spatial limit, would be both interesting challenge and promising research direction. Second, the method currently lacks explicit control over the granularity of part partitioning, except the masks-to-parts setting, where masks be provided as way of control. In the mnage-to-parts and object-to-parts settings, the decomposition of parts can vary. In future iterations, one may incorporate high-level controls for granularity levels, such as 'simple", 'medium', and 'complicated', to make the system more flexible and interactive. Finally, the model learns part distributions directly from the training data, which introduces the risk of bias being inherited from the dataset.
Broader Impact. Although our model is trained on large amount of data, it may still exhibit biases that reflect imbalances in the underlying distribution. These biases can influence downstream tasks and should be carefully evaluated before practical deployment. To mitigate potential misuse or harmful applications, we recommend implementing safeguards and conducting thorough audits to ensure responsible usage. Additionally, the training process requires substantial computational resources, particularly terms of GPU usage. This raises concerns about energy consumption and the associated environmental impact, which should be taken into account when scaling deploying model in real-world settings.
26


Reviews:
[{"review_id": "DAoTvJoCdc", "reviewer": "Reviewer_V2su", "summary": "This paper introduces an autoregressive model for generating compositional 3D objects part by part. To achieve this, the authors first parameterize 3D objects into a latent space using the existing 3DShape2VecSet representation. They then train a latent diffusion model to progressively predict the latent code of each 3D part, conditioned on input evidence and the latents of previously generated parts. While the experimental results demonstrate plausible generation outcomes, the proposed method does not present significant technical innovation. Additionally, the overall writing quality of the paper could be improved.", "questions": "Here are my specific questions and suggestions:\n\n1. In the Introduction, it would be helpful to provide a visual example to illustrate the statement that \"the concatenation of two codes can be decoded into the union of the corresponding surfaces.\" Moreover, although the importance of concatenation is emphasized, the actual implementation does not use concatenation but instead re-encodes the known parts. What is the purpose of discussing concatenation in the Introduction if it is not used in practice?\n\n2. The paper claims that the method can automatically generate 3D objects part by part, but does not describe a termination strategy. How does the model determine when the scene has been fully generated?\n\n3. In Sec 3.1, the function ‘sample_N x’ would be clearer if written as ‘sample(N, x)’ to indicate its inputs. Additionally, the point set P is sampled from object x, and then ~P is sampled from P. Why not sample M points directly from x? What is the rationale behind this two-step sampling process?\n\n4. What is the architecture of the decoder D? How is it trained? What is the underlying principle that enables it to achieve the results shown in Fig. 3? Please provide a detailed discussion and explanation.\n\n5. In Sec 3.3, the process involves decoding, sampling, merging, and re-encoding latent codes. How does this approach differ from simply concatenating the latents? It seems that encoding a small 3D part and encoding a complex scene with the same encoder may yield different results. Please provide comparative experiments to demonstrate the differences between these two strategies.\n\n6. How are different parts defined in the training data? How do you ensure that each part has a consistent semantic meaning?\n\n7. The writing, especially in the Methods section, is somewhat verbose and difficult to follow. I recommend revising and refining this section to more clearly define key concepts and articulate the core methodology.\n\n8. Missing citations or comparisons:\n- MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation\n- Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly\n- CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image\n- Part123: Part-aware 3D Reconstruction from a Single-view Image\n\n9. Minor Issues and Typos:\n- The function ‘sample_N’ is defined multiple times in Sec 3.3.\n- The pipeline shown in Fig 2 is not referenced or explained in the main text.", "limitations": "No, in the conclusion, the authors need to provide a necessary discussion of the limitations of the method.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe method achieves impressive results in generating compositional 3D objects.\n\nWeaknesses:\nThe technical novelty of the paper is limited, as the approach essentially fine-tunes a latent diffusion model. Besides, many technical details are unclear, and the paper lacks necessary ablation studies and relevant citations.", "was_revised": true, "final_justification": "The authors' response has largely addressed my concerns. I agree to accept the paper contingent upon the promised content being incorporated into the revised camera-ready version.", "rebuttal": "> The technical novelty of the paper is limited, as the approach essentially fine-tunes a latent diffusion model. Besides, many technical details are unclear, and the paper lacks necessary ablation studies and relevant citations.\n\nWe respectfully disagree with the point and would like to summarize our contributions as follows:\n\n- **First to explore auto-regressive 3D part generation**, demonstrating the potential of auto-regressive generation for scalability and broader applications such as scene and city generation. Other works on part generation are *not* autoregressive. We specifically designed the re-encoding schema for efficiency as noted by Reviewer kSwB. \n\n- **No reliance on external 2D segmentation models**: Unlike most prior works, our method enables the model to learn 3D part decomposition directly, without additional 2D masks.\n\n- **Unified model for diverse part generation and decomposition tasks**: We show that a single model can generalize across a wide range of decomposition tasks, taking as input different modalities (3D models, images, and masks). No other part generator is this flexible.\n\n- **Compositionality of VecSet representation**: As noted by Reviewer kSwB and di68, we are the first to demonstrate that the VecSet representation supports compositional generation. This is not only helpful for part generation but also crucial for scaling to larger and more complex scenes, as shown in the examples of scene and city generation.\n\n> In the Introduction, it would be helpful to provide a visual example to illustrate the statement that \"the concatenation of two codes can be decoded into the union of the corresponding surfaces.\" Moreover, although the importance of concatenation is emphasized, the actual implementation does not use concatenation but instead re-encodes the known parts. What is the purpose of discussing concatenation in the Introduction if it is not used in practice?\n\nThank you for your suggestion! We have provided a visual example in Figure 3 and we plan to include more examples and discussion in the revision. The point is to show that the VecSet representation has good local properties which are suitable for part generation as each part is represented by an independent VecSet. However, instead of naively concatenating part codes, we opted for re-encoding the previously generated parts into a fixed-length VecSet, which allows us to bound the number of tokens entering cross-attention (lines 201–203) and results in improved efficiency. In particular, when the number of parts is large (the city generation example), simple concatenation would lead to a huge computation cost and would easily go out of memory.\n\nTo further explore this idea, we also experimented with a latent fuser module (a 6-layer Perceiver-style network) that explicitly fuses the *concatenated* VecSet tokens into a fixed-length latent. We provide the performance comparison in the following table. All models are trained for 150 epochs with 512 tokens per part and the same training setup. As shown in the table, all three methods achieve similar results, with re-encoding performing slightly better than the other two methods.\n\n| Method | IoU ↑ | F-Score ↑ | Chamfer Distance ↓|\n|-|-|-|-|\n| Re-encode |   0.627 | 0.815 | 0.055 | \n| Concat | 0.611 | 0.804 | 0.059 | \n| Latent Fuser |  0.608 | 0.802 | 0.061 | \n\nWe also provide GPU memory and runtime analysis for the different approaches, as shown below. Decoding time is not included, since all methods use the same mesh decoding process for each part. The experiments are conducted on a single NVIDIA A100 GPU (80GB) using PyTorch 2.7.1 with F.scaled_dot_product_attention.\n\n| #Part | Method  | Inference GPU Memory (GB) | Training GPU Memory (GB) | Inference Speed (sec/step) | Training Speed (sec/step) |\n|-------|----------------|---------------------------|---------------------------|-----------------------------|----------------------------|\n| 5     | Re-encode      | 17.9                      | 58.6                      | 0.616                       | 1.101                      |\n|       | Concat         | 30.3                      | 58.8                      | 0.737                       | 1.378                      |\n|       | Latent Fuser   | 30.3                      | 58.6                      | 0.622                     | 1.181                      |\n| 10    | Re-encode      | 17.9                      | 58.6                      | 0.626                       | 1.101                      |\n|       | Concat         | 33.3                      | 63.6                      | 0.956                       | 1.845                      |\n|       | Latent Fuser   | 33.3                      | 58.6                      | 0.644                       | 1.303                      |\n| 30    | Re-encode      | 17.9                      | 58.6                      | 0.626                       | 1.101                      |\n|       | Concat         | 45.4                      | OOM                       | 1.826                       | OOM                        |\n|       | Latent Fuser   | 45.3                      | 58.6                      | 0.736                       | 1.799                      |\n\nFrom the table above, we observe that concatenating all latents (Concat) significantly increases memory usage and slows down training, even when using the latest efficient PyTorch implementation. This is mainly because the number of tokens grows rapidly with the number of parts. To address this, we adopt the re-encoding strategy (Re-encode), where both runtime and memory usage remain stable even with a large number of parts, since the mesh joining step is negligible in time and GPU memory. All parts are re-encoded into a fixed-length VecSet regardless of their number.\n\n> The paper claims that the method can automatically generate 3D objects part by part, but does not describe a termination strategy. How does the model determine when the scene has been fully generated?\n\nThis is described in L210-211: the model terminates by generating the [EoT] token, which we simply define to be a zeroed-out shape vector $z$. \n\n>  In Sec 3.1, the function ‘sample_N x’ would be clearer if written as ‘sample(N, x)’ to indicate its inputs. Additionally, the point set P is sampled from object x, and then ~P is sampled from P. Why not sample M points directly from x? What is the rationale behind this two-step sampling process?\n\nAs described in L158-159, the VecSet representation encodes all the $N$ points -- it randomly selects a smaller subset $M \\subset N$ of these $N$ points as a \"support set\" (\"query\") as the input of encoder, but then encodes *all* the $N$ points via cross-attention (a component in the encoder function). This is just how the standard VecSet encoder is defined [29,55,59,60]. \n\n> What is the architecture of the decoder D? How is it trained? What is the underlying principle that enables it to achieve the results shown in Fig. 3? Please provide a detailed discussion and explanation.\n\nAs detailed in L846-858 in the supplementary material, the decoder architecture is similar to the 3DShape2VecSet [59], with the major difference of a larger model, training loss, data augmentation and the use of an SDF decoder instead of an opacity decoder. As to why the compositionality works, this appears to be an emerging property of VecSet itself, as we do not do special training to achieve it (which, we think, makes it an interesting observation). The likely reason is that the transformer is trained to operate on sets -- for example, it is permutation equivariant -- and that the elements in these sets are actually \"semantically\" close to 3D points because they start as such in the encoder. So, just like the union of sets of 3D points is well defined, the union of these sets of code vectors is also meaningful to the model. Note that this is non-trivial, as the concatenation of the latent codes must then be decoded in a new, overall consistent SDF function for the full object.\n\n> How are different parts defined in the training data? How do you ensure that each part has a consistent semantic meaning?\n\nWe define parts in the same way as PartGen [5]: these are artist-like parts, where the semantics are implicitly derived from how 3D artists naturally decompose objects. As described in Lines 261–268, each artist-created asset is stored in glTF/GLB format, which contains multiple meshes (i.e., parts) along with the hierarchical structure as defined by the artist. In practice, this setup captures a diverse range of plausible decompositions, which our model can sample from due to its probabilistic nature.\n\n> The writing, especially in the Methods section, is somewhat verbose and difficult to follow. I recommend revising and refining this section to more clearly define key concepts and articulate the core methodology.\n\nThank you, we will revise the method section to include more details.\n\n> Missing citations or comparisons.\n\nThank you for the constructive suggestions. We will cite all the recommended papers accordingly. We will add a detailed discussion section with related works. In fact, Part123 is already cited as [31]. The scope of MIDI, Deep Prior Assembly, and CAST differs from ours, as these works primarily focus on the instance level rather than the part level, making direct comparisons difficult. Furthermore, CAST was released on arXiv in February and is not open-sourced yet. As for Part123, the method only infers surface decompositions without producing complete parts, and it relies on an external 2D segmentation model such as SAM. In addition, MIDI is limited to generating up to five instances due to its reliance on costly attention between latent codes, whereas our method can efficiently generate tens of part tokens.\n\n> Minor Issues and Typos.\n\nThanks! We will correct them in the revision."}, {"review_id": "wKIrvtPhdk", "reviewer": "Reviewer_kSwB", "summary": "This paper proposes a new part-based 3D generator, named AutoPartGen. AutoPartGen leverages the locality property of the vecset-based 3D VAE latent space and trains a flow model to autoregressively generate part latents. The authors conduct several experiments to validate the proposed method.", "questions": "See the weaknesses, especially the part-order ambiguity.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The observation that locality, or at least compositionality, is well supported by the vecset representation is insightful. Interestingly, the authors demonstrate that directly concatenating two different latents can be decoded together.\n\n2. The proposed autoregressive generation process is reasonable. Additionally, AutoPartGen re-encodes all previously generated meshes to obtain more compact conditional latents, enhancing efficiency.\n\n3. The presentation is clear and easy to follow.\n\nWeaknesses:\n\n1. AutoPartGen suffers from part order ambiguity. The authors do not clarify how the model defines the autoregressive order. From my perspective, the conditionally masked image $J$ could provide some information on which part to generate. However, the masked image may contain limited information when the part size is small.\n\n2. More ablation experiments are expected, particularly regarding the re-encoding strategy to inject previous information.\n\n3. It is unclear how AutoPartGen generates texture, as the vecset VAE does not encode appearance information.", "was_revised": true, "final_justification": "Most of my concerns are addressed by the rebuttal. Therefore, I tend to maintain my score.", "rebuttal": "> AutoPartGen suffers from part order ambiguity. The authors do not clarify how the model defines the autoregressive order. From my perspective, the conditionally masked image $J$ could provide some information on which part to generate. However, the masked image may contain limited information when the part size is small.\n\nThank you for your helpful suggestion. In the masks-to-parts setting, the order of the input masks determines the order in which parts are generated. As described in the paper, we provide both the unmasked and masked images to help the model learn the spatial relationship between the parts and the whole object in 3D space. In our experiments, we found that the model can accurately infer very small parts (less than 0.1% of the object’s volume) with very small masks. In addition, the users could interactively adjust the image guidance scale to control the effect of masks and images (L213-217). \n\nIn the automatic setting, we use a predefined order during training, which the model also follows during inference. Specifically, the training assets are defined in a canonical space, and parts are sorted based on their bounding boxes in the following order: bottom to top (Z), left to right (X), and front to back (Y), following Blender’s ZXY axis convention. We first compare the minimum Z values of the parts; if they are similar, we then compare the minimum X values, and finally, the minimum Y values if needed. This \"*lexicographic*\" ordering is similar to the strategy used in autoregressive mesh generation models [*1, *2].\n\nWe have revised the paper accordingly to include more details about the part order. \n\n> More ablation experiments are expected, particularly regarding the re-encoding strategy to inject previous information.\n\nThanks! We have conducted additional experiments to further explore this direction. Specifically, we compare the re-encoding strategy with two alternatives: simply concatenating all part tokens, and using a latent fuser module based on a 6-layer Perceiver-style network that compresses the concatenated VecSet into a fixed-length latent. The performance comparison between these three approaches is shown in the table below. All models are trained for 150 epochs using 512 tokens per part under the same training setup. As shown in the table, all three methods achieve similar results, with re-encoding performing slightly better than the other two methods.\n\n\n| Method | IoU ↑ | F-Score ↑ | Chamfer Distance ↓|\n|-|-|-|-|\n| Re-encode |  0.627 | 0.815 | 0.055 | \n| Concat | 0.611 | 0.804 | 0.059 | \n| Latent Fuser |  0.608 | 0.802 | 0.061 | \n\nWe also provide GPU memory and runtime analysis for the different approaches, as shown below. Decoding time is not included, since all methods use the same mesh decoding process for each part. The experiments are conducted on a single NVIDIA A100 GPU (80GB) using PyTorch 2.7.1 with F.scaled_dot_product_attention.\n\n| #Part | Method  | Inference GPU Memory (GB) | Training GPU Memory (GB) | Inference Speed (sec/step) | Training Speed (sec/step) |\n|-------|----------------|---------------------------|---------------------------|-----------------------------|----------------------------|\n| 5     | Re-encode      | 17.9                      | 58.6                      | 0.616                       | 1.101                      |\n|       | Concat         | 30.3                      | 58.8                      | 0.737                       | 1.378                      |\n|       | Latent Fuser   | 30.3                      | 58.6                      | 0.622                     | 1.181                      |\n| 10    | Re-encode      | 17.9                      | 58.6                      | 0.626                       | 1.101                      |\n|       | Concat         | 33.3                      | 63.6                      | 0.956                       | 1.845                      |\n|       | Latent Fuser   | 33.3                      | 58.6                      | 0.644                       | 1.303                      |\n| 30    | Re-encode      | 17.9                      | 58.6                      | 0.626                       | 1.101                      |\n|       | Concat         | 45.4                      | OOM                       | 1.826                       | OOM                        |\n|       | Latent Fuser   | 45.3                      | 58.6                      | 0.736                       | 1.799                      |\n\nFrom the table above, we observe that concatenating all latents (Concat) significantly increases memory usage and slows down training, even when using the latest efficient PyTorch implementation. This is mainly because the number of tokens grows rapidly with the number of parts. To address this, we adopt the re-encoding strategy (Re-encode), where both runtime and memory usage remain stable even with a large number of parts, since the mesh joining step is negligible in time and GPU memory. Regardless of how many parts are involved, they are re-encoded into a fixed-length VecSet.\n\n> It is unclear how AutoPartGen generates texture, as the vecset VAE does not encode appearance information.\n\nSimilar to VecSet [59] and other related works [28, 29, 55, 60, 61, *1, *2], our model focuses solely on generating 3D geometry without textures. The primary focus of this paper is on modeling shape, and the VecSet representation does not include appearance information. However, VecSet is typically used in conjunction with texture generation models such as TextureGen [*3] or MV-Adapter [*4], as demonstrated in [28, 29, 60], to generate appearance after geometry is established. We plan to further explore this direction in future work, aiming to jointly model both geometry and appearance.\n\n[*1] Siddiqui, Yawar, et al. \"Meshgpt: Generating triangle meshes with decoder-only transformers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[*2] Chen, Yiwen, et al. \"Meshanything: Artist-created mesh generation with autoregressive transformers.\" arXiv preprint arXiv:2406.10163 (2024).\n\n[*3] Bensadoun, Raphael, et al. \"Meta 3D TextureGen: Fast and consistent texture generation for 3d objects.\" arXiv preprint arXiv:2407.02430 (2024).\n\n[*4] Huang, Zehuan, et al. \"MV-Adapter: Multi-view consistent image generation made easy.\" arXiv preprint arXiv:2412.03632 (2024)."}, {"review_id": "mat359yLHa", "reviewer": "Reviewer_di68", "summary": "The method is about generating 3D objects/scenes composed of many parts. The training of the method contains two stages: 1) part autoencoder; 2) conditioned diffusion model. When doing inference, an object is generated part-by-part autoregressively. Extensive experiments are done to prove the effectiveness of the method, including object- and scene-level generation.", "questions": "I still do not understand how the parts are ordered. Usually, we need an order for the elements used in autoregressive models, like the natural language order or rasterization order for images. However, from the context, I do not get what the order is here.\n\nThe inference includes a (autoregressive) set of diffusion generation processes. I would expect it would be very slow. It would be better if there were a time analysis of the inference cost.", "limitations": "The method takes really lots of GPUs (128 for the autoencoder and 256 for the diffusion models) to train. This could be a major limitation for most universities (which do not have access to large computing clusters). However, this is not a real drawback. On the contrary, it shows the scaling ability.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The writing is clear. The core intuition is well-described in the introduction. It is based on a simple observation of the combinatorial property of 3DShape2VecSet (see below). The outer level of the generation is done in an autoregressive way, i.e., part-by-part. The inner level is done using diffusion models in the latent space of parts.\n\nThe property described in Fig 3 and Sec 3.1 is a very interesting observation. This not only helps the part-aware generation as shown in this paper, but also might be helpful in scaling 3DShape2VecSet to a larger latent space while preserving high-quality details. Actually, I am quite surprised about the property. It would be better if there were some math analysis about the property.\n\nThe main experiments are about object-level generation, including different conditions (images and objects). The experiment setup makes sense and the results are strong. There are also applications on scene-level generation.", "was_revised": true, "final_justification": "Thanks for the response. I recommend acceptance. The authors delivered some interesting results.", "rebuttal": "> The property described in Fig 3 and Sec 3.1 is a very interesting observation. This not only helps the part-aware generation as shown in this paper, but also might be helpful in scaling 3DShape2VecSet to a larger latent space while preserving high-quality details. Actually, I am quite surprised about the property. It would be better if there were some math analysis about the property.\n\nThank you, we also find this property very interesting. As to why the compositionality works, this appears to be an emerging property of VecSet itself as we do not do special training to achieve it (which, we think, makes it even more interesting). The likely reason is that the transformer is trained to operate on sets -- for example, it is permutation equivariant -- and that the elements in these sets are actually \"semantically\" close to 3D points because they start as such in the encoder. So, just like the union of sets of 3D points is well defined, the union of these sets of code vectors is also meaningful to the model. Note that this is non-trivial, as the concatenation of the latent codes must then be decoded in a new, overall consistent SDF function for the full object. \n\nA recent work [*1] also has a related observation, as it found that the spatial queries of the decoder and shape latents show strong locality. Specifically, they find that nearby queries have high attention to a similar small group of latent tokens, which are then further used to accelerate the decoding process.\n\n> I still do not understand how the parts are ordered. Usually, we need an order for the elements used in autoregressive models, like the natural language order or rasterization order for images. However, from the context, I do not get what the order is here.\n\nThank you for your helpful suggestion. In the masks-to-parts scenario, the order of the input masks determines the order in which parts are generated. In the automatic setting, we use a predefined order during training, which the model also follows during inference. Specifically, the training assets are defined in a canonical space, and parts are sorted based on their bounding boxes in the following order: bottom to top (Z), left to right (X), and front to back (Y), following Blender’s ZXY axis convention. We first compare the minimum Z values of the parts; if they are similar, we then compare the minimum X values, and finally, the minimum Y values if needed. This \"*lexicographic*\" ordering is similar to the strategy used in autoregressive mesh generation models [*2, *3].\n\n\n> The inference includes a (autoregressive) set of diffusion generation processes. I would expect it would be very slow. It would be better if there were a time analysis of the inference cost.\n\nThe inference time of AutoPartGen primarily depends on the number of parts being generated, increasing approximately linearly since the time cost remains constant per part. We report a detailed time analysis in the table below. Not that each part generation involves several sequential steps: (1) encoding the input conditions (e.g., images and previously generated shape); (2) running latent diffusion; (3) decoding the latent into an SDF grid; and (4) extracting the mesh using Marching Cubes.\n\n| Encode Conditions | Diffusion  | Part Decoding |  Marching Cubes  |\n|-------|-------|-------|-------|\n|0.13s| 12.2s| 36.3s| 2.1s|\n\nAs shown in the table, the current bottleneck of the pipeline lies in the diffusion process and decoding. To accelerate inference, we can adopt FlashVDM [*1], which reduces the decoding runtime from 36.3s to <10s. Additionally, efficiency can be further improved by employing a distillation strategy in the diffusion process, as demonstrated in [*1].\n\n> The method takes really lots of GPUs (128 for the autoencoder and 256 for the diffusion models) to train. This could be a major limitation for most universities (which do not have access to large computing clusters). However, this is not a real drawback. On the contrary, it shows the scaling ability.\n\nWe agree that our training requires substantial resources (128 GPUs for the autoencoder and 256 for the diffusion model), which may not be accessible to all institutions. However, our goal is to demonstrate the scaling potential of the method, especially for highly detailed 3D generation and scene generation. Smaller-scale versions can be trained with fewer resources by reducing model and batch sizes, though this typically comes at the cost of reduced performance. We briefly experimented with training on 64 GPUs and found that the model converges more slowly, with a ~0.03 IoU drop compared to the model trained with the same number of iterations on 256 GPUs. \n\n[*1] Lai, Zeqiang, et al. \"Unleashing vecset diffusion model for fast shape generation.\" arXiv preprint arXiv:2503.16302 (2025).\n\n[*2] Siddiqui, Yawar, et al. \"Meshgpt: Generating triangle meshes with decoder-only transformers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[*3] Chen, Yiwen, et al. \"Meshanything: Artist-created mesh generation with autoregressive transformers.\" arXiv preprint arXiv:2406.10163 (2024)."}, {"review_id": "N7WUNOwJDa", "reviewer": "Reviewer_bEVk", "summary": "This work presents an auto-regressive method for 3D shape part generation. The approach dynamically decides which part to be generated and the number of generated parts. Results demonstrate it can achieve composable object generation without too much overlap.", "questions": "Basically, I encourage the authors to put more effort into improving the writing quality and paper organization. I did not think the paper is clear enough for readers to understand the technical details and reproduce the method. \nThe author mentioned the ambiguous problem for part generation. I wonder how the method addresses them.", "limitations": "It would be good to see discussions about failure cases.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Generating part-level objects is essential for many downstream tasks. \n+ The overall autoregressive generation is reasonable and novel.\n+ The generated parts have less overlap and are of better quality.\n\nWeakness:\n- The paper and its motivation are not well organized. For example, I did not see why supporting different modal inputs is challenging in the design, but this is emphasized in the introduction. Moreover, I lost myself when going through technical details. For example, how to generate \\hat{z}? If each step takes a point as input, this should be reflected in the pipeline, and there should be more descriptions about how the points are sampled. Line 162: Should there be an additional \\hat{P} for the SDF function? It is unclear to me how the diffusion process is used. In Fig. 3, the symbol z^(1) is not defined, and there are inconsistent symbol usages for z in the method part. In line 181, is the equation correct? Is there any way to ensure the \\hat{z} and the generated parts are consistent? \n- The evaluation is not solid enough. If you have ground truth parts, why not evaluate the mIoU for all ablations and comparisons? For the present results, it would be hard for readers to judge the advantage of the method. Moreover, is there any specific reason why there is less overlap for the reported results? \n- I did not see specific designs of AR models for part generations, and the shape generation is mainly based on Shape2VecSet.", "was_revised": true, "final_justification": "Basically, I think it requires a careful revision and reconsideration about its core contributions. I do not object to its acceptance contingent on the following revisions. \n\n1) A more focused contribution on diffusion-based AR models and their technical novelty. \n\n2) A thorough revision for its technical method part to ease the understanding. \n\n3) Evaluations on the role of diffusion parts and the advantage of compositional shape2vecset methods over other general ones. If there are other technical claims, a careful ablation over their alternatives should be evaluated.", "rebuttal": "> I did not see why supporting different modal inputs is challenging in the design, but this is emphasized in the introduction. \n\nAs in VecSet [59] and other prior works, each latent z is *generated* by a denoising diffusion model. The difference compared to prior work is how these generators are prompted (conditioned). In the original VecSet and other 3D generation works [18, 43, 59, 60, 61], the prompt is just an image. In our case, it can be an image, an image and a mask, and/or an encoded 3D shape, depending on which specific task one wants to solve. Hence, we design a unified model to handle different inputs in just one network. It is non-trivial that the same set of weights can generate parts across all these different scenarios and input modalities, as this requires training on diverse tasks using a unified training signal and supporting a general inference pipeline.\n\n> how to generate \\hat{z}? \n\nTo generate $\\hat{z}$, users can either directly provide an unmasked image to our model or encode a 3D shape into $\\hat{z}$. For example, one can use an existing 3D scan as shown in the paper (Section 4.2), or assets generated by other methods (L205-207).\n\n> If each step takes a point as input, this should be reflected in the pipeline, and there should be more descriptions about how the points are sampled. Line 162: Should there be an additional \\hat{P} for the SDF function?\n\nWe will clarify this process in the revision. As described in L151–154, the decoder is a function that takes a 3D point as input, along with the latent code (i.e., the encoded shape), and outputs the signed distance from the point to the shape surface. To extract the mesh, we evaluate this function over a grid of points and apply Marching Cubes to the resulting SDF values.\n\n> It is unclear to me how the diffusion process is used.\n\nThe diffusion model is used to generate part latents, where each latent represents a distinct part of the object, conditioned on previous part latents and a choice of other modalities such as 3D shape, image, or masked image.\n\n> In Fig. 3, the symbol z^(1) is not defined, and there are inconsistent symbol usages for z in the method part. \n\nThank you for the suggestion. We will revise the paper to clarify these notations. Briefly:\n\n- $z^{(i)}$: latent representation of the $i$-th part (L192). \n- $z$: latent representation of a part or the entire object, depending on context (L186–187). \n- $z^{(1,\\dots,k-1)}$: latent representation of the combined mesh formed by parts 1 through $k-1$ (L203). \n- $\\hat{z}$: latent representation of the whole object (L206). \n- $z_t$: noised version of $z$ used during the diffusion process (L179). \n\n> In line 181, is the equation correct?\n\nThank you for spotting this! The equation should be $v(t, z_t,\\epsilon) = (z_t - \\sqrt{\\alpha_t} \\epsilon)/\\sqrt{1 - \\alpha_t}$. We have corrected this in the paper.\n\n> Is there any way to ensure the \\hat{z} and the generated parts are consistent?\n\nWe do this by explicitly conditioning the next part generation process on both the latent representation of previous generated parts $z^{(1,\\dots,k-1)}$ and the whole object $\\hat{z}$ (L205-211). \n\n> The evaluation is not solid enough. If you have ground truth parts, why not evaluate the mIoU for all ablations and comparisons? For the present results, it would be hard for readers to judge the advantage of the method. \n\nWe have included quantitative results, including IoU, in our ablation study (see Tables 2 and 3 in the supplementary material). In addition, we have conducted additional experiments in response to other reviewers, which you can refer to.\n\n> Moreover, is there any specific reason why there is less overlap for the reported results?\n\nThis is a result of our autoregressive design, as discussed in L295–298 and illustrated in Figure 8. Unlike parallel extraction, where parts are predicted independently, the autoregressive approach allows the model to capture dependencies between parts, helping to avoid undesirable overlaps. Theoretically, the autoregressive model learns the joint probability distribution over all parts, whereas the parallel approach models only the marginal distributions.\n\n\n> I did not see specific designs of AR models for part generations, and the shape generation is mainly based on Shape2VecSet. \n\nThank you for the comment. While our shape representation builds on Shape2VecSet, we introduce three key architectural changes to enable autoregressive part generation:\n- **Autoregressive Diffusion Formulation:** We formulate the part generation process as an autoregressive diffusion model, which is novel in the context of 3D part generation. Unlike standard diffusion models that generate latent all at once, our approach generates each part sequentially, allowing the model to capture the dependencies between parts.\n- **Efficient Encoding of Previous Parts:** We explore how to best encode previously generated parts, as detailed in our additional ablation studies (see response to other reviewers). We propose a compact and efficient strategy to incorporate the information from earlier parts without increasing the token count or computational cost.\n- **Unified Formulation and Modeling:** We design a unified model and formulation that supports various input modalities, including 3D shapes, images, and masks, making the approach flexible and broadly applicable."}, {"review_id": "xcA78bUvWw", "reviewer": "Reviewer_Hib7", "summary": "The method generates 3D objects by dividing them into smaller parts and synthesizing each part independently. It supports generation from either an image or a 3D object, with optional part-specific masks. The process follows an autoregressive strategy, where each part is conditioned on all previously generated parts.", "questions": "1. Generation Order: How is the order of part generation determined? Can we predict which parts are generated first? During training, how do you decide the sequence of parts?\n2. Canonical Positioning: Is the canonical position of each part defined based on the final assembled object, or can the object shape and part positions be learned independently?\n3. Detail Level of Parts: How can we assess the level of detail in generated parts? For example, in the room scene (Fig. 1 middle), the “chair” part is generated as a whole, but generating a chair object as parts typically involves distinct components like handles and wheels.\n4. Necessity of Part-wise Generation: If the model is capable of generating full objects, what is the motivation for generating parts separately?", "limitations": "1. The method cannot handle objects without clear parts, such as natural elements like seas or rivers, or highly complex objects like trees with many small leaves.\n2. The paper does not address texture generation, which limits the realism and completeness of the generated 3D parts.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. New approach for generating 3D assets, by autoregressive generating parts of the object. This approach can provide more information about the generated object about it's decomposition.\n2. Three tasks involving part generation. image-to-parts, object-to-parts, masks-to-parts.\n\nWeaknesses:\n1. The method cannot handle objects without clear parts, such as natural elements like seas or rivers, or highly complex objects like trees with many small leaves.\n2. Applying the same resolution to all parts may result in some parts being overly detailed, while others appear low-resolution, leading to inconsistent quality.\n3. The paper does not address texture generation, which limits the realism and completeness of the generated 3D parts.", "was_revised": true, "final_justification": "i see no reason to change my rating", "rebuttal": "> The method cannot handle objects without clear parts, such as natural elements like seas or rivers, or highly complex objects like trees with many small leaves.\n\nThanks! It is not clear how one would decompose *stuff* categories like \"water\" into parts, so we think it is legitimate to leave this out of scope. As for the object complexity, the main problem is one of *scalability* rather than of principle. Given a tree, AutoPartGen can extract the canopy, trunk, roots, and branches, but it is not trained to extract thousands of small leaves. However, recent high-resolution 3D generators [*1] suggest that this might be possible in the future.\n\n> Applying the same resolution to all parts may result in some parts being overly detailed, while others appear low-resolution, leading to inconsistent quality.\n\nThis is a good point, and allocating a variable number of tokens to different parts is certainly an interesting direction for future work. In our current setup, we follow the original 3DShape2VecSet [59] design and adopt a fixed-length VecSet representation for all parts. As in the original paper [59], the object is encoded into a fixed-length latent regardless of the object complexity. This approach is simple and robust, as demonstrated in prior works [29, 55, 59, 60].\n\n> The paper does not address texture generation, which limits the realism and completeness of the generated 3D parts.\n\nSimilar to VecSet [59] and other related works [28, 29, 55, 60, 61], our model focuses solely on generating 3D geometry without textures. The VecSet representation, which we build on, does not include appearance information. However, VecSet is typically used in conjunction with texture generation models such as TextureGen [*2] or MV-Adapter [*3], as demonstrated in [28, 29, 60], to generate appearance after geometry is established. We plan to further explore this direction in future work, aiming to jointly model both geometry and appearance.\n\n\n> Generation Order: How is the order of part generation determined? Can we predict which parts are generated first? During training, how do you decide the sequence of parts?\n\nThank you for your helpful suggestion. In the masks-to-parts scenario, the order of the input masks determines the order in which parts are generated. In the automatic setting, we use a predefined order during training, which the model also follows during inference. Specifically, the training assets are defined in a canonical space, and we sort the parts based on their bounding boxes in the order of bottom to top (Z), left to right (X), and front to back (Y), following Blender’s ZXY axis convention. We first compare the minimum Z values of the parts; if they are similar, we then compare the minimum X values, and finally, the minimum Y values if needed.  This \"lexicographic\" ordering is similar to the strategy used in autoregressive mesh generation models [*4, *5].\n\n> Canonical Positioning: Is the canonical position of each part defined based on the final assembled object, or can the object shape and part positions be learned independently?\n\nThe canonical position and orientation of each part in our method are defined based on the final assembled object. Generating the object shape and parts independently would require a different design paradigm. While some scene generation works [*6, *7, *8] explore independent instance generation followed by placement, such approaches often suffer from inaccurate spatial relationships. In contrast, for part-level modeling, maintaining correct relative positions is crucial since parts must be composed into a coherent object. Defining part size, position, and orientation based on the assembled object allows us to preserve spatial consistency and eliminates the need for a separate optimization step during assembly.\n\n> Detail Level of Parts: How can we assess the level of detail in generated parts? For example, in the room scene (Fig. 1 middle), the “chair” part is generated as a whole, but generating a chair object as parts typically involves distinct components like handles and wheels.\n\nThank you, this is an interesting question. In our current setup, the level of detail in part decomposition is implicitly determined by the granularity of artist-annotated parts in the training data. For example, if the full object is a chair, the decomposition typically includes parts like legs, cushion, and backrest. However, if the full asset contains multiple objects, such as chairs and tables, the decomposition often reflects object-level grouping. In practice, as in PartGen [5], our model can learn all such decompositions, if given in the training data, and can generate different ones simply by resampling. In addition, if the user wants a more fine-grained decomposition, the output part latent can be fed back into AutoPartGen as input (object-to-parts).\n\nIn general, our model learns this granularity directly from the data distribution, and so a different granularity can be learned by tweaking the training data.\n\nIn addition, we provide a masks-to-parts setting as described in the paper, where users can control the decomposition level using 2D masks. These masks can be manually annotated or generated using segmentation pipelines like SAM.\n\n> Necessity of Part-wise Generation: If the model is capable of generating full objects, what is the motivation for generating parts separately?\n\nThank you. While both previous works and our method can generate a whole object, part-level generation is motivated by several practical and technical needs (L17-26):\n- **3D editing and customization**: In real-world applications, part-based generation enables intuitive editing. Users can modify or replace specific components (e.g., change the legs of a chair or the handle of a mug) without regenerating the entire object. \n- **Animation and articulation**: Many objects, especially those with moving parts (like doors, arms, or wheels), require separate motion for each part. Part-wise generation is essential for producing objects that can be easily rigged and animated.\n- **Robotic manipulation and spatial intelligence**: Robots often need to identify and interact with functional parts of objects, such as handles, lids, or levers. Explicit part representations make it easier for robots to understand object functionality and graspable regions. In addition, part-wise generation supports scalable data creation for robotics, as robots typically interact with objects at the part level during tasks like grasping, assembly, and tool use [*9, *10].\n- **Material assignment**: Different parts of an object often have different physical properties and materials (e.g., a chair with a metal frame and fabric seat). Part-wise generation makes it easier to assign materials in a semantically consistent way. \n- **More detailed generation**: Since each part is generated with different VecSet latents, the model can allocate capacity to capture fine-grained geometry per part. This leads to higher overall fidelity and expressiveness than generating a coarse full-object mesh in fixed-length VecSet.\n- **Fabrication, including 3D printing**: Many fabrication workflows, including 3D printing and physical assembly, require objects to be split into discrete parts. Part-wise generation aligns naturally with this process, enabling easier design, modification, and physical realization of objects. \n\n\n[*1] Chen, Yiwen, et al. \"Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention.\" arXiv preprint arXiv:2507.17745 (2025).\n\n[*2] Bensadoun, Raphael, et al. \"Meta 3d texturegen: Fast and consistent texture generation for 3d objects.\" arXiv preprint arXiv:2407.02430 (2024).\n\n[*3] Huang, Zehuan, et al. \"Mv-adapter: Multi-view consistent image generation made easy.\" arXiv preprint arXiv:2412.03632 (2024).\n\n[*4] Lai, Zeqiang, et al. \"Unleashing vecset diffusion model for fast shape generation.\" arXiv preprint arXiv:2503.16302 (2025).\n\n[*5] Siddiqui, Yawar, et al. \"Meshgpt: Generating triangle meshes with decoder-only transformers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[*6] Wang, Kai, et al. \"Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks.\" ACM Transactions on Graphics (TOG) 38.4 (2019): 1-15.\n\n[*7]  Nie, Yinyu, et al. \"Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[*8] Gao, Daoyi, et al. \"Diffcad: Weakly-supervised probabilistic cad model retrieval and alignment from an rgb image.\" ACM Transactions on Graphics (TOG) 43.4 (2024): 1-15.\n\n[*9] Liu, Weiyu, et al. \"Composable part-based manipulation.\" arXiv preprint arXiv:2405.05876 (2024).\n\n[*10] Huang, Haoxu, et al. \"Copa: General robotic manipulation through spatial constraints of parts with foundation models.\" 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024."}]

Author rebuttal:

Meta-review: This paper introduces AutoPartGen, an auto-regressive framework for 3D shape part generation that directly identifies and generates parts from input conditions (e.g., images or shapes) without requiring external segmentation modules. The method consists of two training stages: (1) a part autoencoder to learn compact representations of shape components, and (2) a conditioned diffusion model for autoregressive generation. During inference, the model generates 3D objects sequentially, part by part, enabling fine-grained control over structure.

The expert reviewers’ final recommendations are 3x Borderline Accept, 1x Accept, and 1x Borderline Reject, reflecting a generally positive but cautious reception after careful deliberation and consideration of the authors’ rebuttal.

The key strengths are: The method demonstrates a novel end-to-end approach for part-aware 3D generation, avoiding reliance on pre-defined or externally computed segmentations; The re-encoding strategy introduced in the rebuttal shows promising results in terms of efficiency and performance, with stable runtime and memory usage even for high-part-count models.

After a thorough discussion and a careful review of the authors' rebuttal, two primary concerns were raised:
1. **Part Order Ambiguity**:  Several reviewers questioned how the model defines its autoregressive generation order, a critical detail that was not sufficiently clarified in the original submission. While the authors address this in the rebuttal, the discussion of order ambiguity should be included in the revised version.  
2. **Insufficient Ablation Study**: The initial experiments lacked the necessary ablations to isolate and validate the contributions of the paper’s core technical components. In response, the authors added experiments comparing the proposed re-encoding strategy against alternatives. These new results support the technical contribution, showing marginal improvements in performance and consistent scalability, which helps strengthen the case for the method.

While the proposed approach offers a compelling direction for structured 3D generation, the concerns around part-ordering and the depth of analysis remain significant. The authors have made meaningful progress in addressing these issues through the rebuttal, particularly by enhancing the ablation study. Therefore, despite the remaining borderline sentiment from some reviewers, the paper is recommended for acceptance. The final version should incorporate the clarifications and new experimental results presented in the rebuttal.
Decision: Accept (poster)