PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
Authors: Yuchen Lin
Chenguo Lin
Panwang Pan
Honglei Yan
Feng Yiqiang
Yadong MU
Katerina Fragkiadaki
Venue: NeurIPS 2025 poster ()
Keywords: 3D Generation
3D Part-level Object Generation
3D Scene Generation
Generative Models
Primary area: applications
TL;DR: PartCrafter is a structured 3D generative model that jointly generates multiple parts and objects from a single RGB image in one shot.

Abstract: We introduce PartCrafter, the first structured 3D generative model that jointly synthesizes multiple semantically meaningful and geometrically distinct 3D meshes from a single RGB image. Unlike existing methods that either produce monolithic 3D shapes or follow two-stage pipelines, i.e. first segmenting an image and then reconstructing each segment, PartCrafter adopts a unified, compositional generation architecture that does not rely on pre-segmented inputs. Conditioned on a single image, it simultaneously denoises multiple 3D parts, enabling end-to-end part-aware generation of both individual objects and complex multi-object scenes. PartCrafter builds upon a pretrained 3D mesh diffusion transformer (DiT) trained on whole objects, inheriting the pretrained weights, encoder, and decoder, and introduces two key innovations: (1) A compositional latent space, where each 3D part is represented by a set of disentangled latent tokens; (2) A hierarchical attention mechanism that enables structured information flow both within individual parts and across all parts, ensuring global coherence while preserving part-level detail during generation. To support part-level supervision, we curate a new dataset by mining part-level annotations from large-scale 3D object datasets. Experiments show that PartCrafter outperforms existing approaches in generating decomposable 3D meshes, including parts that are not directly visible in input images, demonstrating the strength of part-aware generative priors for 3D understanding and synthesis. Code and training data are released.

PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
Yuchen Lin 1.3 Chenguo Lin Panwang Pt
Honglei Yan", Yiqiang Feng", Yadong Mu', Katerina Fragkiadaki
Equal contribution Project lead
Peking University, BByteDance, 'Carnegie Mellon University
attps://wget gi pprojects/partcrfftee
Figure I: PARTCRAFTER is a structured 3D generative model that jointly generates multiple parts and objects from single RGB image in one shot, without the need for segmented image inputs.
Abstract
We introduce PARTCRAFTER, the first structured 3D generative model that jointly synthesizes multiple semantically meaningful and geometrically distinct 3D meshes from single RGB image. Unlike existing methods that either produce monolithic 3D shapes or follow two-stage pipelines, i.e. first segmenting an image and then reconstructing each segment, PARTCRAFTER adopts unified, compositional generation architecture that does not rely on pre-segmented inputs. Conditioned single image, simultaneously denoises multiple 3D parts, enabling end-to- end part-aware generation of both individual objects and complex multi-object scenes. PARTCRAFTER builds upon a pretrained 3D mesh diffusion transformer (DiT) trained on whole objects, inheriting the pretrained weights, encoder, and decoder, and introduces two key innovations: (1) compositional latent space, where each 3D) part is represented by a set of disentangled latent tokens; (2) A hierarchical attention mechanism that enables structured information flow both within individual parts and across all parts, ensuring global coherence while preserving part-level detail during generation. To support part-level supervision, curate new dataset by mining pert-level annotations from large-scale 3D object datasets. Experiments show that MARTCRAFTED outperforms existing approaches in generating decomposable 3D meshes, including parts that not directly visible in imput images, demonstrating the strength of part-aware generative priors for 3D understanding and synthesis. Code and training data are released.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
central organizing principle in perception is the role of objects and parts semantillyy coherent units that serve as compositional building blocks for higher-level cognitive tasks, including language, planning, and reasoning. This part-based structure facilitates generalization, as parts can be indepen- dently interpreted, recombined, and reused across different contexts. In contrast, most contemporary neural networks lack the ability to form and manipulate such structured, symbol-like entities.
In 3D generation, diffusion-based generative models have shown strong capabilities in synthesizing entire 3D object meshes from scratch from images [1, 2, 3]. However, these models typically operate at the whole-object level and do not support part-level decomposition. This limitation restricts their applicability in downstream tasks such as texture mapping, animation, physical simulation, and scene editing, Some recent efforts have attempted to address this by first segmenting images into semantic parts and then reconstructing each part in 3D [4, 5, 6, 7J. However, this two-stage pipeline suffers from errors in segmentation, extensive computational costs for extra segmentation models, and difficulties in scaling up, thereby limiting both robustness and fidelity.
We introduce PART a structured generative model for 3D scenes that enables part-level generation from a single RGB image through a compositional latent space. PARTCRAFTER jointly generates multiple distinct 3D parts by binding each to a dedicated set of latent variables. This disentanglement allows parts be independently edited, removed, added without disrupting the rest of the scene. PARTCRAFTER builds upon large-scale pretrained 3D object latent diffusion models, which represent object meshes as sets of latent tokens [8] aligned either explicitly [3] or implicitly [1, 2] to regions in 3D space. PARTCRAFTER restructures these pretrained models into a compositional architecture equipped with varying number of latent token sets. It guides each latent token set during the denoising process to associate with a particular 3D part entity, through a novel local-global attention mechanism that facilitates both intra-part and inter- part information flow, in an identity-awe awwrre and permutation invarrian way. A shared decoder then maps each latent set into a coherent 3D mesh. We initialize the model's encoder, decoder, and denoising transformer using weights pretrained on whole-object mesh generation tasks [1]. When conditioned on single RGB image, the model produces structured 3D outputs with coherent pert-level decomposition, eliminating the need for brittle
pipelines.
To support training, we construct a large-scale dataset by mining part-level annotations from existing 3D object repositories. Many assets in datasets like Objaverse [9] contain part information in their GLTF metadata, as they are often authored using modular components. Rather than flattening these into single meshes, as done in prior work, we retain their part annotations. Our curated dataset merges Objaverse [9]. ShapeNet [10], and the Amazon Berkeley Objects (ABO) dataset [11], resulting in a rich collection of part-annotated 3D models suitable for learning compositional generation. As for scene-level generation, we leverage the existing 3D scene dataset 3D-Front [12] for training.
We evaluate PARTCRAFTER on both 3D part-level object generation and 3D scene reconstruction, and compare it against existing two-stage methods [6. 7] that first segment the input image and then reconstruct each segment. Our results show that PARTCRAFTER achieves higher generation quality and better efficiency. As shown in our experimental section, PARTCRAFTER invisible 3D structures from single image. It can be equally well used at the object or scene level, which makes a universal model for 3D scene reconstruction. Notably, PARTCRAFTER surpasses its underlying 3D object generative model on object reconstruction fidelity, showing that understanding the compositional structure of objects enhances the quality of 3D generation.
can automatically infer
In summary, our contributions are as follows:
We propose PARTCRAFTER, structured 3D) generation model with explicit pert-level binding, capable of generating semantically meaningful 3D) components to compose an object or a scene from image prompts, without any segmentation input.
We design a novel compositional DiT architecture with structured latent spaces, identity- aware part-global attention, and shared decoders.
A new dataset with part annotations is curated from existing 3D assets.
PARTCRAFTER achieves -ate-of-thhe-rt performance on structured 3D generation tasks, demonstrating strong results on both individual objects and complex multi-object scenes. Comprehensive ablation studies validate the contribution of each component in our approach.

---

2
Related Work
3D Object Generation Previous works on 3D object generation have adopted various representa-
tions, including voxels [13, 14, 15], point clouds [16, 17, 18], signed distance fields (SDFs) [19, 20], neural radiance fields (NeRFs) [21, 22. 23, 24, 25I, and 3D Gaussian splitting (3DGS) [26, 27, 28]. We focus on 3D meshes for their compatibility with real-world 3D content creation pipelines, One line of mesh generation works produces explicit mesh structures autoregressively [29, 30, 31, 32, 33, 34]. The other line of 3D mesh generation methods uses latent diffusion models [3, 8, 35, 36, 37], and has demonstrated strong performance, particularly when training high-capacity diffusion models across targe-scale datasets [1, 2, 38, 39, 40]. PARTCRAFTER builds upon the latter line of 3D mesh generative models. However, these methods typically generate 3D objects as holistic entities, neglecting the natural part-based decomposition that exists in 3D object datasets and is commonly employed by human artists during creation. PARTCRAFTER addresses this limitation.
3D Part-level Object Generation 3D part-level object generation is a long-standing problem in computer vision. One line of work [41, 42] has focused on assembling existing parts by inferring the right translations and orientations. Another line of work, which is more related our work, generates 3D geometry of parts and their structure to form a complete object. Different primitives have been used as representation, such as point clouds [43], voxels [44], implicit neural fields [45, 46, 47, 48], and other parametric representations [49, 50, 51J. These works are limited by either the scale of datasets [10, 52] or the expressiveness of the representation. To improve the visual quality and utilize prior knowledge of 2D pretrained models, several recent works [4. 5] propose to leverage multi-view diffusion and 2D segmentation models [53] to generate NeRF [21] or NeuS [54] with part labels. Concurrent work HoloPart [6] proposes first segment the 3D object mesh into parts and then refine the parts' geometry with a pretrained 3D DIT [1]. Although effective, these methods are heavily dependent the segmentation quality and are difficult to scale up. In contrast, our method can generate structured 3D part-level objects in one shot without any 2D or 3D segmentation.
3D Scene Generation Prior work on 3D object-composed scene generation typically extracts abstract representations such layouts [55, 56], graphs 58], or segmentations [59, 60, 61, 62, 63, 64] to model object relationships, and then generates [65, 66, 67] or retrieves [68, 68] objects based on these representations and input conditions. The closest work to ours is MIDI [7], which generates compositional 3D scenes by segmenting the input image [69] and prompting object-level 3D diffusion models with region segments. However, MIDI requires all components to be visible in the image, hindering its ability to segment and reconstruct occluded parts, which is a shared limitation across methods. PARTCRAFTER is inspired by MIDI but overcomes
all two-stage aegment-and-reconstunn this limitation. Without any additional segmentation model, PARTCRAFTER is capable of generating parts even when they are not visible in the conditioning image prompt. PARTCRAFTER demonstrates that explicit segmentation is not a necessary prerequisite for structured 3D scene generation.
3 PARTCRAFTER! A Compositional 3D Diffusion Model
The architecture of PARTCRAFTER is illustrated in Figure 2. It is a compositional generative model that generates structured 3D assets by simultaneously denoising several part-specified latent token sets. Specifically, given prompt RGB image e and a user-specified number of parts N. PARTCRAFTER generates a structured 3D asset O = [ e.g., part-decomposable object or a 3D scene composed of object parts, where each part p; is separate, semantically meaningful component of the asset. 3D meshes are utilized as the output representation. At inference time, all part meshes P = [V.FF)) F.} are generated and decoded simultaneously. where V; E IR. V.x is the vertex set and F NFx33 is the face set of the i-th part. Notably, coordinates of each part's vertices are in the same global canonical space [-1, 1/3 allowing for easy assembly of the parts into a complete object or scene, without any need for additional transformations.
PARTCRAFTER builds upon a pretrained 3D object mesh generation model [1] by utilizing and reassembling its encoders, decoders, and DiT blocks into a compositional multi-entity generation architecture, We provide description of object-level 3D generation models in Section 3.1, followed by detailed presentation of our proposed structured multi-entity model in Section 3.2.
3

---

Ieport trange
Patchily
R/v
Encoder
Part: decomposable
Monolithic object
GL (W))
3DVAE
10
Noise Latents
Gioial Apterion Ameetton
local
3D Native DIT
Figure 2: Architecture of PARTCRAFTER, Our model utilizes local-global attention to capture both part-level and global features. Part ID embeddings and incorporation of the image condition into both local and global features ensure the independence and semantic coherence of the generated parts.
3.1 Preliminary: Diffusion Transformer for 3D Object Mesh Generation
Pretrained 3D object generation models have shown impressive performance in generating high- quality 3D objects. Our model specifically builds on top of TripoSG [1], a state-of-the-art 3D image-to-mesh generative model. TripoSG first encodes 3D shapes into set of latent vectors with transformer-basee Variational Autoencoder (VAE) using 3DShape2VecsSe VecSet [8]. The decoder of the VAE uses Signed Distance Function (SDF) representation, which enables sharper geometry and avoids aliasing. A rectified flow model trained on the VAE latents to generate new 3D shapes from noise. The model is conditioned on single input image using DINOv2 [70] features, injected via cross-attention every transformer block. It is trained on million high-quality 3D shapes curated from Objaverse [9, 71] and ShapeNet [10] through rigorous four-stag stur-stage pipeline involving quality scoring, filtering out noisy data, mesh repair, and conversion to a watertight SDF-compatible format.
3.2 PARTCRAFTER
The design of PARTCRAFTER is motivated by our observation that foundation 3D latent diffusion models such as TripoSG [1], although trained on object-level training examples, can be applied effectively out-of-the-box to auto-encode 3D parts as well, without normalizing the input 3D mesh to the center of the canonical space (as done for 3D object meshes). We thus adapt the neural components of TripoSG build part-decomposable 3D asset reconstruction model.
Compositional Latent Space The key insight of 3D object mesh generative models with a set of latents is that each token in the latent space of the pretrained 3D VAE [1, 2, 8] is implicitly associated with an area of the canonical 3D space. PARTCRAFTER expands the latent space of monolithic object generative models with multiple sets of latents, each taking care of separate 3D part. Specifically, an object or a scene represented as a set of parts O { and each part p; is represented by set of latent tokens z; = {zig } E IR K XC where is the number of tokens in the component's set and C' is the dimension of each token. To distinguish across parts, add learnable part identity embedding e; C to the tokens each part p;. These embeddings are randomly initialized and optimized during training. To support the inherent permutation invariance of the parts, shuffle the order of the parts in training. Since z; represents the features of the i-th part, we construct the global 3D asset tokens simply by concatenating the tokens of all parts, i.c., Z {z 1.W E XXX
Local-Global Denoising Transformer We fuse information across sets latent tokens to enable both local-level and global-level reasoning. We apply local attention independently the tokens of each part. This captures localized features within each part, ensuring that their internal structure remains distinct. After capturing part-level features, apply global attention over the entire set of

---

3D Assets Statistics
3D Assets with Part Information
Part Count
Figure 3: Dataset Overview. Large-scale 3D object datasets [9, 10, 11] often contain rich part annotations. The pie chart and bin chart visualize the distribution an object's part count.
tokens E to model global interactions across parts. The attention operations are defined as:
A, local softmax
2
E RRXX
A global softmax
zz" ERNKNNK (I)
where A denotes the resulting attention maps. This hierarchical attention structure captures both fine-gre grained part details and global 3D context, enabling effective information exchange between local and holistic representations. We adopt the DiT architecture with long skip connections [35, 72, 73], as in TripoSG [1], and replace its original attention module with our part-global attention mechanism.
We inject DINOv2 [70] features of the condition image c into both levels of attention. Specifically, we use cross-attention within both the local and global attention. This dual-conditioning design enables the model to align the overall part-based composition with the input image while ensuring that each part remains semantically meaningful. Our design choices are validated in Section 4.3.
Training Objective We train PARTCRAFTER by rectified flow matching [74, 75, 76, 77], which maps the noisy Gaussian distribution to the data distribution in linear trajectory. Specifically, given an object latent Zo ] and condition image c, we perturb the latents by adding Gaussian
noise (O, I) noise level i, yielding noisy latent representation Z, ( 1Z + (1
The model is then trained to predict the velocity term e Zo given the noisy latent Z, at the noise level I, conditioned on the condition image by minimizing the following objective:
Low Eg,,,t [[( Za) vo (Z,, (2,,t,c)ll)
(2)
where vo is the velocity prediction. Importantly, the noise level t shared across all parts of the 3D scene object ensure consistent trajectory sampling.
3.3 Dataset Curation
3D object datasets such Objaverse [9] and Objaverse-XL [71] have made millions of 3D assets available to the research community. While previous works [1, 3, 38) have primarily focused on directly utilizing these datasets [9, 10, 11, 52, 71, 78] for 3D whole-object generation,
observe that the rich part-level metadata included in many of these 3D models offers an opportunity to enable more structured 3D generation. As shown Figure 3, over half of the objects in a subset [79] of Objaverse [9] contain explicit part annotations. These part labels often originate from artists' workflows, where objects intentionally decomposed into semantically meaningful components to facilitate modular design, such as modeling a pair of scissors using two distinct blades. To train our model, we curate dataset by combining multiple sources [9, 10, 11, 79], yielding 130,000 3D objects, of which 100,000 contain multiple parts. We further refine this dataset by filtering based on texture quality, part count, and average part-level Intersection over Union (IoU) to ensure high-quality supervision. The resulting dataset comprises approximately 50,000 lartela objects and 300,000 individual parts. For 3D scenes, we utilize the existing 3D object-composed scene dataset 3D-Front [12]. Additional dataset statistics and filtering criteria are detailed in Appendix A.
5

---

3.4 Implementation Details
We modify the 21 DiT blocks in TripoSG [1] by alternating their original attention processors with our proposed local-global attention mechanism. Specifically, global-level attention is applied to DiT blocks with even indices, while local-level attention is used in the odd-indexed blocks, following the long-cut strategy. We validate this architectural design in Section 4.3. PARTCRAFTER CRATER is trained on H20 GPUs with a batch size of 256 by fully finetuning the pretrained TripoSG [1]. We first train base model for up to 8 parts on our curated part-level object dataset at learning rate of 1e-4 for 5K iterations. For part-decomposable objects, we then finetune the base model to support up to 16 parts. For object-composed scenes, we further adapt the base model to the 3D-Front [12] dataset for up to 8 objects. Both finetuning processes last for 5K iterations at a reduced learning rate of 5e-5. We include 30% monolithic objects in training for regularization. This curriculum training strategy avoids loss spikes and catastrophic forgetting during the training process. The whole training process takes about 2 days. We use 512 tokens for each part, which we find is sufficient to represent part geometry and semantics. We evaluate PARTCRAFTER on a test set of about 2K data samples.
Experiments
Our experiments aim to answer the following questions: (1) How does PARTCRAFTER perform in part-level reconstruction of objects and scenes compared to existing state-of-the-art models that first segment and then reconstruct parts at the object and scene level? (2) Can PARTCRAFTER reconstruct parts that are not visible in the image prompt? (3) How do results vary with different numbers of parts? (4) What are the contributions of design choices in our local-glt local-global denoising transformer?
Baselines To the best of our knowledge, PARTCRAFTER is the first work to generate 3D part-level object meshes from a single image. Recent works Part123 [4] and PartGen [5] reconstruct 3D neural fields [21, 54] from images, which are not directly comparable to our work that focuses on meshes. We consider the following baselines: (1) HoloPart [6] on object level, which is a concurrent work that first segments given 3D object mesh and then completes the segment-e parts into fine-grained meshes. We adapt HoloPart to our setting by utilizing TripoSG [1] to generate a mesh from an image, and apply HoloPart get the part meshes. For fair comparison, we align the number of tokens in TripoSG with our method, that is, N x 512 tokens for N parts. (2) MIDI [7] on scene level, which reconstructs multi-instance 3D scenes using object segmentation prompts. We provide ground truth segmentation masks for MIDI, while PARTCRAFTER does not need any segmentation.
Evaluation Metrics We measure the generation result of structured 3D assets in both the global (object or scene) and part level. (1) Fidelity of the generated mesh. Since we do not have the correspondence between the generated and ground truth parts, we simply concatenate the parts to form a single mesh. We evaluate the fidelity of generated 3D meshes by L2 Chamfer Distance (CD) and F-Score with threshold of 0.1. Lower Chamfer Distance and higher F-Score indicate higher similarity between the generated and ground truth meshes. (2) Geometry Independence of Generated Part Meshes. We use the Average Intersection over Union (IoU) to evaluate the geometry independence of generated part meshes. We compute the average IoU between each generated part by voxelizing the canonical space into 64 64 64 grids. Lower IoU indicates less overlap between generated parts, thus demonstrating better part independence. The optimal metrics are reached when generated parts are non-intersecting and can be composed into an object similar to the ground truth. We report the average generation time of objects or scenes with 4 parts in the test set on an H20 GPU.
4.1 3D Part-level Object Generation
We evaluate the performance of PARTCRAFTER on the 3D part-level object generation task. As shown in Table 1, PARTCRAFTER outperforms HoloPart by a large margin in both object-level and pert-level metrics. Given an image, PARTCRAFTER is able to generate a 3D pert-decomposable mesh with high fidelity and geometry independence in seconds. HoloPart requires substantially more time to segment the object mesh and its segmentation process suffers due to the lower geometry quality of the generated mesh compared to real artistic meshes, which hinders its performance. Notably, PARTCRAFTER surpasses our backbone model TripoSG [1] in object-level metrics, even when we align the number of tokens in TripoSG with our method. This suggests that our local- global attention mechanism enables better object-level representation learning, thereby enhancing

---

Table 1: Evaluation on 3D Part-level Object Generation. We report evaluation result on Obja-
verse [9], ShapeNet [10], and ABO [11]. We denote TripoSG* [1] as the number-of-token of-token-aligned
backbone model. Higher F-Score and lower CD, IoU indicate better results. Best results are bolded.
3D Part-level
Objaverse [9]
ShapeNet [10]
ABO [11]
Time
Generation
J. CD
1 F-Score
ToU
J. CD
1 F-Score
4 JoU
CCD
F-Score
J. ToU
Dataset
0.0796
0.1827
/
0.0137
TripoSG [1]
0.3104
0,5940
/
0.3751
0,5050
0.2017
0.7096
TripoSG* [1]
0.1821
0.7115
/
0.3301
0.5589
0.1503
0.7723
HoloPart [6]
0.1916
0.6916
0.0443
0.3511
0.5498
0.1
0.1338
0.8093
0.0449
18min
Ours
0.1726
0.7472
0.0359
0.3205
0.5668
0.0293
0.1047
0.8617
0.0243
34s
Table Evaluation on 3D Object-Composed Scene Generation. We report evaluation results on
3D-From [12], as well as on challenging subset of 3D-Front [12] characterized by severe occlusions.
3D-Front [12]
3D-Front (Occluded) [12]
3D Scene Generation
+. Run Time
CD
F-Score
loU
CD
F-Score
| loU
MIDI [7]
0.1602
0.7931
0.0013
0.2591
0,6618
0.0020
80s
Ours
0.1491
0.8148
0.0034
0.1508
0.7800
0.0035
34s
the generation process through improved structural understanding. We also observe that PARTCRAFTER performs worse on ShapeNet [10] than on Objaverse [9] and ABO [1]], primarily due to the backbone's degraded performance on ShapeNet. We present qualitative results in Figure show that PARTCRAFTER infer parts invisible in the conditioning image and generate 3D
part-level objects. We further provide 3D object texture generation results in Appendix D.
4.2 3D Object-Composed Scene Generation
We conduct experiments 3D generation using the 3D-Front [12] dataset. As shown in Table PARTCRAFTER outperforms MIDI [7] reconstruction fidelity metrics. While MIDI [7] uses ground truth segmentation masks for evaluation, PARTCRAFTER does not require any segmentation. To further validate our method, we select a subset of 3D scenes in 3D-Front [12] with severe occlusion, where the ground truth segmentation masks cannot segment all objects. We observe that the performance of MIDI [7] degrades significantly in these cases, while PARTCRAFTER still maintains a high level of generation quality. MIDI [7] slightly surpasses PARTCRAFTER in IoU metrics, primarily due to the fact that ground truth 2D segmentation masks are used in their pipeline. Qualitative results are provided in Figure 5 demonstrate that PARTCRAFTER can recover complex 3D structures and generate high-quality meshes for 3D scenes, all from a single image.
For additional qualitative results of our method, please check Appendix E and the supplementary file.
4.3 Ablations
We present quantitative results in Table to validate the effectiveness of each component in our method. All experimental settings consistent with our base model, using maximum of 8 parts and 512 tokens per part, trained for 5K iterations with a learning rate of Ic-4 and a batch size of 256.
Necessity of Local-Global Attention To assess the necessity of local-global attention, we conduct ablation experiments by removing either the local attention or the global attention. We observe that the model completely collapses without local attention and fails to generate any meaningful 3D mesh (Exp. 2). In contrast, removing global attention still allows the model to produce 3D meshes, but the outputs lack part decomposition, exhibit significant geometry overlap, and result in notably higher IoU values (Exp. 3). These findings demonstrate that local-global attention essential for learning meaningful 3D representations and capturing the 3D structural relationships globally.
7

---

Input
HoloPart
Ours
Input
HoloPart
Ours
Figure 4: Qualitative Results on 3D Part-Level Object Generation. We present visualization results of HoloPart [6] and our method. Different colors indicate different parts of generated objects.
Input
MIDI
Ours
Input
MIDI
Ours
Figure 5: Qualitative Results 3D Scene Generation. We present visualization results of MIDI [7] and our method. Different colors indicate different objects the generated 3D scene.
Necessity of Part Identity Control We remove part identity embeddings from attentions, and model collapses due to ambiguity in distinguishing between different parts (Exp. 1). We further explore how to incorporate control signals into the local-global attention (Exp. 4). Enabling cross- attention in the local attention improves mesh fidelity but reduces geometry independence, while enabling it in the global attention enhances geometry independence at the cost fidelity. Based on these observations, adopt cross-attention in both local and global attention modules to strike balance between fidelity and independence, achieving the best overall performance.
Order of Local-Global Attention Since our DiT follows a U-Net-like architecture with long skip connections, we investigate how the ordering of local-global attention affects performance (Exp. 5). Across the 21 DiT blocks, approximately balance the number of local and global attention modules. explore three different configurations for placing global-level attention: (1) in the middle, (2) at the beginning and end, and (3) in alternating pattern. Among these, the alternating configuration

---

Table 3: Ablation Study for PARTCRAFTER, We report evaluation results on Objaverse [9] dataset.
Exp- ID
Part Emb
Sclf Local-Attm
Self Global-Attn
Cross Local-Attn
Cross Global-Attn
Global- Attn Order
CD
F-Score
loU
Middle
0.3143
0.4978
0.1401
#1
Middle
0.1711
0.7374
0.0814
X
/
Middle
0.4632
0.2327
0.0541
#2
Middle
0.1711
0.7374
0.0814
X
Middle
0.2606
0.5978
0.1602
M3
Middle
0.1711
0.7374
0.0814
Middle
0.1744
0.7334
0.0869
#4
Middle
0.1847
0.7188
0.0824
Middle
0.1711
0.7374
0.0814
Middle
0.1711
0.7374
0.0814
#5
Sides
0.1901
0.7114
0.0336
Alternating
0.1781
0.7212
0.0518
Ours
Alternating
0.1781
0.7212
Input
Part
Parts
Parts
Parts
Parts
Parts
Parts
8 Parts
AIII
Figure 6: Qualitative Results on Different Number Parts. We show that PARTCRAFTER can generate reasonable results with different granularities of part decomposition given the same image.
yields the best trade-off between local-level and global-level metrics, suggesting that this arrangement enables more effective information exchange and integration across both levels.
Specified Number of Parts We present qualitative results Figure to show that PARTCRAFTER can generate reasonable results with a varied number of parts given the same image. It validates that PARTCRAFTER captures 3D features hierarchically and handles different granularities of part decomposition, which can be useful feature for downstream applications and commercial use.
5 Conclusion
In this work, we propose PARTCRAFTER, a novel 3D native structured generative model. Trained on our curated part-level 3D mesh dataset, PARTCRAFTER reconstructs part-level objects and scenes without relying on any 2D or 3D segmentation information that existing models need. PARTCRAFTER validates the feasibility of integrating 3D structural understanding into the generative process.
Limitations and Future Works PARTCRAFTER is trained on 50K part-level data, which is rela- tively small compared to that used to train 3D object generation models (typically millions.. Future works can consider scaling up DiT training with more data of higher quality.
Broader Impact We conducted a foundational study in 3D computer vision. While our large- scale training may have environmental implications, we believe the benefits advancing 3D vision justify further exploration. The importance of the problem studied is discussed in the Introduction.

---

Acknowledgment
This research is supported by grant from Bytedance PICO (No. C1202406071057993)
References
[1] Yangguang Li, Zi- Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, et al. Triposg: figh-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv preprint arXiv:2502 200,22502.06608,, 2025.
[2] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG), 2024.
[3] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01 24...4//200000 506, 2024.
[4] Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, and Wenping Wang. Part123: part-aware 3d reconstruction from a single-view image. In ACM SIGGRAPH 2024 Conference Papers, 2024.
[5] Minghao Chen, Roman Shapovalov, Iro Laina, Tom Monnier, Jianyuan Wang, David Novotny, and Andrea Vedaldi. Partgen: Part-level 3d generation and reconstruction with multi-view diffusion models. arXiv preprint arXiv:2412./8800 2024.
[6] Yunhan Yang, Yuan-Chen Guo, Yukun Huang. Zi-Xin Zou, Zhipeng Yu, Yangguang Li, Yan- Pei Cao, and Xihui Liu. Holopart: Generative 3d part amodal segmentation. arXiv preprint arXiv:2504.0m 07943, 2025.
[7] Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang. Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. Midi: Multi-instance diffusion for single image to 3d scene generation. arXiv preprint arXiv:2412.03558, 2024.
[8] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG), 2023.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
[11] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022.
[12] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[13] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European Conference on Computer Vision (ECCV), 2016
[14] Aditya Sanghi, Rao Fu, Vivian Liu, Karl DD Willis, Hooman Shayani, Amir Khasahmadi, Srinath Sridhar, and Daniel Ritchie Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
10

---

[15] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point- voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[17] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeie, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[18) Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint 1rX1v:2212.08751, 2022.
[19) Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[20] Yen-Chi Cheng, Hsin-Ying Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Represent- ing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[22] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[23] Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. In International Conference on Learning Representations (ICLR), 2024.
[24] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-vies -pews gener- ation and large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
[25] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
[26] Bowen Zhang, Yiji Cheng, Jiaolong Yang. Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube; A structured and explicit radiance representation for 3d generative modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[27] Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, and Tong He. Gvgen: Text-to-3d generation with volumetric representation. In European Conference on Computer Vision (ECCV), 2024.
[28] Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, and Yadong Mu. Diffsplat: Re- purposing image diffusion models for scalable gaussian splat generation. arXiv preprint arXiv:2501. 16764, 2025.
[29] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

---

[30] Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, al. Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406. (00r:2206./0133,, 2024.
[31] Jiaxiang Tang, Zhaoshuo Li, Zekun Hao, Xian Liu, Gang Zeng, Ming- Yu Liu, and Qinsheng Zhang. Edgerunner: Augo-regressive auto-encoder for artistic mesh generation. arXiv preprint arXiv:2409. /8/14, 2024.
[32] Zhengyi Wang, Jonathan Lorraine, Yikai Wang, Hang Su, Jun Zhu, Sanja Fidler, and Xiaohui Zeng. Llama-mesh: Unifying 3d mesh generation with language models. arXiv preprint arXiv:2411.09595 2024.
[33] Ruowen Zhao, Junliang Ye, Zhengyi Wang, Guangee Liu, Yiwen Chen, Yikai Wang, and Jun Zhu. Deepmesh: Auto-regressive artist-mesh creation with reinforcement learning. arXiv preprint arXiv:2503.15265 2025.
[34] Xianglong He, Junyi Chen, Di Huang. Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, and Yangguang Li. Meshcraft: Exploring efficient and controllable mesh generation with flow-based dits. arXiv preprint arXiv:2503.2300 23022,, 2025.
[35] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape- image-text aligned latent representation. Advances in neural information processing systems, 2023.
[36] Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, and Yao Yao. Direct3d: Scalable mmage-to-3d generation via 3d latent diffusion transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[37] Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng, and Ping Tan. Dora: Sampling and benchmarking for 3d shape variational auto-encoders. arXiv preprini 24X2::2412.17800 2024.
[38] Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng. Mingxin Yang, Sheng Zhang, Xianghui Yang, al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501.122000 2025.
[39] Xianglong He, Zi-Xin Zou, Chia-Hao Chen, Yuan-Chen Guo, Ding Liang, Chun Yuan, Wanli Ouyang, Yan-Pei Cao, and Yangguang Li. Sparseflex: High-resolution and arbitrary-topology 3d shape modeling, arXiv preprint 2rXiv:2503.21732 2025.
[40] Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han. Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging. arXiv preprint arXiv:2503. 22236, 2025.
[41] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems, 2020.
[42] Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[43] George Kiyohiro Nakayama, Mikaela Angelina Uy. Jiahui Huang, Shi-Min Hu, Ke Li, and Leonidas Guibas. Difffacto: Controllable part-based 3d point cloud generation with cross diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023.
[44] Zhijie Wa, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Sagnet: Structure-aware generative network for 3d-shape modeling. ACM Transactions on Graphics (TOG), 2019.
[45] Dmitry Petrov, Matheus Gadelha, Radomír Méch and Evangelos Kalogerakis. Anise: Assembly- based neural implicit surface reconstruction. IEEE Transactions on Visualization and Computer Graphics, 2023.
12

---

[46] Connor Lin, Niloy Mitra, Gordon Wetzstein, Leonidas J Guibas, and Paul Guerrero. Neuform: Adaptive overfitting for neural shape editing. Advances in Neural Information Processing Systems, 35:15217-15229, 2022.
[47] Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. Salad: Part-level latent diffu- sion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2023.
[48] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. arXiv preprint arXiv: 00575, 2019.
[49] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7 7154-7164, 2019.
[50] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[51] Jingwen Ye, Yuze He, Yanning Zhou, Yiqin Zhu, Kaiwen Xiao, Yong-Jin Liu, Wei Yang, and Xiao Han. Primitiveanything: Human-crafted 3d primitive assembly generation with auto-regressive transformer. arXiv preprint MXXi::2505.04220 2025.
[52] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918, 2019.
[53] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafsonn, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision. 2023.
[54] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv: 106. 106.1999, 2021.
[55] Chenguo Lin and Yadong Mu. Instructscene: dustruction-driven 3d indoor scene synthesis with semantic graph prior. arXiv preprint arXiv:240..04117, 2024.
[56] Chenguo Lin, Yuchen Lin, Panwang Pan, Xuanyang Zhang, and Yadong Mu. Instructlay- out: Instruction-drrive 2d and 3d layout synthesis with semantic graph prior. arXiv preprint arXiv:2407.07588. 2024.
[57] Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, 2024.
[58] Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang, Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Componerf: Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv preprint arXiv:2303. 13843, 2023.
[59] Tao Chu, Pan Zhang, Qiong Liu, and Jiaqi Wang. Buol: A bottom-up framework with occupancy- aware lifting for panoptic 3d scene reconstruction from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[60] Daoyi Gao, Dávid Rozenberszki, Stefan Leutenegger, and Angela Dai. Diffcad: Weakly- supervised probabilistic cad model retrieval and alignment from rgb image. ACM Transac- tions on Graphics (TOG), 2024.
[61] Can Gümeli, Angela Dai, and Matthias Nießner. Roca: Robust cad model retrieval and alignment from single image. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022.
13

---

[62] Weicheng Kuo, Anelia Angelova, Tsung- Yi Lin, and Angela Dai. Mask2cad: 3d shape prediction by learning to segment and retrieve. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, 2020.
[63] Haonan Han, Rui Yang, Huan Liao, Jiankai Xing, Zunnan Xu, Xiaoming Yu, Junwei Zha, Xiu Li, and Wanhua Li. Reparo: Compositional 3d assets generation with differentiable 3d layout alignment. arXiv preprint arXiv:2405. 18525, 2024.
[64] Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Zero-shot scene reconstruction from single images with deep prior assembly. arXiv preprint arXiv:2410. 159711 2024.
[65] Manuel Dahnert, Ji Hou, Matthias Nießner, and Angela Dai. Panoptic 3d scene reconstruction from a single rgb image. Advances in Neural Information Processing Systems, 2021.
[66] Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Combo- verse: Compositional 3d assets creation using spatially-aware diffusion guidance. In European Conference Computer Vision, 2024.
[67] Andreea Dogaru, Mert Özer, and Bernhard Egger. Generalizable 3d scene reconstruction via divide and conquer from single view. arXiv preprint (ixiv:2444403422.. 2024.
[68] Hamid Izadinia, Qi Shan, and Steven M Seitz. Im2cad. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[69] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- worldworld models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024.
[70] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 07X1v:230..07193, 2023.
[71] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[72] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
[73] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[74] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for nigh-resolution image synthesis. In International Conference on Machine Learning (ICMI.), 2024.
[75] Michael S Albergo and Eric Vanden-Eijnden Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209. /5571, 2022.
[76] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03003, 2022.
[77] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arX1v:2210.02747, 2022.
[78] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions Graphics (ToG), 2016.
14

---

[79] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. European Conference Computer Vision (ECCV), 2024.
[80] Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y Lam, Yan-Pei Cao, and Xihui Liu. Sampart3d: Segment any part in 3d objects. arXiv preprint arXiv:2411.77184, 2024.
[81] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10901-10911, 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses the limitations of the work performed by the authors in Section
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code and data will be made available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
soniiiiscsycc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpp.cccppbblll/// http:ccccpppccccccccceeccccccccc htpppullccuuuddeessssepppppepsel guides/ /CodeSumii ssi ssisssoolooooooooo oonooiiyyyyloooooooomoooomiiimlp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The paper does not report information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ptttps://neurr ccc/pblic//att etthiisuueeelii ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ /0tp:/pp..../ooteeeeeee///////// for what should or should not be described.
22

---

A Dataset Details
We collect our part-level dataset from Objaverse [9] (ODC-By v1.0 License), ShapeNet-Core (Custom License), and Amazon Berkeley Objects [11] (CC-BY 4.0 License). We use high-quality subset of Objaverse provided by LGM [79]. We filtered out objects without textures and selected those with fewer than 16 parts and a maximum IoU of less than 0.1. The resulting dataset comprises approximately 50,000 part-labeled objects and 300,000 individual parts. We adopt an additional 30,000 monolithic objects as regularization. We use the 3D-Front (CC-BY-NC 4.0 License) dataset processed by MIDI [7]. We will release our dataset under a CC-BY 4.0 license.
B Implementation Details
Our model builds TripoSG [1] (MIT License). As for part-level object generation, adapt HoloPart [6] (MIT License) into a generative pipeline. Specifically, we first generate a 3D mesh from the input image using TripoSG and then use HoloPart generate a part-level object. We use the same 3D segmentation model, SAMPart3D [80] (MIT License), as Holopart. As for 3D scene generation, adopt MIDI [7] (MIT License) baseline. We will release our code under MIT license.
C Real-world Results
We train PARTCRAFTER on synthetic rendered images, which allows us to have a large and diverse dataset. To test the boundary of our model's generalization ability, we test PARTCRAFTER on real- world images from the CO3D [81] dataset. As expected, the performance was lower compared to synthetic images due to the domain gap. To address this, we explored transferring the style of real- world images to make them look more like images rendered from a graphics engine using recent image editing models, such as GPT-4o. Specifically, we use the prompt: "Preserve all details and perform image-to-imuge style transfer to convert the image inlo the style of 3D rendering (Objaverse-styl rendering)." " We find that our model performs significantly better in the style-trane style-transeeredd images. Therefore, propose two-stage pipeline for real-world image inference: first, an image editing model to transfer the style of the input image, and then use PARTCRAFTER to generate the 3D part-level object. As shown in Figure 7, this simple pipeline works surprisingly well.
D Texture Generation
As shown in Figure 8, we further generate textures for the generated 3D part-level objects using an off-the-shelf texture generation model Hunyuan3D-2 [38] (Hunyuan3D-2 License). Although Hunyuan3D-2 is trained on monolithic objects, it performs well on part-level objects. Since we know which part each vertex belongs to, we can manipulate the generated UV map accordingly. assigning textures to their corresponding parts. Therefore, our method is capable of generating 3D objects composed of multiple distinct parts with respective textures, all from a single image. Compared to previous methods that generate shape and texture for monolithic objects, our method produces part-aware 3D objects with distinct textures for each component. Because prior approaches treat the object single whole, textures often suffer from artifacts such as color bleeding between parts, and the resulting shapes may lack physically plausible part-level structure. contrast, our method ensures more accurate texture assignment and generates 3D structures that better reflect the compositional nature real-world objects. Thanks to these features, we believe PARTCRA holds great potential for downstream applications such as real-to-simulatio transfer and robotics training, where accurate and part-aware 3D representations with textures are crucial.
E
More Results
We provide more generation results on 3D part-level object in Figure 9, 10, 11, and 12 and 3D object-composed scene in Figure 13. For better visualization results, please see the supplementary files and our project website: https: h/tp:://ggggm//ee github. h/ppoo/ecta/pprtccattee.
23

---

Input
RMBG
RMBG-Result
Stylized
Stylized Result
Figure 7: Qualitative Results on 3D Part-Level Object Generation from Real-War World Images. We use GPT-4o to transfer the style of real-world images to make them look like images rendered from graphics engine. Then, we use PARTCRAFTER to generate 3D part-level objects.
Input
Ours
Textured
Input
Ours
Textured
Figure 8: Qualitative Results on 3D Textured Part-Level Object Generation.
24

---

Input
Ours
Figure 9: More results of image- image-conditioned 3D part-level object generation of PARTCRRATER.
25

---

Input
Ours
Figure 10: More results image-conditioned 3D part-level object generation of PARTCRAFTER.
26

---

Input
Ours
Figure 11: More results Congs--oondiiionce 3D part-level object generation of PARTCRAFTER.
27

---

Input
Ours
Figure 12: More results image-conditioned 3D part-level object generation of PARTCRAFTER.
28

---

Input
Ours
Figure 13: More results image-conditioned 3D scene generation of PARTCRRATERR
29


Reviews:
[{"review_id": "45U0m1MPHO", "reviewer": "Reviewer_JUky", "summary": "This paper proposes a framework to generate 3D meshes with parts using diffusion models directly. The model encodes different parts of the objects into different sets of latent vectors, learning an image-conditional diffusion model in the latent space. The diffusion model utilizes local attention within parts and global attention across parts. The paper shows applications in generating part-level meshes with image condition and 3D object-level scene generation.", "questions": "1)For training samples that have fewer or more parts than the hyper-parameter N, how do you deal with it?\n\n2)How do you control the number of parts in inference? For example, for a model trained with Z = {z_i} ∈ ℝ^{NK×C}, how do you generate fewer than N parts in inference? What does the shape of Z look like in that case? \n\n3)Is the model able to generate parts inside the objects?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1)The experiments ablate different components of the framework. \n\n2)The generated shapes have high quality, and the details of the shapes are well preserved. \n\n3)Inference speed is fast. \n\nCons:\n\n1)There is not enough description about the details of the VAE. What do the encoder and decoder look like? \n\n2)Missing details of the model design, for example, \n- what’s the value of C (dimensions of the tokens)? \n- How many vertices and faces are generated? \n- Are you using a different number of vertices/faces per shape? How is that controlled in inference?", "was_revised": true, "final_justification": "The missing details are added by the authors.", "rebuttal": "Thank you for your feedback. We are encouraged to know that you recognize the high quality of generation results and the fast inference speed of our method. Below, we provide clarifications for the concerns raised. We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have. \n\n**1. Missing details of the VAE.**\n\nWe directly leverage the pretrained TripoSG[1] VAE and do not fine-tune it. \n\nGiven a 3D mesh, the VAE first densely samples ~204,800 surface points, each associated with local geometric features such as 3D coordinates and surface normals. From these, it subsamples a smaller set (512 or 2048) to serve as latent queries. Using a set of cross-attention layers, these queries attend to the full dense point set (used as keys and values), allowing the transformer encoder to fuse global shape context into the latent tokens. \n\nThe decoder processes these tokens through self-attention, then predicts a truncated signed distance function (TSDF) for query positions via cross-attention. Then, meshes are extracted using Marching Cubes from the decoded TSDF. The VAE uses multi-resolution training, sharing weights across architectures that use varying number of point latents. This enables extrapolation to a higher number of latents at inference time. \n\nThe VAE’s loss function includes SDF loss, surface normal loss (using ground-truth normals for fine details), eikonal regularization, and KL regularization. \n\n**2. Missing details of the model design.**\n- The value of $C$ (the token channel dimension). \n    \n    Following the TripoSG VAE, we set $C$ to $64$. \n\n- How many vertices and faces are generated?\n\n    The number of vertices and faces generated depends on the resolution used when applying the Marching Cubes algorithm to extract the mesh from the predicted TSDF. Higher resolution settings will result in more vertices and faces, capturing finer geometric details, while lower resolutions will produce fewer, leading to a coarser mesh.\n\n- How do we determine the number of generated vertices and faces? \n\n    The specific number isn't fixed but is determined by the user-specified resolution parameter during the mesh extraction step. This allows flexibility to balance between computational efficiency (fewer vertices/faces) and geometric fidelity (more vertices/faces) based on application needs.\n\nFor more details, please refer to the original paper of TripoSG[1].\n\n**3. How to deal with a different number of parts during training?**\n\nWe're sorry for the confusion. **The number of parts is not a fixed hyperparameter of our model but rather a variable that can change per datapoint.** Thanks to the use of attention layers, the model can handle varying numbers of parts in the form of a varying number of latent sets in its input. In each training step, the model processes the input object with its actual number of annotated parts, allowing it to learn to handle different configurations. In other words, we train a single model that can handle a varying number of parts per data point both during training and inference.\n\n**4. How do we control the number of parts during inference?**\n\nDuring inference, the number of parts can be varied by controlling the number of latent sets $N$ in the model. Each latent set corresponds to a part, so adjusting $N$ allows users to specify how many parts they want the model to generate. \n\n**5. Is the model able to generate parts inside the objects?**\n\n**Yes, that model can generate parts inside objects.** For example, in the first example in the \"Image to 3D Part-Level Object Generation\" section of the supplementary website (a LEGO character), the model generates slot-like structures that are inside the object. These internal components are clearly visible in the provided split-view video, which shows the internal details alongside the external structure, demonstrating the model's capability to generate both exterior and interior parts of objects. \n\n\nWe will add all the aforementioned discussions to the revision of this manuscript. \n\n---\n[1] Li et al., TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models, arXiv 2025"}, {"review_id": "RrYY2doJvM", "reviewer": "Reviewer_ynhN", "summary": "This paper proposes a method to unify the architecture for generating decomposable meshes conditioned on an image. Concretely, the authors propose a one-stage pipeline to generate part mesh instead of following a first segment, then repair the parts paradigm. The method is based on the diffusion network, and the proposed compositional latent space to enable global and local attention for information within and across parts. The author also curated a part-based dataset from open-source datasets, as claimed to be part of the contribution. The authors also compare both single mesh and scene-based part generation with SoTA methods like HoloPart.", "questions": "--The part dataset is constructed by parsing the node, how to make sure the node grouping is meaningful?\n\n--In Figure 6, it shows the model can generate variable length of parts, is the training also randomly select the number of parts, or does it learn to adapt to the number of parts?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n--The work adopts TripoSG’s 3D VAE representation, which associates input point cloud locations to a set of latents; thus, the parts can use a shared canonical space instead of renormalizing into a canonical space separately. This representation allows the model to generate parts with transformation baked in and does not need to predict additional transformation information for each part.\n\n--The adaptation of global and local attention to model global part interaction and per-part details is novel. The author also provides a detailed ablation study on the importance of using dual attention.\n\nWeakness:\n--The author adapts the TripoSG VAE representation and found that even when the VAE is trained on a single mesh, it can still reconstruct part-based meshes. This is a strong assumption, but there is no validation in the experiment to prove it. It would be great to show the reconstruction accuracy for parts with TripoSG VAE to validate the assumption.\n\n--The current pipeline cannot control the number of parts and the semantic labels, which are important in practice.", "was_revised": true, "final_justification": "The author's response addresses some of my concerns, and this work is promising for the future part-based 3D generative model; hence, I decided to increase my original rating.", "rebuttal": "We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged that you recognize the novelty of our proposed method, including the adaptation of the pretrained VAE and the local-global attention mechanism. Below, we provide clarifications for the concerns raised. We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.\n\n**1. Validation of the TripoSG VAE's ability to reconstruct parts.**\n\nThank you for pointing this out! Per your suggestion, we have conducted additional experiments to validate the TripoSG VAE's ability to reconstruct parts while being trained on objects. We calculate the Chamfer Distance (CD) and F-Score between the reconstructed mesh and the ground truth mesh. Lower CD and higher F-Score indicate better reconstruction quality. We conduct evaluations on both the whole-object level and the part level. The following table shows the results:\n\n| Level   |   CD $\\downarrow$ |   F-Score $\\uparrow$ |\n|---------|-------------------|----------------------|\n| Object  |            0.0153 |               0.9824 |\n| Part    |            0.0413 |               0.9398 |\n\nThe CD and F-Score on the part level are only slightly worse than those on the whole-object level. We will add this experiment in the main paper. It's fair since we do not fine-tune the pretrained VAE, and the VAE is trained on whole objects. The results indicate that the TripoSG VAE can effectively reconstruct parts, achieving a close performance comparable to the whole object.\n\n**2. Control of the number of parts and the semantic labels during inference.**\n\nThe number of parts can be controlled by adjusting the number of latent sets $N$ in the model. Each latent set corresponds to a part, so adjusting $N$ allows users to specify how many parts they want the model to generate. \n\nAbout the semantic labels, the current version of our model does not support explicit semantic control during inference. We have found that a two-stage pipeline could be used to achieve this, where the first stage generates a semantic label map for the image, and the second stage uses this label map in place of the image prompt to guide the 3D part mesh generation. We've tried this approach with GPT-4o’s image editing function for the image label map generation, and it works well. We will add the results of this two-stage pipeline to the revision of this manuscript since we cannot update the PDF during the rebuttal phase. \n\n**3. How to ensure the part decomposition is meaningful in the dataset?**\n\nWe build our part-level dataset by using existing part annotations in Objaverse. Intuitively, artists construct 3D models using semantically meaningful parts (e.g., a human figure with arms, hair, torso) for modular design. This ensures part decompositions align with intuitive, real-world structures and their semantic components. \n\n**4. Is the training randomly selecting the number of parts, or does it learn to adapt to the number of parts?** \n\nWhile the model does not explicitly predict the number of parts, it is trained with objects that contain different numbers of parts. The number of parts is not a fixed hyperparameter of our model but rather a variable that can change per datapoint. Thanks to the use of attention layers, the model can handle varying numbers of parts in the form of a varying number of latent sets in its input. In each training step, the model processes the input object with its actual number of annotated parts, allowing it to learn to handle different configurations. In other words, **we train a single model that can handle a varying number of parts per data point both during training and inference.**\n\nDuring inference, the number of parts can be controlled by controlling the number of latent sets $N$ in the model. Each latent set corresponds to a part, so adjusting $N$ allows users to specify how many parts they want the model to generate. \n\nWe will add all the aforementioned discussions to the revision of this manuscript."}, {"review_id": "77LqoGZfTk", "reviewer": "Reviewer_BRws", "summary": "This paper proposes the first part-aware native 3D generator. The model builds upon a pre-trained whole-object generator and introduces two key innovations: a compositional latent space and a local-global attention mechanism. In addition, the authors curate a 3D dataset with part annotations to facilitate training. Experimental results demonstrate the model's superiority over traditional two-stage pipelines in terms of both efficiency and part segmentation quality.", "questions": "- How does the number of parts affect the model’s performance in terms of both generation quality and computational cost? A more thorough analysis would be helpful.\n\n- Could you provide more details on how part annotations were mined from Objaverse? Given that its file structure is artist-defined and hierarchical, how was this handled to ensure consistency?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper presents a novel approach to part-aware 3D generation. Unlike conventional two-stage pipelines, the proposed method generates part-decomposable objects in an end-to-end fashion. This design enhances both efficiency and segmentation quality.\n\n- The experimental evaluation is comprehensive, covering both object- and scene-level domains. The results show significant improvements in part segmentation quality compared to prior methods.\n\n- The paper is well-organized and clearly written, making it easy to follow.\n\n## Weaknesses\n\n- The paper lacks a detailed analysis of how the number of parts affects both time cost and generation quality. For example, in Figure 6 (Lego figurine), the hand shapes deteriorate in the 2-part and 4-part settings, indicating instability.\n\n- The model’s ability to handle occluded or invisible parts at the object level remains uncertain. For instance, in Figure 2, the left arm in the conditioning image is partially occluded by fire, and the generated 3D asset fails to reconstruct a complete left arm.\n\n- The paper lacks failure cases.\n\n- The choice of evaluation metric could be improved. While IoU is used to evaluate part-level quality, it may not capture semantic correctness. For example, parts with low IoU might be semantically meaningless.", "was_revised": true, "final_justification": "This is a novel method for part-aware 3D object generation. I believe it will serve as a strong baseline for future work and be beneficial to the community. Therefore, I am inclined to recommend acceptance.", "rebuttal": "Thank you for your insightful and valuable feedback. We are encouraged to know that you recognize the novelty of our proposed method, including the design of the one-stage end-to-end pipeline, the compositional latent space, and the local-global attention mechanism. Below, we provide clarifications for the concerns raised. We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.\n\n**1. How the number of parts affects both time cost and generation quality?**\n- Computational cost. \n\n    We conducted experiments to analyze the impact of the number of parts on computational cost. As shown in the table below, the time cost increases with the number of parts. The inference time is still far lower than that of HoloPart (about 18 minutes), even with 16 parts, demonstrating the efficiency of our method. \n\n    | Number of Parts | Inference Time (s) $\\downarrow$ | \n    |-----------------|---------------------|\n    | 1               | 11.67                |\n    | 2               | 18.91                |\n    | 4               | 32.99                |\n    | 8               | 69.43                |\n    | 16              | 131.1                |\n\n- Generation quality.\n\n    The generation quality depends on many factors, including the complexity of the input image, the number of parts, the number of tokens, the denoising steps, and the classifier-free guidance scale. \n    \n    As shown in Figure 6, the quality roughly remains consistent across different numbers of parts, with some variations in detail and structure. Intuitively, an input image should have a range of suitable numbers of parts for optimal generation quality. If the number of parts is too small, the model may not be able to capture the full complexity of the object, leading to lower quality generation. Conversely, if the number of parts is too large, it may introduce unnecessary complexity and noise, also degrading quality. \n    \n    We have found that the number of tokens and denoising steps has a more significant impact on generation quality than the number of parts. In the submission version, we set the number of tokens to 512 and the denoising steps to 50. Since submission, we have scaled the model to 1024 tokens and 100 denoising steps, which improves the generation quality and geometry details, such as the hands and fingers of the generated human characters. The following table shows the evaluation results on the Objaverse dataset with the scaled-up model: \n\n    | Model | CD $\\downarrow$ | F-Score $\\uparrow$ | IoU $\\downarrow$ |\n    |-------|-----------------|--------------------|------------------|\n    | Before Scaling | 0.1726          | 0.7472             | 0.0359           |\n    | After Scaling  | **0.1360**          | **0.7942**             | **0.0315**           |\n    \n- Incomplete arm mesh in Figure 2\n\n    Thank you for pointing this out. We conducted further experiments and identified two contributing factors: (1) a large classifier-free guidance (CFG) scale, which causes the model to overfit to the input image, and (2) an insufficient number of tokens and denoising steps, which limits the refinement of the generated mesh.\n    This issue does not occur in the scaled-up version of our model, as mentioned earlier. \n\nWe will include a discussion of these findings in the revised manuscript, as we are unable to update the PDF during the rebuttal phase. \n\n\n**2. The model’s ability to handle occluded or invisible parts at the object level.**\n\n**The model can generate occluded, invisible or fully internal to the object parts.** In the car example of Figure 4, not all wheels are visible in the input image, yet the model successfully generates all four wheels. In the first example in the \"Image to 3D Part-Level Object Generation\" section of the supplementary website (a LEGO character), the model generates slot-like structures that are inside the object.  This demonstrates that the model can infer and generate parts that are not directly visible in the input image, leveraging its understanding of object structure and context. \n\nWe are working on benchmarking visible versus internal versus occluded part 3D reconstructions, and we will add those in the revised manuscript per your request.\n\n**3. Failure cases.**\n\nThank you for pointing this out! In our experiments, the failure cases are mainly due to the following three reasons. (1) Unsatisfactory geometry details, such as nonsmooth surfaces or floating parts. This is often caused by insufficient denoising steps or too small number of tokens. (2) Unsatisfactory part decomposition, such as generating parts that are not semantically meaningful. This is often caused by the input image not providing enough information for the model to infer the correct part decomposition, such as an input image of a rock with no clear part structure. (3) Out-of-distribution images, such as real-world images, can also lead to failure cases. Currently, we train our model on synthetic rendered images, which allows us to have a large and diverse dataset. We've conducted experiments on real-world images. The results are not as good as those on synthetic images due to the domain gap. We have solved this issue by doing style transfer through automated image editing in order to transfer the real-world images into in-domain 3D-rendering style images. We find this strategy works well. \n\nWe will add visualizations of all failure cases to our project page and the revision of this manuscript, since we cannot update the PDF during the rebuttal phase. \n\n**4. The choice of the IoU metric for evaluating the geometric independence of generated part meshes.** \n\nThank you for your valuable feedback. We agree that each metric has its limitations. To ensure a comprehensive evaluation, we employ multiple metrics beyond IoU. As shown in Table 1, we report performance using Chamfer Distance and F-score, in line with established practices in part-based 3D reconstruction, such as Holopart and MIDI. We would be happy to include any additional metrics you may suggest.\n\n**5. Details on how part annotations were mined from Objaverse? How was this handled to ensure consistency?**\n\nWe simply use `parts = mesh.dump()` where `mesh` is a `trimesh.Scene` object. The `dump()` method in trimesh is used to decompose a `trimesh.Scene` object into multiple `trimesh.Trimesh` objects, each representing a part of the original mesh. Internally, a `Scene` consists of multiple geometries, each with its own transformation (position, rotation, scale) and potentially corresponding to a semantically meaningful component of the overall object. When `dump()` is called, it iterates through all geometries in the scene, applies their transformations to bring them into world coordinates, and returns each one as an individual mesh. \n\nIntuitively, artists construct 3D models using semantically meaningful parts (e.g., a human figure with arms, hair, and torso) for modular design, which ensures part decompositions align with intuitive, real-world divisions. We do not perform consistency checks on the part annotations, as humans may have different interpretations of what constitutes a part. Thus, we let the model learn from the existing structure in Objaverse, which results in more diverse, realistic, and human-like part decompositions.\n\nWe will add all the aforementioned discussions to the revision of this manuscript."}, {"review_id": "aSjhsVJqA7", "reviewer": "Reviewer_bQqp", "summary": "This paper proposed a method to generate 3D objects from single view image with part-level decomposition. There are two key innovations: (1) A compositional latent space where each 3D part is represented by a set of disentangled latent tokens; (2) A hierarchical attention mechanism that enables structured information flow both within individual parts and across all parts. The experiment shows the good performance of the proposed method.", "questions": "1. The paper only presented good cases. What would be the border of the model capability?\n2. Please explain in more details the evaluation process for Table 1. Specifically, are there inner structures in the GT mesh, and does this incurs unfairness that make TripoSG have disadvantage in metrics.", "limitations": "No obvious limitation.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method show promising results\n2. The ablation is comprehensive. \n\nWeakness:\n1. concerns about data leak: \n1.1 the method is initialized with TripoSG, which utilized many data sources for training their model including objaverse and objaverse xl, so that even the author made a test split from their filtered data, the objects are highly likely have been part of the training set of its pretrained backbone model. \n1.2 Weapon, axe, chairs, sword are really very abundant in the objaverse dataset, and many of such objects are just the same whitebox mesh with different textures, so that it is not safe and convincing to demonstrate the effectiveness with these objects.\n\n2. Fairness of evaluation: The paper reported 3D Part-level Object Generation result on Objaverse (Table 1). It includes TripoSG while TripoSG was trained on Objaverse. Therefore, it is weird if the method can perform better than TripoSG*. However, the author explains in the paper that the evaluation was performed on part-level objects, which implies that the GT objects all have inner (occluded) structures. However, TripoSG was trained on watertight objects with no inner structure. This may present a significant unfairness in evaluation.\n\n3. It would be good if a pipeline figure can be self-contained. However, it is not clean what does Part-decomposable object and Monolithic object means and what is their purpose if we only read the figure and its caption.\n\n4. insufficient evaluation: following weakness 1, the presented visual results are mainly cartoon-style, game-style, or simple, static, convex and common real-world objects. This may be insufficient to demonstrate that the proposed method is truly effective on real use cases. Usually, many method claimed good performance fails if being tested out of the above scoop. To demonstrate that the proposed method is really effectiveness, the author should present more examples, including 1) images captured in real-world, 2) objects with some dynamics (e.g. a realistic-style image about a dog running to the camera); 3) objects with complex spatial inter-part relationship.", "was_revised": true, "final_justification": "The authors addressed my concerns comprehensively. I lean to accept this paper.", "rebuttal": "We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the innovation of our proposed compositional latent space and the local-global attention mechanism. Below, we provide clarifications for the concerns raised. We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.\n\n**1. Data Leakage between the train and test sets.**\n\n- The method is initialized with TripoSG, so that even if the author made a test split from their filtered data, the objects are highly likely to have been part of the training set of its pretrained backbone model. \n\n   We sincerely thank you for your insightful observation. TripoSG[1] is trained on Objaverse, Objaverse-XL, and ShapeNet. In the manuscript, we have provided results on ABO (object level) and 3D-Front (scene level). We show our method works well on these datasets, which are not part of the training set of TripoSG. We will only focus on these evaluations since they are not contaminated. Meanwhile, our baselines (HoloPart and MIDI) are also fine-tuned from TripoSG, so comparing with them is still fair. \n\n- Weapons, axes, chairs, and swords are really abundant in the objaverse dataset, so it is not safe and convincing to demonstrate the effectiveness with these objects.\n\n    We agree that these objects are abundant in the Objaverse dataset. We've provided results on a wide range of objects, such as human characters, vehicles, and furniture, to demonstrate the effectiveness of our method in the Figures and Supplementary website. \n\n**2. It is weird if the method can perform better than the number-of-token-aligned TripoSG. Is the evaluation fair since TripoSG is trained on watertight objects with no inner structure?**\n\nThank you for raising this important point. We acknowledge that the presence of internal structures in the ground-truth objects affects the evaluation due to the symmetric nature of the Chamfer Distance. This oversight was on our part. To address this, we are currently re-evaluating our method against TripoSG* at the whole-object level, after removing internal structures from both the ground-truth meshes and the generated part-based meshes. This will ensure a fair comparison at the object level. The updated results will be included in the revised manuscript.\n\nWe would also like to clarify that our model is specifically designed for part-based generation. We reported the object-level scores to provide a point of reference against a widely recognized baseline. We appreciate your feedback, which has helped us identify and correct this evaluation issue. \n\n**3. In the pipeline figure, it is not clear what a Part-decomposable object and a Monolithic object mean, and what their purpose is if we only read the figure and its caption.** \n\nWe apologize for the confusion. In the pipeline figure, \"Part-decomposable object\" refers to objects that are composed of multiple distinct parts, and \"Monolithic object\" refers to objects that are not decomposed into parts. The purpose of this distinction is to highlight that our method can handle part-level generation compared to many previous monolithic generation methods. We'll add \"PartCrafter can generate structured 3D models with multiple semantic parts (Part-decomposable objects), in contrast to previous approaches that generate entire shapes as single entities (Monolithic objects)\" to the caption of the pipeline figure. \n\n**4. Insufficient evaluation.**\n- Real-world images.\n\n    Thank you for your suggestion! We agree that evaluating our model on real-world images offers a more comprehensive assessment of model performance. Currently, we train our model on synthetic rendered images, which allows us to have a large and diverse dataset. Since submission, we've tested our model on real-world images from the CO3D[2] dataset. As expected, the performance was lower compared to synthetic images due to the domain gap. \n\n    To address this, we explored transferring the style of real-world images to make them look more like images rendered from a graphics engine using recent image editing models, such as GPT-4o. Specifically, we used the prompt: “Preserve all details and perform image-to-image style transfer to convert the image into the style of a 3D rendering (Objaverse-style rendering).” We found that our model performed significantly better in the style transferred images. \n\n    We are working on further enhancing real-world performance by augmenting our training data with these style-transferred images. By leveraging image editing models (e.g., GPT-4o) to adapt the rendered images to real-looking ones while preserving fine-grained structural details, we aim to narrow the synthetic-to-real domain gap and improve the model’s generalization.\n\n    We will add these visualizations to our project page and the revision of this manuscript, since we cannot update the PDF during the rebuttal phase. \n\n- Objects with dynamics. \n\n    Currently, our method focuses on static objects, an assumption widely adopted in many 3D generation approaches. We acknowledge that dynamic objects lie beyond the scope of the current work, and we leave their exploration to future research.\n\n- Objects with complex spatial structure. \n\n    Since spatial structural complexity is difficult to define and quantify directly, we use the number of parts as a proxy to represent different levels of complexity. To this end, we conduct experiments on Objaverse to evaluate the model's performance under varying numbers of parts. The following table shows the results: \n\n    | Number of Parts | CD $\\downarrow$ | F-Score $\\uparrow$ | IoU $\\downarrow$ |\n    |-------------|--------|---------|---------|\n    |           2 | 0.1310 |  0.8037 |  0.0418 |\n    |           4 | 0.1476 |  0.7735 |  0.0249 |\n    |           6 | 0.1352 |  0.7924 |  0.0198 |\n    |           8 | 0.1401 |  0.7897 |  0.0227 |\n    |          10 | 0.1319 |  0.7908 |  0.0211 |\n    |          12 | 0.1366 |  0.7837 |  0.0200 |\n    |          14 | 0.1528 |  0.7756 |  0.0221 |\n    |          16 | 0.1417 |  0.7881 |  0.0198 |\n\n    From the table, we observe that the model's performance remains relatively stable across different numbers of parts, suggesting robustness to varying levels of spatial complexity.\n\n\n**5. The border of the model's capability.**\n\nWe analyze the border of the model's capability in two aspects: (1) the capability of the current model by failure case analysis and (2) the potential of the method by scaling up the model. \n\n- The capability of the current model. \n\n    In our experiments, the failure cases are mainly due to the following three reasons. (1) Unsatisfactory geometry details, such as nonsmooth surfaces or floating parts. This is often caused by insufficient denoising steps or too small number of tokens. (2) Unsatisfactory part decomposition, such as generating parts that are not semantically meaningful. This is often caused by the input image not providing enough information for the model to infer the correct part decomposition, such as an input image of a rock with no clear part structure. (3) Out-of-distribution images, such as real-world images, can also lead to failure cases. Currently, we train our model on synthetic rendered images, which allows us to have a large and diverse dataset. We've conducted experiments on real-world images. The results are not as good as those on synthetic images due to the domain gap. We have solved this issue by doing style transfer through automated image editing in order to transfer the real-world images into in-domain 3D-rendering style images. We find this strategy works well. \n\n    Despite these limitations, our method demonstrates strong performance on a wide range of objects, including human characters, vehicles, and furniture. We hope PartCrafter can serve as a solid foundation for future research in part-level 3D generation. And we will release the training, inference code, the model weights, and the part-level dataset to facilitate future research. \n\n- The potential of the method. \n\n    Due to the resource constraints and the time limit, the submission version of our method is trained on a subset of Objaverse without fine-tuning the pretrained VAE. We believe that the method has great potential to be scaled up. By using more training data, fine-tuning the pretrained VAE, and scaling up the model, we can achieve even better performance. \n\n    In the submission version, we set the number of tokens to 512 and the denoising steps to 50. Since submission, we have scaled the model to 1024 tokens and 100 denoising steps, which improves the generation quality and geometry details. The following table shows the evaluation results on the Objaverse dataset with the scaled-up model: \n\n    | Model | CD $\\downarrow$ | F-Score $\\uparrow$ | IoU $\\downarrow$ |\n    |-------|-----------------|--------------------|------------------|\n    | Before Scaling | 0.1726          | 0.7472             | 0.0359           |\n    | After Scaling  | **0.1360**          | **0.7942**             | **0.0315**           |\n\nWe will add all the aforementioned discussions to the revision of this manuscript. \n\n---\n[1] Li et al., TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models, arXiv 2025\n\n[2] Reizenstein et al., Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction, ICCV 2021"}, {"review_id": "9Es4zhovpg", "reviewer": "Reviewer_EYFc", "summary": "The paper introduces PartCrafter, a novel structured 3D generative model that creates semantically meaningful and geometrically distinct 3D parts from a single RGB image—without requiring pre-segmented inputs. This paper proposes a unified, end-to-end architecture based on a compositional latent space and hierarchical attention. It builds on a pretrained 3D diffusion transformer, enabling generation of coherent multi-part objects and scenes. A new dataset with part-level annotations is also curated to support training. Experiments show PartCrafter outperforms previous approaches in both single object and multi-object scenarios.", "questions": "I would recommend the authors to consider the following questions:\n\n1. During evaluation, how is the number of generated parts determined? Is this parameter fixed across all experiments when comparing with baseline methods, or does it vary depending on each individual object?\n\n2. The paper currently uses IoU metrics to measure the geometric independence of generated part meshes. However, HoloPart employs IoU to evaluate the semantic accuracy of generated parts by comparing them against ground truth parts. Could the authors consider incorporating similar semantic accuracy evaluation metrics to provide a more comprehensive assessment of part generation quality?\n\n3. Please provide additional discussion regarding the controllability of the model, particularly addressing how users can effectively control the generation process during inference.\n\n4. It would be valuable to include more extensive results demonstrating in-the-wild generation capabilities and part editing functionality to better assess the method's practical applicability.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The paper demonstrates that foundation 3D latent diffusion models (e.g., TripoSG) can effectively auto-encode 3D parts without requiring centered input meshes. This insight enables the introduction of a compositional latent space with multiple 3D part latents, allowing end-to-end generation of multi-part objects without pre-segmented inputs.\n- The authors contribute a valuable dataset with part-level annotations that will benefit future research in this domain.\n- The proposed method achieves superior fidelity in generated 3D parts compared to existing state-of-the-art approaches.\n\n**Weaknesses:**\n- While the end-to-end approach is innovative, the controllability of the model remains a concern. Although Figure 6 demonstrates the model's ability to vary the number of generated parts, it is unclear whether users must experiment with different part counts during inference to achieve desired results.\n- The semantic consistency of generated parts appears inconsistent across different objects. For instance, Figure 10 shows that ears are sometimes connected to the head and sometimes disconnected, indicating potential issues with part boundary consistency.\n- The paper lacks evaluation on in-the-wild images, limiting assessment of real-world applicability.\n- The paper does not address part editing capabilities, particularly for scenarios where users wish to modify non-isolated parts (e.g., editing a foot that is connected to a leg). This represents a significant practical limitation.", "was_revised": true, "final_justification": "Thanks the authors for their rebuttal. However, I still have concerns regarding the controllability of the proposed methods. As mentioned by the authors, users need to specify the number of parts they wish to create. Given that there is no explicit semantic guarantee with the proposed methods, users might need multiple attempts to achieve their desired results. Additionally, non-isolated parts cannot be directly edited. Therefore, I will maintain my original rating of borderline reject.", "rebuttal": "We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the insight that pretrained VAE can be adapted to reconstruct parts and the contribution of the part-level dataset. We are also glad that you find the generation results to be impressive compared to the baselines. Below, we provide clarifications for the concerns raised. We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.\n\n**1. The controllability of the model. It is unclear whether users must experiment with different part counts during inference to achieve desired results.** \n\nThe number of parts is explicitly controllable by setting the number of latent sets $N$ in the model, where each latent set corresponds to a distinct part. This allows users to directly specify the desired part count during inference. While an input image may have a range of suitable part counts, this flexibility is intentional: it enables users to generate objects with varying levels of detail and complexity, catering to different downstream needs such as coarse vs. fine-grained editing. Our qualitative results (Figure 6) demonstrate that the model produces reasonable decompositions across different $N$, validating this design as a feature rather than a limitation.\n\n**2. The semantic consistency of the generated parts.** \n\nIntuitively, the part decomposition of a 3D object can be varied, and the model can generate different part decompositions for the same object. This also corresponds to the fact that humans may have different interpretations of what constitutes a part. Users can generate part-level objects with different seeds to obtain different part decompositions and select the one that best fits their needs. \n\n**3. In-the-wild generation capability.**\n\nThank you for your suggestion! We agree that evaluating our model on real-world images offers a more comprehensive assessment of model performance. Currently, we train our model on synthetic rendered images, which allows us to have a large and diverse dataset. Since submission, we've tested our model on real-world images from the CO3D[1] dataset. As expected, the performance was lower compared to synthetic images due to the domain gap.\n\nTo address this, we explored transferring the style of real-world images to make them look more like images rendered from a graphics engine using recent image editing models, such as GPT-4o. Specifically, we used the prompt: “Preserve all details and perform image-to-image style transfer to convert the image into the style of a 3D rendering (Objaverse-style rendering).” We found that our model performed significantly better in the style-transferred images. \n\nWe are working on further enhancing real-world performance by augmenting our training data with these style-transferred images. By leveraging image editing models (e.g., GPT-4o) to adapt the rendered images to real-looking ones while preserving fine-grained structural details, we aim to narrow the synthetic-to-real domain gap and improve the model’s generalization.\n\nWe will add these visualizations to our project page and the revision of this manuscript, since we cannot update the PDF during the rebuttal phase. \n\n**4. Part editing functionality, particularly for scenarios where users wish to modify non-isolated parts.**\n\nWe appreciate your inquiry into the part editing functionality of our model. Our approach can naturally support transforming, substituting, and removing isolated parts. However, for non-isolated parts, the current model does not support direct editing. It may require first performing 2D image editing, such as editing a foot that is connected to a leg on the image, and then using the edited image to guide the part generation. \n\n**5. During evaluation, how is the number of generated parts determined?** \n\nWe simply use the ground truth number of parts for each object in the dataset. The number of parts is not fixed, but varies depending on each individual object. \n\n**6. Use IoU metrics to measure the semantic accuracy of generated parts.**\nThank you for the valuable suggestion. We've conducted additional experiments to evaluate the semantic accuracy using IoU metrics on Objaverse. Higher IoU indicates better semantic accuracy. The following table shows the results: \n| Model | IoU $\\uparrow$ |\n|-------|------------------|\n| HoloPart | 0.4359 |\n| PartCrafter | **0.5770** |\n\nThese results demonstrate that our method achieves better semantic accuracy than HoloPart, indicating the effectiveness of our part-level generation approach.\n\nWe will add all the aforementioned discussions to the revision of this manuscript. \n\n--- \n[1] Reizenstein et al., Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction, ICCV 2021"}]

Author rebuttal:

Meta-review: The paper presents an end-to-end method to generate meshes divided into parts, claimed to be the first of this kind. The idea is to rely on a compositional latent space that represents each 3D part as disentangled latent tokens, and a hierarchical local-global attention mechanism. Authors start from multiple publicly available datasets to curate a 3D dataset with parts annotations.

Before the discussion, the opinions on the paper were particularly borderline; there is a general appreciation for the results and comparison with the state of the art, but I also see there have been several suggestions for further analysis, especially about the number, the controllability, and the generality of the generated parts. 

After the discussion, opinions changed more toward acceptance. In summary, Reviewers ynhN, bQqp, and JUky are in favor of acceptance, while Reviewer BRws leans toward it: PartCrafter is recognized as the first end-to-end approach for part-based meshes generated from a single image, and the results have been found to be of high quality.
Instead, Reviewer EYFc , leans toward rejection. Their main concern lies in the controllability of the number of parts, which has to be manually selected by the user. They also have concerns about the editing of the results, though it is not directly the focus of the work.

The AC believes that prompting the user for inputting a part number is a substantial limitation, which might hamper the applicability of the method. The authors suggest this can be inferred by images, but the automatization of this might also not be trivial (e.g., estimation of internal parts), and it is completely left unexplored by the paper. On the other hand, the addressed task is relevant, the experiments provide sufficient evidence about the method's performance and the validation of the individual design components, and it is a fair first step, paving the way for future works. The curated dataset also provides a valuable contribution, instrumental in developing the research field. Hence, the AC leans toward acceptance.
Decision: Accept (poster)