DSPart: A Large-scale Diffusion-generated Synthetic Dataset with Annotations from 3D Parts

Authors: Jiawei Peng, Yining Sun, Ju He, Jieneng Chen, Prakhar Kaushik, Wufei Ma, Yi Zhang, Jiahao Wang, Angtian Wang, Xiaoding Yuan, Qihao Liu, Adam Kortylewski, Yaoyao Liu, Alan Yuille
Primary area: datasets and benchmarks
Keywords: Synthetic dataset, Semantic Part Segmentation, 3D Parts Annotation

Abstract:
Object parts provide representations that enable a detailed and interpretable understanding of object structures, making part recognition crucial for various real-world applications. However, acquiring pixel-level part annotations is both expensive and time-consuming. Rendering 3D object models with their 3D part annotations is a promising solution since it allows the generation of unlimited synthetic data samples with precise 3D control and accurate part segmentation masks. Nevertheless, these synthetic datasets suffer from a lack of realism, resulting in large domain gaps. In this paper, we present a large-scale realistic synthetic dataset with part annotations, namely Diffusion-generated Synthetic Parts (DSPart), for both rigid objects and animals. For images in DSPart, we obtain 2D part masks from 3D part annotations by leveraging recent advances in diffusion models with 3D control. In addition to offering more diverse and realistic textures, prior knowledge of diffusion models enables the object to exhibit more physically realistic interactions with the ground plane and other spatial contexts. We annotate $475$ representative shape instances from $50$ object categories for DSPart-Rigid and use $3,065$ high-quality SMAL models fitted poses from $40$ animal categories for DSPart-Animal. Experimental results demonstrate the potential of our dataset in training robust part segmentation models, effectively bridging the gap between synthetic and real-world data.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730336878209

thread_type:
official_review

<thread_object>
creator: reviewer_ytbv

<content>
summary:
This paper proposes two datasets of 3D object parts, DSPart-Rigid and DSPart-Animal. For DSPart-Rigid, it annotates 475 representative shape instances from 50 object categories. For DSPart-Animal, it creates one single 3D part annotation since all SMAL models share the same vertex IDs, and uses 3,065 SMAL models fitted poses from 40 animal categories. It uses a diffusion model, 3D-DST (Ma et al., 2024), to render the realistic images. To further filter out low-quality images for animals, this paper proposes a filter, Part-attention Regressor Filter, which estimates the accuracy of articulations. In total, this data sets contains 48K synthetic images of rigid objects and 52K synthetic images of animals. To evaluate the quality of the dataset, this paper trains the models on its datasets in both synthetic-only and synthetic-to-real scenarios to evaluate their performance on PartImageNet (He et al., 2022) and PascalPart (Chen et al., 2014).

strengths:
1. The proposed filter for animals, Part-attention Regressor Filter, significantly improves the quality of the generated data, as shown in Table 5.

weaknesses:
1. If we compare the results of the model trained only on real-world data and the model trained only on synthetic data, the difference is huge, especially for animals (Table 3 & 4). This might mean that the proposed dataset and dataset creation methods are still far behind the real-world hand data, and might not be scalable as described.
2. Although the proposed filter, PRF, takes a significant portion of the paper, it seems not used in producing the final datasets (see numbers in Table 3 "DSPart-Animal" and Table 5 "Human"). Therefore, it is questionable whether the method is scalable as claimed. I wish it could be further clarified.

questions:
1. In Table 5, it shows that the PRF-filtered result under the UDA setting is close to the human-filtered result, except background. Is there any insights why background is special?
2. It is interesting that, in Table 4 (Syn-only), the IoU of back legs and the tails are significantly worse than the previous synthesized dataset, CC-SSL. Is there any insights on that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730340348154

thread_type:
official_review

<thread_object>
creator: reviewer_hvmi

<content>
summary:
The paper introduces a synthetic dataset called DSPart, designed to improve part-based recognition models. DSPart includes synthetic images with part-level annotations derived from 3D models, specifically addressing two key groups: rigid objects (DSPart-Rigid) and animals (DSPart-Animal). With annotating 3.5K 3D models, it generates 100K realistic synthetic images with masks. Extensive experiments demonstrate that DSPart surpasses existing synthetic datasets in part segmentation tasks.

strengths:
1. The paper introduces an innovative approach that utilizes 3D part annotations to generate realistic synthetic images with corresponding part masks. This method enables the generation of an unlimited number of synthetic data samples using a limited set of annotated 3D shapes, significantly reducing the cost and time required for annotating large quantities of 2D images.
2. Sufficient experiments on the 2D part segmentation task to demonstrate the effectiveness of the DSPart dataset.
3. DSPart addresses limitations in previous synthetic datasets by using diffusion models to generate realistic textures and physically consistent contexts, making the synthetic images more reflective of real-world conditions.

weaknesses:
1. The method is challenging to generalize due to the difficulty in defining annotations for parts in other categories. For instance, providing semantic-level part definitions for data in the large-scale 3D dataset Objaverse[1] is particularly challenging. Is it able to consider a part definition approach similar to SAM?
2. Generating images with 2D part annotations using standard 3D parts can increase data volume; however, annotating 3D data is more challenging and requires annotators to possess advanced tool usage skills. This, in turn, limits the scalability of the data to some extent.
3. Each generated image contains a single, clear object positioned in the center. However, in real images, there may be multiple objects that are not necessarily centered or clearly defined. This indicates that some domain gaps still exist.

[1] Deitke M, Schwenk D, Salvador J, et al. Objaverse: A universe of annotated 3d objects[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 13142-13153.

questions:
1. Is it able to adjust poses of 3D shapes to increase the diversity of 3D data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532969504

thread_type:
official_review

<thread_object>
creator: reviewer_xgjx

<content>
summary:
This paper presents a new part-segmentation synthetic dataset via 2D diffusion models, the results show that it is possible to use diffusion models to generate more diverse data samples for part segmentation task.

strengths:
1. Show the potentional on using synthetic data for part segmentation.

weaknesses:
1. Lack of details on how to use diffusion models to generate data samples, which base model is used. Why not use a ControlNet to generate controllable data pairs.

2. The scale of the generated data is relatively small, which makes it hard to verify the generalization capability of the model when trained on  larger data scales.

questions:
1. Given the categories included in DSPart-Rigid and DSPart-Animal, to what extent is this dataset likely to be generalizable to other object categories or part segmentation tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694129389

thread_type:
official_review

<thread_object>
creator: reviewer_qefu

<content>
summary:
The paper introduces a framework (DSPart), including a filtering approach (PRF), for generating a large-scale synthetic dataset for 2D object part segmentation (DSPart-Rigid and DSPart-Animal). The results demonstrate that training on the generated dataset achieves superior performance compared to existing synthetic datasets (UDAPart and CC-SSL), in both supervised (SegFormer) and unsupervised (DAFormer) setups.

strengths:
1.	The paper is well-written and easy to follow.

2.	The paper presents an effective and easily applicable method (DSPart) for utilizing an existing generation approach (3D-DST) and existing 3D CAD models (e.g., ShapeNet and Objaverse), along with additional annotations (e.g., 3D parts), to generate pixel-level part-based datasets.

3.	The scale and diversity of the generated dataset (DSPart-Rigid and DSPart-Animal) are impressive, comprising 100k images, 90 object categories, and over 3.5k 3D models.

4.	Training on the generated dataset, whether or not UDA is applied, typically demonstrates superior results on real part-based datasets (e.g., PartImageNet) compared to training on existing synthetic datasets.

weaknesses:
Major:

1.	Despite its simplicity, the proposed framework (DSPart) shows limited novelty from a technical perspective. It appears more like a direct application of an existing framework (3D-DST), enhanced by incorporating additional annotations (e.g., 3D parts) and a wider variety of 3D models (e.g., animals). Although the paper claims that the extension is non-trivial (Line 107), it would be beneficial if the paper could justify this claim by highlighting its technical innovations.

2.	The paper demonstrates that the proposed filtering mechanism (PRF) outperforms the existing one (KCF). However, this comparison seems unfair and potentially lacks meaningful insight for several reasons. First, PRF trains models for filtering using reliable human-filtered data (e.g., 13k images) that KCF does not have access to, making it unsurprising that PRF performs better. Additionally, the paper does not quantify the contribution of each component within PRF, such as the use of two PARE models, three different metrics, and various thresholds. It is possible that PRF’s superior performance is simply due to its access to more reliable data. Finally, the paper does not provide visual examples to illustrate why PRF outperforms KCF. Including examples of data filtered by PRF but not by KCF, or showcasing PRF's failure cases, would add clarity and strengthen the explanation.

3.	Although the paper shows that using the DSPart dataset for training generally yields better performance than other existing synthetic datasets, it does not fully explain why the DSPart dataset is superior, which is essential for the community to understand the results. For example, as shown in Table 2, training models on DSPart-Rigid consistently outperforms training models on UDAPart. Is this advantage due to the realism of the generated images, the diversity of the 3D CAD models, the increased number of viewpoints, or the total number of generated images? Investigating different versions of DSPart-Rigid could be insightful, such as generating DSPart-Rigid with the same set of 3D CAD models used by UDAPart, varying the number of viewpoints, generating different numbers of images, or controlling texture diversity through varied prompts.

Minor:

4.	In Table 5, the paper shows that PRF is more efficient (20 minutes for inference) compared to KCF (50 training hours). However, since PRF requires human-filtered data for training 3D pose estimation models, it would be fairer to include this as part of PRF’s overall cost (e.g., 100 hours human labor). While the paper mentions that PRF only needs a pre-trained model and can be applied to the generated images at scale, this is true only if the human-filtered data used by PRF effectively captures the entire animal pose space, which may not be the case.

5.	The paper defines five super-categories (car, airplane, bicycle, boat, and tool) for rigid objects in the proposed dataset (DSPart-Rigid). However, the tool category is never used in the paper.

6.	The paper utilizes only a single architecture for both the fully supervised setup (SegFormer) and the UDA setup (DAFormer). Including additional architectures or UDA methods would strengthen some of the paper’s claims, such as the claim that DSPart-Rigid enables better knowledge transfer from the synthetic to the real domain for UDA (Lines 355-357).

questions:
1.	Please clarify the technical novelty of the proposed framework in response to Weaknesses-1. Highlighting the key advancements distinguishing 3D-DST from the proposed framework would be beneficial.

2.	Please provide additional details about PRF in response to Weaknesses-2 and Weaknesses-4. This may include explaining why PRF outperforming KCF is still meaningful, even if PRF has access to more reliable data, quantifying the contribution of each component within PRF, and providing visual examples.

3.	Please fully justify why the DSPart dataset is superior. As suggested in Weaknesses-3, exploring different versions of DSPart-Rigid could be insightful.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper presents DSPart, a dataset of Diffusion-generated Synthetic Parts, consisting of 100K images with part masks generated from 3D models.  Part annotations are provided on the 3D models and used to render part segmentations that corresponds to the realist images generated using 3D-DST [Ma et al 2024]. 

The DSPart dataset is composed of DSPart-Rigid (48K images generated from 475 rigid 3D models over 50 categories) and DSPart-Animal (52K images generated from 3,065 models from Animal3D that are created by taking the SMAL model and fitting it to 40 animal categories).  The 475 rigid 3D models are annotated with fine-grained parts. For DSPart-Animal, a single part annotation for the SMAL model is sufficient, but a filtering phase is needed to filter out low quality generated images. 

Experiments compare whether using the proposed dataset to train part segmentation network can improve performance.  Results show that combining the proposed dataset with existing training dataset can increase accuracy.    

**Contributions**
- The main contribution of the work is the proposed method of generating aligned natural images and pixel-level part segmentation using annotated 3D models and 3D-DST.  
- The work build on existing components, with the main novel component being the proposed filtering mechanism

**Strengths**
1. Paper is well-written and easy to follow [qEFu]
2. Proposed method is effective at providing pixel-level part segmentation (given part annotated 3D models) [qEFu,hvMi]
3. Experiments demonstrates that including the dataset in training can help improve part segmentation performance [qEFu,xgjX,hvMi] 
4. Using generated data for training part segmentators can potentially provide unlimited data [hvMi]
5. Some reviewers felt the scale and diversity of the generated dataset was sufficient [qEFu]

**Weaknesses**

Reviewers expressed the following concerns:
1. Concerns about the scalability of the approach
   - Some reviewers find the scale of generated data to be small [xgjX]
   - Generalizability to other categories unclear [xgjX]
   - The proposed method relies on having annotated parts on 3D models which may be hard to obtain [hvMi]
   - As annotating 3D parts is more challenging than annotating 2D parts, there is doubts about the scalability of the approach [hvMi]
   - Results showing the effectiveness of the DSPart data is with human filter (not the proposed filter mechanism) [yTbV]
2. Concerns that the generated images are not of sufficient quality as real-world images
   - Performance on model trained on real images is still much better than using synthetic images [yTbV]
   - The generated images do not reflect objects in context which are more natural [hvMi]
3. Other concerns:
   - Limited technical contribution, as the work is largely based on 3D-DST. [qEFu]   
   - Concerns about fairness of comparison of the proposed filtering mechanism [qEFu]
   - Lack of insight about what aspect of DSPart provides the most benefit to guide how to scale up the generation [qEFum,xgjX]

**Recommendation**

While reviewers appreciated the potential of using generated data to train better part segmentation models, most reviewers did not find the contributions to be sufficient for acceptance at ICLR (with 3 reviewers giving the paper a rating of 5, and one a rating of 6).  

The main concern is the scalability of the proposed approach, as it requires annotated 3D parts, and human filtering of images was used for the final dataset.  Due to the concerns expressed by the reviewers, the AC recommends reject.