SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GPT-4 (creative thinker): Here are 1000 examples for you:
1. A man in a sharp grey suit, crisp white shirt, and black polished shoes
strides confidently down a city street
2. A college student walks down the street wearing a casual outfit,
consisting of a cozy hoodie, slim-fit jeans, and a pair of trendy sneakers.
3. A young man in a casual outfit strolls down the sidewalk, wearing a
denim jacket over a graphic tee, paired with black jeans.
ControlNet (editor)
DatasetDM (annotator) :
Human Pose
Figure 4: Collaboration between GPT-4 and Diffusion Model. Large language models, e.g., GPT-4, can provide diverse and unrestricted prompts, enhancing the diversity of generated images.
Semantic Mask
Instance Mask
DeepFashion Mask
Grey suit  $\rightarrow$  White suit city street  $\rightarrow$  forest
Zero-Shot Segmentation	Long-Tail Segmentation
Seen Class	Unseen Class	Head Class	Tail Class
aeroplane (0), bicycle (1), bird (2), boat (3), bottle (4), bus (5), car (6), cat (7), chair (8), cow (9), diningtable (10), dog (11), horse (12), motorbike (13), person (14)	potted plant (15), sheep (16), sofa (17), train (18), tvmonitor (19)	aeroplane (0), bicycle (1), bird (2), boat (3), bottle (4), bus (5), car (6), cat (7), chair (8), cow (9)	diningtable (10), dog (11), horse (12), motorbike (13), person (14), potted plant (15), sheep (16), sofa (17), train (18), tvmonitor (19)
Table 1: Details for Zero-Shot and Long-tail Segmentation on VOC 2012 [15].
text prompts, which average around 100 prompts for each dataset. For each inference, a random prompt is sampled from this set.
Architecture and Training Details. Stable diffusion V1 [41] model pre-trained on the LAION5B [45] dataset is used as our text-to-image diffusion model. The decoder architecture of Mask2Former [8] was selected as the base architecture for our P-Decoder. And we use 100 queries for the segmentation task. For all tasks, we train DatasetDM for around  $50k$  iterations with images of size  $512 \times 512$ , which only need one Tesla V100 GPU, and lasted for approximately 20 hours. Optimizer [36] with a learning rate of 0.0001 is used.
Downstream Tasks Evaluation. To comprehensively evaluate the generative image of DatasetDM, we conduct seven groups of experiments for the supported six downstream tasks. Semantic Segmentation. Pascal-VOC 2012 [15] (20 classes) and Cityscapes [11] (19 classes), as two classical benchmarks are used to evaluate. We synthesized  $2k$  images for each class in both datasets, resulting in a total of  $40k$  and  $38k$  synthetic images for Pascal-VOC 2012 and Cityscapes, respectively. The synthetic data is subsequently utilized to train Mask2former [8] and compared to its real data counterpart on a limited dataset setting (around 100 images). Instance Segmentation. For the COCO2017 [33] benchmark, we synthesized  $1k$  images for each class, resulting in a total of  $80k$  synthetic images. Similarly, Mask2former [8], as the baseline, is used to evaluate the synthetic data. We evaluate only the class-agnostic performance, where all the 80 classes are assigned the same class ID. Depth Estimation. We synthesized a total of  $80k$  synthetic images for NYU Depth V2 [46]. And using Depthformer  $[30]^2$  to evaluate our synthetic data. Pose Estimation. We generated a set of  $30k$  synthetic images for COCO2017 Pose dataset [33] and employed HRNet [48] as the baseline model to assess the effectiveness of our approach. Zero-Shot Semantic Segmentation. Following Li et al. [31], Pascal-VOC 2012 [15] (20 classes) is used to evaluate. We train DatasetDM with only 15 seen categories, where each category including 30 real images, and synthesized a total of  $40k$  synthetic images for 20 categories. Long-tail Semantic Segmentation. The categories of VOC 2012 are divided into head (20 images each class) and tail classes (2 images each class). Then we train DatasetDM with these data and generate synthetic data. Human Semantic Segmentation. We synthesized a total
$^{2}$ https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox
Figure 5: Examples of generated data for DatasetDM. Our method can produce semantic/instance segmentation, depth, and human pose annotation across various domains.
method	VOC (Semantic Seg.)/%	COCO2017 (Instance Seg.)/%	NYU Depth V2 (Depth Est.)	COCO2017 (Pose Est.)/%
# real	# synth.	mIoU	# real	# synth.	AP	# real	# synth.	REL ↓	# real	# synth.	AP
Baseline	100	-	65.2	400	-	14.4	50	-	0.31	800	-	42.4
DatasetDM	100	40k	78.5	400	80k	26.5	50	35k	0.21	800	80k	47.5
Table 2: Downstream Tasks. 'real' and 'synth.' denote real and synthetic images, respectively. The backbones of baselines for four tasks are 'Swin-B', 'Swin-B', 'Swin-L', and 'HR-W32', respectively.
of  $20k$  synthetic images for DeepFashion-MM [27] (24 classes). Mask2former [8] is used to evaluate the synthetic data. We split DeepFashion-MM into a set of 100 training images, and 12,500 testing images. Further details, such as prompts, can be found in the supplementary material.
Class Split for Zero-Shot and Long-Tail Segmentation. Table 1 provides a comprehensive overview of the class distribution for both zero-shot and long-tail scenarios. The division for zero-shot classes is consistent with previous studies [5, 53, 31]. The configuration for long-tail data distribution is firstly established in this paper.
Table 2 provides a basic comparison of the selected four downstream tasks. More additional experiments can be found in Table 3 and Table 4, as well as in the Supplementary Material (i.e., Pose Estimation, Depth Estimation, Zero-Shot Segmentation, Fashion Segmentation, and others).
Semantic Segmentation. Table 4 displays the performance on VOC 2012. Using only 100 real images (5 images per class), training with purely synthetic data from DatasetDM achieves a  $73.7\%$  mIoU, an  $8.5\%$  improvement compared to using real data alone. Moreover, when jointly training with the 100 real images, further improvement is observed, resulting in a mIoU of  $78.5\%$ . Table 5 displays the performance on Cityscapes [11]. To compare with previous methods [55], we also conducted experiments with a 9-classes division. DatasetDM demonstrates consistent advancements over the baseline or prior SOTA, achieving up to a  $10\%$  improvement in mIoU under each experimental setup.
Instance Segmentation. Table 3 presents three distinct training settings, encompassing variations in the backbone and the number of training images. Regardless of the chosen setting, DatasetDM consistently achieves an improvement of approximately  $10\%$ . Employing 800 training images (10 images per class) and the Swin-B backbone, DatasetDM yields a  $12.1\%$  increase in Average Precision (AP), resulting in a final AP of  $26.5\%$ .
Depth Estimation. Table 2 presents a concise comparison between synthetic and real data on the NYU Depth V2 dataset [46]. Detailed information (e.g., backbone, other metrics) can be found in the supplementary material. When trained with 50 images, DatasetDM can achieve a  $10\%$  improvement compared to training solely with real images.
method	backbone	# real image	# synthetic image	AP	\( AP^{50} \)	\( AP^{75} \)	\( APS \)	\( AP^M \)	\( AP^L \)
Baseline	R50	400	-	4.4	9.5	3.5	1.1	3.3	12.1
DatasetDM	R50	-	80k (R:400)	12.2	24.3	10.9	1.6	11.3	30.9
DatasetDM	R50	400	80k (R:400)	14.8	29.7	13.0	2.3	15.1	36.0
Baseline	Swin-B	400	-	11.3	23.0	9.6	3.2	10.1	27.1
DatasetDM	Swin-B	-	80k (R:400)	17.6	34.1	15.8	3.4	17.8	39.5
DatasetDM	Swin-B	400	80k (R:400)	23.3	43.0	22.2	7.7	26.1	48.7
Baseline	Swin-B	800	-	14.4	28.8	12.7	5.6	15.7	29.2
DatasetDM	Swin-B	800	80k (R:800)	26.5	46.9	25.8	7.7	29.8	53.3
Table 3: Instance segmentation on COCO val2017. 'R:' denotes the real data used to train.
method	backbone	# real image	# synthetic image	Sampled Classes for Comparison/%	mIoU
bird	cat	bus	car	dog
Baseline	R50	100	-	54.8	53.3	69.3	66.8	24.2	43.4
DatasetDM (ours)	R50	-	40k (R:100)	84.7	74.4	86.0	79.2	63.7	60.3
DatasetDM (ours)	R50	100	40k (R:100)	81.7	82.3	87.7	77.9	69.3	66.1
Baseline	Swin-B	100	-	54.4	68.3	86.5	71.8	49.1	65.2
DatasetDM (ours)	Swin-B	-	40k (R:100)	93.4	94.5	93.8	78.8	79.6	73.7
DatasetDM (ours)	Swin-B	100	100 (R:100)	83.9	71.0	82.9	78.0	39.5	67.9
DatasetDM (ours)	Swin-B	100	400 (R:100)	86.9	92.0	90.8	82.6	86.7	76.1
DatasetDM (ours)	Swin-B	100	40k (R:100)	86.7	93.8	92.3	88.3	87.1	78.5
Baseline	Swin-B	10.6k (full)	-	93.7	96.5	90.6	88.6	95.7	84.3
DatasetDM (ours)	Swin-B	10.6k (full)	40k (R:100)	93.9	97.6	91.9	89.4	96.1	85.4
Table 4: Semantic segmentation on VOC 2012. 'R' refers to the number of real data used to train.
Human Pose Estimation. For the human pose estimation task on the COCO2017 dataset, DatasetDM demonstrates significant improvements compared to the baseline trained on 800 real images, achieving a  $5.1\%$  increase, as illustrated in Table 2. Similar to depth estimation, additional information can be found in the supplementary material.
Zero Shot and Long-tail Segmentation. Table 7 displays the results of experiments related to zero-shot and long-tail segmentation. Our model, DatasetDM, notably alleviates the effects of long-tail distribution by synthesizing a substantial amount of data for rare classes, leading to an improvement of up to  $20\%$  in mIoU. Details for both tasks can be found in the supplementary material.
Method	Zero-Shot Setting	Long-tail Setting
seen	unseen	harm.	head	tail	mIoU
Baseline(no Syn.)	61.3	10.7	18.3	61.2	44.1	52.6
Li et al. [31]	62.8	50.0	55.7	-	-	-
DiffuMask [53]	71.4	65.0	68.1	-	-	-
DatasetDM	78.8	60.5	68.4	73.1	66.4	70.0
Table 7: Zero Shot and Long-tail Segmentation on VOC 2012. Zero Shot: following priors [31, 53], we train DatasetDM with only 15 seen categories, and tested for 20 categories. Long-tail Setting: the 20 categories are divided into head (10 classes, 20 images each class) and tail classes (10 classes, 2 images each class).
Figure 6: Ablation of Prompt Length.
Diffusion Time Steps. Table 6a depicts the influence of visual features derived from various diffusion time steps, and the maximum sampling step is 1000. We observe that a large step results in adverse outcomes, whereas the performance with a smaller step tends to remain relatively stable.
Cross-Attention Fusion. Fig. 6b demonstrates that the cross attention maps,  $\mathcal{F}$  in step 1 can yield a modest improvement, roughly  $1\%$ . Interestingly, it appears that as the step size increases, the benefit becomes less pronounced. Indeed, when the step size surpasses 500, it may even result in detrimental effects. From this perspective, the utility of the cross-attention map is limited.
Training Set Size. Additional training data for DatasetDM can further improve synthetic data, as shown in Table 6c. The increase of the training data from 60 to 400 images precipitates the most conspicuous improvement, subsequently reaching a saturation point. With 1k training images, the
method	backbone	# real image	# synthetic image	8 Classes/%	19 Classes/%
vehicle	mIoU	car	bus	bicycle	mIoU
Baseline	R50	9	100k+ (R:16)	84.3	71.5	82.8	22.3	42.4	36.8
HandsOff [55]	R101	16	100k+ (R:16)	-	55.1	-	-	-	-
HandsOff [55]	R101	50	100k+ (R:16)	-	60.4	-	-	-	-
DatasetDM (ours)	R50	-	38k (R:9)	86.9	69.5	83.3	8.3	53.5	34.2
DatasetDM (ours)	R50	9	38k (R:9)	88.6	76.7	85.6	28.9	56.5	42.1
DatasetDM (ours)	R101	9	38k (R:9)	88.9	77.5	85.9	27.9	60.4	43.7
Baseline	Swin-B	9	-	84.1	74.5	83.3	27.7	42.0	41.1
DatasetDM (ours)	Swin-B	-	38k (R:9)	85.7	73.3	84.3	20.3	29.1	37.3
DatasetDM (ours)	Swin-B	9	38k (R:9)	89.4	80.0	87.2	30.2	66.5	47.4
Table 5: Semantic segmentation on Cityscapes for two different split settings: 8 and 19 categories. 'vehicle', 'car', 'bus', and 'bicycle' are sampled classes for presentation.
step	car	dog	mIoU
1	88.3	87.1	78.5
100	88.1	87.2	78.5
200	88.0	86.6	78.3
500	87.2	84.9	76.8
800	86.3	83.4	76.1
(a) Visual Features  $\mathcal{F}$
step	car	dog	mIoU
-	88.1	87.0	77.6
1	88.3	87.1	78.5
200	87.7	87.0	78.0
500	87.2	86.3	77.5
800	87.1	86.1	77.1
(b) Cross Attention  $\hat{\mathcal{A}}$
# train im.	syn.	joint
60	71.4	77.1
100	73.7	78.5
200	74.4	79.4
400	76.4	80.4
1,000	78.4	81.1
(c) Size of Train Set.
prompt (# num.)	car	dog	mIoU
Human (100)	84.9	84.5	76.6
GPT-4 (100)	85.2	86.1	77.1
GPT-4 (200)	88.0	86.2	77.3
GPT-4 (500)	88.1	87.1	78.5
GPT-4 (1k)	88.3	87.1	78.5
(d) Prompt Candidates.
Table 6: DatasetDM Ablation on Pascal-VOC 2012 for semantic segmentation. Swin-B is used as the backbone. 100 real images are used for (a), (b), and (d). 'Syn.' and 'Joint' denote training with only synthetic data and joint training with real data, respectively.
performance escalates to an impressive  $81\%$ , demonstrating competitive prowess for the application. Notably, 1k training images representing roughly  $10\%$  of the original data, is still relatively diminutive.
Prompt from Language Model. Candidate Number. We also investigated the impact of the number of prompt candidates, as depicted in Table 6d. With the current configurations, an increase in the number of prompts can potentially lead to a performance improvement of  $2\%$ . Word Number of Each Prompt. We simply study the effect of the length of prompt in Fig. 6. An increase in text length from 5 to 10 yields approximately  $4\%$  enhancement. When the text length surpasses 10, the performance appears to plateau. We argue that the upper limit is due to the current capacity of generative models.
In this study, we investigate using a perception decoder to parse the latent space of an advanced diffusion model, extending the text-to-image task to a new paradigm: text-guided data generation. Training the decoder requires less than  $1\%$  of existing labeled images, enabling infinite annotated data generation. Experimental results show that the existing perception models trained on synthetic data generated by DatasetDM exhibit exceptional performance across six datasets and five distinct downstream tasks. Specifically, the synthetic data yields significant improvements of  $13.3\%$  mIoU for semantic segmentation on VOC 2012 and  $12.1\%$  AP for instance segmentation on COCO 2017. Furthermore, text-guided data generation offers additional advantages, such as a more robust solution for domain generalization and enhanced image editing capabilities. We hope that this research contributes new insights and fosters the development of synthetic perception data.
W. Wu, C. Shen's participation was supported by the National Key R&D Program of China (No. 2022ZD0118700). W. Wu, H. Zhou's participation was supported by the National Key Research and Development Program of China (No. 2022YFC3602601), and the Key Research and Development Program of Zhejiang Province of China (No. 2021C02037). M. Shou's participation was supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF13-2021-0008, and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).
[48] K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693-5703, 2019.
[49] X. Wang, L. Jing, Y. Lyu, M. Guo, J. Wang, H. Liu, J. Yu, and T. Zeng. Deep generative mixture model for robust imbalance classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[50] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[51] G. Wilson and D. J. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
[52] J. Wu, H. Fang, Y. Zhang, Y. Yang, and Y. Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. arXiv preprint arXiv:2211.00611, 2022.
[53] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303.11681, 2023.
[54] Z. Wu, L. Wang, W. Wang, T. Shi, C. Chen, A. Hao, and S. Li. Synthetic data supervised salient object detection. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5557-5565, 2022.
[55] A. Xu, M. I. Vasileva, A. Dave, and A. Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. arXiv preprint arXiv:2212.12645, 2022.
[56] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803, 2023.
[57] M. Zhang, Y. Zhou, J. Zhao, Y. Man, B. Liu, and R. Yao. A survey of semi-and weakly supervised semantic segmentation of images. Artificial Intelligence Review, 53:4259-4288, 2020.
[58] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
[59] W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and J. Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153, 2023.
[60] Y. Zhao, Q. Ye, W. Wu, C. Shen, and F. Wan. Generative prompt model for weakly supervised object localization. arXiv preprint arXiv:2307.09756, 2023.
[61] X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270, 2022.