SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details: We build our framework on PyTorch deep learning framework [52] and Stable Diffusion [5] version 2.1-base with  $T = 100$  timesteps. We construct the masks using optimal values for  $\tau$ ,  $\alpha$ , and  $\beta$ , which are defined in Sec. 6.2. Regarding semantic segmenter, we employ the DeepLabV3 [15] and Mask2Former [24] segmenter implemented in the MMSegmentation framework [53]. We use the AdamW optimizer with a learning rate of  $1e^{-4}$  and weight decay of  $1e^{-4}$ . For other hyper-parameters, we follow standard settings in MMSegmentation.
Quantitative results: Tab. 1 compares the results of DeepLabV3 [15] and Mask2Former [24] trained on the real training set, a synthetic dataset of DiffuMask [8], and the synthetic dataset of Dataset
Table 1: Comparison in mIoU between training DeepLabV3 [15] and Mask2Former [24] on the real training set, the synthetic dataset of DiffuMask [8], and the synthetic dataset of Dataset Diffusion.
Segmenter	Backbone	VOC dataset	COCO dataset
Training set	Val	Test	Training set	Val
DeepLabV3	ResNet50	VOC's training (11.5k images)	77.4	75.2	COCO's training (2017: 118k images)	48.9
DeepLabV3	ResNet101	79.9	79.8	54.9
Mask2Former	ResNet50	77.3	77.2	57.8
Mask2Former	ResNet50	DiffuMask [8] (60k images)	57.4	-	-	-
DeepLabV3	ResNet50	Dataset Diffusion (40k images)	61.6	59.0	Dataset Diffusion (80k images)	32.4
DeepLabV3	ResNet101	64.8	64.6	34.2
Mask2Former	ResNet50	60.2	60.5	31.0
Table 2: Performance of different text prompt selections. Red: class names, blue: similar terms.
Method	Example	mIoU (%)
1: Simple text prompts	a photo of an aeroplane	54.7
2: Captions only	a large white airplane sitting on top of a boat	50.8
3: Class labels only	aeroplane boat	57.4
4: Simple text prompts + class labels	a photo of an aeroplane; aeroplane boat	57.6
5: Caption + class labels	a large white plane sitting on top of a boat; aeroplane boat	62.0
Diffusion. On VOC, our approach yields satisfactory results of  $64.8\mathrm{mIoU}$  when compared to the real training set of  $79.9\mathrm{mIoU}$ . Further, ours outperforms DiffuMask by a large margin of  $4.2\mathrm{mIoU}$  using the same Resnet50 backbone. The detailed IoU of each class is reported in the Supp. Also, Dataset Diffusion achieves a promising result of  $34.2\mathrm{mIoU}$  compared to  $54.9\mathrm{mIoU}$  of real COCO training set. These results demonstrate the effectiveness of Dataset Diffusion, although the gaps with the real dataset are still substantial, i.e.,  $15\mathrm{mIoU}$  in VOC and  $20\mathrm{mIoU}$  in COCO. This is due to the fact that the image content of COCO is more complex than that of VOC, reducing the ability of Stable Diffusion to produce images with the same level of complexity. We will discuss more in Sec. 5.
Qualitative results on the validation set of VOC are shown in Fig. 5. In Fig. 5a, the synthetic images and their corresponding masks are utilized for training the semantic segmenter. The first two rows (1, 2) serve as excellent examples of successful segmentation, while the last two rows (3, 4) demonstrate failure cases. In certain instances, the self-training technique proves effective in rectifying mis-segmented objects (as seen in rows 2 and 3). However, it can also adversely impact the original masks when dealing with objects of small size (as observed in row 4). In Fig. 5b, our predicted segmentation results on the validation set of VOC exhibit varying outcomes. The first three rows exhibit satisfactory results, with the predicted masks closely aligning with the ground truth. Conversely, the last three rows illustrate failure cases resulting from multiple small objects (row 4) and the presence of intertwined objects (rows 5 and 6).
We conduct all ablation study experiments on the text prompts described in Sec. 3.1. Additionally, we report the results with  $20\mathrm{k}$  images using the initial mask generated by Dataset Diffusion without using the self-training technique or test-time augmentation unless indicated in each experiment.
Effect of text prompt selection. Tab. 2 compares different text prompt selection methods. Our class-prompt appending technique outperforms the text prompts using captions or class labels only. Specifically, the class-prompt appending technique increases the performance by 11.2 and  $4.6\mathrm{mIoU}$  over the "caption-only" and "class-label-only" text prompts, respectively. Class-prompt appending also outperforms the simple text prompts by  $7.3\mathrm{mIoU}$ . These results indicate that our text prompt selection method can help SD generate datasets with both diversity and accurate attention.
Effects of different components of stage 2 and stage 3 in Fig. 2 on the overall performance are summarized in Tab. 3. Using only cross-attention results in a low performance of  $44.8\mathrm{mIoU}$  as the
Figure 5: (a) Row 1 (R1) and R2 are successful cases, while R3 and R4 demonstrate failures. Self-training helps correct mis-segmented objects in some cases (R2 and R3) but can harm the original mask for small objects (R4). (b) R1 to R3 show accurate results, closely matching the GT. R4 to R6 reveal failure cases due to numerous small objects (R4) and intertwined objects (R5 and R6).
(b) Segmentation results on VOC's validation set
Table 3: Impact of cross-attention, self-attention, uncertainty, self-training, and test time augmentation (TTA) (refer to Sec. 3.2, Sec. 3.3). TTA includes multi-scale and input flipping at test time.
Cross-attention	Self-attention	Uncertainty	Self-training	TTA	mIoU (%)
✓	44.8
✓	✓	61.0
✓	✓	✓	62.0
✓	✓	✓	✓	62.7
✓	✓	✓	✓	✓	64.3
cross-attention map is coarse and inaccurate (as illustrated in Fig. 4). Using self-attention refinement boosts the performance significantly to  $61.0\mathrm{mIoU}$ . Also, using other techniques like uncertainty-aware loss, self-training, and test time augmentation help improve performance incrementally.
Effect of different feature scales used for aggregating self-attention and cross-attention maps is shown in Tab.4. As can be seen, for the cross-attention map, choosing too small and too large feature scales both hurt the performance since the former lacks details while the latter focuses on fine details instead of object shape. For the self-attention map, using the scale of 32 gives slightly better results.
Hyper-parameters selection for mask generation (Sec. 3.2). We conduct sensitivity analysis on  $\tau$ ,  $\alpha$ , and  $\beta$  to determine the optimal values in Tab. 5. Tab. 5a shows the results of choosing  $\tau$  (with fixed  $\alpha = 0.5, \beta = 0.6$ ) with the best result with  $\tau = 4$ . A too-large value of  $\tau = 5$  decreases the performance as the refined cross-attention map tends to spread out the whole image rather than the object only. Additionally, Tab. 5b exhibits the analysis on the  $(\alpha, \beta)$  range given the fixed  $\tau = 4$ , the range of  $(0.5 - 0.6)$  achieves the best performance of  $62.0\mathrm{mIoU}$ .
Table 4: Study on different feature scales
Cross-attention	Self-attention
32	64
8	39.7	38.1
16	62.0	59.6
32	52.8	50.9
64	35.4	31.5
16, 32	59.7	57.3
16, 32, 64	59.1	57.2
Table 5: Hyper-parameters for mask generation.
(a) Analysis of  $\tau$  with  $\alpha = 0.5$  and  $\beta = 0.6$
τ	0	1	2	3	4	5
mIoU (%)	44.8	59.5	60.5	60.2	62.0	60.5
(b) Analysis of  $(\alpha ,\beta)$  given  $\tau = 4$
α - β	0.4-0.5	0.5-0.6	0.4-0.6
mIoU (%)	59.5	62.0	60.7
Figure 6: Left: Correlation maps at some positions with others, extracted from a self-attention map. Right: Failure cases of SD when generating images with multiple objects. Red: classes are missed.
A group of lawn chairs sitting on top of a beach; person umbrella chair
An unusual looking red bus going down a road.; person car bus
A close up of a cat on a desk near a sandwich; cat apple sandwich
A person on a skateboard on the road; person car skateboard
Limitations: While our method is effective for generating synthetic datasets, there are some limitations to consider. Our primary reliance on Stable Diffusion [5] for image generation can result in difficulties with producing complex scenes. First, when given a text prompt that involves three or more objects, the diffusion model may only produce an image depicting two or three objects as exemplified in Fig. 6 - Right. However, there is ongoing research to improve the quality of the diffusion model and to incorporate stronger guidance, such as layout or box conditions, which shows promise in addressing this issue. Second, it is worth noting that in some cases, our Dataset Diffusion may not produce high-quality segmentation masks when objects are closely intertwined, as seen in Fig.5a with the example of a man riding a horse. Third, the bias in the LAION-5B dataset, on which Stable Diffusion was trained, may be transferred to the generated dataset. This is the current limitation of Stable Diffusion as it was trained on a large-scale uncurated dataset like LAION-5B. However, there are several studies addressing the bias problem in generative models [54-56] focusing on enhancing fairness and reducing biases in generative models. We believe that these studies and future work on the topic of fairness in GenAI will help to mitigate the bias in the generated images.
Conclusion: We have presented our novel framework - Dataset Diffusion - which enables the generation of synthetic semantic segmentation datasets. By leveraging Stable Diffusion, Dataset Diffusion can produce high-quality semantic segmentation and visually realistic images from specified object classes. Throughout our experiments, we have demonstrated the superiority of Dataset Diffusion over the concurrent method, DiffuMask, achieving an impressive mIoU of 64.8 in VOC and 34.2 in COCO. This remarkable advancement paves the way for future research endeavors focused on the creation of large-scale datasets with precise annotations using generative models.
[49] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022. 4, 6
[50] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013. 6
[51] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference on computer vision, pages 991-998. IEEE, 2011. 6
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. ArXiv, abs/1912.01703, 2019. 6
[53] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020.6
[54] Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint arXiv:2302.10893, 2023. 9
[55] Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to-image generation. arXiv preprint arXiv:2308.00755, 2023.
[56] Xingzhe Su, Wenwen Qiang, Zeen Song, Hang Gao, Fengge Wu, and Changwen Zheng. Manifold-guided sampling in diffusion models for unbiased image generation. arXiv preprint arXiv:2307.08199, 2023. 9
In this supplementary material, we first elaborate on our implementation details in Sec. 6.1 Then, we present additional results from our ablation study in Sec. 6.2, focusing on two key aspects: the impact of timesteps on aggregating attention maps and the influence of the number of generated images on the segmentation results. Next, we provide a comprehensive per-class IoU using PASCAL VOC test set and COCO2017 val set in Sec. 6.3. After that, we present the quantitative results of Dataset Diffusion in other image domains in Sec. 6.4. Finally, we provide more qualitative results in Sec. 6.5.
6.1 Elaboration of Implementation Details
Other text prompt selection methods. In this part, we discuss how we implement other text prompt selection methods in the first ablation study in the main paper. We note that the difference between these methods and Dataset Diffusion lies in the prompt construction and the cross-attention aggregation. For the 'simple text prompts' method, we construct prompts as "A photo of a [class name]" with the cross-attention map at the token "[class name]". For the 'caption only' method, the class labels are not appended to the captions as in Dataset Diffusion with the cross-attention at the class category token. In this way, the terms that do not match exactly with the class names, e.g., "aeroplane" vs. "airplane"), will be ignored. For the "class labels only" method, we use the prompts as "[class name 1] [class name 2] ...” with the cross-attention at the class tokens. For compound class names such as "dining table", we take the mean of the cross-attention maps at all positions.
More on Sec 3.1. We have observed a major issue with Stable Diffusion. It often fails to generate an adequate number of target classes when multiple object classes are provided in the text prompts, particularly in the case of COCO. To avoid this issue, we introduce a limit on the number of class labels, denoted as  $k$ . If the number of classes  $M$  in an image exceeds this threshold, we select the top- $k$  classes based on their frequency. Then, we generate  $k$  simpler text prompts, each containing only one class from  $C_i$  classes of image  $i$ , ranked by the least frequent classes. The simple text prompt used is "a photo of a [class name]; [class name]". This approach ensures that the generated images are more faithful to the provided text prompts, and it facilitates the creation of a more diverse set of text prompts that includes both simple prompts and real captions. This strategy helps to enhance the quality and coverage of the synthetic dataset, mitigating the issue of missing target classes in the generated images.
Computation details. We conduct our experiments on NVIDIA A100 40G GPUs. Generating a 40k image dataset with Stable Diffusion V2.1 takes about 30 hours and training the DeepLabV3 for 20k iterations takes about 3 hours on a single GPU.
6.2 Additional Ablation Studies
Study on different ranges of timesteps to aggregate attention maps. In the main paper, we only experiment with aggregating self-attention and cross-attention maps over all the timesteps. Here, we provide an ablation study in Tab. 6 to show that the variation in timesteps has a minimal impact on the results. As can be seen, averaging all the timesteps yields the best performance, and the maximum decrease is only  $0.5\mathrm{mIoU}$ .
Study on numbers of generated image-mask pairs in the synthetic dataset. We evaluate the performance of our semantic segmenter on three numbers of generated images: 10k, 20k, and 40k. The results are shown in Tab. 7. By using a four times larger training dataset (from 10k to 40k), we only gain  $1.0\%$  in mIoU. We expect the gain to be smaller when increasing the number of training images further since the system performance is reaching its limit. Because the gain is small compared to the additional computation cost, we do not examine our system when using more than 40k images. We further experiment with the pseudo masks generated using a semantic segmenter well-trained on the real Pascal VOC 2012 dataset to make predictions on our synthetic images. We consider them highly accurate pseudo masks. The results in Tab. 8 show that the performance of the semantic segmenter training on 40k images with these masks yields a 1.4 mIoU gain compared to when training on 10k images, suggesting this small gain comes solely from the mask quality, not from increasing number of generated images.
Table 6: The impact of different ranges of timesteps to aggregate attention maps.
Timesteps	100-88	88-75	75-50	50-0	100-0
mIoU (%)	61.9	61.5	61.5	61.8	62.0
Table 7: The impact of different numbers of generated images in the synthetic dataset.
# images	10k	20k	40k
mIoU (%)	63.8	64.3	64.8
Table 8: Different # generated images with masked produced by DeepLabV3 trained on real data.
# images	10k	20k	40k
mIoU (%)	71.2	71.6	72.6
6.3 Detailed Per-class IoU on the VOC and COCO datasets
The detailed per-class IoUs of the COCO 2017 dataset are shown in Tab. 9. Furthermore, in Tab. 10, we report the mIoU on the VOC test set of each class when training DeepLabV3 on our synthetic dataset and the Pascal VOC training set. We observe that for the class "tv/monitor", the cross-attention frequently focuses on the boundary of the monitor rather than capturing the entire objects, and the self-attention map cannot help alleviate this issue, resulting in poorly generated masks (as illustrated in Fig. 7). This behavior could explain the significant performance drop of that class compared to the results trained on the real dataset.
6.4 Other Image Domains
Driving Scenes (Tab. 11): Our synthetic dataset exhibits a performance gap with real datasets (3k real images + manual labeling) of approximately  $19\mathrm{mIoU}$ . This discrepancy parallels those observed in the VOC and COCO datasets, as previously discussed in our main paper. Notably, our synthetic dataset performs comparably with real datasets for specific classes such as bicycle, car, and person. The process of generating images for this dataset is similar to those of VOC and COCO.
Facial Part Segmentation (Tab. 12): A performance gap of around  $16.5\mathrm{mIoU}$  (for 5k images) exists between real and synthetic datasets. This disparity is reasonable, considering our approach's zero-shot nature, where training images are generated based on text prompts. Compared to the faces generated in DatasetGAN whose training images are from CelebA-HQ-Mask, the faces generated by Dataset Diffusion are often not well-aligned. However, Dataset Diffusion still competes favorably (78.2 vs 87.0). Furthermore, DatasetGAN requires a number of manual-labeled images (20 in this case) for generating new images and segmentations. Hence, for a fair comparison, we combine our synthetic data with 20 labeled images, and DatasetDiffusion surpasses DatasetGAN (89.9 vs. 87.0). An example of text prompt used is "A portrait photo of a young woman; hair eye nose ear mouth" where hair, eye, nose, ear, mouth are parts' names.
Satellite/Aerial Images (Tab. 13): Dataset Diffusion's generated image quality and segmentations do not align with DroneDeploy's standards, leading to a mIoU gap of 37.4. This discrepancy stems from the limited presence of aerial/satellite images in SD's training set (LAION-5B). The text prompts hinder the generation of images that match DroneDeploy's style. The results are presented with the prompt: "An aerial view of a [building, clutter, water, vegetation, ground, car]" and 15k images.
6.5 More Qualitative Results
We first demonstrate the qualitative results of the segmenter on the VOC val set in Fig. 8 and COCO 2017 val set in Fig. 9. Next, in Figs. 10, 11, and 12, we present further qualitative results to illustrate the effectiveness of Dataset Diffusion in generating scenes with different complexities. In Fig. 10, a simple prompt with a single object is depicted, and Dataset Diffusion performs remarkably well
Table 9: COCO 2017's per-class IoU with DeepLabV3 trained on COCO's training set (118k images) and Dataset Diffusion (100k images).
Class	COCO	Dataset Diffusion	Class	COCO	Dataset Diffusion
background	89.1	80.9	wine glass	48.8	29.3
person	80.6	50.5	cup	44.8	13.7
bicycle	64.7	39.9	fork	18.0	11.1
car	53.1	35.5	knife	19.0	6.1
motorcycle	76.9	62.6	spoon	15.8	7.7
airplane	78.0	62.2	bowl	38.0	24.4
bus	77.2	64.8	banana	62.0	49.0
train	77.2	58.7	apple	40.1	34.9
truck	52.8	36.8	sandwich	40.9	19.4
boat	53.1	42.5	orange	64.4	48.4
traffic light	60.5	31.0	broccoli	51.0	36.8
fire hydrant	84.6	55.9	carrot	41.8	22.1
stop sign	90.5	42.9	hot dog	49.8	30.9
parking meter	68.3	51.0	pizza	67.8	44.1
bench	46.8	27.5	donut	62.7	48.3
bird	66.9	47.2	cake	51.3	30.1
cat	79.9	64.0	chair	34.3	10.5
dog	74.5	46.5	couch	52.3	29.5
horse	77.4	60.5	potted plant	26.1	4.5
sheep	79.7	63.1	bed	58.3	42.0
cow	75.0	63.9	dining table	39.6	14.6
elephant	87.4	76.6	toilet	70.0	51.3
bear	88.4	78.1	tv	63.8	6.3
zebra	88.2	81.8	laptop	65.8	42.5
giraffe	82.5	77.0	mouse	60.3	12.7
backpack	23.1	7.5	remote	48.6	21.2
umbrella	69.3	51.2	keyboard	55.5	32.6
handbag	18.9	0.04	cell phone	57.1	35.7
tie	6.7	4.6	microwave	59.9	39.1
suitcase	65.8	29.8	oven	50.1	17.8
frisbee	55.8	29.1	toaster	0.5	6.4
skis	22.4	7.2	sink	54.9	21.4
snowboard	48.7	21.9	refrigerator	70.3	20.1
sports ball	42.1	24.5	book	37.3	23.2
kite	46.7	34.2	clock	64.0	24.3
baseball bat	23.1	8.3	vase	48.0	39.4
baseball glove	59.8	2.5	scissors	59.1	25.1
skateboard	45.3	22.0	teddy bear	70.8	46.1
surfboard	60.2	43.4	hair drier	0.08	2.6
tennis racket	72.6	15.7	toothbrush	36.6	21.9
bottle	37.5	20.1	mIoU (%)	54.9	34.2
in this scenario. In Fig. 11, more complex scenes are shown, and Dataset Diffusion still produces acceptable results. However, the limitations of Stable Diffusion become apparent in Fig. 12, where generated scenes are complex but often lack the objects explicitly mentioned in the prompt. This highlights the challenges Stable Diffusion faces when generating scenes with multiple objects.
Table 10: VOC's per-class IoU with DeepLabV3 trained on VOC's training set and Dataset Diffusion.
Class	VOC	Dataset Diffusion	Class	VOC	Dataset Diffusion
aeroplane	93.6	81.6	diningtable	58.9	42.9
bicycle	43.8	35.8	dog	89.7	71.8
bird	93.4	73.3	horse	93.4	78.2
boat	67.0	62.2	motorbike	90.9	80.6
bottle	78.5	72.6	person	87.1	70.8
bus	95.9	85.5	pottedplant	68.2	53.9
car	90.7	64.8	sheep	90.9	77.8
cat	94.9	78.2	sofa	58.7	41.8
chair	36.8	21.6	train	84.9	72.7
cow	89.4	69.2	tv/monitor	74.3	29.6
mIoU (%)	79.8	64.6
Figure 7: The poorly generated masks for the class "tv/monitor".
Table 11: Cityscapes Object Segmentation Results
Training set	Bicycle	Bus	Car	Motorbike	Person	Train	mIoU
Cityscapes	78.0	89.7	95.4	69.8	82.5	84.6	83.3
Dataset Diffusion	72.9	57.7	93.7	39.8	82.1	39.9	64.4
Table 12: CelebA-HQ-Mask Facial Part Segmentation Results
Training set	# Samples	Hair	Eye	Nose	Ear	Mouth	mIoU
CelebA-HQ-Mask	5k labeled	98.9	98.0	99.5	77.9	99.4	94.7
Dataset Diffusion	5k synthetic	97.2	74.5	85.1	41.2	93.1	78.2
DatasetGAN	5k synthetic + 20 labeled	97.5	95.3	96.6	48.1	95.7	87.0
Dataset Diffusion	5k synthetic + 20 labeled	97.9	95.3	97.2	61.5	97.6	89.9
Table 13: DroneDeploy Results
Training set	Building	Clutter	Vegetation	Water	Ground	Car	mIoU
DroneDeploy	47.8	32.3	64.4	73.9	84.6	65.5	61.4
Dataset Diffusion	21.9	13.6	28.8	10.1	58.7	11.0	24.0
Figure 8: Segmentation results on VOC's val set. Three separate columns show examples of simple scenes with a single object, quite complex scenes with multiple objects, and very complex scenes with intertwined objects, respectively. In each example from left to right: test image, ground truth segmentation, and our predicted mask.
Figure 9: Segmentation results on COCO 2017. Please refer to the caption of Fig. 8 for more details.
A photo of keyboard; keyboard
A photo of mouse; mouse
A photo of orange; orange
A photo of apple; apple
Figure 10: Dataset Diffusion can generate high-quality image-mask pairs with a simple text prompt containing a single object. Each row shows two examples with text prompts underneath. From left to right: generated image, initial mask with uncertain regions in white, and final mask after self-training.
A photo of cow; cow
A photo of remote; remote
Two men in cowboy hats tying the legs of a cow.; person cow
A man hanging onto his little boy teaching him to ski; person skis
A bowl with nuts and bananas next to a bunch of bananas.; bowl banana
A man that is walking across a concrete pad; person horse