Dataset Diffusion: Diffusion-based Synthetic Data Generation for Pixel-Level Semantic Segmentation
Authors: Quang Ho Nguyen
Truong Tuan Vu
Anh Tuan Tran
Khoi Nguyen
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep learning; Diffusion Models; Semantic Segmentation; Text-to-Image
TL;DR: We propose a new approach for generating semantic segmentation labels from Stable Diffusion along with images.

Abstract: Preparing training data for deep vision models is a labor-intensive task. To address this, generative models have emerged as an effective solution for generating synthetic data. While current generative models produce image-level category labels, we propose a novel method for generating pixel-level semantic segmentation labels using the text-to-image generative model Stable Diffusion (SD). By utilizing the text prompts, cross-attention, and self-attention of SD, we introduce three new techniques: class-prompt appending, class-prompt cross-attention, and self-attention exponentiation. These techniques enable us to generate segmentation maps corresponding to synthetic images. These maps serve as pseudo-labels for training semantic segmenters, eliminating the need for labor-intensive pixel-wise annotation. To account for the imperfections in our pseudo-labels, we incorporate uncertainty regions into the segmentation, allowing us to disregard loss from those regions. We conduct evaluations on two datasets, PASCAL VOC and MSCOCO, and our approach significantly outperforms concurrent work. Our benchmarks and code will be released at https://github.com/VinAIResearch/Dataset-Diffusion.

Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation
Quang Nguyen 1.2 Truong Vul- Anh Tran'
Khoi Nguyen'
VinAI Research, Ho Chi Minh City University of Technology. VNU-HCM
Abstract
Preparing training data for deep vision models is a labor-intensive task. To ad- dress this, generative models have emerged as an effective solution for generating synthetic data. While current generative models produce image-level category labels, we propose novel method for generating pixel-level semantic segmen- tation labels using the text-to-image generative model Stable Diffusion (SD). By utilizing the text prompts, cross-attention, and self-attention of SD, we introduce three new techniques: class-prompt appending, class-prompi cross-attention, and self-attention exponentiation. These techniques enable us to generate segmentation maps corresponding to synthetic images. These maps training semantic segmenters, eliminating the need for labor- intensive pixel-wise annotation. To account for the imperfections in pseudo-labels, uncertainty regions into the segmentation, allowing us to disregard loss from those regions. We conduct evaluations and our approach significantly outperforms concurrent work. Our benchmarks and code will be released at
as pseudo-labels for
incorporate
datasets, PASCAL VOC and MSCOCO,
Diffuusion.
Introduction
Semantic segmentation is fundamental task in computer vision. Its objective is to assign semantic labels to each pixel an image, making it crucial for applications such as autonomous driving, scene comprehension, and object recognition. However, one of the primary challenges in semantic segmentation is the high cost associated with manual annotation. Annotating large-scale datasets with pixel-level labels is labor-intensive, time-consuming. and requires substantial human effort.
To address this challenge, an alternative strategy involves leveraging generative models to synthesize datasets with pixel-level labels. Past research efforts have utilized Generative Adversarial Networks (GANs) to effectively generate synthetic datasets for semantic segmentation, thereby mitigating the reliance on manual annotation [1-3]. However, GAN models primarily concentrate on object-centric images and have yet to capture the intricate complexities present real-world scenes.
On the other hand, text-to-image diffusion models have emerged as a promising technique for generating highly realistic images from textual descriptions [4-7]. These models possess unique characteristics that make them well- rell-suited for the generation of semantic segmentation datasets. Firstly, the text prompts used input to these models can serve as valuable guidance since they explicitly specify the objects to be generated. Secondly, the application of cross and self-attention maps in the image generation process endows these models with informative spatial cues, enabling precise extraction of object positions within the generated images. By leveraging these characteristics of text-to-image diffusion models, the concurrent works Diffu- Mask [8] and DiffusionSeg [9] effectively generate pairs of synthetic images and corresponding
First two authors contribute equally. The work done during Quang Nguyen's internship at VinAl Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Target classes: person" "dog". "horse"
Test Image
Dataset Generated Images
Semantic Segmenter Training
Semantic Segmenter
Segmentations
Segmentation
Figure 1: Overview of our Dataset Diffusion for synthetic dataset generation. (Left) Given the target classes, our framework generates high-fidelity images with their corresponding pixel-level semantic segmentations. These segmentations serve pseudo-labels for training semantic segmenter. (Right) The trained semantic segmenter is able to predict the semantic segmentation test image.
segmentation masks. DiffuMask achieves this by utilizing straightforward text prompts, such as "a photo of [class name] [background description] ", to generate image and segmenta- tion mask pairs. Meanwhile, DiffusionSeg focuses on creating synthetic datasets that address the challenge of object discovery, which involves identifying salient objects within an image. While these approaches successfully produce images paired with their corresponding segmentation masks, they are currently limited to generating single object segmentation mask per image.
In this paper, present Dataset Diffusion, novel framework for synthesizing high-quality semantic segmentation datasets, shown in Fig. 1. Our approach focuses on generating realistic images depicting scenes with multiple objects, along with precise segmentation masks. We introduce two techniques: class-prompt appending, which encourages diverse object classes in the generated images, and class-prompt cross-atted cots-natention, enabling more precise attention to each object within the scene. We also introduce self-attention exponentiation, simple refinement method using self-attention maps to enhance segmentation quality. Finally, we employ the generated data to train semantic segmenter using uncertainty-ant aware segmentation loss and self-training.
To evaluate the quality of the synthesized datasets, introduce benchmark datasets: synth-VOC and synth-COCO. These benchmarks utilize two well-established semantic segmentation datasets, namely PASCAL VOC [10] and COCO [11], standardize the text prompt inputs and ground-truth segmentation evaluation. On the synth- VOC benchmark, Dataset Diffusion achieves an impressive mloU of 64.8, outperforming DiffuMask [8] by substantial margin. On the synth-COCC benchmark. the DeepLabV3 model trained on our synthesized dataset achieves noteworthy results of 34.2 in mloU compared the model trained on real images with full supervision.
In summary, the contributions of our work are as follows:
We present framework that effectively employs state-of-the-arr text-to-image diffusion model to generate synthetic datasets with pixel-level annotations.
We introduce a simple and effective text prompt design that facilitates the generation of complex and realistic images, closely resembling real-world scenes.
We propose straightforward method that utilizes self and cross-attention maps to achieve highly accurate segmentation, thereby improving the quality and reliability of the synthesized datasets. We introduce synth-V synth-VOC and synth- month-COCO benchmarks for evaluating the performance of semantic segmentation dataset synthesis.
In the following, Sec. reviews prior work, Sec. 3 describes our proposed framework, and Sec. presents our experimental results, Finally, Sec. concludes with some remarks and discussions.
Related Work
Semantic segmentation is a critical computer vision task that involves classifying each pixel in
an image to specific class label. Popular semantic segmentation approaches include the fully convolutional network (FCN) [12] and its successors, such as Deeplab [13], DeepLabV2 [14],
DeepLabv3 [15], DeepLaby3+ [16], UNet [17]. SegNet [18], PSPNet [19], and HRNet [20]. Recently,

---

transformer-basee approaches like SETR [21], Segmenter [22], SegFormer [23], and Mask2Former [24] have gained attention for their superior performance over convolution- based approaches. In our framework, we focus on generating synthetic datasets that can be used with any semantic segmenter, so we use Dev3 and Mask2Former as they are commonly used.
Text-to-image diffusion models have revolutionized image generation research, moving beyond simple class-conditioned more complex lext-conditioned image generation. Examples include GLIDE [25]. Imagen [6], Stable Diffusion (SD) [5], Dall-E [4], eDiff-I [7], and Muse [26]. These models can generate images with multiple objects interacting with each other, more closely resembling real-world images rather than the single object-centric images generated by prior generative models. Our Dataset Diffusion marks milestone synthetic dataset generation literature, moving from image-level annotation to pixel-level annotation. We utilize Stable Diffusion [5] in our framework, as it is the only open-sourced pretrained text-to-image diffusion model available at the time of writing.
Diffusion models for segmentation. Diffusion models have proven effective for semantic, instance, and panoptic segmentation tasks. These models either use input images to condition the mask- denoising process [27-33], or employ pretrained diffusion models as feature extractors [34-37]. However, they still require ground-truth (GT) segmentation for training. In contrast, our framework utilizes only a pretrained SD to generate semantic segmentation without GT labels.
Generative Adversarial Networks (GANs) for synthetic segmentation datasets. GANs have been employed in the generation of synthetic segmentation datasets, as demonstrated in previous works such as [1, 3, 38, 39]. However, these approaches primarily focus on object-centric images, where a single mask is segmented for the salient object or specific parts of common objects like faces, cars, horses, exemplified in [2]. In contrast, our framework is designed to generate semantic segmentations for more complex images, where multiple objects interact with each other at the scene level. Furthermore, while some techniques [38, 39] support foreground/backkround subtraction, and others [1, 3] still require human annotations, our objective is to generate semantic segmentations for multiple object classes in each image without the need for human involvement.
Diffusion models for synthetic data generation have been used to improve the performance of image classification [40, 41], domain adaptation for classification [42, 43], and zero/few-shor learning [44-4]]. However, these methods produce only image-level annotations as augmentation datasets. In contrast, framework produces pixel-level annotations, which is considerably more challenging.
Recently, there have been concurrent works [8. 9] that utilize Stable Diffusion (SD) for generating object segmentation without any annotations. However, they focus on segmenting single object in an image rather than multiple objects. Their text-prompt inputs to SD are simple, usually "a photo of [class name]"', The semantic segmenter trained on these annotations can segment multiple objects to some extent. Our framework, on the other hand, employs more complex text prompts where multiple objects can coexist and interact, making it more suitable for the semantic segmentation task in real-world images.
3 Dataset Diffusion
Problem setting: Our objective is to generate synthetic dataset D = (I,, S consisting of high-fidelity images I and pixel-level semantic masks S. These images and masks capture both the semantic and location information of the target classes C = {c1, C2, .... Cx) ). where K represents the number of classes. The purpose of constructing this dataset is to train a semantic segmenter without relying human annotation.
In our approach, we follow threc-step process. Firstly, we prepare relevant text prompts P containing the target classes (Sec. 3.1). Secondly, using Stable Diffusson (SD) as our model, we generate images I E IR #x'x'' and their corresponding semantic segmentations S; E {O...., K "xW where 0 represents the background class (Sec. 3.2). These images and segmentations form the synthetic dataset D. Lastly, we train a semantic segmenter on and evaluate its performance on the test set of standard semantic segmentation datasets (Sec. 3.3). It is worth noting that our approach primarily focuses on segmenting common objects everyday scenes, where the SD model excels, rather than specialized domains like medical or acrial images. The overall framework is depicted in Fig, 2.
3

---

Text Prompt Preparation
Segmentation Generation
Semantic Segmenter Training
Clasees sagment:
Generated images
"person", "dog". "lable", "motorbike", "horae"
Goncrata captions from LLM
Generate rotrieve captions from VOC or GOCO
Semantic Segmenter
Self-trannig
Self-ettenlion
OR
Linceraiinty-wwan aworr segmentation loss
Text prompt Coption + Cleeses labels: "A man is running with two dogs: dog, person"
Stable Diffusion
Cross vross-attention
map
Refined cross-
attention map
Segmentaaiins
Figure 2: Three stages of Dataset Diffusion. In the first stage, the target classes are provided, and text prompts are generated using language models such ChatGPT [48]. Real captions (for COCO) or image-based captions (for VOC) can also be used for prompt generation to ensure standard evaluation. The text prompts are then augmented with the target class labels to avoid missing objects. In the second stage, given the augmented text prompt, a frozen Stable Diffusion [5] is employed to generate image and its self- and cross-attention maps. The cross-attention map for each target class is refined using the self-attention map to match the object's 's shape. Finally, the generated images and corresponding semantic segmentations are used to train a semantic segmenter with uncertainty-awer loss and the self-training technique.
3.1 Preparing Text Prompts for Stable Diffusion
To prepare prompts containing a given list of classes for SD, one option to utilize a large language model (LLM) such ChatGPT [48] to generate the sentences, similar to the method described in [9]. This approach can be valuable in worl-world applications.
However, for evaluating the quality of the synthetic dataset, we need to rely on standard datasets for semantic segmentation like PASCAL VOC [10] or COCO [11] to create standardized benchmarks. In this regard, we propose using the provided or generated captions of the training images in these datasets as the text prompts for SD. This is solely for the purpose of standard benchmarking where the text prompts are fixed, and we do not utilize real images image-label associations in our synthetic dataset generation. We call these new benchmarks as Synth-VOC. and synth-COCO.
When using the COCO dataset, we can rely on the provided captions to describe the training images. However, in the case of the PASCAL VOC dataset, which lacks captions, we employ a state-of-the-an image captioner like BLIP [49] to generate captions for each image. However, we encountered several issues with the provided or generated captions. Firstly, the text prompts may not use the exact terms as the target class names C provided in the dataset. For instance, terms like "man' and "'oomnn' may be used instead of "person'' or "bike" instead of "bicycle", resulting in mismatch with the target classes. Secondly, many captions do not contain all the classes that are actually present in the images (as illustrated Fig. 3). This leads to a shortage of text prompts for certain classes, affecting the generation process for those particular classes.
To address the issues, we propose a method that leverages the class labels provided by the datasets. We append the provided (or generated) captions P. with the class labels, creating new text prompts p', that explicitly incorporate all the target classes C, [c:: : where M is the number of classes in image i. This is achieved through the text appending operation or class-promph appending technique: P', = CP. For example, in the case of the left image in Fig. 3, the final text prompt would be "a photograph of a kitchen inside a house; bottle mi crovvee sink refrigerator". This ensures that the new text prompts encompass all the target classes, addressing the issue of mismatched or missing class names in the captions.
3.2 Generating Segmentation from Self and Cross-attention Maps
We build our segmentation generator on Stable Diffusion (SD) by leveraging its self and cross- attention layers. Given a text prompt p' first encoded by text encoder into text embedding E Axxd with the text length A and the number of dimensions de, SD seeks to output the final latent state 20 E Hxxx xds where H, W, d2 are height, width, and number of channels of zo. reflecting the content encoded in from the initial latent state 27 N/ I) after I denoising steps.

---

Welcoma to Scatlone
Caption A photograph of kitceen inside house. Provided classes: bottle, microwave, sinil, refrigerator
Caption bike leaning against a sign in Scotandd Provided classes: bicycle backpack, bottle
Caption: A man riding a dirt bike in forest. Provided classes: person, motorcycie
Common issues of using provided (or generated) captions. Red classes are often missing from the captions, resulting in lack of text prompts for those classes. Blue classes may have different terms used in the captions, causing discrepancy between the target class names and the text prompts.
Figure 3:
Generated Image
Cross-attention
Uncertertyy
Figure 4: Given a text prompt "A bike is parked in a room; bicycle", we obtain the gener- ated image, cross attention map, enhanced cross-attention map by the self-attention with {1.2, 4} described in the Eq. (3), and mask with uncertainty value (white region) by Eq. (4) and Eq. (5).
At each denoising step t, a UNet architecture with layers of self and cross-attention is used to transform to 2t-1- In particular, layer / and time step t, the self-attention layer captures the pairwise similarity between positions within latent state 2 in order to enhance the local feature with the global context in 2 2/41. In the meantime, the cross-an cross-attention layer models the relationship between each position of the latent state zf and each token of the text embedding so that _(+1 can express more of the content encoded in
Formally, the self-attention map A [0, 1 HW and cross-attention map A 10, IA at layer and time step t are computed as follows:
(2KKT) Vah
12KTT
A
Softmax
A/'= =Softmax
(I)
Vai
where Q2, K2, Ke are the query of key of 2, and key of c, respectively, obtained by linear projections and taken as inputs to the attention mechanisms, and dj # features at layer I.
Since only want to obtain the cross-attention map of the class labels C₁ of image i for semantic segmentation, we introduce class-prompt cross-altention that similar to cross-attention Eq. (1) but produced by only taking the softmax over the class name part C rather than entire of the text prompt P',. In practice, cross-attention maps while the original text prompt P', ; for generating images keeps unchanged. After this, we obtain A E [0, HW xM where M is the number of classes in the image.
form
text prompt P. C, just for the purpose of extracting the
With the observation that using different ranges of timesteps only affects the final result marginally, (provided in Supp.), we average these cross and self-attention maps over layers and timesteps:
T
As LXT
A E
Ac
1 LXT
Σ 3
(2)
1=
Although the cross-attention maps Ac already exhibit the location of the target classes in the image, they are still coarse-grained and noisy, as illustrated in Fig. Thus, we propose use the self- attention map As (as illustrated in Fig. 6 Leet) to enhance Ag for a more precise object location. This is because the self-attention maps capturing the pairwise correlations among positions within the latent can help propagate the initial cross-attention maps the highly similar positions, e.g., son-salient parts of the object, thereby enhancing their quality. Therefore, we propose self- attention exponentiation where the self-attention map As is powered before multiplying the cross-attons artsstattention map Ac as:
A (As)² Aci
A 1
(3)
5

---

Next, we aim to identify two matrices: V [0. 1 representing the objectness value at each location (the higher the objectness, the more likely that location contains an object), and E (1...., M indicating which objects in the class labels C; that each location could be. To obtain those, we perform the pixel-wise arg max and max operator (over the category M dimension):
S = arg max
max m
At a location æ in the map V. if its value is less than a threshold, one can set its label to the background class O. However, we find that using a fixed threshold does not work for all images. Instead, we use a lower threshold for certain background decisions and higher threshold / for certain foreground decisions. Any value that falls inside the range (a, (3) expresses an uncertain mask prediction with value U 255. That is, the final mask S is illustrated in the last image of Fig. and calculated as:
{{ 0
if 12 Sa i < V4 < B, otherwise.
S,
U Sp
(5)
3.3 Training Semantic Segmenter on Generated Segmentation
Given the synthetic images I and semantic segmentation masks S, we train a semantic segmenter &b with an uncertainty- aware cross- cross-entropy loss. Specifically, for pixels marked as uncertain, we ignore the loss from those as: Σ. (S, * CC, Sz), where the indication function, Lce is the cross entropy loss, and S = (Z) is the predicted segmentation from the generated image I.
We further enhance the segmentation mask S by the self-training technique [50]. That is, after being trained with S, the segmenter & makes its own prediction on I as pseudo labels S without uncertainty value U. Finally, the final semantic segmenter 4' is the segmenter & trained again on S*.
4
Experiments
Datasets: We evaluate our Dataset Diffusion on two datasets: PASCAL VOC 2012 [10] and COCO 2017 [11]. The PASCAL VOC 2012 dataset has 20 object classes and background class. For standard semantic segmentation evaluation, this dataset is usually augmented with the SBD dataset [51] to have a total of 12, 046 training, 1, 449 validation, and 1, 456 test images. We additionally augment the training images with captions generated from BLIP [49]. The COCO 2017 dataset contains 80 object classes and background class with 118, 288 training and 5K validation images, along with provided captions for each image. is worth noting that we only use the image-level class annotation to form the text prompts described in Sec. 3.1.
We introduce the of our prepared text prompts along with the validation of each dataset as synth-VOC. and synth-COCO the two benchmarks for evaluation of semantic segmentation dataset synthesis. To create balance synthetic dataset among classes, we generate 2k images per object class for PASCAL VOC, resulting in total of 40k image-mask pairs and about 1 images per object class for COCO, resulting in a total of 80k image-mask pairs. the number of text prompts associated with certain class is insufficient, we use more random seeds to generate more images.
Evaluation metric:
We evaluate the performance of Dataset Diffusion using the mean Intersec- tion over Union (mloU) metric. The ml) score measures the overlap between the predicted segmentation masks and the ground truth masks for each class and takes the average across all classes.
Implementation details: We build our framework on PyTorch deep learning framework [52] and Stable Diffusion [5] version 2. 1-base with T = 100 timesteps. We construct the masks using optimal values for 7, a, and B3, which are defined in Sec. 6.2. Regarding semantic segmenter, we employ the DeepLabV3 [15] and Former [24] segmenter implemented in the MMSegmentation framework [53]. We use the AdamW optimizer with learning rate of le and weight decay of 14- For other hyper-parameters, follow standard settings in MMSegmentation.
4.1 Main Results
Quantitative results: Tab. compares the results of Deepl [15] and Mask2Former [24] trained on the real training set, synthetic dataset of DiffuMask [8], and the synthetic dataset of Dataset

---

Table Comparison in mloU between training DeepLabV3 [15] and Mask2Former [24]] on the real training set, the synthetic dataset of DiffuMask [8]. and the synthetic dataset of Dataset Diffusion.
VOC dataset
COCO dataset
Segmenter
Backhone
Training set
Val
Test
Training set
Val
DeepLabV3
ResNet50
77.4
75.2
48.9
VOC's training
COCO's training
DecpLabV3
ResNet101
79.9
79.8
54.9
(11.5k images)
(2017: 118k images)
Mask2Former
ResNet50
77.3
77.2
57.8
DiffuMask [8] (60k images)
Mask2Former
ResNet50
57.4
D3
ResNet50
61.6
59.0
32.4
Dataset Diffusion
Dataset Diffusion
DeepLaabV3
ResNet101
(40k images)
64.8
64.6
(80A images)
34.2
Mask2Former
ResNet50
60.2
60.5
31.0
Table 2: Performance of different text prompt selections. Red: class names, blue: similar terms.
Method
Example
mIoU (%)
1: Simple text prompts
photo of an aeroplane
54.7
2: Captions only
a large white airplane sitting on top of a boat
50.8
3: Class labels only
aeroplane hoat
57.4
4: Simple text prompts + class labels a photo of an acroplane; acroplane boat
57.6
5: Caption class labels
a large white plane sitting on top a boat; aeroplane boat
62.0
Diffusion. On VOC, our approach yields satisfactory results of 64.8 mIoU when compared to the real training set of 79.9 mloU. Further, ours outperforms DiffuMask by a large margin of 4.2 mloU using the same Resnet50 backbone. The detailed loU of each class is reported in the Supp. Also, Dataset Diffusion achieves a promising result of 34.2 mloU compared 54.9 mIoU of real COCO training set. These results demonstrate the effectiveness of Dataset Diffusion, although the gaps with the real dataset are still substantial, i.e., 15 mloU in VOC and 20 mloU in COCO. This is due the fact that the image content of COCO is more complex than that of VOC, reducing the ability of Stable Diffusion to produce images with the same level of complexity. We will discuss more in Sec. 5.
Qualitative results on the validation set of VOC are shown in Fig. 5. In Fig. 5a, the synthetic images and their corresponding masks are utilized for training the semantic segmenter. The first two rows (1, 2) serve as excellent examples of successful segmentation, while the last two rows (3, 4) demonstrate failure cases. In certain instances, the self-training technique proves effective in rectifying mis-segmented objects (as seen in rows 2 and 3). However, it can also adversely impact the original masks when dealing with objects of small size (as observed in row 4). In Fig. 5b, predicted segmentation results on the validation set of VOC exhibit varying outcomes. The first three rows exhibit satisfactory results, with the predicted masks closely aligning with the ground truth. Conversely, the last three rows illustrate failure cases resulting from multiple small objects (row 4) and the presence of intertwined objects (rows and 6).
4.2 Ablation Study
We conduct all ablation study experiments the text prompts described in Sec. 3.1. Additionally, we report the results with 20k images using the initial mask generated by Dataset Diffusion without using the self-training technique or test-time augmentation unless indicated in each experiment.
Effect of text prompt selection. Tab. 2 compares different text prompt selection methods. Our class-prompy appending technique outperforms the text prompts using captions or class labels only. Specifically, the class-prompp appending technique increases the performance by 11.2 and 4.6 mloU over the "caption-only" and dilas--abbelonlyy text prompts, respectively. Class-prompt appending also outperforms the simple text prompts by 7.3 mloU. These results indicate that our text prompt selection method can help SD generate datasets with both diversity and accurate attention.
Effects of different components of stage and stage in Fig. 2 on the overall performance are summarized in Tab. 3. Using only cross-attention results in low performance of 44.8 mloU as the
7

---

Generated image
Mask with uncertainty
Mask after well- self-training
Test
Ground -ruth mask
Our predicted mask
parked farixe, bicycle"
"A Aam wallingg down adessals with dog, person dog"
"Alwing with couch, and coffee tablic voda chair dining table"
(a) Our synthetic images and segmentation masks
(b) Segmentation results on VOC's validation set
Figure 5: (a) Row (R1) and R2 are successful cases, while R3 and R4 demonstrate failures. Self- training helps correct mis-segmented objects in some cases (R2 and R3) but can harm the original mask for small objects (R4). (b) RI R3 show accurate results, closely matching the GT. R4 R6 reveal failure cases due to numerous small objects (R4) and intertwined objects (R5 and R6).
Table 3: Impact of cross-attention, self-attention, uncertainty, self-training, and test time augmentation (TTA) (refer to Sec. 3.2, Sec. 3.3). TTA includes multi-scale and input flipping at test time.
Cross-attention Self-attention
Uncertainty
Self-training
TTA
mloU (%)
44.8
61.0
62.0
62.7
64.3
cross-attention map is coarse and inaccurate (as illustrated in Fig. 4). Using self-attention refinement boosts the performance significantly to 61.0 mloU. Also, using other techniques like uncertainty- aware loss, self-training, and test time augmentation help improve performance incrementally.
Effect of different feature scales used for aggregating self-attention and cross-attention maps is shown in Tab.4. As can be seen, for the cross-attention map. choosing too small and too large feature scales both hurt the performance since the former lacks details while the latter focuses on fine details instead of object shape. For the self-attention map, using the scale of 32 gives slightly better results.
Hyper-parameters selection for mask generation (Sec. 3.2). We conduct sensitivity analysis on 7, a, and 3 to determine the optimal values in Tab. 5. Tab. 5a shows the results of choosing (with fixed a = 0.5, 3 0.6) with the best result with T = A too-large value of - = decreases performance as the refined coos-attenttin map tends to spread out the whole image rather than the object only. Additionally, Tab. exhibits the analysis on the (ax, G) range given the fixed r the range of (0.5 0.6) achieves the best performance of 62.0 mloU.

---

Table 4: Study on different feature scales
Table (a) Analysis of with a = 0.5 and 8 = 0.6
for mask generation.
Sell-attention
Cross-attention
32
64
1
2
3
4
5
39.7
38.1
mIoU (%)
44.8
59.5
60.5
60.2
62.0
60.5
62.0
59.6
32
52.8
50.9
(b) Analysis of (a, () given
64
35.4
31.5
a
0.4-0.5
0.5-0.6
0.4-0.6
16, 32
59.7
57.3
16, 32, 64
59.1
57.2
mloU (%)
59.5
62.0
60.7
bown cheins An umumaal looking red bus of beson; person umbrolia chair
down road; porson car bus
close up a call deok near sondw oh; cat apple sandwich
A person on abalsbowe on Fi roact person car skateboard
Figure 6: Left: Correlation maps positions with others, extracted from self-attention map.
Right: Failure cases of SD when generating images with multiple objects. Red: classes are missed.
5 Discussion and Conclusion
While our method is effective for generating synthetic datasets, there are some lim- itations consider. Our primary reliance on Stable Diffusion [5] for image generation can result in difficulties with producing complex scenes. First, when given text prompt that involves three or more objects, the diffusion model may only produce an image depicting two or three objects as exemplified in Fig. 6 Right. However, there is ongoing research to improve the quality of the diffusion model and incorporate stronger guidance, such as layout or box conditions, which shows promise in addressing this issue. Second, it is worth noting that in some cases, our Dataset Diffusion may not produce high-quality segmentation masks when objects are closely intertwined, as seen in Fig.5a Sa with the example of man riding horse. Third, the bias in the LAION-5B dataset, on current
Limitations:
which Stable Diffusion was trained, may transferred to the generated dataset. This limitation of Stable Diffusion as it was trained on a large-scale uncurated dataset like LAION-5B. However, there are several studies addressing the bias problem in generative models [54-56] focusing enhancing fairness and reducing biases in generative models. We believe that these studies and future work on the topic of fairness in GenAI will help to mitigate the bias in the generated images,
We have presented our novel framework Dataset Diffusion which enables the generation of synthetic semantic segmentation datasets. By leveraging Stable Diffusion, Dataset Diffusion can produce high-quality semantic segmentation and visually realistic images from specified object classes. Throughout our experiments, we have demonstrated the superiority of Dataset Diffusion over the concurrent method, DiffuMask, achieving an impressive mloU of 64.8 in VOC and 34.2 in COCO. This remarkable advancement paves the way for future research endeavors focused on the creation of lcrge-scale datascts with precise annotations using generative models.
Conclusion:

---

References
[1] Yuxuan Zhang, Huan Ling, Jun Gao, K. Yin, Jean-Francois Lafleche, Adela Barriuso, A. Tor- ralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. Computer Vision And Pattern Recognition, 2021. 1,3
[2] Nontawat Tritrong, Pitchaporn Rewathowomwong, and Supasorn Suwajanakom. Repurposing gans for one-shot semantic part segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4475-4485, June 2021.
[3] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. Computer Vision And Pattern Recognition, 2022. 1,3
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06 125525 2022. 1,3
[5] Robin Rombach, Blattmann. Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10674-10685, 2021. 3, 4, 6,9
[6] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 11:36499-364999 2022.
[7] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yo Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble expert denoisers. ArXiv, abs/2211.0133 2022. 1,3
[8] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303. 11681, 2023. 2,
[9] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang. Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv:2303.008// 2023. 1,3,4
[10] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision, 11 111(1):98-136 January 2015. 2, 4, 6
[11] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014. 2, 4,6
[12] Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431-3440, 2014. 2
[13] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Loddon Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062,, 2014.
[14] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin P. Murphy, and Alan Loddon Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence,
40:834 848, 2016.
[15] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. ArXiv, abs/ 706.05587.. 2017. 2, 6,
10

---

[16] Liang- Ching-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision, 2018. 2
[17] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, (tbs/1555.0459) 2015.
[18] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:2481 1295, 2015. 2
[19] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Jia. Pyramid scene parsing network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6230-6233 6239, 2016. 2
[20] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence,
43(100:3349-3364 2020. 2
[21] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. Torr, and Li Zhang. Rethinking semantic segmenta- tion from a
perspective with transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6877-6886, 2020. 3
[22] Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Trans- former for semantic segmentation. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 7242-7252, 2021.
[23] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, José Manuel Álvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. ArXiv, 1as/2105.11203. 2021. 3
[24] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 1280-1289, 2021. 3, 6,
[25] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2021.
[26) Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming Yang, Kevin P. Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. ArXiv, abs/2301.007 100004, 2023. 3
[27] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhen- guo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303. 17559, 2023. 3
[28] Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. Advances in Neural Information Processing Systems, 35:8702-8716, 2022.
[29] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390 2021.
[30] Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffusioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.02773, 2022.
[31] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffusion models for implicit image segmentation ensembles. International Conference on Medical Imaging with Deep Learning, pages 1336-1348. PMLR, 2022.
11

---

[32] Minh-Quan Le, Tam V Nguyen, Trung-Nghia Le, Thanh- Toan Do, Minh N Do. and Minh-Trie Triet Tran. Maskdiff:: Modeling mask distribution with diffusion probabilistic model for few-shot instance segmentation. arXiv preprint arXiv:2303.05 105, 2023.
[33] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv: Arxiv- 21 22..03122, 20
[34] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv: Arxiv-2303.04803 2023. 3
[35] Koutilya Pnvr, Bharat Singh, Pallabi Ghosh, Behjat Siddiquie, and David Jacobs. Ld-znet: latent diffusion approach for text-based image segmentation. arXiv preprint arXiv: Arxiv- 23003123333 2023.
[36] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301.0522. 2023.
[37] Raphael Tang, Akshat Pandey, Zhiying Jiang, Gefei Yang, K. V. S. Manoj Kumar, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention. ArXiv, abs/2210.04885, 2022.
[38] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. arXiv preprint arXiv: Arxiv-2006.044 2001-2000.04488, 2020.
[39] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an un- supervised image segmenter in each of your deep generative models. arXiv preprint arXiv: Arxiv-2105.08127, 2021. 3
[40] Shekoofeh Azizi, Simon Korbllihh, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint 2xXiv:2304.08666 2023. 3
[41] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 3
[42] Jianhao Yuan, Francesco Pinto, Adam Davies, Aarushi Gupta, and Philip Torr. Not just pretty pictures: Text-to-imae -o-image generators enable interpretable interventions for robust representations. arXiv preprint arXiv:2212.11237, 2022. 3
[43] Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv: Arxiv-2302.02503, 2023.
[44] Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi-supervised learners benefit mutually with few labels, arXiv preprint 10X5 2023. 3
[45] Jordan Shipard, Amnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Boost- ing zero-shot classification with synthetic data diversity via stable diffusion. arXiv preprint arXiv:2302.0322 2023.
[46] Aniket Roy, Anshul Shah, Ketul Shah, Anirban Roy, and Rama Chellappa. Difffalig:: Few-shot learning using diffusion based synthesis and alignment. arXiv preprint arXiv:2212.05404, 2022.
[47] Jordan Shipard, Amold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Boost- ing zero-shot classification with synthetic data diversity via stable diffusion. arXiv preprint arXiv:2302.03298 2023. 3
[48] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.087744 2023.
12

---

[49] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- lainual understanding and generation. In International Conference Machine Learning, pages 12888-12900. PMLR, 2022. 4, 6
[50] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. JCML, volume 3, page 896, 2013. 6
[51] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference on computer vision, pages 991-998. IEEE, 2011.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. ArXiv, abs/1912.01703, 2019.
[53] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: gi thub. wwwm/ppen-mmmll/mmeeeeeeeeeeeeee hwm.mmaammmmmmmtttt...n... 2020.
[54] Felix Friedrich, Patrick Schramosski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint arXiv:2302. /0893, 2023. 9
[55] Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to- image generation. arXiv preprint arXiv:2308.000 2023.
[56) Xingzhe Su, Wenwen Qiang. Zeen Song, Hang Gao, Fengge Wu, and Changwen Zheng. Manifold-guided sampling in diffusion models for unbiased image generation. arXiv preprint a7771:2307.08119, 2023. 9
13

---

6 Supplementary Material
In this supplementary material, we first elaborate on our implementation details Sec. 6.1 Then, we present additional results from our ablation study in Sec. 6.2, focusing on two key aspects: the impact of timesteps on aggregating attention maps and the influence of the number of generated images on the segmentation results. Next, we provide comprehensive per-class IoU using PASCAL VOC test set and COC02017 val set in Sec. 6.3, After that, we present the quantitative results of Dataset Diffusion in other image domains in Sec. 6.4. Finally, we provide more qualitative results in Sec. 6.5.
6.1 Elaboration of Implementation Details
Other texf prompt selection methods. In this part, we discuss how we implement other text prompt selection methods in the first ablation study in the main paper. We note that the difference between these methods and Dataset Diffusion lies in the prompt construction and the cross-attention aggregation. For the 'simple text prompts' method, we construct prompts as "A photo of [class name]" with the cross-attention map at the token "[class name]". For the 'caption only' method, the class labels are not appended to the captions as Dataset Diffusion with the cross- attention at the class category token. In this way, the terms that do not match exactly with the class names, c.g., "aeroplane" vs. "airplanc"), will be ignored. For the "class labels only" method, we use the prompts "[class name 1] [class name 2]
with the cross-attention at the class tokens. For compound class names such as "dining table", we take the mean of the cross-attention maps at all positions.
More on Sec 3.1. We have observed a major issue with Stable Diffusion. It often fails to generate an adequate number of target classes when multiple object classes are provided in the text prompts, particularly in the case of COCO. To avoid this issue, we introduce a limit on the number of class labels, denoted as k. If the number of classes M in an image exceeds this threshold, we select the top-k classes based on their frequency. Then, we generate simpler text prompts, each containing only one class from C, classes of image i, ranked by the least frequent classes. The simple text prompt used is "a photo of a [class name]; ; [class name]". This approach ensures that the generated images are more faithful to the provided text prompts, and it facilitates the creation of more diverse set of text prompts that includes both simple prompts and real captions. This strategy helps enhance the quality and coverage of the synthetic dataset, mitigating the issue of missing target classes in the generated images.
Computation details. We conduct our experiments on NVIDIA A100 40G GPUs. Generating a 40k image dataset with Stable Diffusion V takes about 30 hours and training the DeepLabV3 for 20k iterations takes about 3 hours single GPU.
6.2 Additional Ablation Studies
Study on differrnt ranges of timesteps to aggregate attention maps. In the main paper, we only experiment with aggregating self-attention and cross-attention maps over all the timesteps. Here, we provide an ablation study in Tab. 6 to show that the variation in timesteps has minimal impact on the results. As can be seen, averaging all the timesteps yields the best performance, and the maximum decrease is only 0.5 mloU.
Study numbers of generated image-mask pairs in the synthetic dataset. We evaluate the performance of our semantic segmenter on three numbers of generated images: 10k, 20k, and 40k. The results are shown in Tab. 7. By using four times larger training dataset (from 10k to 40k), we only gain 1.0% in mloU. We expect the gain to be smaller when increasing the number of training images further since the system performance reaching its limit. Because the gain small compared to the additional computation cost, we do not examine our system when using more than 40k images. We further experiment with the pseudo masks generated using a semantic segmenter well-trained on the real Pascal VOC 2012 dataset to make predictions on our synthetic images. We consider them highly accurate pseudo masks. The results Tab. show that the performance of the semantic segmenter training on 40k images with these masks yields 1. mloU gain compared to when training on 10k images, suggesting this small gain comes solely from the mask quality, not from increasing number of generated images.
14

---

Table The impact of different ranges of timesteps to aggregate attention maps.
Timesteps
100-88
88-75
75-50
50-0
100-0
mloU (%)
61.9
61.5
61.5
61.8
62.0
Table 7: The impact of different numbers of generated images in the synthetic dataset.
# images
10k
20k
40k
mIoU (%)
63.8
64.3
64.8
Table 8: Different # generated images with masked produced by DeepLabV3 trained on real data.
images
10k
20k
40k
mloU (%) 71.2
71.6
72.6
6.3 Detailed Per-class IoU the VOC and COCO datasets
The detailed per-class loUs of the COCO 2017 dataset are shown in Tab. 9. Furthermore, Tab. 10, report the mloU on the VOC fest set of each class when training Lab3 dataset and the Pascal VOC training set. We observe that for the class "tv/monitor", the cross-attention frequently focuses on the boundary of the monitor rather than capturing the entire objects, and the self-attention map cannot help alleviate this issue, resulting in poorly generated masks (as illustrated in Fig. 7). This behavior could explain the significant performance drop of that class compared the results trained on the real dataset.
synthetic
6.4 Other Image Domains
Driving Scenes (Tab. 11): Our synthetic dataset exhibits performance gap with real datasets (3k real images manual labeling) of approximately 19 mloU. This discrepancy parallels those observed in the VOC and COCO datasets, as previously discussed in our main paper. Notably, our synthetic dataset performs comparably with real datasets for specific classes such as bicycle, car, and person. The process of generating images for this dataset is similar to those of VOC and COCO.
Facial Part Segmentation (Tab. 12): performance gap of around 16.5 mloU (for 5k images) exists between real and synthetic datasets. This disparity is reasonable, considering our approach's 's zero-shot nature, where training images are generated based text prompts. Compared to the faces generated in DatasetGAN whose training images are from CelebA-HQ-Masce the faces generated by Dataset Diffusion are often not well-alin well-aligned. However, Dataset Diffusion still competes favorably (78.2 vs 87.0). Furthermore, DatasetGAN requires a number of manual- labeled images (20 in this case) for generating new images and segmentations. Hence, for a fair comparison, we combine our synthetic data with 20 labeled images, and DatasetDiffusiio surpasses DatasetGAN (89.9 vs. 87.0). An example of text prompt used "A portrait photo of a young woman; hair eye nose ear mouth" where hair, eye, nose, ear, mouth are parts" names.
Satellite/Aerial Images (Tab. 13): Dataset Diffusion's generated image quality and segmentations do not align with DroneDeploy's 's standards, leading to mloU gap of 37.4. This discrepancy stems from the limited presence of aerial/satellite images in SD's s training set (LAION-5B). The text prompts hinder the generation of images that match DroneDeploy's style. The results are presented with the prompt: "An aerial view of a [building, clutter, water, vegetation, ground, car]" and 15k images.
6.5 More Qualitative Results
We first demonstrate the qualitative results of the segmenter on the VOC val set in Fig. and COCO 2017 val set in Fig. 9. Next, in Figs. 10, 11, and 12, we present further qualitative results to illustrate the effectiveness of Dataset Diffusion in generating scenes with different complexities. Fig. 10, simple prompt with a single object is depicted, and Dataset Diffusion performs remarkably well
15

---

Table COCO 2017's per-class IoU with DeepLabV3 trained on COCO's training set (118k images) and Dataset Diffusion (100k images).
Class
COCO
Dataset Diffusion
Class
COCO
Dataset Diffusion
background
89.1
80.9
wine glass
48.8
29.3
person
80.6
50.5
cup
44.8
13.7
bicycle
64.7
39.9
fork
18.0
11.1
53.1
35.5
knifc
19.0
6.1
motorcycle
76.9
62.6
spoon
15.8
7.7
airplane
78.0
62.2
bowl
38.0
24.4
bus
77.2
64.8
banana
62.0
49.0
train
77.2
apple
40.1
34.9
truck
52.8
36.8
sandwich
40.9
19.4
boat
53.1
42.5
orange
64.4
48.4
traffic light
60.5
31.0
broccoli
51.0
36.8
fire hydrant
84.6
55.9
carrot
41.8
22.1
stop sign
90.5
42.9
hot dog
49.8
30.9
parking meter
68.3
51.0
pizza
67.8
44.1
bench
46.8
27.5
donut
62.7
48.3
bird
66.9
47.2
cake
51.3
30.1
79.9
64.0
chair
34.3
10.5
dog
74.5
46.5
couch
52.3
29.5
horse
77.4
60.5
potted plant
26.1
4.5
sheep
79.7
63.1
bed
58.3
42.0
cow
75.0
63.9
dining table
39.6
14.6
clephant
87.4
76.6
toilet
70.0
51.3
bear
88.4
78.1
63.8
6.3
zebra
88.2
81.8
laptop
65.8
42.5
giraffe
82.5
77.0
60.3
12.7
backpack
23.1
7.5
remote
48.6
21.2
umbrella
69.3
51.2
keyboard
55.5
32.6
handbag
18.9
0.04
cell phone
57.1
35.7
tie
6.7
4.6
microwave
59.9
39.1
suitcase
65.8
29.8
oven
50.1
17.8
frisbee
55.8
29.1
toaster
0.5
6.4
skis
22.4
7.2
sink
54.9
21.4
snowhoard
48.7
21.9
refrigerator
70.3
20.1
sports ball
42.1
24.5
book
37.3
23.2
kite
46.7
34.2
clock
64.0
24.3
baseball bat
23.1
8.3
vase
48.0
39.4
baseball glove
59.8
scissors
59.1
25.1
skateboard
45.3
22.0
teddy bear
70.8
46.1
surfboard
60.2
43.4
hair drier
0.08
2.6
tennis racket
72.6
15.7
toothbrush
36.6
21.9
bottle
37.5
20.1
mIoU (%)
54.9
34.2
in this scenario. In Fig. 11, more complex scenes are shown, and Dataset Diffusion still produces acceptable results. However, the limitations of Stable Diffusion become apparent Fig. 12, where generated scenes are complex but often lack the objects explicitly mentioned in the prompt. This highlights the challenges Stable Diffusion faces when generating scenes with multiple objects.
16

---

Table 10: VOC's per-class IoU with DeepLabV3 trained VOC's training and Dataset Diffusion.
Class
VOC
Dataset Difflusion
Class
VOC
Dataset Diffusion
aeroplane
93.6
81.6
diningtable
58.9
42.9
bicycle
43.8
35.8
dog
89.7
71.8
bird
93.4
73.3
93.4
78.2
boat
67.0
62.2
motorbike
90.9
80.6
bottle
78.5
72.6
person
87.1
70.8
bus
95.9
85.5
pottedplant
68.2
53.9
car
90.7
64.8
sheep
90.9
77.8
cat
94.9
78.2
sofa
58.7
41.8
chair
36.8
21.6
train
84,9
72.7
cow
89.4
69.2
tymonitor
74.3
29.6
mloU (%)
79.8
64.6
Figure 7: The poorly generated masks for the class "tv/monitor".
Table 11: Cityscapes Object Segmentation Results
Training set
Bicycle
Bus
Car
Motorbike
Person
Train
mloU
Cityscapes
78.0
89.7
95,4
69.8
82.5
84.6
83.3
Dataset Diffusion
72.9
57.7
93.7
39.8
82.1
39.9
64.4
Table 12: CelebA-HQ-Mask Facial Part Segmentation Results
Training
Samples
Hair
Eye
Nose
Bar
Mouth
mloU
CelebA-HQ-Man
5k labeled
98.9
98.0
99.5
77.9
99.4
94.7
Dataset Diffusion
5k synthetic
97.2
74.5
85.1
41.2
93.1
78.2
DatasetGAN
5k synthetic 20 labeled
97.5
95.3
96.6
48.1
95.7
87.0
Dataset Diffusion
5k synthetic 20 labeled
97.9
95.3
97.2
61.5
97.6
89.9
Table 13: DroneDeploy Results
Training set
Building
Clutter
Vegetation
Water
Ground
Car
mloU
Depnoyeploy Dataset Diffusion
47.8
32.3
64.4
73.9
84.6
65.5
61.4
21.9
13.6
28.8
10.1
58.7
11.0
24.0
17

---

Figure 8: Segmentation results on VOC's val set. Three separate columns show examples of simple scenes with single object, quite complex scenes with multiple objects, and very complex scenes with intertwined objects, respectively. In each example from left to right: test image, ground truth segmentation, and our predicted mask.
18

---

Figure 9: Cegmenniaiion results on COCO 2017. Please refer to the caption of Fig. 8 for more details.
19

---

photo keyboard; keyboard
A photo of mouse; mouse
photo of orange; orange
A photo of oow; cow
photo
remote;
Figure 10: Dataset Diffusion generate high-quality image-mask pairs with simple text prompt containing a single object. Each row shows two examples with text prompts underneath. From left to right: generated image, initial mask with uncertain regions in white, and final mask after self-training.
Two cowboy hats the
motoreycle the rain;;
hanging boy teaching
placing green
bowl with nuts and bananas neat to bunch bananas;; bowl banana
man that le walking across a concrete pad: person horee
A man with glaases eating a alice of pizza person pizza
A yellow and blue plane on rumway; person airplane
Dataset Diffusion still works well with some scenes containing multiple objects. Please refer to the caption of Fig. 10 for more details.
Figure 11:
20

---

A pile of boxes topped with ripe bananes.; person banana chair
large herd cows being led down the road by people; person horse cow
citting
small dog cathing fricbee field; friebee
very
It. hydrant stop sign
An ozen standing in front door next to bicycle; bicycle bench cow
Figure 12:
When the prompt becomes excessively complex, Stable Diffusion faces challenges in generating accurate images, i.e., not including all the objects mentioned in the caption, resulting in the absence of these objects' masks. Please refer to the caption of Fig. 10 for more details.
21


Reviews:
[{"review_id": "79mXKRNC5i", "reviewer": "Reviewer_gEQM", "summary": "This work is about automatically generating synthetic data for training semantic segmentation models. Such synthetic data includes realistic input images along with their corresponding ground truth semantic masks. The authors employ text-to-image diffusion models (Stable Diffusion in this work) and propose a specific prompting and ground truth mask generation. The input prompts for Stable Diffusion are a concatenation of an image-level caption (given or generated) as well as the list of category names contained in an image. The ground truth segmentation masks are extracted by combining the self-attention (image tokens) and cross-attention (image and class name text tokens) maps. The evaluation is done on two datasets, Pascal and COCO, and shows superior accuracy compared to one concurrent work (DiffuMask). The ablation study demonstrates the (positive) impact of all aspects of the proposed framework.", "questions": "**Questions:**\n- The feature scales in Table 4 look sensitive. Do you get a similar conclusion for other datasets?\n\n**Suggestions:**\n- L35: I think it would be useful to the reader if some high-level context is given on how DiffuMask (or the proposed method) achieves generation of segmentation masks.\n- L147: Shouldn't $M$ rather be $M_i$ if it is image-dependent?\n- L148: Does concatenating two strings really need a method name like \"text appending operation\" or \"class-prompt appending technique\"? It's a very basic operation.", "limitations": "Yes, limitations have been addressed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The general problem of generating synthetic training data for segmentation models from vision-language models is interesting and promising.\n- The proposed prompting for Stable Diffusion is simple but effective.\n- The paper is well written and easily comprehensible. The figures give a good overview and also explain each of the proposed aspects of the work well.\n- All aspects of the proposed solution are evaluated in the ablation study.", "weaknesses": "- L123: Unfortunately, I think creating synthetic data is most useful for exactly those applications that are not based on everyday scenes. There are plenty of datasets with everyday scenes already available, like Pascal, COCO or ADE20K.\n- Figure 2 and L133: Doesn't using captions from VOC and COCO bias the whole system and make it unfair when evaluating on those same dataset? Other methods that do not leverage these captions may be at a disadvantage. Using only generated captions would be fine, though. It would have been great to see the difference in final accuracy for ground truth captions and generated captions.\n- Table 1: The comparison to prior work is lacking. There is only a single comparison to a prior/concurrent work. Did DiffuMask [8] also use self-training? And why is there no comparison to DiffusionSeg [9]? Why is there no comparison to GAN-based methods for synthetic dataset generation? Is there a way to use the evaluation protocol from [8, 9] for a comparison?\n\n**Post-rebuttal:**\nI acknowledge that I read all reviews and the author's feedback. The author's feedback clarified several of my concerns and I raised my rating to \"borderline accept\". I'd love to see a discussion about the limitations (and potentially some supporting numbers) in the paper, as discussed in the comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I think creating synthetic data is most useful for exactly those applications that are not based on everyday scenes. There are plenty of datasets with everyday scenes already available, like Pascal, COCO, or ADE20K.** \\\n**A1:** Thanks for your question. We want to clarify that our approach support all kinds of image domains a text-to-image model, i.e., Stable Diffusion can generate. In the paper, we present the experiments on everyday scenes like VOC and COCO since we only have the ground-truth semantic segmentation from this image domain for evaluation. Furthermore, even for the everyday image domain, there are still some cases such as imbalance data distribution, long-tail distribution of object categories, or rare classes that these datasets cannot represent but our approach can generate synthetic datasets for these cases very well as exemplified by some examples of rare classes in Fig. 2 of the global attached pdf. We also select 5 rare classes in the LVIS dataset to train a semantic segmenter on limited real training images (30 images in total) and a sufficient synthetic set (250 images with 50 images each) and report the results in Tab. A below. Our synthetic data generation is an effective tool for tackling rare classes.\n\n**Table A: Results on five rare classes of the LVIS dataset**\n|  |horse buggy | garbage | hippopotamus | dice | mallet | mIoU\n|:-------- |:--------: |:--------:|:--------:|:--------:|:--------:|:--------:|\n| Training set of LVIS | **84.1** | 19.6 | 39.7 | 42.9 | 21.9 | 41.6\n| Dataset Diffusion | 82.6 | **68.9** | **63.8** | **64.2** | **43.0** | **64.5**\n\n**Q2: Figure 2 and L133: Doesn't using captions from VOC and COCO bias the whole system and make it unfair when evaluating on those same dataset? Other methods that do not leverage these captions may be at a disadvantage. Using only generated captions would be fine, though. It would have been great to see the difference in final accuracy for ground truth captions and generated captions.**\\\n**A2:** Thanks! We consider two kinds of generated captions: from an image captioner and from an LLM like ChatGPT. For the former, we already tested it in VOC experiments since we do not have GT captions for this dataset. For the latter, we show the results on VOC in Tab. B below with generated 40k prompts using ChatGPT. We observe no considerable differences in performance between the two generated captions. \nAn example of the text prompt generated by ChatGPT is \"A yellow compact car driving through a city next to buses; car, bus\". On COCO, we are the first to report the results of using synthetic data to train a semantic segmenter, thus, no comparison is available. It's worth noting that our proposed evaluation protocol based on the image caption is aimed to be a standard benchmark for future work in this direction. That is, we primarily focus on better segmentation techniques rather than on better prompt engineering with ChatGPT. \n\n**Table B: Results in mIoU (%) on the VOC test set with caption generated by BLIP and text prompt generated by ChatGPT with DeepLabV3 without TTA and self-training**\n\n| Source captions | ResNet50 | ResNet101 |\n|:-------- |:--------:|:--------:|\n| Captions from BLIP | 58.5 | 62.2\n| Prompts from ChatGPT | 58.3 | 61.2\n\n\n**Q3: Table 1: The comparison to prior work is lacking. There is only a single comparison to a prior/concurrent work. Did DiffuMask [8] also use self-training? Why is there no comparison to DiffusionSeg [9]? Is there a way to use the evaluation protocol from [8, 9] for a comparison?** \\\n**A3:** It's worth noting that we are one of the first works exploring the direction of generating a synthetic dataset for semantic segmentation using a text-to-image diffusion model. Other concurrent works like [8, 9] are published on Arxiv and have not published their code yet. Among them, only DiffuMask [8] can be directly compared to ours and we follow their evaluation setting on VOC. [8] also reported results with the self-training strategy on VOC. On the other hand, Diffusionseg [9] focuses on saliency detection instead and did not release code Therefore, we cannot compare our approach with them.\n\n**Q4: Why is there no comparison to GAN-based methods for synthetic dataset generation?** \\\n**A4:** GAN-based approaches like DatasetGAN [1], BigDatasetGAN [3], [38], and [39] are tested on different tasks such as part segmentation and keypoint detection in [1], and single object segmentation or saliency detection in [3], [38], and [39]. Also, it's not trivial to modify their codes to work with multiclass semantic segmentation like in VOC or COCO. Therefore, we cannot compare directly with these GAN-based approaches. However, we still try our best to adapt [39] to work with VOC and report the results in Tab. C, where 2 classes \"person\" and \"horse\" are excluded. The results from the table demonstrate the superior performance of our approach (a diffusion-based method) over the GAN-based approach for multiclass semantic segmentation.\n\n**Table C: Comparison results with [39] on 18 VOC classes**\n|Method|mIoU|\n|:--------:| :--------:|\n|[39] (GAN-based)|20.3|\n|Dataset Diffusion|**62.5**|\n\n**Q5: The feature scales in Table 4 look sensitive. Do you get a similar conclusion for other datasets?** \\\n**A5:** Yes, we also get a similar conclusion on COCO with 20k captions. We conduct this ablation study with the same setting as Tab. 4. We also achieve the best results when using a cross-attention map at 16 resolution and a self-attention map at 32 resolution as shown in Tab. D below. Hence, the results are consistent among VOC and COCO.\n\n**Table D: Study on different feature scales on COCO**\n|Cross-attention|Self 32|Self 64|\n| :--------: | :--------: | :--------: |\n| 8        |16.2    | 15.5\n| 16       |**25.1**    | 23.9\n| 32       |21.6    | 21.7\n| 64       |15.7    | 15.2\n| 16,32    |24.2    | 23.5\n| 16,32,64 |23.7    | 23.8\n\n**Suggestions**: Thanks so much, we will revise the main paper accordingly"}, {"review_id": "1yoIf90K0V", "reviewer": "Reviewer_KgKS", "summary": "The paper addresses the problem of training data preparation for machine learning tasks using generative models. The paper proposes a way to generate pixel-level semantic segmentation dataset using Stable Diffusion. Given a set of target classes, chatGPT produces input text prompts that along with real captions are used to prompt stable diffusion which can sample images. To generate the corresponding segmentation maps, the self-attention map is used to refine the cross-attention map  for each of the target classes. The estimated segmentation masks are then used to train a semantic segmentation network using uncertainty aware loss and self-training methodology.\nThe efficacy of the method is presented by experiments using PASCAL VOC and COCO dataset. The existing datasets are further enriched by including the captions generated from BLIP. Comparisons are made by training DeepLab V3 and Mask2Former on the real dataset, synthetic dataset from DiffuMask, and the proposed methodology for generating the dataset. Ablations along impact of different design choices is clearly represented in Table 3, along with feature scale in table 4, and hyperparameters for defining the uncertainty in the generated mask. ", "questions": "1. Can the uncertainty parameters also be conditioned on the input text prompt and the generated segmentation mask using existing segmentation data and its augmentation?\n2. Minor suggestion: Please add a few more examples to Figure 6 as there is still some space. ", "limitations": "Yes. Sufficient discussion was included.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well presented in terms of motivations first, followed by the technical details defined in detail to reproduce the results. The idea of using Stable Diffusion as a data source is not novel as presented in StableRep and instructPix2Pix, though the urrent paper lays out the method to use its intermediate activations for generating synthetic segmentation masks for training. The comparisons are promising and furthermore the ablations provide enough justifications for the design choice. The method is well described to the extent where it is sufficient for reproducing the results. The presented results are commendatble and in-line with other results on the use of synthetic datas for downstream tasks and representation learning [StableRep]. I am positive about the the use of stable diffusion priors for etracting useful training data. While the results in thsi paper don't beat training with real dataset from COCO and PASCAL VOC, but it the results point to the promise of synthetic data and priros of generative models.\n", "weaknesses": "1. The method requires a fixed set of test classes to be defined beforehand. This is a limitation as the method cannot be used to generate segmentation masks for unseen classes or extend it in an open-vocabulary manner [See OpenSeg].\n2. I would like some more discussion on why self-attention has the information to further refine the cross-attention map. It seems like because of the NxN structure of the self-attention map, understanding the process that leads to refinement would be helpful for the reader.\n3. How sensitive is the method qualitatively to the choice of the hyperparameters for defining the uncertainty in the generated mask? It would be helpful to see how the uncertainty parameters impact the segmentation masks qualitatively.\n4. Since the model is heavily dependent on quality of data in LAION dataset, the bias in the data probably transfers to generated dataset? A section in the main paper or supplementary about some of the known biases in the generated dataset would be helpful. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The method requires a fixed set of test classes to be defined beforehand. This is a limitation as the method cannot be used to generate segmentation masks for unseen classes or extend it in an open-vocabulary manner [See OpenSeg]**.\\\n**A1:** Since our method is one of the first works using text-to-image model to generate images and semantic segmentation, we evaluate with the standard semantic segmentations setting and have not considered the unseen classes or open-vocab setting. Moreover, given the synthetic dataset, we can treat it as a real dataset and apply SOTA approaches on unseen classes or open-vocab settings such as OpenSeg. We believe they will totally works fine. However, it is out of the scope of this paper and it might make paper more complicated and cluttered since we focus on the quality of generated synthetic dataset.\n\n**Q2: I would like some more discussion on why self-attention has the information to further refine the cross-attention map. It seems like because of the NxN structure of the self-attention map, understanding the process that leads to refinement would be helpful for the reader.**\\\n**A2:** We really appreciate your comment. Cross-attention maps capture the correlation between each position of the latent representation and the tokens of the text embedding. However, the cross-attention map only capture salient parts of the object and ignore non-salient ones. In these cases, the self-attention maps with the ability to capture the pairwise correlations among positions within the latent representation can help propagate the initial cross-attention maps to the highly similar positions, e.g., non-salient parts of the object, thereby enhancing their quality. Also, we provide an illustration of correlation maps extracted from self-attention maps in Fig. 1 in the global attached PDF.\n\n**Q3: How sensitive is the method qualitatively to the choice of the hyperparameters for defining the uncertainty in the generated mask? It would be helpful to see how the uncertainty parameters impact the segmentation masks qualitatively.**\\\n**A3**: Thanks! We provide qualitative results with different hyperparameters for defining the uncertainty in generated masks in the Fig. 3 and Fig. 4 of the global attached pdf.\n\n**Q4: Since the model is heavily dependent on quality of data in LAION dataset, the bias in the data probably transfers to generated dataset?**\\\n**A4**: We really appreciate your insight. Yes, the bias in the LAION dataset may be transfered to the generated dataset. This is the current limitation of Stable Diffusion as it was trained on a large-scale uncurated dataset like LAION. However, there are several studies addressing the bias problem in generative models:\n+ **Seshadri, Preethi, Sameer Singh, and Yanai Elazar. \"The Bias Amplification Paradox in Text-to-Image Generation.\" arXiv preprint arXiv:2308.00755 (2023)**: examines bias amplification in text-to-image generation, focusing on gender biases. \n+ **Friedrich, Felix, et al. \"Fair diffusion: Instructing text-to-image generation models on fairness.\" arXiv preprint arXiv:2302.10893 (2023)**: mainly discusses biases related to genders and human behavior.\n+ **Su, Xingzhe, et al. \"Manifold-Guided Sampling in Diffusion Models for Unbiased Image Generation.\" arXiv preprint arXiv:2307.08199 (2023)**: proposes a method to estimate the data manifold from the training data. The data manifold is then used as a constraint to guide the sampling process in diffusion models that can mitigate the general data bias.\n\nWe believe that with these studies and future work on the topic of fairness in GenAI will help to mitigate the bias in the generated images. We will include this discussion in the revised version.\n\n**Q5: Can the uncertainty parameters also be conditioned on the input text prompt and the generated segmentation mask using existing segmentation data and its augmentation?**\\\n**A5**: If we understand your question correctly, you meant that can we have an adaptive threshold for each text prompt rather than a fixed threshold? Yes, we can adapt the threshold with the given text prompt. However, it is not straightforward to do it without significant modification such as a network to predict the adaptive theshold given the text prompts and produced segmentation map. To train the network, it requires a separate dataset which is also non-trivial effort. Therefore, we opt to use a fixed threshold for all text prompts as proposed. However, we still think your idea is intersting and worth more future work on it. Thanks!\n\n**Q6: Minor suggestion: Please add a few more examples to Figure 6 as there is still some space.** \\\n**A6**: Thank you for the suggestion.\n"}, {"review_id": "IFvgghwUDT", "reviewer": "Reviewer_2Ket", "summary": "The present paper sims to solve the issue of expensive annotation in dense prediction tasks through generating of synthetic images and masks by utilizing frozen a frozen stable diffusion model. First, the authors use the captions extracted from the BLIP or COCO original dataset to generate images by stable diffusion, then combine self-attention and cross-attention maps to produce semantic masks. Finally, the segmenter is supervised by the generated masks with an uncertainty-aware operation, and subsequently self-trained on pseudo labels predicted by the segmenter after first-stage training. This work introduces a simple yet effective method of utilizing text-driven stable diffusion to synthesize images and masks, as opposed to relying heavily on labour-intensive annotations, yielding impressive results on VOC and COCO validation datasets.", "questions": "Questions:\n    1. This work lacks sufficient comparisons of finetuning on real data after training on synthesized data which has been provided in DiffuMask. \n    2. The key contribution of this work lies in reducing annotation costs, yet it significantly increases computational costs during training. It would be more convincing to see further comparisons with other generative or self-supervised methods on semantic segmentation tasks in terms of both performance and efficiency.\n    3. Is Table 1 a fair comparison with the DiffuMask method? Were self-training and TTA applied in the results of this method?\n    4. Since the stable diffusion is totally fixed, how about the results of crossing datasets?\n    5. The submission could benefit from meticulous proof-reading and clearer claims. For instance, there are inconsistencies between the values in Table 1 and those reported in the ablation study. Additionally, could the authors specify the type of segmenter and backbone used in the ablation study?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n    1. This work applies the pretrained stable diffusion model to generate synthesized VOC and COCO datasets, which also enables evaluation of the diffusion model’s capability for real-world scenes generation and generalization. This approach could substantially reduce the annotation cost in semantic segmentation task while achieving impressive performance on VOC and COCO validation dataset. \n    2. For text prompt generation, this work uses the original captions from COCO dataset and leverages BLIP model to generate captions for VOC, and further introduces a calibration operation to address issues of mismatched and missing class names. Compared with DiffuMask which primarily focus on generating one, this work can generate more complex scene encompassing multiple categories.\n    3. For synthesized semantic masks generation, this work ensembles the attention maps of both self-attention and cross-attention. Additionally, they employ uncertainty-aware segmentation loss to alleviate unconfident parameters updates.", "weaknesses": "Weaknesses:\n\tThis work proposes a simple but effective strategy to leverage text-driven stable diffusion however lacks novelty. Generating data via diffusion models might introduce computational burdens but holds the potential for effectively addressing the problem of imbalanced data distribution. Moreover, the submitted version lacks of experiments, as detailed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: This work proposes a simple but effective strategy to leverage text-driven stable diffusion however lacks novelty.** \\\n**A1:** Our proposed method introduces a simple and unique mask generation process by combining self-attention maps powered to $\\tau$ then multiplied with cross-attention maps. To the best of our knowledge, no prior work has explored this particular technique. Therefore, it cannot be considered as lack of novelty.\n\n**Q2: This work lacks sufficient comparisons of finetuning on real data after training on synthesized data which has been provided in DiffuMask**\\\n**A2:** We provide the comparisons of fine tuning on real data after training on synthesized data in Tab. E below. We conduct this experiment with Mask2Former segmenter and ResNet-50 backbone. Training directly on 5k real data yielding 77.0 in mIoU (\\%), while pre-tranining on synthetic data generated by DiffuMask with 60k images then fine-tuning on 5k real data yields a very marginal result of 77.6 in mIoU (\\%). \nThus, once the real data provided, it is not necesarry to pretrain on synthetic data as proposed in one of the experiments of DiffuMask. Therefore, we decide not to include any real data in our approach or evaluation to merely study the quality of the synthetic data. However, to fulfill the request, we also run Mask2Former pretrained on our synthetic data and obtain slightly better results than that of DiffuMask.\n\n\n**Table E: Comparisons of fine-tuning on real data after pre-training on synthetic data.**\n|Pretrained on synthetic data| Real data | mIoU (\\%)\n| :--------: | :--------: | :--------:\n| Not used |VOC 5k| 77.0 |\n|DiffuMask (60k images) |VOC 5k|77.6 |\n|Dataset Diffusion (40k images) |VOC 5k | **78.0** |\n\n\n**Q3: The key contribution of this work lies in reducing annotation costs, yet it significantly increases computational costs during training.**\\\n**A3**: Our approach only employs pretrained Stable Diffusion without retraining or finetuning it, so it is a quite affordable and fully-automated process to generate synthetic data. In contrast, real dataset requires  manual data collection and annotation, which is arguably cost more to have same number of annotated images as synthetic dataset. Furthermore, by employing synthetic data, we can effectively address the issue of imbalanced data distribution as suggested by the reviewer.\n\n\n**Q4: It would be more convincing to see further comparisons with other generative or self-supervised methods on semantic segmentation tasks in terms of both performance and efficiency.**\\\n**A4:** For other generative models like GAN, please refer to A5 of reviewer *gEQM*. For self-supervised methods, we provide the comparison in Tab. F below. It can be seen that our synthetic data generation approach significantly outperforms self/un-supervised approaches using real images with a very large margin. \n\n\n**Table F: Comparisons with other self-supervised methods on VOC2012 *val set***\n| Method|mIoU (\\%)| \n| -------- |:--------:| \n|CLIP$_\\text{py}$ ViT-B|54.6\n|MaskDistill+CRF|48.9\n|Leopart|47.2\n|MaskDistill|45.8\n|Dataset Diffusion (ours)|64.8\n\n    \n**Q5: Is Table 1 a fair comparison with the DiffuMask method? Were self-training and TTA applied in the results of this method?**\\\n**A5**: In Table 1, we ensure a fair comparison with DiffuMask [8] because both methods train the segmenter with only pure synthetic data. In addition, DiffuMask did use the self-training, however, TTA was not mentioned. To address this potential discrepancy and ensure a comprehensive analysis, we present additional results without employing TTA in Tab. G below. As can be seen, we still outperform the version of DiffuMask with a margin of 2.0 mIoU even when not using TTA.\n\n**Table G: Comparative results without using TTA.**\n\n|Training set|Segmenter|Backbone|mIoU\n| -------- | -------- | -------- |:--------:\n| VOC|DeepLabV3|ResNet50|76.2\n| VOC|DeepLabV3|ResNet101|78.7\n| Dataset Diffusion|DeepLabV3|ResNet50|59.9\n| Dataset Diffusion|DeepLabV3|ResNet101|63.1\n| DiffuMask|Mask2Former|ResNet50|57.4\n| Dataset Diffusion|Mask2Former|ResNet50|59.4\n\n\n**Q6: Since the stable diffusion is totally fixed, how about the results of crossing datasets?**\\\n**A6**: Our method is a dataset-agnostic approach like DiffuMask, hence we only depends on given class names only. Therefore, training with our synthetic data then evaluating on different datasets such as COCO or VOC can be considered as cross-dataset evaluation already. \nNevertheless, we also provide the results of cross-dataset evaluation following the setting of DiffuMask [8] for completeness purpose in Tab. I of A8 of reviewer *wdax*.\n\n\n**Q7: The submission could benefit from meticulous proof-reading and clearer claims. For instance, there are inconsistencies between the values in Table 1 and those reported in the ablation study. Additionally, could the authors specify the type of segmenter and backbone used in the ablation study?**\\\n**A7:** Thanks! To clarify, the ablation study was conducted using 20k images without self-training and test-time augmentation (TTA) unless explicitly stated in the experiment (L247) whereas the main results presented in Table 1 were achieved by utilizing 40k captions, self-training, and TTA. We used DeepLabV3 segmenter and ResNet101 as backbone for the ablation study."}, {"review_id": "Wqv7Ul4bRV", "reviewer": "Reviewer_wdax", "summary": "- The paper introduces a method of generating synthetic training data using Stable Diffusion (SD) for semantic segmentation.\n    \n- The class labels are appended to captions which is then used as text prompt to SD to generate synthetic image.\n    \n- The segmentation map is generated by refining the cross-attention map (using only the class name as text prompt) using the exponentiated self-attention map.\n    \n- The generated segmentation masks with uncertainty regions are used as pseudo-labels for training segmentation models.\n    \n- Self-training is performed and the resulting model is evaluated using Test Time Augmentation", "questions": "- is TTA used when evaluating other methods as well?\n    \n- visualization of self-attention maps would be good to have\n    \n- [8] shows results on open vocabulary segmentation and domain generalization as well, which would make the work more comprehensive", "limitations": "Limitations have been discussed in the paper.   \n\n- limited to domains/classes for which stable diffusion can generate images\n    \n- limited by the complexity of images stable diffusion can generate\n    \n- segmentation masks obtained from the method can be noisy/low-quality\n    \n- amount of synthetic data possible to generate depends on the inference speed of SD", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Use of self-attention to improve the obtained segmentation mask\n    \n- Including uncertain regions in the generated segmentation masks is a good idea as they are extracted from attention maps and hence not high-quality\n    \n- Ablation of different components in Table 3 helps understand relative contributions (though more details on the ablation experiment setup would be helpful for the reader)\n    \n- Visualization of failure cases", "weaknesses": "- L107: “Their text-prompts inputs to SD are simple” - which is inaccurate as the other methods also explore different ways of prompting. Similar to this work, the use of ChatGPT to generate prompts is also explored in [9]\n    \n- Finetuning on a small amount of real data improves the performance of [8] significantly and it reaches mIoU higher than training on real data only. Similar experiment here would show how a small amount of real data can be leveraged\n    \n- Ablation: simple text prompt with all class labels (i.e. Row1 + Row3 in Table 2)\n    \n    - In Table 2, the biggest boost seems to come from using all class labels\n        \n    - If simple text prompt with all class labels work well, we could do away with the time consuming extra step of using ChatGPT, BLIP, for generating complex prompts\n        \n- The idea of using self-attention for improving the segmentation mask obtained from cross-attention is also explored in [9] -- not referenced in text when discussing\n    \n- Missing comparison with [9]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: L107: “Their text-prompts inputs to SD are simple” - which is inaccurate as the other methods also explore different ways of prompting. Similar to this work, the use of ChatGPT to generate prompts is also explored in [9].**\\\n**A1:** What we meant by saying these text-prompt inputs to SD are simple is that only a single object for an image is considered in these prompts, and not simple in the prompt construction. Thank you for pointing out, we will make it clearer in the revised version.\n\n**Q2: Finetuning on a small amount of real data improves the performance of [8] significantly and it reaches mIoU higher than training on real data only. Similar experiment here would show how a small amount of real data can be leveraged.**\\\n**A2:** Thanks for your comments. Actually, our finding is opposite as shown in Tab. E of reviewer *2Ket*. Pretraining on synthetic dataset does not help much in the presence of real data. It is not to mention, when real data are available, we can do other strategies to improve the performance with small labeled images such as few-shot learning or semi-supervised learning. Thus, we decided not to involve any real data in our approach and evaluation. \n\n\n**Q3: Ablation: simple text prompt with all class labels (i.e. Row1 + Row3 in Table 2). In Table 2, the biggest boost seems to come from using all class labels. If simple text prompt with all class labels work well, we could do away with the time consuming extra step of using ChatGPT, BLIP, for generating complex prompts**\\\n**A3:** Yes, using all class labels can boost the performance significantly compared to just using the simple prompts. However, there is still a very big gap between all class labels alone and combining them with captions (57.4 vs 62.0 in mIoU).\n\n\n**Q4: The idea of using self-attention for improving the segmentation mask obtained from cross-attention is also explored in [9] -- not referenced in text when discussing**\\\n**A4:** Thanks for pointing out, we will include in the revised version. DiffusionSeg [9] uses self-attention and cross-attention to generate segmentation mask, however, in a very different way from our approach. In particular, self-attention is employed for constructing the pixel-wise objectness and pair-wise affinity. Given that information, the energy function for each mask can be defined. The mask is then generated by minimizing that function using an off-the-shelf graph cut algorithm. In contrast, our method is much simpler by just multiplying powered self-attention with cross attention to obtain refined cross attention.\n\n\n**Q5: Missing comparison with [9]**\\\n**A5:** Please refer to A3 of reviewer *gEQM*.\n\n\n**Q6: is TTA used when evaluating other methods as well?**\\\n**A6:** Please refer to A5 of reviewer *2Ket*.\n\n\n**Q7: visualization of self-attention maps would be good to have**\\\n**A7:** Please refer to A2 of reviewer *KgKS*.\n\n\n**Q8: [8] shows results on open vocabulary segmentation and domain generalization as well, which would make the work more comprehensive**\\\n**A8:** Thank you for your question. We agree that results on open vocabulary segmentation and domain generalization would make our work more comprehensive in terms of comparison with previous work [8].\nFor open-vocabulary segmentation, experiments in [8] (Tab. 3) is conducted as follows: training on synthetic dataset of 20 classes of VOC and testing on all 20 classes and split the results into two groups: 15 seen classes and 5 unseen classes. It is not the standard setting where the model is trained on seen classes and tested on unseen classes. For completeness, we provide the results of our approach on the same setting as [8] in Tab. H. \n\n\n**Table H: Zero-shot evaluation** \n| | Segmenter | Seen | Unseen | Harmonic\n| :--------: | :--------: | :--------: | :--------: | :--------:\n| DiffuMask 60k | Mask2Former | 60.8 | 50.4 | 55.1\n| Dataset Diffusion 40k | Mask2Former | **62.7** | **50.9** | **56.2**\n\nFor domain generalization or cross-dataset setting, our method is a dataset-agnostic approach, as we only depend on given class names only. Therefore, training with our synthetic data then evaluating on different datasets such as COCO or VOC can be considered as cross-dataset evaluation already. Since [8] has not released the code yet, we conduct the following similar experiments. For each dataset in Tab. I, we extract the subset of six shared classes between VOC and Cityscapes, i.e., bicycle, bus, car, motorbike, person (human + rider), and train. Results in Tab. I suggest that our generated data could achieve competitive results on VOC and Cityscapes dataset, compared to using other cross-domain training data.\n\n**Table I: Cross-dataset evaluation**\n|Train set|Test set| bicycle | bus | car | motorbike | person | train |mIoU|\n| -------- | -------- | :--------: | :--------: | :--------: | :--------: | :--------:| :--------: | :--------:\n| Cityscapes|VOC|61.3|73.2|57.2|69.8|89.9|61.2|68.7\n| Dataset Diffusion|VOC|75.9|92.3|88.8|87.1|93.6|92.1|88.3\n| VOC|Cityscapes|74.5|48.4|94.8|49.8|87.5|19.0 |62.3\n| Dataset Diffusion|Cityscapes|71.2|44.7|92.0|27.8|79.2|5.0 | 53.3"}]

Author rebuttal:
We thank all reviewers for their positive feedback. All the reviewers recognize the problem of generating synthetic training data for semantic segmentation from text-to-image diffusion model is interesting and promising. In addition, they also find that our paper is well-written and easy to follow. Moreover, reviewers *2Ket* and *wdax* praise the proposed approach incorporating self-attention and cross-attention maps to generate the semantic masks is simple but effective, can generate more complex semantic segmentation than DiffuMask. They also agree that employing uncertainty-aware segmentation is a good idea to alleviate the imperfect semantic masks. Furthermore, reviewers *gEQM*, *KgKS*, and *wdax* also compliment our comprehensive ablation study. Also, *gEQM* and *2Ket* also appreciate our proposed simple but effective prompting for Stable Diffusion. Finally, *wdax* comments that we have a good visualization of failure cases. Below, we address other comments point by point. 

Meta-review: The reviewers found the paper to be well written, include comprehensive controlled experiments, and presenting a simple & novel approach. They were mostly concerned about the dependence on the initial dataset for training the underlying diffusion model and how it might affect performance on rare classes. Another concern was the missing comparison to GAN-based approaches. Concerns by the reviewers were sufficiently addressed in the extensive rebuttal. The reviewers and the AC appreciated the additional experiments and results. Adding them in combination with a discussion of limitations (i.e. performance depends on how well SD captures the respective domain) will strengthen the paper substantially and push the paper over the acceptance threshold.
Decision: Accept (poster)