DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
Authors: Weijia Wu
Yuzhong Zhao
Hao Chen
Yuchao Gu
Rui Zhao
Yefei He
Hong Zhou
Mike Zheng Shou
Chunhua Shen
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model; Text-guided dataset generation

Abstract: Current deep networks are very data-hungry and benefit from training on large-scale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, we present DatasetDM, a generic dataset generation model that can produce diverse synthetic
images and the corresponding high-quality perception annotations (e.g., segmentation masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module. Training the decoder only needs less than 1% (around 100 images) of manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models on downstream tasks. To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic15
segmentation, instance segmentation, and depth estimation. Notably, it achieves 1) state-of-the-art results on semantic segmentation and instance segmentation; 2) significantly more efficient and robust in domain generalization than the real data; 3) state-of-the-art results in zero-shot segmentation setting; and 4) flexibility for efficient application and novel task composition (e.g., image editing)

DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
Weijia Wu1.3
Yuzhong Zhao²
Hao Chen
Yuchao Gu
Rui Zhao
Yefei He
Hong Zhou
Mike Zheng Shou
Chunhua Shen
'Zhejiang University, China
Uniiversity of Chinese Academy Sciences, China
SShw Lab, National University of Singapore
*Ant Group
Abstract
Current deep networks are very data-hungry and benefit from training on large- scale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, present DatasetDM, a generic dataset generation model that can produce diverse synthetic images and the corresponding high-quality perception annotations (e.g., segmenta- tion masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as ac- curate perception annotations using decoder module. Training the decoder only needs less than % (around 100 images) manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models for downstream tasks.
To showcase the power of the proposed approach, dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation. Notably, it achieves (1) state-of-the-art results on semantic segmentation and instance segmentation; (2) significantly more robust domain generalization than using the real data alone; and state-of-the-at results in zero-shot segmentation setting; and (3) flexibility for efficient application and novel task composition (e.g., image editing).
generate datasets with rich
The project website is at: weijiawu, ieiiaauu.gttubb io/ MooDatasetDMM
Introduction
Modern deep-learning models for perception tasks often require a large amount of labeled data to achieve good performance. Unfortunately, collecting large-scale data and labeling the corresponding pixel-level annotations is time-consuming and expensive process. For example, collecting images of urban driving scenes requires physical car infrastructure. and labeling segmentation annotation for a single urban image in Cityscapes [10] can take up to 60 minutes. Moreover, in certain specialized domains, such medical or human facial data, collecting relevant information can be challenging even impossible, owing to privacy concerns or other factors. The above challenges can be a barrier to advancing artificial intelligence in computer vision.
To reduce costs, many previous researchers have primarily focused on weakly supervised [57 and
unsupervised solutions [51] to address the problem. For instance, certain segmentation priors 22 32
use weak or inexpensive labels to train robust segmentation models. With the advancement of
H. Zhou and M. Shou are the corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Spaleic lmngs
Sanark Scpnerates
Inance Segrosssion
Duphi Exirnice
Fabls: Scraanaiie
Ope-Voechday Segaszzion Trassfomer
Image
Perception Annotations
TES
Ibooieether SOTA
0.31
Draliillaio
Derbect
201TAPT)
Trainin Set
Imagee
Iaaga
300
<50
Figure 1: Synthetic Data from DatasetDM. The high-quality and infinitely numerous synthetic images with perception annotation can yield significant improvements for various downstream tasks.
generative models, such as DALL-E [0]. Stable Diffusion 41 some researchers have begun to explore the potential of synthetic data, attempting use it assist in training models, or
replace real data for perception task. Most works focus on classification, face recognition [34] 49 (2],
salient object detection [54] and segmentation tasks 31 53 58), with only minority trying to
address problems such as human pose estimation [88 or medical image analysis [21] In the era of
29 are recognized as pioneering works that utilize the feature space of pre-trained GANs and design a shallow decoder for generating pixel-level labels in the context of segmentation tasks. Following the two works, Ha (55) extends this approach to multitasking scenarios, such as depth estimation. However, these methods still suffer from three major drawbacks: 1) Due the limitations of the representation ability of early (up to 2020) GAN models, the quality of the synthesized data is often dissatisfactory, leading to an inferior performance on downstream tasks. 2) These methods primarily focus on independent downstream tasks and no one tries to explore a unified data synthesis paradigm with a generalized decoding framework. 3) The training cost is still relatively high, while these methods do not make full use of the visual knowledge contained within the latent codes of powerful text-to- text-to-image models.
GANs, DatasetGAN [58] and BigDatasetGAN
By leveraging large-scale datasets of image-text pairs (e.g., LAIONSB 455). recent text-to-image diffusion models (e.g., Stable Diffusion (41] present phenomenal power in generating diverse and high-fidelity images with rich texture, diverse content, and reasonable structures. The phenomenon suggests that large text-to-image diffusion models can implicitly learn valuable, and rich high-level and low-level visual representations from massive image-text pairs. It is natural ask: Can we leverage the knowledge learned by these models to generate perception annotations and extend the paradigm of text-to-image generation text-to text-to-data generation?
In this paper, built upon the powerful text-to-image diffusion model, we present DatasetDM, a generalized dataset generation model that can produce unlimited number of synthetic images and the corresponding perception annotations, as shown in Fig. [ The key to our approach is a unified perception decoder, namely P-Decoder, that decodes the latent code of the diffusion model to perception annotations. Inspired by align//nstruct tuning from L.M [50], a method for inducing output following capability with minimal human-labeled, we only use less than 1% manually labeled images to train the decoder, enabling infinite annotated data generation. The generated data can subsequently be utilized train any perception models for various downstream tasks, including but not limited to segmentation, depth estimation, and pose estimation. To maintain the robust image generation capabilities of the diffusion model, we freeze the model weights and use image inversion to extract the latent code, i.e., multi-scale features, which are then fed into the unified P-Decoder. The P-Decoder accommodates various downstream tasks within a unified transformer framework. with only minor variations as depicted in Fig. 3
To summarize, our contributions are four-fold:
We introduce DatasetDM: a versatile dataset generation model featuring a perception decoder capable of producing unlimited quantity of high-fidelity synthetic images, along with various perception annotations, including depth, segmentation, and human pose estimation.
Visual align/instruct tuning, method for inducing output following capability with minimal human-labeled data. With less than 1% of the original dataset, i.e., around 100 images,

---

DatasetDM pushes the limits of text-to-ing -iaae generation, pioneering novel paradigm: text-to-data generation. This breakthrough is made possible by leveraging the rich latent code of the pre-trained diffusion model.
Experiments demonstrate that the existing perception models trained on synthetic data generated by DataseiDM exhibit outstanding performance on six datasets across five different downstream tasks. For instance, the synthetic data delivers remarkable gains of 13.3% mloU and 12.1% AP for semantic segmentation on VOC 2012 and instance segmentation on COCO 2017, respectively.
Text-guided data generation allows for the generation of a diverse range of data, which has been shown provide more robust solution for domain generalization and long-tail data distribution. Moreover, DatasetDM offers a flexible approach for novel task composition, as exemplified by its ability to facilitate image editing (see Fig.
2 Related Work
Text-to-Image Generation. Several mainstream methods exist for the task, including Generative Adversarial Networks (GANs)[I],, Variational Autoencoders (VAEs) flow-based models [[2), and Diffusion Probabilistic Models (DPMs) 47
25 20]. Recently, as a likelihood-based method, diffusion models have gained significant attention with promising generation abilities. They match the underlying data distribution by learning reverse noising process. Thanks to the high-qu high-qualtyy image synthesis and the stability of training, diffusion models are quickly emerging as a new in the field of image synthesis.
41
frontier [41 40 26 43
Text- sutdeg diffusion models are used for text-conditional image generation, where a text prompt T guides the generation of content-related images J from random Gaussian noise 2. Visual and textual embeddings are typically fused using cross-attention. Recent large text-to-image diffusion models, such as Stable Diffusion [41] of Stability AI, DALL-E2 44) of OpenAI, and Imagen (33 of Google, have shown powerful performance in generating diverse and high-fidelity images with rich textures and reasonable structures. Their impressive synthesis capabilities suggest that these models can implicitly learn valuable representations with the different semantic granularity from large-scale image-text text pairs. In this paper, we leverage these learned representations (latent codes), to extend the paradigm of -ext-to-image generation to text-to-data generation.
Synthetic Datasets for Perception Task. Previous studies in dataset synthesis, such as Virtual KITTI 17 and Virtual KITTI 2 77, primarily rely on 3D computer simulations to address standard 2D computer vision problems, such as object detection (38], scene understanding 44,, and optical flow estimation (6) However, these methods are limited by the domain of 3D models and cannot be generalized to arbitrary scenes and open-set categories. For instance, Virtual KITTI is exclusively focused on autonomous driving scenes and supports only 20 commonly occurring categories, which cannot be extended to open-scene domains like the COCO benchmark (33].
In contrast, synthetic data generated using generation models (i.e., GAN 119 35 55 and Diffusion Model 47) are more flexible and can support a wider range of tasks and open-world scenes for various downstream tasks, such as classification task 22 face recognition 22. salient object
detection [54], semantic segmentation [29 58 3 60). and human pose [18]. Inspired by the success of large-scale generative models, such as Stable Diffusion
41 trained on massive datasets like LAION5B 45] recent studies have begun to explore the potential of powerful pre-trained diffusion generative models. Based on DDPM 25 DatasetDDPM
3 design several CNN layers as the annotation decoder to generate semantic segmentation data by performing a multi-class task. Li et al. [31] utilized Stable Diffusion and Mask R-CNN pre-trained on the COCO dataset [33] to design and train a grounding module for generating images and semantic segmentation masks. DiffuMask (53] produces synthetic image and semantic mask annotation by exploiting the potential of the cross-attention map between text and image from the text-supervised pre-trained Stable Diffusion model. The above methods focus on semantic segmentation, which cannot handle tasks such as instance segmentation. In this paper, we take a further step by utilizing generalized perception decoder to parse the rich latent space of the pre-trained diffusion model, enabling the generation of perception for a variety of downstream tasks, including depth, segmentation, and human pose.
Diffusion Model for Perception Task. Some recent works
has also attempted to directly employ diffuuson models for perceptual tasks. VPD [59] explores the direct application of pre-trained
[59 56 52
3

---

Stable Diffusion to design perception models. ODISE [56] unify pre- te-irrined text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Different from these approaches, our focus lies in synthetic data augmentation for perception tasks, and design unified transformer-basee decoder to enhance more perception tasks from data aspect.
3 Methodology
3.1 Formulation
Given a language prompt 8, text-guided diffusion models generate content-related images J xxx from a random Gaussian noise 2 ~ N).. The standard text- rext-guided image denoising processing can be formulated as: 7 = (221(2, S), where (rar(-) refers to a pre-trained text-to-image diffusion model. In this paper, we adopt Stable Diffusion [41] as the base for the diffusion model, which consists of three components: a text encoder (-)-) for embedding prompt 8; a pre-trained
variational autoencoder (VAE) [4] that encodes E(. and decodes DD-) latent code of images; and a coneccondiiionll UNet (ca(.) (-) 42 that gradually denoises the latent vectors. To fuse visual and textual embeddings, cross-attention layers are typically used in the UNet for each denoising step. The denoising process is modeled as a Markov chain: 2t-1 = f(x,, E8). where D; denote latent code at timestep i, and E [1, 77. The latent noise at the final timestep T, denoted ær, is equivalent the random Gaussian noise z. f-) is the denoising function [25].
In this paper, we design perception decoder that can effectively parse the latent space of the UNet <,(x,, L, (8)). By doing so, we extend the lext-to-imuge generation approach to text-to-data paradigm:
[J, 11:k) 112 8),
(1)
where IIA denotes the corresponding perception annotations, and is the number of the supported downstream tasks. In fact, the paradigm can support any image-level perception task, such as semantic segmentation, instance segmentation, pose estimation, and depth estimation.
3.2 Method Overview
This paper introduces novel paradigm called text-to-data generation, which extends text-guided diffusion models trained on large-scale image-text pairs. Our key insight is that using a small amount of real data (using less than 1% existing labeled dataset) and a generic perception decoder interpret the diffusion latent spaces, results in the generation of infinite and diverse annotated data. Then the synthetic data can be used to train any existing perception methods and apply them to real images.
The proposed DatasetDM framework, presented in Fig. 2 comprises two stages. 1) The first stage- using diffusion inversion ($3.3) obtain the latent code of the real image and extract the text-image representations ($3.3). These representations and their corresponding annotations then used to train the perception decoder ($3.4). 2) The second Inferncnee ($3.5) uses GPT-4 to enhance the diversity of data and generates abundant images, while the P-Decoder produces corresponding perception annotations such masks and depth maps.
3.3 Hypercolumn Representation Extraction
The first step in the training stage of DatasetDM is extract the hypercolumn representation of real images from the latent space of the diffusion model, as shown in Fig. 2[a). To achieve this, we employ the diffusion inversion technique, which involves adding certain level of Gaussian noise to the real image and then extracting the features from the UNet during the denoising process.
Image Inversion for Diffusion Model. Give real image X " from the training set, the diffusion inversion (diffusion forward process) a process that approximates the posterior distribution q( (zo)) where 2p = E(X). This process is not trainable and is based fixed
Markov chain that gradually adds Gaussian noise to the image, following pre-defined noise schedule
BI...., Br (25
(x,1111-1) = N(x;; VI 1.

---

Preezt "hoto of and noloriike
Text Eneader
Concat
Notio
Crows Amtention Mays
P- Decoder
L
Uncadon
X
Pereeption
Dillision Phozos
Dansidng U-Nar (hr)
Few Images
MuttiSaale Fesure Maps
(4) Trinning D with few manually labeled images (less than 1% original data)
OpenAl
moddrn
Percegtion Annctations
Cross Attesticn Steph
Denuiiing L-hett
Goussiun Noose (Ge, X,)
Mali-Scalc
D
DecodenVVA
(b) Generating Image and corresponding Perception Annotation
Infinite Syntheric Emegges
Figure The overall framework of Dat DatasetDM consists of two main steps: Training. Using diffusion inversion to extract the latent code from small amount of data and then train the perception decoder. 2) Text-guided data generation. large language model such as GPT-4 is utilized to prompt infinite and diverse data generation for various downstream tasks.
where f represents the 1-th time step, and we set it to for single forward pass in paper. A single forward pass for visual representation extraction usually provides two advantages, i.e., faster convergence and better performance (56).
Texe-Image Representation Fusion. With the latent code æ, of the real image and the corresponding language prompt 8, we extract the multi-scale feature maps and cross attention maps from the UNet Eg as follows:
(F.A] = cg(x,, To()))
where 8 for training is simply defined using template "a photo of a [CLS] [CLS] ...". During the data generation phase, GPT-4 is used to provide diverse prompt languages. The variable F denotes the multi-scale feature maps from four layers of the U-Net, corresponding to four different resolutions, i.e., 8 x 8, 16 x 16, 32 x 32, and 64 x 64, as illustrated in Fig. 2 Additionally, represents the cross artsstattntionn maps of text-to-image from the 16 cross-attention layers the U-Net,
QK" where d is the latent projection dimension. We Va
which implement the function A softmax
group the 16 cross-attention maps into groups with the same resolutions, and compute their average within each group, which results in the average cross- catss-attntiion maps A.
have proved the effectiveness of class-discriminative and localization of cross-attention map between visual embedding and the conditioning text features. Thus we concatenate the cross-attention maps A and the multi-scale feature maps to obtain the final extracted hyper-column representation, and further use a 1 x convolution to fuse them: F Conv([F,,A]]
Prior works 53 59 24
3.4 Perception Decoder
The P-Decoder is utilized to translate the representation 5 into perception annotations, which are not limited to specific type for each downstream task. To achieve this goal, inspired by previous works g 61. we devised a generalized architecture. This architecture is depicted Fig. [] with only minor variations (i.e., whether startup some layers) for each downstream task. For example, the pixel decoder and transformer decoder are required for generic segmentation, while only the pixel decoder necessary for depth and pose estimation.
Generic Image Segmentation. In Fig. 3 (a), we present the adaptation for semantic and instance segmentation tasks, which includes two components: the pixel decoder and the transformer decoder.
5

---

Sceard weweeeaeeeeeee
Pota Podissc
Adcasse bligs
Adcmtic Mups
Transdomner Decode:
Copy Lagar
Transformnr Decoddrr
Corry Eager X+ Ubllappl
plappl
Fiod Deoakr Fodter Mays
Piai Deoolrr
Piud
Tust Eascda
Learabli Queact
Temollle Qatan
Cangrey:
(a) Segmeentation
(b) Open- Vocabulary Segreniatioon
(c) Depth/Poss Pose Estimalion
Figure 3: Various types of tasks with our proposed P-Decoder. The proposed decoder is generalized architecture for the six supported tasks, with only minor variations required for different downstream applications, ie., determining whether to activate certain layers.
Similar to Mask2former [8] the transformer decoder comprises stack of transformer layers with cross-attention and self-attention. This component refines the queries and renders the outputs. The pixel decoder made up of multiple upsampling CNN layers, and it used to obtain per-pixel embeddings. Given the representation & and N learnable queues (Qo, Q1..Qut J as input, the decoder outputs N binary masks O 01. ON E {), 1 along with the corresponding category. This is achieved through simple matrix multiplication between the outputs of the transformer decoder and the pixel decoder. Following Mask2former [8] one query is responsible for class instance for semantic and instance segmentation, respectively. To optimize the mask prediction, we use the binary cross-ents cessspy loss [9] and dice loss 33 On the other hand, the cross-entropy loss is used for classification prediction.
Segmentation. Based on the architecture of generic segmentation, semantics- related query concept is proposed to support the apen-vocabulary segmentation task, as shown in Fig (b.. During the training stage, given the category name (Co, Ck...Cr] (e.g., dog, cat) on an image from the training set, we use the text encoder re(-), e.g., CLIP [39] to encode them into the embedding space. Then concatenating them with the queries to equip the semantics of class vocabulary to the query as follows:
Q = (MLP((QQ,7o(C)))
where Q; and Cj is the i-th query embedding and j-th class. MLP refers to a learned MLP, used to fuse the class embedding and learnable query embedding. Thus DatasetDM can generate an open-vocabulary mask by incorporating new class name, as illustrated in Fig. (b).
Depth and Pose Estimation. For depth and pose estimation, the output format is predetermined, eliminating the need to differentiate between classes or instances. In this context, the pixel decoder is only required to predict fixed number of maps O E MM xxx The value of M is set either or 17 (corresponding to 17 human key points), depending on whether the task is depth or human pose estimation. For human pose estimation, we use the mean squared error as the loss function, and the ground truth heatmaps are generated by applying 2D Gaussian with standard deviation of pixel centered on each key point. As for depth estimation, we update the loss function from the classic scale-invariant error 330 13
3.5 Prompting Text-Guided Data Generation
In Fig. (b), we present the inference pipeline for text-guided data generation. There two main differences compared to the training phase: firstly, the prompts come from a large language model instead of a fixed template, and secondly, the denoising process is extended to 7' steps to obtain the synthetic images. Large language model, i.e., GPT-4, is adopted to enhance the diversity of generative data, while recent works [16, 23 4 have proven their powerful understanding and adaptability for the real world. As shown in Fig. 4 instead of the template-based based prompts from humans, we guide GPT-4 to produce diverse, and infinite prompts. For different downstream tasks and datasets, we give different guided prompts for GPT4 For example, as for the urban scene of Cityscapes [10], the simple guided prompt is like 'Please provide 100 language descriptions of urban driving scenes for the Cityscapes benchmark, containing a minimum of 15 words each. These descriptions will serve as guide for Stable Diffusion in generating images." In this approach, we collected L

---

Collaboration between the creative thinker, the painler, the annotator and editor
Human: Please grovide multiple deseciptions For the appearance of one person with a focus on their elottin, pants, and shoes. These descriptions will serve is guide
for Stable Dicfusion in generating correspandigg images.
GPT4 (creative thinker). Hare 1000 examples
Stable Diffusionn (painte)
Kienn shopp gnay suit, enio wnite dirrt, ane bock polshed shate
soniitentt
carsiiingg of neodie, alrefi fi jeans und pair of freelly aneaties,
A young man in cesual culit strals donn the sieesil,, weating Jean.
devnn jockel a graphic lec, paree
Datsse (onmotator)
Non (afiter)
Human Pose
Dopth
Mask
Instance
DeepFashion
Whine
city dtroet
Figure 4: Collaboration between GPT-4 and Diffusion Model. Large language models, e.g., GPT-4, can provide diverse and unrestricted prompts, enhancing the diversity of generated images.
Zero-Sbot Segmentation
Long-Tail Segmentation
Seen Class
Unseen Class
Head Class
Tail Class
aerplanne (0), bicycle (i), bird (2), boar (3),
potted plant (15), shoep (16), sofa (17), train (18),
acroplanc (0), bicycle (I), bird
diningtable (10), dog (1)), horse
bottle (4), bus (5), car (6), cat (7), chair (8),
(2). boat (3), bottle (4), bus (5),
(12). metorbike (13), person (14), car (6), (7), chair (8), cow
(9), diningtable (10), dog (11), horse tymonitor (19)
potted plant (15), sheep (16). sofa (17), train (18), tymonitor (19)
(12), matorbike (13), person (14)
Table 1: Details for Sero-Shot and Long-tail Segmentation VOC 2012 [15.
text prompts, which average around 100 prompts for each dataset. For each inference, a random
prompt is sampled from this set.
4 Experiments
4.1 Implementation details
Architecture and Training Details. Stable diffusion V1 model pre-trained on the LAIONSB [5 dataset is used as our text-to-image diffusion model. The decoder architecture of Fark2Former g was selected as the base architecture for our P-Decoder. And we use 100 queries for the segmentation task. For all tasks, we train DatasetDM for around 50k iterations with images of size 512 x 512, which only need one Tesla V100 GPU, and lasted for approximately 20 hours. Optimizer [36] with learning rate of 0.0001 is used.
Downstream Tasks Evaluation. To comprehensively evaluate the generative image of DatasetDM, we conduct seven groups of experiments for the supported six downstream tasks. Semantic Segmenta- tion. Pascal-VOO 2012 [5] (20 classes) and Cityscapes [T (19 classes), as two classical benchmark are used to evaluate. We synthesized 2k images for each class in both datasets, resulting in a total of 40k and 38k synthetic images for Pascal-VOc 2012 and Cityscapes. respectively. The synthetic data is subsequently utilized to train Mask2former g and compared to its real data counterpart on a limited dataset setting (around 100 images). Instance Segmentation. For the COC02017 (3 benchmark, we synthesized 1 images for each class, resulting in total of 80k synthetic images. Similarly, Mask2former [8), as the baseline, is used to evaluate the synthetic data. We evaluate only the class- performance, where all the 80 classes are assigned the same class ID. Depth Estimation. We synthesized a total of 80k synthetic images for NYU Depth V2 [46). And using Depthformer [30] to evaluate our synthetic data. Pose Estimation. We generated a set of 30k syn- thetic images for COCO2017 Pose dataset [33] and employed HRNet [48] as the baseline model to assess the effectiveness of our approach. Zero-Shot Semantic Segmentation. Following Li et al. BI], Pascal- VOC 2012 [3] (20 classes) is used to evaluate. We train DatasetDM with only 15 seen categories, where each category including 30 real images, and synthesized total of 40k synthetic images for 20 categories. Long-tail Semantic Segmentation. The categories of VOC 2012 are divided into head (20 images each class) and tail classes (2 images each class). Then we train DatasetDM with these data and generate synthetic data. Human Semantic Segmentation. We synthesized a total
7

---

GPT4:
with bise
yellow sports cor driving on brightly street
careee whine car and grzy alog 200093
Urban Domain
Zero Shot Segmentation Iron Man
stims light with joots and tow-lop down city street
(c) Human-Centric Domain
Figure 5: Examples of generated data for Dat Our method can produce semantic/instance segmentation, depth, and human pose annotation across various domains.
VOC (Semantic Seg.
COCO2017 (Instance Seg.
NYU Depth V2 (Depth Est.)
COCO2017 (Pose Est.
mothod
# real
synth.
mloU
# real
# synth.
AP
#real
# #sytth
REL
#raal
#synth.
AP
Baseline
100
65.2
400
14.4
50
0.31
800
42.4
DatasetDM
26.5
Table 2: Downstream Tasks. 'real' and "synth.' denote real and synthetic images, respectively. The backbones of baselines for four tasks are 'Swin-B', 'Swin-B', 'Swin-L', and 'HR-W32', respectively.
of 20k synthetic images for DM synthetic data. We split Dee into a set of 100 training images, and 12,500 testing images. Further details, such prompts, can be found in the supplementary material.
27 (24 classes). Mask2former is used to evaluate
g
Class Split for Zero-Shot and Long-Taie Tail Segmentation. Table[[povvdess a comprehensive overview of the class distribution for both zero-shot and taig-tail scenarios. The division for zero-shot classes consistent with previous studies firstly established in this paper.
The configuration for long-tail data distribution
4.2 Main Results
Table 2
provides basic comparison of the selected four downstream tasks. More additional exper- iments can be found in Table and Table 4 as well as in the Supplementary Material (i.e., Pose Estimation, Depth Estimation, Zero-Shot Segmentation, Fashion Segmentation, and others).
Semantic Segmentation. Table g displays the performance on VOC 2012. Using only 100 real images (5 images per class), training with purely synthetic data from DatasetDM achieves 73.7% mloU, an 8.5% improvement compared to using real data alone. Moreover, when jointly training with the 100 real images, further improvement is observed, resulting in mloU of 78.5%. displays the performance on Cityscapes [11]. To compare with previous methods [55], we also conducted experiments with 9-classes division. DatasetDM demonstrates consistent advancements
displays
the baseline or prior SOTA, achieving up to a 10% improvement in mloU under each experimental setup.
Instance Segmentation. Table presents three distinct training settings, encompassing variations in the backbone and the number of training images. Regardless of the chosen setting, DatasetDM consistently achieves improvement of approximately 10%. Employing 800 training images (10 images per class) and the Swin-B backbone, DatasetDM yields a 12.1% increase in Average Precision (AP), resulting in final AP of 26.5%.
Depth Estimation. Table 2 presents a concise comparison between synthetic and real data on NYU Depth V2 dataset [46] Detailed information (e.g., backbone, other metrics) can be found in the supplementary material. When trained with images, DatasetDM can achieve 10% improvement compared to training solely with real images.

---

method
backbons
real image
# syuthetic image
AP
Apt
Ap'S
AP3
APM
AP
12.1
DataselDM
R50
80% (R:4000)
12.2
24.3
10.9
1.6
11.3
30.9
DatasetDM
R50
400
80k (R:400)
14.8
29.7
13.0
2.3
15.1
36.0
Baseline
Swin-B
400
11.3
23.0
9.6
3.2
10.1
27.1
DatasetDM
Swin-B
80k (R:400)
17.6
34.1
15.8
3.4
17.8
39.5
DatasetDM
Swin-B
400
80k (R:400)
23.3
43.0
22.2
7.7
26.1
48.7
Baseline
Swin-B
800
28.8
3.6
29.2
DataselDM
Swin-B
(R:800)
Table 3: Instance segmentation on COCO va12017.
R:
denotes the real data used to train.
Sampled Classess for Comparison/S
method
backbone
imuge
synihetic imuge
dog
Baseline
R50
100
54.8
53.3
69.3
66.8
24.2
43.4
DataseDDM (ours)
R50
40k (R:100)
84.7
74.4
86.0
79.2
63.7
60.3
DataseLDM (ours)
R50
100
40k (R:100)
81.7
82.3
87.7
77.9
69.3
66.1
Baseline
Swin
100
54.4
68.3
86.5
71.8
49.1
65.2
DatasetDM (ours)
Swin-B
40k (R:100)
DatasetDM (ours)
Swin-B
100
D (ours)
Swin-B
(R:100)
76.
DatasetDM (ours)
Swin-B
40k (R:100)
93.8
92.3
78.5
Baseline
Swin-B
10.6k (full)
93.7
96.5
90.6
88.6
95.7
84.3
DataseDDM (ours)
Swin B
10.6k (full)
40k (R:100)
93.9
97.6
91.9
89.4
96.1
85.4
Table 4: Semantic segmentation on VOC 2012.
'R: refers to the number of real data used to train.
Human Pose Estimation. For the human pose estimation task on the COCO2017 dataset, DatasetDM demonstrates significant improvements compared to the baseline trained on 800 real images, achieving 5.1% increase, illustrated in Table Similar depth estimation, additional information can be found in the supplementary material.
Zero Shot and Long-tail Segmentation. Table displays the results of experiments related to zero- shot and long-tail segmentation. Our model, DatasetDM, notably alleviates the effects of long-tail distribution by synthesizing substantial amount of data for rare classes, leading an improvement of up to 20% in mloU, Details for both tasks can be found in the supplementary material.
4.3 Ablation studies
Zero-Shot Setting
Long dnil Setting
Method
harm.
head
tail
mloU
95
Baseline(no Syn.)
61.3
10.7
18.3
61.2
44.1
52.6
er [1
55.7
DiffuMask 53
71.4
65.0
68.1
mloU
85
DatasetDM
78.8
60.5
68.4
73.1
66.4
70.0
80
Table Zero Shot and Long-tail Segmentation on VOC
2012. Zero Shot: following priors [3I (3). train DatasetDM with only 15 seen categories, and tested for 20 categories. Long-tail Setting: the 20 categories are di- vided into head (10 classes, 20 images each class) and tail classes (10 classes, images each class).
70
15
Word Number of Prompt
Figure Ablation of Prompt Length.
Diffusion Time Steps. Table/6a the influence of visual features derived from various diffusion time steps, and the maximum sampling step is 1000. We observe that large step results in adverse outcomes, whereas the performance with a smaller step tends to remain relatively stable.
Cross-Attention Fusion. Fig. 6b demonstrates that the cross attention maps, F in step I can yield modest improvement, roughly 1%. Interestingly. it appears that as the step size increases, the benefit becomes less pronounced. Indeed, when the step size surpasses 500, it may even result in detrimental effects. From this perspective, the utility of the cross-attention map is limited.
Training Set Size. Additional training data for DatasetDM can further improve synthetic data, as shown in Table (c) The increase of the training data from 60 to 400 images precipitates the most conspicuous improvement, subsequently reaching saturation point. With Ik training images, the

---

8 Classes/%
19 Classes/%
method
hackhone
# real image
synthetic image
vehicle
mloU
bus
bicycle
mloU
Baseline
R50
9
100k+ (R:16)
84.3
71.5
82.8
22.3
42.4
36.8
HandsOff 55
R101
16
100k (R:16)
55.1
HandsOff 65
R101
30
100k (R:16)
60.4
D (burs)
R50
38k (R:9)
86.9
69.5
83.3
8.3
53.5
34.2
DatasetDM (burs)
R50
9
38k (R:9)
88.6
76.7
85.6
28.9
$6.5
42.1
DatasetDM (ours)
R101
9
38k (R.9)
88,9
77.5
85.9
27.9
60.4
43,7
Baseline
Swin-B
84.1
74,5
83.3
27.7
DatasetD (ours)
Swin-B
(R:9)
85.7
73.3
DatasetDM (ours)
(R:9)
89.4
80.0
Table 5: Semantic segmentation on Cityscapes for two different split settings: and 19 categories.
'vehicle', 'car', 'bus', and "bicycle" are sampled classes for presentation.
dog
mloU
stop
car
dog
mloU
# train im.
syn.
joint
prompt num.)
dog
mloU
88.3
87.1
88.1
87.0
77.6
60
71.4
77.1
Human (100)
84.9 84.5
76.6
100
88.1
87.2
78.5
88.3
87.1
78.5
100
73.7
78.5
GPT4 (100)
85.2 86.1
77.1
200
88.0
86.6
78.3
200
87.7
87.0
78.0
200
74.4
79.4
GPT- (200)
88.0 86.2
77.3
500
87.2
84.9
76.8
87.2
86.3
77.5
400
76.4
80.4
GPT-4 (500)
88.1 87.1
78.5
800
86.3
83.4
76.1
87.1
1,000
78.4
81.1
GPT-4 (Ik)
88.3 87.1
78.5
(a) Visual Features 5. (b) Cross Attention A. (c) Size of Train Set.
(d) Prompt Candidates.
Table 6: DatasetDM Ablation on Pascal-VOC 2012 for semantic segmentation. Swin-B is used as the backbone. 100 real images are used for (a), (b), and (d). 'Syn.' and 'Joint" denote training with only synthetic data and joint training with real data, respectively.
performance escalates to an impressive 81%, demonstrating competitive prowess for the application. Notably, 1k training images representing roughly 10% of the original data, is still relatively diminutive.
Prompt from Language Model. Candidate Number: We also investigated the impact of the number of prompt candidates, as depicted in Table 6dd With the current configurations, an increase in the number of prompts can potentially lead a performance improvement of 2%. Word Number of Each Prompt. We simply study the effect of the length of prompt in Fig. 6 An increase in text length from 5 to 10 yields approximately 4% enhancement. When the text length surpasses 10, the performance appears to plateau. We argue that the upper limit is due to the current capacity of generative models.
5 Conclusion
In this study, we investigate using perception decoder parse the latent space of advanced diffusion model, extending the text-to-image task to a new paradigm: text-guided data generation. Training the decoder requires less than 1% of existing labeled images, enabling infinite annotated data generation. Experimental results show that the existing perception models trained on synthetic data generated by DatasetDM exhibit exceptional performance across six datasets and five distinct downstream tasks. Specifically, the synthetic data yields significant improvements of 13.3% mloU for semantic segmentation on VOC 2012 and 12.1% AP for instance segmentation on COCO 2017. Furthermore, text-guided data generation offers additional advantages, such as more robust solution for domain generalization and enhanced image editing capabilities. We hope that this research contributes new insights and fosters the development of synthetic perception data.
Acknowledgements
W. Wu, C. Shen's participation was supported by the National Key R&D Program of China (No. 2022ZD01 18700). W. Wu, H. Zhou's participation was supported by the National Key Research and Development Program of China (No. 2022YFC3602001), and the Key Research and Development Program of Zhejiang Province of China (No. 2021C02037). M. Shou's participation was supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF13-202) -0008. and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).
10

---

References
[1] J. Ahn and S. Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4981-4990, 2018.
[2] P. Akiva and K. Dana. Towards single stage weakly supervised semantic segmentation. arXiv preprint 10Xiv:2166. 10309, 2021.
[3] D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko. Label-efficient semantic segmenta-
tion with diffusion models. arXiv preprint arXiv:21
2021.
[4] S. Bubeck, Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, Lee. Y. Lee, Y. Li,
S. Lundbbrrg et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint VrXiv:2303.12711 2023.
[5] M. Bucher, T.-H. Vu, M. Cord, and Pérez. Zero-shot semantic segmentation. NeurIPS, 2019.
[6] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Itll, October 7-13, 2012, Proceedings, Part 12, pages 611 625. Springer, 2012.
[7] Y. Cahon, N. Murray, and M. Humenherger. Virtual kitti 2. arXiv preprint arXiv:2001. 10773, 2020.
[8] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1290-1299, 2022.
[9] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17664-17875 2021.
[10] Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[11] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban seene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-32233 2016.
[12] L. Dinh, Krueger, and Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.88 1410.85 16, 2014.
[13] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[14] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12888 2021.
[15] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-333 2010.
[16] L. Floridi and M. Chiriatti. Gpt-3: Its nature, scope. limits, and consequences. Minds and Machines, 30:681-694, 2020.
[17] A. Gaidon, Q. Wang. Y. Cabon, and E. Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 43440-3349, 2016.
[18] Y. Ge, 7hao, and I.. Itti. Pose augmentation: Class-agnostic object pose transformation for object recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 138-155. Springer. 2020.
[19] I. Goodfellow, J. Pouget-Abaal Generative adversarial networks. Communications of the ACM, 63(1 (3(11):139-14) 2020.
M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.
[20] Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, et al. Mi show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305. 18292, 2023.
[21] K. Han, Y. Xiong, C. You, P. Khosravi, S. Sun, X. Yan, J. Duncan, and X. Xic. Medgen3d: A deep generative framework for paired 3d image and mask generation. arXiv preprint arXiv:2304. 04 0006000446660000 2023.
[22] R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv: 200.0722 2022.
[23] A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, Y. J. Kim, M. Afify, and H. H. Awadalla. How good are gpt models at machine translation? comprehensive evaluation. arXiv preprint arXiv:2302.092 09210. 2023,
11

---

[24] A. Hertz, R. Mokady, J. Tenenbaum. K. Aberman, Y. Pritch, and D. Cohen-Or. editing with cross attention control. arXiv preprini arXiv:220..01622 2022.
Prompt-to-promp image
[25] Ho, A. Jain, and P. Abbeel. Denoising diffusion prohabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[26] J. Ho, C. Saharia, W. Chan, D. J. Fieet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23(47):1-33, 2022.
[27] Y. Jiang. S. Yang. H. Qiu, W. Wu, C. C. Loy, and Z. Liu. Tex2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022.
[28] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprini arXiv:1312.6114 2013.
[29] D. Li, H. Ling. S. W. Kim, K. Kreis, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21330-21340, 2022.
[30] Z. Li, Z. Chen, X. Liu, and J. Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation, arXiv preprint arXiv:2203.142) 2022.
[31] Z. Li, Q. Zhou, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301 05221, 2023.
[32] D. Lin, J. Dai, J. Jia, K. He, and Sun. Scribblesup: deribble-superviig convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3159-3167 2016.
[33] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zätnick. Microsoft
coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part 13, pages 740-755 Springer, 2014.
[34] Z. Lin, H. Liang, G. Fanti, and V. Sekar. Raregan: Generating samples for rare classes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 7506-7515, 2022.
[35] H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler. Editgan: editing. Advances in Neural Information Processing Systems, 34: 14331-1116 2021.
semantic image
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101. 2017.
[37] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourhh international conference on 3D vision (3DV), pages 565-571. IEEE, 2016.
[38] B. Pepik, M. Stark, P. Gehler, and B. Schiele. Teaching 3d geometry to deformable part models. In 2012 IEEE conference on computer vision and pultern recognition, pages 3362-3369. IEEE, 2012.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[40] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06 25, 2022.
[41] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent difftusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pateen Recognition, pages 10695, 2022.
[42] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Assisted Dnervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[43] C. Saharia, W. Chan, S. Saxena, L. Li. J. Whang. E. L. Denton, K. Ghasemipour, R. Gontijo Lopes. B. Karagol Ayan. Salimans, et al. Photorealistic text-to-image diffussio models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 334994,3444222 2022.
[44] S. Satkin, J. Lin, and M. Hebert. Data-driven scene understanding from 3d models. 2012.
[45] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta,
C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint 2:Xiv:2200.0840 2022.
[46] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rghd images. ECCV (5), 7576:746-7777 76074 2012.
[47] J. Sohl-Dickstein E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256 2265. PMLR, 2015.
12

---

[48] K. Sun. B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 5693-5703, 2019.
[49] X. Wangg L. Jing, Y. Lyu, M. Guo, J. Wang, H. Liu, J. Yu, and T. Zeng. Deep generatiee mixture model for robust imbalance classification. IEEE Transactions Pattern Analysis and Machine Intelligence. 2022.
[50] Y. Wang. Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:22 10560, 2022.
[51] G. Wilson and Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-6,, 2020.
[52] J. Wu. H. Fang. Y. Zhang. Y. Yang. and Y. Xu. Medsegdift: Medical image segmentation with diffusion prohahilistic model. arXiv preprint arXiv:22 1.006] 1,022.
2022.
[53] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixcl-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303. 1681, 2023.
[54] Z. Wu, L.. Wang, W. Wang, T. Shi, C. Chen, A. Hao, and Li. Synthetic data supervised salient object detection. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5557-5565, 2022.
[55] A. Xu, M. I. Vasileva, A. Dave, and A. Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. arXiv preprint arXiv:2212.12666 2022.
[56] J. Xu, S. Liu, Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303. 04803, 2023.
[57] M. Zhang. Y. Zhou, J. Zhao, Y. Man, B. Liu, and R. Yao. A survey of semi-and weakly supervised semantic segmentation of images. Artificial Intelligence Review, 53:4259-4425 4288, 2020.
[58] Y. Zhang, H. Ling, J. Gao, K. Yin. J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
[59] W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and Lu. Unleashing text-to text-to-image diffusion models for visual perception. arXiv preprint arXiv:23033.22/55: 2023.
[60] Y. Zhao, Q. Ye, W. Wu, C. Shen, and F. Wan. Generative prompt model for weakly supervised object localization. arXiv preprint wwXXvv2307.097555 2023.
[61] X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arxiv:2212. 2022.
13


Reviews:
[{"review_id": "1xBppqitC7", "reviewer": "Reviewer_21oc", "summary": "Large-scale datasets with quality annotations are often costly and time-consuming to collect and annotate. Since for most applications, large amounts of annotated data are required by deep neural networks, the authors propose a framework for text-guided synthetic data generation - called DatasetDM - which can produce an infinite number of images and their corresponding annotations, such as semantic segmentation masks or depth maps. Their method builds upon a pretrained diffusion model (specifically, Stable Diffusion) which is extended to include a simple decoder tasked with translating the latent space of the diffusion model’s time-conditional UNet to what the authors call “perception annotations”. The method is evaluated on several downstream tasks: semantic segmentation, instance segmentation, depth map prediction, human pose estimation, and zero-shot semantic segmentation. For each proposed downstream task, the authors use less than 1% of the training dataset to train the perception decoder on multi-scale text-to-image representations, and demonstrate competitive or improved performance over baselines.", "questions": "- Please see a list of suggested baseline comparisons in the previous section which I believe would better support the paper's claims.\n\n- How were the L(~100) prompts generated per dataset created? What was the process of quality control on the generated prompts to ensure the desired diversity in class vocabulary/labels in each respective dataset and task?\n\n- In Table 4, the difference in performance with an increased number of training images seems to demonstrate an increasing trend. When does performance peak?\n\n- Prompt length seems to be of crucial contribution to performance improvements - how feasible is it to provide high-quality, diverse prompts in practice, for any dataset and task? Will the prompt candidates need a human-in-the-loop quality control procedure?", "limitations": "Nothing of note with regard to potential negative societal impact of the proposed work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper attacks a problem relevant to the field: how to efficiently generate pairs of synthetic images and corresponding labels for a variety of core tasks within computer vision, with a framework that requires only minor modifications in architecture or representation depending on the task.\n\n- The framework seems adaptable to several tasks of interest in computer vision: depth map prediction, semantic segmentation, instance segmentation, open-vocabulary version of the latter, and pose estimation.\n\n- The paper is clearly written and easy to follow.\n\n- Results across several downstream tasks (specifically semantic segmentation and instance segmentation) show impressive improvement over baselines.\n\n- Interesting observation that training on synthetic and real data in the ablation studies section helps raise mIOU (Table 4) by a nontrivial amount.\n", "weaknesses": "The evaluation methodology and experimentation proposed in the paper is generally sound, but there are a few notes I'd like to make on the choice of baselines and dataset selection, which would strengthen the contribution claims of the paper.\n\nChoice of baselines / experiments \n\n- The authors cite the recently published work HandsOff [1], which proposes a very similar dataset generation paradigm – although utilizing the latent space of a pretrained StyleGAN model instead of Stable Diffusion, and using GAN inversion instead of diffusion inversion to create a hypercolumn representation per pixel (that is then fed through a simple label generator to produce similar “perception labels”: semantic segmentation masks, depth maps, human pose keypoint prediction, etc.). \n\n- HandsOff uses fewer images for all downstream tasks (16 or 50 depending on the dataset and task), and also uses the same datasets for evaluation on downstream task performance as other cited works like DatasetGAN [2], BigDatasetGAN [3], and EditGAN [4]. In addition, [1], [2] and [4]  also share the same class consolidation in each dataset, making comparison easier. A comparison on similar datasets and similar class structure to these papers (eg. CelebA-Mask-HQ [6] or DeepFashion-MM [9]) with a similar number of training images (instead of the variable “1% of the training dataset” the authors use, which they say averages to around 100 training images) could be added in the performance comparison to establish more firmly a SOTA claim across the proposed tasks.\n\n- I would like to see a comparison against DatasetDDPM [7] – a diffusion-model-based synthetic dataset generation method akin to the proposed approach and concurrent work – to give the reader a more comprehensive understanding of model performance. However, as per the supplementary material of [1], HandsOff actually establishes SOTA performance over all of [2], [4], and [7] with fewer labelled images, thus making a comparison against DatasetDDPM redundant. This could challenge the claim the authors make on L42-44: “Due to the limitations of the representation ability of previous GAN models, the quality of the synthesized data is often dissatisfactory, leading to an inferior performance on downstream tasks”. I see a comparison against HandsOff in the supplementary on Cityscapes semantic segmentation, where the proposed model does indeed outperform, but the same is missing for the pose estimation evaluation on DeepFashion-MM [9]. The best result presented is mIOU of 45.1 with 100 labelled images in 24 classes and with the Swin-B backbone, while HandsOff presents best mIOU of 68.4 with 50 labelled images in 8 classes and with the ResNet151 backone. \n\n- In summary, what would be beneficial to add to the baselines section is a fair comparison on pose estimation on DeepFashion-MM [9] against HandsOff, and a comparison on semantic segmentation on CelebA-Mask-HQ [6] (where HandsOff outperforms DatasetDDPM, another diffusion-based data generation framework).\n\nMissing references \n\n- EditGAN [4] and DatasetDDPM [7] should be added to the list of citations.\t\n\nMinor \n\n- The authors refer to results on the Cityscapes dataset typically used for semantic segmentation evaluation (L242), and outline an example of a generated prompt inputted to Stable Diffusion to generate images akin to the urban driving scenes (L226+). I did not see a reference to the supplementary for a table of the results; I also think mIOU improvement is significant and could be added to Table 3 in the main paper.\n\n- L60: possible typo: “... with minimal human-labeled…”\n\n- L241 states: “To comprehensively evaluate the generative image of DatasetDM, we conduct seven groups of experiments for the supported six downstream tasks”, but L261 says “Tab. 1 provides a basic comparison of the selected four downstream tasks.” I also found it confusing from the introduction and abstract to form an expectation as to which downstream tasks the authors evaluated on, and whether they achieve SOTA performance on all or some. It would be helpful to enumerate specific tasks with SOTA results in the abstract and in the claims section of the introduction.\n\nReferences\n\n[1] A. Xu, M. I. Vasileva, A. Dave, and A. Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.\n\n[2] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.\n\n[3] ] D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n[4] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. EditGAN: HighPrecision Semantic Image Editing. In Advances in Neural Information Processing Systems (NeurIPS), 2021.\n\n[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.\n\n[6] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.\n\n[7] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-Efficient Semantic Segmentation with Diffusion Models. In International Conference on Learning Representations (ICLR), 2022. \n\n[8] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n[9] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers for the valuable comments. \n\n**Weakness 1-1**: Comparison against Handsoff at the **method** levels.\n**R1**: We provide a comparison as follows:\n\n- **Task.** HandsOff designs several CNN layers for the decoder, which cannot handle tasks such as instance segmentation and open-vocabulary segmentation. In contrast, we design a transformer-based approach to accommodate these tasks, emphasizing that instance segmentation and open-vocabulary segmentation are more challenging compared to semantic segmentation.\n\n- **Architecture.** Our method and HandsOff are completely different for the architecture, even in the choice of feature extraction models (GAN and Diffusion). Moreover, the decoders have fundamental differences, with one using several CNN layers and the other one utilizing a transformer-based approach (see Fig.2 and Fig. 3). \n\n**Weakness 1-2**: Comparison against Handsoff at the **experiment** level.\n**R2**: In fact, our method outperforms HandsOff by a large margin as follows: \n\n- **Semantic Segmentation on DeepFashion-MM.** See Tab. 1 of the Rebuttal PDF, under the same setting, using fewer synthetic data (40k v.s 100k+), we can achieve an improvement of nearly `3%`.\n\n- **Semantic Segmentation on Cityscapes.** See Tab. 2 of the Rebuttal PDF (Information from Tab. 3 of Supplementary Material). With 9 real images, our method surpasses Handsoff using more data (50 images), achieving `17.1%` mIoU improvement for the 8-class split. HandsOff did not provide the corresponding performance for the 19-class split.\n\n**Weakness 1-3**: Handsoff challenges the claim the authors make on L42-44.\n**R3**: In fact, this is an open and unresolved issue (GAN or diffusion, who is better), and we would like to share my perspective:\n\n- **GAN v.s. Diffusion.** Currently, most of the SOTA models are diffusion-based, including well-known ones such as Stable Diffusion by Stability AI [3], Imagen by Google [4], and DALL·E 2 by OpenAI [5]. Some researchs[6] has attempted to compare Diffusion and GAN and determine which is better for generation, but there seems to be no definitive conclusion yet.\n\n- **Experiment.** The comparison between Handsoff and DatasetDDPM cannot conclusively determine whether GAN are superior to diffusion model. Firstly, they only compared one dataset, CelebAMask-HQ (Tab. 1 of Handsoff), which contains only human face data. Secondly, both StyleGAN used in Handsoff and DDPM[2] used in DatasetDDPM were proposed in 2020. Newer models like Stable Diffusion have achieved significantly better results.\n\n- **Correction.** Our claim may not be the most appropriate, and we will revise it to 'Due to the limitations of the representation ability of early (up to 2020) GAN models.'\n\n**Weakness 2**: Comparison against DatasetDDPM [1] \n**R4**: We provide the following comparisons between similarities and differences:\n\n- **Different Task**. DatasetDDPM focuses solely on semantic segmentation. In contrast, we propose a versatile dataset generation model that can simultaneously handle `six` tasks, including instance segmentation, and human pose estimation(see Fig. 1, Tab. 5).\n\n- **Different Architecture**. DatasetDDPM utilizes five features extracted from the UNet (DDPM[2]), while we use four features (L173) from the UNet of Stable Diffusion. Additionally, the perception decoder is entirely different. DatasetDDPM employs several CNN layers to perform a multi-class task for one mask, which cannot handle tasks such as instance segmentation, and open-vocabulary segmentation. In contrast, we use a unified transformer-based decoder (see Fig. 3 and Fig. 4) that can simultaneously handle six downstream tasks.\n\n- **Diffusion-based (similarity)**. Although the diffusion models we use are different (DDPM and Stable Diffusion), they both belong to the diffusion models.\n\n**Summary**: Our method and DatasetDDPM[1] have fundamental differences in tasks and core contributions (perception decoder architecture). \n\n**Weakness 3**: Experiment for Celebi-Mask-HQ.\n**R5**. We presented experiments in the rebuttal PDF (Tab. 5). We achieve an improvement of nearly 1.5%.\n\n**Weakness 4**: Missing references and Minor.\n**R6**: Thank you for the valuable suggestion. We will cite and discuss relevant works in the latest version and make minor corrections.\n\n**Weakness 5**: I did not see a table of the Cityscapes.\n**R7**: Supplementary Material Table 3 and Section 2.4.\n\n**Weakness 6**: Confusion.\n**R8**: A total of six downstream tasks, Tab. 1 presents the basic four of them, and more are available in Tab. 5 and the supplementary material.\n\n**Q1**:Baseline comparisons \n**R9**: See R1 - R5.\n\n**Q2**:How were the prompts generated? \n**R10**: In the supplementary materials, we provide detailed prompt examples for each dataset and task, as well as the generation process in Tab. 11 and Fig. 4. \n\n**Q3**:In Table 4, When does performance peak?\n**R11**: More experimental results can be found in Tab. 3 (increasing real data) and Tab. 4 (increasing synthetic data) of the rebuttal PDF. In brief, when we continuously increase the real data, the performance keeps improving (no peak). When we continue to increase the synthetic data, as the data exceeds 100k, further data increase has little impact on result improvement.\n\n**Q4**:Will the prompt candidates need a human-in-the-loop quality control procedure?\n**R12**: As shown in Fig. 4 of the supplementary materials, humans only need to provide a few `simple prompts` without any other manual intervention. These prompts can be completed in just `a few minutes`.\n\n[1]  Label-Efficient Semantic Segmentation with Diffusion Models. ICLR 2022\n\n[2] Denoising diffusion probabilistic models. NeurIPS 2020\n\n[3] High-resolution image synthesis with latent diffusion models.CVPR. 2022\n\n[4] Photorealistic text-to-image diffusion models with deep language understanding.NeurIPS 2022\n\n[5] Zero-shot text-to-image generation. ICML, 2021.\n\n[6] Diffusion models beat gans on image synthesis. NeurIPS 2021"}, {"review_id": "l31F5m07Nu", "reviewer": "Reviewer_rde2", "summary": "The paper presents a framework based on pre-trained Diffusion Models to synthesize images along with their perception annotation for various downstream perception tasks. With limited real labeled images as a baseline, the extra synthetic training pairs can boost the performance by a large margin.", "questions": "How about comparing the performance of the same amount of real and synthetic training pairs? For example, 1) what is the result under 100 real training images? 2) under 100 synthetic training images **only**, 3) how about 10,000 real or synthetic training pairs?", "limitations": "Yes, they have been discussed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation to synthesize perception datasets is promising.\n- The designed P-decoder is generic by taking advantage of the well-trained features from Diffusion Models.\n- Various downstream tasks are validated.", "weaknesses": "My major concern comes from the experimental settings, especially the number of used real labeled images. I understand that, with limited real labeled images (e.g., 50 or 100 images), the performance boosted by extra synthetic pairs can look more appealing, because the baseline achieved by 50 labeled images is extremely weak.\n\nHowever, considering a real-world scenario, we really hope to use the synthetic training pairs to boot **an already well-established baseline with all available real training pairs**, for example, using all the 10,000+ training images on Pascal VOC. Will the proposed DatasetDM further boost this strong and realistic baseline?\n\nAnother concern is that, what if the generated synthetic images do not belong to the same domain as original images? I am worrying the performance may be even degraded in some domains that are not so common as COCO and Pascal VOC when integrating the synthetic images, such as medical and remote sensing applications. Are there any mechanisms to avoid this?\n\n---\n\n**After discussion:**\n\nI agree with most of the authors' claims about the differences between real and synthetic images. However, as for the second difference \"low-quality synthetic images result in high-quality pseudo labels\", I politely disagree with it. Even though real unlabeled images are more challenging to assign pseudo labels, they are much more informative to learn than synthetic images (also supported by results in Response 3 above, 100 real images: 65.3 vs. 100 synthetic images: 40.1). For the hard-to-predict regions, we can easily skip them during learning by setting a pixel-wise threshold for them, which is a common practice in semi-supervised learning. I believe the reported COCO results above can be much higher when applying a stronger semi-supervised framework and set an ideal threshold.\n\nThe contribution of this work seems to only lie in replacing the real unlabeled images with synthetic unlabeled images, which can be deemed as a side work of semi-supervised learning paradigms. Moreover, there is no strong evidence to prove that synthetic images can play a superior role to real unlabeled images. There is also a severe domain gap issue when using these generative models for some rare domains, such as medical images and remote sensing images.\n\nBased on the above considerations, I would change my rating to 4: Borderline Reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers for the valuable comments. \n\n**Weakness 1**: Using the synthetic training pairs to boot an already well-established baseline with all available real training pairs, for example, using all the 10,000+ training images on Pascal VOC. \n\n**Response 1**: Thank you for the valuable suggestions. We provide the relevant experiments of well-established baselines for Pascal VOC as follows:\n\n| | 100 real image | 100 real image, 40k synthetic images | 1k real image | 1k real image, 40k synthetic images | 11.5k real image (full)| 11.5k real image, 40k synthetic images|\n|  ----  | ----  |----  |----  | ----  | ----  |----  |\n| mIoU | 65.2 | 78.5(`+13.3`)| 79.0 | 81.1(`+2.1`)| 84.3 | 85.4 (`+1.1`)\n\nAs real data increases, the gain from synthetic data gradually diminishes. However, for Pascal VOC, even with the full dataset (11.5k), our synthetic data still provides a 1.1% gain. Additionally, we would like to further emphasize some points for the real-world scenario:\n\n- **Annotation cost.** The annotation cost of 10k+ real masks is enormous, and reducing the annotation cost is the main motivation of this paper(L23-33). In real-world applications, the annotation cost is often the most significant factor. An example is SAM[1], which achieved promising performance using 1B masks and 11M images. According to the SAM paper (Sec. 5 Segment Anything Data Engine), the data annotation for the first stage requires 2091 working days (8 hours per day), meaning that `100 workers would need to annotate for at least 21 days each`. \n- **New insight.** I believe our work can provide valuable insights to the community. Our work is just an initial validation, as discussed in the Supplementary Materials, Section 5 \"Limitation & Future Work.\" We anticipate that more powerful generation models can lead to further performance improvements in the future. \n\n**Weakness 2**: I am worrying the performance may be even degraded in some domains that are not as common as COCO and Pascal VOC when integrating synthetic images, such as medical and remote sensing applications. Are there any mechanisms to avoid this?\n\n**Response 2**: This is a good question. The authors provided responses from three perspectives: domain gap, solutions, and whether a unified domain is necessary.\n\n- **Text-guided generation(Gap).** In fact, the text-guided generation capability of Stable Diffusion is already quite impressive. Please refer to Fig. 5 of paper, Fig. 1, and Fig. 2 of the supplementary material for some examples. Even in the presence of errors in the generated data, the domain gap remains very small. An example can be found in Supplementary Material, Fig. 5, where providing the prompt `a blue sedan waits at a stoplight` did not result in Stable Diffusion generating the `blue sedan`. This bias is weak and unlikely to lead to significant domain gaps, such as generating medical or remote sensing images.\n\n- **Stronger generation model(Solution).** To further align the domain and reduce the domain gap between real and synthetic data, one can utilize a more powerful generation model, such as IF[2]. The IF model exhibits superior semantic alignment, achieving higher clip score performance. (see Fig. 5 of the supplementary material).\n\n- **Special or General Data (Discussion).** A recent trend is the development of general foundational models, such as the GPT series[3][4] and SAM[5] (Some studies[5][6] have shown promising performance in medical and remote sensing tasks). In such cases, it is not necessary to completely unify the domain gap of data. Instead, a very large model can be designed and trained with a million-level dataset, allowing us to cover various scenarios, including COCO, Pascal VOC, medical, and remote sensing domains.\n\n**Question 1**: How about comparing the performance of the same amount of real and synthetic training pairs?  \n\n**Response 3**: We provide an ablation study for the same number of real and synthetic training pairs as follows:\n\n| | 100 real image | 100 synthetic images | 100 real image, 100 synthetic images| 1k real image| 1k synthetic images| 1k real image, 1k synthetic images|1k real image, 40k synthetic images| 11.5k real image|11.5k real image, 40k synthetic images| \n|  ----  | ----  |----  |----  | ----  | ----  |----  |----  |----  |----  |\n| mIoU | 65.3 | 40.1 | 67.9(`+2.6`)| 79.0 |68.8 | 80.1(`+1.1`)  | 81.1(`+2.1`) | 84.3 | 85.4 (`+1.1`)\n\nWe would like to further emphasize some points:\n\n- **Data cost.** In real-world applications, there is no reason to limit the quantity of synthetic data. When using 8 GPUs to generate 1,000 images, it takes approximately 3 minutes, and generating 40,000 data takes half a day. Moreover, regardless of the amount of synthetic data, their manual annotation cost remains the same, requiring only 100 real images and annotations.\n\n- **Same setup as prior[7][8][9].** Our work adopts a setting in line with prior research, employing a limited set of real data to synthesize a vast amount of data for augmentation. The comparative studies in these related works[7][8][9] also generate synthetic data volumes ranging from 5k to 200k.\n\n[1] Segment anything. arXiv preprint  (2023).\n\n[2] Deep-floyd IF\n\n[3] Training language models to follow instructions with human feedback. NeurIPS (2022).\n\n[4] Language models are few-shot learners. NeurIPS (2020).\n\n[5]Medical sam adapter: Adapting segment anything model for medical image segmentation.arXiv preprint (2023).\n\n[6] Scaling-up remote sensing segmentation dataset with segment anything model. arXiv preprint (2023).\n\n[7] Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint (2023).\n\n[8] HandsOff: Labeled dataset generation with no additional human annotations. CVPR. 2023.\n\n[9] Datasetgan: Efficient labeled data factory with minimal human effort. CVPR. 2021."}, {"review_id": "VO7DgsJwS3", "reviewer": "Reviewer_PHiv", "summary": "With the success of text-to-image generation models, the authors propose a novel approach to text-to-data that can produce synthetic data by adding a perception decoder (P-Decoder) on top of Stable Diffusion, which only requires a few labeled images for training. The authors show various experimental results to demonstrate the effectiveness of the proposed approach.", "questions": "Please check the comments in the Weaknesses section.", "limitations": "The authors haven't discussed the limitations of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n\n2. The idea of text-to-data is interesting, and the proposed approach is easy to train with 1% of labeled data.", "weaknesses": "1. In Section 4, the DatasetDM results are compared to the baselines trained with less than 1000 real images, which is unfair regarding the number of training samples (e.g. 400 vs. 80k). To claim the effectiveness of the proposed model, more realistic baselines with the same number of training samples must be considered.\n\n2. One of the contributions of the work is that the DatasetDM model can generate an unlimited amount of synthetic data. However, the authors do not conduct an analysis of how the number of generated samples affects the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers for the valuable comments. \n\n**Weaknesses 1**: In Section 4, the DatasetDM results are compared to the baselines trained with less than 1000 real images, which is unfair regarding the number of training samples (e.g. 400 vs. 80k). To claim the effectiveness of the proposed model, more realistic baselines with the same number of training samples must be considered.\n\n**Response 1**: Thank you for your constructive feedback. But, we believe that our experimental comparisons are reasonable, several crucial reasons are outlined below:\n\n- **Same Data cost.** We only use 400 real images to train and generate 80k synthetic images, meaning that the annotation cost for the 80k synthetic dataset is equivalent to that of the 400 real images. This ensures a fair comparison. The motivation of the paper is to reduce the annotation cost for perception tasks (see lines 22-31). Additionally, for datasets like VOC 2012, only 11.5k real images are available for training, not 80k.\n\n- **Same experimental setup as prior works[1][2][3].** Our setting is entirely in line with previous works[1][2][3], which all explore using few real data to perform few data tuning[4][5], and obtain a large amount of effective synthetic data at low cost. Among these works, HandsOff [3] (CVPR2023) is most similar to our setting, where their baseline also uses a small number of real images to control data cost. Under the same setting, for Semantic segmentation on Cityscapes, we outperform HandsOff by `17.1%` mIoU (see Supplementary Material, Table 3).\n\n- **The same number of training samples.** Here, we also provide the ablation study for the number of training images to further analyze the performance of our method. As shown in the table below (Semantic segmentation on VOC 2012), with the same amount of synthetic data(100 images), we can achieve a `2.7%` mIoU improvement, and using 400 synthetic data leads to a `10.9%` improvement. Even when using 11.5k real data (full training images, more realistic baselines), our synthetic data still achieve a `1.1%` increase in mIoU.\n\nNote: The annotation cost for the 400 synthetic data is the same as for 100 synthetic data, and the computational cost only requires a few more minutes of inference, which can be entirely negligible.\n\n| | 100 real image |  100 real image, 100 synthetic images|  100 real image, 400 synthetic images| 1k real image|  1k real image, 1k synthetic images|1k real image, 40k synthetic images| 11.5k real image| 11.5k real image, 40k synthetic images|\n|  ----  | ----  |----  |----   | ----  |----  |----  |----  |----  |\n| mIoU | 65.2  | 67.9(`+2.7`)| 76.1(`+10.9`)| 79.0 | 80.1(`+1.1`)  | 81.1(`+2.1`) | 84.3 | 85.4(`+1.1`)\n\n\n**Weaknesses 2**: The authors do not conduct an analysis of how the number of generated samples affects the performance.\n\n**Response 2**: Thank you for your constructive suggestion. We provide the ablation study of the number of generated samples for Semantic segmentation on VOC 2012 as follows:\n\n|Class| Baseline (100 real image )| only 40k synthetic images | 100 real image, 100 synthetic images| 100 real image, 400 synthetic images|100 real image, 40k synthetic images | 100 real image, 100k synthetic images| 100 real image, 200k synthetic images|\n|  ----  | ----  |----  |----  | ----  | ----  | ----  |----  |\n| mIoU | 65.2 | 73.7|  67.9(`+2.7`)| 76.1(`+10.9`) | 78.5(`+13.3`)| 78.7(`+13.5`)| 78.8(`+13.6`)\n\nIt can be observed that synthetic data can lead to a significant mIoU improvement of up to 13.6%. However, as the data volume exceeds 50k, the gain from synthetic data gradually becomes less pronounced.\n\n\n**Limitations  1**: The authors haven't discussed the limitations of the work.\n\n**Response 1**: We have already provided one section to discuss the limitations of this work in Supplementary Material Sec. 5.2 Limitation & Future Work (L207-221). We provide extensive experiments and analysis, including `12` experimental tables, `4` experimental data setting tables, and `11` figures. Due to space constraints, we have to include discussions for limitations in the supplementary material.\n\n\n[1] Zhang, Yuxuan, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. \"Datasetgan: Efficient labeled data factory with minimal human effort.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155. 2021.\n\n[2] Li, Daiqing, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. \"BigDatasetGAN: Synthesizing ImageNet with pixel-wise annotations.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21330-21340. 2022.\n\n[3] Xu, Austin, Mariya I. Vasileva, Achal Dave, and Arjun Seshadri. \"HandsOff: Labeled dataset generation with no additional human annotations.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7991-8000. 2023.\n\n[4] Jia, Menglin, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. \"Visual prompt tuning.\" In European Conference on Computer Vision, pp. 709-727. Cham: Springer Nature Switzerland, 2022.\n\n[5] Liu, Haotian, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n\n"}, {"review_id": "XWmxaA6NCa", "reviewer": "Reviewer_DG4j", "summary": "The paper introduces a novel dataset generation method producing diverse synthetic images along with perception annotations including semantic/instance segmentation, and depth/pose estimation. The technique leverages a pretrained text-to-image generation model, i.e., Stable Diffusion, to extract cross-attention maps and multi-scale feature maps as features input to multiple trainable decoders for different tasks. By training these add-on decoders on only a small number of labeled images (typically 100-800 images), the model can generate annotations along with the synthetic images. This method is evaluated on many tasks and achieves promising results. ", "questions": "+ After averaging the cross-attention maps within certain resolutions, do the authors fuse the averaged cross-attention maps with the corresponding feature maps in the same resolution or fuse two modalities in the same resolution first? \n+ Refer to Sec. 3.2 in the supplementary document, how can the authors generate the semantic/instance masks with the class prediction and mask predictions? How can the authors choose the threshold to obtain the final masks?\n+ Do the authors observe any failure cases when generating the annotations?\n+ Does the proposed method work for all classes in PASCAL-VOC 2012 or COCO2017 (the authors can show the class-specific results)?\n", "limitations": "No discussion on the method's limitations was provided. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "+ This paper addresses an interesting and impactful problem.\n+ The proposed method can generate pseudo-labels for several tasks segmentation, depth estimation, etc. with a small number of labeled images.\n", "weaknesses": "+ Limited novelty: the presented method is fairly similar to [a], published more than 2 months prior to this submission. The most significant difference is that in [a], the authors work directly on real images while this method synthesizes new images to train a corresponding network to produce prediction. No mentions or discussions are provided.\n+ The current setting is equivalent to the semi-supervised segmentation, the authors should compare with several semi-supervised methods in the corresponding tested perception tasks. \n+ The authors should include experiments to examine the contribution of cross–attention maps or multi-scale feature maps alone and in combination with ablation studies to analyze the contribution of each module to the overall performance. \n\nMinor: \n+ In L194, the authors use the notation Q for the learnable queues, however, in L207, the authors notated Q as query embedding. I would suggest using different Q_{x} for separate notations. \n+ The mask generation process should be (briefly) discussed in the main paper instead of the supplementary documents.\n\n[a] Zhao, Wenliang, et al. \"Unleashing text-to-image diffusion models for visual perception.\" arXiv preprint arXiv:2303.02153 (2023).\n\n---\nAfter reading the rebuttal, the highest score that I can give for the paper is 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers for the valuable comments. \n\n**Weaknesses 1**: Comparison with VPD (novelty). **R 1**: We want to emphasize that our work and VPD[1] (unpublished) are concurrent efforts. Besides, we believe that our work and VPD are entirely different for the following reasons:\n\n- **Different Task**. Our work is a synthetic data augmentation for perception tasks, while VPD is a diffusion-based perception algorithm.  In addition, we support more tasks than VPD, including Pose Estimation, Fashion Segmentation, and Open-Vocabulary Segmentation, which are not merely simple extensions but require corresponding improvements in the perception decoder structure.\n\n- **Different Architecture**.  The most crucial design component in VPD or our work is the perception decoder, and in this aspect, we are completely different. We design a transformer-based approach to model multiple tasks (Figure 3), while VPD employs existing decoder models (CNN-based), such as FPN[6] for semantic segmentation, using another CNN-based decoder for referring segmentation. The task-specific decoder of VPD utilizes multiple architectures but does not support instance segmentation.\n\n- **Training strategy**. We explore few-shot tuning by generating more synthetic data for auxiliary training. VPD, on the other hand, focuses on training with the entire dataset. Our emphasis is on performance improvement under limited data cost, while VPD serves as a performance method with abundant data.\n\n- **No mentions or discussions**. We have cited VPD and mentioned it in the context of the Text-Image Representation Fusion (L178-180), which is the closest connection we have with VPD. We did not claim that the cross attention is our contribution (L69-85), while its performance has already been validated by VPD[1], DiffuMask[2], and P2P[3]. \n\n**Summary**: Except for the feature extraction(common operations, similar to Prior[1][4][5]), and the usage of cross attention (not our contribution), all other parts (`task` and `decoder structure`) are completely different from VPD. Our `unique` feature (L69-85) is a versatile dataset generation model, where a transformer decoder can simultaneously generate various perception annotations.\n\n**Weaknesses 2**: Comparison with the semi-supervised segmentation. **R 2**: In some downstream tasks, we have provided some weakly supervised comparisons, including Zero-Shot Semantic Segmentation in Table 5 and Table 9 (supplementary material), where ZegFormer, and SIGN are weakly supervised methods. \n\nIn addition, we would like to emphasize that the current setting is `different` from weakly supervised segmentation. Existing weakly supervised methods, such as Zegformer and SIGN, utilize the entire training set with 10k+ images for PASCAL-VOC 2012 and do not use mask annotations, whereas we only use 100 images and utilize mask annotations. Therefore, a direct comparison between the two settings is not appropriate.\n\nIn comparison, some works for synthetic data augmentation (e.g., DiffuMask [2], Li et al. [5], and HandsOff [7]) are more reasonable for comparison. These methods all explore synthetic data augmentation for perception tasks and have similar settings. Especially HandsOff and our approach both explore training with limited data to generate more augmented data. Under the same setting, for Semantic segmentation on Cityscapes, we outperform HandsOff by **17.1%** mIoU (see Supplementary Material, Table 3).\n\n**Weaknesses 3**: Related experiments for cross–attention maps or multi-scale feature maps. **R 3**: We have provided the relevant experiments in Tab 4 (a) and (b), where the impact of using cross-attention maps or multi-scale feature maps with different steps on the results is shown. The results show that using the cross-attentions from appropriate step can lead to a gain of approximately 1%.\n\n**Question 1**: Details for fusing two modalities\n**R4**: We fuse features and cross-attention maps at different resolutions and then use them. Specifically, given four feature maps at different resolutions $\\mathcal{F}_1, \\mathcal{F}_2, \\mathcal{F}_3, \\mathcal{F}_4$, and four cross-attention maps at different resolutions $\\mathcal{\\hat{A}}_1, \\mathcal{\\hat{A}}_2, \\mathcal{\\hat{A}}_3, \\mathcal{\\hat{A}}_4$, we first fuse the features for each resolution. For the $8 \\times 8$ resolution, we obtain $\\mathcal{\\hat{F}}_1=\\text{Conv}\n([\\mathcal{F}_1,\\mathcal{\\hat{A}}_1])$. Then, we use the four final features ($\\mathcal{\\hat{F}}_1,\\mathcal{\\hat{F}}_2,\\mathcal{\\hat{F}}_3,\\mathcal{\\hat{F}}_4$) for the perception tasks.\n\n**Question 2**: How can the authors choose the threshold to obtain the final masks? **R5**: The process of mask generation is the same as in the baseline Mask2Former (L246), where we did not select specific mask thresholds. The thresholds for both instance/semantic masks and classification are set to 0.5. \n\n**Question 3**: Failure Cases. **R6**: We present some failure cases in the author rebuttal PDF (Fig. 1, Fig. 2).\n\n**Question 4**: Class-specific results for PASCAL-VOC 2012. **R7**: Please refer to the author rebuttal PDF for all 21 classes (Tab. 6).\n\n**Limitations 1**: No discussion of limitations. **R8**: We have already provided one section to discuss the limitations in Supplementary Material Sec. 5.2 (L207-221).\n\n\n[1] \"Unleashing text-to-image diffusion models for visual perception.\" arXiv preprint  (2023).\n\n[2] \"Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models.\" arXiv preprint  (2023).\n\n[3] \"Prompt-to-prompt image editing with cross attention control.\" arXiv preprint (2022).\n\n[4] \"Open-vocabulary panoptic segmentation with text-to-image diffusion models.\" CVPR. 2023.\n\n[5] \"Guiding text-to-image diffusion model towards grounded generation.\" arXiv preprint (2023).\n\n[6] \"Panoptic feature pyramid networks.\" CVPR. 2019.\n\n[7] \"HandsOff: Labeled dataset generation with no additional human annotations.\" CVPR. 2023.\n"}, {"review_id": "s23DD9Uy0P", "reviewer": "Reviewer_aPc6", "summary": "The paper proposes a new framework called DatasetDM that leverages the rich visual knowledge contained in diffusion models to generate synthetic data and corresponding annotations for various perception tasks, The core component of DatasetDM is the P-Decoder for generating perception annotations. The primary process is as follows: 1) During the training stage, a pre-defined text template is employed to extract feature maps and cross-attention maps from the pre-trained Stable Diffusion UNet, which are then concatenated to create a feature representation. The feature representation is then utilized to train P-Decoder, which employs a unified architecture that can adapt to various perception tasks; 2)During the generation stage, random sampling is conducted from a vast quantity of prompts provided by GPT-4 to enhance diversity. Subsequently, new images are generated by the VAE decoder of vanilla Stable Diffusion, while corresponding annotations are generated by the P-Decoder. The synthetic data and annotations will be utilized for various downstream tasks. ", "questions": "1) The paper requires further clarification on how the correspondence between generated images and annotations is ensured. Specifically, within the framework of DatasetDM, new images are generated by the VAE decoder of vanilla Stable Diffusion, while perception annotations are generated by the proposed P-Decoder， which utilizes a distinct architecture and does not share the same input with the former.\n\n2) Please further clarify why training the P-Decoder only requires less than 1% of manually labeled images. Visual align/instruct tuning is listed in the contribution, while not elaborated in the methodology.\n\n3) It is advised to utilize appropriate evaluation metrics or conduct further visualization experiments to provide an intuitive observation of the quality of synthetic data and the accuracy of the annotations.\n\n4) Sometimes Stable Diffusion may encounter difficulty in comprehending the given text prompt, resulting in unexpected image generation, I am curious whether such instances occurred during the experiments. Furthermore, is there a potential for repetition in the vast quantity of prompts generated by GPT-4? As In the ablation study, the performance improvement appears to plateau when the number of prompts is sufficiently large.\n\n5) There are spelling errors in Fig 2 a) and Fig 4, where \"DatasetDB\" should be corrected to \"DatasetDM\".\n", "limitations": "The limitations of the paper are not explicitly addressed, and it would be beneficial to add a section about limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a novel approach to leveraging the visual representations contained in the pre-trained Stable Diffusion for solving multiple perception tasks. The methodology is well-organized and clearly illustrated. Notably, there is limited research focusing on the utilization of the rich latent space in diffusion models for downstream vision tasks. Thus, the methodology presented in the paper is informative and enlightening.", "weaknesses": "1) The methodology lacks some novelty, as it bears a resemblance to the paper VPD[a]. Specifically, the method employed in DatasetDM for extracting visual representations from Stable Diffusion is nearly identical to that of VPD and only with minor differences in implementation details, including using a similar technique to extract multi-scale feature maps and cross-attention maps and employing the same manipulation of concatenation. Additionally, DatasetDM adopts a resembling design to VPD, as VPD also employs task-specific decoders for perception tasks and also uses a unified architecture with minor task-specific differences for the architecture design of the decoder.\n2) The experiments are insufficient. Firstly, to further illustrate the effectiveness of synthetic data, an ablation on the number of synthetic images can be conducted, especially investigating the performance of using a small number of synthetic images(e.g., 400 real images and 400 synthetic images versus 400 real images only). Secondly, more SOTA methods can be included as baseline to further evaluate the effectiveness of DatasetDM. Furthermore, evaluation on some less common datasets is necessary, which are more likely to face issues of data and annotation scarcity\n[a] \"Unleashing Text-to-Image Diffusion Models for Visual Perception\"(VPD)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers for the valuable comments. \n\n**Weakness 1**: Comparison with VPD.\n**R1**: We want to emphasize that our work and VPD (unpublished) are concurrent works. And we argue that our work is not a resemblance for the following reasons:\n\n- **Extracting features.**  Using the diffusion model to extract features is a common operation, and VPD is not the first work to do so. Previous works[1][2] have also employed this approach, similar to using a backbone to extract features. The contribution of this paper (L69-85) lies in the design of the versatile dataset generation model.\n\n- **Cross-attention.** We did not claim that the cross attention is our contribution (L69-85), while its performance has already been validated by VPD and DiffuMask[3]. We have cited VPD and mentioned it in the context of the cross-attention maps (L178-180). \n\n- **Perception decoder**. The two decoders are completely different. We design a transformer-based architecture (Fig. 3), while VPD uses existing decoder models (CNN-based), i.e., FPN. VPD does not support instance segmentation, as it predicts a single mask for multi-class classification. Meanwhile, we support a broader range of tasks, including pose estimation, zero-shot, and long-tail segmentation, which can not be supported by VPD. After carefully reviewing the VPD paper and official coding, we are confident in this observation.\n\n- **Training**. We explore few-shot tuning by generating more synthetic data for auxiliary training. VPD, on the other hand, focuses on training with the entire dataset.\n\n- **Different Task**. Our work focuses on synthetic data augmentation, while VPD is a perception model. \n\n**Summary**: Overall, we have `numerous differences` with VPD for the core `contribution` (L69-85 in our paper), with the most significant being the dissimilarity in decoder structure and the supported tasks. VPD can not support for instance segmentation, pose estimation, zero-shot segmentation, and long-tail segmentation.\n\n**Weakness 2-1**: Ablation on the number of synthetic images. **R2**: See Tab. 4 of  the rebuttal PDF for the ablation study on VOC 2012. Even using only 100 synthetic images (equivalent to the number of real images), we can still achieve a `2.6%` mIoU improvement. With 400 images, a significant improvement of `10.9%` can be achieved.\n\nAdditional points worth mentioning include: **Computing cost.** The only difference between 100, 400, and 40k synthetic images is the inference computational cost, which ranges from a few minutes to half a day. In practical applications, there is no reason to limit the use of only 100 images. **Compare with the prior[2][3][4].** Previous highly relevant works, such as DiffuMask[3], and HandsOff[4], all generate at least thousands of synthetic data. \n\n**Weakness 2-2**: More SOTA methods. **R3**: We have provided extensive experiments with **three** SOTA methods (L244-L259)  across **six** different tasks. Here we additionally provide a pose estimation method, SimCC[5], on the COCO val2017 dataset(see Tab. 7 of the rebuttal PDF)\n\n\n**Weakness 2-3**: Less common datasets. **R4**: We are conducting relevant experiments on LVIS[10] and will strive to provide before the rebuttal deadline .\n\n**Q 1**: Correspondence between images and annotations. **R5**:  The inputs of VAE and decoder belong to the same set of features. The decoder utilizes $F$ (L172), which are the Multi-Scale Feature Maps of the UNet (see Fig. 2). The feature $F$ with $64\\times64$ resolution at the 1000-th denoising step also is the input for the VAE decoder. \n\n**Q 2-1**: Only requires less than 1% of data. **R6**:  Because the used Stable Diffusion trained on 5 billion data (LAION5B, L48), it possesses strong semantic understanding capabilities, similar to the principles behind using language models(GPT-3[8], Self-instruct[9]) for aligning/instruct tuning, where large models exhibit robust generalization abilities.\n\n**Q 2-2**: Visual align/instruct tuning. **R7**: The tuning is more like a training strategy where pre-trained large models[8][9] on massive datasets exhibit strong generalization abilities, and for a specific task, only a small amount of task-specific data is needed to achieve good results. In our work (Fig. 2), we only need to train the decoder.\n\n**Q 3**:  Visualization. **R8**: Bad cases see Fig. 1 and Fig. 2 of rebuttal PDF, good cases see Fig. 4, Fig. 5 of paper, and Fig. 1, Fig. 2 of the supplementary material.\n\n**Q 4-1**:  Unexpected generation from prompt. **R9**: Good question. The phenomenon exists. We have discussed this in the Supplementary Material, Sec. 5.2, and Fig. 5, where providing a prompt `a blue sedan waits at a stoplight` resulted in Stable Diffusion not generating the blue sedan. One solution is to use more powerful generation models, such as IF[7].\n\n**Q 4-2**:  Repetitive prompt. **R10**: Almost no repeated prompts since we only generate 100 prompts for each category (L227). The performance plateau is determined by various factors, including the quality of images and the accuracy of masks. Therefore, increasing prompts may not necessarily improve the performance plateau.\n\n[1] Open-vocabulary panoptic segmentation with text-to-image diffusion models.CVPR. 2023.\n\n[2] Guiding text-to-image diffusion model towards grounded generation.\" arXiv preprint (2023).\n\n[3] Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint (2023).\n\n[4] HandsOff: Labeled dataset generation with no additional human annotations. CVPR. 2023.\n\n[5] Simcc: A simple coordinate classification perspective for human pose estimation.\" ECCV, 2022.\n\n[6] lvis: a dataset for large vocabulary instance segmentation.\n\n[7] deep-floyd/IF\n\n[8] Language models are few-shot learners.\" (2020): 1877-1901.\n\n[9] Self-instruct: Aligning language model with self generated instructions.\" arXiv preprint (2022).\n\n[10] lvis: a dataset for large vocabulary instance segmentation."}]

Author rebuttal:
Thanks to all reviewers and ACs for the valuable comments and suggestions. We have summarized and addressed the main concerns raised by each reviewer as follows:

**To Reviewer 21oc.** Our work differs from Handsoff and DatasetDDPM primarily in terms of uniqueness and contribution (lines 69 - 85). The key distinctions lie in the fact that we cater to `a wider range of tasks`, unlike Handsoff and DatasetDDPM, which cannot handle instance and open-vocabulary segmentation. Additionally, there are fundamental `differences in perception decoder`. While we utilize a unified transformer decoder, the other two approaches rely on CNN-based methods. Besides, our method `outperforms` HandsOff by a large margin (see Tab. 1, Tab. 2 and Tab. 5 of the Rebuttal PDF).

**To Reviewer rde2.** Our synthesized data can boost an already well-established baseline with all available real training data (`11.5k`) to achieve a `1.1%` increase in mIoU on VOC 2012. Furthermore, we would like to emphasize that the primary motivation of this paper (lines 23-33) is to decrease annotation costs and offer valuable insights to the community, highlighting the strong potential of synthesized data.

**To Reviewer PHiv.** Thank you for your constructive feedback for the number of training samples. However, in fact, training with 400 real images compared to 80k synthetic images is `fair`, as their annotation costs are consistent. The motivation of the paper is to decrease the cost of data (refer to L 22-31). And our approach `aligns` completely with prior studies [1][2][3], all of which investigate utilizing a small quantity of real data to acquire a substantial amount of effective synthetic data at a low cost. Furthermore, in Semantic Segmentation on VOC 2012, using the same quantity of synthetic data (100 images) as the real data, we attain a `2.7%` increase in mIoU, and with 400 synthetic data, there's a `10.9%` enhancement. Even with the use of 11.5k real data (full training images, more realistic baselines), our synthetic data still yield a `1.1%` mIoU improvement.

**To Reviewer DG4j and aPc6.** Firstly, we want to emphasize that our work and VPD (unpublished) are concurrent works. Secondly, our work and theirs are fundamentally different in terms of `tasks`, core `contributions`, and `architecture`. Here are some comparisons:

- **Different Task**. Our work is a synthetic data augmentation for perception tasks, while VPD is a diffusion-based perception algorithm. In addition, we support more tasks than VPD, including Pose Estimation, Fashion Segmentation, and Open-Vocabulary Segmentation, which are not merely simple extensions but require corresponding improvements in the perception decoder structure.

- **Different Architecture**. The most crucial design component in VPD or our work is the perception decoder, and in this aspect, we are completely different. We design a transformer-based approach to model multiple tasks (Fig. 3 of paper), while VPD employs existing decoder models (CNN-based), such as FPN for semantic segmentation. The task-specific decoder of VPD utilizes multiple architectures, and can not support instance segmentation and open-vocabulary segmentation.

- **Training strategy**. We explore few-shot tuning by generating more synthetic data for auxiliary training. VPD, on the other hand, focuses on training with the entire dataset.

- **Summary**. We have numerous differences with VPD for core contribution (L69-85 of our paper), with the most significant being the dissimilarity in decoder structure and the supported tasks. VPD can not support for instance segmentation, pose estimation, zero-shot segmentation, and long-tail segmentation.

**To all Reviewers.** We sincerely thank and appreciate the diligent efforts and valuable insights from all the reviewers for our paper. The authors look forward to further responses and discussions with you in the author-reviewer discussion period.


Best regards,

Authors of paper 5020.


[1] Datasetgan: Efficient labeled data factory with minimal human effort. CVPR. 2021.

[2] BigDatasetGAN: Synthesizing ImageNet with pixel-wise annotations. CVPR. 2022.

[3] HandsOff: Labeled dataset generation with no additional human annotations. CVPR. 2023.

Meta-review: This paper received three positive reviews and two negative reviews. Since this paper is borderline, the AC did a detailed review of the paper, the authors' feedback, and the reviewers' comments and discussions. One of the concerns of the reviewers who gave a negative score is the difference from the VPD; the VPD was indeed submitted to arXiv more than two months before NeurIPS, but given that the upload date is March 3, it can be considered almost concurrent work. The authors have also carefully explained the differences from the VPD in their feedback, and have performed many comparison experiments in the limited time available to demonstrate the differences. The proposed method loses when trained with only the same number of synthetic data as real data, but this is not a problem since the number of synthetic data can be easily increased.

In conclusion, the ACs consider this paper to be accepted, but strongly recommend that the reviewers' comments be addressed in the camera-ready paper, for example by describing the differences from VPD and including additional experiments.
Decision: Accept (poster)