Title: DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Model; Text-guided dataset generation
Abstract: Current deep networks are very data-hungry and benefit from training on large-scale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, we present DatasetDM, a generic dataset generation model that can produce diverse synthetic
images and the corresponding high-quality perception annotations (e.g., segmentation masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module. Training the decoder only needs less than 1% (around 100 images) of manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models on downstream tasks. To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic15
segmentation, instance segmentation, and depth estimation. Notably, it achieves 1) state-of-the-art results on semantic segmentation and instance segmentation; 2) significantly more efficient and robust in domain generalization than the real data; 3) state-of-the-art results in zero-shot segmentation setting; and 4) flexibility for efficient application and novel task composition (e.g., image editing)

DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models

Abstract

Current deep networks are very data-hungry and benefit from training on largescale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, we present DatasetDM, a generic dataset generation model that can produce diverse synthetic images and the corresponding high-quality perception annotations ( e.g. , segmentation masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module. Training the decoder only needs less than 1 % (around 100 images) manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models for downstream tasks.

To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation. Notably, it achieves (1) state-of-the-art results on semantic segmentation and instance segmentation; (2) significantly more robust on domain generalization than using the real data alone; and state-of-the-art results in zero-shot segmentation setting; and (3) flexibility for efficient application and novel task composition ( e.g. , image editing).

The project website is at: weijiawu.github.io/DatasetDM .

1 Introduction

Modern deep-learning models for perception tasks often require a large amount of labeled data to achieve good performance. Unfortunately, collecting large-scale data and labeling the corresponding pixel-level annotations is a time-consuming and expensive process. For example, collecting images of urban driving scenes requires physical car infrastructure, and labeling a segmentation annotation for a single urban image in Cityscapes [10] can take up to 60 minutes. Moreover, in certain specialized domains, such as medical or human facial data, collecting relevant information can be challenging or even impossible, owing to privacy concerns or other factors. The above challenges can be a barrier to advancing artificial intelligence in computer vision.

To reduce costs, many previous researchers have primarily focused on weakly supervised [57] and unsupervised solutions [51] to address the problem. For instance, certain segmentation priors [1, 2, 32] use weak or inexpensive labels to train robust segmentation models. With the advancement of

Open-Vocabulary

Segmentation:

Transformer

Image

Baseline/Prev. SOTA

DataDiffusion

VOC 2012 (mIoU

↑

)

100 Images

65.2

78.5

COCO 2017 (AP

↑

)

800 Images

14.4

26.5

Dataset

NYU Depth v2 (AbsRel

↓

)

50 Images

0.31

0.22

Perception Annotations

COCO 2017 Pose (AP

↑

)

800 Images

43.4

48.9

DeepFashion-MM (mIoU

↑

)

100 Images

VOC Zero Shot (mIoU

↑

)

450 Images

40.1

45.1

68.1

68.4

generative models, such as DALL-E [40], Stable Diffusion [41], some researchers have begun to explore the potential of synthetic data, attempting to use it to assist in training models, or even replace real data for perception task. Most works focus on classification, face recognition [34, 49, 22], salient object detection [54] and segmentation tasks [31, 53, 58], with only a minority trying to address problems such as human pose estimation [18] or medical image analysis [21]. In the era of GANs, DatasetGAN [58] and BigDatasetGAN [29] are recognized as pioneering works that utilize the feature space of pre-trained GANs and design a shallow decoder for generating pixel-level labels in the context of segmentation tasks. Following the two works, Hands-off [55] extends this approach to multitasking scenarios, such as depth estimation. However, these methods still suffer from three major drawbacks: 1) Due to the limitations of the representation ability of early (up to 2020) GAN models, the quality of the synthesized data is often dissatisfactory, leading to an inferior performance on downstream tasks. 2) These methods primarily focus on independent downstream tasks and no one tries to explore a unified data synthesis paradigm with a generalized decoding framework. 3) The training cost is still relatively high, while these methods do not make full use of the visual knowledge contained within the latent codes of powerful text-to-image models.

By leveraging large-scale datasets of image-text pairs ( e.g. , LAION5B [45]), recent text-to-image diffusion models ( e.g. , Stable Diffusion [41]) present phenomenal power in generating diverse and high-fidelity images with rich texture, diverse content, and reasonable structures. The phenomenon suggests that large text-to-image diffusion models can implicitly learn valuable, and rich high-level and low-level visual representations from massive image-text pairs. It is natural ask: Can we leverage the knowledge learned by these models to generate perception annotations and extend the paradigm of text-to-image generation to text-to-data generation?

In this paper, built upon the powerful text-to-image diffusion model, we present DatasetDM, a generalized dataset generation model that can produce an unlimited number of synthetic images and the corresponding perception annotations, as shown in Fig. 1. The key to our approach is a unified perception decoder, namely P-Decoder, that decodes the latent code of the diffusion model to perception annotations. Inspired by align/instruct tuning from LLM [50], a method for inducing output following capability with minimal human-labeled, we only use less than 1 % manually labeled images to train the decoder, enabling infinite annotated data generation. The generated data can subsequently be utilized to train any perception models for various downstream tasks, including but not limited to segmentation, depth estimation, and pose estimation. To maintain the robust image generation capabilities of the diffusion model, we freeze the model weights and use image inversion to extract the latent code, i.e. , multi-scale features, which are then fed into the unified P-Decoder. The P-Decoder accommodates various downstream tasks within a unified transformer framework, with only minor variations as depicted in Fig. 3.

To summarize, our contributions are four-fold:

Weintroduce DatasetDM: a versatile dataset generation model featuring a perception decoder capable of producing an unlimited quantity of high-fidelity synthetic images, along with various perception annotations, including depth, segmentation, and human pose estimation.

Visual align/instruct tuning, a method for inducing output following capability with minimal human-labeled data. With less than 1% of the original dataset, i.e. , around 100 images,

DatasetDM pushes the limits of text-to-image generation, pioneering a novel paradigm: text-to-data generation . This breakthrough is made possible by leveraging the rich latent code of the pre-trained diffusion model.

Experiments demonstrate that the existing perception models trained on synthetic data generated by DatasetDM exhibit outstanding performance on six datasets across five different downstream tasks. For instance, the synthetic data delivers remarkable gains of 13 . 3% mIoU and 12 . 1% AP for semantic segmentation on VOC 2012 and instance segmentation on COCO 2017, respectively.

Text-guided data generation allows for the generation of a diverse range of data, which has been shown to provide a more robust solution for domain generalization and long-tail data distribution. Moreover, DatasetDM offers a flexible approach for novel task composition, as exemplified by its ability to facilitate image editing (see Fig. 4).

2 Related Work

Text-to-Image Generation. Several mainstream methods exist for the task, including Generative Adversarial Networks (GANs)[19], Variational Autoencoders (VAEs)[28], flow-based models [12], and Diffusion Probabilistic Models (DPMs) [47, 41, 25, 20]. Recently, as a likelihood-based method, diffusion models have gained significant attention with promising generation abilities. They match the underlying data distribution by learning to reverse a noising process. Thanks to the high-quality image synthesis and the stability of training, diffusion models are quickly emerging as a new frontier [41, 40, 26, 43] in the field of image synthesis.

Text-guided diffusion models are used for text-conditional image generation, where a text prompt P guides the generation of content-related images I from a random Gaussian noise z . Visual and textual embeddings are typically fused using cross-attention. Recent large text-to-image diffusion models, such as Stable Diffusion [41] of Stability AI, DALL-E2 [40] of OpenAI, and Imagen [43] of Google, have shown powerful performance in generating diverse and high-fidelity images with rich textures and reasonable structures. Their impressive synthesis capabilities suggest that these models can implicitly learn valuable representations with the different semantic granularity from large-scale image-text pairs. In this paper, we leverage these learned representations (latent codes), to extend the paradigm of text-to-image generation to text-to-data generation.

Synthetic Datasets for Perception Task. Previous studies in dataset synthesis, such as Virtual KITTI [17] and Virtual KITTI 2 [7], primarily rely on 3D computer simulations to address standard 2D computer vision problems, such as object detection [38], scene understanding [44], and optical flow estimation [6]. However, these methods are limited by the domain of 3D models and cannot be generalized to arbitrary scenes and open-set categories. For instance, Virtual KITTI is exclusively focused on autonomous driving scenes and supports only 20 commonly occurring categories, which cannot be extended to open-scene domains like the COCO benchmark [33].

In contrast, synthetic data generated using generation models ( i.e. , GAN [19, 35, 55] and Diffusion Model [47]) are more flexible and can support a wider range of tasks and open-world scenes for various downstream tasks, such as classification task [22], face recognition [22], salient object detection [54], semantic segmentation [29, 58, 3, 60], and human pose [18]. Inspired by the success of large-scale generative models, such as Stable Diffusion [41], trained on massive datasets like LAION5B [45], recent studies have begun to explore the potential of powerful pre-trained diffusion generative models. Based on DDPM [25], DatasetDDPM [3] design several CNN layers as the annotation decoder to generate semantic segmentation data by performing a multi-class task. Li et al. [31] utilized Stable Diffusion and Mask R-CNN pre-trained on the COCO dataset [33] to design and train a grounding module for generating images and semantic segmentation masks. DiffuMask [53] produces synthetic image and semantic mask annotation by exploiting the potential of the cross-attention map between text and image from the text-supervised pre-trained Stable Diffusion model. The above methods focus on semantic segmentation, which cannot handle tasks such as instance segmentation. In this paper, we take a further step by utilizing a generalized perception decoder to parse the rich latent space of the pre-trained diffusion model, enabling the generation of perception for a variety of downstream tasks, including depth, segmentation, and human pose.

Diffusion Model for Perception Task. Some recent works [59, 56, 52] has also attempted to directly employ diffusion models for perceptual tasks. VPD [59] explores the direct application of pre-trained

Stable Diffusion to design perception models. ODISE [56] unify pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Different from these approaches, our focus lies in synthetic data augmentation for perception tasks, and design a unified transformer-based decoder to enhance more perception tasks from data aspect.

3 Methodology

3.1 Formulation

Given a language prompt S , text-guided diffusion models generate content-related images I ∈ R H × W × 3 from a random Gaussian noise z ∼ N ( 0 , I ) . The standard text-guided image denoising processing can be formulated as: I = Φ T2I ( z , S ) , where Φ T2I ( · ) refers to a pre-trained text-to-image diffusion model. In this paper, we adopt Stable Diffusion [41] as the base for the diffusion model, which consists of three components: a text encoder τ θ ( · ) for embedding prompt S ; a pre-trained variational autoencoder (VAE) [14] that encodes E ( · ) and decodes D ( · ) latent code of images; and a time-conditional UNet ( ϵ θ ( · ) ) [42] that gradually denoises the latent vectors. To fuse visual and textual embeddings, cross-attention layers are typically used in the UNet for each denoising step. The denoising process is modeled as a Markov chain: x t -1 = f ( x t , ϵ θ ) , where x t denote latent code at timestep t , and t ∈ [1 , T ] . The latent noise at the final timestep T , denoted as x T , is equivalent to the random Gaussian noise z . f ( · ) is the denoising function [25].

In this paper, we design a perception decoder that can effectively parse the latent space of the UNet ϵ θ ( x t , t, τ θ ( S )) . By doing so, we extend the text-to-image generation approach to a text-to-data paradigm:

where P 1: k denotes the corresponding perception annotations, and k is the number of the supported downstream tasks. In fact, the paradigm can support any image-level perception task, such as semantic segmentation, instance segmentation, pose estimation, and depth estimation.

3.2 Method Overview

This paper introduces a novel paradigm called text-to-data generation , which extends text-guided diffusion models trained on large-scale image-text pairs. Our key insight is that using a small amount of real data (using less than 1% existing labeled dataset) and a generic perception decoder to interpret the diffusion latent spaces, results in the generation of infinite and diverse annotated data. Then the synthetic data can be used to train any existing perception methods and apply them to real images.

The proposed DatasetDM framework, presented in Fig. 2, comprises two stages. 1) The first stageTraining -involves using diffusion inversion (§3.3) to obtain the latent code of the real image and extract the text-image representations (§3.3). These representations and their corresponding annotations are then used to train the perception decoder (§3.4). 2) The second stageInference (§3.5)-uses GPT-4 to enhance the diversity of data and generates abundant images, while the P-Decoder produces corresponding perception annotations such as masks and depth maps.

3.3 Hypercolumn Representation Extraction

The first step in the training stage of DatasetDM is to extract the hypercolumn representation of real images from the latent space of the diffusion model, as shown in Fig. 2(a). To achieve this, we employ the diffusion inversion technique, which involves adding a certain level of Gaussian noise to the real image and then extracting the features from the UNet during the denoising process.

Image Inversion for Diffusion Model. Give a real image X ∈ R H × W × 3 from the training set, the diffusion inversion (diffusion forward process) is a process that approximates the posterior distribution q ( x 1: T | x 0 ) , where x 0 = E ( X ) . This process is not trainable and is based on a fixed Markov chain that gradually adds Gaussian noise to the image, following a pre-defined noise schedule β 1 , . . . , β T [25]:

Denoising U-Net (

𝑡

!

)

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Prompt:    '

Photo of a person and a motorbike

'

Diffusion Process

ℰ

Encoder(VAE)

𝑥

!

𝑥

"

Add Noise

Multi-Scale Feature Maps

Cross Attention Maps

Text Encoder

ℒ

🔥

P - Decoder

(a) Training DatasetDM with few manually labeled images (less than 1% original data)

Denoising U-Net

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Decoder(VAE)

𝒟

𝓏

(

i.e.,

𝑥

#

)

Multi-Scale Feature Maps (

𝑡

!

Step)

Cross Attention Maps (

𝑡

!

Step)

ChatGPT/GPT4

1

. 'Two cars are parked on the urban street.'

2.

'A busy city street with multiple lanes of traffic'.

…

random sampling

Various Perception Annotations

(b) Generating Image  and corresponding Perception Annotation

Text Encoder

❄

❄

Freeze

❄

🔥

Trainable

🔥

P - Decoder

OpenAI

Perception Loss

Few Images

…

Infinite Synthetic Images

×𝑡

C

C

C

Concat

一

一

S

where t represents the t -th time step, and we set it to 1 for a single forward pass in our paper. A single forward pass for visual representation extraction usually provides two advantages, i.e. , faster convergence and better performance [56].

Text-Image Representation Fusion. With the latent code x t of the real image and the corresponding language prompt S , we extract the multi-scale feature maps and cross attention maps from the UNet ϵ θ as follows:

where S for training set is simply defined using a template ' a photo of a [CLS] 1 , [CLS] 2 , ... '. During the data generation phase, GPT-4 is used to provide diverse prompt languages. The variable F denotes the multi-scale feature maps from four layers of the U-Net, corresponding to four different resolutions, i.e. , 8 × 8 , 16 × 16 , 32 × 32 , and 64 × 64 , as illustrated in Fig. 2. Additionally, A represents the cross-attention maps of text-to-image from the 16 cross-attention layers in the U-Net, which implement the function A = softmax ( QK T √ d ) , where d is the latent projection dimension. We group the 16 cross-attention maps into 4 groups with the same resolutions, and compute their average within each group, which results in the average cross-attention maps ˆ A .

Prior works [53, 59, 24] have proved the effectiveness of class-discriminative and localization of cross-attention map between the visual embedding and the conditioning text features. Thus we concatenate the cross-attention maps ˆ A and the multi-scale feature maps F to obtain the final extracted hyper-column representation, and further use a 1 × 1 convolution to fuse them: ˆ F = Conv ([ F , ˆ A ]) .

3.4 Perception Decoder

The P-Decoder is utilized to translate the representation ˆ F into perception annotations, which are not limited to a specific type for each downstream task. To achieve this goal, inspired by previous works [8, 61], we devised a generalized architecture. This architecture is depicted in Fig. 3, with only minor variations ( i.e. , whether to startup some layers) for each downstream task. For example, the pixel decoder and transformer decoder are required for generic segmentation, while only the pixel decoder is necessary for depth and pose estimation.

Generic Image Segmentation. In Fig. 3-(a), we present the adaptation for semantic and instance segmentation tasks, which includes two components: the pixel decoder and the transformer decoder.

(a) Generic Segmentation

(c) Depth/Pose Estimation

Transformer

Decoder

Learnable Queues

Feature Maps

Attention Maps

Semantic/Instance Masks

C

×2

UpSample

⊗

Current

Feature

×3

Category

⊗

(b) Open-Vocabulary Segmentation

Transformer

Decoder

Attention Maps

Semantic/Instance Masks

C

×2

UpSample

⊗

Current

Feature

×3

⊗

Learnable Queues

Feature Maps

Category: dog, cat, car, …

Text Encoder

C

Feature Maps

Depth/Pose Prediction

C

×2

UpSample

⊗

Current

Feature

×3

Attention Maps

Pixel Decoder

Pixel Decoder

Pixel Decoder

Similar to Mask2former [8], the transformer decoder comprises a stack of transformer layers with cross-attention and self-attention. This component refines the queries and renders the outputs. The pixel decoder is made up of multiple upsampling CNN layers, and it is used to obtain per-pixel embeddings. Given the representation ˆ F and N learnable queues { Q 0 , Q 1 ...Q T } as input, the decoder outputs N binary masks O = o 1 , · · · , o N ∈ { 0 , 1 } N × H × W , along with the corresponding category. This is achieved through simple matrix multiplication between the outputs of the transformer decoder and the pixel decoder. Following Mask2former [8], one query is responsible for a class or instance for semantic and instance segmentation, respectively. To optimize the mask prediction, we use the binary cross-entropy loss [9] and dice loss [37]. On the other hand, the cross-entropy loss is used for classification prediction.

Open-Vocabulary Segmentation. Based on the architecture of generic segmentation, a semanticsrelated query concept is proposed to support the open-vocabulary segmentation task, as shown in Fig 3-(b). During the training stage, given the category name { C 0 , C 1 ...C K } ( e.g. , dog, cat ) on an image from the training set, we use the text encoder τ θ ( · ) , e.g. , CLIP [39] to encode them into the embedding space. Then concatenating them with the queries to equip the semantics of class vocabulary to the query as follows:

where Q i and C j is the i -th query embedding and j -th class. MLP refers to a learned MLP, used to fuse the class embedding and learnable query embedding. Thus DatasetDM can generate an open-vocabulary mask by incorporating a new class name, as illustrated in Fig. 5 (b).

Depth and Pose Estimation. For depth and pose estimation, the output format is predetermined, eliminating the need to differentiate between classes or instances. In this context, the pixel decoder is only required to predict a fixed number of maps O ∈ R M × H × W . The value of M is set to either 1 or 17 (corresponding to 17 human key points), depending on whether the task is depth or human pose estimation. For human pose estimation, we use the mean squared error as the loss function, and the ground truth heatmaps are generated by applying a 2D Gaussian with a standard deviation of 1 pixel centered on each key point. As for depth estimation, we update the loss function from the classic scale-invariant error [30, 13].

3.5 Prompting Text-Guided Data Generation

In Fig. 2 (b), we present the inference pipeline for text-guided data generation. There are two main differences compared to the training phase: firstly, the prompts come from a large language model instead of a fixed template, and secondly, the denoising process is extended to T steps to obtain the synthetic images. Large language model, i.e. , GPT-4, is adopted to enhance the diversity of generative data, while recent works [16, 23, 4] have proven their powerful understanding and adaptability for the real world. As shown in Fig. 4, instead of the template-based prompts from humans, we guide GPT-4 to produce diverse, and infinite prompts. For different downstream tasks and datasets, we give different guided prompts for GPT-4. For example, as for the urban scene of Cityscapes [10], the simple guided prompt is like ' Please provide 100 language descriptions of urban driving scenes for the Cityscapes benchmark, containing a minimum of 15 words each. These descriptions will serve as a guide for Stable Diffusion in generating images. ' In this approach, we collected L

Collaboration between the creative thinker, the painter, the annotator and editor

Human

: Please provide multiple descriptions for the appearance of one person with a focus on their clothing, pants, and shoes. These descriptions will serve as a guide

for Stable Diffusion in generating corresponding images.

creative thinker

)

:   Here are 1000 examples for you:

DatasetDM (

annotator

)

:

Stable Diffusion (

painter

)

:

1.

A man in a sharp grey suit, crisp white shirt, and black polished shoes

strides confidently down a city street

2.

A college student walks down the street wearing a casual outfit,

consisting of a cozy hoodie, slim-fit jeans, and a pair of trendy sneakers.

3.

A young man in a casual outfit strolls down the sidewalk, wearing a

denim jacket over a graphic tee, paired with black jeans.

…

ControlNet (

editor

)

:

Grey suit -> White suit

Human Pose                          Depth                     Semantic Mask              Instance Mask           DeepFashion Mask

city street -> forest

Zero-Shot Segmentation

Long-Tail Segmentation

Seen Class

Unseen Class

Head Class

Tail Class

aeroplane (0), bicycle (1), bird (2), boat (3),

bottle (4), bus (5), car (6), cat (7), chair (8),

cow (9) , diningtable (10), dog (11), horse

(12), motorbike (13), person (14)

potted

plant

(15),

sheep

(16), sofa (17), train (18),

tvmonitor (19)

aeroplane (0),

bicycle (1),

bird

(2), boat (3), bottle (4), bus (5),

car (6), cat (7), chair (8), cow (9)

diningtable (10), dog (11), horse (12), motorbike (13), person (14), potted plant (15), sheep (16), sofa (17), train (18), tvmonitor (19)

text prompts, which average around 100 prompts for each dataset. For each inference, a random prompt is sampled from this set.

4 Experiments

4.1 Implementation details

Architecture and Training Details. Stable diffusion V1 [41] model pre-trained on the LAION5B [45] dataset is used as our text-to-image diffusion model. The decoder architecture of Mask2Former [8] was selected as the base architecture for our P-Decoder. And we use 100 queries for the segmentation task. For all tasks, we train DatasetDM for around 50 k iterations with images of size 512 × 512 , which only need one Tesla V100 GPU, and lasted for approximately 20 hours. Optimizer [36] with a learning rate of 0 . 0001 is used.

Downstream Tasks Evaluation. To comprehensively evaluate the generative image of DatasetDM, we conduct seven groups of experiments for the supported six downstream tasks. Semantic Segmentation. Pascal-VOC 2012 [15] (20 classes) and Cityscapes [11] (19 classes), as two classical benchmark are used to evaluate. We synthesized 2 k images for each class in both datasets, resulting in a total of 40 k and 38 k synthetic images for Pascal-VOC 2012 and Cityscapes, respectively. The synthetic data is subsequently utilized to train Mask2former [8] and compared to its real data counterpart on a limited dataset setting (around 100 images). Instance Segmentation. For the COCO2017 [33] benchmark, we synthesized 1 k images for each class, resulting in a total of 80 k synthetic images. Similarly, Mask2former [8], as the baseline, is used to evaluate the synthetic data. We evaluate only the class-agnostic performance, where all the 80 classes are assigned the same class ID. Depth Estimation. We synthesized a total of 80 k synthetic images for NYU Depth V2 [46]. And using Depthformer [30] 2 to evaluate our synthetic data. Pose Estimation. We generated a set of 30 k synthetic images for COCO2017 Pose dataset [33] and employed HRNet [48] as the baseline model to assess the effectiveness of our approach. Zero-Shot Semantic Segmentation. Following Li et al. [31], Pascal-VOC 2012 [15] (20 classes) is used to evaluate. We train DatasetDM with only 15 seen categories, where each category including 30 real images, and synthesized a total of 40 k synthetic images for 20 categories. Long-tail Semantic Segmentation. The categories of VOC 2012 are divided into head (20 images each class) and tail classes (2 images each class). Then we train DatasetDM with these data and generate synthetic data. Human Semantic Segmentation. We synthesized a total

:

A white car and

a gray car drive along

a busy street.

seed

: 200098

:

A man with a

crew cut in a white polo

shirt, light wash jeans,

and white low-top

sneakers confidently

down a city street.

seed

: 107474

:

A nighttime cityscape

with a blue luxury car and a

yellow sports car driving on a

brightly lit street.

seed

:

619177

(a)

Urban City Domain

(b)

Zero Shot Segmentation:

Iron Man

(c)

Human-Centric Domain

of 20 k synthetic images for DeepFashion-MM [27] (24 classes). Mask2former [8] is used to evaluate the synthetic data. We split DeepFashion-MM into a set of 100 training images, and 12,500 testing images. Further details, such as prompts, can be found in the supplementary material.

Class Split for Zero-Shot and Long-Tail Segmentation. Table 1 provides a comprehensive overview of the class distribution for both zero-shot and long-tail scenarios. The division for zero-shot classes is consistent with previous studies [5, 53, 31]. The configuration for long-tail data distribution is firstly established in this paper.

4.2 Main Results

Table 2 provides a basic comparison of the selected four downstream tasks. More additional experiments can be found in Table 3 and Table 4, as well as in the Supplementary Material ( i.e. , Pose Estimation, Depth Estimation, Zero-Shot Segmentation, Fashion Segmentation, and others).

Semantic Segmentation. Table 4 displays the performance on VOC 2012. Using only 100 real images (5 images per class), training with purely synthetic data from DatasetDM achieves a 73 . 7% mIoU, an 8 . 5% improvement compared to using real data alone. Moreover, when jointly training with the 100 real images, further improvement is observed, resulting in a mIoU of 78 . 5% . Table 5 displays the performance on Cityscapes [11]. To compare with previous methods [55], we also conducted experiments with a 9-classes division. DatasetDM demonstrates consistent advancements over the baseline or prior SOTA, achieving up to a 10% improvement in mIoU under each experimental setup.

Instance Segmentation. Table 3 presents three distinct training settings, encompassing variations in the backbone and the number of training images. Regardless of the chosen setting, DatasetDM consistently achieves an improvement of approximately 10% . Employing 800 training images (10 images per class) and the Swin-B backbone, DatasetDM yields a 12 . 1% increase in Average Precision (AP), resulting in a final AP of 26 . 5% .

Depth Estimation . Table 2 presents a concise comparison between synthetic and real data on the NYU Depth V2 dataset [46]. Detailed information ( e.g. , backbone, other metrics) can be found in the supplementary material. When trained with 50 images, DatasetDM can achieve a 10% improvement compared to training solely with real images.

Human Pose Estimation. For the human pose estimation task on the COCO2017 dataset, DatasetDM demonstrates significant improvements compared to the baseline trained on 800 real images, achieving a 5 . 1% increase, as illustrated in Table 2. Similar to depth estimation, additional information can be found in the supplementary material.

Zero Shot and Long-tail Segmentation. Table 7 displays the results of experiments related to zeroshot and long-tail segmentation. Our model, DatasetDM, notably alleviates the effects of long-tail distribution by synthesizing a substantial amount of data for rare classes, leading to an improvement of up to 20% in mIoU. Details for both tasks can be found in the supplementary material.

4.3 Ablation studies

5

10

15

20

25

30

70

75

80

85

90

95

Word Number of Prompt

mIoU

Bird

Cat

Person

Diffusion Time Steps. Table 6a depicts the influence of visual features derived from various diffusion time steps, and the maximum sampling step is 1000. We observe that a large step results in adverse outcomes, whereas the performance with a smaller step tends to remain relatively stable.

Cross-Attention Fusion. Fig. 6b demonstrates that the cross attention maps, F in step 1 can yield a modest improvement, roughly 1% . Interestingly, it appears that as the step size increases, the benefit becomes less pronounced. Indeed, when the step size surpasses 500, it may even result in detrimental effects. From this perspective, the utility of the cross-attention map is limited.

Training Set Size. Additional training data for DatasetDM can further improve synthetic data, as shown in Table 6c. The increase of the training data from 60 to 400 images precipitates the most conspicuous improvement, subsequently reaching a saturation point. With 1k training images, the

performance escalates to an impressive 81% , demonstrating competitive prowess for the application. Notably, 1k training images representing roughly 10% of the original data, is still relatively diminutive.

Prompt from Language Model . Candidate Number. We also investigated the impact of the number of prompt candidates, as depicted in Table 6d. With the current configurations, an increase in the number of prompts can potentially lead to a performance improvement of 2% . Word Number of Each Prompt. We simply study the effect of the length of prompt in Fig. 6. An increase in text length from 5 to 10 yields approximately 4% enhancement. When the text length surpasses 10 , the performance appears to plateau. We argue that the upper limit is due to the current capacity of generative models.

5 Conclusion

In this study, we investigate using a perception decoder to parse the latent space of an advanced diffusion model, extending the text-to-image task to a new paradigm: text-guided data generation. Training the decoder requires less than 1% of existing labeled images, enabling infinite annotated data generation. Experimental results show that the existing perception models trained on synthetic data generated by DatasetDM exhibit exceptional performance across six datasets and five distinct downstream tasks. Specifically, the synthetic data yields significant improvements of 13 . 3% mIoU for semantic segmentation on VOC 2012 and 12 . 1% AP for instance segmentation on COCO 2017. Furthermore, text-guided data generation offers additional advantages, such as a more robust solution for domain generalization and enhanced image editing capabilities. We hope that this research contributes new insights and fosters the development of synthetic perception data.

References

J. Ahn and S. Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4981-4990, 2018.

P. Akiva and K. Dana. Towards single stage weakly supervised semantic segmentation. arXiv preprint arXiv:2106.10309 , 2021.

D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126 , 2021.

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

M. Bucher, T.-H. Vu, M. Cord, and P. Pérez. Zero-shot semantic segmentation. NeurIPS , 2019.

D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12 , pages 611-625. Springer, 2012.

Y. Cabon, N. Murray, and M. Humenberger. Virtual kitti 2. arXiv preprint arXiv:2001.10773 , 2020.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022.

B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:17864-17875, 2021.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems , 27, 2014.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010.

L. Floridi and M. Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines , 30:681-694, 2020.

A. Gaidon, Q. Wang, Y. Cabon, and E. Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4340-4349, 2016.

Y. Ge, J. Zhao, and L. Itti. Pose augmentation: Class-agnostic object pose transformation for object recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16 , pages 138-155. Springer, 2020.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305.18292 , 2023.

K. Han, Y. Xiong, C. You, P. Khosravi, S. Sun, X. Yan, J. Duncan, and X. Xie. Medgen3d: A deep generative framework for paired 3d image and mask generation. arXiv preprint arXiv:2304.04106 , 2023.

R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574 , 2022.

A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, Y. J. Kim, M. Afify, and H. H. Awadalla. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210 , 2023.

A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res. , 23(47):1-33, 2022.

Y. Jiang, S. Yang, H. Qiu, W. Wu, C. C. Loy, and Z. Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG) , 41(4):1-11, 2022.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21330-21340, 2022.

Z. Li, Z. Chen, X. Liu, and J. Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. arXiv preprint arXiv:2203.14211 , 2022.

Z. Li, Q. Zhou, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301.05221 , 2023.

D. Lin, J. Dai, J. Jia, K. He, and J. Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3159-3167, 2016.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Z. Lin, H. Liang, G. Fanti, and V. Sekar. Raregan: Generating samples for rare classes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7506-7515, 2022.

H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems , 34:16331-16345, 2021.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) , pages 565-571. IEEE, 2016.

B. Pepik, M. Stark, P. Gehler, and B. Schiele. Teaching 3d geometry to deformable part models. In 2012 IEEE conference on computer vision and pattern recognition , pages 3362-3369. IEEE, 2012.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

S. Satkin, J. Lin, and M. Hebert. Data-driven scene understanding from 3d models. 2012.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. ECCV (5) , 7576:746-760, 2012.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5693-5703, 2019.

X. Wang, L. Jing, Y. Lyu, M. Guo, J. Wang, H. Liu, J. Yu, and T. Zeng. Deep generative mixture model for robust imbalance classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

G. Wilson and D. J. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST) , 11(5):1-46, 2020.

J. Wu, H. Fang, Y. Zhang, Y. Yang, and Y. Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. arXiv preprint arXiv:2211.00611 , 2022.

W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303.11681 , 2023.

Z. Wu, L. Wang, W. Wang, T. Shi, C. Chen, A. Hao, and S. Li. Synthetic data supervised salient object detection. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5557-5565, 2022.

A. Xu, M. I. Vasileva, A. Dave, and A. Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. arXiv preprint arXiv:2212.12645 , 2022.

J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803 , 2023.

M. Zhang, Y. Zhou, J. Zhao, Y. Man, B. Liu, and R. Yao. A survey of semi-and weakly supervised semantic segmentation of images. Artificial Intelligence Review , 53:4259-4288, 2020.

Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10145-10155, 2021.

W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and J. Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153 , 2023.

Y. Zhao, Q. Ye, W. Wu, C. Shen, and F. Wan. Generative prompt model for weakly supervised object localization. arXiv preprint arXiv:2307.09756 , 2023.

X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework called DatasetDM that leverages the rich visual knowledge contained in diffusion models to generate synthetic data and corresponding annotations for various perception tasks, The core component of DatasetDM is the P-Decoder for generating perception annotations. The primary process is as follows: 1) During the training stage, a pre-defined text template is employed to extract feature maps and cross-attention maps from the pre-trained Stable Diffusion UNet, which are then concatenated to create a feature representation. The feature representation is then utilized to train P-Decoder, which employs a unified architecture that can adapt to various perception tasks; 2)During the generation stage, random sampling is conducted from a vast quantity of prompts provided by GPT-4 to enhance diversity. Subsequently, new images are generated by the VAE decoder of vanilla Stable Diffusion, while corresponding annotations are generated by the P-Decoder. The synthetic data and annotations will be utilized for various downstream tasks.\n\nSTRENGTHS:\nThe paper proposes a novel approach to leveraging the visual representations contained in the pre-trained Stable Diffusion for solving multiple perception tasks. The methodology is well-organized and clearly illustrated. Notably, there is limited research focusing on the utilization of the rich latent space in diffusion models for downstream vision tasks. Thus, the methodology presented in the paper is informative and enlightening.\n\nWEAKNESSES:\n1) The methodology lacks some novelty, as it bears a resemblance to the paper VPD[a]. Specifically, the method employed in DatasetDM for extracting visual representations from Stable Diffusion is nearly identical to that of VPD and only with minor differences in implementation details, including using a similar technique to extract multi-scale feature maps and cross-attention maps and employing the same manipulation of concatenation. Additionally, DatasetDM adopts a resembling design to VPD, as VPD also employs task-specific decoders for perception tasks and also uses a unified architecture with minor task-specific differences for the architecture design of the decoder.\n2) The experiments are insufficient. Firstly, to further illustrate the effectiveness of synthetic data, an ablation on the number of synthetic images can be conducted, especially investigating the performance of using a small number of synthetic images(e.g., 400 real images and 400 synthetic images versus 400 real images only). Secondly, more SOTA methods can be included as baseline to further evaluate the effectiveness of DatasetDM. Furthermore, evaluation on some less common datasets is necessary, which are more likely to face issues of data and annotation scarcity\n[a] \"Unleashing Text-to-Image Diffusion Models for Visual Perception\"(VPD)\n\nQUESTIONS:\n1) The paper requires further clarification on how the correspondence between generated images and annotations is ensured. Specifically, within the framework of DatasetDM, new images are generated by the VAE decoder of vanilla Stable Diffusion, while perception annotations are generated by the proposed P-Decoder， which utilizes a distinct architecture and does not share the same input with the former.\n\n2) Please further clarify why training the P-Decoder only requires less than 1% of manually labeled images. Visual align/instruct tuning is listed in the contribution, while not elaborated in the methodology.\n\n3) It is advised to utilize appropriate evaluation metrics or conduct further visualization experiments to provide an intuitive observation of the quality of synthetic data and the accuracy of the annotations.\n\n4) Sometimes Stable Diffusion may encounter difficulty in comprehending the given text prompt, resulting in unexpected image generation, I am curious whether such instances occurred during the experiments. Furthermore, is there a potential for repetition in the vast quantity of prompts generated by GPT-4? As In the ablation study, the performance improvement appears to plateau when the number of prompts is sufficiently large.\n\n5) There are spelling errors in Fig 2 a) and Fig 4, where \"DatasetDB\" should be corrected to \"DatasetDM\".\n\nLIMITATIONS:\nThe limitations of the paper are not explicitly addressed, and it would be beneficial to add a section about limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel dataset generation method producing diverse synthetic images along with perception annotations including semantic/instance segmentation, and depth/pose estimation. The technique leverages a pretrained text-to-image generation model, i.e., Stable Diffusion, to extract cross-attention maps and multi-scale feature maps as features input to multiple trainable decoders for different tasks. By training these add-on decoders on only a small number of labeled images (typically 100-800 images), the model can generate annotations along with the synthetic images. This method is evaluated on many tasks and achieves promising results.\n\nSTRENGTHS:\n+ This paper addresses an interesting and impactful problem.\n+ The proposed method can generate pseudo-labels for several tasks segmentation, depth estimation, etc. with a small number of labeled images.\n\nWEAKNESSES:\n+ Limited novelty: the presented method is fairly similar to [a], published more than 2 months prior to this submission. The most significant difference is that in [a], the authors work directly on real images while this method synthesizes new images to train a corresponding network to produce prediction. No mentions or discussions are provided.\n+ The current setting is equivalent to the semi-supervised segmentation, the authors should compare with several semi-supervised methods in the corresponding tested perception tasks. \n+ The authors should include experiments to examine the contribution of cross–attention maps or multi-scale feature maps alone and in combination with ablation studies to analyze the contribution of each module to the overall performance. \n\nMinor: \n+ In L194, the authors use the notation Q for the learnable queues, however, in L207, the authors notated Q as query embedding. I would suggest using different Q_{x} for separate notations. \n+ The mask generation process should be (briefly) discussed in the main paper instead of the supplementary documents.\n\n[a] Zhao, Wenliang, et al. \"Unleashing text-to-image diffusion models for visual perception.\" arXiv preprint arXiv:2303.02153 (2023).\n\n---\nAfter reading the rebuttal, the highest score that I can give for the paper is 4.\n\nQUESTIONS:\n+ After averaging the cross-attention maps within certain resolutions, do the authors fuse the averaged cross-attention maps with the corresponding feature maps in the same resolution or fuse two modalities in the same resolution first? \n+ Refer to Sec. 3.2 in the supplementary document, how can the authors generate the semantic/instance masks with the class prediction and mask predictions? How can the authors choose the threshold to obtain the final masks?\n+ Do the authors observe any failure cases when generating the annotations?\n+ Does the proposed method work for all classes in PASCAL-VOC 2012 or COCO2017 (the authors can show the class-specific results)?\n\nLIMITATIONS:\nNo discussion on the method's limitations was provided.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
With the success of text-to-image generation models, the authors propose a novel approach to text-to-data that can produce synthetic data by adding a perception decoder (P-Decoder) on top of Stable Diffusion, which only requires a few labeled images for training. The authors show various experimental results to demonstrate the effectiveness of the proposed approach.

STRENGTHS:
1. The paper is well-written and easy to follow. 

2. The idea of text-to-data is interesting, and the proposed approach is easy to train with 1% of labeled data.

WEAKNESSES:
1. In Section 4, the DatasetDM results are compared to the baselines trained with less than 1000 real images, which is unfair regarding the number of training samples (e.g. 400 vs. 80k). To claim the effectiveness of the proposed model, more realistic baselines with the same number of training samples must be considered.

2. One of the contributions of the work is that the DatasetDM model can generate an unlimited amount of synthetic data. However, the authors do not conduct an analysis of how the number of generated samples affects the performance.

QUESTIONS:
Please check the comments in the Weaknesses section.

LIMITATIONS:
The authors haven't discussed the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework based on pre-trained Diffusion Models to synthesize images along with their perception annotation for various downstream perception tasks. With limited real labeled images as a baseline, the extra synthetic training pairs can boost the performance by a large margin.\n\nSTRENGTHS:\n- The motivation to synthesize perception datasets is promising.\n- The designed P-decoder is generic by taking advantage of the well-trained features from Diffusion Models.\n- Various downstream tasks are validated.\n\nWEAKNESSES:\nMy major concern comes from the experimental settings, especially the number of used real labeled images. I understand that, with limited real labeled images (e.g., 50 or 100 images), the performance boosted by extra synthetic pairs can look more appealing, because the baseline achieved by 50 labeled images is extremely weak.\n\nHowever, considering a real-world scenario, we really hope to use the synthetic training pairs to boot **an already well-established baseline with all available real training pairs**, for example, using all the 10,000+ training images on Pascal VOC. Will the proposed DatasetDM further boost this strong and realistic baseline?\n\nAnother concern is that, what if the generated synthetic images do not belong to the same domain as original images? I am worrying the performance may be even degraded in some domains that are not so common as COCO and Pascal VOC when integrating the synthetic images, such as medical and remote sensing applications. Are there any mechanisms to avoid this?\n\n---\n\n**After discussion:**\n\nI agree with most of the authors' claims about the differences between real and synthetic images. However, as for the second difference \"low-quality synthetic images result in high-quality pseudo labels\", I politely disagree with it. Even though real unlabeled images are more challenging to assign pseudo labels, they are much more informative to learn than synthetic images (also supported by results in Response 3 above, 100 real images: 65.3 vs. 100 synthetic images: 40.1). For the hard-to-predict regions, we can easily skip them during learning by setting a pixel-wise threshold for them, which is a common practice in semi-supervised learning. I believe the reported COCO results above can be much higher when applying a stronger semi-supervised framework and set an ideal threshold.\n\nThe contribution of this work seems to only lie in replacing the real unlabeled images with synthetic unlabeled images, which can be deemed as a side work of semi-supervised learning paradigms. Moreover, there is no strong evidence to prove that synthetic images can play a superior role to real unlabeled images. There is also a severe domain gap issue when using these generative models for some rare domains, such as medical images and remote sensing images.\n\nBased on the above considerations, I would change my rating to 4: Borderline Reject.\n\nQUESTIONS:\nHow about comparing the performance of the same amount of real and synthetic training pairs? For example, 1) what is the result under 100 real training images? 2) under 100 synthetic training images **only**, 3) how about 10,000 real or synthetic training pairs?\n\nLIMITATIONS:\nYes, they have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLarge-scale datasets with quality annotations are often costly and time-consuming to collect and annotate. Since for most applications, large amounts of annotated data are required by deep neural networks, the authors propose a framework for text-guided synthetic data generation - called DatasetDM - which can produce an infinite number of images and their corresponding annotations, such as semantic segmentation masks or depth maps. Their method builds upon a pretrained diffusion model (specifically, Stable Diffusion) which is extended to include a simple decoder tasked with translating the latent space of the diffusion model’s time-conditional UNet to what the authors call “perception annotations”. The method is evaluated on several downstream tasks: semantic segmentation, instance segmentation, depth map prediction, human pose estimation, and zero-shot semantic segmentation. For each proposed downstream task, the authors use less than 1% of the training dataset to train the perception decoder on multi-scale text-to-image representations, and demonstrate competitive or improved performance over baselines.\n\nSTRENGTHS:\n- The paper attacks a problem relevant to the field: how to efficiently generate pairs of synthetic images and corresponding labels for a variety of core tasks within computer vision, with a framework that requires only minor modifications in architecture or representation depending on the task.\n\n- The framework seems adaptable to several tasks of interest in computer vision: depth map prediction, semantic segmentation, instance segmentation, open-vocabulary version of the latter, and pose estimation.\n\n- The paper is clearly written and easy to follow.\n\n- Results across several downstream tasks (specifically semantic segmentation and instance segmentation) show impressive improvement over baselines.\n\n- Interesting observation that training on synthetic and real data in the ablation studies section helps raise mIOU (Table 4) by a nontrivial amount.\n\nWEAKNESSES:\nThe evaluation methodology and experimentation proposed in the paper is generally sound, but there are a few notes I'd like to make on the choice of baselines and dataset selection, which would strengthen the contribution claims of the paper.\n\nChoice of baselines / experiments \n\n- The authors cite the recently published work HandsOff [1], which proposes a very similar dataset generation paradigm – although utilizing the latent space of a pretrained StyleGAN model instead of Stable Diffusion, and using GAN inversion instead of diffusion inversion to create a hypercolumn representation per pixel (that is then fed through a simple label generator to produce similar “perception labels”: semantic segmentation masks, depth maps, human pose keypoint prediction, etc.). \n\n- HandsOff uses fewer images for all downstream tasks (16 or 50 depending on the dataset and task), and also uses the same datasets for evaluation on downstream task performance as other cited works like DatasetGAN [2], BigDatasetGAN [3], and EditGAN [4]. In addition, [1], [2] and [4]  also share the same class consolidation in each dataset, making comparison easier. A comparison on similar datasets and similar class structure to these papers (eg. CelebA-Mask-HQ [6] or DeepFashion-MM [9]) with a similar number of training images (instead of the variable “1% of the training dataset” the authors use, which they say averages to around 100 training images) could be added in the performance comparison to establish more firmly a SOTA claim across the proposed tasks.\n\n- I would like to see a comparison against DatasetDDPM [7] – a diffusion-model-based synthetic dataset generation method akin to the proposed approach and concurrent work – to give the reader a more comprehensive understanding of model performance. However, as per the supplementary material of [1], HandsOff actually establishes SOTA performance over all of [2], [4], and [7] with fewer labelled images, thus making a comparison against DatasetDDPM redundant. This could challenge the claim the authors make on L42-44: “Due to the limitations of the representation ability of previous GAN models, the quality of the synthesized data is often dissatisfactory, leading to an inferior performance on downstream tasks”. I see a comparison against HandsOff in the supplementary on Cityscapes semantic segmentation, where the proposed model does indeed outperform, but the same is missing for the pose estimation evaluation on DeepFashion-MM [9]. The best result presented is mIOU of 45.1 with 100 labelled images in 24 classes and with the Swin-B backbone, while HandsOff presents best mIOU of 68.4 with 50 labelled images in 8 classes and with the ResNet151 backone. \n\n- In summary, what would be beneficial to add to the baselines section is a fair comparison on pose estimation on DeepFashion-MM [9] against HandsOff, and a comparison on semantic segmentation on CelebA-Mask-HQ [6] (where HandsOff outperforms DatasetDDPM, another diffusion-based data generation framework).\n\nMissing references \n\n- EditGAN [4] and DatasetDDPM [7] should be added to the list of citations.\t\n\nMinor \n\n- The authors refer to results on the Cityscapes dataset typically used for semantic segmentation evaluation (L242), and outline an example of a generated prompt inputted to Stable Diffusion to generate images akin to the urban driving scenes (L226+). I did not see a reference to the supplementary for a table of the results; I also think mIOU improvement is significant and could be added to Table 3 in the main paper.\n\n- L60: possible typo: “... with minimal human-labeled…”\n\n- L241 states: “To comprehensively evaluate the generative image of DatasetDM, we conduct seven groups of experiments for the supported six downstream tasks”, but L261 says “Tab. 1 provides a basic comparison of the selected four downstream tasks.” I also found it confusing from the introduction and abstract to form an expectation as to which downstream tasks the authors evaluated on, and whether they achieve SOTA performance on all or some. It would be helpful to enumerate specific tasks with SOTA results in the abstract and in the claims section of the introduction.\n\nReferences\n\n[1] A. Xu, M. I. Vasileva, A. Dave, and A. Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.\n\n[2] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.\n\n[3] ] D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n[4] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. EditGAN: HighPrecision Semantic Image Editing. In Advances in Neural Information Processing Systems (NeurIPS), 2021.\n\n[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.\n\n[6] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.\n\n[7] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-Efficient Semantic Segmentation with Diffusion Models. In International Conference on Learning Representations (ICLR), 2022. \n\n[8] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n[9] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.\n\nQUESTIONS:\n- Please see a list of suggested baseline comparisons in the previous section which I believe would better support the paper's claims.\n\n- How were the L(~100) prompts generated per dataset created? What was the process of quality control on the generated prompts to ensure the desired diversity in class vocabulary/labels in each respective dataset and task?\n\n- In Table 4, the difference in performance with an increased number of training images seems to demonstrate an increasing trend. When does performance peak?\n\n- Prompt length seems to be of crucial contribution to performance improvements - how feasible is it to provide high-quality, diverse prompts in practice, for any dataset and task? Will the prompt candidates need a human-in-the-loop quality control procedure?\n\nLIMITATIONS:\nNothing of note with regard to potential negative societal impact of the proposed work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}