Title: ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Object Detection Dataset Generation

Complex Scene Synthesis

Domain-Specific

Diffusion Models
Abstract: Modern diffusion-based image generative models have made significant progress and become promising to enrich training data for the object detection task. However, the generation quality and the controllability for complex scenes containing multi-class objects and dense objects with occlusions remain limited. This paper presents ODGEN, a novel method to generate high-quality images conditioned on bounding boxes, thereby facilitating data synthesis for object detection. Given a domain-specific object detection dataset, we first fine-tune a pre-trained diffusion model on both cropped foreground objects and entire images to fit target distributions. Then we propose to control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. ODGEN exhibits robustness in handling complex scenes and specific domains. Further, we design a dataset synthesis pipeline to evaluate ODGEN on 7 domain-specific benchmarks to demonstrate its effectiveness. Adding training data generated by ODGEN improves up to 25.3% mAP@.50:.95 with object detectors like YOLOv5 and YOLOv7, outperforming prior controllable generative methods. In addition, we design an evaluation protocol based on COCO-2014 to validate ODGEN in general domains and observe an advantage up to 5.6% in mAP@.50:.95 against existing methods.

avatar

Abstract

Modern diffusion-based image generative models have made significant progress and become promising to enrich training data for the object detection task. However, the generation quality and the controllability for complex scenes containing multiclass objects and dense objects with occlusions remain limited. This paper presents ODGEN, a novel method to generate high-quality images conditioned on bounding boxes, thereby facilitating data synthesis for object detection. Given a domainspecific object detection dataset, we first fine-tune a pre-trained diffusion model on both cropped foreground objects and entire images to fit target distributions. Then we propose to control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. ODGEN exhibits robustness in handling complex scenes and specific domains. Further, we design a dataset synthesis pipeline to evaluate ODGEN on 7 domain-specific benchmarks to demonstrate its effectiveness. Adding training data generated by ODGEN improves up to 25.3% mAP@.50:.95 with object detectors like YOLOv5 and YOLOv7, outperforming prior controllable generative methods. In addition, we design an evaluation protocol based on COCO-2014 to validate ODGEN in general domains and observe an advantage up to 5.6% in mAP@.50:.95 against existing methods.

1 Introduction

Object detection is one of the most widely used computer vision tasks in real-world applications. However, data scarcity poses a significant challenge to the performance of state-of-the-art models

like YOLOv5 [25] and YOLOv7 [54]. With the progress of generative diffusion models, such as DALL-E [13], and Stable Diffision [44], which can generate high-quality images from text prompts, recent works have started to explore the potential of synthesizing images for perceptual model training. Furthermore, methods like GLIGEN [33], ReCo [61], and ControlNet [62] provide various ways to control the contents of synthetic images. Concretely, extra visual or textual conditions are introduced as guidance for diffusion models to generate objects under certain spatial constraints. Therefore, it becomes feasible to generate images together with instance-level or pixel-level annotations.

Nevertheless, it remains challenging to generate an effective supplementary training set for real-world object detection applications. Firstly, large-scale pre-trained diffusion models are usually trained on web crawl datasets such as LAION [48], whose distributions may be quite distinct from specialist domains. The domain gap could undermine the fidelity of generated images, especially the detailed textures and styles of the objects, e.g., the tumor in medical images or the avatar in a video game. Secondly, the text prompts for object detection scenes may contain multiple categories of subjects. It could cause the "concept bleeding" [12, 40, 68] problem for modern diffusion models, which is defined as the unintended merging or overlapping of distinct visual elements in an image. As a result, the synthetic images can be inconsistent with pseudo labels used as generative conditions. Thirdly, overlapping objects, which are common in object detection datasets, are likely to be merged as one single object by existing controllable generative methods. Lastly, some objects may be neglected by the diffusion model so that no foreground is generated at the conditioned location. All of the limitations can hinder the downstream object detection performance.

To address these challenges, we propose ODGEN, a novel method to generate synthetic images with a fine-tuned diffusion model and object-wise conditioning modules that control the categories and locations of generated objects. We first fine-tune the diffusion model on a given domain-specific dataset, with not only entire images but also cropped foreground patches, to improve the synthesis quality of both background scene and foreground objects. Secondly, we design a new text prompt embedding process. We propose to encode the class name of each object separately by the frozen CLIP [41] text encoder, to avoid mutual interference between different concepts. Then we stack the embeddings and encode them with a newly introduced trainable text embedding encoder. Thirdly, we propose to use synthetic patches of foreground objects as the visual condition. Each patch is resized and pasted on an empty canvas according to bounding box annotations, which deliver both conceptual and spatial information without interference from overlap between objects. In addition, we train foreground/background discriminators to check whether the region of each pseudo label contains a synthetic object. If no object can be found, the pseudo label will be filtered.

We evaluate the effectiveness of our ODGEN in specific domains on 7 subsets of the Roboflow-100 benchmark [6]. Extensive experimental results show that adding our synthetic data improves up to 25.3% mAP@.50:.95 on YOLO detectors, outperforming prior controllable generative methods. Furthermore, we validate ODGEN in general domains with an evaluation protocol designed based on COCO-2014 [37] and gain an advantage up to 5.6% in mAP@.50:.95 than prior methods.

The main contributions of this work can be summarized as follows:

We propose to fine-tune pre-trained diffusion models with both cropped foreground patches and entire images to generate high-quality domain-specific target objects and background scenes.

We design a novel strategy to control diffusion models with object-wise text prompts and synthetic visual conditions, improving their capability of generating and controlling complex scenes.

We conduct extensive experiments to demonstrate that our synthetic data effectively improves the performance of object detectors. Our method outperforms prior works on fidelity and trainability in both specific and general domains.

2 Related Works

Diffusion models [7, 20, 29, 38, 49, 50] have made great progress in recent years and become mainstream for image generation, showing higher fidelity and training stability than prior generative models like GANs [4, 16, 26, 27, 28] and VAEs [30, 43, 52]. Latent diffusion models (LDMs) [44] perform the denoising process in the compressed latent space and achieve flexible generators conditioned on inputs like text and semantic maps. LDMs significantly improve computational efficiency and enable training on internet-scale datasets [48]. Modern large-scale text-to-image diffusion models

including eDiff-I [2], DALL·E [42], Imagen [46], and Stable Diffusion [40, 44] have demonstrated unparalleled capabilities to produce diverse samples with high fidelity given unfettered text prompts.

Layout-to-image generation synthesizes images conditioned on graphical inputs of layouts. Pioneering works [22, 35] based on GANs [16] and transformers [53] successfully generate images consistent with given layouts. LayoutDiffusion [66] fuses layout and category embeddings and builds object-aware cross-attention for local conditioning with traditional diffusion models. LayoutDiffuse [24] employs layout attention modules for bounding boxes based on LDMs. MultiDiffusion [3] and BoxDiff [58] develop training-free frameworks to produce samples with spatial constraints. GLIGEN [33] inserts trainable gated self-attention layers to pre-trained LDMs to fit specific tasks and is hard to generalize to scenes uncommon for pre-trained models. ReCo [61] and GeoDiffusion [5] extend LDMs with position tokens added to text prompts and fine-tunes both text encoders and diffusion models to realize region-aware controls. They need abundant data to build the capability of encoding the layout information in text prompts. ControlNet [62] reuses the encoders of LDMs as backbones to learn diverse controls. However, it still struggles to deal with some complex cases. MIGC [67] decomposes multi-instance synthesis to single-instance subtasks utilizing additional attention modules. InstanceDiffusion [55] adds precise instance-level controls, including boxes, masks, and scribbles for text-to-image generation. Given annotations of bounding boxes, this paper introduces a novel approach applicable to both specific and general domains to synthesize high-fidelity complex scenes consistent with annotations for the object detection task.

Dataset synthesis for training object detection models. Copy-paste [9] is a simple but effective data augmentation method for detectors. Simple Copy-Paste [14] achieves improvements with a simple random placement strategy for objects. Following works [13, 65] generate foreground objects and then paste them on background images. However, generated images by these methods usually have obvious artifacts on the boundary of pasted regions. DatasetGAN [64] takes an early step to generate labeled images automatically. Diffusion models have been used to synthesize training data and benefit downstream tasks including object detection [5, 10, 11, 63], image classification [1, 8, 18, 34, 47, 51], and semantic segmentation [15, 23, 31, 36, 39, 56, 57, 59, 60]. Modern approaches for image-label pairs generation can be roughly divided into two categories. One group of works [11, 15, 31, 36, 56, 59, 63] first generate images and then apply a pre-trained perception model to generate pseudo labels on the synthetic images. The other group [5, 23, 39] uses the same strategy as our approach to synthesize images under the guidance of annotations as input conditions. The second group also overlaps with layout-to-image approaches [24, 33, 55, 61, 62, 66, 67]. Our approach should be assigned to the second group and is designed to address challenging cases like multi-class objects, occlusions between objects, and specific domains.

3 Method

This section presents ODGEN, a novel approach to generate high-quality images conditioned on bounding box labels for specific domains. Firstly, we propose a new method to fine-tune the diffusion model in Sec. 3.1. Secondly, we design an object-wise conditioning method in Sec. 3.2. Finally, we introduce a generation pipeline to build synthetic datasets for detector training enhancement in Sec. 3.3.

3.1 Domain-specific Diffusion Model Fine-tuning

Modern Stable Diffusion [44] models are trained on the LAION-5B [48] dataset. Therefore, the textual prompts and generated images usually follow similar distributions to the web crawl data. However, real-world object detection datasets could come from very specific domains. We fine-tune the UNet of the Stable Diffusion to fit the distribution of an arbitrary object detection dataset. For training images, we use not only the entire images from the dataset but also the crops of foreground objects. In particular, we crop foreground objects and resize them to 512 × 512 . In terms of the text input, as shown in Fig. 2 (a), we use templated condition "a <scene> " named c s for entire images and "a <classname> " named c o for cropped objects. If the names are terminologies, following the subject-driven approach in DreamBooth [45], we can employ identifiers like "[V]" to represent the target objects and scenes, improving the robustness to textual ambiguity under the domain-specific context. x t s and x t o represent the input of scenes x s and objects x t added to noises at time step t . ϵ s and ϵ o represent the added noises following normal Gaussian distributions. The pre-trained Stable

ing labels scene'

ine-tune cle'

Diffusion Model convert

Class labels

stack

Synthesized objects

Fine-tuned

Diffusion Model

'A  motorcycle  and  vehicles

in  a  traffic  scene'

CLIP

Text

Encoder

TEXT LIST

IMAGE LIST

Image

Encoder

concat in channel

& padding in channel

Text

Embed

Encoder

random noise

predicted noise

reconstruction loss with

foreground reweighted

(c)

(b)

Trainable weights

Frozen weights

Synthesized sample

[ 'A  motorcycle',

'A  vehicle',

'A  vehicle' ]

TEXT LIST

padding empty

strings

ControlNet

IMAGE LIST

bounding box positions

crop and resize

'A  traffic scene'

'A  motorcycle'

shared weights

Diffusion

Model

Diffusion

Model

predicted noise

predicted noise

reconstruction

loss

random noise

random noise

reconstruction

loss

(a)

from (b)

from (b)

motorcycle

vehicle

vehicle

Fine-tuned

Diffusion Model

class

labels

Object-wise Conditioning Module

Diffusion parameterized by θ is fine-tuned with the reconstruction loss as:

where λ controls the relative weight for the reconstruction loss of scene images. τ is the frozen CLIP text encoder. Our approach guides the fine-tuned model to capture more details of foreground objects and maintain its capability of synthesizing the complete scenes.

3.2 Object-wise Conditioning

ControlNet [62] can perform controllable synthesis with visual conditions. However, the control can be challenging with an increasing object number and category number due to the "concept bleeding" phenomenon. Stronger conditions are needed to ensure high-fidelity generation. To this end, we propose a novel object-wise conditioning strategy with ControlNet.

Text list encoding. As shown in Fig. 2 (b) and (c), given a set of object class names and their bounding boxes, we first build a text list consisting of each object with the fixed template "a <classname> ". Then the text list is padded with empty texts to a fixed length N and converted to a list of embeddings by a pre-trained CLIP text tokenizer and encoder. The embeddings are stacked and encoded by a 4-layer convolutional text embedding encoder, and used as the textual condition for ControlNet. The text encoding of native ControlNet compresses the information in global text prompts altogether, resulting in mutual interference between distinct categories. To alleviate this "concept bleeding" problem of multiple categories, our two-step text encoding enables the ControlNet to capture the information of each object with separate encoding.

'

<classname 1>  and  <classname 2>

and  <classname 3>  in  a  <scene>

'

Text Prompt

'

a  <classname 1>

'

'

a  <classname 2>

'

'

a  <classname 2>

'

'

a  <classname 3>

'

Image List

Synthesized

Samples

Detection Datasets

sample

Class-wise Statistics

Bounding box area

Bounding box aspect ratio

Bounding box location

Bounding box number

classname

1

classname

2

classname

3

classname

2

Text List

ControlNet

Fine-tuned

Diffusion Model

Cropped

Objects

Filtered Labels

Object distribution estimation

Dataset Synthesis

Corrupted label filtering

Fore/Back-

ground

Discriminator

Image list encoding. As shown in Fig. 2 (b) and (c), for each item in the text list, we use the fine-tuned diffusion model to generate images for each foreground object. We then resize each generated image and paste it on an empty canvas based on the corresponding bounding box. The set of pasted images is denoted as an image list, which contains both conceptual and spatial information of the objects. The image list is concatenated and zero-padded to N in the channel dimension and then encoded to the size of latent space by an image encoder. Applying an image list rather than pasting all objects on a single image can effectively avoid the influence of object occlusions.

Optimization. With a pair of image and object detection annotation, we generate the text list c tl and image list c il as introduced in this section. The input global text prompt c t of the diffusion model is composed of the object class names and a scene name, which is usually related to the dataset name and fixed for each dataset. The ControlNet, along with the integrated encoders, are trained with the reconstruction loss. In addition, the weights of foreground regions are enhanced to produce more realistic foreground objects in complex scenes. The overall loss function is formulated as:

where M represents a binary mask with 1 on foreground pixels 0 on background pixels. ⊙ represents element-wise multiplication, and γ controls the foreground re-weighting.

3.3 Dataset Synthesis Pipeline for Object Detection

Our dataset synthesis pipeline is summarized in Fig. 3. We generate random bounding boxes and classes as pseudo labels based on the distributions of the training set. Then, the pseudo labels are converted to triplets: <image list, text list, global text prompt>. ODGEN uses the triplets as inputs to synthesize images with the fine-tuned diffusion model. In addition, we filter out the pseudo labels in which areas the foreground objects fail to be generated.

Object distribution estimation. To simulate the training set distribution, we compute the mean and variance of bounding box attributes and build normal distributions. In particular, given a dataset with K categories, we calculate the class-wise average occurrence number per image µ = ( µ 1 , µ 2 , . . . , µ K ) and the covariance matrix Σ to build a multi-variable joint normal distribution N ( µ , Σ ) . In addition, for each category k , we build normal distributions N ( µ x , σ 2 x ) and N ( µ y , σ 2 y )

MRI

Image

Apex

Game

Underwater

Cotton

ODGEN

ControlNet

GLIGEN

ReCo

Annotation

GeoDiffusion

positive

avatar

avatar

scallop

scallop

scallop

holothurian

G-arboreum

G-arboreum

G-herbaceum

for the top-left coordinate of the bounding boxes, N ( µ area , σ 2 area ) for box areas, and distributions N ( µ ratio , σ 2 ratio ) for aspect ratios.

Image synthesis. We first sample the object number per category from the K -dimensional joint normal distributions. Then for each object, we sample its bounding box location and size from the corresponding estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts. Finally, we use ODGEN to generate images with the triplets.

Corrupted label filtering. There is a certain possibility that some objects fail to be generated in synthetic images. While utilizing image lists alleviates the concern to some extent, the diffusion model may still neglect some objects in complex scenes containing dense or overlapping objects. As a result, the synthesized pixels could not match the pseudo labels and will undermine downstream training performance. We fine-tune a ResNet50 [17] with foreground and background patches cropped from the training set to classify whether the patch contains an object. The discriminator checks whether objects are successfully synthesized in the regions of pseudo labels, rejects nonexistent ones, and further improves the accuracy of synthetic datasets.

4 Experiments

We conduct extensive experiments to demonstrate the effectiveness of the proposed ODGEN in both specific and general domains. FID [19] is used as the metric of fidelity. Mean average precisions (mAP) of YOLOv5s [25] and YOLOv7 [54] trained with synthetic data are used to evaluate the trainability, which concerns the usefulness of synthetic data for detector training. Our approach is implemented with Stable Diffusion v2.1 and compared with prior controllable generation methods based on Stable Diffusion, including ReCo [61], GLIGEN [33], ControlNet [62], GeoDiffusion [5], InstanceDiffusion [55], and MIGC [67]. Native ControlNet doesn't support bounding box conditions. Therefore, we convert boxes to a mask C sized H × W × K , where H and W are the image height and width, and K is the class number. If one pixel ( i, j ) is in n boxes of class k , the C [ i, j, k ] = n . YOLO models are trained with the same recipe as Roboflow-100 [6] for 100 epochs to ensure convergence. The length of text and image lists N used in our ODGEN is set to the maximum object number per image in the training set.

4.1 Specific Domains

Roboflow-100 [6] is used for the evaluation in specific domains. It consists of 100 object detection datasets of various domains. We select 7 representative datasets (RF7) covering video games, medical imaging, and underwater scenes. To simulate data scarcity, we only sample 200 images as the training set for each dataset. The whole training process including the fine-tuning on both cropped objects and entire images and the training of the object-wise conditioning module, only depends on the 200 images. λ in Eq. (1) and γ in Eq. (2) are set as 1 and 25. We first fine-tune the diffusion model according to Fig. 2 (a) for 3k iterations. Then we train the object-wise conditioning modules on a V100 GPU with batch size 4 for 200 epochs, the same as the other methods to be compared.

4.1.1 Fidelity

For each dataset in RF7, we compute the FID scores of 5000 synthetic images against real images, respectively. As shown in Tab. 1, our approach outperforms all the other methods. We visualize the generation quality in Fig. 4. GLIGEN only updates gated self-attention layers inserted to Stable Diffusion, making it hard to be generalized to specific domains like MRI images and the Apex game. ReCo and GeoDiffusion integrate encoded bounding box information into text tokens, which require more data for diffusion model and text encoder fine-tuning. With only 200 images, they fail to generate objects in the given box regions. ControlNet, integrating the bounding box information into the visual condition, presents more consistent results with the annotation. However, it may still miss some objects or generate wrong categories in complex scenes. ODGEN achieves superior performance on both visual effects and consistency with annotations.

4.1.2 Trainability

To explore the effectiveness of synthetic data for detector training, we train YOLO models pre-trained on COCO with different data and compare their mAP@.50:.95 scores in Tab. 2. In the baseline setting, we only use the 200 real images for training. For the other settings, we use a combination of 200 real and 5000 synthetic images. Our approach gains improvement over the baseline and outperforms all the other methods.

In addition, we add experiments with larger-scale datasets with 1000 images sampled from the Apex Game and the Underwater datasets. The training setups are kept the same as the training on 200 images. We conduct experiments on ODGEN, ReCo, and GeoDiffusion. ReCo and GeoDiffusion

ODGEN

ReCo

GLIGEN

ControlNet

GeoDiffusion

MIGC

Annotation

cat

bed

bottle

cell phone

cup

may benefit from larger-scale training datasets since they need to fine-tune more parameters in both the UNet in Stable Diffusion and the CLIP text encoder. GLIGEN struggles to adapt to new domains. ControlNet performs worse on layout control than ODGEN. Therefore, these two methods are not included in this part. The corrupted label filtering step is not used for any method. Results in Tab. 3 show that the baselines trained on real data only become stronger with larger-scale datasets while our ODGEN still benefits detectors with synthetic data and outperforms other methods.

4.2 General Domains

COCO-2014 [37] is used to evaluate our method in general domains. As the scenes in this dataset are almost covered by the pre-training data of Stable Diffusion, we skip the domain-specific fine-tuning stage and train the diffusion model together with the object-wise conditioning modules. We train our ODGEN on the COCO training set with batch size 32 for 60 epochs on × 8 V100 GPUs as well as GLIGEN, ControlNet, and GeoDiffusion. Officially released checkpoints of ReCo (trained for 100 epochs on COCO), MIGC (trained for 300 epochs on COCO), and InstanceDiffusion (trained on self-constructed datasets) are employed for comparison. γ in Eq. (2) is set as 10 for our ODGEN.

4.2.1 Fidelity

We use the annotations of the COCO validation set as conditions to generate 41k images. The FID scores in Tab. 4 are computed against the COCO validation set. ODGEN achieves better FID results than the other methods. We provide a typical visualized example in Fig. 5. Given overlapping bounding boxes of multiple categories, ODGEN synthesizes all the objects of correct categories and accurate positions, outperforming the other methods. More samples are added in Appendix G.

4.2.2 Trainability

We use 10k annotations randomly sampled from the COCO validation set to generate a synthetic dataset of 10k images. The YOLO models are trained on this synthetic dataset from scratch and evaluated on the other 31k images in the COCO validation set. As shown in Tab. 4, ODGEN achieves significant improvement over the other methods in terms of mAP.

We further conduct experiments by adding 20k synthetic images to the 80k training images. We train YOLO models from scratch on the COCO training set (80k images) as the baseline and on the same 80k real images + 20k synthetic images generated by different methods for comparison. We use the labels of 20k images from the COCO validation set as conditions to generate the synthetic set and use the other 21k real images for evaluation. The results are shown in Tab. 5. It shows that ODGEN improves the mAP@.50:95 by 0.5% and outperforms the other methods.

We observe that YOLO models trained on synthetic data only fall behind models trained on real data. We add experiments with different training and validation data combinations in Tab. 6. YOLO models trained on real images show better generalization ability and achieve close results when tested on real and synthetic data. YOLO models trained on synthetic data only get significantly better results when tested on synthetic data than when generalized to real data. It indicates that noticeable domain gaps still exist between real and synthetic data, which may be limited by the generation quality of modern Stable Diffusion models and are promising to be narrowed with future models.

4.3 Ablation Study

Image list and text list in object-wise conditioning modules. ODGEN enables object-wise conditioning with an image list and a text list as shown in Fig. 2. We test the performance of

Image List:

Image List:

Image List:

Image List:

Image

eList:

ImageList:

TextList:

Text List:

Text List:

Text List:

Text List:

Text List:

bus

traffic light

bus

traffic light

motorcycles

motorcycles

motorcycle

motorcycle

motorcycle

motorcycle

vehicle

vehicle

trafficlight

traffic

ight

𝛾 = 0

𝛾 = 25

𝛾 = 50

𝛾 = 100

vehicle

vehicle

bus

Annotation

using global text prompts in the placement of text lists or pasting all the foreground patches on the same empty canvas in the placement of image lists for the ablation study. Experiments are performed on the MRI and Road Traffic datasets in RF7. The MRI dataset only has two categories and almost no overlapping objects, whereas the road traffic dataset has 7 categories and many occluded objects. From the results in Tab. 7, it can be found that using image lists or text lists brings moderate improvement on MRI images and significant improvement on the more complex Road Traffic dataset. Visualization in Fig. 6 shows that the image list improves the fidelity under occlusion, and the text list mitigates the mutual interference of multiple objects.

Foreground region enhancement. We introduce a reweighting term controlled by γ for foreground objects in Eq. (2). Tab. 8 shows that appropriate γ values can improve both the fidelity and the trainability since the foreground details are better generated. However, increasing the value will undermine the background quality, as visualized in Fig. 6.

Corrupted label filtering. Results in Tab. 9 show that the corrupted label filtering step helps improve the mAP@.50:95 of ODGEN by around 1-2%. The results of ODGEN without the corrupted label filtering still significantly outperform the baseline setting in Tab. 2.

5 Conclusion

This paper introduces ODGEN, a novel approach aimed at generating domain-specific object detection datasets to enhance the performance of detection models. We propose to fine-tune the diffusion model on both cropped foreground objects and entire images. We design a mechanism to control the foreground objects with object-wise synthesized visual prompts and textual descriptions. Our work significantly enhances the performance of diffusion models in synthesizing complex scenes characterized by multiple categories of objects and bounding box occlusions. Extensive experiments demonstrate the superiority of ODGEN in terms of fidelity and trainability across both specific and general domains. We believe this work has taken an essential step toward more robust image synthesis and highlights the potential of benefiting the object detection task with synthetic data.

References

S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research , 2023.

Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference on Machine Learning , pages 1737-1752, 2023.

A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

K. Chen, E. Xie, Z. Chen, Y. Wang, L. Hong, Z. Li, and D.-Y. Yeung. Geodiffusion: Text-prompted geometric con-trol for object detection data generation. In International Conference on Learning Representations , 2024.

F. Ciaglia, F. S. Zuppichini, P. Guerrie, M. McQuade, and J. Solawetz. Roboflow 100: A rich, multi-domain object detection benchmark. arXiv preprint arXiv:2211.13523 , 2022.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

L. Dunlap, A. Umino, H. Zhang, J. Yang, J. E. Gonzalez, and T. Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. Advances in Neural Information Processing Systems , 36, 2024.

N. Dvornik, J. Mairal, and C. Schmid. Modeling visual context is key to augmenting object detection datasets. In Proceedings of the European Conference on Computer Vision , pages 364-380, 2018.

H. Fang, B. Han, S. Zhang, S. Zhou, C. Hu, and W.-M. Ye. Data augmentation for object detection via controllable diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1257-1266, 2024.

C. Feng, Y. Zhong, Z. Jie, W. Xie, and L. Ma. Instagen: Enhancing object detection by training on synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14121-14130, 2024.

W. Feng, X. He, T.-J. Fu, V. Jampani, A. R. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations , 2022.

Y. Ge, J. Xu, B. N. Zhao, N. Joshi, L. Itti, and V. Vineet. Dall-e for detection: Language-driven compositional image synthesis for object detection. arXiv preprint arXiv:2206.09592 , 2022.

G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y . Lin, E. D. Cubuk, Q. V . Le, and B. Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2918-2928, 2021.

R. Gong, M. Danelljan, H. Sun, J. D. Mangas, and L. Van Gool. Prompting diffusion representations for cross-domain semantic segmentation. arXiv preprint arXiv:2307.02138 , 2023.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems , 27, 2014.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. QI. Is synthetic data from generative models ready for image recognition? In The Eleventh International Conference on Learning Representations , 2022.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30, 2017.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems , 36, 2024.

M. Jahn, R. Rombach, and B. Ommer. High-resolution complex scene synthesis with transformers. arXiv preprint arXiv:2105.06458 , 2021.

Y. Jia, L. Hoyer, S. Huang, T. Wang, L. Van Gool, K. Schindler, and A. Obukhov. Dginstyle: Domaingeneralizable semantic segmentation with image diffusion models and stylized semantic control. In Synthetic Data for Computer Vision Workshop@ CVPR 2024 , 2023.

C. Jiaxin, L. Xiao, S. Xingjian, H. Tong, X. Tianjun, and L. Mu. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.08908 , 2023.

G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, Y. Kwon, K. Michael, J. Fang, C. Wong, Z. Yifu, D. Montes, et al. ultralytics/yolov5: v6. 2-yolov5 classification models, apple m1, reproducibility, clearml and deci. ai integrations. Zenodo , 2022.

T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems , 34:852-863, 2021.

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4401-4410, 2019.

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8110-8119, 2020.

D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in Neural Information Processing Systems , 34:21696-21707, 2021.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

N. Kondapaneni, M. Marks, M. Knott, R. Guimaraes, and P. Perona. Text-image alignment for diffusionbased perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13883-13893, 2024.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation. In International conference on machine learning , pages 1288812900. PMLR, 2022.

Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y . J. Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22511-22521, 2023.

Z. Li, Y. Li, P. Zhao, R. Song, X. Li, and J. Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint arXiv:2305.12954 , 2023.

Z. Li, J. Wu, I. Koh, Y . Tang, and L. Sun. Image synthesis from layout with locality-aware mask adaption. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13819-13828, 2021.

Z. Li, Q. Zhou, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Open-vocabulary object segmentation with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7667-7676, 2023.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

D. Peng, P. Hu, Q. Ke, and J. Liu. Diffusion-based image translation with label guidance for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 808-820, 2023.

D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations , 2024.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , pages 1278-1286. PMLR, 2014.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-toimage diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

M. B. Sarıyıldız, K. Alahari, D. Larlus, and Y. Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8011-8021, 2023.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265, 2015.

Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems , 33:12438-12448, 2020.

B. Trabucco, K. Doherty, M. A. Gurinas, and R. Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations , 2023.

A. Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems , 33:19667-19679, 2020.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao. Yolov7: Trainable bag-of-freebies sets new state-of-theart for real-time object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7464-7475, 2023.

X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6232-6242, 2024.

W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems , 36:54683-54695, 2023.

W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixellevel annotations for semantic segmentation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1206-1217, 2023.

J. Xie, Y. Li, Y . Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7452-7461, 2023.

J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2955-2966, 2023.

L. Yang, X. Xu, B. Kang, Y. Shi, and H. Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. Advances in Neural Information Processing Systems , 36, 2024.

Z. Yang, J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan, Z. Liu, C. Liu, M. Zeng, et al. Reco: Regioncontrolled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14246-14255, 2023.

L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

M. Zhang, J. Wu, Y. Ren, M. Li, J. Qin, X. Xiao, W. Liu, R. Wang, M. Zheng, and A. J. Ma. Diffusionengine: Diffusion model is scalable data engine for object detection. arXiv preprint arXiv:2309.03893 , 2023.

Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10145-10155, 2021.

H. Zhao, D. Sheng, J. Bao, D. Chen, D. Chen, F. Wen, L. Yuan, C. Liu, W. Zhou, Q. Chu, et al. X-paste: revisiting scalable copy-paste for instance segmentation using clip and stablediffusion. In International Conference on Machine Learning , pages 42098-42109. PMLR, 2023.

G. Zheng, X. Zhou, X. Li, Z. Qi, Y. Shan, and X. Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22490-22499, 2023.

D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang. Migc: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6818-6828, 2024.

J. Zhu, H. Ma, J. Chen, and J. Yuan. Isolated diffusion: Optimizing multi-concept text-to-image generation training-freely with isolated diffusion guidance. arXiv preprint arXiv:2403.16954 , 2024.

A Limitations

Despite the significant improvement achieved by this work, we find it remains challenging to achieve comparable training performance as real data with merely synthetic data. We consider the limitation to be mainly caused by two reasons. Firstly, the generation fidelity is still limited by modern diffusion models. For instance, VAEs in Stable Diffusion are not good enough to deal with complex text in images, constraining the performance on video game datasets. ODGEN is promising to achieve further improvements with more powerful diffusion models in the future. Secondly, this paper mainly focuses on designing generative models with given conditions. However, the other parts in the dataset synthesis pipeline are not fully optimized, which are interesting to be explored in future work.

B Broader Impact

We introduce a novel approach for object detection datasets synthesis, achieving improvement in fidelity and trainability for both specific and general domains. Our approach is more prone to biases caused by training data than typical AI generative models since it shows robustness with limited data of specific domains. Therefore, we recommend practitioners to apply abundant caution when dealing with sensitive applications to avoid problems of races, skin tones, or gender identities.

C Supplemental Ablations

Detector training w/o real data. In the main paper, we train the detectors with 200 real images and 5000 synthetic images for RF7 benchmarks. In this ablation study, we conduct experiments with the 5000 synthetic images only to explore the impact of the absence of real data for training detectors. As shown in Tab. 10, for easy datasets with fewer objects, like MRI image and Cotton, training with 5000 synthetic data only achieves better results than training with 200 real samples only. Increasing the dataset size can improve the model performance because the generation quality is good enough in these cases. Nevertheless, for complex scenes like Robomaster and Aquarium, training with synthetic data only fails to achieve better results. It may be caused by the limited generation quality of modern diffusion models. For instance, the V AE in Stable Diffusion is not good at synthesizing texts in images, which is common in the Robomaster dataset. Besides, FID results in Tab. 1 indicate that there still exist gaps between the distributions of real and synthetic images. The performance of our approach is promising to be further improved in the future with more powerful generative diffusion models. In addition, the current dataset synthesis pipeline is not fully optimized and cannot reproduce the distributions of foreground objects perfectly, which may also lead to worse performance of training on synthetic data only with the RF7 datasets.

Number of synthesized samples. We ablate the number of synthesized samples used for detector training and provide quantitative results in Tab. 11. With increasing amounts of synthesized samples, detectors achieve higher performance and get very close results between 5000 and 10000 synthetic samples.

Category isolation for datasets synthesis. As illustrated in the main paper, modern diffusion models have the limitation of "concept bleeding" when multiple categories are involved in the generation of one image. An alternative method could be only generating objects of the same category in one image. We select 3 multi-class datasets and compare this approach against the proposed method in ODGEN. As shown in Tab. 12, ODGEN achieves better results on most benchmarks than generating samples with isolated categories. This is an expected result since generating objects of various categories in one image should be better aligned with the test set.

Corrupted label filtering. We design this step to filter some labels when objects are not generated successfully, which may be caused by some unreasonable boxes obtained with the pipeline in Fig. 3. For experiments on COCO, this step is not applied to any method since we directly use labels from the COCO validation set and hope to synthesize images consistent with the real-world labels. For RF7 experiments, this step is only applied to our ODGEN in Tab. 2. For fair comparison, we skip this step on RF7 to compare the generation capability of different methods fairly and provide results in Tab. 13. It shows that the corrupted label filtering step only contributes a small part of the improvement. Without this step, our method still outperforms the other methods significantly. In addition, as illustrated in Appendix A, the current dataset synthesis pipeline is designed to compare different methods and can be improved further in future work.

Foreground region enhancement on COCO. We set γ as 10 in Eq. (2) in our experiments trained on COCO. We add ablations with γ value of 0 to ablate foreground region enhancement on COCO and provide quantitative results in Tab. 14. It shows that the foreground region enhancement helps ODGEN achieve better results, especially the layout-image consistency reflected by mAP results.

Fine-tuning on both cropped foreground objects and entire images. For specific domains, taking the Robomaster dataset in RF7 as an example, most images in the dataset contain multiple categories of objects like "armor", "base", "watcher", and "car", which are either strange for Stable Diffusion or different from what Stable Diffusion tends to generate with the same text prompts. For models fine-tuned on entire images only, they cannot obtain direct guidance on which parts in entire images correspond to these objects. As a result, the fine-tuned models cannot synthesize correct images for foreground objects given text prompts like "a base in a screen shot of the Robomaster game". We provide FID results of foreground objects synthesized by models fine-tuned on both entire images and cropped foreground objects (text prompts are composed of object names and the scene name,

"A bicycle in a traffic scene

"Crosswalks in a traffic scene”

"A fire hydrant in a traffic scene'

"Traffic lights in a traffic scene'

"A vehicle in a traffic scene'

"A bus in a traffic scene”

e.g., "a car and a base in a screen shot of the Robomaster game") and models fine-tuned on entire images only in Tab. 15. It shows that the proposed approach helps fine-tuned models generate images of foreground objects significantly better. Similarly, we also provide FID results of the Road Traffic dataset in Tab. 16, which is relatively more familiar for Stable Diffusion than the Robomaster dataset. Results also show the improvement provided by our approach compared with models fine-tuned on entire images only. Visualized examples produced by the model fine-tuned on entire images only are shown in Fig. 8. Such models struggle to capture the information of foreground objects and fail to produce images needed to build image lists for our ODGEN.

D Implementation Details

D.1 Global Text Prompt

All the methods included for comparison in our experiments share the same textual descriptions of the foreground objects and entire scenes in the global text prompts for a fair comparison. ReCo [61] and GeoDiffusion [5] further add the location information of foreground objects to their global text prompts following their approaches. For general domains, we concatenate the object class names into a sentence as the global text prompt. For specific domains, we concatenate the object class names and the scene name into a sentence as the global text prompt.

D.2 ODGEN (ours)

We follow the default training setting used by ControlNet [62], including the learning rates and the optimizer to train our ODGEN models and provide fair comparison with ControlNet.

Offline foreground objects synthesis. To improve the training efficiency, we synthesize 500 foreground object images for each category offline and build a synthetic foreground pool. During the training of object-wise conditioning modules, foreground images are randomly sampled from the pool to build the image list.

Image encoder. ControlNet uses a 4-layer convolutional encoder to encode input conditions sized 512 to 64 × 64 latents. The input channels are usually set as 1 or 3 for input images. Our ODGEN

follows similar methods to convert the input conditions. The key difference is that we use image lists of synthesized objects as input conditions. The image lists are padded to N and concatenated in the channel dimension for encoding. In this case, we expand the input channels of the encoder to 3 N and update the channels of the following layers to comparable numbers. We provide the channels of each layer in Tab. 17. The convolutional kernel size is set as 3 for all layers. We design different encoder architectures according to the maximum object numbers that can be found in a single image. For datasets like MRI in which most images contain only one object, we can use a smaller encoder to make the model more lightweight. For a generic model trained on large-scale datasets, we can fix the architectures of encoders with a large N to make it applicable to images containing many objects.

Text embedding encoder. Similar to the image list, the text list is also padded to the length N with empty text. Each item in the text list in encoded by the CLIP text encoder to an embedding. With the CLIP text encoder used in Stable Diffusion 1.x and 2.x models, we get N text embeddings sized ( batch size , 77 , 1024) . Then we stack the embeddings to size ( batch size , N, 77 , 1024) and use a 4-layer convolutional layer to compress the information into one embedding. The input channel is N and the output channels of following layers are ⌊ N/ 2 ⌋ , ⌊ N/ 4 ⌋ , ⌊ N/ 8 ⌋ , 1 . We set the convolutional kernel size as 3, the stride as 1, and use zero padding to maintain the sizes of the last two dimensions.

Foreground/Background discriminator. We train discriminators by fine-tuning ImageNet pretrained ResNet50 on foreground and background patches randomly cropped from training sets. They are split for training, validating, and testing by 70%, 10%, and 20%. The model is a binary classification model that only predicts whether a patch contains any object. The accuracy on test datasets is over 99% on RF7. Therefore, we can confidently use it to filter the pseudo labels.

D.3 Other Methods

ReCo [61] & MIGC [67] & GeoDiffusion [5] are layout-to-image generation methods based on Stable Diffusion. For the comparison on RF7, we use the official codes of ReCo and GeoDiffusion to fine-tune the diffusion models on domain-specific datasets and generate synthetic datasets for experiments. For the comparison on COCO, we directly use their released pre-trained weights trained on COCO. MIGC only releases the pre-trained weight but not the code, so we only compare it with our method on the COCO benchmark. ReCo and GeoDiffusion fine-tune both the text encoder and the UNet of Stable Diffusion. MIGC only fine-tunes the UNet.

InstanceDiffusion [55] depends on multiple formats of inputs, including segmentation masks, boxes, and scribbles, to train its UniFusion module. However, the RF7 datasets of specific domains do not provide segmentation masks and scribbles. Therefore, it is only employed for comparison on COCO with its open-source checkpoint.

GLIGEN [33] is an open-set grounded text-to-image generation methods supporting multiple kinds of conditions. We use GLIGEN for layout-to-image generation with bounding boxes as conditions. GLIGEN is designed to fit target distributions by only fine-tuning the inserted gated self-attention layers in pre-trained diffusion models. Therefore, we fix the other layers in the diffusion model during fine-tuning on RF7 or COCO.

ControlNet [62] is initially designed to control diffusion models by fixing pre-trained models, reusing the diffusion model encoder and fine-tuning it to provide control. In our experiments, to make fair comparisons, we also fine-tune the diffusion model part to fit it to specific domains that can be different from the pre-training data of Stable Diffusion. The diffusion model remains trainable for experiments on COCO as well. The native ControlNet is originally designed as an image-to-image generation method. As illustrated in Sec. 4, we provide a simple way to convert boxes to masks. Given a dataset containing K categories, an image sized H × W , and B objects with their bounding boxes bbox and classes cls , we convert them to a mask M in the following process:

E Datasets Details

RF7 datasets. We employ the RF7 datasets from Roboflow-100 [6] to evaluate the performance of our ODGEN on specific domains. Here, we add more details of the employed datasets, including Apex Game, Robomaster, MRI Image, Cotton, Road Traffic, Aquarium, and Underwater. We provide the number of images for each dataset, including the train, validation, and test sets in Tab. 18. We use the first 200 samples in the image list annotation provided by Roboflow-100 for each dataset to build the training set for our experiments. The full validation set is used as the standard to choose the best checkpoint from YOLO models trained for 100 epochs. The full test set is used to evaluate the chosen checkpoint and provide mAP results shown in this paper. Visualized examples with annotations are shown in Fig. 9. The object categories of each dataset are provided in Tab. 19. Some images in the Apex Game and Robomaster datasets contain complex and tiny text that are hard to generate by current Stable Diffusion.

F Supplemental Related Works

Copy-paste [14, 65] method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets used in this paper. Besides, our approach also serves as a controllable image generation method that only needs bounding box labels to produce realistic images, which is hard to achieve by copy-paste. InstaGEN [11] is mainly designed for the open-vocabulary object detection task. It generates images randomly and annotates them with a grounding head trained with another pre-trained detector. Earlier works like LayoutDiffusion [66] are not implemented with latent diffusion models and thus are not included for comparison. Works [23, 39, 56] are designed for semantic segmentation, while our ODGEN is designed for object detection.

positive

negative

negative

positive

positive

positive

Real MRI Samples

avatar

avatar

avatarvatan

a

object

avatar

20

Real Apex Game Samples

安国

舒正正证

45

base

armor

8

armc

car

une-red

armor

car

armor-

Real Robomaster Samples

G-barbadense

G-herbaceum

G-arboreum

G-arboreum

G-arboreum

G-herbaceum

shutterstrck

Real Cotton Samples

firehydrants

bicycles

vehiclevehicles

vehiclesvehicle

motorcycles

crosswalks

Real Road Traffic Samples

rapenguin

fish

stingray

puffin

IS

ish

starfish

starfish

penguin

starfish

Jfish

2

fishh

fisfish

starfish

starfish

Real Aquarium Samples

echinusainusinus

hohalathurian

holothurian

echinus

Echinus

echihusholothurian

holothurian

holothurian

holothurian

holothurian

holothurian

holathurian

holothurian

echinus"

echinus

starfish

工

holothurian

echinus

holothurian

口

echinus

Real Underwater Samples

G Supplemental Experiments

mAP@.50 on RF7. Tab. 20 provides quantitative results for trainability in terms of mAP@.50 as supplements to Tab. 2. Our approach achieves improvement over training on real data only and outperforms other methods.

Visual relationship between image lists and synthesized images. The image list is designed to provide information on category and localization for controlling foreground object layouts. The synthesized images share the same category with objects in generated images and are pasted to

the same position as those in generated images to build image lists. The objects in image lists and synthesized images may all be apples or cakes but have different shapes and colors, as shown by the visualized samples provided in Fig. 10.

Detector performance improvement with synthetic data. We provide several visualized samples in Fig. 11 and show that the synthetic data helps detectors detect some occluded objects.

Generalization to novel categories. We use ODGEN trained on COCO to synthesize samples containing subjects not included in COCO, like moon, ambulance, tiger, et al. We show visualized samples in Fig. 12. It shows that ODGEN trained on COCO can control the layout of novel categories. However, we find that the generation quality is not very stable, which may be influenced by the fine-tuning process on the COCO dataset. In future work, we hope to get a powerful and generic model that is robust to the most common categories in daily life.

Computational cost. Our ODGEN needs to synthesize images of foreground objects to build image lists, leading to higher computational costs for training and inference. Therefore, we propose to generate an offline image library of foreground objects to accelerate the process of building image lists. With the offline library, we can randomly pick images from it to build image lists instead of synthesizing new images of foreground objects every time.

Taking the model trained on the COCO dataset as an example, our ODGEN shares a very close scale of parameters with ControlNet (Trainable parameters: ODGEN 1231M v.s. ControlNet 1229M, parameters in the UNet of Stable Diffusion are included). We provide the training time for 1 epoch on COCO with 8 V100 GPUs of different methods in Tab. 21.

In the inference stage (Fig. 2(c)), our ODGEN pads both the image and text lists to a fixed length. Therefore, the computational cost for inference with an offline library doesn't increase significantly with more foreground objects. Other methods like InstanceDiffusion [55] and MIGC [67] need more time for training and inferencing with more objects. Taking models trained on COCO as examples, to generate an image with 6 bounding boxes on a V100 GPU, ODGEN takes 10 seconds, ControlNet takes 8 seconds, and InstanceDiffusion takes 30 seconds.

Motoreyele

Motoreyele

/ehiele

Apple

Cake

Surfboard

TrafficLight

Traffie Light

山

Vehicle

Vehicle

Vehicle

Vehicle

Person

Crosswalk

tie

Vehicle

Vehicle

Crosswalk

Crosswalk

Person

Synthesized Images

SynthesizedImages

used in Image Lists

During the inference process, we compare using the same offline image library as training against using a totally different image library. We get very close results, as shown in Tab. 22. Both outperform other methods significantly, as shown in Tab. 4. It indicates that ODGEN can extract category information from synthesized samples of foreground objects in image lists instead of depending on certain images of foreground objects. ODGEN is not constrained by the image library of foreground objects used in training and can be generalized to other newly generated offline image libraries consisting of novel synthesized samples of foreground objects, which ensures that offline image libraries increase ODGEN's computational efficiency without reducing its practicality.

Quantitative evaluation for concept bleeding. This work mainly focuses on the concept of the bleeding problem of different categories of objects. The mAP results of YOLO models trained on synthetic images only in Tab. 4 can be a proxy task to prove that the concept bleeding is alleviated since our ODGEN achieves state-of-the-art performance in the layout-image consistency of complex scene generation conditioned on bounding boxes. This section adds quantitative evaluation with BLIPVQA[21], which employs the BLIP [32] model to identify whether the contents in synthesized images are consistent with text prompts. The results are provided in Tab. 23. Our ODGEN outperforms other methods and gets results close to ground truth (real images from the COCO validation set sharing the same labels with synthetic images).

ODGEN trained on COCO with Stable Diffusion v1.5. Our ODGEN is implemented with Stable Diffusion v2.1 in Sec. 4. We add experiments of implementing ODGEN with Stable Diffusion v1.5

moon

ambulance

mountain

tree

ball

口

tiger

“An avatar in a screen shot of the

“An object in a screen shot of the

"A screen shot of the Apex game"

Apex game"

Apex game'

“A jellyfish in a photo of aquarium'

"A starfish in a photo of aquarium'

"A photo of aquarium"

"Traffic lights in a traffic scene'

“A vehicle in a traffic scene”

"A traffic scene'"

trained on COCO for comparison in this part. Results are provided in Tab. 24. Our ODGEN achieves similar results with different versions of Stable Diffusion models.

Visualization. We provide extensive visualized examples of ODGEN to demonstrate its effectiveness. In Fig. 13, we provide visualized samples produced by models fine-tuned on both cropped foreground objects and entire images. It shows that the fine-tuned models are capable of generating diverse foreground objects and complete scenes. In Fig. 14, we show generated samples of the specific domains in RF7. It can be seen that ODGEN is robust to complex scenes composed of multiple categories, dense and overlapping objects. Besides, ODGEN shows strong generalization capability to various domains.

'

a negative area

in an MRI image

'

'

an avatar in a screen

shot of the Apex game

'

'

a base and cars

in a screen shot of

the robomaster game

'

'

echini and a holothurian

in an underwater scene

'

'

jellyfishes

in a photo of aquarium

'

'

a motorcycle and

a vehicle in a

traffic scene

'

'

G-arboreum in

a photo of cotton

'

'

a positive area

in an MRI image

'

'

a rune-red

in a screen shot of

the robomaster game

'

'

G-herbaceum in

a photo of cotton

'

'

a starfish and a scallop

and an echinus in

an underwater scene'

'

a stingray and a shark in

a photo of aquarium

'

'

traffic lights and

vehicles in a

traffic scene

'

'

an avatar and an object

shot of the Apex game

'

positive

rune-red

G-herbaceum

G-herbaceum

scallop

echinus

starfish

positive

shark

stingray

vehicle

vehicle

traffic light

traffic light

vehicle

avatar

object

Synthesized

MRI Image

Samples

negative

Synthesized

Robomaster

Samples

car

car

car

base

Synthesized

Cotton

Samples

G-arboreum

G-arboreum

Synthesized

Underwater

Samples

negative

echinus

echinus

echinus

holothurian

Synthesized

Aquarium

Samples

jellyfish

jellyfish

jellyfish

jellyfish

jellyfish

Synthesized

Road Traffic

Samples

motor-

cycle

vehicle

Synthesized

Apex Game

Samples

avatar

'

a negative area

in an MRI image

'

'

a base

in a screen shot of

the robomaster game

'

'

G-herbaceum in

a photo of cotton

'

starfishes and a scallop

in an underwater scene

'

'

a fish and a shark in

a photo of aquarium

'

'

crosswalks and

vehicles in a

traffic scene

'

'

an avatar in a screen

shot of the Apex game

'

negative

base

G-herbaceum

G-herbaceum

starfish

starfish

scallop

negative

fish

shark

vehicle

vehicle

vehicle

crosswalk

avatar

'

a positive area

in an MRI image

'

'

a base and cars

in a screen shot of

the robomaster game

'

'

G-arboretum and

G-barbadense

in a photo of cotton

'

'

holothurians in an

underwater scene

'

'

fishes in a

photo of aquarium

'

'

a fire hydrant

in a traffic

road scene

'

'

an object in a screen

shot of the Apex game

'

positive

car

base

car

car

G-arboreum

G-barbadense

holothurian

holothurian

holothurian

positive

fish

fish

fish

fish

fish

fish

fire hydrant

object

'

a positive area

in an MRI image

'

positive

'

cars and a rune-blue

in a screen shot of

the robomaster game

'

car

car

rune-blue

'

G-arboreun in

a photo of cotton

'

G-arboreum

'

echini and a starfish in

an underwater scene

'

starfish

echinus

echinus

'

puffins in a

photo of aquarium

'

puffin

puffin

puffin

puffin

'

motorcycles in

a traffic

scene'

motorcycle

motor-

cycle

'

avatars in a screen

shot of the Apex game

'

avatar

avatar

Sunentk

口

ReCo

GLIGEN

ControlNet

GeoDiffusion

MIGC

ODGEN

Annotation

dining table

bottle

wine glass

cup

sports ball

tennis racket

person

hot dog

person

bowl

broccoli

bench

ReCo

GLIGEN

ControlNet

GeoDiffusion

MIGC

ODGEN

Annotation

cell phone

laptop

chair

person

baseball bat

person

baseball glove

kite

person

car

ReCo

GLIGEN

ControlNet

GeoDiffusion

MIGC

ODGEN

Annotation

clock

surfboard

person

tie

person

cell phone

book

vase

chair

ODGEN

Annotation

ODGEN

Annotation

ODGEN

Annotation

horse

person

truck

tv

person

remote

cake

apple

dining table

bird

skateboard

couch

tie

person

bottle

sink

potted plant

remote

tv

bed

keyboard

cup

oven

sink

microwave

dining table

bowl

spoon

banana

chair

umbrella

spoon

cup

dining table

pizza

bottle

bowl

cup

dining table

Annotation

Seed 1

Seed 2

Seed 3

Seed 4

Label

chair

couch

tv

cell phone

bottle

sink

toilet

cup

donut

dining table

motorcycle

pizza

cake

bowl

laptop

tv

keyboard

bowl

broccoli

spoon

dining table

yleoy

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of this work in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided details of our implementation in both the main paper and the supplementary to ensure reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: This paper do not include new data. The code is not included yet. We have provided enough details for implementation to ensure reproducibility. We will release the code and models soon.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details in our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Statistical significance of the experiments are not included. The experiments cost large amounts of computational resources. For example. our ODGEN on COCO is trained for 14 days with 8 V100 GPUs. Besides, the experiments on both specific and general domains including 8 datasets and two proxy detection models show the same trend. In addition, the improvement of our method is significant compared with prior methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included the information of compute resources in our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research obeys the code of ethics of NeurIPS.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts of this paper in discussed in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We have not designed safeguards yet but our model can share the same safeguards with modern large-scale text-to-image models like Stable Diffusion.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We respect the license and terms of all the assets used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of controllable image generation from bounding boxes and text prompts for training object detectors. To fine-tune diffusion models to specific domains effectively and deal with the challenge of concept bleeding, this paper proposes a method called ODGEN. In the proposed method, the diffusion model is first fine-tuned on both entire images and cropped foreground regions. Then, text lists and image lists are utilized as input for ControlNet to avoid the concept bleeding and generate high-quality images. The experimental results show that the proposed method outperforms the state-of-the-art methods for controllable image generation on seven types of datasets in terms of FID score. In addition, the object detectors trained on the generated images from the proposed method achieve better performance than other methods.\n\nSTRENGTHS:\ni) The proposed method is quite simple yet effective. Its relative ease of implementation is helpful for computer vision practitioners and future researchers. Also, the task tackled in this paper is of practical benefit.\n\nii) This paper is well-written and easy to follow. The motivation for tackling the task is clearly described in Sec. 1, and related works are well-summarized in Sec. 2. The proposed method is clearly explained at both the idea level and procedure level in Sec. 3.\n\niii) The experiments demonstrate the high performance of the proposed method. The proposed method achieved better FID scores than other controllable image generation methods on seven types of object detection datasets. Additionally, training with synthetic images from the proposed method improved the performance of YOLO-based object detectors significantly.\n\nWEAKNESSES:\nMy concerns are mainly about the lack of detailed analysis:\n\nI) This paper claims that fine-tuning with both cropped foreground regions and entire images is one of its contributions. However, an ablation study to confirm the effectiveness of this approach is missing.\n\nii) From Fig. 4, it looks like the corrupted label filtering is performed only for the proposed method (ODGEN). Because the corrupted label filtering is a simple technique and can be applied to other methods as well, the comparisons between ODGEN and other methods not only without filtering but also with filtering should be provided for fair comparisons.\n\niii) This paper provides only the results with 200 real images. I'm curious to know how the performance of the proposed method and other methods change when more number of real images are available for training.\n\niv) In Sec. 1, the concept bleeding is raised as one of the challenges in this task. Although Fig. 6 provides an example, I would like to see a quantitative evaluation to check whether it was effectively addressed by the proposed method.\n\nQUESTIONS:\nSee iii) in Weakness.\nHow is the performance of the proposed method and other methods when more real images are available?\n\nLIMITATIONS:\nThe limitations are adequately discussed in Sec. A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is about generating synthetic (annotated) data for object detection with diffusion models that are tuned for a specific image domain. Generating synthetic annotated data can be useful for model training in situations where data is scarce. The authors propose a specific pipeline and new modules to generate such data. First, an off-the-shelf diffusion model is fine-tuned on a specific domain, for instance a specifc object detection dataset. This is done both with the full images and crops of foreground objects (via the given bounding boxes). The statistics of object categories, number of instances, bounding boxes are estimated. Then, a new layout is sampled from the distribution, the class-specific objects are individually synethesized with the tuned diffusion model. These are individually placed at their sampled bounding box location to build a so-called image list. Then, a ControlNet model is used that encodes this image list along with a text list of the object categories, which outputs the final synthesized image. The newly designed text and image encoders for the controlnet are trained. Finally, a verification step is done at the object level (classifying if the generated image contains the desired object inside a certain bounding box). The synthesized data is evaluated with the FID metric, and also used to train object detectors, which are then evaluated with mAP.\n\nSTRENGTHS:\n- The problem the paper tries to address is important and has several use cases, maybe even beyond what the paper lists.\n- I especially like the adaptation to specific domains, which is certainly relevant for many real-world use cases.\n- I enjoyed reading the paper, it's well written, has a good flow, it was easy to understand.\n- The results seem to be clearly better than prior works like GLIGEN.\n\nWEAKNESSES:\n- Positioning with respect to some prior work.\n    - In the related work section, the paragraphs on layout-to-image generation & dataset synthesis contain some highly related works (like InstanceDiffusion or others listed as the second group of data synthesis approaches). However, these works are only briefly described but not contrasted against the proposed method. So, what's the difference to InstanceDiffusion, for example?\n    - Why are there no comparisons to InstanceDiffusion in the experiments, or even simpler data augmentation techniques like \"copy-paste\"?\n- The object distributions are simplified compared to real distributions.\n    - Using a Gaussian distribution for a discrete variable seems odd. There will be some probability to sample a negative number and if you clip, you ultimately end up with a different distribution than what was estimated, I guess.\n    - It seems that all estimated distributions regarding the bounding boxes are independent. I assume that location and areas are dependent. Are the distributions modeled independently on purpose? As in, is it better for simulation to have independent distributions? Or is this just done for simplicity? And was the impact of this evaluated empirically?\n- Missing details on the usefulness of synthetic data\n    - For the domain specific domain, it seems that the diffusion models were also tuned only with the 200 selected domain-specific images (according to line 212). I found this information to be crucial in judging whether or not the experiments are valid. I suggest to highlight this aspect already in Section 4.1 - and maybe also discuss why this information is curcial. My thinking here is that if you used more data for tuning the diffusion model than you use to train the detectors, it would be unclear where performance gains come from. It could just come from the additional data used to train the diffusion model, rather than the images being synthetic.\n    - Also, does the same conclusion on object detector performance gains from Table 2 also hold when scaling up real and synthetic images? From a practical point of view, annotating 800 more images with bounding boxes would likely be affordable in most cases. So, I'm missing an experiment like in Table 2, but with more images for both real and synthetic images, e.g., 1000 and 25000 real and synthetic images.\n    - For the general domain, it's great to see in Table 3 that the detector is better when trained on ODGEN data compared to any other synthetic data. However, the real reference points should be (a) the 10k real images and (b) the combination of 10k real and K synthetic images.\n\nQUESTIONS:\n- When you fine-tune the diffusion model with driving data, don't you have issues with limited diversity? I understand that you might only be interested in the same label spaces as defined in the dataset. But for real applications, one is often interested in simulating something that is missing in a dataset. For instance, a driving dataset may contain lots of regular cars, vans and trucks. But you also want to generate emergency vehicles like police cars or ambulances.\n- Does the filtering rate of the \"corrupted label filtering\" step correlate with object occlusions? As in, are there more issues (during either the generation or even the classification/filtering) for occluded/overlapping objects?\n- An interesting application of this paper could be to improve language-based object detectors like in [A]. This work relies on GLIGEN and could benefit from ODGEN. The difference would be that the general semantic knowledge of the diffusion model would be leveraged.\n\nReferences:\n- [A] Generating Enhanced Negatives for Training Language-Based Object Detectors. Zhao et al. CVPR'24\n\nLIMITATIONS:\nThe limitations are adequately discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
ODGEN uses a diffusion-based generation model to create novel images to train object detectors. Object bounding boxes along with the object's textual description are given as a conditioning for the generation step. With these generated images they can improve the detector's performance.

STRENGTHS:
The proposed generation pipeline can be used to enrich the training dataset with more high-quality images. Since multiple objects can exist in the same image, using bounding boxes along with the textual description seems to be the better approach.  

The paper is well written and the evaluation is done with standard datasets. The improvement over the baselines shows the efficacy of the approach.

WEAKNESSES:
1. There is a high generation cost when there are multiple objects in the same image, which reduces the practicality of the proposed approach. 
2. It would be interesting to see if the generated images improve the detector's performance where it originally failed due to lack of data, such as when objects are partially occluded.  
3. How are the statistics of the object layout taken? 
4. Why only 5k images were generated? Can we see some study of gradually adding synthetic data, like 1k,5k,10k,20k, and some plateauing of the detector performance?
4. It will be only fair to compare with other generation approaches with the same post-generation filtering being applied to them. How does ODGEN compare then?
4. In line 111, it is misleading to say "a new method to fine-tune the diffusion model", as it is nothing more than just fine-tuning.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
Please refer to the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the ODGEN method to generate high-quality images conditioned on bounding boxes. ODGEN fine-tunes Stable Diffusion (SD) on domain-specific datasets to enhance image quality in specialist domains, designing a novel strategy to control SD with object-wise text prompts and synthetic visual conditions to alleviate 'concept bleeding'. Experimental results on the COCO-2014 dataset demonstrate that ODGEN surpasses other methods in control capability. Additionally, the authors design a dataset synthesis pipeline using ODGEN, showing that using additional generated training data can improve the performance of state-of-the-art (SOTA) object detectors.\n\nSTRENGTHS:\n+ The ODGEN method proposed in this paper outperforms previous methods and shows significant improvements.\n+ This paper is well organized and written. The overall paper is easy to follow.\n\nWEAKNESSES:\nThere are some unclear implementation details, and a few experiments are missing. Please refer the Questions.\n\nQUESTIONS:\n- When fine-tuning the pre-trained diffusion model, the authors use not only the entire images from the dataset but also the crops of foreground objects (i.e., resized to 512 x 512). I'm curious if this approach would help in generating particularly small objects.\n\n- During actual inference, what is the visual relationship between the image list $c_{il}$ (i.e., in Eqn. 2) and the final generated image? The authors should provide some visual examples to illustrate this.\n\n- The authors mention that the object-wise conditioning used in the paper can alleviate \"concept bleeding\" in multi-instance generation. I believe it is necessary to conduct experiments on the COCO-MIG benchmark [65], which assesses the model's ability to control both positioning and attributes, to elaborate on the attribute binding capability of the ODGEN.\n\n    [65] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR'24\n\n- Compared to GLIGEN and MIGC, ODGEN achieves better results in overlapping cases. Which module's design primarily contributes to this improvement? As this is a major issue in the industry, I hope the authors can emphasize this in the methods section.\n\n- Have the authors considered open-sourcing the code and model? I believe this would greatly benefit the community.\n\nLIMITATIONS:\n- The proposed method requires fine-tuning the entire Stable Diffusion model and training an additional ControlNet. This often requires significant computational resources, making reproduction challenging. \n- I hope the authors can open-source the corresponding model and code, as this would greatly benefit the community.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt proposes a novel data synthesis pipeline, ODGEN, which uses diffusion models to generate high-quality and controllable datasets for object detection.\nThey first fine-tune the pre-trained diffusion models on both cropped foreground objects and entire images. Next, they control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions.\n\nFor each image, the object number per category, bounding box location and size are sampled from estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts which are fed to ODGEN to generate the image.\n\nSTRENGTHS:\nThe method addresses a practical problem in machine learning, and new effective techniques are employed in the application of diffusion models to achieve significant better synthesis quality than existing approaches.\n\nFor example, two-step encoding (CLIP text tokenizer -> stacking -> encoder) of the textual condition enables the ControlNet to capture the information of each object with separate encoding and alleviates the concept bleeding problem of multiple categories. Also, encoding individual image lists rather than pasting all objects on a single image can effectively avoid the influence of object occlusions.\n\nThe experimental validation is thorough, covering multiple benchmarks and providing detailed comparisons with baseline methods.\n\nAblation analysis is insightful.\n\nWEAKNESSES:\nAs mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.\n\nFor the data scarcity experiment, authors only sample 200 images as the training set for all datasets, which is interesting but not sufficient for the analysis. More extensive evaluation can be done with higher ratios of real:synthetic data for datasets like ApexGame, Robomaster, and Underwater (and even larger datasets like COCO) .\n\nRestricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.\n\nODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.\n\nThere are no comparisons provided with the fully supervised approach (models trained on complete real training set) on representative datasets. This would help evaluate the trainability of the proposed approach over data augmentation techniques such as copy-paste.\n\nThe computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.\n\nQUESTIONS:\nAs mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.\n\nRestricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.\n\nODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.\n\nThe computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-31"}