ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models

Authors: jingyuan  zhu
shiyu  li
yuxuan  liu
jian  yuan
ping  huang
jiulong  shan
huimin  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 14:46:40

Abstract: Modern diffusion-based image generative models have made significant progress and become promising to enrich training data for the object detection task. However, the generation quality and the controllability for complex scenes containing multi-class objects and dense objects with occlusions remain limited. This paper presents ODGEN, a novel method to generate high-quality images conditioned on bounding boxes, thereby facilitating data synthesis for object detection. Given a domain-specific object detection dataset, we first fine-tune a pre-trained diffusion model on both cropped foreground objects and entire images to fit target distributions. Then we propose to control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. ODGEN exhibits robustness in handling complex scenes and specific domains. Further, we design a dataset synthesis pipeline to evaluate ODGEN on 7 domain-specific benchmarks to demonstrate its effectiveness. Adding training data generated by ODGEN improves up to 25.3% mAP@.50:.95 with object detectors like YOLOv5 and YOLOv7, outperforming prior controllable generative methods. In addition, we design an evaluation protocol based on COCO-2014 to validate ODGEN in general domains and observe an advantage up to 5.6% in mAP@.50:.95 against existing methods.

Main paper: 
 introduction: 
 Object detection is one of the most widely used computer vision tasks in real-world applications. However, data scarcity poses a significant challenge to the performance of state-of-the-art models including eDiff-I [2] , DALL•E [42] , Imagen [46] , and Stable Diffusion [40, 44] have demonstrated unparalleled capabilities to produce diverse samples with high fidelity given unfettered text prompts.Layout-to-image generation synthesizes images conditioned on graphical inputs of layouts. Pioneering works [22, 35] based on GANs [16] and transformers [53] successfully generate images consistent with given layouts. LayoutDiffusion [66] fuses layout and category embeddings and builds object-aware cross-attention for local conditioning with traditional diffusion models. LayoutDiffuse [24] employs layout attention modules for bounding boxes based on LDMs. MultiDiffusion [3] and BoxDiff [58] develop training-free frameworks to produce samples with spatial constraints. GLIGEN [33] inserts trainable gated self-attention layers to pre-trained LDMs to fit specific tasks and is hard to generalize to scenes uncommon for pre-trained models. ReCo [61] and GeoDiffusion [5] extend LDMs with position tokens added to text prompts and fine-tunes both text encoders and diffusion models to realize region-aware controls. They need abundant data to build the capability of encoding the layout information in text prompts. ControlNet [62] reuses the encoders of LDMs as backbones to learn diverse controls. However, it still struggles to deal with some complex cases. MIGC [67] decomposes multi-instance synthesis to single-instance subtasks utilizing additional attention modules. InstanceDiffusion [55] adds precise instance-level controls, including boxes, masks, and scribbles for text-to-image generation. Given annotations of bounding boxes, this paper introduces a novel approach applicable to both specific and general domains to synthesize high-fidelity complex scenes consistent with annotations for the object detection task.Dataset synthesis for training object detection models. Copy-paste [9] is a simple but effective data augmentation method for detectors. Simple Copy-Paste [14] achieves improvements with a simple random placement strategy for objects. Following works [13, 65] generate foreground objects and then paste them on background images. However, generated images by these methods usually have obvious artifacts on the boundary of pasted regions. DatasetGAN [64] takes an early step to generate labeled images automatically. Diffusion models have been used to synthesize training data and benefit downstream tasks including object detection [5, 10, 11, 63] , image classification [1, 8, 18, 34, 47, 51] , and semantic segmentation [15, 23, 31, 36, 39, 56, 57, 59, 60] . Modern approaches for image-label pairs generation can be roughly divided into two categories. One group of works [11, 15, 31, 36, 56, 59, 63] first generate images and then apply a pre-trained perception model to generate pseudo labels on the synthetic images. The other group [5, 23, 39] uses the same strategy as our approach to synthesize images under the guidance of annotations as input conditions. The second group also overlaps with layout-to-image approaches [24, 33, 55, 61, 62, 66, 67] . Our approach should be assigned to the second group and is designed to address challenging cases like multi-class objects, occlusions between objects, and specific domains. 
method: 
 This section presents ODGEN, a novel approach to generate high-quality images conditioned on bounding box labels for specific domains. Firstly, we propose a new method to fine-tune the diffusion model in Sec. 3.1. Secondly, we design an object-wise conditioning method in Sec. 3.2. Finally, we introduce a generation pipeline to build synthetic datasets for detector training enhancement in Sec. 3.3.Modern Stable Diffusion [44] models are trained on the LAION-5B [48] dataset. Therefore, the textual prompts and generated images usually follow similar distributions to the web crawl data. However, real-world object detection datasets could come from very specific domains. We fine-tune the UNet of the Stable Diffusion to fit the distribution of an arbitrary object detection dataset. For training images, we use not only the entire images from the dataset but also the crops of foreground objects. In particular, we crop foreground objects and resize them to 512 × 512. In terms of the text input, as shown in Fig. 2 (a), we use templated condition "a <scene>" named c s for entire images and "a <classname>" named c o for cropped objects. If the names are terminologies, following the subject-driven approach in DreamBooth [45] , we can employ identifiers like "[V]" to represent the target objects and scenes, improving the robustness to textual ambiguity under the domain-specific context. x t s and x t o represent the input of scenes x s and objects x t added to noises at time step t. ϵ s and ϵ o represent the added noises following normal Gaussian distributions. The pre-trained Stable Diffusion parameterized by θ is fine-tuned with the reconstruction loss as:EQUATIONwhere λ controls the relative weight for the reconstruction loss of scene images. τ is the frozen CLIP text encoder. Our approach guides the fine-tuned model to capture more details of foreground objects and maintain its capability of synthesizing the complete scenes.ControlNet [62] can perform controllable synthesis with visual conditions. However, the control can be challenging with an increasing object number and category number due to the "concept bleeding" phenomenon. Stronger conditions are needed to ensure high-fidelity generation. To this end, we propose a novel object-wise conditioning strategy with ControlNet.Text list encoding. As shown in Fig. 2 (b) and (c), given a set of object class names and their bounding boxes, we first build a text list consisting of each object with the fixed template "a <classname>".Then the text list is padded with empty texts to a fixed length N and converted to a list of embeddings by a pre-trained CLIP text tokenizer and encoder. Image list encoding. As shown in Fig. 2 (b) and (c), for each item in the text list, we use the fine-tuned diffusion model to generate images for each foreground object. We then resize each generated image and paste it on an empty canvas based on the corresponding bounding box. The set of pasted images is denoted as an image list, which contains both conceptual and spatial information of the objects. The image list is concatenated and zero-padded to N in the channel dimension and then encoded to the size of latent space by an image encoder. Applying an image list rather than pasting all objects on a single image can effectively avoid the influence of object occlusions.Optimization. With a pair of image and object detection annotation, we generate the text list c tl and image list c il as introduced in this section. The input global text prompt c t of the diffusion model is composed of the object class names and a scene name, which is usually related to the dataset name and fixed for each dataset. The ControlNet, along with the integrated encoders, are trained with the reconstruction loss. In addition, the weights of foreground regions are enhanced to produce more realistic foreground objects in complex scenes. The overall loss function is formulated as:EQUATIONwhere M represents a binary mask with 1 on foreground pixels 0 on background pixels. ⊙ represents element-wise multiplication, and γ controls the foreground re-weighting.Our dataset synthesis pipeline is summarized in Fig. 3 . We generate random bounding boxes and classes as pseudo labels based on the distributions of the training set. Then, the pseudo labels are converted to triplets: <image list, text list, global text prompt>. ODGEN uses the triplets as inputs to synthesize images with the fine-tuned diffusion model. In addition, we filter out the pseudo labels in which areas the foreground objects fail to be generated.Object distribution estimation. To simulate the training set distribution, we compute the mean and variance of bounding box attributes and build normal distributions. In particular, given a dataset with K categories, we calculate the class-wise average occurrence number per image µ = (µ 1 , µ 2 , . . . , µ K ) and the covariance matrix Σ to build a multi-variable joint normal distribution N (µ, Σ). In addition, for each category k, we build normal distributions N (µ x , σ for the top-left coordinate of the bounding boxes, N (µ area , σ 2 area ) for box areas, and distributions N (µ ratio , σ 2 ratio ) for aspect ratios. Image synthesis. We first sample the object number per category from the K-dimensional joint normal distributions. Then for each object, we sample its bounding box location and size from the corresponding estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts. Finally, we use ODGEN to generate images with the triplets.Corrupted label filtering. There is a certain possibility that some objects fail to be generated in synthetic images. While utilizing image lists alleviates the concern to some extent, the diffusion model may still neglect some objects in complex scenes containing dense or overlapping objects. As a result, the synthesized pixels could not match the pseudo labels and will undermine downstream training performance. We fine-tune a ResNet50 [17] with foreground and background patches cropped from the training set to classify whether the patch contains an object. The discriminator checks whether objects are successfully synthesized in the regions of pseudo labels, rejects nonexistent ones, and further improves the accuracy of synthetic datasets. 
experiments: 
 We conduct extensive experiments to demonstrate the effectiveness of the proposed ODGEN in both specific and general domains. FID [19] is used as the metric of fidelity. Mean average precisions (mAP) of YOLOv5s [25] and YOLOv7 [54] trained with synthetic data are used to evaluate the trainability, which concerns the usefulness of synthetic data for detector training. Our approach is implemented with Stable Diffusion v2.1 and compared with prior controllable generation methods based on Stable Diffusion, including ReCo [61] , GLIGEN [33] , ControlNet [62] , GeoDiffusion [5] , InstanceDiffusion [55] , and MIGC [67] . Native ControlNet doesn't support bounding box conditions. Therefore, we convert boxes to a mask C sized H ×W ×K, where H and W are the image height and width, and K is the class number. If one pixel (i, j) is in n boxes of class k, the C[i, j, k] = n. YOLO models are trained with the same recipe as Roboflow-100 [6] for 100 epochs to ensure convergence. The length of text and image lists N used in our ODGEN is set to the maximum object number per image in the training set. Roboflow-100 [6] is used for the evaluation in specific domains. It consists of 100 object detection datasets of various domains. We select 7 representative datasets (RF7) covering video games, medical imaging, and underwater scenes. To simulate data scarcity, we only sample 200 images as the training set for each dataset. The whole training process including the fine-tuning on both cropped objects and entire images and the training of the object-wise conditioning module, only depends on the 200 images. λ in Eq. ( 1) and γ in Eq. ( 2) are set as 1 and 25. We first fine-tune the diffusion model according to Fig. 2 (a) for 3k iterations. Then we train the object-wise conditioning modules on a V100 GPU with batch size 4 for 200 epochs, the same as the other methods to be compared.For each dataset in RF7, we compute the FID scores of 5000 synthetic images against real images, respectively. As shown in Tab. 1, our approach outperforms all the other methods. We visualize the generation quality in Fig. 4 . GLIGEN only updates gated self-attention layers inserted to Stable Diffusion, making it hard to be generalized to specific domains like MRI images and the Apex game.ReCo and GeoDiffusion integrate encoded bounding box information into text tokens, which require more data for diffusion model and text encoder fine-tuning. With only 200 images, they fail to generate objects in the given box regions. ControlNet, integrating the bounding box information into the visual condition, presents more consistent results with the annotation. However, it may still miss some objects or generate wrong categories in complex scenes. ODGEN achieves superior performance on both visual effects and consistency with annotations.To explore the effectiveness of synthetic data for detector training, we train YOLO models pre-trained on COCO with different data and compare their mAP@.50:.95 scores in Tab. 2. In the baseline setting, we only use the 200 real images for training. For the other settings, we use a combination of 200 real and 5000 synthetic images. Our approach gains improvement over the baseline and outperforms all the other methods.In addition, we add experiments with larger-scale datasets with 1000 images sampled from the Apex Game and the Underwater datasets. The training setups are kept the same as the training on 200 images. We conduct experiments on ODGEN, ReCo, and GeoDiffusion. ReCo and GeoDiffusion COCO-2014 [37] is used to evaluate our method in general domains. As the scenes in this dataset are almost covered by the pre-training data of Stable Diffusion, we skip the domain-specific fine-tuning stage and train the diffusion model together with the object-wise conditioning modules. We train our ODGEN on the COCO training set with batch size 32 for 60 epochs on ×8 V100 GPUs as well as GLIGEN, ControlNet, and GeoDiffusion. Officially released checkpoints of ReCo (trained for 100 epochs on COCO), MIGC (trained for 300 epochs on COCO), and InstanceDiffusion (trained on self-constructed datasets) are employed for comparison. γ in Eq. ( 2) is set as 10 for our ODGEN.We use the annotations of the COCO validation set as conditions to generate 41k images. The FID scores in Tab. 4 are computed against the COCO validation set. ODGEN achieves better FID results than the other methods. We provide a typical visualized example in Fig. 5 . Given overlapping bounding boxes of multiple categories, ODGEN synthesizes all the objects of correct categories and accurate positions, outperforming the other methods. More samples are added in Appendix G. We observe that YOLO models trained on synthetic data only fall behind models trained on real data.We add experiments with different training and validation data combinations in Tab. 6. YOLO models trained on real images show better generalization ability and achieve close results when tested on real and synthetic data. YOLO models trained on synthetic data only get significantly better results when tested on synthetic data than when generalized to real data. It indicates that noticeable domain gaps still exist between real and synthetic data, which may be limited by the generation quality of modern Stable Diffusion models and are promising to be narrowed with future models. Image list and text list in object-wise conditioning modules. ODGEN enables object-wise conditioning with an image list and a text list as shown in Fig. 2 . We test the performance of From the results in Tab. 7, it can be found that using image lists or text lists brings moderate improvement on MRI images and significant improvement on the more complex Road Traffic dataset. Visualization in Fig. 6 shows that the image list improves the fidelity under occlusion, and the text list mitigates the mutual interference of multiple objects. Foreground region enhancement. We introduce a reweighting term controlled by γ for foreground objects in Eq. (2). Tab. 8 shows that appropriate γ values can improve both the fidelity and the trainability since the foreground details are better generated. However, increasing the value will undermine the background quality, as visualized in Fig. 6 .Corrupted label filtering. Results in Tab. 9 show that the corrupted label filtering step helps improve the mAP@.50:95 of ODGEN by around 1-2%. The results of ODGEN without the corrupted label filtering still significantly outperform the baseline setting in Tab. 2. 
conclusion: 
 This paper introduces ODGEN, a novel approach aimed at generating domain-specific object detection datasets to enhance the performance of detection models. We propose to fine-tune the diffusion model on both cropped foreground objects and entire images. We design a mechanism to control the foreground objects with object-wise synthesized visual prompts and textual descriptions. Our work significantly enhances the performance of diffusion models in synthesizing complex scenes characterized by multiple categories of objects and bounding box occlusions. Extensive experiments demonstrate the superiority of ODGEN in terms of fidelity and trainability across both specific and general domains. We believe this work has taken an essential step toward more robust image synthesis and highlights the potential of benefiting the object detection task with synthetic data. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: We have not designed safeguards yet but our model can share the same safeguards with modern large-scale text-to-image models like Stable Diffusion.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We respect the license and terms of all the assets used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Despite the significant improvement achieved by this work, we find it remains challenging to achieve comparable training performance as real data with merely synthetic data. We consider the limitation to be mainly caused by two reasons. Firstly, the generation fidelity is still limited by modern diffusion models. For instance, VAEs in Stable Diffusion are not good enough to deal with complex text in images, constraining the performance on video game datasets. ODGEN is promising to achieve further improvements with more powerful diffusion models in the future. Secondly, this paper mainly focuses on designing generative models with given conditions. However, the other parts in the dataset synthesis pipeline are not fully optimized, which are interesting to be explored in future work.We introduce a novel approach for object detection datasets synthesis, achieving improvement in fidelity and trainability for both specific and general domains. Our approach is more prone to biases caused by training data than typical AI generative models since it shows robustness with limited data of specific domains. Therefore, we recommend practitioners to apply abundant caution when dealing with sensitive applications to avoid problems of races, skin tones, or gender identities.Detector training w/o real data. In the main paper, we train the detectors with 200 real images and 5000 synthetic images for RF7 benchmarks. In this ablation study, we conduct experiments with the 5000 synthetic images only to explore the impact of the absence of real data for training detectors. As shown in Tab. 10, for easy datasets with fewer objects, like MRI image and Cotton, training with 5000 synthetic data only achieves better results than training with 200 real samples only. Increasing the dataset size can improve the model performance because the generation quality is good enough in these cases. Nevertheless, for complex scenes like Robomaster and Aquarium, training with synthetic data only fails to achieve better results. It may be caused by the limited generation quality of modern diffusion models. For instance, the VAE in Stable Diffusion is not good at synthesizing texts in images, which is common in the Robomaster dataset. Besides, FID results in Tab. 1 indicate that there still exist gaps between the distributions of real and synthetic images. The performance of our approach is promising to be further improved in the future with more powerful generative diffusion models. In addition, the current dataset synthesis pipeline is not fully optimized and cannot reproduce the distributions of foreground objects perfectly, which may also lead to worse performance of training on synthetic data only with the RF7 datasets. Number of synthesized samples. We ablate the number of synthesized samples used for detector training and provide quantitative results in Tab. 11. With increasing amounts of synthesized samples, detectors achieve higher performance and get very close results between 5000 and 10000 synthetic samples.Category isolation for datasets synthesis. As illustrated in the main paper, modern diffusion models have the limitation of "concept bleeding" when multiple categories are involved in the generation of one image. An alternative method could be only generating objects of the same category in one image. We select 3 multi-class datasets and compare this approach against the proposed method in ODGEN. As shown in Tab. 12, ODGEN achieves better results on most benchmarks than generating samples with isolated categories. This is an expected result since generating objects of various categories in one image should be better aligned with the test set. Corrupted label filtering. We design this step to filter some labels when objects are not generated successfully, which may be caused by some unreasonable boxes obtained with the pipeline in Fig. 3 . For experiments on COCO, this step is not applied to any method since we directly use labels from the COCO validation set and hope to synthesize images consistent with the real-world labels.For RF7 experiments, this step is only applied to our ODGEN in Tab. 2. For fair comparison, we skip this step on RF7 to compare the generation capability of different methods fairly and provide results in Tab. 13 . It shows that the corrupted label filtering step only contributes a small part of the improvement. Without this step, our method still outperforms the other methods significantly. In addition, as illustrated in Appendix A, the current dataset synthesis pipeline is designed to compare different methods and can be improved further in future work.Foreground region enhancement on COCO. We set γ as 10 in Eq. ( 2) in our experiments trained on COCO. We add ablations with γ value of 0 to ablate foreground region enhancement on COCO and provide quantitative results in Tab. 14. It shows that the foreground region enhancement helps ODGEN achieve better results, especially the layout-image consistency reflected by mAP results. Fine-tuning on both cropped foreground objects and entire images. For specific domains, taking the Robomaster dataset in RF7 as an example, most images in the dataset contain multiple categories of objects like "armor", "base", "watcher", and "car", which are either strange for Stable Diffusion or different from what Stable Diffusion tends to generate with the same text prompts. For models fine-tuned on entire images only, they cannot obtain direct guidance on which parts in entire images correspond to these objects. As a result, the fine-tuned models cannot synthesize correct images for foreground objects given text prompts like "a base in a screen shot of the Robomaster game". We provide FID results of foreground objects synthesized by models fine-tuned on both entire images and cropped foreground objects (text prompts are composed of object names and the scene name, All the methods included for comparison in our experiments share the same textual descriptions of the foreground objects and entire scenes in the global text prompts for a fair comparison. ReCo [61] and GeoDiffusion [5] further add the location information of foreground objects to their global text prompts following their approaches. For general domains, we concatenate the object class names into a sentence as the global text prompt. For specific domains, we concatenate the object class names and the scene name into a sentence as the global text prompt.We follow the default training setting used by ControlNet [62] , including the learning rates and the optimizer to train our ODGEN models and provide fair comparison with ControlNet.Offline foreground objects synthesis. To improve the training efficiency, we synthesize 500 foreground object images for each category offline and build a synthetic foreground pool. During the training of object-wise conditioning modules, foreground images are randomly sampled from the pool to build the image list.Image encoder. ControlNet uses a 4-layer convolutional encoder to encode input conditions sized 512 to 64 × 64 latents. The input channels are usually set as 1 or 3 for input images. Our ODGEN follows similar methods to convert the input conditions. The key difference is that we use image lists of synthesized objects as input conditions. The image lists are padded to N and concatenated in the channel dimension for encoding. In this case, we expand the input channels of the encoder to 3N and update the channels of the following layers to comparable numbers. We provide the channels of each layer in Tab. 17. The convolutional kernel size is set as 3 for all layers. We design different encoder architectures according to the maximum object numbers that can be found in a single image. For datasets like MRI in which most images contain only one object, we can use a smaller encoder to make the model more lightweight. For a generic model trained on large-scale datasets, we can fix the architectures of encoders with a large N to make it applicable to images containing many objects. RF7 datasets. We employ the RF7 datasets from Roboflow-100 [6] to evaluate the performance of our ODGEN on specific domains. Here, we add more details of the employed datasets, including Apex Game, Robomaster, MRI Image, Cotton, Road Traffic, Aquarium, and Underwater. We provide the number of images for each dataset, including the train, validation, and test sets in Tab. 18. We use the first 200 samples in the image list annotation provided by Roboflow-100 for each dataset to build the training set for our experiments. The full validation set is used as the standard to choose the best checkpoint from YOLO models trained for 100 epochs. The full test set is used to evaluate the chosen checkpoint and provide mAP results shown in this paper. Visualized examples with annotations are shown in Fig. 9 . The object categories of each dataset are provided in Tab. 19 . Some images in the Apex Game and Robomaster datasets contain complex and tiny text that are hard to generate by current Stable Diffusion.Copy-paste [14, 65] method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets used in this paper. Besides, our approach also serves as a controllable image generation method that only needs bounding box labels to produce realistic images, which is hard to achieve by copy-paste. InstaGEN [11] is mainly designed for the open-vocabulary object detection task. It generates images randomly and annotates them with a grounding head trained with another pre-trained detector. Earlier works like LayoutDiffusion [66] are not implemented with latent diffusion models and thus are not included for comparison. Works [23, 39, 56] are designed for semantic segmentation, while our ODGEN is designed for object detection. G Supplemental Experiments mAP@.50 on RF7. Tab. 20 provides quantitative results for trainability in terms of mAP@.50 as supplements to Tab. 2. Our approach achieves improvement over training on real data only and outperforms other methods.Visual relationship between image lists and synthesized images. The image list is designed to provide information on category and localization for controlling foreground object layouts. The synthesized images share the same category with objects in generated images and are pasted to During the inference process, we compare using the same offline image library as training against using a totally different image library. We get very close results, as shown in Tab. 22. Both outperform other methods significantly, as shown in Tab. 4. It indicates that ODGEN can extract category information from synthesized samples of foreground objects in image lists instead of depending on certain images of foreground objects. ODGEN is not constrained by the image library of foreground objects used in training and can be generalized to other newly generated offline image libraries consisting of novel synthesized samples of foreground objects, which ensures that offline image libraries increase ODGEN's computational efficiency without reducing its practicality. Quantitative evaluation for concept bleeding. This work mainly focuses on the concept of the bleeding problem of different categories of objects. The mAP results of YOLO models trained on synthetic images only in Tab. 4 can be a proxy task to prove that the concept bleeding is alleviated since our ODGEN achieves state-of-the-art performance in the layout-image consistency of complex scene generation conditioned on bounding boxes. This section adds quantitative evaluation with BLIP-VQA [21] , which employs the BLIP [32] model to identify whether the contents in synthesized images are consistent with text prompts. The results are provided in Tab. 23. Our ODGEN outperforms other methods and gets results close to ground truth (real images from the COCO validation set sharing the same labels with synthetic images).ODGEN trained on COCO with Stable Diffusion v1.5. Our ODGEN is implemented with Stable Diffusion v2.1 in Sec. 4. We add experiments of implementing ODGEN with Stable Diffusion v1.5 Visualization. We provide extensive visualized examples of ODGEN to demonstrate its effectiveness. In Fig. 13 , we provide visualized samples produced by models fine-tuned on both cropped foreground objects and entire images. It shows that the fine-tuned models are capable of generating diverse foreground objects and complete scenes. In Fig. 14 , we show generated samples of the specific domains in RF7. It can be seen that ODGEN is robust to complex scenes composed of multiple categories, dense and overlapping objects. Besides, ODGEN shows strong generalization capability to various domains. Answer: [No] Justification: This paper do not include new data. The code is not included yet. We have provided enough details for implementation to ensure reproducibility. We will release the code and models soon. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper receives all positive ratings, where all the reviewers agree that the rebuttal and replies have addressed the major concerns (two reviewers upgraded the rating). Overall, the paper proposes a data generation pipeline via the diffusion model for domain-specific object detection. Comprehensive experiments are provided to validate the effectiveness of the proposed framework. The AC carefully checked the paper, reviews, and rebuttal, and agree with the reviewers about the positive rating. Hence the AC recommends the acceptance decision. It would be highly recommended that the authors include the suggested feedback in the final version (especially the experiments with more real and synthetic data to showcase a broader usage) and release the code to foster the progress in this community.

Reviews: review: paper_summary: paper_summary: It proposes a novel data synthesis pipeline, ODGEN, which uses diffusion models to generate high-quality and controllable datasets for object detection.
They first fine-tune the pre-trained diffusion models on both cropped foreground objects and entire images. Next, they control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions.

For each image, the object number per category, bounding box location and size are sampled from estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts which are fed to ODGEN to generate the image.
strength_weakness: strength_weakness: As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.

For the data scarcity experiment, authors only sample 200 images as the training set for all datasets, which is interesting but not sufficient for the analysis. More extensive evaluation can be done with higher ratios of real:synthetic data for datasets like ApexGame, Robomaster, and Underwater (and even larger datasets like COCO) .

Restricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.

ODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.

There are no comparisons provided with the fully supervised approach (models trained on complete real training set) on representative datasets. This would help evaluate the trainability of the proposed approach over data augmentation techniques such as copy-paste.

The computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.
questions: questions: As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.

Restricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.

ODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.

The computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes the ODGEN method to generate high-quality images conditioned on bounding boxes. ODGEN fine-tunes Stable Diffusion (SD) on domain-specific datasets to enhance image quality in specialist domains, designing a novel strategy to control SD with object-wise text prompts and synthetic visual conditions to alleviate 'concept bleeding'. Experimental results on the COCO-2014 dataset demonstrate that ODGEN surpasses other methods in control capability. Additionally, the authors design a dataset synthesis pipeline using ODGEN, showing that using additional generated training data can improve the performance of state-of-the-art (SOTA) object detectors.
strength_weakness: strength_weakness: There are some unclear implementation details, and a few experiments are missing. Please refer the Questions.
questions: questions: - When fine-tuning the pre-trained diffusion model, the authors use not only the entire images from the dataset but also the crops of foreground objects (i.e., resized to 512 x 512). I'm curious if this approach would help in generating particularly small objects.

- During actual inference, what is the visual relationship between the image list $c_{il}$ (i.e., in Eqn. 2) and the final generated image? The authors should provide some visual examples to illustrate this.

- The authors mention that the object-wise conditioning used in the paper can alleviate "concept bleeding" in multi-instance generation. I believe it is necessary to conduct experiments on the COCO-MIG benchmark [65], which assesses the model's ability to control both positioning and attributes, to elaborate on the attribute binding capability of the ODGEN.

    [65] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR'24

- Compared to GLIGEN and MIGC, ODGEN achieves better results in overlapping cases. Which module's design primarily contributes to this improvement? As this is a major issue in the industry, I hope the authors can emphasize this in the methods section.

- Have the authors considered open-sourcing the code and model? I believe this would greatly benefit the community.
limitations: limitations: - The proposed method requires fine-tuning the entire Stable Diffusion model and training an additional ControlNet. This often requires significant computational resources, making reproduction challenging. 
- I hope the authors can open-source the corresponding model and code, as this would greatly benefit the community.
paper_summary: paper_summary: ODGEN uses a diffusion-based generation model to create novel images to train object detectors. Object bounding boxes along with the object's textual description are given as a conditioning for the generation step. With these generated images they can improve the detector's performance.
strength_weakness: strength_weakness: 1. There is a high generation cost when there are multiple objects in the same image, which reduces the practicality of the proposed approach. 
2. It would be interesting to see if the generated images improve the detector's performance where it originally failed due to lack of data, such as when objects are partially occluded.  
3. How are the statistics of the object layout taken? 
4. Why only 5k images were generated? Can we see some study of gradually adding synthetic data, like 1k,5k,10k,20k, and some plateauing of the detector performance?
4. It will be only fair to compare with other generation approaches with the same post-generation filtering being applied to them. How does ODGEN compare then?
4. In line 111, it is misleading to say "a new method to fine-tune the diffusion model", as it is nothing more than just fine-tuning.
questions: questions: Please refer to the weakness section.
limitations: limitations: Please refer to the weakness section.
paper_summary: paper_summary: This work is about generating synthetic (annotated) data for object detection with diffusion models that are tuned for a specific image domain. Generating synthetic annotated data can be useful for model training in situations where data is scarce. The authors propose a specific pipeline and new modules to generate such data. First, an off-the-shelf diffusion model is fine-tuned on a specific domain, for instance a specifc object detection dataset. This is done both with the full images and crops of foreground objects (via the given bounding boxes). The statistics of object categories, number of instances, bounding boxes are estimated. Then, a new layout is sampled from the distribution, the class-specific objects are individually synethesized with the tuned diffusion model. These are individually placed at their sampled bounding box location to build a so-called image list. Then, a ControlNet model is used that encodes this image list along with a text list of the object categories, which outputs the final synthesized image. The newly designed text and image encoders for the controlnet are trained. Finally, a verification step is done at the object level (classifying if the generated image contains the desired object inside a certain bounding box). The synthesized data is evaluated with the FID metric, and also used to train object detectors, which are then evaluated with mAP.
strength_weakness: strength_weakness: - Positioning with respect to some prior work.
    - In the related work section, the paragraphs on layout-to-image generation & dataset synthesis contain some highly related works (like InstanceDiffusion or others listed as the second group of data synthesis approaches). However, these works are only briefly described but not contrasted against the proposed method. So, what's the difference to InstanceDiffusion, for example?
    - Why are there no comparisons to InstanceDiffusion in the experiments, or even simpler data augmentation techniques like "copy-paste"?
- The object distributions are simplified compared to real distributions.
    - Using a Gaussian distribution for a discrete variable seems odd. There will be some probability to sample a negative number and if you clip, you ultimately end up with a different distribution than what was estimated, I guess.
    - It seems that all estimated distributions regarding the bounding boxes are independent. I assume that location and areas are dependent. Are the distributions modeled independently on purpose? As in, is it better for simulation to have independent distributions? Or is this just done for simplicity? And was the impact of this evaluated empirically?
- Missing details on the usefulness of synthetic data
    - For the domain specific domain, it seems that the diffusion models were also tuned only with the 200 selected domain-specific images (according to line 212). I found this information to be crucial in judging whether or not the experiments are valid. I suggest to highlight this aspect already in Section 4.1 - and maybe also discuss why this information is curcial. My thinking here is that if you used more data for tuning the diffusion model than you use to train the detectors, it would be unclear where performance gains come from. It could just come from the additional data used to train the diffusion model, rather than the images being synthetic.
    - Also, does the same conclusion on object detector performance gains from Table 2 also hold when scaling up real and synthetic images? From a practical point of view, annotating 800 more images with bounding boxes would likely be affordable in most cases. So, I'm missing an experiment like in Table 2, but with more images for both real and synthetic images, e.g., 1000 and 25000 real and synthetic images.
    - For the general domain, it's great to see in Table 3 that the detector is better when trained on ODGEN data compared to any other synthetic data. However, the real reference points should be (a) the 10k real images and (b) the combination of 10k real and K synthetic images.
questions: questions: - When you fine-tune the diffusion model with driving data, don't you have issues with limited diversity? I understand that you might only be interested in the same label spaces as defined in the dataset. But for real applications, one is often interested in simulating something that is missing in a dataset. For instance, a driving dataset may contain lots of regular cars, vans and trucks. But you also want to generate emergency vehicles like police cars or ambulances.
- Does the filtering rate of the "corrupted label filtering" step correlate with object occlusions? As in, are there more issues (during either the generation or even the classification/filtering) for occluded/overlapping objects?
- An interesting application of this paper could be to improve language-based object detectors like in [A]. This work relies on GLIGEN and could benefit from ODGEN. The difference would be that the general semantic knowledge of the diffusion model would be leveraged.

References:
- [A] Generating Enhanced Negatives for Training Language-Based Object Detectors. Zhao et al. CVPR'24
limitations: limitations: The limitations are adequately discussed in the appendix.
paper_summary: paper_summary: This paper tackles the task of controllable image generation from bounding boxes and text prompts for training object detectors. To fine-tune diffusion models to specific domains effectively and deal with the challenge of concept bleeding, this paper proposes a method called ODGEN. In the proposed method, the diffusion model is first fine-tuned on both entire images and cropped foreground regions. Then, text lists and image lists are utilized as input for ControlNet to avoid the concept bleeding and generate high-quality images. The experimental results show that the proposed method outperforms the state-of-the-art methods for controllable image generation on seven types of datasets in terms of FID score. In addition, the object detectors trained on the generated images from the proposed method achieve better performance than other methods.
strength_weakness: strength_weakness: My concerns are mainly about the lack of detailed analysis:

I) This paper claims that fine-tuning with both cropped foreground regions and entire images is one of its contributions. However, an ablation study to confirm the effectiveness of this approach is missing.

ii) From Fig. 4, it looks like the corrupted label filtering is performed only for the proposed method (ODGEN). Because the corrupted label filtering is a simple technique and can be applied to other methods as well, the comparisons between ODGEN and other methods not only without filtering but also with filtering should be provided for fair comparisons.

iii) This paper provides only the results with 200 real images. I'm curious to know how the performance of the proposed method and other methods change when more number of real images are available for training.

iv) In Sec. 1, the concept bleeding is raised as one of the challenges in this task. Although Fig. 6 provides an example, I would like to see a quantitative evaluation to check whether it was effectively addressed by the proposed method.
questions: questions: See iii) in Weakness.
How is the performance of the proposed method and other methods when more real images are available?
limitations: limitations: The limitations are adequately discussed in Sec. A.

Comments: title: Supplemental: synthesizing overlapping objects
comment: Dear Reviewer FcFR,

Thank you for your positive feedback and efforts in reviewing this paper again!

Authors
Thanks again for the response. Because my remaining concern has been addressed, I'm happy to raise my rating.
Dear Reviewer FcFR,

Thanks for your positive feedback. We are glad that most of your concerns have been well addressed. We add a detailed explanation for your remaining concern. 

1. $\textbf{Specific Domains}$: 

Taking the Robomaster dataset in RF7 as an example, most images in the dataset contain multiple categories of objects like "armor", "base", "watcher" and "car",  which are either strange for Stable Diffusion or different from what Stable Diffusion tends to generate with the same text prompts. For the model fine-tuned on entire images only, it cannot obtain guidance on which parts in entire images correspond to these objects. As a result, the fine-tuned models cannot synthesize correct images for foreground objects given text prompts like "a base in a screen shot of the robomaster game". We agree with you that "adding experimental results with the models fine-tuned on entire images only" can make our statement more convincing. We provide FID results of foreground objects synthesized by models fine-tuned on both entire images and cropped foreground objects (text prompts are composed of the names of objects in the image and the scene name, e.g., "a car and a base in a screen shot of the robomaster game"), and models fine-tuned on entire images only, in the table below for comparison:

Table: FID ($\downarrow$) results of foreground objects in the Robomaster dataset synthesized by models fine-tuned on different data. 
Object categories | watcher | armor | car | base | rune 
-------|--------|-------|-------|--------|-------
Models fine-tuned on entire images only | 384.94 | 532.56 | 364.46 | 325.79 | 340.11
Models fine-tuned on both cropped objects and entire images | $\textbf{124.45}$ | $\textbf{136.27}$ | $\textbf{135.66}$ | $\textbf{146.50}$ | $\textbf{128.95}$

It shows that our approach helps fine-tuned models generate images of foreground objects better. Similarly, we also provide results on the Road Traffic dataset, which is relatively more familiar for Stable Diffusion than the Robomaster dataset.

Table: FID ($\downarrow$) results of foreground objects in the Road Traffic dataset synthesized by models fine-tuned on different data. 
Object categories | vehicle | traffic light | motorcycle | fire hydrant | crosswalk | bus | bicycle 
-------|--------|-------|-------|--------|-------|-------|-------
Models fine-tuned on entire images only | 241.64 | 240.55 | 213.90 | 205.22 | 253.40 | 238.63 |153.76
Models fine-tuned on both cropped objects and entire images | $\textbf{105.27}$ | $\textbf{90.71}$ | $\textbf{143.30}$ | $\textbf{53.05}$ | $\textbf{97.33}$ | $\textbf{118.58}$ | $\textbf{65.61}$ 

Results also show improvement of our approach compared with models fine-tuned on entire images only. Since links are not allowed in the responses, we cannot provide supplemental visualized results here but will add them to the revised manuscript. 

2. $\textbf{General Domains}$: 

The fine-tuning step is designed for specific datasets. As for datasets of general domains like COCO, Stable Diffusion can generate the same categories of objects as the objects in the COCO dataset without fine-tuning. Therefore, we skip the fine-tuning step and directly use Stable Diffusion to synthesize images of foreground objects and use them to build image lists. The results in our paper show superior improvement achieved by our ODGEN compared with prior works.
﻿
Thanks for your valuable suggestions and efforts in reviewing this paper again!
﻿
Authors
I would like to thank the authors for their careful rebuttal. My concerns ii)-iv) have been well addressed. Regarding i), I'm still not convinced. I think the claim "the models only fine-tuned on entire images have difficulties in generating foreground objects required by image lists since they cannot bind the prompts to corresponding objects in entire images that may contain multiple categories of objects." may be true. However, the evidence to support this claim (experimental results with the models fine-tuned on entire images only) should be provided.

Therefore, I keep my rating as Borderline Accept although I'm positive about accepting this paper.
Dear Reviewer FcFR,

We are writing to kindly remind you that the time left for discussion is no more than 12 hours. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments again.

Authors.
Dear Reviewer Fyyh,

We are writing to kindly remind you that the time left for discussion is no more than 12 hours. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments again.

Authors.
Dear Reviewer N8et,

We are glad that all your concerns have been solved. We appreciate your efforts in reviewing our paper and providing feedback during the discussion period.

Authors
Thanks for your efforts in rebuttal. The response has addressed my concerns.
After reading other reviewers' comments and the response, I would like to raise my rating.
Dear Reviewer PYXX,

We are glad that all your concerns have been solved. We appreciate your efforts in reviewing our paper and providing feedback during the discussion period.

Authors
Thanks for the rebuttal; After reading them I don't have any further questions.
Dear Reviewer PYXX,

We are writing to kindly remind you that the time left for discussion is only about one day. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments.

Authors.
Dear Reviewer N8et,

We are writing to kindly remind you that the time left for discussion is only about one day. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments.

Authors.
Dear Reviewer 2vbs,

We are glad that all your concerns have been solved. We appreciate your efforts in reviewing our paper and providing feedback during the discussion period.

Authors
I confirm that I read the rebuttal. I appreciate the author's detailed response. I have no further questions to the authors at this point.
Compared with other cases, cases with overlapping objects are more complex and challenging to synthesize. As shown by the visualized samples provided in Fig.5, 10, 11, and 12 in our paper, our approach has made great progress in generating overlapping objects and outperforms prior works on layout-image consistency.