ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models
Authors: JingYuan Zhu
Shiyu Li
Yuxuan Liu
Jian Yuan
Ping Huang
Jiulong Shan
Huimin Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Object Detection Dataset Generation
Complex Scene Synthesis
Domain-Specific
Diffusion Models
Primary area: diffusion_based_models
TL;DR: We propose a novel method to control diffusion models with bounding box labels, exhibiting robustness in handling complex scenes and specific domains and enabling detector enhancement with synthetic data.

Abstract: Modern diffusion-based image generative models have made significant progress and become promising to enrich training data for the object detection task. However, the generation quality and the controllability for complex scenes containing multi-class objects and dense objects with occlusions remain limited. This paper presents ODGEN, a novel method to generate high-quality images conditioned on bounding boxes, thereby facilitating data synthesis for object detection. Given a domain-specific object detection dataset, we first fine-tune a pre-trained diffusion model on both cropped foreground objects and entire images to fit target distributions. Then we propose to control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. ODGEN exhibits robustness in handling complex scenes and specific domains. Further, we design a dataset synthesis pipeline to evaluate ODGEN on 7 domain-specific benchmarks to demonstrate its effectiveness. Adding training data generated by ODGEN improves up to 25.3% mAP@.50:.95 with object detectors like YOLOv5 and YOLOv7, outperforming prior controllable generative methods. In addition, we design an evaluation protocol based on COCO-2014 to validate ODGEN in general domains and observe an advantage up to 5.6% in mAP@.50:.95 against existing methods.

ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models
Jingyuan Zhu'
Shiyu Li
Yuxuan Liu?
Jian Yuan'
Ping Huang²
Jiulong Shan
Huimin Ma
Tsinghua University, China
Apple
UUniversity of Science and Technology Beijing
oehinas
nnttrryyde
callep
Wallop belitbaria
sturish
matorcycles traffii
cresswelks
scalleps and holathurion
traffic
Miject
pongsin
negetive image'
object
ponguins
Apex game
photo
in photo of aquerium
Figure 1: The proposed ODGEN enables controllable image generation from bounding boxes and text prompts. It can generate high-quality data for complex scenes, encompassing multiple categories, dense objects, and occlusions, which can be used enrich the training data for object detection.
Abstract
Modern diffusion based image generative models have made significant progress and become promising to enrich training data for the object detection task. However, the generation quality and the controllability for complex scenes containing multi- class objects and dense objects with occlusions remain limited. This paper presents ODGEN, novel method to generate high-quality images conditioned bounding boxes, thereby facilitating data synthesis for object detection. Given a domain- specific object detection dataset, first fine-tune pre-trained diffusion model on both cropped foreground objects and entire images to target distributions. Then propose to control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. ODGEN exhibits robustness in handling complex scenes and specific domains. Further, design a dataset synthesis pipeline to evaluate ODGEN on 7 domain-specific benchmarks to demonstrate its effectiveness. Adding training data generated by ODGEN improves up to 25.3% mAP@ A.A..50:95 .5:.95 with object detectors like YOLOv5 and YOLOv7, outperforming prior controllable generative methods. In addition, design an evaluation protocol based on COCO-2014 4 to validate ODGEN in general domains and observe an advantage up to 5.6% in mAP@ 550:95 against existing methods.
Introduction
Object detection is of the most widely used computer vision tasks in real-world applications. However, data scarcity poses a significant challenge to the performance of Stae-oofthe--ae models
Ma the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

With the progress of generative diffusion models, such as which can generate high-quality images from text prompts, recent works have started to explore the potential of synthesizing images for perceptual model training. Furthermore, methods like GLIGEN 33], ReCo [6]], and ControlNet 62] provide various ways to control the contents of synthetic images. Concretely, extra visual or textual conditions are introduced as guidance for diffusion models to generate objects under certain spatial constraints. Therefore, becomes feasible to generate images together with instance-level or pixel-level annotations.
like YOLOv5 25 and YOLOv7 154
DALL-E [3 and Stable Diffision 44
Nevertheless, it remains challenging to generate an effective supplementary training set for object detection applications. Firstly, large-scale pre- pre-trained diffusion models are usually trained on web crawl datasets such as LAION [48]. whose distributions may be quite distinct from specialist domains. The domain gap could undermine the fidelity of generated images, especially the detailed textures and styles of the objects, e-g., the tumor medical images or the avatar in video game. Secondly, the text prompts for object detection scenes may contain multiple categories of subjects. could cause the "concept bleeding" [12 40 68 problem for modern diffusion models, which is defined as the unintended merging overlapping of distinct visual elements in image. As result, the synthetic images can be inconsistent with pseudo labels used as generative conditions. Thirdly, overlapping objects, which are common in object detection datasets, are likely to be merged one single object by existing controllable generative methods. Lastly, some objects may be neglected by the diffusion model so that no foreground is generated at the conditioned location. All of the limitations can hinder the downstream object detection performance.
To address these challenges, we propose ODGEN, a novel method to generate synthetic images with a fine-tuned diffusion model and object-wise conditioning modules that control the categories and locations of generated objects. We first fine-tune the diffusion model on given domain-specine dataset, with not only entire images but also cropped foreground patches, to improve the synthesis quality of both background scene and foreground objects. Secondly, we design new text prompt embedding process. We propose to encode the class name of each object separately by the frozen text encoder, to avoid mutual interference between different concepts. Then we stack the embeddings and encode them with a newly introduced trainable text embedding encoder. Thirdly, propose to synthetic patches of foreground objects the visual condition. Each patch is resized and pasted on an empty canvas according to bounding box annotations, which deliver both conceptual and spatial information without interference from overlap between objects. In addition, we train foreground/hackgronn
CLIP 41
discriminators to check whether the region of each pseudo label contains synthetic object. If no object can be found, the pseudo label will be filtered.
We evaluate the effectiveness of our ODGEN in specific domains on 7 subsets of the Roboflow-100 benchmark (6). Extensive experimental results show that adding our synthetic data improves up to 25.3% mAP@ 50:995 on YOLO detectors, outperforming prior controllable generative methods. Furthermore, validate ODGEN in general domains with evaluation protocol designed based on COCO-2014 (37) and gain an advantage up to 5.6% in mAP@ 50:995 than prior methods.
The main contributions of this work can be summarized follows:
We propose to fine-tune pre-trained diffusion models with both cropped foreground patches and entire images to generate high-quality domain-specifi target objects and background scenes.
We design novel strategy to control diffusion models with object-wise text prompts and synthetic visual conditions, improving their capability of generating and controlling complex scenes.
We conduct extensive experiments to demonstrate that our synthetic data effectively improves the performance of object detectors. Our method outperforms prior works on fidelity and trainability in both specific and general domains.
2 Related Works
Diffusion models 20 29
49 50 have made great progress in recent years and become mainstream for image generation. showing higher fidelity and training stability than prior generative models like GANs 4 16
38,
26
27
28 and VAËs 30 52 Latent diffusion models (LDMss) [44
perform the denoising process in the compressed latent space and achieve flexible generators condi- tioned on inputs like text and semantic maps. I.MSs significantly improve computational efficiency and enable training on internet-ccale datasets [88 Modern large-scale text-to- --igaaa diffusion models

---

including I (2), DALL-E 42 Imagen [46. and Stable Diffusion 40 44 have demonstrated unparalleled capabiliities to produce diverse samples with high fidelity given unfettered text prompts.
Layout-to-image generation synthesizes images conditioned on graphical inputs of layouts. Pi-
based on GANs [66 and transformers [53] successfully generate images consistent with given layouts. LayoutDiffusion [66] fuses layout and category embeddings and builds object-aware cross-attention for local conditioning with traditional diffusion models. LiyoutDif- fuse (24] employs layout attention modules for bounding boxes based on LDMs. MultiDiffussin B
oneering works 22 35
and Biff [58] develop training-free free frameworks produce samples with spatial constraints.
GLIGEN 33 inserts trainable gated self-attention layers to pre-trained LDMs to fit specific tasks and
hard to generalize to scenes uncommon for pre-trained models. ReCo 61] and GeoDiffusion [5] extend LDMs with position tokens added to text prompts and fine-tunes both text encoders and diffusion models to realize region-aware controls. They need abundant data to build the capability of encoding the layout information in text prompts. ControlNet [62] reuses the encoders of L.MMs as backbones to learn diverse controls. However, still struggles to deal with some complex cases. MIGC 667 decomposes multi-instance synthesis to single-instance subtasks utilizing additional attention modules. InstanceDiffusion [55] adds precise instance-level controls, including boxes, masks, and scribbles for text-to-image generation. Given annotations of bounding boxes, this paper introduces a novel approach applicable to both specific and general domains to synthesize sigh-fidelity complex scenes consistent with annotations for the object detection task.
Dataset synthesis for training object detection models. Copy-paste ( is simple but effective data augmentation method for detectors. Simple Copy-Paste [14] achieves improvements with a simple random placement strategy for objects. Following works [ Gs generate foreground objects and then paste them on background images. However, generated images by these methods usually have obvious artifacts on the boundary of pasted regions. DatasetGAN [64] takes an early step to generate labeled images automatically. Diffusion models have been used to synthesize training data and benefit downstream tasks including object detection
5
10
63
image classification I [
3
51]
and semantic segmentation [15 23
31
36
39
56
57
59
60) Modern approaches for
pairs generation can be roughly divided into two categories. One group of works [[] [5|
B1 36
56 [59 663 first generate images and then apply pre-trained perception model to generate pseudo labels on the synthetic images. The other group [5] [3] 39] uses the same strategy as our approach to synthesize images under the guidance of annotations as input conditions. The second group also
overlaps with layout-to-image approaches 244 33 55 61 62 66, 67). Our approach should be
assigned to the second group and is designed to address challenging cases like multi-class objects, occlusions between objects, and specific domains.
Method
This section presents ODGEN, a novel approach to generate high-quality images conditioned on bounding box labels for specific domains. Firstly, we propose new method to fine-tune the diffusion model in Sec. 3.1 Secondly, we design an object-wise conditioning method in Sec. 3.2 Finally, we introduce generation pipeline to build synthetic datasets for detector training enhancement
in Sec. 3.3
3.1 Domain-specific Diffusion Model Fine-tuning
Modern Stable Diffusion 44 models trained on the LAION-SB 48 dataset. Therefore, the textual prompts and generated images usually follow similar distributions to the web crawl data. However, real-world object detection datasets could come from very specific domains. We fine-tune the UNet of the Stable Diffusion to fit the distribution of an arbitrary object detection dataset. For training images, we use not only the entire images from the dataset but also the crops of foreground objects. In particular, we crop foreground objects and resize them to 512 x 512. In terms of the text input, as shown in Fig. (a), we use templated condition "a <scene>" named c, for entire images and "a <classname>* named for cropped objects. If the names are terminologies, following the subject-driven approach in DreamBooth [45), we can employ identifiers like "[V]" to represent the target objects and scenes, improving the robustness to textual ambiguity under the domain-specine context. o and & represent the input of scenes x, and objects I; added to noises at time step l. 6, and e, represent the added noises following normal Gaussian distributions. The pre-trained Stable
3

---

random noise
preficred noise
reconstruction
TEXT ('A motorcycle, A vehicle",
Dirlision Model
labels
Difiasion Model
traffic scene'
AA vehicle') ]
crop and resize
shared weights
bounding hos posiiios
motorcycle
Diclusion
Model
IMAGE
Symthesized objects
loss
moise
prediesed noise
reconstraction loss with foreground reweighted
Fine-tured
'A motorcycle and vehicles traffic
from
TEXT LIST
padding empty strings
CLIP
stack
Text Emhad Encoder
Synthesized sumple
Encuder
Trainable weights
from
IMAGE LIST
channel packing in channel
Encuder
Fruzen weights
Object-wise Conditioning Module
Figure 2: ODGEN training pipeline: (a) pre-trained diffusion model is fine-tuned detection dataset with both entire images and cropped foreground patches. (b) text list is built based on class labels. The fine-tuned diffusion model in stage (a) used generate a synthetic object image for each text. Generated object images are resized and pasted on empty canvases per box positions, constituting an image list. (c) The image list concatenated in the channel dimension and encoded as conditions for ControlNet. The text list is encoded by the CLIP text encoder, stacked, and encoded again by the text embedding encoder as inputs for ControlNet.
Diffusion parameterized by is fine-tuned with the reconstruction loss as:
Free -EpgLeesNN [lke, - 4, [[]ss - co(o, (c..)))
+
where À controls the relative weight for the reconstruction loss of scene images. 7 is the frozen CLIP text encoder. Our approach guides the fine-tuned model to capture more details of foreground objects and maintain its capability of synthesizing the complete scenes.
Object-wise Conditioning
ControlNet (62) can perform controllable synthesis with visual conditions. However, the control can be challenging with an increasing object number and category number due "concept bleeding" phenomenon. Stronger conditions are needed to ensure high-fidelity generation. To this end, we propose a novel object-wise conditioning strategy with ControlNet.
Text list encoding, As shown in Fig. (b) and (c), given a set of object class names and their bounding boxes, we first build Then the text list is padded with empty texts to fixed length N and converted to list of embeddings by pre-trained CLIP text tokenizer and encoder. The embeddings are stacked and encoded by l-yyr convolutional text embedding encoder, and used as the textual condition for ControlNet. The text encoding of native ControlNet compresses resulting in mutual interference between distinct categories. To alleviate this "concept bleeding" problem of multiple categories, our two-step text encoding enables the ControlNet to capture the information of each object with separate encoding.
list consisting of each object with the fixed template "a <classname>?
information in global text prompts altogether,

---

Object distribution estimation
Corrupted label filtering
Filtered Labels
Bounding number Bounding box area
Bounding box aspect ratio Bounding box location
Fore/Back-
ground Discriminator
Detection Datasets
Class-wise Statistics
Gaussian Distributions
Dataset Synthesis
sample
Cropped Objects
a <classname
<classname 1> and <classname 2> and <classname in a <scene> Text Prompi
classname
cclassname 2>
<classname 2>'
<classname 3>
classname
Text List
Fine-tuned Diffusion Model
Synthesized
classname
classname
ControlNet
Samples
Image List
Pscudo Labels
Pipeline for object detection dataset synthesis. Yellow block: estimate Gaussian distribu- tions for the bounding box number, area, aspect ratio, and location based on the training set. Blue block: sample pseudo labels from the Gaussian distributions and generate conditions including text and image lists to synthesize novel images. Pink block: train classifier with foreground and background patches randomly cropped from the training set and use it to filter pseudo labels that failed to be synthesized. Finally, the filtered labels and synthetic images compose datasets.
Figure 3:
Image list encoding. As shown in Fig. (b) and (c), for each item in the text list, use the fine-tuned diffusion model to generate images for each foreground object. We then resize each generated image and paste it an empty canvas based the corresponding bounding box. The set of pasted images is denoted as an image list, which contains both conceptual and spatial information of the objects. The image list concatenated and zero-padded to N in the channel dimension and then encoded to the size of latent space by an image encoder. Applying an image list rather than pasting all objects on single image can effectively avoid the influence of object occlusions.
Optimization. With a pair of image and object detection annotation, we generate the text list cu and image list ca introduced in this section. The input global text prompt c of the diffusion model is composed of the object class names and a scene name, which is usually related to the dataset name and fixed for each dataset. The ControlNet, along with the integrated encoders, are trained with the reconstruction loss. In addition, the weights of foreground regions are enhanced to produce more realistic foreground objects in complex scenes. The overall loss function is formulated as:
Lrecon Eus.ccc.onn.c.e.../..... [[]e eo (xr, L Ci, Cut, Ga]
(2)
Ccontrol Creeon + Crecon C M
where M represents a binary mask with on foreground pixels 0 on background pixels. @ represents element-wise multiplication, and controls the foreground re- re-weighting.
3.3 Dataset Synthesis Pipeline for Object Detection
Our dataset synthesis pipeline is summarized in Fig. 3 We generate random bounding boxes and classes pseudo labels based on the distributions of the training set. Then, the pseudo labels converted to triplets: <image list, text list, global text prompt>. ODGEN uses the triplets as inputs to synthesize images with the fine-tuned diffusion model. In addition, we filter out the pseudo labels in which areas the foreground objects fail to be generated.
Object distribution estimation. To simulate the training set distribution, we compute the mean and variance of bounding box attributes and build normal distributions. In particular, given dataset with K categories, we calculate the class-wise average occurrence number per image μ (ii P2,..... : (x) and the covariance matrix Σ to build multi-variable joint normal distribution Mu, E). In addition, for each category k, we build normal distributions N (Uz, oz) and N N( og)
5

---

Annotation
ReCo
GLIGEN
ControlNet
GeoDiffusion
ODGEN
MRI Image
positive
Apex Game
scallop
Underwater
bolotburiur
scallop
scallop
Grair boreum
Cotton
G-arboreust
G Caceraacum
Figure 4:
Comparison between ODGEN and other methods under the same condition shown in the first column. ODGEN can be generalized to specific domains and enables accurate layout control.
for the top-left coordinate of the bounding boxes, N (Marea, Parea) for box areas, and distributions N (prratoo oatsoo) for aspect ratios.
Image synthesis. We first sample the object number per category from the K-dimensional joint normal distributions. Then for each object, we sample its bounding box location and size from the corresponding estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts. Finally, we use ODGEN generate images with the triplets.
Corrupted label filtering. There is certain possibility that some objects fail to be generated in synthetic images. While utilizing image lists alleviates the concern to some extent, the diffusion model may still neglect some objects complex scenes containing dense or overlapping objects. As result, the synthesized pixels could not match the pseudo labels and will undermine downstream training performance. We fine-tune a ResNet50 [17] with foreground and background patches cropped from the training set to classify whether the patch contains object. The discriminator checks whether objects are successfully synthesized in the regions of pseudo labels, rejects nonexistent ones, and further improves the accuracy of synthetic datasets.
Experiments
We conduct extensive experiments demonstrate the effectiveness of the proposed ODGEN in both specific and general domains. FID 19 is used as the metric of fidelity. Mean average precisions (mAP) of YOLOv5s (25 and YOLOv7 (54) trained with synthetic data are used evaluate the trainability, which concerns the usefulness of synthetic data for detector training. Our approach is implemented with Stable Diffusson v2.1 and compared with prior controllable generation methods based on Stable Diffusion, including ReCo [61] GLIGEN (33. ControlNet 62], GeoDiffusion (5). Diffusio (55), and MIGC [67]. Native ControlNet doesn't 't support bounding box conditions. Therefore, we convert boxes to mask C' sized H x W x K, where H and W are the image height and width, and K is the class number. one pixel (i,j) is boxes of class k, the Cli, [[i,j, k) = n. YOLO models are trained with the same recipe as Roboflow-1ln 100 6 for 100 epochs to ensure convergence. The length text and image lists N used in our ODGEN set to the maximum object number per image in the training set.

---

Table FID (4) scores computed over 5000 images synthesized by each approach on RF7 datasets. ODGEN achieves better results than the other on all 7 domain- datasets.
Datasets
ReCo
GLIGEN
ControlNet
GeoDiffusion
ODGEN
Apex Game
88.69
125.27
97.32
120.61
58.21
Robomaster
70.12
167.44
134,92
76.81
57.37
MRI Image
202.36
270.52
212.45
341.74
93.82
Cotton 108.55
89.85
196.87
203.02
85.17
Road Traffic
80.18
98.83
162.27
68.11
63.52
Aquarium 122.71
98.38
146.26
162.19
83.07
Underwater 73.29
147.33
126.58
125.32
70.20
Table 2: mAP@ 550:95 (1) of YOLOv5s YOLOV7 on RF7. Baseline models are trained with 200 real images only, whereas the other models are trained with 200 real 5000 synthetic images from various methods. ODGEN leads to the biggest improvement all domain-specific datasets.
Bascline
ReCo
GLIGEN
ControlNet GeoDiffusion
ODGEN
real synth #
200 + 0
200 5000
200 + 5000
200 + 5000
200 + 5000
200 5000
Apex Game
38.3/47.2
25.0//1.5
24.87 32.5
33.8 42.7
29.27
39,9 52.6
Robomaster
27.2
18.2/279
25.1
24.9
18.2
39.6/
MRI Image
37.6/27.4
32.7/38.3
32.3/25.9
44.7 /37.2
42.0/38.9
46.1 / 41.5
Cotton 16.7
29.3/
28.0/39.0
22.6/
30.2/36.0
42.0 / 43,2
Road Traffic
34.3
/
30.5
29.4
39.2 / 43.8
Aquarium
30.0/29.6
33.8/
24.1
18.2/25.6
21.6 30.9
32.2 38.5
Underwater
16.7/
13.7/
14.9/ 18.5
1
13.8/
19.2 / 22.0
4.1 Specific Domains
Roboflow-10k 100 (6] used for the evaluation in specific domains. It consists of 100 object detection datasets of various domains. We select representative datasets (RF7) covering video games, medical imaging, and underwater scenes. To simulate data scarcity, we only sample 200 images as the training set for each dataset. The whole training process including the fine-tuning on both cropped objects and entire images and the training of the object-wise conditioning module, only depends on the 200 images. À in Eq. (I) and in Eq. (2) are set as I and 25. We first fine-tune the diffusion model according Fig. (a) for 3k iterations. Then we train the object-wise conditioning modules on V100 GPU with batch size for 200 epochs, the same as the other methods to be compared.
4.1.1 Fidelity
For each dataset in RF7, we compute the FID scores of 5000 synthetic images against real images, respectively. As shown Tab. I our approach outperforms all the other methods. We visualize the generation quality in Fig. 4 GLIGEN only updates gated self-attention layers inserted to Stable Diffusion, making hard to generalized specific domains like MRI images and the Apex game. ReCo and GeoDiffusion integrate encoded bounding box information into text tokens, which require more data for diffusion model and text encoder fine-tuning. With only 200 images, they fail to generate objects in the given box regions. ControlNet, integrating the bounding box information into the visual condition, presents more consistent results with the annotation. However, it may still miss some objects generate wrong categories in complex scenes. ODGEN achieves superior performance on both visual effects and consistency with annotations.
4.1.2 Trainability
To explore the effectiveness synthetic data for detector training, we train YOLO models pre-trained on COCO with different data and compare their mAP @ .50:95 scores in Tab. In the baseline setting, we only use the 200 real images for training. For the other settings, we use combination of 200 real and 5000 synthetic images. Our approach gains improvement over the baseline and outperforms the other methods.
In addition, we add experiments with larger-scale datasets with 1000 images sampled from the Apex Game and the Underwater datasets. The training setups are kept the same as the training on 200 images. We conduct experiments ODGEN, ReCo, and GeoDiffusion. ReCo and GeoDiffusion
7

---

Table 3: mAP@ / m 50:995 (f) results ODGEN trained on larger-scale datasets of 1000 real images. The top 3 rows show results of YOLOv5s and the bottom 3 rows show results of YOLOv7. Baseline models are trained with 1000 real images only, whereas the other models are trained with 1000 real + 5000 / 10000 synthetic images from various methods. ODGEN leads to more significant improvement than other methods.
Datasets
Apex Game
Apex Game
Apex Game
Underwater
Underwater
Underwater
real + synth
0
1000 5000
+000 10000
1000 +
1000 + 5000
1000 + 10000
ReCo
53.5
78.7 46.99
82.0/
55.6/ 29.2
55.1 28,4
53.9/29.1
GeoDiffusion
53.5
80,0/47.2
82,5 / 47.5
55.6 29.2
54.2/ 27.9
54.3 28.0
ODGEN
83.2/
33.3/5355
83.6 / 53.6
55.6 29.2
59.6 32.5
56.3/ 29.8
ReCo
83.87 55.0
80.5 50.7
79.2 49.9
54.67 28.3
56.5 28.7
56.47 30.1
GeoDiffusion
83.8 55.0
81.2 51.0
81.0 50.5
54.6 / 28.3
57.0 28.9
55.8 28.9
ODGEN
83.8 55.0
84,4 55.2
84.0 / 55.0
54.6 / 28.3
58.2 / 29.8
62.1 31.8
Annotation
ReCo
GLIGEN
ControlNet
GeoDiffusion
MIGC
ODGEN
bottle
cell phone
food
Figure 5: Visualized results comparison for models trained on COCO. ODGEN is better qualified for
synthesizing complex scenes with multiple categories of objects and bounding box occlusions.
may benefit from larger-scale training datasets since they need to fine-tune more parameters in both the UNet in Stable Diffusion and the CLIP text encoder. GLIGEN struggles adapt to new domains. ControlNet performs worse on layout control than ODGEN. Therefore, these two methods are not included this part. The corrupted label filtering step is not used for any method. Results in Tab. show that the baselines trained real data only become stronger with larger-scale datasets while our ODGEN still benefits detectors with synthetic data and outperforms other methods.
Table 4: FID (L) and mAP (1) of YOLOv5s / YOLOV7 on COCO. FID is computed with 41k synthetic images. For mAP, YOLO models are trained from scratch on 10k synthetic images and validated on 3 31k real images. ODGEN outperforms all the other methods in terms of both fidelity and trainability.
Geo-
Instance-
Metries
ReCo
GLIGEN
Control
MIGC
ODGEN
Diffusion
Diffusion
FID
18,36
26.15
25.54
30.00
21.82
23.29
16.16
mAP@.
7.60 11.01
6.7079.42
1.43/1.15
9.54/16.01 5.94/9.21
10.00/17.10
18.90
24.40
mAP@ ..A...00:95
3,82 / 5.29
3.56 4.66
0.52/0.33 0.38
2.37 /4.44
4.67 /8.65
5.42 10.20
9.70
14.20
4.2 General Domains
COCO-2014 37 is used to evaluate method in general domains. As the scenes in this dataset almost covered by the pre-training data of Stable Diffusion, we skip the domain-st -spccifc fine-tuning stage and train the diffusion model together with the object-wise conditioning modules. We train our ODGEN on the COCO training set with batch size 32 for 60 epochs on x8 V100 GPUs as well GLIGEN, ControlNet, and GeoDiffusion. Officially released checkpoints of ReCo (trained for 100 epochs on COCO), MIGC (trained for 300 epochs on COCO), and InstanceDifflusio (trained on self-constructer datasets) employed for comparison. y in Eq. (2) is set 10 for ODGEN.
4.2.1 Fidelity
We use the annotations of the COCO validation set as conditions to generate 41k images. The FID scores Tab. Jrre computed against the COCO validation set. ODGEN achieves better FID results than the other methods. We provide typical visualized example in Fig. 5 Given overlapping bounding boxes of multiple categories, ODGEN synthesizes all the objects of correct categories and accurate positions, outperforming the other methods. More samples added in Appendix G

---

Table 5: mAP@@ .50 (1) and mAP@ mnA...50:.95 (f) YOLOv5s/ YOLOV7 on the COCO dataset. Baseline models are trained with 80k images from the COCO training set, whereas the other models are trained
with the same 80k real 20k synthetic images. ODGEN outperforms baseline and the other methods.
Metrics
Baseline
ReCo
GLIGEN
ControlNet
mAP@ .50
51.5764.5
50.8/64.3
50.9/64.2
50.1/64.3
mAP@ 50:95
32.6/45.4
31.8/45.2
31.9 / /4..2
31.0/45..
Metrics
GeoDiffusion
MIGC
InstanceDiffusion
ODGEN
@AP@. ,50
51.2/64.4
64.6
51.5 /4.6
mAP@ 50:995
32.1/45.2
42.5
32.6/ 45.6
33.1/ /4..9
4.2.2 Trainability
We use 10k annotations randomly sampled from the COCO validation set to generate a synthetic dataset of 10k images. The YOLO models are trained this synthetic dataset from scratch and evaluated on the other 31k images in the COCO validation set. As shown in Tab. ODGEN achieves the other methods in terms of mAP.
significant improvement
We further conduct experiments by adding 20k synthetic images to the 80k training images. We train YOLO models from scratch the COCO training set (80k images) the baseline and on the same 80k real images 20k synthetic images generated by different methods for comparison. We use the labels of 20k images from the COCO validation set conditions to generate the synthetic set and use the other 21k real images for evaluation. The results are shown in Tab. 5 It shows that ODGEN improves the mAP@ .50:95 by 0.5% and outperforms the other methods.
observe that YOLO models trained on synthetic data only fall behind models trained on real data. We add experiments with different training and validation data combinations in Tab. 6 YOLO models trained on real images show better generalization ability and achieve close results when tested on real and synthetic data. YOLO models trained on synthetic data only get significantly better results when tested on synthetic data than when generalized real data. It indicates that noticeable domain gaps still exist between real and synthetic data, which may be limited by the generation quality of modern Stable Diffusion models and are promising to be narrowed with future models.
Table 6: mAP@ .50 (1) and mAP@. MAP@.50:95 (f) of YOLOv5s / YOLOv7 trained from scratch and validated real synthetic COCO validation set. 10k images used for training and the other 31k images are used for validation. Real represents real images in the COCO validation set and synthetic represents images synthesized by our ODGEN using the same labels.
Train Validate
mAP@ .50 (t)
mAP@_500 (t)
Train Validate inA@@_50 (f)
mAP@. .5:995 (1)
Real Real 29.40 41.70
15,807 26.30
Synthetic Real 18,90/ 24.40
9.70 14.20
Synthetic
29.40 39.90
15,00/ 23.20
Synthetic Synthetic
37,90/ 45.40
20.40 27.10
Table 7: Using the image list (II.) and the text list (TL) benefits FID and mAP (YOLOv5s YOLOv7). They especially helpful for the Road Traffic dataset which has more categories and occlusions.
Table 8: Proper y values in Eq. (2) benefit
both FID and mAP (YOLOv5s / YOLOv7) of synthetic images, while overwhelming values lead to degeneration.
Datasets
IL
TL
FID())
mAP@ AAP@.50:955 (t)
Datasets
7
FID
mAP@ mAP@.500.5 (f)
X
98.29
44.6/39.9
0
83.94
30.87 34.5
MRI Image
X
V
95.67
44.9 41.4
Aqua- rium
25
83.07
32,2 38.5
X
99.16
46.2 41.0
50
87.00
32.1 / 38.0
V
V
93.82
46.1 41.5
100
90.13
29.9/35.5
X
67.40
35.5
0
66.65
36,97 39.5
Road Traffic
x
Y
66.48
26.5
Road
25
63.52
39.2 / 43.8
V
x
65.80
32.3/39.1
Traffic
50
66.46
36.6/ /3.99
V
63.52
39.2/ 43.8
100
72.18
32.9 03..1
4.3 Ablation Study
Image list and text list in object-wise conditioning modules. conditioning with image list and text list as shown in Fig.
ODGEN enables wije
We test the performance of

---

Image List:
Image List:
Image List:
X
Image List: V
Image List:
Image List: Text List:
Text List:
Text List:
Text List:
Text List:
Text List:
X
traffii light
srathic Tight
motorryy
trifficc
Figure 6: Visualized ablations. Left: using neither image nor text lists, traffic light generated in the position of bus, and car is generated in the position of a traffic light; Middle: not using image list, two occluded motorcycles are merged as one; Right: not using text list, motorcycle is generated in the position of vehicle. Using both image and text lists generates correct results.
bus
vehicle
vehicle
Annotation
==
25
= 50
= 100
Figure With increasing 7 value, our approach produces higher-quality foreground objects. However, overwhelming leads to blurred background and degenerated fidelity.
using global text prompts in the placement of text lists or pasting all the foreground patches on the same empty canvas in the placement of image lists for the ablation study. Experiments performed on the MRI and Road Traffic datasets in RF7. The MRI dataset only has two categories and almost no overlapping objects, whereas the road traffic dataset has 7 categories and many occluded objects. From the results in Tab. 7 it can be found that using image lists or text lists brings moderate improvement on MRI images and significant improvement on the more complex Road Traffic dataset. Visualization in Fig. shows that the image list improves the fidelity under occlusion, and the text list mitigates the mutual interference of multiple objects.
Foreground region enhancement. We introduce a re- weighting term controlled by y for foreground objects in Eq. 2 Tab. [8] shows that appropriate y values can improve both the fidelity and the trainability since the foreground details are better generated. However, increas- ing the value will undermine the background quality, visualized in Fig. 6
Table 9: mAP (YOLOv5s/ / YOLOv7) of the corrupted label filtering for ODGEN.
Label
Datasets
mAP@.50:.55 50: (t)
filtering
42.07 43.2
Cotton
X
40.5 / 42,1
Corrupted label filtering. Results in Tab. show that the corrupted label filtering step helps improve the mAP@ 50:95 of ODGEN by around 1-2%. The results of ODGEN without the corrupted label filtering still signif- icantly outperform the baseline setting in Tab.
Robo-
39,6 34.7
master
X
Under-
V
19.2/ 22.0
water
X
18.9/210
5 Conclusion
This paper introduces ODGEN, a novel approach aimed at generating domain-specific object detection datasets to enhance the performance of detection models. We propose fine-tune the diffusion model on both cropped foreground objects and entire images. We design a mechanism to control the foreground objects with object-wise synthesized visual prompts and textual descriptions. Our work significantly enhances the performance of diffusion models in synthesizing complex scenes characterized by multiple categories of objects and bounding box occlusions. Extensive experiments demonstrate the superiority of ODGEN in terms of fidelity and trainability across both specific and general domains. We believe this work has taken an essential step toward more robust image synthesis and highlights the potential of benefiting the object detection task with synthetic data.
10

---

Acknowledgements
We thank support from National Natural Science Foundation of China under Grant 62227801 and National Science and Technology Major Project under Grant 20222D0017902.
References
[1] S. Azizi, S. Korblith, C. Saharia, M. Norouzi, and D. J. Flect. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Researck, 2023.
[2] Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro,
et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint
arXiv:2211 1.00124, 2022.
[3] O. Bar-Tal, L. Yariv, Y. Lipman, and Dekel. Multidiffusion: fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference on Machine Learning, pages 1737-1752, 2023.
[4] A. Brock, Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[5] K. Chen, F. Xie, 7. Chen, Y. Wang, I.. Hong. Z. Li, and D.-Y. Yeung. Geodiffusion: Text-prompted geomet- ric con-trol for object detection data generation. In International Conference Learning Representations, 2024.
[6] F. Ciaglia, F. S. Zuppichini, P. Guerrie, M. McQuade, and J. Solawetz. Roboflow 100: A rich, multi-domain object detection benchmark. arXiv preprint arXiv:2211.3352 2022.
[7] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 2021.
[8] L. Dunlap, A. Umino, H. Zhang, J. Yang, J. E. Gonzalez, and T. Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. Advances in Neural Information Processing Systems, 36, 2024.
[9] N. Dvornik, J. Mairal, and C. Schmid. Modeling visual context is key augmenting object detection datasets. In Proceedings of the European Conference on Computer Vision, pages 364-380 2018.
[10] H. Fang, B. Han, S. Zhang. S. Zhou, C. Hu, and W.-M. Yc. Data augmentation for object detection via controllable diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1257-1266, 2024.
[11] C. Feng, Y. Zhong, Jie, W. Xie, and L.. Ma. Instagen: Enhancing object detection by training on synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14121- 14121-14130, 2024.
[12] W. Feng, X. Hc, T.-J. Fu, V. Jampani, A. R. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference Learning Representations, 2022.
[13] Y. Ge. J. Xu, B. N. Zhao, N. Joshi, L. Itti, and Vineet. Dall-e for detection: Language-driven compositional image synthesis for object detection, arXiv preprint arXi::2206.0955 09592, 2022.
[14] G. Ghiasi, Y. Cui. A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph. Simple copy copy-paste paste strong data augmentation method for instance segmentation. Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 2918- 2928, 2021.
[15] R. Gong, M. Danelljan, H. Sun, D. Mangas, and L. Van Gool. Prompting diffusion representations for cross-domain semantic segmentation. arXiv preprint arX0v:2307.001333 2023.
[16] I. Goodfellow, J. Pbadet-Abadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
[17] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] R. He, Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. QL Is synthetic data from generative models ready for image recognition? In The Eleventh International Conference on Learning Representations, 2022.
[19] M. Heusel, I.. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances Neural Information Processing Systems, 30. 2017.
[20] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-668 165006661111100 2020.

---

[21] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. Tži-compbench: A comprehensive benchmark for open-world compositional lext-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[22] M. Jahn, R. Rombach, and B. Ommer. High-resolution complex scene synthesis with transformers. arXiv preprint arXiv:2105.064555 2021.
[23] Y. Jia, L. Hoyer, S. Huang. T. Wang. L. Van Gool, K. Schindler, and A. Obukhov. Dginstyle: Domain- generalizable semantic segmentation with image diffusion models and stylized semantic control. In Syntheric Data for Computer Vision Workshop@ CVPR 2024, 2023.
[24] C. Jiaxin, L. Xiao, S. Xingjian, H. Tong. X. Tianjun, and L. Mu. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.08900 2023.
[25] G. Jocher, A. Chaurasia, A. Stoken, J. Borovee, Y. Kwon, K. Michael, J. Fang, C. Wong, Z. Yifu, D. Montes, et al. ultralyties/yolov5:: v6. 2-yolov5 classification models, apple ml, reproducibility, clearml and deci. integrations. Zenodo, 2022.
[26] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alies-free generative adversarial networks. Advances Neural Information Processing Systems, 34:852-863, 2021.
[27] T. Karras, S. Laine, and T. Aila. style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[28] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020.
[29] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in Neural Information Processing Systems, 1422169--21701 2021.
[30] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.66 14, 2013.
[31] N. Kondapaneni, M. Marks, M. Knott, Guimaraes, and Perona. Text- Text-image alignment for diffusion- based perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 13883-13893, 2024.
[32] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International conference on machine learning, pages 12888- 12900. PMLR, 2022.
[33] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang. J. Gao, C. Li, and Y. J. Lee, Gligen: Open-set grounded text-to- image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22511-22521, 2023.
[34] Z. Li, Y. Li, P. Zhao, R. Song, X. Li, and J. Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint S.ti::2305.19944 2023.
[35] Z. Li, J. Wu, I. Koh, Y. Tang, and L. Sun. Image synthesis from layout with vecality-aware mask adaption. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13819-13822 13828, 2021.
[36] Z. Li, Q. Zhou, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Open-vocabulary object segmentation with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7667-7676, 2023.
[37] T.-Y. Lin, M. Maire, S. Belongic, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[38] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[39] D. Peng, P. Hu, Q Ke, and J. Liu. Diffusion- hiffusiio-based image translation with label guidance for domain adaptive semantic segmentation. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 808-820, 2023.
[40] D. Podell, English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[42] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint arXiv:2204.061 06122 2022.
12

---

[43] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pages 1278-1286 PMLR. 2014.
[44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[45] N. Ruiz. Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 10, 2023.
[46] C. Saharia, W. Chan, Saxena, L. Li, J. Whang, L. Denton, K. Ghasemipour, Gontijo Lopes,
B. Karagol Ayan. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[47] B. Sanyıldız, K. Alahari, D. Larlus, and Kalantidis. Fake till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 801 1- 101201 2023.
[48] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta,
C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 2022.
[49] J. Sohl-Dickstein E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265, 2015.
[50] Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems, 33: 1243--14488 2020.
[51] B. Trabucco, K. Doherty, M. A. Gurinas, and R. Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[52] A. Vahdat and J. Kautz. Nvac: A deep hicrarchical variational autoencoder. Advances in Neural Information Processing Systems, 33: 19667-9967.,, 2020.
[53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[54] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao. Yolov7: Trainable hag-of-frechies new state-of-the- art for real-time object detectors. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7464-7475, 2023.
[55] X. Wang, T. Darrell, Rambhatla, R. Girdhar, and I. Misra. Instancediffusson: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2232-6242, 2024.
[56] Wu, Zhao, H. Chen, Y. Gu, R. Zhao, He, H. Zhou, M. Z. Shou, and C. Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36:54683-54695, 2023.
[57] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Symthesizing images with pixel- level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1206-1217, 2023.
[58] J. Xic, Y. Li, Y. Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452-7461, 2023.
[59] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to- -mage diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[60] I... Yang, X. Xu, B. Kang, Y. and H. Zhao. Freemask: Synthetic images with dense annotations make
stronger segmentation models. Advances in Neural Information Processing Systems, 36, 2024.
[61] Z. Yang, J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan, Z. Liu, C. Liu, M. Zeng, et al. Reco: Region- controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14246- 14255, 2023.
[62] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[63] M. Zhang, J. Wu, Y. Ren. M. Li. J. Qin. X. Xiao, W. Liu, R. Wang. M. Zheng, and J. Ma. Diffusionengine: Diffusion model is scalalle data engine for object detection. arXiv preprint arXiv:2309.0338 2023.
13

---

[64] Y. Zhang, H. Ling, J. Gao, K. Yin. J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10145-10155, 2021.
[65] H. Zhao, D. Sheng. J. Bao, D. Chen, D. Chen, F. Wen, L. Yuan, C. Liu, W. Zhou, Q. Chu, et al. X-paste: revisiting scalable copy-paste for instance segmentation using clip and stablediffusion. In International Conference on Machine Learning, pages 42098 42109. PMLR, 2023.
[66] G. Zheng, X. Zhou, X. Li, Z. Qi, Y. Shan, and X. Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490-22499, 2023.
[67] D. Zhou, L. F. Ma, X. Zhang, and Y. Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6818-6828, 2024.
[68] J. Zhu, H. Ma, J. Chen, and Yuan. Isolated diffusion: Optimizing multi-concept text-to-image generation training- freelyy with isolated diffusion guidance. arXiv preprint wrXiv:2443.1699.5 2024.
14

---

A Limitations
Despite the significant improvement achieved by this work, we find remains challenging to achieve comparable training performance as real data with merely synthetic data. We consider the limitation be mainly caused by two reasons. Firstly, the generation fidelity is still limited by modern diffusion models. For instance, VAEs in Stable Diffusion are not good enough to deal with complex text in images, constraining the performance on video game datasets. ODGEN promising to achieve further improvements with more powerful diffusion models in the future. Secondly, this paper mainly focuses on designing generative models with given conditions. However, the other parts in the dataset synthesis pipeline are not fully optimized, which are interesting to be explored in future work.
B Broader Impact
We introduce a novel approach for object detection datasets synthesis, achieving improvement in fidelity and trainability for both specific and general domains. Our approach is more prone biases caused by training data than typical Al generative models since shows robustness with limited data of specific domains. Therefore, we recommend practitioners to apply abundant caution when dealing with sensitive applications to avoid problems of races, skin tones, or gender identities.
C Supplemental Ablations
Detector training w/o real data. In the main paper, we train the detectors with 200 real images and 5000 synthetic images for RF7 benchmarks. In this ablation study, we conduct experiments with the 5000 synthetic images only to explore the impact of the absence of real data for training detectors. As shown in Tab. [0] for easy datasets with fewer objects, like MRI image and Cotton, training with 5000 synthetic data only achieves better results than training with 200 real samples only. Increasing the dataset size can improve the model performance because the generation quality good enough in these cases. Nevertheless, for complex scenes like Robomaster and Aquarium, training with synthetic data only fails to achieve better results. It may be caused by the limited generation quality modern diffusion models. For instance, the VAE in Stable Diffusion is not good at synthesizing texts in images, which is common in the Robomaster dataset. Besides, FID results in Tab. T Iidiiaate that there still exist gaps between the distributions of real and synthetic images. The performance of our approach is promising to be further improved in the future with more powerful generative diffusion models. In addition, the current dataset synthesis pipeline is not fully optimized and cannot reproduce the distributions of foreground objects perfectly, which may also lead to worse performance of training on synthetic data only with the RF7 datasets.
Table 10: Ablations of using real data only and using synthetic data only during detector training. (Metric: YOLOv5s YOLOv7)
Metrics
mAP@.5M .50 (1)
mAP@ ..5099995 (?)
real + synth
+00+0
5000
200
5000
200+0
0+5500
200 5000
Robomaster
50.3/48.8
35.9
67.4/65.4
27.2/
18.7/21.0
39.6/
MRI Image
55.1
59,8 55.2
68.1
37.6 27.4
41.3//5.3
46.1 /41.5
Cotton
39.4/32.0
43.1 /43.2
55.9/55.0
16.7 / 20.5
29.5
42.0/43.2
Aquarium
53,8/53.0
4
62.6 66.5
30.0 29.6
23.8/
3 38.5
Number of synthesized samples. We ablate the number of synthesized samples used for detector training and provide quantitative results in Tab. [I] With increasing amounts of synthesized samples, detectors achieve higher performance and get very close results between 5000 and 10000 synthetic samples.
Category isolation for datasets synthesis. As illustrated in the main paper, modern diffusion models have the limitation of "concept bleeding" when multiple categories are involved the generation of one image. An alternative method could be only generating objects of the same category in one image. We select 3 multi-class datasets and compare this approach against the proposed method in ODGEN. As shown in Tab. 12 ODGEN achieves better results on most benchmarks than generating samples with isolated categories. This an expected result since generating objects of various categories in one image should be better aligned with the test set.
15

---

Table 11: Ablations of the number of synthetic samples (Metric: mAP@. .50 (1) of YOLOvvs/ YOLOv7, real images # 200). Results are very
Table 12: Ablations of category isolation dur-
ing datasets synthesis (Metric: mAP@. (f) of YOLOvvs YOLOv7, real + synth images 200 5000). It hinders the performance in most cases.
close between 5k and 10k synthetic samples.
synth #
Robomaster
Cotton
Aquarium
50.3748.8
29.4/32.0
53.8/5300
Split Categories
True
False
1000
52.2/61.7
54.0/51.3
55.0 63.8
Road Traffic
63.4/70.3
66.8 70.4
3000
61.9/61.1
62.6/564 56.4
56.8 / 65.8
Aquarium
63.2/63.3
62.6 / 66.5
5000
65.4
63.9 / 55.0
62.6/ 66.5
Underwater Object
41.1 41.1/42.0
40.0 / 44.8
10000
65.4
64.4/566
62.8 66.3
Table 13: mAP@ .50:95 (1) of YOLOv5s / YOLOv7 on RF7. Baseline models are trained with 200 real images only, whereas the other models are trained with 200 real 5000 synthetic images from various methods. The corrupted label filtering step not applied to all methods. ODGEN still leads to the biggest improvement all 7 domain-specific datasets.
Baseline
ReCo
GLIGEN
ControlNet
GeoDiffusion
ODGEN
real synth
200 + 0
200 5000
200 + 5000
200 + 5000
200 5000
200 5000
Apex Game
38.3/47.2
25.0//1.5
24.87 32.5
33.8 42.7
29.27
39.8/ 52.6
Robomaster
27.2 26.5
18.2/
19.1/25.0
24.4/
18.2
39.0 33.3
MRI Image
37.6/27.4
32.3/38.3
32.3
44.7 /3.22
42.0/38.9
46.1 / 44.5
Cotton 20.7
29.3/
28.0 39.0
22.1135.1
30.2/36.0
40.5 / 42.1
Road Traffic
35.3/41.0
22.8
22.2 29.5
22.1 30.5
/
38.2 43.2
Aquarium
29.6
23.8/34.
24.1 32.2
25.2
21.6/ 30.9
32.0 / 38.4
Underwater 16.7/
13.7/
14.9/ 18.5
15.5 17.8
13.8 17.2
18.9 / 21.6
Corrupted label filtering. We design this step to filter some labels when objects are not generated successfully, which may be caused by some unreasonable boxes obtained with the pipeline in Fig. For experiments on COCO, this step is not applied to any method since we directly use labels from the COCO validation set and hope to synthesize images consistent with the real-world labels. For RF7 experiments, this step is only applied to our ODGEN in Tab. 2 For fair comparison, we skip this step on RF7 to compare the generation capability of different methods fairly and provide results in Tab. 13 It shows that the corrupted label filtering step only contributes a small part of the improvement. Without this step. method still outperforms the other methods significantly. In addition, as illustrated in Appendix A the current dataset synthesis pipeline is designed to compare different methods and can be improved further in future work.
Foreground region enhancement on COCO. We set y as 10 in Eq. (2) in our experiments trained on COCO. We add ablations with 7 value of 0 to ablate foreground region enhancement on COCO and provide quantitative results Tab. [4] It shows that the foreground region enhancement helps ODGEN achieve better results, especially the layout-image consistency reflected by mAP results.
Table 14: FID (L) and mAP (f) of YOLOv5s / YOLOv7 on COCO. FID computed with 41k synthetic images. For mAP, YOLO models are trained from scratch on 10k synthetic images and validated on 31k real images. Foreground region enhancement contributes the improvement of both FID and mAP results.
Method
FID (4)
mAP@.50 (t)
mAP@ 50:95 (t)
ODGEN w/ foreground region enhancement
16.16
18,90 / 24.40
9.70 14.20
ODGEN w/o foreground region enhancement
16.45
16,90 / 23.10
8.72 / 13.50
Fine-tuning on both cropped foreground objects and entire images. For specific domains, taking the Robomaster dataset in RF7 an example, most images in the dataset contain multiple categories of objects like "armor", ; "base", "watcher", and "car", which are either strange for Stable Diffusion or different from what Stable Diffusion tends to generate with the same text prompts. For models fine-tuned on entire images only, they cannot obtain direct guidance on which parts in entire images correspond to these objects. As result, the fine-tuned models cannot synthesize correct images for foreground objects given text prompts like "a base in screen shot of the Robomaster game". We provide FID results of foreground objects synthesized by models fine-tuned on both entire images and cropped foreground objects (text prompts composed of object names and the scene name,
16

---

"A bicycle in traflic scene"
"Crosswalks traffic scene"
"A fire hydrant in traflic scene"
"Traffic lights in traffic scene"
"A vehicle traffic soene"
"A bus in traffic scene"
Visualized examples produced by the model fine-tuned on entire images from the Road Traffic dataset only. It fails to produce images of foreground objects correctly.
Figure 8:
e.g., "a car and a base in a screen shot the Robomaster game") and models fine-tuned on entire images only in Tab. 15| It shows that the proposed approach helps fine-tuned models generate images of foreground objects significantly better. Similarly, we also provide FID results the Road Traffic dataset in Tab. [6] which is relatively more familiar for Stable Diffusion than the Robomaster dataset. Results also show the improvement provided by our approach compared with models fine-tuned on entire images only. Visualized examples produced by the model fine-tuned entire images only shown in Fig. g Such models struggle to capture the information of foreground objects and fail to produce images needed build image lists for our ODGEN.
Table 15: FID (L) results of foreground objects in the Robomaster dataset synthesized by models fine-tuned on different data.
Object categories
watcher
base
Entire images only
384.96
532.56
364.46
325.79
340.
Entire images and cropped objects
124.45
136.27
135.66
146.50
128.95
Table 16: FID (L) results of foreground objects in the Road Traffic dataset synthesized by models
fine-tuned on different data.
Ohject categories
vehicle
traffic light
motor-
fire
cross- walk
bus
hicycle
cycle
hydrant
Entire images only
241.64 105.27
240.55
213.90
205.22
253.40
238.63 118.58
153.76
Entire images and cropped objects
90.71
143.30
53.05
97.33
65.61
D
Implementation Details
D.1 Global Text Prompt
All the methods included for comparison in our experiments share the same textual descriptions of the foreground objects and entire scenes in the global text prompts for fair comparison. ReCo [61] and GeoDiffusion (5) further add the location information of foreground objects to their global text prompts following their approaches. For general domains, we concatenate the object class names into sentence as the global text prompt. For specific domains, we concatenate the object class names and the scene name into a sentence as the global text prompt.
D.2 ODGEN (ours)
We follow the default training setting used by ControlNet (62), including the learning rates and the optimizer to train our ODGEN models and provide fair comparison with ControlNet.
Offline foreground objects synthesis. To improve the training efficiency, we synthesize 500 foreground object images for each category offline and build synthetic foreground pool. During the training of object-wise conditioning modules, foreground images are randomly sampled from the pool to build the image list.
Image encoder. ControlNet uses a 4-layer convolutional encoder to encode input conditions sized 512 to 64 x 64 latents. The input channels usually set or 3 for input images. Our ODGEN
17

---

follows similar methods to convert the ippu conditions. The key difference is that we use image lists of synthesized objects as input conditions. The image lists are padded N and concatenated in the channel dimension for encoding. In this case, we expand the input channels of the encoder to 3N and update the channels of the following layers to comparable numbers. We provide the channels of each layer in Tab. 17 The convolutional kernel size is set as 3 for all layers. We design different encoder architectures according to the maximum object numbers that can be found can use smaller encoder to
single image. For
datasets like MRI in which most images contain only one object, make the model more lightweight. For a generic model trained on large-scale datasets, we can fix architectures of encoders with large N make it applicable to images containing many objects.
Table 17: Detailed input and output channels of each convolutional layer in the image encoder used in ODGEN. As the maximum object number per image for different datasets varies, we select different N and thus different channel numbers.
Layer
Layer 2
Layer 3
Layer 4
Datasets
N
Input
Output
Input
Output
Input
Output
Input
Output
Apex Game
6
18
32
32
96
96
128
128
256
Robomaster
17
51
64
64
96
96
128
128
256
MRI Image
2
6
16
16
32
32
96
96
256
Cotton
27
81
96
96
128
128
192
192
256
Road Traffic
19
57
64
64
96
96
128
128
256
Aquarium
56
168
168
168
192
192
224
224
256
Underwater
237
237
237
256
256
256
256
256
COCO
93
297
297
297
256
256
256
256
256
Text embedding encoder. Similar to the image list, the text list is also padded to the length N with empty text. Each item in the text list in encoded by the CLIP text encoder an embedding. With the CLIP text encoder used in Stable Diffusion 1.x and 2.x models, we get N text embeddings sized (batch size, 77, 1024). Then 4-layer convolutional layer to compress the information into one embedding. The input channel N and the output channels of following layers are [N/2], [NV/4], [N/8], We the convolutional kernel size as 3, the stride as 1, and use zero padding to maintain the sizes of the last two dimensions.
stack the embeddings to size (batch size, N. 77, 1024) and
Foreground/Backgroumm discriminator. We train discriminators by fine-tuning ImageNet pre- trained ResNet50 on foreground and background patches randomly cropped from training sets. They are split for training, validating, and testing by 70%, 10%, and 20%. The model is binary classification model that only predicts whether a patch contains any object. The accuracy on test datasets over 99% on RF7. Therefore, we can confidently it filter the pseudo labels.
D.3 Other Methods
ReCo 61 & MIGC 667 & GeoDiffusion 5 are layout-to-imant generation methods based on the official codes of ReCo and GeoDiffusion
Stable Diffusion. For the comparison RF7, to fine-tune the diffusion models on domain-specift datasets and generate synthetic datasets for experiments. For the comparison COCO, we directly use their released pre- traitreindd weights trained on COCO. MIGC only releases the pre-trained weight but not the code, so we only compare it with method the COCO benchmark. ReCo and GeoDiffusion fine-tune both the text encoder and the UNet of Stable Diffusion. MIGC only fine-tunes the UNet.
Diffason (55] depends multiple formats of inputs, including segmentation masks, boxes, and scribbles, to train its UniFusion module. However, the RF7 datasets specific domains do not provide segmentation masks and scribbles. Therefore, it is only employed for comparison on COCO with its opun-source checkpoint.
GLIGEN 33 is an open-set grounded text-to-image generation methods supporting multiple kinds of conditions. use GLIGEN for layout-to-image generation with bounding boxes as conditions. GLIGEN is designed to fit target distributions by only fine-tuning the inserted gated self-attention layers in pre-trained diffusion models. Therefore, we fix the other layers in the diffusion model during fine-tuning on RF7 or COCO.
18

---

ControlNet [62] is initially designed to control diffusion models by fixing pre-trained models, reusing the diffusion model encoder and fine-tuning provide control. In our experiments, to make fair comparisons, we also fine-tune the diffusion model part to fit it to specific domains that can be different from the pre-training data of Stable Diffusion. The diffusion model remains trainable for experiments on COCO as well. The native ControlNet is originally designed as image-to-imag generation method. As illustrated in Sec. 4 we provide a simple way to convert boxes to masks. Given dataset containing K categories, imgge sized H x W, and B objects with their bounding boxes bbor and classes cls, we convert them to mask in the following process:
Algorithm 1 Convert bounding boxes to mask for ControlNet
M = zeros H, W. K)
for i 11...... . do
coordleft: coordrop. coordright, coordhottom bboz
= clssi]
M Mccordeop coordbattom, coordleeft coordright, k]+
end for
Table 18: The number of images in the 7 subsets of Roboflow-100 100 used to compose the RF7 datasets.
Datasets
Train
Validation
Test
Apex Game
2583
415
691
Robomaster
1945
278
556
MRI Image
253
39
79
Cotton
367
20
19
Road Traffic
494
133
187
Aquarium
448
63
127
Underwater
5320
760
1520
E Datasets Details
RF7 datasets. We employ the RF7 datasets from Roboflow-100 6 to evaluate the performance of our ODGEN on specific domains. Here, we add more details of the employed datasets, including Apex Game, Robomaster, MRI Image, Cotton, Road Traffic, Aquarium, and Underwater. We provide the number of images for each dataset, including the train, validation, and test sets in Tab. [8] We use the first 200 samples in the image list annotation provided by Roboflow- for each dataset to build the training set for our experiments. The full validation set is used as the standard to choose the best checkpoint from YOLO models trained for 100 epochs. The full test set used to evaluate the chosen checkpoint and provide mAP results shown in this paper. Visualized examples with annotations shown in Fig. 9 The object categories each dataset are provided Tab. I9 Some images in the Apex Game and Robomaster datasets contain complex and tiny text that are hard to generate by current Stable Diffusion.
Supplemental Related Works
Copy-paste 14 65 method requires segmentation masks get the cropped foreground objects, which are not provided by the RF7 datasets used in this paper. Besides, our approach also serves as controllable image generation method that only needs bounding box labels produce realistic images, which is hard to achieve by capte.. InstaGEN [II] is mainly designed for the open-vocabulary object detection task. It generates images randomly and annotates them with a grounding head trained with another pre-trained detector. Earlier works like LayoutDiffusion [66] are not implemented with
latent diffusion models and thus are not included for comparison. Works 23 39 56) are designed for semantic segmentation, while our ODGEN is designed for object detection.
19

---

begatia
TOOHIN
ROGILRO
ROG
Real MRI Samples
Real Apex Game Samples
T
XXEE EEL
BUDGSROO
Real Robomaster Samples
Camorrumm
Real Cotton Samples
Rolsreyck
Real Road Traffic Samples
Dengue
BUZEO
Real Aquarium Samples
echnutinn af
echinus
Bthnus
Real Underwater Samples
Figure 9: Visualized examples with object detection annotations of RF7 datasets.
G
Supplemental Experiments
mAP@.5 on RF7. Tab. 20 provides quantitative results for trainability in terms of mAP@@ .50 supplements Tab. 2 Our approach achieves improvement over training real data only and outperforms other methods.
Visual relationship between image lists and synthesized images. The image list is designed to provide information on category and localization for controlling foreground object layouts. The synthesized images share the same category with objects in generated images and are pasted to
20

---

Datasets
Class Names
Apex Game
Avatar, Object
Robomaster
Armor, Base, Car, Rune, Rune-blue, Rune-gray, Rune-grey, Rune-red, Watcher
MRI Image
Negatíve, Positive G-arboreum, G-barbadense, G-herbaceum, G-hirsitum
Cotton Road Traffic
Bicycles, Buses, Crosswalks, Fire hydrants, Motorcycles, Traffic lights, Vehicles
Aquarium
Fish, Jellyfish, Penguin, Puffin, Shark, Starfish, Stingray Echinus, Holothurian, Scallop, Starfish, Waterweeds
Underwater
Table 19: Class names of the RF7 datasets.
Table 20: mAP@. ,0 (t) of YOLOv5s YOLOv7 on RF7. Baseline models are trained with 200
real images only, whereas the other models are trained with 200 real 5000 synthetic images from
various methods. ODGEN leads to the biggest improvement on all domain-specifie datasets.
Baseline
ReCo
GLIGEN
ControlNet
GeoDiffusion
ODGEN
real synth #
200 0
+0000000
+ 5000
200 + 5000
200 5000
200 5000
Apex Game
78.4
53.3/60.0
53.3/57.0
59.3 70.3
55.87
69.1/82.1
Robomaster
50.3/48/
41.6/54.4
38.7/
44.4
41.5/45.7
67.4 / 65,4
MRI Image
55.1 /44.7
59.7/
52.1
64.7/55.4
61.6/56.1
68.1 / 61.3
Cotton
39.4/32.0
42.7/53.4
41.8/55.0
38.3
49.6/49.4
63.9 / 55.0
Traffic
61.1 68.1
42.6/52.4
45.0 54.5
/
38.2 54.6
66.8 / 70.4
Aquarium
53.8/
45.8/59.0
51.2
3
4 54.8
62.6 / 66.5
Underwater
35.7/39.2
34.5
33.67
34.3
31.8/7.6
40.0 / 44.8
the same position those in generated images to build image lists. The objects image lists and synthesized images may all be apples cakes but have different shapes and colors, as shown by the visualized samples provided in Fig. TO
Detector performance improvement with synthetic data. We provide several visualized samples in Fig. and show that the synthetic data helps detectors detect occluded objects.
Generalization to novel categories. We use ODGEN trained on COCO to synthesize samples containing subjects not included in COCO, like moon, ambulance, tiger, et al. We show visualized samples in Fig. 12 It shows that ODGEN trained on COCO can control the layout of novel categories. However, we find that the generation quality is not very stable, which may be influenced by the fine-tuning process on the COCO dataset. In future work, we hope to get a powerful and generic model that is robust to the most common categories in daily life.
Computational cost. Our ODGEN needs to synthesize images of foreground objects to build image lists, leading to higher computational costs for training and inference. Therefore, we propose to generate an offline image library of foreground objects to accelcrate the process of building image lists. With the offline library, synthesizing new images of foreground objects every time.
randomly pick images from it to build image lists instead of
Taking the model trained on the COCO dataset as an example, our ODGEN shares a very close scale of parameters with ControlNet (Trainable parameters: ODGEN 1231M v.s. ControlNet 1229M, parameters in the UNet of Stable Diffusion are included). We provide the training time for 1 epoch on COCO with V100 GPUs of different methods in Tab. 21
Table 21: Training time for I epoch on COCO with 8 V100 GPUs.
Method
ReCo
GLIGEN
ControlNet
GeoDiffusion
ODGEN
Training Time 3 hours
7 hours
4.5 hours
3.2 hours
5.6 hours
In the inference stage (Fig. ()), our ODGEN pads both the image and text lists to a fixed length. Therefore, the computational cost for inference with an offline library doesn't 't increase significantly with more foreground objects. Other methods like InstanceDiffusion (55) and MIGC 67 need more
time for training and inferencing with more objects. Taking models trained on COCO examples, to generate an image with bounding boxes on V100 GPU, ODGEN takes 10 seconds, ControlNet takes 8 seconds, and InstanceDiffusion takes 30 seconds.
21

---

Apple
Cake
Surfboard
Person
tie
Person
Synthesized Images used in Image Lists
YOLO Synthetic Data YOLO Synthetic Data
Synthesized Images
Figure 11: Detection results of YOLO de- tectors trained w/ or w/o ODGEN syn- thetic data. The synthesized samples of ODGEN help detectors perform better partially occluded objects.
images used to build image lists. Right: corresponding synthesized samples. The syn- thesized images of foreground objects share only the same category with objects synthesized samples but not colors or shapes.
Figure 10: Left:
During inference process, we compare using same offline image library as training against using totally different image library. We get very close results, shown in Tab. 22] Both outperform other methods significantly, as shown in Tab. 4 It indicates that ODGEN can extract category information from synthesized samples of foreground objects in image lists instead of depending on certain images of foreground objects. ODGEN is not constrained by the image library of foreground objects used in training and can be generalized other newly generated offline image libraries consisting novel synthesized samples foreground objects, which ensures that offline image libraries increase ODGEN's computational efficiency without reducing its practicality.
Table 22: FID (1) and mAP (f) of YOLOv5s YOLOv7 for ODGEN trained on COCO. FID is computed with 41k synthetic images. For mAP, YOLO models trained from scratch on 10k synthetic images and validated on 31k real images. Different offline image libraries are applied for inference.
Offline Image Library
FID (L)
mAP@ 50 (f)
mAP@@ ,5:955 (T)
Same training Different from training
16.01
18.90 9970
24.40 / 14.20
16.16
18.60 9.52
24.20/ / 14.10
Quantitative evaluation for concept bleeding. This work mainly focuses on the concept of the bleeding problem of different categories objects. The mAP results of YOLO models trained on synthetic images only in Tab, can be proxy task to prove that the concept bleeding is alleviated since our ODGEN achieves state-of- the-art performance in the layout-image consistency complex scene generation conditioned on bounding boxes. This section adds quantitative evaluation with BLIP- VQA 21), which employs the BLIP 32] model to identify whether the contents in synthesized images consistent with text prompts. The results are provided in Tab. 23 Our ODGEN outperforms other methods and gets results close to ground truth (real images from the COCO validation set sharing the same labels with synthetic images).
ODGEN trained on COCO with Stable Diffusion v1.5. Our ODGEN implemented with Stable Diffusion v in Sec. We add experiments of implementing ODGEN with Stable Diffusion v1.5
22

---

moon
ambulance
mountain
tree
ball
tiger
Figure 12:
Visualized samples containing novel categories of foreground objects generated by ODGEN trained the COCO dataset. Stable Diffusion used to generate images of the foreground objects to build image lists for inference. It shows that our ODGEN can control the layout of novel categories that were never seen its training process.
"An avalar in screen shot of the Apex game"
"An object in a screen shot of the Apex game"
"A screen shot of the Apex game"
"Alellyfihh in a photo of aquariuu"
"A starfish in a photo of aquarium"
"A photo of aquarium"
8
"Traffic lights a traffic scene"
"A vehicle in a traffic scene"
"A traffic scene"
Figure 13: Visualized samples generated by the models fine-tuned both cropped foreground objects and entire images from RF7 datasets (200 images for each dataset). Text prompts are provided under figures. Columns 1-6 are generated foreground objects and columns 7-9 are generated entire images. Models fine-tuned on entire images only cannot synthesize foreground objects required by image lists since they cannot bind the prompts corresponding objects in entire images that may contain multiple categories of objects.
trained COCO for comparison in this part. Results provided in Tab. 24 Our ODGEN achieves similar results with different versions Stable Diffusion models.
Visualization. We provide extensive visualized examples of ODGEN to demonstrate its effectiveness. In Fig. [3] we provide visualized samples produced by models fine-tuned on both cropped foreground objects and entire images. It shows that the fine-tuned models are capable of generating diverse foreground objects and complete scenes. Fig. 14 we show generated samples of specific domains in RF7. It can be seen that ODGEN is robust to complex scenes composed of multiple categories, dense and overlapping objects. Besides, ODGEN shows strong generalization capability to various domains.
23

---

Synthesized MRI Samples
PURLON
negetive MRI image
positive
image'
image
image'
Synthesized Apex Game Samples
ovator
avatar
object screeh the Apex gome
avator
Apex game
Apex gome"
Apex gome
Apex game
Synthesized
Robomaster
Samples
base and shot
rune-ree red shot game'
base and cars
base shot of
cors and rune-blue shot of
shot of game
game
game'
game'
Synthesized
Cotton Samples
G- G-basaall sulcsse
C-
G-arboreum
6-herbaceum photo
G boretum und b-borbadonse
6-herbaceum
G-arboreun in photo cotton
photo
Trame light
Synthesized Road Traflic Samples
cycle
melee light
motorcycle end vehicle treffic
"taaffic lights and vehicles in
fire hydrant in traffic road
cr crosswalks walk and vehicles traffi
motorcycles in traffic
traffic scene
parfin
Synthesized Aquarium Samples
Esh
U4A
Jellyfishes aquarium
shingray and shark
Fissess
fish and shark k photo equarium
puffins
Synthesized Underwater Samples
he
techini and a holothurian
ta starfish and scallop and echinus undorwater
holothurians in underwater
starfishes and a scallop underwater scene
technii and starfish underwater
in underwoter
Figure 14: Additional visualized samples produced by approach in various specific domains. Annotations of bounding boxes are marked on samples and global text prompts are provided below corresponding samples. Our approach can be generalized various domains and is compatible with complex scenes with multiple objects, multiple categories, and bounding box occlusions.
In Fig. Fig. 16 and Fig. we provide supplemental visualized comparisons between ODGEN and the other methods the COCO-2014 dataset. ODGEN achieves the best image-annotation consistency
maintains high generation fidelity. More generated samples of ODGEN are shown in Fig. 18 In addition, also show different examples produced by ODGEN on the same conditions in Fig. to show its capability of generating diverse results.

---

Anmolation
ReCo
GLIGEN
ControlNet
GeoDinussio
MIGC
ODGEN
dining bable
bonil
cup
CE
ball
CE
Nes
person
hot
ponson
bowd
broccoli
Visualized comparison results the COCO-2014 dataset. The annotations used as conditions to synthesize images are shown in the first column.
Figure 15:
Table 23: BLIP-VQA (t) results averaged over 41k images synthesized with different methods using the labels from the COCO validation set.
Method
ReCo
GLIGEN
ControlNet
GeoDiffusion
BLIP- VQA (t)
0.2027
0.2281
0.2461
0.2114
Method
MIGC
InstanceDiffusion
ODGEN
Ground Truth (reference)
BLIP-VQA
0,2314
0.2293
0.2716
0.2745
Table 24: FID (L) and mAP (1) of YOLOv5s / YOLOv7 COCO. FID is computed with 41k
synthetic images. For mAP, YOLO models are trained from scratch on 10k synthetic images and validated 31 real images.
Method
FID (L)
mAP@ 50 (f)
mAP@ 50:95 (?)
ODGEN hased on Stable Diffusion v2.1
16.16
18.90
24,40
9.70 14.20
ODGEN hased Stable Diffusion v1.5
15.93
18.20
24.50
9.39 14.30
25

---

Annotation
ReCo
GLIGEN
ControlNet
GeaDiffusion
MIGC
ODGEN
cell phone
laptop
bencch
Loddy
ponon
boseball Bet
baseball glove
posson
Adl
Figure 16: Visualized comparison results the COCO-2014 dataset. The annotations used as conditions to synthesize images are shown in the first column.
26

---

Annotation
ReCo
GLIGEN
ControlNet
GeaDiffusion
MIGC
ODGEN
clock
suithoard
person
cell phone
porson
book
diair
Figure 17: Visualized comparison results the COCO-2014 dataset. The annotations used as conditions to synthesize images are shown in the first column.
27

---

Annotation
ODGEN
Annotation
ODGEN
Annotation
ODGEN
horse
cake
truck
dining table
person
apple
bird
dining table
bow!
couch
spoon
person
skateboard
banana
bottle
chair
pizza
umbrella
potted plant
dining
dining table
spoon
bowl
sink
cup
remote
keyboard
oven
cup
Figure 18: Visualized results of ODGEN on COCO-2014 dataset. In every grid, the annotation used conditions to synthesize images are shown in the top-left corner.
28

---

Label
Annotation
Seed
Seed
Seed
Seed
laptop
keyboard
chair
DV
couch
spoon
dining table
broccoli
cell phone
bottle
soillet
sink
cup
dining table
motorcycle
cake
bowl
SIE Maraals
teddy hear
Figure 19: Visualized results of our ODGEN on the COCO-2014 dataset generated from the same conditions. The annotation is placed on the first column.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided details of our implementation in both the main paper and the supplementary to ensure reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [No]
Justification: This paper do not include new data. The code is not included yet. We have provided enough details for implementation to ensure reproducibility. We will release the code and models soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
puls www.guiieeeeeeeooo / ss www.noolsssssooooocooooooooolloo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https)
//nips httppppllllllgclp wttp//uideeeeeee ides Ctee//oodSSubm wesessubiisss i onPol enPolicyy icyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Statistical significance of the experiments are not included. The experiments cost large amounts of computational resources. For example. our ODGEN COCO is trained for 14 days with 8 V100 GPUs. Besides, the experiments on both specific and general domains including 8 datasets and two proxy detection models show the same trend. In addition, the improvement of our method is significant compared with prior methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the information of compute resources in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research obeys the code of ethics of NeurlPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of this paper in discussed in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
Justification: We have not designed safeguards yet but our model can share the same safeguards with modern large-scale text-to-image models like Stable Diffusion.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We respect the license and terms of all the assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssww www.osocittcceeeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "mONdYWzpod", "reviewer": "Reviewer_Fyyh", "summary": "It proposes a novel data synthesis pipeline, ODGEN, which uses diffusion models to generate high-quality and controllable datasets for object detection.\nThey first fine-tune the pre-trained diffusion models on both cropped foreground objects and entire images. Next, they control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions.\n\nFor each image, the object number per category, bounding box location and size are sampled from estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts which are fed to ODGEN to generate the image.", "questions": "As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.\n\nRestricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.\n\nODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.\n\nThe computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method addresses a practical problem in machine learning, and new effective techniques are employed in the application of diffusion models to achieve significant better synthesis quality than existing approaches.\n\nFor example, two-step encoding (CLIP text tokenizer -> stacking -> encoder) of the textual condition enables the ControlNet to capture the information of each object with separate encoding and alleviates the concept bleeding problem of multiple categories. Also, encoding individual image lists rather than pasting all objects on a single image can effectively avoid the influence of object occlusions.\n\nThe experimental validation is thorough, covering multiple benchmarks and providing detailed comparisons with baseline methods.\n\nAblation analysis is insightful.", "weaknesses": "As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed.\n\nFor the data scarcity experiment, authors only sample 200 images as the training set for all datasets, which is interesting but not sufficient for the analysis. More extensive evaluation can be done with higher ratios of real:synthetic data for datasets like ApexGame, Robomaster, and Underwater (and even larger datasets like COCO) .\n\nRestricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this.\n\nODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper.\n\nThere are no comparisons provided with the fully supervised approach (models trained on complete real training set) on representative datasets. This would help evaluate the trainability of the proposed approach over data augmentation techniques such as copy-paste.\n\nThe computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.\n\n$\\textbf{1. Encoder architectures:}$ \n\nIn this paper, we change the channel number according to the maximum object numbers that can be found in a single image. For datasets like MRI in which most images contain only one object, we can use fewer channels to make the model more lightweight. For a generic model trained on large-scale datasets, we can set the channel number with a higher enough value that can cover most circumstances. For example, we are working on a model trained on about 3000k images and set the input channels as 150 to make it applicable to images containing at most 50 foreground objects.\n\n$\\textbf{2. More extensive evaluation with larger-scale datasets:}$ \n\nSee the experiments provided in parts 2 and 3 in the global response.\n\n$\\textbf{3. ``Synthetic only\" not performing well:}$\n\nTo the best of our knowledge, the current synthetic data are mostly used to serve as augmentation data instead of training detectors on them only. A similar conclusion was drawn by prior work like GeoDiffusion [a]. Although our approach has achieved improvement in complex scene synthesis, there still exists noticeable gaps between real data and synthetic data. For example, we train ODGEN on 80k images from COCO and synthesize 10k images to train YOLO models from scratch. However, compared with the YOLO trained on 10k real images from scratch, the YOLO model trained on synthetic data only still falls behind (YOLOv7 mAP@.50: trained on 10k synthetic images only 24.40 v.s. trained on 10k real images only 53.20, same labels applied). We look forward to further improvement in future work with more powerful generative models and control methods to narrow the gaps. Besides, as illustrated in the limitations part (Suppl. A), the current dataset synthesis pipeline (Fig. 3 in our paper) is not fully optimized and cannot reproduce the distributions of foreground objects completely, which may also lead to worse performance of training on synthetic data only on RF7 datasets (labels used for COCO experiments are directly obtained from real images). \n\n$\\textbf{4. FID and mAP:}$ \n\nFID is used to evaluate the distance of distributions between generated samples and real samples by extracting features from images and computing the mean and covariance values. However, it's hard for FID to reflect the condition(layout)-image consistency, which is required by the object detection task and can be reflected by mAP results. Taking Fig. 5 in our paper as an example, ReCo failed to generate the bed within the orange box or generate cups within the purple boxes. Similar misalignment phenomena can be found in Fig. 10, 11, and 12 as well. It will hurt the detector's training and make ReCo perform much worse on trainability (in terms of mAP) than our ODGEN.\n\n$\\textbf{5. Comparison with the fully supervised approach:}$ \n\nWe presented fully supervised experiments with training on real data only as the baseline on RF7 datasets in Tab. 2 of our paper. Similar experiments on the COCO dataset are added to part 2 of the global response. The copy-paste method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets employed in this paper. Besides, our ODGEN can be implemented without segmentation masks. \n\n$\\textbf{6. Computational complexity and training time:}$ \n\nTaking the model for the COCO dataset as an example, our ODGEN shares a very close scale of parameters with ControlNet (Trainable parameters: ODGEN 1231M v.s. ControlNet 1229M. parameters in the UNet of Stable Diffusion are included). We provide the training time for 1 epoch on COCO with 8 V100 GPUs of different methods in the Table below (the training code of MIGC is not open-source and thus not included here):\n\nMethod | ReCo | GLIGEN | ControlNet | GeoDiffusion | ODGEN (ours)\n----------|---------|-------|----------|---------|----------\nTraining time (hrs) | 3 | 7 | 4.5 | 3.2 | 5.6\n\n[a] GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation, ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mbv0NxZqDt", "reviewer": "Reviewer_N8et", "summary": "This paper proposes the ODGEN method to generate high-quality images conditioned on bounding boxes. ODGEN fine-tunes Stable Diffusion (SD) on domain-specific datasets to enhance image quality in specialist domains, designing a novel strategy to control SD with object-wise text prompts and synthetic visual conditions to alleviate 'concept bleeding'. Experimental results on the COCO-2014 dataset demonstrate that ODGEN surpasses other methods in control capability. Additionally, the authors design a dataset synthesis pipeline using ODGEN, showing that using additional generated training data can improve the performance of state-of-the-art (SOTA) object detectors.", "questions": "- When fine-tuning the pre-trained diffusion model, the authors use not only the entire images from the dataset but also the crops of foreground objects (i.e., resized to 512 x 512). I'm curious if this approach would help in generating particularly small objects.\n\n- During actual inference, what is the visual relationship between the image list $c_{il}$ (i.e., in Eqn. 2) and the final generated image? The authors should provide some visual examples to illustrate this.\n\n- The authors mention that the object-wise conditioning used in the paper can alleviate \"concept bleeding\" in multi-instance generation. I believe it is necessary to conduct experiments on the COCO-MIG benchmark [65], which assesses the model's ability to control both positioning and attributes, to elaborate on the attribute binding capability of the ODGEN.\n\n    [65] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR'24\n\n- Compared to GLIGEN and MIGC, ODGEN achieves better results in overlapping cases. Which module's design primarily contributes to this improvement? As this is a major issue in the industry, I hope the authors can emphasize this in the methods section.\n\n- Have the authors considered open-sourcing the code and model? I believe this would greatly benefit the community.", "limitations": "- The proposed method requires fine-tuning the entire Stable Diffusion model and training an additional ControlNet. This often requires significant computational resources, making reproduction challenging. \n- I hope the authors can open-source the corresponding model and code, as this would greatly benefit the community.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The ODGEN method proposed in this paper outperforms previous methods and shows significant improvements.\n+ This paper is well organized and written. The overall paper is easy to follow.", "weaknesses": "There are some unclear implementation details, and a few experiments are missing. Please refer the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.\n\n$\\textbf{1. Fine-tuning on foreground objects:}$\n\nBoth the model fine-tuning and the object-wise conditioning module are designed to enhance foreground object generation but not limited to small objects. These two parts are conducted in sequence and contribute to synthesizing both large and small objects together. Given an object detection dataset, we fine-tune Stable Diffusion with the proposed approach to make it capable of synthesizing all kinds of foreground objects and provide visual prompts for the training of the object-wise conditional module. The image lists in the object-wise conditioning module provide the information of category and position for foreground objects. As shown by the visualized samples and quantitative results, our approach obtains improvement in generating dense and small objects compared with other methods and achieves better layout-image consistency. \n\n$\\textbf{2. Visual relationship between image lists and synthesized images:}$ \n\nThe image list is designed to provide the information of category and localization for the control of foreground object layouts. The synthesized images share the same category with objects in generated images and are pasted to the same position as the objects in generated images to build image lists. The objects in image lists and objects in synthesized images may all be apples or cakes but have different shapes and colors. More visualized samples are provided in Fig. 2 of the PDF file enclosed in the global response.\n\n$\\textbf{3. Attribute binding:}$ \n\nOur approach is designed to synthesize object detection datasets that focus on the categories of foreground objects. As a result, the attribute binding is not in the scope now. Our work focuses on the concept bleeding problem of categories while MIGC pays additional attention to attributes. Taking the right part of Fig. 6 in our paper as an example, our ODGEN fixes the concept bleeding problem of categories and generates the concepts of motorcycle and vehicle correctly. We consider to address the challenge of attribute binding in future work to obtain ODGEN models with powerful attribute binding capability by training on datasets containing annotations of attributes (e.g., Visual Genome [a]). We will also highlight the difference between ODGEN and MIGC in the revised manuscript.\n\n$\\textbf{4. Overlapping objects:}$ \n\nThe image list in the object-wise conditioning module contributes to the improvements in object occlusion. Our ODGEN introduces guidance for objects with synthesized visual prompts. We employ an image list to paste the visual prompt for each object separately on different empty canvases. As a result, ODGEN has access to the localization of overlapping objects with visual knowledge in the image list. We ablate this design in the ablations part (the middle part of Fig. 6 in our paper) to show that it contributes to the synthesis of overlapping objects. Experiments also show that our method contributes to the improvement in complex scene synthesis compared with prior works. We will emphasize this part in the method section of the revised manuscript.\n\n$\\textbf{5. Open-source:}$ \n\nWe have plans to open-source the code and model weights.\n\n[a] Krishna R, Zhu Y, Groth O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 2017, 123: 32-73.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sY9UGz0Pn6", "reviewer": "Reviewer_PYXX", "summary": "ODGEN uses a diffusion-based generation model to create novel images to train object detectors. Object bounding boxes along with the object's textual description are given as a conditioning for the generation step. With these generated images they can improve the detector's performance.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed generation pipeline can be used to enrich the training dataset with more high-quality images. Since multiple objects can exist in the same image, using bounding boxes along with the textual description seems to be the better approach.  \n\nThe paper is well written and the evaluation is done with standard datasets. The improvement over the baselines shows the efficacy of the approach.", "weaknesses": "1. There is a high generation cost when there are multiple objects in the same image, which reduces the practicality of the proposed approach. \n2. It would be interesting to see if the generated images improve the detector's performance where it originally failed due to lack of data, such as when objects are partially occluded.  \n3. How are the statistics of the object layout taken? \n4. Why only 5k images were generated? Can we see some study of gradually adding synthetic data, like 1k,5k,10k,20k, and some plateauing of the detector performance?\n4. It will be only fair to compare with other generation approaches with the same post-generation filtering being applied to them. How does ODGEN compare then?\n4. In line 111, it is misleading to say \"a new method to fine-tune the diffusion model\", as it is nothing more than just fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.\n\n$\\textbf{1. Generation cost:}$\n\nBefore training and inference, we can generate an offline library of foreground objects to accelerate the process of building image lists. With the offline library, we can randomly pick images from it to build image lists instead of synthesizing new images of foreground objects every time. In the generation stage (Fig. 2c in our paper), our approach pads both the image and text lists to a fixed length. Therefore, the computational cost for inference with offline libraries doesn't increase with more foreground objects. Other methods like InstanceDiffusion [a] and MIGC [b] need more time for training and inferencing with more objects. Taking the models trained on COCO as an example, to generate an image with 6 bounding boxes on a V100 GPU, ODGEN takes 10 seconds, ControlNet takes 8 seconds, and Instance Diffusion takes 30 seconds.\n\nDuring the inference, we compare using the same offline image library as training against using a totally different image library. We get very close results as shown in the table below (mAP metrics are provided in the YOLOv5s/YOLOv7 format):\n\nOffline Image Library | FID ($\\downarrow$) | mAP@.50 ($\\uparrow$) | mAP@.50:95 ($\\uparrow$)\n-----|-----|-----|-----\nSame as training | 16.01 | 18.90/9.70 | 24.40/14.20\nDifferent from training | 16.16 | 18.60/9.52 | 24.20/14.10\n\nBoth outperform other methods significantly as shown in Tab. 3 of our paper. It indicates that ODGEN is capable of extracting category information from synthesized samples of foreground objects in image lists instead of depending on certain images of foreground objects. ODGEN is not constrained by the image library of foreground objects used in training and can be generalized to other newly generated offline image libraries consisting of novel synthesized samples of foreground objects, which ensures that the use of offline libraries won't reduce the practicality of ODGEN.\n\n$\\textbf{2. Detector performance improvement with synthetic data:}$ \n\nWe provide several visualized detection results of YOLO models trained with and without synthetic data in Fig. 3 of the PDF file enclosed in the global response. It shows that the synthetic data helps detectors detect some occluded objects.\n\n$\\textbf{3. Object layout statistics:}$ \n\nAs illustrated in Sec. 3.3 (Line 167-173) in our paper, the statistics of the object layout are taken from the training dataset. For example, ODGEN trained on 200 images takes the statistics from the 200 images.\n\n$\\textbf{4. The number of synthetic images:}$ \n\nWe have added the ablations of the number of training samples (1k, 3k, 5k, and 10k) in Tab. 1 in the PDF file attached to the global response. It shows that using 5k synthetic images gets results close to using 10k synthetic images and outperforms using 1k or 3k synthetic images under our experimental setups.\n\n$\\textbf{5. Post-generation filtering step:}$ \n\nSee the analysis and experiments provided in part 1 of the global response.\n\n$\\textbf{6. Expression:}$ \n\nWe will change the expression to \"We propose to fine-tune the diffusion model with domain-specific images and cropped foreground objects\" in the revised manuscript to clarify the difference between our approach and the fine-tuning method on entire images only.\n\n[a] InstanceDiffusion: Instance-level Control for Image Generation, CVPR 2024\n\n[b] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDlPL1BqTC", "reviewer": "Reviewer_2vbs", "summary": "This work is about generating synthetic (annotated) data for object detection with diffusion models that are tuned for a specific image domain. Generating synthetic annotated data can be useful for model training in situations where data is scarce. The authors propose a specific pipeline and new modules to generate such data. First, an off-the-shelf diffusion model is fine-tuned on a specific domain, for instance a specifc object detection dataset. This is done both with the full images and crops of foreground objects (via the given bounding boxes). The statistics of object categories, number of instances, bounding boxes are estimated. Then, a new layout is sampled from the distribution, the class-specific objects are individually synethesized with the tuned diffusion model. These are individually placed at their sampled bounding box location to build a so-called image list. Then, a ControlNet model is used that encodes this image list along with a text list of the object categories, which outputs the final synthesized image. The newly designed text and image encoders for the controlnet are trained. Finally, a verification step is done at the object level (classifying if the generated image contains the desired object inside a certain bounding box). The synthesized data is evaluated with the FID metric, and also used to train object detectors, which are then evaluated with mAP.", "questions": "- When you fine-tune the diffusion model with driving data, don't you have issues with limited diversity? I understand that you might only be interested in the same label spaces as defined in the dataset. But for real applications, one is often interested in simulating something that is missing in a dataset. For instance, a driving dataset may contain lots of regular cars, vans and trucks. But you also want to generate emergency vehicles like police cars or ambulances.\n- Does the filtering rate of the \"corrupted label filtering\" step correlate with object occlusions? As in, are there more issues (during either the generation or even the classification/filtering) for occluded/overlapping objects?\n- An interesting application of this paper could be to improve language-based object detectors like in [A]. This work relies on GLIGEN and could benefit from ODGEN. The difference would be that the general semantic knowledge of the diffusion model would be leveraged.\n\nReferences:\n- [A] Generating Enhanced Negatives for Training Language-Based Object Detectors. Zhao et al. CVPR'24", "limitations": "The limitations are adequately discussed in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem the paper tries to address is important and has several use cases, maybe even beyond what the paper lists.\n- I especially like the adaptation to specific domains, which is certainly relevant for many real-world use cases.\n- I enjoyed reading the paper, it's well written, has a good flow, it was easy to understand.\n- The results seem to be clearly better than prior works like GLIGEN.", "weaknesses": "- Positioning with respect to some prior work.\n    - In the related work section, the paragraphs on layout-to-image generation & dataset synthesis contain some highly related works (like InstanceDiffusion or others listed as the second group of data synthesis approaches). However, these works are only briefly described but not contrasted against the proposed method. So, what's the difference to InstanceDiffusion, for example?\n    - Why are there no comparisons to InstanceDiffusion in the experiments, or even simpler data augmentation techniques like \"copy-paste\"?\n- The object distributions are simplified compared to real distributions.\n    - Using a Gaussian distribution for a discrete variable seems odd. There will be some probability to sample a negative number and if you clip, you ultimately end up with a different distribution than what was estimated, I guess.\n    - It seems that all estimated distributions regarding the bounding boxes are independent. I assume that location and areas are dependent. Are the distributions modeled independently on purpose? As in, is it better for simulation to have independent distributions? Or is this just done for simplicity? And was the impact of this evaluated empirically?\n- Missing details on the usefulness of synthetic data\n    - For the domain specific domain, it seems that the diffusion models were also tuned only with the 200 selected domain-specific images (according to line 212). I found this information to be crucial in judging whether or not the experiments are valid. I suggest to highlight this aspect already in Section 4.1 - and maybe also discuss why this information is curcial. My thinking here is that if you used more data for tuning the diffusion model than you use to train the detectors, it would be unclear where performance gains come from. It could just come from the additional data used to train the diffusion model, rather than the images being synthetic.\n    - Also, does the same conclusion on object detector performance gains from Table 2 also hold when scaling up real and synthetic images? From a practical point of view, annotating 800 more images with bounding boxes would likely be affordable in most cases. So, I'm missing an experiment like in Table 2, but with more images for both real and synthetic images, e.g., 1000 and 25000 real and synthetic images.\n    - For the general domain, it's great to see in Table 3 that the detector is better when trained on ODGEN data compared to any other synthetic data. However, the real reference points should be (a) the 10k real images and (b) the combination of 10k real and K synthetic images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.\n\n$\\textbf{1. Comparison with InstanceDiffusion:}$\n\nWe didn't include InstanceDiffusion since it consists of an UniFusion module which at least requires bounding box labels and segmentation masks as training data. However, our approach and other methods included for comparison only need bounding box labels. We add an additional comparison by employing the open-source InstanceDiffusion model to generate images following the same setups as Tab. 3 in our paper (mAP results are provided in the YOLOv5s/YOLOv7 format):\n\nMethod | FID($\\downarrow$) | mAP@.50($\\uparrow$) | mAP@.50:95($\\uparrow$)\n--------|--------|--------|--------\nMIGC | 21.82 | 9.54/16.01 | 4.67/8.65\nInstanceDiffusion | 23.29 | 10.00/17.10 | 5.42/10.20\nODGEN (ours) | $\\textbf{16.16}$ | $\\textbf{18.90}$/$\\textbf{24.40}$ | $\\textbf{9.70}$/$\\textbf{14.20}$\n\nWe also include InstanceDiffusion in the experiments mentioned in the second part of the global response (YOLO models trained on 80k real images v.s. YOLO models trained on 80k real images + 20k synthetic images). Our approach outperforms InstanceDiffusion as shown by the results in Tab. 2 in the PDF file attached to the global response.\n\nOther works in the second group of data synthesis approaches: Earlier work like LayoutDiffusion [64] is not implemented with latent diffusion models and thus is not included for comparison. Works [21, 37] are designed for the semantic segmentation task while our ODGEN is designed for the object detection task.\n\n$\\textbf{2. Comparison with copy-paste:}$\n\nThe copy-paste method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets used in this paper. Besides, our approach only requires bounding box labels, which are easier to annotate than masks.\n\n$\\textbf{3. Datasets synthesis pipeline design:}$\n\nWe agree that the current method cannot exactly reproduce the foreground object distributions in the training datasets. The number of objects in an image is sampled from a joint distribution across different categories while the positions and sizes of bounding boxes are independent. The current dataset synthesis pipeline is designed for simplicity and serves as a method to compare the fidelity and trainability of different methods. As illustrated in the limitations part (Suppl. A), the dataset synthesis pipeline has not been fully optimized yet, which would be interesting to explore in future work.\n\n$\\textbf{4. Details of the training data:}$\n\nThe whole training process on domain-specific datasets, including the fine-tuning on both cropped objects and entire images and the training of the object-wise conditioning module, only depends on the 200 images used for the baseline YOLO detectors training. Besides, the foreground/background discriminator used for corrupted label filtering is also trained and evaluated on the 200 images. We will highlight this point in the revised manuscript to make it clearer. Thanks for your valuable advice.\n\n$\\textbf{5. Experiments with more training data:}$ \n\nSee the experiments added in the parts 2 and 3 of the global response.\n\n$\\textbf{6. Experiments for the trainability evaluation on COCO:}$\n\nSee the experiments of training YOLO with \"80k real v.s. 80k real + 20k synthetic\" images added in part 2 of the global response. The current experiments of trainability on COCO (Tab. 3 in our paper) are designed to evaluate the trainability of using synthetic data only. We train YOLO models on synthetic datasets and evaluate them on real COCO data (sampled from the COCO validation set). We find that results of different methods have high degrees of discrimination and our approach achieves improvements by a large margin compared with existing methods.\n\n$\\textbf{7. Generalization to novel categories:}$ \n\nIt is an interesting topic to evaluate the generalization to novel categories not included in the training process. We use ODGEN trained on COCO to synthesize samples containing objects not included in COCO like moon, ambulance, and tiger et al. We show visualized samples in Fig. 4 in the PDF file enclosed in the global response. It shows that ODGEN trained on COCO is capable of controlling the layout of novel categories. However, we find that the generation quality is not stable enough, which may be impacted by the fine-tuning process on the COCO dataset. We are also trying to train a generic ODGEN on about 3000k images covering more than 3000 categories. We hope to get a powerful and generic model covering most common categories in future work. \n\n$\\textbf{8. Corrupted label filtering:}$\n\nThe label filtering step is not applied to experiments on COCO. The filtering rate of the \"corrupted label filtering\" step is fixed for experiments on RF7. The proposed discriminator is only designed to justify foreground and background and doesn't discriminate specific categories. It is designed to filter the boxes in which no objects are synthesized. Even if object A is partially occluded by object B, the discriminator is accessible to the information of object B and still judges it as foreground. Therefore, we don't find apparent performance drops on occluded objects. The accuracy of discriminators on the test sets sampled from the RF7 datasets is over 99% (details are provided in Suppl. D.2, Line 505-509). \n\n$\\textbf{9. Benefiting other works with our ODGEN}$ \n\nIt's good to know that work [A] may be benefited from this paper. We also hope that ODGEN can help and inspire the community in the future.\n\n[21]  Y. Jia, et al. Dginstyle: Domain generalizable semantic segmentation with image diffusion models and stylized semantic control. CVPR 2024 workshop.\n\n[37] D. Peng, et al. Diffusion-based image translation with label guidance for domain adaptive semantic segmentation. ICCV 2023.\n\n[64] G. Zheng, et al. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jntDA0kf0b", "reviewer": "Reviewer_FcFR", "summary": "This paper tackles the task of controllable image generation from bounding boxes and text prompts for training object detectors. To fine-tune diffusion models to specific domains effectively and deal with the challenge of concept bleeding, this paper proposes a method called ODGEN. In the proposed method, the diffusion model is first fine-tuned on both entire images and cropped foreground regions. Then, text lists and image lists are utilized as input for ControlNet to avoid the concept bleeding and generate high-quality images. The experimental results show that the proposed method outperforms the state-of-the-art methods for controllable image generation on seven types of datasets in terms of FID score. In addition, the object detectors trained on the generated images from the proposed method achieve better performance than other methods.", "questions": "See iii) in Weakness.\nHow is the performance of the proposed method and other methods when more real images are available?", "limitations": "The limitations are adequately discussed in Sec. A.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i) The proposed method is quite simple yet effective. Its relative ease of implementation is helpful for computer vision practitioners and future researchers. Also, the task tackled in this paper is of practical benefit.\n\nii) This paper is well-written and easy to follow. The motivation for tackling the task is clearly described in Sec. 1, and related works are well-summarized in Sec. 2. The proposed method is clearly explained at both the idea level and procedure level in Sec. 3.\n\niii) The experiments demonstrate the high performance of the proposed method. The proposed method achieved better FID scores than other controllable image generation methods on seven types of object detection datasets. Additionally, training with synthetic images from the proposed method improved the performance of YOLO-based object detectors significantly.", "weaknesses": "My concerns are mainly about the lack of detailed analysis:\n\nI) This paper claims that fine-tuning with both cropped foreground regions and entire images is one of its contributions. However, an ablation study to confirm the effectiveness of this approach is missing.\n\nii) From Fig. 4, it looks like the corrupted label filtering is performed only for the proposed method (ODGEN). Because the corrupted label filtering is a simple technique and can be applied to other methods as well, the comparisons between ODGEN and other methods not only without filtering but also with filtering should be provided for fair comparisons.\n\niii) This paper provides only the results with 200 real images. I'm curious to know how the performance of the proposed method and other methods change when more number of real images are available for training.\n\niv) In Sec. 1, the concept bleeding is raised as one of the challenges in this task. Although Fig. 6 provides an example, I would like to see a quantitative evaluation to check whether it was effectively addressed by the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.\n\n$\\textbf{1. Ablations for fine-tuning on both cropped foreground objects and entire images:}$\n\nThe fine-tuning process is the basis of the training of the following object-wise conditioning module. The fine-tuning on both cropped foreground objects and entire images makes the fine-tuned models capable of synthesizing the images of foreground objects, which are needed by building image lists for the next step. We provide some visualized samples in Fig. 1 of the PDF file attached to the global response. It shows that the fine-tuned model is capable of generating diverse foreground objects and complete scenes. The training of the object-wise conditioning module is designed to realize layout control based on the fine-tuned model and uses the image lists containing synthesized images of foreground objects as conditions. Therefore, it's hard to conduct an ablation study that fine-tunes the diffusion model without cropped foregrounds. Because the models only fine-tuned on entire images have difficulties in generating foreground objects required by image lists since they cannot bind the prompts to corresponding objects in entire images that may contain multiple categories of objects.\n\n$\\textbf{2. Post-processing step with the corrupted label filtering:}$ \n\nSee the analysis and experiments provided in part 1 of the global response. We provide results of all methods without post-processing to compare the generation capability directly and fairly.\n\n$\\textbf{3. Experiments with a larger number of real images:}$ \n\nSee the experiments provided in the parts 2 and 3 of the global response. We provide experiments with different methods trained on the COCO training set (80k images) and add experiments of our ODGEN trained on 1000 images from the Apex Game and Underwater Object datasets.\n\n$\\textbf{4. Quantitative evaluation of concept bleeding:}$ \n\nThis work mainly focuses on the concept bleeding problem of different categories. The mAP of YOLO models trained on synthetic images only in Tab. 3 of our paper can be a proxy task to prove that concept bleeding is alleviated since our approach achieves state-of-the-art performance in the layout-image consistency of complex scene generation conditioned on bounding boxes. In addition, we add the quantitative evaluation with BLIP-VQA ($\\uparrow$) [a] which employs the BLIP model to identify whether the contents in synthesized samples are consistent with the text prompts. The results are shown in the table below. Our ODGEN outperforms other methods and gets results close to ground truth (real images from the COCO dataset sharing the same labels with synthetic images). The results are averaged over 41k synthetic images following the labels of the COCO validation set.\n\nMethod | ReCo | GLIGEN | ControlNet | GeoDiffusion | MIGC | InstanceDiffusion | ODGEN (ours) | Ground Truth (provided as reference)\n------|------|-------|-------|-------|------|-------|------|------\nBLIP-VQA ($\\uparrow$) | 0.2027 | 0.2281 | 0.2461 | 0.2114 | 0.2314 | 0.2293 | $\\textbf{0.2716}$ | 0.2745 \n\n[a] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu, T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, Advances in Neural Information Processing Systems, vol. 36, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for your efforts in reviewing this paper and providing so many valuable comments. We are glad that all reviewers acknowledge that the performance of our work is better than prior methods on the box-to-image generation task and most reviewers find our paper well-written and easy to follow. Besides, reviewer 2vbs and FcFR point out that the challenge tackled by this paper has many practical use cases, even beyond what this paper lists. We provide detailed responses to resolve reviewers' concerns and will add the discussion and experiments in the rebuttal to the revised manuscript.

We first address several common concerns and provide supplemental figures and tables in the attached PDF:

$\textbf{1. Post-processing with the corrupted label filtering (Reviewer PYXX and FcFR):}$

We design this step to filter some labels when objects are not generated successfully, which may be caused by some unreasonable boxes obtained with the pipeline in Fig. 3 in our paper. For COCO experiments (Tab. 3 in our paper), this step is not applied to any method since we directly use labels from the COCO validation set and hope to synthesize images consistent with the real-world labels. For RF7 experiments (Tab. 2 in our paper), this step is only applied to our ODGEN. For fair comparison, we skip this step on RF7 to compare the generation capability of different methods fairly. We provide results in the Table below:

Table: mAP@.50:.95 ($\\uparrow$) of YOLOv5s/YOLOv7 on RF7. ODGEN (with or without post-processing) leads to greater improvement than other methods on all 7 datasets.
Datasets | Baseline | ReCo | GLIGEN | ControlNet | GeoDiffusion | ODGEN w/o post-processing | ODGEN w/ post-processing
-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|------------------
real + synth #|200 + 0|200 + 5000|200 + 5000|200 + 5000|200 + 5000|200 + 5000|200 + 5000
Apex Game|38.3/47.2|25.0/31.5|24.8/32.5|33.8/42.7|29.2/35.8|$\textbf{39.8}$/$\textbf{52.6}$|$\textbf{39.9}/\textbf{52.6}$
Robomaster|27.2/26.5|18.2/27.9|19.1/25.0|24.4/32.9|18.2/22.6|$\textbf{39.0}$/$\textbf{33.3}$|$\textbf{39.6}/\textbf{34.7}$ 
MRI Image|37.6/27.4|42.7/38.3|32.3/25.9|44.7/37.2| 42.0/38.9|$\textbf{46.1}$/$\textbf{41.5}$|$\textbf{46.1}$/$\textbf{41.5}$ 
Cotton|16.7/20.5|29.3/ 37.5|28.0/39.0|22.6/35.1|30.2/36.0|$\textbf{40.5}$/$\textbf{42.1}$|$\textbf{42.0}$/$\textbf{43.2}$
Road Traffic|35.3/41.0|22.8/29.3|22.2/29.5|22.1/30.5|17.2/29.4|$\textbf{38.2}$/$\textbf{43.2}$|$\textbf{39.2}$/$\textbf{43.8}$ 
Aquarium|30.0/29.6|23.8/34.3|24.1/32.2|18.2/25.6|21.6/30.9|$\textbf{32.0}$/$\textbf{38.4}$|$\textbf{32.2}$/$\textbf{38.5}$
Underwater|16.7/19.4|13.7/15.8|14.9/18.5|15.5/17.8|13.8/17.2|$\textbf{18.9}$/$\textbf{21.6}$|$\textbf{19.2}$/$\textbf{22.0}$

It shows that the corrupted label filtering step only contributes a small part of the improvement. Without this step, our method still outperforms the other methods significantly. Results of the other methods were provided in Tab. 6 of our paper as well. In addition, as illustrated in the limitations part (Suppl. A), the current dataset synthesis pipeline is designed to compare different methods and can be improved further in future work. We don't see this part as the main contribution of our paper.

$\textbf{2. Scaling up on COCO (Reviewer Fyyh and 2vbs):}$

We conduct experiments by adding 20k synthetic images to the 80k training images. We train YOLO models on the COCO training set (80k images) as the baseline and on the same 80k real images + 20k synthetic images generated by different methods for comparison. The COCO validation set contains 41k images, we use the labels of 20k images of them as conditions to generate the synthetic set and use the other 21k real images for evaluation. The results are shown in Tab. 2 in the attached PDF file. It shows that ODGEN improves the mAP@.50:95 by 0.5\% and outperforms the other methods.

$\textbf{3. Scaling up on RF7 (Reviewer Fyyh, 2vbs, and FcFR):}$

Apart from the experiments on COCO provided above, we add additional experiments with 1000 training images from the Apex Game and the Underwater Object dataset here. We provide the results of YOLO models trained on the combination of real and synthetic data as below. We conduct experiments on ODGEN, ReCo, and GeoDiffusion. ReCo and GeoDiffusion may benefit from larger-scale training datasets since they need to fine-tune more parameters in both the UNet in Stable Diffusion and the CLIP text encoder. GLIGEN struggles to adapt to new domains and ControlNet performs worse on layout control than ODGEN. So they are not included in this part (also limited by computational resources and time in rebuttal). The corrupted label filtering step is not used for any method.

Table. mAP@.50/mAP@.50:.95 ($\uparrow$) results of ODGEN trained on larger-scale datasets.
Datasets|Apex|Apex|Apex|Underwater|Underwater|Underwater
------|--------|--------|--------|--------|--------|--------
real + synth #|1000 + 0|1000 + 5000|1000 + 10000|1000 + 0|1000 + 5000|1000 + 10000|
YOLOv5s ODGEN (Ours)|83.2/53.5|$\textbf{83.3}$/$\textbf{53.5}$|$\textbf{83.6}$/$\textbf{53.6}$|55.6/29.2|$\textbf{59.6}$/$\textbf{32.5}$|$\textbf{56.3}$/$\textbf{29.8}$ 
YOLOv5s ReCo|83.2/53.5|78.7/46.9|82.0/46.9|55.6/29.2|55.1/28.4|55.9/29.1
YOLOv5s GeoDiffusion|83.2/53.5|80.0/47.2|82.5/47.5|55.6/29.2|54.2/27.9|54.3/28.0
YOLOv7 ODGEN (Ours)|83.8/55.0|$\textbf{84.4}$/$\textbf{55.2}$|$\textbf{84.0}$/$\textbf{55.0}$|54.6/28.3|$\textbf{58.2}$/$\textbf{29.8}$|$\textbf{62.1}$/$\textbf{31.8}$
YOLOv7 ReCo|83.8/55.0|80.5/50.7|79.2/49.9|54.6/28.3|56.5/28.7|56.4/30.1
YOLOv7 GeoDiffusion|83.8/55.0|81.2/51.0|81.0/50.5|54.6/28.3|57.0/28.9|55.8/28.9

When trained on larger datasets, the baselines (trained on real data only) become stronger but our ODGEN still benefits detectors with synthetic data and outperforms other methods.

Meta-review: The paper receives all positive ratings, where all the reviewers agree that the rebuttal and replies have addressed the major concerns (two reviewers upgraded the rating). Overall, the paper proposes a data generation pipeline via the diffusion model for domain-specific object detection. Comprehensive experiments are provided to validate the effectiveness of the proposed framework. The AC carefully checked the paper, reviews, and rebuttal, and agree with the reviewers about the positive rating. Hence the AC recommends the acceptance decision. It would be highly recommended that the authors include the suggested feedback in the final version (especially the experiments with more real and synthetic data to showcase a broader usage) and release the code to foster the progress in this community.
Decision: Accept (poster)