ImaginaryNet: Learning Object Detectors without Real Images and Annotations
Authors: Minheng Ni
Zitong Huang
Kailai Feng
Wangmeng Zuo
Venue: ICLR 2023 poster (2023)
Keywords: Object detection
Visual synthesis
Generative model

Abstract: Without the demand of training in reality, humans are able of detecting a new category of object simply based on the language description on its visual characteristics. Empowering deep learning with this ability undoubtedly enables the neural network to handle complex vision tasks, e.g., object detection, without collecting and annotating real images. To this end, this paper introduces a novel challenging learning paradigm Imaginary-Supervised Object Detection (ISOD), where neither real images nor manual annotations are allowed for training object detectors. To resolve this challenge, we propose ImaginaryNet, a framework to synthesize images by combining pretrained language model and text-to-image synthesis model. Given a class label, the language model is used to generate a full description of a scene with a target object, and the text-to-image model is deployed to generate a photo-realistic image. With the synthesized images and class labels, weakly supervised object detection can then be leveraged to accomplish ISOD. By gradually introducing real images and manual annotations, ImaginaryNet can collaborate with other supervision settings to further boost detection performance. Experiments show that ImaginaryNet can (i) obtain about 75% performance in ISOD compared with the weakly supervised counterpart of the same backbone trained on real data, (ii) significantly improve the baseline while achieving state-of-the-art or comparable performance by incorporating ImaginaryNet with other supervision settings.  Our code will be publicly available at https://github.com/kodenii/ImaginaryNet.

Published as a conference paper at ICLR 2023
IMAGINARYNET: LEARNING OBJECT DETECTORS WITHOUT REAL IMAGES AND ANNOTATIONS
Minheng Ni, Zitong Huang, Kailai Feng & Wangmeng Zuo* Faculty of Computing Harbin Institute of Technology
{mhni, zthuang, k klfeng)estu. hit. eet.eddu.c.. wmzuo@hit.edd..
ABSTRACT
Without the demand of training in reality. humans are able of detecting new category of object simply based on the language description on its visual char- acteristios. Empowering deep learning with this ability undoubtedly enables the neural network to handle complex vision tasks, e.g., object detection, without col- lecting and annotating real images. To this end, this paper introduces novel challenging learning paradigm Munginay--Superviin where neither real images manual annotations allowed for training object detectors. To resolve this challenge, we propose IMAGINARYNET, a framework synthesize images by combining pretrained language model and text-to-image synthesis model. Given a class label, the language model is used to generate full description of scene with target object, and the text-to-image model is deployed to generate a photo-realistic image. With the synthesized images and class labels, weakly supervised object detection can then be leveraged ac- complish ISOD. By gradually introducing real images and manual annotations, IMAGINARY NET can collaborate with other supervision settings to further boost detection performance. Experiments show that IMAGINARYNET can (i) obtain about 75% performance
Object Detection (ISOD),
ISOD compared with the weakly supervised coun- terpart of the same backbone trained on real data, (ii) significantly improve the baseline while achieving state-of-the of-thh-art or comparable performance by incorpo- rating AMAGNARYYNET with other supervision settings. Our code will be publicly available at http://gghbbbbooooooddmm/mgaggg/
INTRODUCTION
Without the demand of training in reality, humans are able of detecting new category of object simply based on the language description on its visual characteristics. Equipping this ability to deep learning may allow the neural network to handle complex vision tasks, e.g., object detection, without real images and annotations. Recently, we witness the rise of Contrastive Language-Image Pre-training (CLIP) (Radford et al. (021), where general knowledge can learned by pre-training and then be applied to various downstream tasks via zero-shot learning or task-specife fine-tuning. Unlike image classification, object detection more challenging and has a larger gap than pre- training tasks. Several methods, such RegionCLIP (Zhong et al. 2022a), ViLD (Gu et al. 202T), and Detic (Zhou et al., 2022). have been suggested to transfer knowledge from pre-trained CLIP (Radford et al.) 20011) some modules of detection. However, real images and annotations still required for some key modules of the object detectors, such as Region Proposal Network (RPN) or Region of Interest (Rol) heads.
In this work, we aim to raise and answer question: given suitable pre-trained models, can we learn object detectors without real images and manual annotations? To this end, we introduce Object Detection (ISOD), where no real im-
novel learning paradigm, ie., Imaginary-Supervisn ages and manual annotations can be used for training object detection. Fortunately, benefited from the progress in vision- language pre-training, ISOD is practically feasible. Here we propose IMAG- INARYNET, framework to learn object detectors by combining pretrained language model as well
"Corresponding Author.

---

Published as a conference paper at ICLR 2023
text-to-image synthesis models. In particular, the text-to-image synthesis model is adopted to generate photo-realistic images, and the language model can be used to improve the diversity and provide class labels for the synthesized images. Then, ISOD can be conducted by leveraging weakly supervised object detection (WSOD) algorithms on the synthesized images with class labels to learn object detectors. We up strong -liP--baeed model the baseline to verify the effectiveness of IMAGINARYNET, Experiments show that IMAGINARYNEL can outperform the CLiP-based model with a large margin. Moreover, IMAGINARYNET obtains about 75% performance in ISOD com- pared with the weakly supervised model of the same backbone trained on real data, clearly showing the feasibility of learning object detection without any real images and manual annotations.
By gradually introducing real images and manual annotations, IMAGINARYNET can collaborate with other supervision settings to further boost detection performance. performance of existing object detection models may be constrained by the limited amount of train- ing data. As a result, we use IMAGINARYNET as a dataset expansion approach to incorporate with real images and manual annotations. Further experiments show that IMAGINARY significantly improves the performance of the baselines while achieving state-of-the-art or comparable perfor- mance in the supervision setting.
worthy noting that the
To sum up. the contributions of this work are as follows:
We propose IMAGINARYNET, a framework to generate synthesized images as well as su- pervision information for training object detector. To the best of our knowledge, we are among the first work to train deep object detectors solely based on synthesized images.
We propose novel paradigm of object detection, Eunginary-Supervicn (ISOD), where no real images and annotations can be used for training object detectors. We set up the benchmark of ISOD and obtain about 75% performance in ISOD when compared with the WSOD model of the same backbone trained on real data.
Object Detection
By incorporating with real images and manual annotations, ImaginaryNet significantly im- proves the baseline model while achieving state-of-the tar or comparable performance.
2
RELATED WORK
2.1 OBJECT DETECTION
Most supervisd object detection methods (FSOD) (Ren et al. 2013; Redmon et al. 2016 Tian ct al. [211 Carion et al.] (2020) rely on large amount of training data with box-level annotations.
To reduce the annotation costs, some works attempt train a detector with incompletely supervised
training data. For example, reakkl-suupervised object detection (WSOD) (Huang et al. 2022 Dong
al. 2021 Bilen & Vedaldi 2016: Tang et al.; 2017) requires only image-level labels. While semi-
supervised object detection (SSOD) (Liu et al. 2021 Xu et al., 2021 Chen et al. 2022 leverages unlabeled data combining with box-level labeled data. Although these works use relatively less or
weak supervision, all of these works still rely on real images and manual annotations. In this paper, we propose ISOD, where no real images and manual annotations can be used for training object detection, thereby saving the demand for data acquisition and annotation costs.
2.2 SIM2REAL
Sim2real aims to use the simulator engines to simulate images for the model. Some works (Akhyani al.) 2022; Sadeghi & Levine, 2016 Wang et al., 2018) use engines simulate data for differ-
ent vision tasks, such as emotion classification. In object detection, Horváth et al. (2022); Borrego al. (2018: : Tremblay et al.) (2018) use simulated images with 2D or 3D engine, e.g., UF4, for industrial object detection. Because of the large domain gap caused by limitation of simulating engine, Sim2real and subsequent domain adaption methods focus on reducing domain gap. How- ever, with the progress in text-to-image synthesis, domain gap of images generated by such models has been largely reduced. The key problem is how to effectively employ text-to-image synthe- sis model for generating diverse images with proper content and quality. To this end, we propose IMAGINARYNET, which can even archive 75% performance in ISOD compared with the weakly supervised counterpart of the same backbone.
2

---

Published as a conference paper at ICLR 2023
ImaginaryNet
boat bird dog
bottle horse
Sample
A photo of bird
bean
REKEN otee
Syn
Toak mmggo
Riorosortaaae ceneraace
Hood
Dosecationn
g
cat
Prefix
yuarinn wilhellers
A thor df 1e9d
Jartingg will ailles
photo standing with others
CLIP
Proposst
O0
I-N(0.1)
photo bird
Languago
Text-to-lmage Synthesis Model
Representation Generator
Figure 1: Overview of IMAGINARY IMAGINARYNET samples the class label randomly and fulfills it to the prefix template. The language model extends the prefix to complete description. The synthesis model generates imaginary images from random noise based on the description. Pro- posal representations
extracted from imaginary images. IMAGINARYNET optimizes Detection Head with proposal representations and class labels. If real data exists, Detection Head will also be optimized based on representations from real images and manual annotations.
2.3 PRE-TRAINED MODELS
Pre- Pre-trained models have shown effectiveness in many vision tasks including object detection. Some CLIP-based models can transfer knowledge from pre-trained CLIP (Radford al.) (021) model. However, real images and manual annotations are still needed for their RPNs or Rol heads. Visual synthesis models, such as GAN (Goodfellow et al. 2016), StyleGAN (Karras et al., 2019), and ImageBART Esser et al. (2021) aim to generate plausible images. In recent years, witness the rise of text-to-image synthesis models with high quality and capability language controlling, such DALL-E (Ramesh al., 2021], Stable Diffusion (Rombach et al.) 2022), and Imagen (Saharia et al. 2022). However, few studies have been given to only using the knowledge from pre-trained text-to-image synthesis models handle complex vision tasks, e,g., object detection.
METHODOLOGY
3.1 PRELIMINARIES AND PROBLEM FORMULATIONS
Object detection aims find each object with the bounding box by E R defined by [minn : Mmin Tmax Umax] that specifies its top-left comner (Imin, Mmia) and its bottom-right comner (Imax: Vmax)., and class label E C in given image E RCxWxfl WI,, where C, W and H denote the channels, width and height of the image and ht denotes the k-th object, respectively.
( Y train the detector, where I is the real In WSOD, we have real dataset DR image and YR "o..... ......... ufa]]
is the corresponding image-level weak teak-annotation. Class-labeled
annotation ye {0, 1} indicates the presence of least one instance of e-th category and |C] is the number of categories. In SSOD, we have real labeled dataset D = {(T. Y and an un-labeled dataset DR. = {..) The real image I,", corresponding instance-level annotations YF = [(b,, with N elements, and unlabeled image I can be accessed. Here, N the number object instances associated with IR. and b; is the i-th object localization bounding box.
In this paper, propose Imaginary-Supervee Object Detection (ISOD), where no real images and annotations can be accessed.
3.2 IMAGINARY GENERATOR
IMAGINARYNET aims to generate synthesized images as well as supervision information for training object detectors. We divide this into series steps: Prompt Generation, 2. Image Generation, and 3. CLIP Filtering.
Prompt Generation Given a class vocab C, NET samples uniformly to get a class label C. Then, the class name of will be fulfilled into prefix phrase, which is "A photo of
3

---

Published as a conference paper at ICLR 2023
Detection Head
Brunch
Detection Head
Branch
Detrction Head
Teacher Branes
Hangeary
Reftre Strnch
Reftne Sterch
Training Imaginary Student Drrnnhh
Training imaginary only
Training with imagiinary class
The structure of Detection Head. Based on different data settings, we use three types of detection heads. In (a) type of Detection Head, no real images and annotations will participate the training process. In types (b) and (c), real images and annotations will be trained with imaginary data together. In type (c), MII. Branch will be disabled due box supervision exists.
Figure 2:
a", obtain the subject description Pe of the imaginary image. To make the scene of image more specific and diverse, Pe will be the guidance prefix the language model G egg. GPT-2 RRafoord et al. 2019), to imagine the full description of the of image. The language model G which is a transformer-bart
autoregressive network, generates the full description token by token until meeting full stop or exceeding the maximum of length ML Let t, G. be the generated full description of the imaginary image.
Image Generation
IMAGINARYNET further obtains the synthesized image by
I= I= (zlte),
(1)
where 2 (O, 1) is random noise sampled from normal distribution. The text-to-image synthe- sis model GV usually adopts a transformer-bra autoregressive model or diffusion network, and generates imaginary image based on z with the guidance of t, in a step by step manner.
CLIP Filtering The language and text-to-image synthesis model may bring in errors during their generation process. To tackle this problem, following the method of CLIP classification (Radford) al. 2021 the CLIP Filter will find out the most matched text in the group (Po c E C). e.g.. ["A photo of aeroplane". "A photo of bicycle", "A photo of bird", ...). with each image to do classification. If the class of the image classified by CLIP is different from its initial class used for generation, we consider this an eπrоr sample and re-generate it from Prompt Generation.
3.3 DETECTION NETWORK
Representation Generator For real image or an imaginary image I, IMAGINARYNET obtains its feature map h = (I), where E' is the image encoder consisting of multi-layerss of convolutional networks. Simultancously, proposals P = R(I) of the image I are extracted by proposal generator R, which adopts selective search module for ISOD) and WSOD for SSOD. IMAGINARYNET applies Rol Pooling (Ren et al. 2015 Girshick] 2015) to each proposal p: in feature map obtain the proposal representation f, Rd Consequently, IMAGI-
RPN without initialization
NARYNET obtains a set of imaginary proposal representations FF { which allows Detection Head learn object detection.
Detection Head and Training Objectives We add object detection head at the end of the Representation Generator, which takes the proposals P and corresponding representations JF as input and predicts the bounding box of an object. IMAGINARYNET uses different Detection Head based on different task settings.
In ISOD, because real images and annotations be accessed. learn the detection head fol- lowing the OICR (Tang et al., (017]) method to predict the classification probability of each proposal, Le., = [IC, SOD
only using the dataset DC = [ w> YO)) generated by IMAGINARYNE: NET. Here, IG denotes an image generated by synethsis model, Y 6 ..... S/ indicates the image- level annatannnotaiion. Following OICR, the object detection head consists of a Multiple Instances Learning (MIL) branch and Refinement branch with the structure shown in Fig (2) During the
We set M 15. We tried different M , but significant gain obtained.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of our IMAGINARY NET(SOOD) with baseline methods on PASCAL VOC 2007. We consider two baseline methods, a strong baseline based on unsupervised CLIP model, and an upper-bound baseline based on OICR trained on 5,000 real images with image-level annotations.
MEtHoos ABRO Bike
BIRD
BOAT BOTTLE
Bus CAR CAT CRAIK Cow TABLE DoG HORSE Morta PERSON FLANT Sitter SOBA TRAIN TV MAP
OICR (WSOD)
64.76 18.16 56.59 20.17 70.13
64.42
19.52
20.12
27.42
69.61
36.28
20.32
22.
13.45
IMAUINRYYNE
45,77
39.79 20.33
50.38
15.95
31,93
training phase, all object representations are first fed into the MIL branch. The MIL branch com- bines two parallel classification streams, each of which contains a softmax function. One softmax function is along the classes dimension to output the score matrix x E |||| and the other is along the object dimension to output the score matrix x E ||P|X|C Subsequently, we compute the element- wise product of these two score matrixes and get the x x @ x x indicates the final object-level classification scores. Then we get the e-th class image-level score 8² by sum over all object: 2 ED "iee- We convert the target label into one-hot form and calculate the Binary Cross Entropy (BCE) loss:
C iigu mil
Ecec No log m + (1 - (c) (1 s
To further improve the performance of the detector, we integrate the self-training methodology and construct the Refinement branch, which has a similar structure as the Rol head of the R-CNN-like detector (Ren et al.] 2015). Following Tang et al. (2017), we keep the proposal with the highest classification score from x as the pseudo-ground-trun (PGT), which are used to optimize the Refinement branch by refinement loss Lref. Finally, the learning objective of our object detection head of ISOD is:
L = C + Lref-
(3)
The whole training pipeline of ISOD is shown in Fig. 2(a). Here we take the OICR as the example and it is feasible employ other WSOD methods for the end of ISOD.
In WSOD, by gradually introducing real images and manual annotations, IMAGINARYNET can collaborate with other more supervision setting to boost detection performance. We learn the de- tection head by Y fI; gWSOD) using generated dataset DC = { Y&)} and real dataset DR = { YB)]. During training, both real images and synthesized images are fed into the Representation Generator and object detection head in order and optimize these two modules via back-propagation. And we only need to slightly modify the training pipeline of object detection head. When real images and image-level annotations are available, as shown in Fig. (bb) we can uniformly sample synthesized and real images and feed them into the WSOD training process.
In SSOD, the real labeled dataset DJ = {(T,, YP)) and real unlabeled dataset D = (TT can be accessed. IMAGINARYNET further generates a dataset DC = { YO) For simplicity, we ignore Y and combine {{IR) and {{ā)) to form the unlabeld dataset D..- We learn the detection head by Y = f[I gSSOD) using the combination of unlabeled dataset Du and real labeled dataset Dl = (CI Y()). When both real images and box-level annotations are available, the detection head consists of a Teacher branch and a Student branch following Unbiased-teacee -eachhertteerrreeeee (Liu et al, 2021). As shown in Fig. (2cc, we simply leverage the teacher-student framework widely adopted in SSOD to train the detection head. The teacher detection network generates pseudo-ground-trute (PGT) for unlabeled real images and imaginary images to train the student detection network. In addition, the student is also trained on real images with box-level ground-truth (GT) by a supervised- learning manner (Ren al. 2015).
3.4 INFERENCE PIPELINE
During inference, the language model, synthesis model and CLIP Filter will be abandoned. Taking a real image as the input, the Representation Generator generates proposal representations, and the Detection Head decodes them to predict the detection bounding box.
5

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of MHAGINAAYNETTWWSOO on PASCAL VOC 2007. All models use 5,000 real data from VOC2007 and MAAGINARYNEI(WSDD leverages extra 5,000 imaginary samples. In comparison, IMAGINARY SSD) achieves ofte-of-theear performance.
MiCHoDS
ALBO Bace
BiKo BoC
BOTTLE
CAN
CHAIR
Cow
TxBLs
MorOk
PARSOX
Sora
TRAIN
MAP
WASDUN OACR
29.41
31-4K
12.1
55.15
42.82
42.03
10.00
15-12
34.52
58.34
16:11
55.60
9:54
14:22
10:21
40 56 70
14.22
54.97 55 32
47.64
29.25
24.94
00:32
54-76
1600
18.16
56 59
20.1
70.15
50.03
64.42
19.52
20.12
40.14
37
68
52.77
41:03
63
IHAGINAYNNET WSDDO
41
16.25
19.77
25.45
50.96
40.56
54,00
44.00
5L12
19:71
16:35
15.00:15:115
INNomARrNar + CICK
56-51
54.87
28.25
22-33
6691
52-35
59:06
18.09
56.49
52
62011
59:12
64:10
57.36
25.05
16.17
14
06.64
61:71
51:30(+3477 (+1.47)
INsuNARYNar W2N
74.30 K1.31
66.36
3432
57,89
75.76
79.33
15.50
32
13-34
46:56
81.42
75,09
75.19
42.82
31.48
59.21
62:81
$1.22
73,97
65,00 (+8.10
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We use GPT-2 (Radford et al. 2019 as the language model and DALLE-mini, which can better follow the language guidance, as the text-to-image synthesis model. We implement the image coder with ResNet50 pretrained on ImageNet dataset. For Proposal Generator, we use Selective Search following W2N (Huang et al. 2022) for obtaining no real images and manual annotations. All training hyper-parameters follow OICR Tang et al. (017) and W2N (Huang et al.) 2022) for a fair comparison. Unless otherwise specified, generate 5,000 imaginary images during training, which has the similar amount of images in comparison to PASCAL VOC2007 trainval set.
4.2 EXPERIMENTAL SETUP
We first compare IMAGINARYNET with the ISOD model to verify whether it is feasible to learn object detectors without real images and manual annotations. To the best of our knowledge, we are among the first to investigate ISOD solely based on synthesized images. So we set up strong baseline based on the CLIP model. For CLIP baseline, we use the Edge Boxes (Zitnick & Dollár, 2014 algorithm to extract potential proposals on VOC test images for CLIP to classify its class, based on its similarity score with pre-designed prompts. Before computing AP, we apply the Non- Maximum Suppression (NMS) operation to remove redundant proposals based on CLIP scores. See Appendix A.1.I for more details.
We also assess EMAGINARYNET by collaborating with WSOD method W2N (Huang et al. 2022 Le., MATINARYNET(WSODD For comparison, we consider WSDDN (Bilen & Vedaldi 2016), OICR (Tang et al. 2017) and W2N (Huang et al., 2022) on real images as the baseline detection networks. Compared with them, IMAGINARY NET in ISOD, i.e., IMAGINARYNET(SOD), adopts similar training pipeline but does not require real images and manual annotations, whereas IMAGI-
NARYNET(WSOD) leverages both real and synthesized images.
4.3
OVERALL RESULTS
4.3.1
IMAGINARY-SUPERVISE OBJECT DETECTION
As shown in Tab. I IMAGINARYNET(SSOD) achieves an mAp of 35.43. This verifies the feasibility of learning object detectors without real images and annotations. Moreover, IMAGI- NARYNET(ISOD) outperforms the CLIP baseline model with a gain of 14.59 in mAP. This fur- ther supports that the competitiveness of pre-trained generative model against pre-trained con- trastive alignment models (e.g., CLIP) for learning object detectors. For most classes, IMAGI- NARYNET(SOD) can obtain a significant performance gain, and IMAGINARYNET(IODD) has much fewer parameter amounts and computational cost in inference because the pre-trained modules can be abandoned after training. Especially, for some classes such cat, the performance improved from 25.92 of CLIP to 60.79 of IMAGINARY The results clearly show the feasibility of IMAGINARYNETTISOD i.e., learning object detectors without real images and manual annotations.
We further compare IMAGINARYNET(SSDD with an upper-bound baseline, WSOD on real images with class-level annotations. For fair comparison, we only generate 5,000 imaginary samples during training. Compared with OICR, which has the same backbone with ours and trained on 5,000 real images and manual annotations WSOD setting, we achieved about 75% performance of it.
We tried Selective Search but Edge Boxes is more effective, so we report the best results.

---

Published as a conference paper at ICLR 2023
OVERALL RESULTS OF TOTAL IMAGINARY SAMPLES
40
35.9
34,97 35,82 36.28
35
33.86
30.09
30
25
Emeginuty Sample Real Sample
20
IK
2K
3K
10K 15K 20K
Figure 3: Visualization of extracted features. We can observe that features can cluster based on the object type. This shows imaginary data contains similar knowledge like real data.
Figure 4: Overall results of different total imaginary samples. The performance improves steadily with the growth of imaginary samples.
photo horse field a chair, which Tuesdoy aftenoonn
photo aeroplane, big big red plane
photo of dog pain
covered by a thin layer of
Figure 5: Visualization of imaginary images. Language model extends the prefix to a full descrip- tion of the scene and the synthesis model generated images followed the description.
Table 3: Comparison of method with real data PASCAL VOC 2007. Imaginary data
can play a similar role in enhancing performance. 10K Imaginary data can improve the model by similar margin compared with 10K real data. Moreover, introducing imaginary data when extra real data exists can further improve the performance.
METHODS
ANNOTATED REAL DATA
ANNOOTATD REAL DATA
UN-REAL DATA
MAP
FASTERRCNN
5K VOC2007
76.00
UNBIASED TEACHER
5K VOC2007
10K VOC2012
81.08 (+5.08)
IMAGINARYNETYSSOD)
SK VOC2007
d
SK IMAGINARY
80.36 (+4.36)
IMAGINARYNETTSSOD)
5K VOC2007
6
10K IMAGINARY
80.60 (44.60) 60)
(MASISARYNETYSSOD)
5K VOC2007
1OK VOC2012
1OK
81.60 (+5.60)
Considering that synthesized images and image-level annotations are free for IMAGINARYNET, our method has the potential to further improve detection accuracy by increasing training samples.
MSCOCO Results We also conduct the experiments on MSCOCO dataset, and obtain similar observations as VOC2007. See Appendix A.2. for more details.
4.3.2 INCORPORATING WITH WSOD ON REAL IMAGES AND MANUAL ANNOTATIONS
We further show that the synthesized images by IMAGINARYNET can collaborate with other super- vision settings real images and manual annotations for boosting detection performance. Here use WSOD as an example. As given in Tab. 2 IMAGINARYNET(WSOD has achieved otate-of- the-art performance in WSOD. This shows the effectiveness of NMAGINARYNET object detectors without real images and annotations, but also is effective in improving the perfor- mance of backbones without increasing parameters and inference cost. TMAGINARYNETWSSOD obtains an mAP of 65.05, which is higher than the state-of-the-art WSOD models.
not only learn
4.4 COMPARISON WITH REAL DATA
How effective IMAGINARYNET in comparison with real data? To answer this question, we con- duct comparison experiments in SSOD setting. We use the same backbone, RESNET5O-FFD as the supervised model FANTERRCNN as well as the SSOD model UNBIASED TEACHER For faimess, we use the same backbone, hyper--arameters, and real data by following the competing methods.
"thhough the performance in its original paper is lower, obtain 81.08 re-training the model.

---

Published as a conference paper at ICLR 2023
Table 4: Comparison of different imaginary samples in ISOD on PASCAL VOC 2007. We set the different scales of samples and report their mAP. The mAP increases with the growth of samples.
ATRO
Rus
CAR
CAT
CHAIR
TWRLS.
HoRsE
PLANT
SOTA
TRAIN
TV
41.26
19
13.62
23.
33.98
17.21
Figure 6: Visualization of detection results on PASCAL VOC 2007 test split. Results show that it is feasible to learn object detectors via IMAGINAYNNETSSOD.
As shown in Tab. S comparing with FASTER-RCNN and UNBIASED TEACHER, IMAGI- NARYNET(SSOD) has achieved comparable performance, about 99.4% performance of UNBIASED TEACHER. This shows that generated data are almost as effective as real data but have much lower cost in data acquisition and higher controllability in class balance. Compared to the baseline UN- BIASED TEACHER, we can observe that generated data are orthogonal with real data, where we can improve the performance by increasing both real and synthesized data.
4.5 VISUALIZATION
4.5.1
EXTRACTED FEATURES
To verify whether the features extracted from imaginary images contain similar information real images, we extracted their features from the same Representation Generator. For real images, we extract 10 features per class as anchors and 10 features per class from imaginary images. We use T-SNE to project them to 2D image.
In Fig. 3 we can observe that features one class from imaginary images are coincident with the feature cluster of the same class from real images. This indicates that features from imaginary images contain similar information in comparison with real images, which explains why IMAGI- NARYNET can learn object detectors even without real images and annotations.
4.5.2 IMAGINARY IMAGES
We randomly sample imaginary images from the synthesis model and corresponding text descrip- tions from the language model to observe how they can play the role of real data. Fig. 5 we can find that text descriptions are quite diverse. Meanwhile, high-quality images from the text-to-image synthesis model follow the description well and contain vivid objects of the corresponding class. Although all images generated are not in real datasets, IMAGINARYNET successfully simulates the scene of the images from object detection datasets. This may explain why IMAGINARYNET can enhance object detectors under settings that contain real images and annotations.
4.5.3
CASE STUDIES
To better reveal the quality of IMAGINARYNET, visualize detection results from VOC2007 test split in ISOD setting. In Fig. 6 many objects are detected correctly. In some hard cases, e.g., the 5-th image which contains many drivers and motors overlapping together, also be detected well with nearly-correct boxes and classes. This proves the effectiveness of IMAGINARYNET in ISOD.
still find some failure cases. For example, in Fig. 6 some motorbikes are not detected well and persons are classified into the wrong class. This is because ISOD is very strict setting, where the model cannot access any
Although AMAGINARYNET shows pretty effectiveness in many
8

---

Published as a conference paper at ICLR 2023
Table 5: Ablation of components in ISOD on PASCAL VOC 2007. We can observe that the
language model improve the results of the model significantly.
Microoos
Blnp BOAT BOTTLE Bus Csa
CHAIR Cow TABLE Doc Hoase Morox
PuAsON
PLANT
Suaar
Sora
TRAIN TV
IMAGAINARY
48.37
42.84
16.14
18:35
46.77 44,57
13:00 79 30.33 52.05 58.38
50.55
22.45
18:55
38.66
40:44
51.95
20 36433 33 36 33:33 15.99 47.54.31.02
who CLIP FiLre
47.18
43-13
35.53
5,33
19:35
32 5M
43
66.73
12:03
43.98 18.13 03 93
36.71
$1.68
19.04
1647
34.13
42.74
04
wio LANGUAGE Moou, 31.32 49.54
32,13
3.24
23.78
43.35
34.86
30.95
13.35
38.99
15:77
38.51 40.82
42.34
15.44
18.39
42.11
38.86
real images and annotations. Nonetheless, IMA still shows that it is feasible learn object detectors in this hard setting. We further analyze this in Appendix
A.3 A.3
5 ABLATION STUDIES
5.1 INFLUENCE OF TOTAL IMAGINARY SAMPLES
Can the model obtain higher performance if IMAGINARYNET imagine more samples? To figure out this, we conduct experiments on ISOD with different scales of imaginary samples. In Tab. and Fig. 4 one can see that the overall performance, mAP, increases steadily to 36.28 finally. The result shows that more imaginary samples can make the detector stronger. Meanwhile, IMAGINARYNET obtains mAP of 25.39 with 1K samples only. This shows the effectiveness of imaginary samples, which train the detector with very limited samples. We also notice the fluctuation of several classes, which may be explained by the instability of class-level annotation learning. In general, more samples will improve the final performance.
5.2 EFFECTIVENESS OF COMPONENTS
Whether the various scene descriptions and error samples affect detector learning? To figure out this question, re-train an IMAGINARYNET without the language model and CLIP Filter. In Tab. 5] we can observe a significant drop in mAP when no language model and CLIP Filter existing. Without CLIP filter, the average performance drops significantly. This show the that error samples may affect the final results. Without language model, the performance of each class also drops in most cases. Such class, like cat, drops over 30 of AP50. The result indicates that language model is also important in generating diverse images. We also notice the unexpected drop in some classes and we believe this can be explained by the failure of the language model to generate a scene similar to the real dataset in some cases or that CLIP reduce the diversity of generated data. We will leave this as future work to generate images that are more suitable for learning detectors.
BROADER IMPACT
IMAGINARYNET is an effective framework to learn detectors without real images and annotations for further improving the performance with real data. It can provide new insight for handling com- plex vision tasks under low-resource regime. For example, it is possible to learn detection or seg- mentation models without any real data. However, the performance of IMAGINARY NET((SDD is still inferior to WSOD and FSOD. Thus, more studies are required to generate diverse and photo- realistic images and to better extract supervision information from the language descriptions and synthesized images. We discuss the limitations in Appendix A.5
7 CONCLUSION
In this paper, we proposed a novel learning paradigm of object detection, Imaginary-Supervise Object Detection (ISOD), where no real images and manual annotations can be used for training object detectors. In particular, images as well as supervision information. Experiments shows that IMAGINARY can obtain about 75% of performance in comparison with the WSOD counterpart trained using real data and image-level annotations. Moreover, IMAGINARYNET can collaborate with other supervision settings real data to further boost detection performance.
presented an IMAGINARYNE: framework to generate synthesized

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work supported by the National Key RD Program of China under Grant No. 121000
REFERENCES
Saba Akhyani, Mehryar Abbasi Boroujeni, Mo Chen, and Angelica Lim. Towards inclusive hri: Using sim2real address underrepresentation in emotion expression recognition. arXiv preprint arXiv:2208.07472. 2022.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2846-2854, 2016.
Joao Borrego, Rui Figuciredo, Atabak Dehban, Plinio Moreno, Alexandre Bernardino, and José Santos-Victon A generic visual perception domain randomisation framework for gazebo. In 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp- 237-242. IEEE, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229 Springer, 2020.
Binbin Chen, Weijie Chen, Shicai Yang, Yunyi Xuan, Jie Song. Di Xie, Shiliang Pu, Mingli Song, and Yucting Zhuang. Label matching semi-supervised object detection. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14381-14390, 2022.
Jaemin Cho, Jie Lei. Hao Tan, and Mohit Bansal. Unifying vision-and-lanne lan-lage tasks via text generation. International Conference on Machine Learning, pp. 1931- 19942. PMLR, 2021.
Bowen Dong, Zitong Huang, Yuelin Guo, Qilong Wang, Zhenxing Niu, and Wangmeng Zuo. Boost- ing weakly supervised object detection via learning bounding box adjusters. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2876-2885, 2021.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Infor- mation Processing Systems, 34:3518-3532, 2021.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2021.
Dániel Horváth, Gábor Erdös, Zoltán Istenes, TomáK Horváth, and Sándor Földi. Object detection using sim2real domain randomization for robotic applications. arXiv preprint arXiv:2208.04111 2022.
Zitong Huang, Yiping Bao, Bowen Dong, Erjin Zhou, and Wangmeng Zuo. W2n: Switching from weak supervision to noisy supervision for object detection. arXiv preprint arXiv:2207. 12104, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEFF/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Yen-Cheng Liu, Chih-Yar Ma, Zijian He. Chia-Waa Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2 09480, 2021.
10

---

Published as a conference paper at ICLR 2023
Teli Ma, Shijie Geng, Mengmeng Wang. Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tailed recognition baseline via vasion-language model. arXiv preprint arXi::2111.14745 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Fereshtch Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. 2016.
arXiv preprint
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / 1487, 2022.
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2843-2851, 2017.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CVF international conference computer vision, pp- 9627-9636, 2019.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 969-977, 2018
Yu- Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- 2018
tion, pp.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang. Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3060-3069, 2021.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao. Han Hu, and Xiang Bai. A simple baseline for open- vocabulayyy semantic segmentation with pre-trained vision- language model. In European Conference on Computer Vision, pp. 736-753. Springer, 2022.

---

Published as a conference paper at ICLR 2023
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liumian Harold Li, Lu- owei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-basce language- image pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16793-16803, June 2022a.
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region- based language-image pretraining. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16793-16803, 2022b.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.022 2200::2200.02005, 2022.
C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In Euro- pean conference on computer vision, pp. 391-405. Springer, 2014.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
A.1.1 CLIP BASELINE
We use ViT-B/16 as the image encoder for CLIP and use Edge Boxes to extract 1000 proposals for each test image. Each proposal will be transformed to 384 x 384 to fit the CLIP input size. Then each proposal will be calculated similarity with the prompt which consists of the prefix "a photo of" and the class name. We select the highest CLIP similarity as the class of the proposal. Before computing AP, we apply the Non-Maximum Suppression (NMS) operation to remove redundant proposals according to their CLIP scores.
A.1.2 CLASS SELECTING IN WSOD
To obtain a better performance in WSOD, the model trained with selected classes of Imaginary samples instead of all Imaginary samples. We first trained the model with all imaginary samples and evaluate the performance of each class in val split. We selected classes that the performance is higher than baseline model. Then we re-trained the model with imaginary samples of selected classes to obtain the best performance. As the results shown Tab. g we can observe that the IMAGINARYNET obtains better performance even without class selecting. Combining with class selecting, we obtain the best performance. We explained this as the gap among Imaginary samples and real samples in some classes, such as boat or train.
Table 6: Comparison of data used in WSOD PASCAL VOC 2007. We can observe that generating selected class improve the performance of IMAGINARYNET significantly.
Me mittiss
CLaSS Stranc DAU Anso Bink Dato Bose Doven Bus CAR CAr CHoc Cov Tura Deu Boxce Monos Passus Plase Sansur Sons TEXIN TV MAP
GCTR
PAAT 16.89 70 78,18 6641 6143
19.87
20.13 d0 37 R8A1 53
IessIAAee Ner OCTR
62 69 81 26.68 55 95 15 51 14
61
63 16
16.75
73
58.68 56.55 13.4 26 da 19 50.19 56.44 51.19
DRAGINAT OCIR
58.51
20.35 27.35 $4.91 60.50
53.01 59.32
64.00
23.63
48.17
A.2 EXTRA COMPARISONS
A.2.1 COMPARISON IMAAGNARY-SUUEERVISDD OBJECT DETECTION ON MSCOCO
Table 7: Comparison of our method in ISOD on MSCOC02014. We obtain a much higher mAP compared with the CLIP baseline. This shows the feasibility of training the detector on harder dataset of 80 classes without any real images and annotations.
METHODS
MAP
CLIP
6.3
IMAGINARY
11.4
We also verified our framework MSCOC02014, which is also common dataset in object detec- tion. As the results in Tab. [7] we outperform the CLIP baseline with nearly two times of mAP. This proves that our framework also effective on the harder dataset, which contains 80 classes.
A.2.2 COMPARISON WITH FINE-TUNED CLIP
Table 8: Comparison with fine-tuned CLIP. IMAGINARYNET still outperforms CLIP even after
fine-tuning CLIP on our generated data,
METHOnS
Areo WEKE BURD Brar Borm Bos
CAR
CNT CHAIR Cow
TARES
Don
House
Moroo
Preson
PLANT
So5s TEAIN
MAP
CLIP
16.30 17.15 19.42
8.80
14:53
30.28
25 83 25.92
20.34
25.00
00
21.55
25:00
20.32
22.59
10.09
13:43
26.3
26.24
27.82
19:26
23.59 20.84
CLIP PINETUNE
18 20.37 15
10
U.T5
33.06
38
37
19
23.79
18 KM
41
29.11
14.11
24,07
29
34
79
21
24.04 21.01
EMAGINARYNET
48.27 34.42 42.84
16.14
50.53
38.66
We think that CLIP cannot use the information from the overlapping proposals because its pre- training is based image-text matching. But the detection head of NT is designed for the training detection task. We conduct the extra comparison by fine-tuning the CLIP model on the same data used for NET
13

---

Published as a conference paper at ICLR 2023
As the Tab. 8 IMAGINARYNET still outperforms CLIP even after fine-tuning CLIP our generated data. But CLIP is still a strong baseline for this task because it uses a large-scale pre-trainin pre-training and can be used to make comparison in ISOD, the strict setting that models cannot access any real images and annotations.
A.2.3 COMPARISON WITH OTHER CLIP- CLIP-BASED DETECTION APPROACHES
We add the extra comparison with these CLIP-based Detection approaches. Due to these models, e.g., RegionCLIP (Zhong al. 2022b), Detic (Zhou et al. 2022), and ViLD (Gu et al. 2021), use different human-annotated real detection data, we adopt the labeled dataset for training each model. For fair comparison, we adopt MAAINNAYNET(SSSDD for leveraging both imaginary data and the same labeled data. We compare results on 20 base class in VOC 2007 test split. For ViLD, we directly provide the results from their paper.
Table 9: Comparison with other clip-based object detection approaches on PASCAL VOC 2007. IMAGINARYNET still obtains the best performance.
METHODS
ANNOTATED REAL DATA
UNANNOTATED REAL DATA UNANNOTATED IMAGINARY DATA MAP
REGIONCLIP
74K COCO-48 165K COCO + 100K LVIS SK VOC 2007 100K LVIS
1.5M IMAGINET
68.35
DETIC
79.44
VILD
78.9
(EASS
5K VOC 2007
5K IMAGINARY
80.36
(MAGINARYNET(SSOD))
5K VOC 2007
10K IMAGINARY
80.60
INAGINARYNEI(SSOD)
5K VOC 2007
10K VOC 2012
10K IMAGINARY
81.60
From the Tab. 9 we can observe IMAGINAYYNET(SSOD) still obtains the best performance. This proves the effectiveness of generated data from IMAGINARYNET.
A.3 FAILURE CASE
From Fig. 8 we can observe that WSOD can better detect objects that are usually accompanying with each other, c.g., rider and horse. This is because we cannot know all the categories appearing in the generation image precisely. For example, contain only a motorbike or a human riding a motorbike. usually, this may not influence the final results because of plenty of training samples. However, for objects accompanying usually, it is easy for the model to consider them as a whole object. This may motivate us explore how control the accompanying in the generation better learn accompanying objects. A modified cross-attentive module for focusing on the key object described in the text may help the model decrease the possibilities of generating accompanying objects
cannot know whether the generated images
two objects are not accompanying
A.4 DISCUSSION
Table 10: Comparison of different prefix on PASVAL VOC 2007. Different prefixes can affect the final performance of detection.
Merions ABRO BiKe BIRD Brxt BoTTL
Bus
CAR
CAT
CHATE
Cow
TABLE
P. DoG
Hoess
Moroe
PrESOs
PLANT
SHEEp
SoFk
TRAIN
TV MAP
Paunx 47.18 43. 35.51 5.33
19.23
32.58
41.23 66.72
12.03
43.98
18.13
43.98
36.71
$1.68
19.04
14.97
34.12
42.74
23.04
33.36
35.25 42.48
Parii 40 77 48 10 35 59
7:35
13
73
45 56
24.24
45.30
13
35.68
15.25
58.59
41 $2
42:24
12,95
16:47
39.64
37.34
24.77
32.30 40.42
Permx 13.22 47
62 16
6.24
21
63
42.10
35
04
3643
11 99
$4.50
14.37
34.77
36.6
44.39
14.14
19.04
38.32
35.41
22. 38
28.44
A.4.1 PREFIX
Prefix is the initial input, which can provide important cues and the language model will extend the prefix to longer sentence. Prefix is used in many zero-shot language generation vision-and- language works (Radford et al. 2021 Wang et al. 2021 Cho et al., 2021). The prefix will provide important cues for language models. Different prefixes will change the results of the language model and the results of the text-to-image synthesis model then as well. It will also affect the final performance of detection.
Many works (Radford et al. 2021 Ma et al. 2021 Xu et al. 2022) prove that some simple prefixes help the multimodal model calculate vision-and-language information better. Following these
14

---

Published as a conference paper at ICLR 2023
IBANK OFP
Figure 7: Composition and extrapolation ability of text-to-image synthesis model. The model can cre- an object with the de- scription only.
Figure 8: Failure case in ISOD. Compared the result of IMAGI- NARYNET in ISOD (left) with the result of OICR in WSOD (right), observe that IMAGNARYNET detect the rider and horse as a whole object. Model of WSOD can better detect objects that are usually accompanying with each other. This may motivate us ex- plore how to control the accompanying in the generation or better learn accompanying objects.
previous works, we consider four representative prefixes as follow: (1) "A photo of a {class}", (2) "A photograph a {class}". (3) "An image of a {class}". and (4) "A {class)".
We conduct the comparison on prefixes in Tab. I0 Each prefix will generate 5K samples for training. To avoid influence from CLIP, which may make the model re-generate images forever when prefix bad, disable CLIP Filter in this experiment.
For the Prefix (1) and (2), the language model can generate the sentence that describes the content of images based on the prefix. Meanwhile, the word in these prefixes, "photo" "photograph", forces the text-to-image synthesis model to generate real-style images like photos. This is also the way for us control the domain gap between the generated images and real images in the dataset.
Prefix (3) obtains a much worse result. We can observe that although the language model still can generate the sentence describing objects well, the text-to- sot-to-immge synthesis model generates many images of cartoon or art style. This leads to a domain gap that makes the model get worse results. Some classes that have significant gap in real and art style, such as cat, drop by huge margin.
Prefix (4) an even worse prefix than (3). Most sentences generated by the language model are not describing the object, e.g., "A person should seek professional assistance to obtain an accommoda- tion.". We did not train our model based on the prefix (4) because it is hard for the text-to-image synthesis model to generate an image based on such a sentence without any object describing.
A.4.2 SYNTHESIS MODEL
Different from class-to-image synthesis models, text-to-image synthesis models have a much stronger capability of composition and extrapolation (Ramesh et al., [021] Rombach et al.) 2022] Saharia et al. 2022). For example, the model may trained with only "black cats" and "white swans", but it can generate "black swans". Meanwhile, text-to-image synthesis models can even create new objects with detailed descriptions. For example, they can generate the image with the de- scription "A photo of blue card on the table with Imm thickness and rounded corner, card numbers in the card with title 'Bank of Card"" (see Fig. 7
Table 11: Comparison of different visual synthesis models in ISOD on PASCAL VOC 2007. We can observe that DALLE-MINI can provide higher performance for IMAGINARYNET
Me Asnu
Bonuu Mus
CHAIR Cow
Mooxe Moroo PeRsoN Suate
STABLE Durusis 35.T1 24.26 13.07
14.95
1.17
34.72
30.76 41.43 9.76 20.65 25.39
36.04
27.37
35.90
10.45
3.78
17.50 36.68 30,89 13.53 23.29 34.12 42.74 23,04 35.36 33.23
DALLE-SINE
47.18 43.13 35.51 5.33
19.23
32.58
41.23 66.72 12,03
48.08
18.13
43.98
36,71
$1.68
19.4
1497
We notice that the generated images are different from different generative models. In this paper, we use two effective open-source visual synthesis models, STABLE DIFFUSION and DALLE- MINI. We conduct comparison ISOD verify the influence of synthesis models. In Tab. II we find that DALLE-MING MINI can provide higher performance for IMAGINARYNET. We explain this as MINL'SMINI'S results contain fewer objects, which easier for the model to learn in the training.
15

---

Published as a conference paper at ICLR 2023
Table 12: Ablation of data augmentation strategies training on PASCAL VOC 2007. We can observe that all augmentation strategies are effective On imaginary samples.
TRAIN TIST ABo. ARRo BIKE BiKD Bost BOTTLS Bus CAR CNT CHAiR Cow TARLE Doo House Morox PexsoN
Plase
Surse
SoPA
TRXIN
TV MAP
$6.05 sa RI 02 65 da $5.55
21:11 56. 23.22 65. 13 67.68 61 24 14.3 15.15 87.37 88
$6.05
$5.65
56.45
61.85
75.04
85.05
16.39 79 82
B5
87.63
79.05
77.51
30.50 31.22
87.83
57.08
58.28
$7.54
59.62
A.4.3 ORTHOGONALITY WITH DATA AUGMENTATIONS
Table 13: The performance of ImaginaryNeet with Active Selection strategy on PASCAL VOC 2007 in ISOD. We can observe significant gain with this loop.
NAAGIRARNNET BAAGINARYNET ACTIVE SELECCION 46.75
40.25 18.04 44.23
16.80
13.62 42.15 21.54 24.75 55.04 36.59
23.35
36.01
41.72 45.01
$3.55
65.25
40.97
22.56
18:35
48.45
46,05
25.85
19.42
14.97
12.56
13.76
47.23
25.67
Whether the data augmentation strategies still work on imaginary images? To answer the question, we conduct several ablation studies with different data augmentation strategies, training augmenta- tion, and testing augmentation. For most WSOD methods, which are also employed into our ISOD setting, no training augmentation needed because it relatively hard learn with class-labeled images. Consequently, we conduct this experiment in SSOD setting. As Tab. T2 shows, we can observe that N obtains better performance after activating training augmen- tation and testing augmentation. These results give strong evidence that our framework is orthogonal with data augmentations. Moreover, orthogonality with data augmentation strategies may imply that imaginary samples have a similar property to real data.
A.4.4 ACTIVE SELECTION
We setup Active Selection strategy by using the detector to find out the samples generated which have the highest loss in a group of samples. We train IMAGINARYNET on 10K data first to initialize it and then continue optimize it with the loop mentioned above on 5K extra samples. The generator will generate 15 samples, and the detector will choose 5K samples with the highest loss. To make a fair comparison, we use an IMAGINARYNET trained with 15K data as the baseline.
From the Tab. 13 we can observe a significant gain by taking into account this Active Sample Selecting strategy. We think the reason for the improvement is similar to the GAN-based network: the goal of the detector is to minimize the loss of the samples, but the goal of the generator is to generate the sample that makes the detector have the highest loss. These adversarial goals make the network more effective.
A.5 LIMITATIONS
We further discuss the limitations and conduct extra ablation studies on the quantity and quality of the generated images from three aspects: (1) sample number, (2) diversity and photo-reality, and (3) generation errors.
A.5.1 SAMPLE NUMBER
As the Tab. 4 one can that IMAGINARYNET obtains the mAP of 36.28 when using 25K gen- erated samples. Nonetheless, the growth of performance has slowed down when the number of generated samples is more than 10K. Thus, we speculate that 36.28 approaches the upper bound of the performance. We note that it may be hard for our method to surpass the method training on full annotated real images. How utilize the generative models effectively for refreshing this upper bound will be question in the future study.
A.5.2 DIVERSITY AND PHOTO-REALITY
The diversity of images generated will help the model learn more knowledge and improve its robust- ness. Diversity is affected by not only the text-to-image synthesis model but also language prompts. This is also the motivation that introduce the language model to the framework to obtain better diversity. Meanwhile, higher photo-reality also helps the model to obtain a better performance due to low domain gap to real images. With the incredible increase in text-to-image synthesis models,
16

---

Published as a conference paper at ICLR 2023
images that are very similar to real photos can be synthesized. Therefore, believe that we obtain images generated with higher and higher photo-reality.
Table 14: Quantification of language and text-to-image generation errors by Clip.
Marnonus
Asao
Bixs
Binto
Boot Borna
Bus
Uas
Can
Caoss
Con
Toars
Doo
Thons Morns Passos PLANT Smasr 5085
Texis
TV
WAP
EaRoa Bucto 2,09
6.56% 11.668 6.96%
9,969 24.809 9.20%
7.50%
17332
13,00%
6.96% 12.80%
9,369 9.28% 36.08% 4.34% 9.12% 1..00
1240%
55.12%
13,79%
A.5.3 GENERATION ERRORS
As the Tab. 14] we find that we have a 13.79% error rate overall and each class has different rate. This limits the performance of IMAGINARYNET. By introducing CLIP Filter in IMAGI- NARYNET, we can increase its performance significantly but the generation errors may still be left some samples.
17


Reviews:
[{"review_id": "wRQ8UzmoKI8", "reviewer": "Reviewer_DNcv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the problem of training an object detector without any real images or annotations. For this end, they propose generating synthetic (realistic) images using a pre-trained text-to-image model conditioned on words for the object categories as interest, and following a weakly-supervised object detection pipeline to train a detector given the generated image and the class labeled that conditioned it. The authors evaluate their approach on PASCAL VOC (and partially on MS COCO) with a comparison to other pre-training based object detectors, weakly-supervised detectors and a supervised method.\n", "summary_of_the_review": "Novel method, strong results. Missing important comparisons to other CLIP-based approaches.", "strength_and_weaknesses": "Strengths:\n+ Annotation is a labor-intensive task and mechanisms for reducing the cost of this process are very valuable.\n\n+ A novel paradigm is introduced for using pre-trained text-to-image models for training object detectors.\n\n+ Strong results over the baseline CLIP-based models.\n\n\nWeaknesses:\n\n1. My only concern with the paper is that it is lacking a comparison with other CLIP-based detection approaches introduced in Introduction.\n\n\nMinor comments:\n\n- \"Humans can easily detect a known concept from language description without the demand of training in reality.\" => I would be very careful with such claims. I kind of disagree with this as there must have been some kind of (pre)training for humans to recognize the concept as known, either the language model, the other objects or the parts of the object/concept etc.\n\n- First paragraph of the Intro: \"RPNs or RoI heads\" => Acronyms are used without introduction.\n- \"which is ”A photo of a”\" => The opening quotation is incorrect. You can use `` in Latex to do this.\n- \"image encoder consisted of multi-layers\" => \"image encoder consisting of multi-layers\".\n- \"a RPN network\" => \"an RPN network\".\n- \"IMAGINARYNET apply RoI Pooling\" => \"IMAGINARYNET applies RoI Pooling\". Please cite the relevant paper here for RoI pooling.\n- \"IMAGINARYNET obtain\" => \"IMAGINARYNET obtains\".\n- \"IMAGINARYNET sample\" => \"IMAGINARYNET samples\". The same problem recurs through out the text. Please check the paper carefully for more.\n- \"OICR(Tang et al., 2017),\" => \"OICR (Tang et al., 2017),\".\n- \"and the structure of them are shown in Fig (2).\" => \"with the structure shown in Fig (2).\"\n- Eq 2, 3: dot should be placed after the eqs.\n- \"the (Tang et al., 2017),\" => \"Tang et al. (2017),\". You can use a different cite command to get Tang et al. (2017).\n- \"for reaching no real images and manual annotations\" => \"for obtaining no real images and manual annotations\".\n- \"the Edge Boxes algorithm\" => Provide citation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dN_8U2jiuED", "reviewer": "Reviewer_w1NM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a framework, named ImaginaryNet, to train object detectors using images generated by a text-to-image model and class labels from a pre-trained language model.\nThis is also a learning scheme, named Imaginary-Supervised Object Detection, for training object detectors without requiring real images and manual annotations. This scheme is based on a weakly-supervised object detection (WSOD) algorithm, i.e. OICR, with the object proposal and class labels obtained from ImaginaryNet.\nThe paper's main contribution is in using the language model and synthesis model to build a learning scheme on top of the existing WSOD scheme without requiring real images or manual annotations.\n", "summary_of_the_review": "My initial rating is a weak reject. The main reason for rejection is that the paper did not put enough focus on the synthesis model. This makes the paper's claim of learning object detectors without real images weaker since there could be an object class that the synthesis model cannot generate without learning it first. In that case, we still need to have real images. For example, learning a \"human hand\" detector would require the synthesis model to generate an image with a \"human hand\".", "strength_and_weaknesses": "Strength:\n1. Using language and synthesis models is a clever idea to improve the existing WSOD algorithm. \n2. Extensive experimental results to demonstrate the strength of the proposed method\n\nWeakness:\n1. The effect of the limitation caused by synthesis models was not well studied and discussed although there is an ablation study on the effectiveness of the language model.\n2. The experiments and ablation studies sections are not well organized and hard to follow. There is some confusion in the settings, e.g. ImaginaryNet ISOD, WSOD, SSOD.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a5XptLgsFS", "reviewer": "Reviewer_va3v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to learn object detectors from text-to-image synthesis data. The idea looks interesting to the community.\nIt first generates sentence description with prefix constraints and GPT-2, then generates imaginary images with DALLE-mini. \nA WSOD is applied on sythensis image. The whole method is named ISOD. Experimental on PASCAL VOC 2007 shows that ISOD achieves close performance to WSOD on real images and annotations, while the performance could be even boosted when gradually increases real images and annotations. ", "summary_of_the_review": "This paper could be viewed as WSOD on image-to-text synthesis datasets. It needs to handle the domain differences between generated images and real images. The idea is overall novel, but the formulation is a bitter weak and the studies also not that comprehensive from my perspective. \n\nI think the idea deserves a top conference, but the paper needs thorough studies to before it could be published. \nI may change my rating based on the rebuttal. \n\n\n==============================\nThe authors give detailed response to all my concerns. I raise the rating from 5=>6. \nHowever, I would seriously suggest the authors make a revision to reformat the story of the work to move some important contents in appendix into the main body. ", "strength_and_weaknesses": "Strong points\n+ The idea looks very interesting to me. \n\nWeak points\n- Some descriptions are not very clear. For instance, \n\n       - what are the prefix in the language models? How different prefix make impacts for the final results? \n\n       - In Table-1, there are some classes (boat, chair, etc) that Imaginary perform worse than CLIP, it lacks explanations. Are these due to the image synthesis errors?\n\n- The idea looks a bit straightforward, but the formulation is a bitter weak. It does not take the language and text-to-image generation errors into considerations. It would be perfect to have a GAN like loop between detectors and generators. \n\n- For illustration part, I would suggest the author to show some cases why ImaginaryNet does not works but WSOD works, so that possible new ideas could be risen from analysis. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xc62lkSmir", "reviewer": "Reviewer_n5tS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a scheme for training object detection networks using self-generated synthetic data.  The system presented herein does not require manually-annotated real images, which are hard to come by.  Rather, the system leverages the spectacular advances in text-to-image generation system to construct visually realistic data that serves the role of traditional object detection datasets.  The proposed scheme is able to benefit from the \"real\" data when available.  The paper also includes a set of experiments that confirm the useful of this approach.", "summary_of_the_review": "The work is interesting and it is moves the field of deep learning in the right direction.  The approach presented here reduces our reliance on manually annotated real data for the task of object detection, which is highly desireable.  At the same time it is somewhat difficult to ascertain what makes this model better than other approaches that also use generated data.  The novelty, I feel, is on the lower-end of the spectrum. ", "strength_and_weaknesses": "The approach presented herein is well-motivated; however, given prior works that also advocate the use of synthetic data for training deep learning model, the novelty-aspect of this work is somewhat low.  Section 2.2, for example, lists some of these works.  Reading this section it seems that existing schemes are not as robust (or useful) for model training simply because the quality, i.e., visual realism, of the generated images do not match \"real\" images.  This subsequently leads to a domain gap, which must be addressed by using manually-annotated real images.  If this is true then one may reasonable assume that with our increased ability to generated visual realistic images, some of the existing methods may work as well as the approach developed in this work.  I feel that the paper needs to do a better job of distinguishing itself from prior art.\n\nAlong the same lines, while the work shows that ImaginaryNet outperfroms CLIP, it is not immediately obvious to me why is it so.  Posing this question another way, do you think CLIP can achieve comparable performance if allowed to train on larger training data (generated data, of course). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster