Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a framework, named ImaginaryNet, to train object detectors using images generated by a text-to-image model and class labels from a pre-trained language model.
This is also a learning scheme, named Imaginary-Supervised Object Detection, for training object detectors without requiring real images and manual annotations. This scheme is based on a weakly-supervised object detection (WSOD) algorithm, i.e. OICR, with the object proposal and class labels obtained from ImaginaryNet.
The paper's main contribution is in using the language model and synthesis model to build a learning scheme on top of the existing WSOD scheme without requiring real images or manual annotations.

strength_and_weaknesses: Strength:
1. Using language and synthesis models is a clever idea to improve the existing WSOD algorithm. 
2. Extensive experimental results to demonstrate the strength of the proposed method

Weakness:
1. The effect of the limitation caused by synthesis models was not well studied and discussed although there is an ablation study on the effectiveness of the language model.
2. The experiments and ablation studies sections are not well organized and hard to follow. There is some confusion in the settings, e.g. ImaginaryNet ISOD, WSOD, SSOD.

clarity,_quality,_novelty_and_reproducibility: The overall quality of the paper is acceptable. More clarity is needed in the experiments and ablation studies. The overall ideas of the work are original, however, the framework is developed by putting together existing models in a pipeline, e.g. language model via GPT-2, text-to-image synthesis model via DALLE-mini, and proposal generator via selective search.
Based on what is described in the paper, the proposed framework should be reproducible.
summary_of_the_review: My initial rating is a weak reject. The main reason for rejection is that the paper did not put enough focus on the synthesis model. This makes the paper's claim of learning object detectors without real images weaker since there could be an object class that the synthesis model cannot generate without learning it first. In that case, we still need to have real images. For example, learning a "human hand" detector would require the synthesis model to generate an image with a "human hand".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a scheme for training object detection networks using self-generated synthetic data.  The system presented herein does not require manually-annotated real images, which are hard to come by.  Rather, the system leverages the spectacular advances in text-to-image generation system to construct visually realistic data that serves the role of traditional object detection datasets.  The proposed scheme is able to benefit from the "real" data when available.  The paper also includes a set of experiments that confirm the useful of this approach.
strength_and_weaknesses: The approach presented herein is well-motivated; however, given prior works that also advocate the use of synthetic data for training deep learning model, the novelty-aspect of this work is somewhat low.  Section 2.2, for example, lists some of these works.  Reading this section it seems that existing schemes are not as robust (or useful) for model training simply because the quality, i.e., visual realism, of the generated images do not match "real" images.  This subsequently leads to a domain gap, which must be addressed by using manually-annotated real images.  If this is true then one may reasonable assume that with our increased ability to generated visual realistic images, some of the existing methods may work as well as the approach developed in this work.  I feel that the paper needs to do a better job of distinguishing itself from prior art.

Along the same lines, while the work shows that ImaginaryNet outperfroms CLIP, it is not immediately obvious to me why is it so.  Posing this question another way, do you think CLIP can achieve comparable performance if allowed to train on larger training data (generated data, of course). 

clarity,_quality,_novelty_and_reproducibility: The quality of the writing is good.  The paper does suffer from low novelty since others have explored similar ideas to train deep learning models.  The engineering apsects of the work are clear; however, I couldn't get an intuitive feel for why the paper outperforms a method such as CLIP.  I feel that the paper will benefit from including a summary of how this work is different from prior work and what aspects of the model leads to better performance.  Is it training?  Is it because the model has a lot more capacity than some other models? Is it because we now have models to generate highly realistic visual data?
summary_of_the_review: The work is interesting and it is moves the field of deep learning in the right direction.  The approach presented here reduces our reliance on manually annotated real data for the task of object detection, which is highly desireable.  At the same time it is somewhat difficult to ascertain what makes this model better than other approaches that also use generated data.  The novelty, I feel, is on the lower-end of the spectrum.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors study the problem of training an object detector without any real images or annotations. For this end, they propose generating synthetic (realistic) images using a pre-trained text-to-image model conditioned on words for the object categories as interest, and following a weakly-supervised object detection pipeline to train a detector given the generated image and the class labeled that conditioned it. The authors evaluate their approach on PASCAL VOC (and partially on MS COCO) with a comparison to other pre-training based object detectors, weakly-supervised detectors and a supervised method.

strength_and_weaknesses: Strengths:
+ Annotation is a labor-intensive task and mechanisms for reducing the cost of this process are very valuable.

+ A novel paradigm is introduced for using pre-trained text-to-image models for training object detectors.

+ Strong results over the baseline CLIP-based models.


Weaknesses:

1. My only concern with the paper is that it is lacking a comparison with other CLIP-based detection approaches introduced in Introduction.


Minor comments:

- "Humans can easily detect a known concept from language description without the demand of training in reality." => I would be very careful with such claims. I kind of disagree with this as there must have been some kind of (pre)training for humans to recognize the concept as known, either the language model, the other objects or the parts of the object/concept etc.

- First paragraph of the Intro: "RPNs or RoI heads" => Acronyms are used without introduction.
- "which is ”A photo of a”" => The opening quotation is incorrect. You can use `` in Latex to do this.
- "image encoder consisted of multi-layers" => "image encoder consisting of multi-layers".
- "a RPN network" => "an RPN network".
- "IMAGINARYNET apply RoI Pooling" => "IMAGINARYNET applies RoI Pooling". Please cite the relevant paper here for RoI pooling.
- "IMAGINARYNET obtain" => "IMAGINARYNET obtains".
- "IMAGINARYNET sample" => "IMAGINARYNET samples". The same problem recurs through out the text. Please check the paper carefully for more.
- "OICR(Tang et al., 2017)," => "OICR (Tang et al., 2017),".
- "and the structure of them are shown in Fig (2)." => "with the structure shown in Fig (2)."
- Eq 2, 3: dot should be placed after the eqs.
- "the (Tang et al., 2017)," => "Tang et al. (2017),". You can use a different cite command to get Tang et al. (2017).
- "for reaching no real images and manual annotations" => "for obtaining no real images and manual annotations".
- "the Edge Boxes algorithm" => Provide citation.
clarity,_quality,_novelty_and_reproducibility: The paper is around %10 of the papers in terms of clarity, quality, novelty and reproducibility.
summary_of_the_review: Novel method, strong results. Missing important comparisons to other CLIP-based approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to learn object detectors from text-to-image synthesis data. The idea looks interesting to the community.
It first generates sentence description with prefix constraints and GPT-2, then generates imaginary images with DALLE-mini. 
A WSOD is applied on sythensis image. The whole method is named ISOD. Experimental on PASCAL VOC 2007 shows that ISOD achieves close performance to WSOD on real images and annotations, while the performance could be even boosted when gradually increases real images and annotations. 
strength_and_weaknesses: Strong points
+ The idea looks very interesting to me. 

Weak points
- Some descriptions are not very clear. For instance, 

       - what are the prefix in the language models? How different prefix make impacts for the final results? 

       - In Table-1, there are some classes (boat, chair, etc) that Imaginary perform worse than CLIP, it lacks explanations. Are these due to the image synthesis errors?

- The idea looks a bit straightforward, but the formulation is a bitter weak. It does not take the language and text-to-image generation errors into considerations. It would be perfect to have a GAN like loop between detectors and generators. 

- For illustration part, I would suggest the author to show some cases why ImaginaryNet does not works but WSOD works, so that possible new ideas could be risen from analysis. 
clarity,_quality,_novelty_and_reproducibility: - The overall idea looks somewhat novel to me, although there are im2real based on Gaming engine before, but using image-to-text, to the best my knowledge, this is the first. 

- Although the paper just consists of several public available modules such as GPT-2, DALLE-MINI, WSOD, it still lacks of details or detailed studies on the language generation prefix. 
summary_of_the_review: This paper could be viewed as WSOD on image-to-text synthesis datasets. It needs to handle the domain differences between generated images and real images. The idea is overall novel, but the formulation is a bitter weak and the studies also not that comprehensive from my perspective. 

I think the idea deserves a top conference, but the paper needs thorough studies to before it could be published. 
I may change my rating based on the rebuttal. 


==============================
The authors give detailed response to all my concerns. I raise the rating from 5=>6. 
However, I would seriously suggest the authors make a revision to reformat the story of the work to move some important contents in appendix into the main body.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: The paper tackles the problem of training object detectors without directly using any annotations or real images. The authors use a text to image generative model to generate images, followed by training a weakly supervised detector. The model is evaluated on COCO and VOC.

*Strengths*: (1) The idea of using text to image generative models for training detectors is interesting. Since these models can be conditioned on object categories, they generate object-centric images. (2) Strong performance compared to recent models like Detic and ViLD. 

*Weaknesses*: (1) The method's performance is bottlenecked by the generative model.  As the authors also note in their response, the performance saturates with many generated samples. (2) The experimental details should be better explained.
justification_for_why_not_higher_score: The paper's presentation (writing quality, clarity) could be greatly improved. Technically, the approach has limitations primarily tied to the generative model's capability. Generated images have been used to train detectors before, which limits the technical novelty.
justification_for_why_not_lower_score: The proposed method achieves really strong performance compared to recent CLIP-based detectors. This makes it practically useful, while also being interesting.
Decision: Accept: poster