Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a training pipeline for better dealing with open-vocabulary object detection. Specifically, it incorporates CLIP as a pre-trained vision-language embedding to significantly expand the potential object detection vocabulary and mitigated performance degradation due to the gap between image-centric and object-centric representation. In this paper three important components are proposed to enhance the system, including (1) region-based knowledge distillation to match the CLIP embedding for image patch (instead of full image) with text, thus the replaced text classification head can better fit the region detection; (2) image-level supervision with pseudo box labels which benefit from a recent multi-modal ViT (MViT) class-agnostic object detector; (3) a weight transfer function which combines (1) and (2) more effectively in a trainable approach. The overall system is validated on multiple benchmarks and achieves state-of-the-art performance, especially for novel object classes. 
strengths_and_weaknesses: Strengths:

- The assumption for the performance degradation from CLIP to object detection is reasonable and the authors propose a solid solution to mitigate the gap. The image-centric and object-centric nature may lead to incompatibility of the visual encoder and textual embedding, thus introducing the $L1$ and $L2$ regularization objectives is intuitively helpful to bridge the gap.
- The weight transfer function seems interesting and shows great potential for merging two components in a more harmonized way. The performance boost is quite significant and similar ideas may inspire other ensembling/multi-component fusion systems.
- The details of the training and evaluation are provided and the paper is easy to follow.

Weaknesses:

- The model did demonstrate significant improvements over other approaches, although it doesn't seem clear to me how much of the distillation from MViT is involved in these improvements. See my Questions.
questions: 1. Although the authors perform detailed ablation studies to validate each component in their system, one critical aspect is not covered which is the performance of MViT on "novel" object proposal/detection. Specifically, since the region proposal is generated from MViT for both class-agnostic and class-specific approaches, one possibility is that the pre-training of MViT may have some leaked information to the student model. The authors did mention that "we ensure that MViT pretraining dataset has no overlap with any of the evaluation datasets in our work." but I'm curious about details about the exclusion -- does it mean the eval images are never included in the pre-training or even the novel object classes are not exposed? Probably an even more careful ablation study would be replacing RPN in the baseline object detection with MViT so the effect of the enhanced proposal can be analyzed. Essentially I'm feeling that the quality of the system is heavily dependent on the $\mathcal{Q}_{pseudo}$ but this paper did not provide sufficient ablations about it.

2. Regarding the weight transfer function, I'm curious how this is trained -- is it being trained in an end-to-end fashion (I presume) together with the full system using loss defined in L229? Does it mean $W_P$ in the weight transfer function is now replaced by $W_\tau$ and $W_D$? Probably it's better to explicitly show this relation in the equation below L229.
limitations: This author does not provide explicit explanations of limitations and societal impacts but I presume this system has similar issues with other systems (e.g. potential societal bias when trained with large-scale automatic collected datasets).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for open-vocabulary detection where the goal is to train an object detector able to detect objects in an image given a text description of the desired object category. The method is evaluated by its ability to detect novel classes for which no bounding box supervision is provided during train time. In more details, during training, one has access to a dataset containing bounding box annotation (e.g. COCO detection) along with a dataset that has only image level annotation (e.g. COCO captioning). The proposed approach relies on 3 core ideas: (1) a region-based knowledge distillation (RKD) approach which aims at enforcing that the features extracted from the detection vision backbone (the student) match the ones of a multimodal CLIP visual encoder (the teacher), (2) A pseudo labelling technique to extract candidate bounding boxes on the dataset containing image level annotation and (3) a method to enforce better combination of the two previous techniques by tying parameters. They report performance on COCO and LVIS and show stronger performance compared to previous approaches. 
strengths_and_weaknesses: **Strengths**

- The paper obtains good results on LVIS and COCO that outperform the state-of-the-art (Detic),
- The ideas are overall well motivated and the method is technically sound 
- The ablation study is overall well conducted (see more suggestions in the Questions section) 

**Weaknesses**

- An important aspect of the method seems to be that it relies on the MViT approach to obtain region proposals (used both in RKD with class agnostic proposals as well as for the pseudo labeling approach, see for Example Table 6 and Table 7 which shows the importance of MViT especially for the AP on novel classes. However MViT is trained on 3 datasets, some of which have bounding box annotation (GQA, VisualGenome and Flickr). The authors should discuss whether or not these datasets contain bounding box annotation for classes that are considered "novel" in the evaluation. If yes then I believe this would be unfair compared to previous approaches. See Questions for more details about this. 

- Related to the previous point, it would be good to list in Table 3, what are the datasets that are used by each methods (e.g. what is the source image-text pairs for all methods etc etc). I encourage the authors to provide this information in the rebuttal as well. This will help to better compare against past work such as Detic.

- The description of the approach is not always clear (see Questions section for a detailed list of those questions). It is important that the authors clarify this during the rebuttal.

- Some additional experiments could be conducted (see suggestions in Questions) 

=== POST REBUTTAL

After reading the rebuttal and other reviews, I am raising my score from 5 to 6. The authors have adequately addressed the important concerns I had about the novel class results. To the best of my knowledge, I believe that the results are now legit. 

questions: **About MViT**

As discussed above, it is unclear if using MViT is coherent with the idea of trying to generalize the method to novel classes. The authors should reply to the following questions:

- Among the datasets used to train MViT, what are the ones that are annotated with bounding box annotations? Among those, are there classes that overlap with the "novel classes" of COCO or LVIS?
- Is it possible to say how much MViT alone is getting on the novel classes of COCO (it seems that it should be possible to evaluate MViT on it)?


**Method clarifications**

There are various things that are not very clear from reading the method section. Questions are listed below:

- L140: what is the difference between $\mathcal{R}$ and $\mathcal{R}(r)$?
- L143: how do you take the cosine similarity against the background vector since it consists of a vector of all zeros (how does the normalization works in that case)?
- L143: related, isnt it bad to assign boxes to background even when they could be from another class which is simply not annotated in the dataset? (e.g. a COCO images could contain objects that are not in the regular 80 COCO classes and therefore not annotated). 
- L 167: How do you obtain the CLIP embedding used for RKD? Do you extract them by cropping the input images and feeding the crop to the CLIP model or do you instead extract a ROI pooled feature from a feature grid computed over the whole image? 
- Figure 2: please add the total number of categories used here. From the matrix it sounds like there are 16 categories (16 * 100 images total) but I could only count 10 colors in the TSNE plot).
- L205: what is N in practice for ImageNet21K and COCO captions? In particular how do you obtain image-level category label from coco captions? I might have missed this but I could not find those details.
- Equation (4): what are the negative classes used in this equation? 
- About the weight transfer function: it is unclear to me why one needs to have 2 different set of weights $W_P$ and $W_D$. The section 3.4. would benefit from better details on what exactly the architecture is and why those weights are different? The notation $f$ is used for the projection function but I found it confusing as I could not exaclty relate what $f$ precisely was with respect to the different weights $W_P$ and $W_D$. See related questions in the experiment section below.

**Question about experiment choices and details**

- Optimization details (beginning of section 4.2). Why does the method consists of multiple phases of training? Isn't it possible to train everything at once with all the additional losses?
- Related to the previous question, if everything was trained together, isnt it possible in theory to completely tie together $W_P$ and $W_D$ so that to have the exact same of weights for going from the visual feature space to the joint text-image CLIP embedding space?
- Related to the previous 2 questions, please provide details about what exactly is trained and what is frozen in the phases of training. 
- Alternative to weight transfer (Table 2): it would be good to add the following two approaches:
    * $W_P=W_D$ by freezing it from the second phase of training 
    * Cotraining with all the losses of L 229

**Minor requests/typos**

- Table 6 does not bold the right numbers. Table 7 is also missing a bold number (Supervised Base on AP_base)
limitations: The limitations are not discussed in the main paper. It would be good to do so: are there novel classes that suffer more than others? If yes, why? What are the important pieces of the method and how robust is the method to failure in those pieces? (e.g. the MViT component etc etc). Potential negative societal impact is also not discussed despite the method potentially being trained on large scale image and text datasets.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an approach for training open-vocabulary object detection models, i.e. models that can detect objects based on a list of object categories that is given at test time, potentially including novel categories not seen during training.

The paper starts from what has become the standard open-vocabulary approach: A Faster-RCNN detection architecture that is combined with a CLIP image/text model to allow open-vocabulary classification. The paper points out that the CLIP model is trained using only image-level supervision, and proposes a two-stage training approach to adapt it for the object-level detection task:

**First training stage: Region-based knowledge distillation:** In this stage, a pre-trained object detector is used to predict class-agnostic boxes on some dataset. A vision/language model (CLIP) is then used to extract image embeddings for the regions defined by the predicted boxes. These embeddings are then used as distillation targets for the region embeddings of the detection model. This approach closely follows the ViLD paper, although the present paper uses a different object detection model for the pseudo-labeling. One innovation is to add an “inter-embedding relationship matching loss”, which minimizes the difference between the similarity matrices of the CLIP (teacher) embeddings and the region embeddings of the student detector.

**Second training stage: Image-level supervision with pseudo-box labels:** In this stage, a pre-trained open-vocabulary detector is used to add box annotations to images from a classification dataset, by using the image-level (classification) labels as queries for the detector. These pseudo-labels are then used to train the classification loss of the detection models. This approach follows the Detic paper, but the present paper adds a new component called "weight transfer function". The weight transfer function conditions the model in the second stage on the embedding projection learned in the first stage, which is supposed to prevent the image-level supervision from destroying the representations learned in the first training stage.
 
The method is evaluated on the COCO and LVIS open-vocabulary benchmarks and compared to relevant prior work. All proposed components are ablated in experiments.


strengths_and_weaknesses: ### Strengths
1. The paper is written very clearly. Even though the proposed methods are quite complex, they are motivated and explained well. It was a pleasure to read.

2. While the method primarily combines existing approaches from ViLD and DETIC, it it contributes important innovations that improve the synergy between these approaches (the weight transfer function and IRM loss).

### Weaknesses
1. In their current form, the presented results are **not open-vocabulary**, in contrast to the claims in the paper: the paper restricts the training data to the target vocabulary (_“Following [2, 1], we use a subset of ImageNet-21K having 997 overlapping LVIS categories and COCO captions dataset for ILS in LVIS and COCO experiments respectively.”_). This vocabulary restriction means that the method is in fact not open-vocabulary, but closed-vocabulary, since the target vocabularies (LVIS and COCO) must be known at training time for this filtering. Please either use a non-restricted training vocabulary that is not specifically tailored to the test data (e.g. all ImageNet-21k labels) for all main results, or explicitly and prominently state that the results are not open-vocabulary. For a method to be “open-vocabulary”, no part of the training pipeline should have knowledge of the label space used during testing. I emphasize that the fact that prior work (Detic) does the same is not a reason to repeat this practice. If knowledge of the test label space is required to reach good results, then these results are not "open-vocabulary", and if such restriction becomes accepted in the literature, then the term "open-vocabulary" becomes meaningless.

2. It is not clear how the MViT model used for pseudo-box prediction is trained. Does this model see any bounding box annotations during training? Is it ensured that none of these bounding box annotations include any of the "novel" classes used for evaluation?

The validity of the paper critically depends on the soundness of the evaluation. If the results hold for rigorous open-vocabulary evaluation, then the work represents and significant contribution to the field.
questions: 1. See "Weakness 1": Provide true open-vocabulary results.
2. See "Weakness 2": Clarify MViT pretraining. If box annotations for "novel" classes were not held out for MViT training, this may inflate results. In this case, please provide results with novel-class annotations removed from all training stages.
3. Is the IRM loss computed for regions from single images, or all regions within a batch?
4. "Complimentary" --> "complementary"
limitations: Currently, the evaluation is not truly open-vocabulary, this needs to be addressed.

Societal implications are sufficiently addressed in the appendix.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work aims at training open-vocabulary object detectors by leveraging CLIP and pretrained multi-modal ViT. This work propose RKD to align the region representations from the RPN and that of CLIP image encoder. In order to better make use of external images (like ImageNet) which only provide image-level annotations, this work proposes to use MViT to generate pseudo boxes and labels and use them to self-train the detector. In addition, this work proposes to adopt a weight transfer function to alleviate the competition between the two losses. The results on COCO has a significant improvement over previous works.
strengths_and_weaknesses: Pros
1. The technical contents of the paper are quite a lot. I can tell that the authors put many efforts on the paper. It includes improvements on both knowledge distillation side (as in ViLD) and pseudo boxes labeling side (as in Detic). It is the first paper in attempt to combine the advantages of both knowledge distillation and self-training.
2. The experiments are throughout, including results on different dataset and the ablation study.
3. The use of weight transfer in this task is interesting and shown to help.

Cons
1. My major concern is the use of MViT that makes the comparison against ViLD and Detic unfair. Because (1) model perspective: MViT can already detect objects, so using MViT in this task is like supervising a detector using another trained detector; (2) data perspective: MViT is pretrained on LMDet dataset, which is very large with human annotations. If the authors can address this concern of mine, I may consider raising the rating.
2. The overall pipeline is quite complex, including the three training stages and how different losses/components work together. Moreover, the hyper-parameters need to be searched for different datasets, which makes me concern about the generalization of the proposed method.
3. It feels like the authors combines many existing methods/models in order to achieve good performance. For example, the IRM loss is just the loss proposed in Similarity-Preserving Knowledge Distillation [39]. Without IRM, the proposed RKD degrades to the knowledge distillation proposed in ViLD. Besides, the weight transfer function is proposed in [41]. 
4. The total training on COCO takes 36 epochs, which is longer than Detic.
5. The advantage of the proposed method on large scale datasets seems insignificant. Although the performance gain on COCO is significant, but the gap between the proposed method and Detic on LVIS and Object365 is marginal (ie. less than 1AP).
6. There are too many abbreviations in the paper, such as RKD, ILS, PIS, IRM loss, which to some extent makes the paper less easy to read.
questions: 1. What is the performance on MViT on COCO or LVIS in the open-vocabulary setting?
2. Do the author have some results without the use of MViT? I would like to see a fair comparison to Detic.
limitations: Yes.
review_title: null

Meta-review: metareview: The paper receives overall positive ratings after rebuttal. The major concern before rebuttal is that the benefits and limitations from using MViT are unclear. The rebuttal has addressed most concerns from reviewers. AC encourages authors to make the final revision with review comments.
confidence: Certain
Decision: Accept