Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to directly learn from large-scale image-text pair data for open-vocabulary object detection. 
It formulates object-language alignment as a set matching problem.
The benefit of such approach is to allowing training on image-text pairs without the help of grounding information.
The author conducts experiments on COCO and LVIS to demonstrate its performance on novel categories.
strength_and_weaknesses: Strength
+ Performance is slightly better than previous methods

Weakness
- The paper lacks necessary mathematics proof on solving object language alignments learning. How does bipartite matching is optimized end-to-end during training? Hungarian matching is not differentiable. How does it guarantee convergence on learning?
- It also lacks implementation details. For example, in table 4, how does one-to-one or one-to-all affect the matching? In table 5, how does image-text pair loss implement?
- For Table 1, compared with Base-only, the proposed alignment loss downgrades the performance on supervised base categories. What is the reason?  
- For Figure 3, the author avoids example with multiple same instances. What will happen when apply to an image "a basket of oranges"?  
clarity,_quality,_novelty_and_reproducibility: The clarity can be improved, especially on the details of method.
summary_of_the_review: Overall, I feel solving object language alignment without grounding information is the correct way. However, this paper doesn't prove the effectiveness of using bipartite matching nor well explain the details. 

After reading the rebuttal, I still don't get enough details on the correctness of the proposed method. Meanwhile, it also raised me another concern on its ability of handling dense detection. Hence, I would not recommend to accpet this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a set-matching approach for open-vocabulary object detection by jointly training on detection and image caption dataset. The approach achieves good results on the existing COCO and LVIS open-vocabulary detection benchmark.
strength_and_weaknesses: Strengths:
* The idea of joint training with caption data is promising as captions are noisier and more scalable than image tags
* Set matching is an intuitive approach to handle region-word alignment
* The label space of COCO/LVIS are not used in training phase which follows the open-vocabulary detection settings
* Performance on COCO/LVIS are both strong and the margin over Detic baseline is clear
* Ablation studies are informative.

Weakness:
* Method - what are the constraints for region-text matching? Are all regions assigned to words or are all words assigned to regions? I can imagine some words or regions are noisy and do not have good matches.
* Results - the scalability of the proposed approach needs further study in the axis of data size (e.g. CC12M, larger web-scale image-text data) and model size (larger backbones). It'd be very useful if this approach scales well with data and model size. A simple study is to subsample CC3M and observe the scaling benefits.
* Looks like the proposed approach compromises base categories compared to the Detic baseline. Table 5 image-text loss only has the same overall AP suggests this as well. Is there any explanation for this?
* Comparison with DetPro is unfair because DetPro uses ViLD detector as baseline while this work uses Detic CenterNet as baseline. Detic papers report a ViLD detector baseline which is lower. A more fair comparison would be to report a Mask R-CNN detector (ViLD style) baseline on LVIS.
* SOTA on LVIS claim is not well supported because this study focuses on R50 backbone only.
* [minor] how much overhead does the set-matching add to the training time?

clarity,_quality,_novelty_and_reproducibility: Clarity - the paper is written clearly and easy to follow.
Quality - the quality is good overall with some weaknesses as indicated above.
Originality - novelty is good because the set-matching loss hasn’t been shown effective in the joint training setup to my knowledge.
Reproducibility - the authors promise code release and provide some details of implementation.


summary_of_the_review: The paper proposes an interesting approach to tackle open-vocabulary detection by set matching loss on caption data. There are some weaknesses in the paper but I think the merits outweigh the downsides and am leaning positive for the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a novel method for open-vocabulary object detection (OVOD), where a model is trained from two datasets, one containing bounding box annotations for a set of base categories, and another one containing only free-form text descriptions (captions) of imagesf. While prior work typically uses grounded annotations (words in captions are associated with bounding boxes) or distills information from image-level pre-trained vision and language models, this work proposes a loss function to directly leverage image-text pairs (without grounding supervision). The architecture of the detector is similar to existing ones, with the only exception that text embeddings are predicted instead of a fixed-size probability distribution over a known label space. The model trains from both detection data and image-text pairs. The losses are the same as in standard detectors when training from detection data. For image-text pairs, a similarity is computed between each noun in the caption and each predicted region (region proposals). Bipartite matching is then applied on the similarity matrix to find a matching between regions and nouns, which then serves as ground truth for a standard binary cross-entropy loss. Experiments on the open-vocabulary setting for both COCO and LVIS datasets show improvement over prior baselines, specifically for novel categories.
strength_and_weaknesses: Strengths:
- The topic is interesting
- The paper is well written, easy to follow, figures a good
- The method is easy and simple, but effective
- The experiments are mostly clear and the method performs well
- Ablation studies are reasonable.

Weaknesses:
- The dataset generalization experiment in Section 4.6 from COCO to Pascal VOC is limited. The 20 Pascal VOC categories are a subset of the 80 COCO categories. The question is how many of the 20 categories are part of the 48 base categories. Still, there will likely only be a few novel categories.
- Did you try a different text embedding architecture that was not jointly trained on images and text? Would this make a difference?
- The related work and the baselines can be improved
  - I think paper [A] (concurrent work) can be discussed in the related work section.
  - The paper [B] is another recent example for OVOD that can be discussed and compared with. It uses pseudo labels and gets strong results on COCO, although in a slightly different setting.
  - The reference "Gu et al." is now an ICLR 2022 paper
- I'm missing a more thorough discussion on the region-word alignment loss
  - It would be good to explain in words how the loss deals with incomplete captions, i.e., when some objects captured in the image are not referred to in the text. Looking at the loss and assuming a perfect matching, it seems such regions would just be ignored by the loss.
  - The nouns in other captions of the same mini-batch (W') may contain the matching (positive) word. How likely does this happen? And why does W' not contain other words of the same caption?
  - How do you deal with plural descriptions in a text, e.g., the caption "Two cats near a dog". Again, I think the loss would just ignore one of the regions that contain a cat (assuming a good matching). Still, I think it would be valuable for the paper to discuss such situations in text.

References:
- [A] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. Golnaz et al. ECCV 2022
- [B] Exploiting Unlabeled Data with Vision and Language Models for Object Detection. Shao et al. ECCV 2022
clarity,_quality,_novelty_and_reproducibility: - Overall, I think the paper is of high quality, clearly written, and novel. The method seems easy to reproduce, but the authors seem to release code upon acceptance.
- There are few parts of the text that can be improved, though:
  - The first paragraph on page 5 about the image-text loss is not clear to me. My assumption is that the whole caption is represented by one embedding vector and the whole image as one region vector, which constitutes another (correct) match. However, I think the text in this paragraph could also be misinterpreted as actually having some image-text pairs with ground truth region-word alignment (like in a semi-supervised setting), which I assume is not the case given the whole motivation of the work.
  - Also the paragraph on "Object vocabulary" on page 5 is not clear to me, without having read the corresponding paragraph in the ablation study. It might help to use the notation introduced in Section 3.1.
  - The one-to-one vs. one-to-many experiment needs more clarification. First, the intuition for doing this is missing. I assume it relates to my comment above on how to deal with plural nouns in the text. From the definition of the loss function in Eq. 2, I assume the relation here means one (word) to many (regions). As a suggestion for further investigation, would it help to identify if nouns are plural or singular, and then choose the appropriate constraints (alignment algorithms) for mapping to either one or multiple regions?
  - I think it should be made more clear in the beginning the that the model trains from both detection and image-caption data. Otherwise, the reader may wonder how the model creates sensible regions in the first place.
  - Typo on page 8 last paragraph: "leaning" -> "learning"
summary_of_the_review: Overall, I think this a high-quality paper with clear impact to the research community on open-vocabulary object detection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to use weak supervision (image-caption pairs) for open-vocabulary object detection (OVOD). This is in fact a problem of weakly-supervised object detection (WSOD). In stead of using the max-score or max-region matching in the literature, this paper proposes to formulate the problem of object-noun matching as a problem of bipartite matching problem, which can be solved by Hungarian algorithm. The experiments on COCO and LVIS have shown that this simple method can have big improvements. 
strength_and_weaknesses: **Strength**

+Since the object-language annotation is expensive, it is reasonable to use weak supervision to improve OVOD performances.

+The use of bipartite matching is reasonable and simple.

+The improvements on COCO and LVIS seem nontrivial.


**Weaknesses**

-Using weak supervision to improve OVOD performance is not new, e.g. in RegionCLIP, GLIP, X-DETR, Detic. But those works only show some minor improvements, unlike 5 points gain on LVIS in this paper. I don't see anything fundamentally different, so I am not sure why the gains in this paper are so big.

-Using bipartite matching for WSOD is not new, as seen in [1]. However, this paper doesn't discuss it.

-This paper only shows the results on a base backbone, e.g. ResNet-50. However, many works have already shown high OVOD performances using some large models, e.g. [2]. I am wondering if the gains of this paper can be generalized to large models. 

-It seems using weak data can easily get 4-5 points improvement, but I don't see the ablations studies of getting those improvements step-by-step

[1] Omni-DETR: Omni-Supervised Object Detection with Transformers, CVPR 2022

[2] Simple Open-Vocabulary Object Detection with Vision Transformers

clarity,_quality,_novelty_and_reproducibility: **Clarity**

This paper is clear and reads easily. But I still have some questions.
- The last three lines of paragraph "multi-modal object detection" is not accurate.

- How to use image-text caption is unclear. Do you extract the global image features by global pooling? And on which layer? More details are needed.

- I am confused about the experiments of Table 3. If you only use category names, do you still use captions? If so, how? And there are only 6750 nouns for CC-3M dataset? And how to interpret those numbers? More details are required.

- What's the row of using nothing in Table 5?

- Do you generate the pseudo labels online? Do you use a student-teacher framework as in many semi-supervised and weakly-supervised object detection papers?

**Novelty**

Overall, I think the novelty is small, given using weak supervision and bipartite matching are already in the literature.

**Reproducibility**

Although the proposed algorithm is conceptually very simple, I am concerned about if we can reproduce the same improvements, given the detailed ablations are missing. Also the paper doesn't mention the code will be released.
summary_of_the_review: The idea is simple and easy to understand, and it gives some big improvements. But the technical novelty is small, no results comparing with the state-of-the-art, and I am concerned about the reproducibility.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper tackles the open-vocabulary object detection problem by leveraging an image-text corpus without any grounding annotation. This differs from prior methods that use some form of grounding annotation or distillation from image-level multi-modal models. Specifically, it proposes a simple method to perform bi-partite matching across a similarity function between nouns in the text and regions in the image, and utilizes that in a standard binary cross-entropy training loss. Also unlike prior methods, this method does not require knowing the label space of the "open" datasets since it mines this from the corpus. Results are shown across COCO and LVIS datasets to show improvements in detection, especially on the novel classes. 

The reviewers all appreciated the topic, the writing (for the most part, see caveats), simple but effective method, and improvements. However, there were a number of concerns pointed out including 1) Novelty - Several prior works use weak supervision or even similar ideas of bi-partite matching (unsurprisingly given the generality of the problem) including weakly supervised object detection, 2) Scale with respect to the text corpus size and better architectures (e.g. Swin), 3) Some performance sacrifice on the base classes, 4) Use of different detector baselines (e.g. vs. DetPro), 5) Limitations of the one-to-one mapping, and 6) A number of other clarifying questions. The authors provided extensive rebuttals, including significant new experiments showing scalability across data and model size and explanations on how the work is situated with respect to the prior works.  

After the rebuttal, the reviewers had an extensive discussions. Some were not convinced especially by the novelty of the method or strength of the results given the simplicity/empirical nature of the method. Others had concerns about the lack of theoretical or other analysis of the way region-text matching works, although of course most prior methods in this subfield also tend to be empirical in nature. In the end, after having read the paper and considered all of the materials (reviews, rebuttals, and discussions), I recommend accepting this paper. While it is true that the paper has some limitations (most notably some sacrifice on the base classes, making it unclear where it stands in the tradeoff compared to prior methods), it offers a different perspective than prior methods which train on a range of data (including with grounding annotations used either in pre-training or during training) and instead considers image-text pairs in a fine-grained way. The method for doing so is simple, as oftentimes deep learning methods tend to be, but the idea for doing it is compelling given that such data also forms the basis of models such as CLIP. The paper also performs "true" open-vocabulary detection (where the label space is not taken from "open" dataset but instead from the collective nouns in the corpus). All of these perspectives will be of significant interest to the community, and given that the authors plan to release the code it could form the foundation upon which further works can build to address some of the weaknesses.  

I do recommend that the authors take the significant discussion to heart and further update the paper, for example running some of the experiments mentioned (e.g. aligning with DetPro) and providing a more extensive exposition on the limitations of the method such as when dealing with potential failure cases of one-to-one mappings. 
justification_for_why_not_higher_score: Overall, while I believe this paper should be accepted, I do not believe it reaches the level of a spotlight or oral paper given the discussion. 
justification_for_why_not_lower_score: Overall, this paper does have some limitations that could be used to justify rejection. However, I believe that the new perspective it adds to the open-vocabulary problem, compared to prior works, would be of interest to the community and could be further built-upon in future works.
Decision: Accept: poster