Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of open world object detection - a lifelong learning system where the object detector is required to detect objects belonging to all classes known so far and “unknown” objects. Unknown objects are annotated and available for training at a later stage. At a given time only a specific set of training data is available with annotations that involve all known classes so far, while all past training data are unavailable. 


The proposed approach builds upon recent work [A], by incorporating a semantic topology in the object feature space (RoI features in Faster R-CNN). While [A] use contrastive clustering to group object features based on object category, and energy based out-of-distribution detection to detect “unknown” objects, the authors propose an end-to-end trainable approach. The proposed approach uses RoI features in two parallel streams, with one focused on clustering operation, while having object category prediction loss on both streams of features. Features are clustered around predetermined semantically meaningful object representations. Pre-trained word embeddings from a language model (CLIP) are used for this purpose. The use of pre-defined semantic anchors allows learned features of known objects to form consistent clusters during incremental learning. 


Experiments are conducted using 80 categories from PASCAL-VOC and MSCOCO datasets. The problem is designed as 4 sequential tasks each involving 20 object categories (same setup as in [A]). Detection performance is measured using mAP, as well as measures that focus on unknown object detection.

[A] Towards Open World Object Detection, Joseph et al., CVPR 2021


main_review: #### Strengths:
1. The idea to use semantic topology to enforce discriminative features that are consistent across incremental learning is an interesting direction for open set detection. The paper is well written and the ideas and experiments are presented with clarity.
2. Experiments are thorough, including ablations that show the effectiveness of different model components. It is interesting to see that using an additional classification loss on a separate stream of clustered RoI object features leads to significant improvement in detection performance.

#### Weaknesses:
1. The proposed approach seems to show significant improvement in detecting unknown objects. However I have a few questions on Table 1 results:

     (a) Wilderness impact values of ORE [A] in Table 1 is ~2 times that of the corresponding values in the [A], while all other results seem comparable. What is the reason for this? 

     (b) Task 2 Wilderness index for ORE [A] is shown as 0.2970. (Should it be 0.0297?) The number seems inconsistent, as it is an order of magnitude higher than other comparable results in the table(for task 1 and task3), and also the corresponding result from [A]. The same result is also shown in Table 3. 
     
      (c) If (a) and (b) do point to some errors, it seems that there are significant gains in A-OSE but much less gain in WI when comparing with [A]. Could this be explained based on the data? 

2. The proposed approach is closely related to [A], which is also the main baseline. However, Fig 1 and Fig 2 show the learned representations using vanilla training strategies using Faster R-CNN vs. using semantic topology. Compactness of feature representation is also obtained in [A], so the figure tends to mislead the reader about prior work. It would be better to see a comparison of feature representations of semantic topology vs. that of ORE.

#### Additional comments / Questions:

1. Does semantic feature classification head also predict for the “unknown” class (unlike RoI feature classification)? It seems to be implied that way, but it is not made explicit in the paper. It would be useful to have a description on exactly how “unknown” objects are detected during inference. 

2. Formula of WI in subsection 4.2 is not correct (-1 should be a separate term, not in the denominator). Also, the citations for WI and A-OSE are swapped in that section.

3. The model is designed such that features of unknown objects will change during incremental learning to cluster around predefined semantic anchors. It would be good to have more discussion on the nature and choice of unknown anchor in the semantic topology. For instance, what is a good choice for the unknown anchor? Does the choice of unknown anchor affect learning?

[A] Towards Open World Object Detection, Joseph et al., CVPR 2021

summary_of_the_review: The proposed approach is significant and relevant in open world object detection, proposing an end-to-end approach that learns incrementally and can detect unknown objects. A key promise of the proposed approach over prior work seems to be the improved ability in detecting unknown objects. However, I have concern regarding the main results reported (see weakness 1), and some other questions. I am willing to raise my rating if the authors can address these.

**[Edit after discussion period: ]** I thank the authors and the other reviewers for active engagement in the discussion. Important related work on zero shot learning was not cited in the preliminary version, which the authors included later on. While there is one point of concern in the paper claims (see below), I think this work provided a solid contribution to open-world detection problem. Hence I would like to raise my rating assuming the authors can make the necessary updates to fix the concern below.

* It was brought to notice by a reviewer the discrepancy between the problem definition in Section 3.1 and the evaluation protocol. While the problem is defined to involve "human user.. annotate (unknown instances) and return back to the model" at each stage, the evaluation protocol involves a different training set involving the unknown instances at each stage. I believe both settings are equally relevant for open-world detection problem, and I would encourage the authors to update the claims to match with the actual evaluation protocol.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a semantic topology embedding for Open-World Object Detection (OWOD) where an object detector identifies objects of unknown classes and incrementally learns to classify them assuming that their annotations are progressively given by humans. To maintain discriminative and consistent relationships among object classes, the authors introduce a semantic topology for the feature space of the detector by constructing pre-deﬁned anchors for categories using a pretrained language model. During training, it enables the detector to distinguish unknown objects out of the known categories and also makes learned features of different classes undistorted during incremental learning. Eperimental results show that the semantic topology improves state-of-the-art open-world object detectors and help the open-world detectors preserve a discriminative and consistent feature space.
main_review: - Marginal novelty. 
While the semantic topology scheme is somewhat new, this work is heavily built on the work of Joseph et al., CVPR’21. Many parts of this paper are recaps of Joseph et al.'s work. Furthermore, leveraging the embeddings of the pretrained language model is popular in the zero-shot learning literature, so I don't think the core idea is very original while it's interesting to see the effect of the simple method. 

- Predefined and rigid semantic topology.
The proposed semantic topology is rigid and not adaptive to the new object classes coming along. While it's surprising that it substantially improves the performance, this predefined and fixed topology is too simplistic and may not scale well to a larger number of classes. 

- Use of a pretrained language model.
The proposed semantic topology requires the use of a pretrained language model, which means an external knowledge source as well as the use of class 'word' information in training. In this aspect, the performance comparison to the previous work can be seen as not fair. While the authors provide the result with random anchors for semantic topology, the results are not complete and also make me intrigued. Since the random anchor assignment may organize the embedding space in a random manner, it may hinder the learning of a proper embedding space. It needs more careful investigation on this issue.  

- Incomplete analysis.
Some important studies on the effects of language models and semantic anchor losses are done only on task #2, which is strange. To see the consistency along with the incremental learning processes, the results need to be shown for all the tasks. In particular, the study on the random anchors needs more clarification and in-depth analyses. How exactly the random vectors are generated? If the random vectors are too close to each other, they may harm the performance, in particular, when the number of classes increases. Did the authors observe such effects? Did the author use a specific random assignment to make the anchors distant from each other?  

- Missing details.
Unknown-aware RPN in Section 3.3 is not clearly described. Is it trained or just borrowed from the work of Joseph et al.'s 2021? Since this looks like one of the core modules in this work, a more detailed description needs to be done for the paper to be self-contained. 

- Etc. 
The title of this work is too general for readers to recognize what this work is about. I suggest changing the title and making it more informative, at least, including the term 'open-world detection'. 

summary_of_the_review: While the proposed semantic topology is simple yet effective, outperforming the state of the art, this work has marginal novelty, clear limitations, and missing analyses. I'm on the borderline, slightly leaning toward rejection. I will make a final decision based on the authors' rebuttal responding to my critiques.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends the recently-introduced object ORE object detector [A]. ORE is trained in an incremental fashion and was shown to minimize the confusion between classes presented in different task sets by adding a contrastive objective to the model training, that pushes features, representing different semantic classes, far away in the feature space. This paper presents the case that combining ORE detector with large-scale language models, trained by aligning textual queries and images, can significantly improve ORE detector results. This is not surprising: it was already shown that such multi-modal language models could be successfully used for detection of novel classes in a zero-shot setting (in which names of target classes are given, but image data for these classes is available during the model training, see [D]).
To the best of my knowledge, the core idea of semantic anchoring using language models was introduced in [C] (see Sec. 3.3 Densely Sampled Embedding Space). In this work, the semantic embedding space is extended with additional data from external sources that contain semantic information about the unseen classes via language embedding to image embedding alignment. This method suggests using CLIP [B] instead (which makes sense and simplifies the knowledge transfer!)

[A] Joseph et al., CVPR’21
[B] Radford et al., ICLR’21
[C] Bansal et al., ECCV’18
[D] Xian et al., CVPR'17
[E] Zheng et al., CVPR’21
main_review: **Pros**

* The core idea introduced in this paper is nice, simple, and intuitive: simply leverage large-scale language models for zero-shot transfer to initialize a large variety of semantic class prototypes in the embedding space. Such augmentation of ORE makes perfect sense, and as verified experimentally, is very effective! 
* This paper tackles and very challenging important problem and the methodology is presented clearly. 
* I like the experimental evaluation; besides showing that leveraging language models for this task improves performance on incremental object detector learning, this paper also nicely ablates the impact on the effect of different language models.
 

**Cons**

* I am confused about the claims that prior work does not encourage intra-class compactness; isn't that exactly what the contrastive loss (in ORE) (in this paper referred to as clustering loss) is supposed to do?
* This paper adopts the task description from [A], claiming that novel objects are recognized as unknowns and should be labeled by annotators and used for re-training. This is not what the proposed method or method being built upon (ORE) does. Both are always only evaluated for the task of incremental learning, in terms of mAP and metrics, that evaluate confusion between known and unknown classes (e.g., Wilderness Impact measure): neither [A] or this paper provide any evidence that the learned detector is effective in detecting novel object instances (and separating them from the background *stuff* classes), and can be reliably be used in (for example) in conjunction with active learning to minimize the annotation effort. In summary, there is no evidence that any of the "new" classes in the new task sets are actually being detected before the network update.  
To justify the claims made in Sec. 3.1, the model would need to label instances as unknowns and transfer semantic labels to these before using them for the model re-training (and not simply usefully labeled images for the new task set). I know that this confusion was not introduced in this paper. Rather, this paper just iterates claims and statements made in [A]. Still, I would suggest not propagating it further and fairly acknowledge that this paper really tackles object detection in an incremental learning setting. 
* The basis of this paper is zero-shot learning: the knowledge about unseen classes is transferred from the language models. Such a paper absolutely must establish links with zero-shot learning and discuss prior work. I can understand that in the current rate of publications in our field one can miss a certain related paper, and I am usually happy to point this out and suggest updating the related work. However, this is not the case here; this paper entirely ignores the field of zero-shot learning, which is a basis for this work. 
I strongly suggest performing a literature review in this field. In terms of zero-shot recognition, a good source would be [D]. Closely related methods on zero-shot detection and segmentation are aforementioned [C], and [E].
* A big part of the paper is 1-1 re-write of sections from [A], e.g., 3.1, 3.3, 4.1 (using own words). A fair thing to do would be to write a very short recap on problem definition and ORE detector, followed by introducing what is novel in this work. Reading this paper largely feels like re-reading [A].
 

**Question:** which semantic classes are used to initialize the anchors in the embedding space?

summary_of_the_review: This paper describes a nice way of leveraging knowledge about various visual concepts, distilled in modern language models, to improve over incremental learning method by [A]. 

However, overall I, unfortunately, need to point out that the proposed (valuable and effective!) extension of ORE is almost exactly what was proposed before in the scope of zero-shot object detection [C], just leveraging the most recent developments in natural language processing. In case my interpretation is wrong I would like to see a very thorough discussion in relation to the aforementioned (that should already have been in the original manuscript), with a focus on what exactly the difference is.  


I still think this paper still carries an interesting message for the community, but (i) link to prior work on ZSL must be clearly established, and (ii) I do not think that ICLR is the right venue for this work in terms of novelty.  


**Post rebuttal** 


I would like to thank the authors for their comments and colleagues for the discussion. First, I would like to clarify that the paper was updated to include a discussion on the closely related topic of ZSL. I take back my comments that the usage for semantic anchors reduces this problem from open-world detection to zero-shot detection. Those were based on my initial miss-conception on how semantic anchors were used. 

However, I still think that the evaluation protocol studies only incremental learning and provides no evidence whether the proposed method can be actually applied to the (significantly more challenging!) problem of open-world detection, and authors actually do acknowledge that in one of their responses. 

As a reviewer, I cannot recommend accepting a paper that (in my view) has a mismatch between paper claims and premise (open world detection) and the actual delivery (incremental learning). My colleagues do not find this point as concerning (esp. given the fact that this issue was inherited from the prior work), therefore I will not strongly argue against acceptance, but I nevertheless wanted to bring this issue to AC's attention.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel method for open-world object detection, where instances of unknown categories need to be identified and annotated data for such new categories need to be integrated into the model in an incremental fashion. Prior work typically separate this problem into two tasks, out-of-distribution detection and incremental/continuous learning. This paper proposes to use fixed semantic anchors for each category, which are embedding vectors from a language model (or randomly generated vectors) for each category.  Importantly, when new data arrives, new embeddings are added while previous ones do not change. This encourages the feature representation to be compact (discriminative) and consistent (over time).
main_review: ### Strengths

- The paper deals with an interesting problem, which has clear practical relevance
- Overall, the approach is well described and motivated, with some details that need further improvement (see below)
- The proposed method seems technically sound
- The results show a clear improvement over prior work


### Weaknesses

#### Clarity

- Eq. 2 is not fully defined. What is the additional semantic feature classification loss exactly? What is its label space? Also the "traditional detection" losses need to be explained because this is a continual learning setup and not a "traditional" detection setup. Are boxes that RPN defines as "unknown" a separate class in the RoI classifier? Or are only the currently known categories used?

- Is there any further discussion about why three classifiers (traditional detection classifier, semantic feature classifier and the clustering itself is essentially also a classifier) are needed, besides the empirical ablation study? Some form of justification would be good.

- The last sentence of section 3.4.1 raises the question what the embedding is for the "unknown" class.


#### Related work

- How is the concept of "semantic anchors" different to approaches in zero-shot learning/detection that also encode the category names with a language model and use the embeddings as the classification/regression targets?  For example [A,B,C]

- Another aspect to consider in the related work are methods that try to combine multiple object detection datasets with different label spaces to create a larger unified label space, like [D,E].


#### Typos, minor comments and suggestions

- "has arisen" in abstract is maybe the wrong word? What about "has raised ... interest"?
- "open-set error" in abstract is unclear for the general reader I guess. I would express this in a different way or give a relative improvement.
- Typo: "making such detectors cannot handle"
- Is there a reference or a discussion to support the statement that the non-end-to-end ORE approach is "far from optimal"? This split into out-of-distribution detection and incremental learning seems natural.
- 3.2: I would write out "SA Head", I assume it means "Semantic Anchor Head".
- Why not use cosine similarity between RoI feature embedding and text embedding and do a cross-entropy loss over the similarities instead of Eq. 1? That's similar to how CLIP aligned images and text.
- 4.1 What does "semantic drifts" mean?
- 4.2 The recall level R is not described what it does
- 4.4 Typo: "at the following task"
- 4.5 Meaning of "group of classes (10, 5 and the last class)" is unclear
- 4.6 Typo: "To be specific, ..."


#### References mentioned above

- [A] Zero-shot object detection. Bansal et al. ECCV'18
- [B] Open-Vocabulary Object Detection Using Captions. Zareian et al. CVPR'21
- [C] Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation. Gu et al. arXiv'21
- [D] Object Detection with a Unified Label Space from Multiple Datasets. Zhao et al. ECCV'20
- [E] Simple multi-dataset detection. Zhou et al. arXiv'21

summary_of_the_review: Overall, I think the paper presents a solid contribution to the task of open-world object detection. Some aspects of the paper need more clarification and some related work is missing that needs to be discussed.
review_title: null

Meta-review: This paper presents work on open-world object detection.  The main idea is to use fixed per-category semantic anchors.  These can be incrementally added to when new data appear.  The reviewers engaged in significant discussion around the paper with many iterations of improvements to the paper.  Initial concerns regarding zero-shot learning were addressed, as were remarks on presentation and claims.

In the end the reviewers were split on this paper.  I recommend to accept the paper on the basis of the semantic topology ideas and the thorough experimental results.

The remaining concern centered around the evaluation protocol used in the paper, which follows that in the literature (e.g. Joseph et al. CVPR 21).  While this is not a fatal flaw, it is an issue with how this genre of methods is evaluated.  It would be good to add discussion to the final paper to highlight this as an opportunity for future work in the field to address.  Specifically, as a reviewer noted "after detecting "unknown" objects in T1, the (hypothetical) annotation process provides boxes for ALL objects of some new classes instead of only for those that have been correctly detected (localized and marked "unknown")."
Decision: Accept (Poster)