Objects in Semantic Topology
Authors: Shuo Yang
Peize Sun
Yi Jiang
Xiaobo Xia
Ruiheng Zhang
Zehuan Yuan
Changhu Wang
Ping Luo
Min Xu
Venue: ICLR 2022 Poster (2022)

Abstract: A more realistic object detection paradigm, Open-World Object Detection, has arised increasing research interests in the community recently. A qualified open-world object detector can not only identify objects of known categories, but also discover unknown objects, and incrementally learn to categorize them when their annotations progressively arrive. Previous works rely on independent modules to recognize unknown categories and perform incremental learning, respectively. In this paper, we provide a unified perspective: Semantic Topology. During the life-long learning of an open-world object detector, all object instances from the same category are assigned to their corresponding pre-defined node in the semantic topology, including the `unknown' category. This constraint builds up discriminative feature representations and consistent relationships among objects, thus enabling the detector to distinguish unknown objects out of the known categories, as well as making learned features of known objects undistorted when learning new categories incrementally. Extensive experiments demonstrate that semantic topology, either randomly-generated or derived from a well-trained language model, could outperform the current state-of-the-art open-world object detectors by a large margin, e.g., the absolute open-set error (the number of unknown instances that are wrongly labeled as known) is reduced from 7832 to 2546, exhibiting the inherent superiority of semantic topology on open-world object detection.

Published as a conference paper at ICLR 2022
OBJECTS IN SEMANTIC TOPOLOGY
Shuo Yang' ! Peize Sun? Yi Jiang' Xiaobo Xia
Ruiheng Zhang
Zehuan Yuan Changhu Wang Ping Luo² Min Xu
University of Technology Sydney
"The University of Hong Kong
ByttDDance Al Lab *University of Sydney Beijing Institute of Technology
ABSTRACT
A more realistic object detection paradigm, Open- Open-World Object Detection, has arised increasing research interests in the community recently. qualified open- world object detector can not only identify objects of known categories, but also discover unknown objects, and incrementally learn to categorize them when their annotations progressively arrive. Previous works rely on independent modules to recognize unknown categories and perform incremental learning, respectively. In this paper, provide unified perspective: Semantic Topology. During the life-long learning of an open-world object detector, all object instances from the same category are assigned their corresponding pre-defined node in the semantic topology, including the 'unknown' category. This constraint builds up discrim- inative feature representations and consistent relationships among objects, thus enabling the detector to distinguish unknown objects out of the known categories, as well as making learned features of known objects undistorted when learning new categories incrementally. Extensive experiments demonstrate that semantic topology, either randomly- gegeerrtd or derived from a well-trained language model, could outperform the current state-of-the-art open-wotd wrendworld object detectors by a large margin, egg., the absolute open-set error (the number of unknown instances that are wrongly labeled as known) is reduced from 7832 to 2546, exhibiting the inherent superiority of semantic topology on open-world object detection.
INTRODUCTION
Object detection, which aims at localizing and classifying objects in a given scene (Felzenszwalb eet al. 2010 Everingham et al. 2010 Lin al.) 2014), is one of the most iconic abilities of biological intelligence. was introduced to the artificial intelligence field to endow an intelligence agent with the ability of scene understanding. Although significant advances have been made improve the object detection system in recent years (Girshick et al., 2014 Ren et al., 2015 Cai & Vasconcelos/
2018 Sun al. 2020b Redmon et al. 2016: Lin al. 2017 Tian al. 2019: Zhou et al.) 2019a Carion et al. 2020 Sun et al.| 2020a), strong assumption that all the objects of interest have been annotated in the training set, i.e., close-set learning, is always made but holds well. All unknown objects are treated as background and ignored by current detectors, making such detectors cannot handle corner cases in many real-world applications such as autonomous driving, where some obstacles are not available during training, but must be detected when intelligent cars are running on the road. However, creating a large-scale dataset that contains annotations for all objects of interest at once is extremely expensive, even impossible.
Superior to current detectors, humans naturally have the ability to discover both known and unknown objects and gradually learn novel concepts motivated by their curiosity. Learning by discovering the unknown is crucial for human intelligence (Livio) 2017 Meacham, 1983), ) and has been considered as key step to achieve artificial general intelligence (AGI) (Goertzel] 2014). Recently, new object
detection paradigm, named Open- World Object Detection, has been established (Joseph et al. 2021
Miller al. 2021: 2018c: Liu al. 2020
mimic this learning procedure.
qualified open-world object detector can not only identify known objects, but also discover object instances of unknown categories and gradually learn to recognize them when their annotations
corresponniig author

---

Published as a conference paper at ICLR 2022
aM 6.
(a) Training on five categories.
(b) Four new categories are introduced.
Figure 1: t-SNE visualization of object features in ORE. The location of previously-kne known (the first five
classes) object features severely distorted when learning new categories.
-.0
(a) Training on five categorics.
(b) Four new categories are introduced.
Figure 2: t-SNE visualization of object features with proposed semantic topology. Both previously- known categories and novel categories are binded their corresponding nodes on semantic topology. maintains the previously-kne -noow category feature topology when learning novel categories.
progressively arrive. The learning of novel categories always in an incremental fashion, where the detector cannot access all old data when training on new categories. This open-world learning setting much more realistic but challenging than previous close-set object detection.
The open-world object detection poses two challenges current detectors, i.e., recognition of unknown categories and incremental learning. the one hand, previous close-set detectors do not explicitly encourage intra-class compactness (Liu et al., [016bb Yang et al., 202Tb). However, compact feature representation is highly required for unknown object recognition (discovery). the known categories occupy most of the feature space, the detector will probably classify the unknown object as one the known categories. On the other hand, the vanilla training strategy the object detector lacks the mechanism prevent 'catastrophic forgetting" in incremental learning (Joseph) et al. 2021 [Shmelkov et al., 2017a Peng et al.] 2020), i.e,, previously-knowe objects features are severely distorted when learning new categories. Consequently, training novel categories weakens the detector's 's ability to detect previously-knowe objects.
Previous works have made efforts to endow the object detector with the capacity of unknown recognition and incremental learning. One of the representative works, ORE (Joseph et al. 2021). designs clustering loss function compact the object features, and involves an energy-based out- of-distribution identification approach (Liu et al. (2000) to detect unknown objects. Combining these two independent technologies step-wise manner makes ORE non-end-to-ens framework. Such solutions for open-world object detection are far from the optimal status. Additionally, ORE (Joseph) al. 202T) doesn't it guarantee feature space topology consistency, which crucial for effective new class learning and avoidance of catastrophic forgetting, shown in Figure [
This paper formalizes unknown recognition and incremental learning proposes much simpler but more effective framework for open- world object detection than prior arts. We propose that an open-world object detector desired to learn a feature space with such characteristics, including (a) discriminativeness: the unknown objects and the objects of new cat- egories couldn't it be overlapped with any grviously-knoww categories the feature space, and (b) consistency: the feature topology of previously worioouusl-kkownnn categories couldn't 't be distorted severely when
unified perspective and
2

---

Published as a conference paper at ICLR 2022
Unknown Aware RPN
box reg.
chssilication
Backhone
Feature
claseifiatiio
Semantic Prepeturr
clustering
semantic anchors
poicd Star:
semantic anchors
lovrtng
novel cless features
PONE
unknownn semamicc anchors
anknown class fentures.
(A) space
Regiateting sertantic classes
Leaming novel classes and stahiliiing feature topolagy
Figure 3: Illustration of novel classes, e.g., chair, and refrigerator, introduced to the open- object detector at specific time point during the training procedure. Each node semantic topology. termed as a semantic anchor, pre-defined by sondoolly-gneratee vector or derived from a well -rainer lan- guage model before starting the training procedure. When the detector learns novel categories, the corresponding semantic anchors registered to the semantic topology firstly, then object features of the same category constrained to close its semantic anchor. At the inference stage, the Rol feature classifier and the semantic feature classifier are ensembled to make predictions.
learning new categories. Our key idea is to pre-define unique and fixed centroid in feature space for each category, including the 'unknown' category, and to push object instances close to their belonging centroids during the life-long learning of an open-world object detector. The pre-defined centroids named 'Semantic Anchors', and all semantic anchors constitute the structure of 'Semantic Topology'. . As shown in Figure 2 all features are binded to their corresponding semantic anchors to satisfy the discriminativenes. previously-knon known objects feature topology is maintained when incrementally learning new categories to satisfy the consistency.
We introduce off-the-shelf pre-trat pre-trained language model to set up the semantic topology. The se- mantic anchor for each category is derived from the language model by embedding the corresponding semantic anchors when the semantic topology gradually grows. In the experiments, we show that by combining current detector with our proposed semantic anchor head, huge improvement in open-world object detection performance across the board is achieved. In addition to the consistent mAP improvement over the whole life of the detector, the ability of unknown recognition of our proposed method outperforms the current state-of-the-ard methods by large margin, e.g., the absolute open-set error is reduced by 2/3, from 7832 to 2546.
category name. By incrementally registering
classes involved,
More importantly, we conduct comparison experiment by randomly generating semantic anchors instead of leveraging language models. Although mondomly-generann anchors do not provide semantic priors, their performance still surpasses state-of-the-art methods. This strongly proves that the topology consistency is the most key characteristic for open-world learning while introducing semantic relationships can further boost the performance.
RELATED WORKS
Object detection
Modern object detection frameworks (Ren et al. 2015:
Redmon et al. 2016 Liu et al. 201 6a; Lin
ct al. 2017 Zhang et al. 20 Redmon et al.
2016 Redmon & Farhadi 201 8018 Girshick 22055 Duan et al. 2019 Tian et al. 2019 Tan et al. 2020 Zhou et al. 201 9b; Jiang et al. 2018 Zhang
et al. 2019b Carion et al. 2020 Sun et al. (020bb take advantage of the high-capacity representation in deep neural networks to localize and classify the target class object in given images and videos.
These eellbistall detectors can achieve excellent performance in the close-set dataset such as
PascalVOC (Everingham et al. 2010), MSCOCO (Lin et al., 2014). However, the detectors can
3

---

Published as a conference paper at ICLR 2022
not handle open-wwe worndworld object detection, which is more common in the real world. To this end, ORE (Joseph et al. 2021) raises and formalizes the open-world object detection problem.
Class incremental learning Class incremental learning aims to learn classifier incrementally to recognize all encountered classes met so far, including previously-knon -nowwn classes and novel classes. Knowledge distillation is commonly adopted to mitigate forgetting old classes, which stores some old class exemplars to fine-tune the model or compute the distillation loss. iCaRL (Rebuffi et al.) 2017) maintains an "episodic memory" of the exemplars and incrementally adds novel class examples into the memory. Then the nearest neighbor classifier can be obtained incrementally. LwF (Li & Hoiem 2017) proposes a modified cross- cross-entropy loss to preserve the knowledge the previous task. BiC Wu et al. 2019) points out the data imbalance between old classes and new classes causes the network's ' prediction biased towards new classes. However, the existing class incremental methods cannot handle the open-world problem, where the classifier should identify those 'unknown" classes and incrementally learns to recognize them, but existing methods would recognize unknown objects as background.
Open-set Learning In open-set learning, the knowledge contained in the training set is incomplete, Le., the examples encountered during inference may belong category that does not appear in the training set. OpenMax (Bendale & Boult 2016) uses a Weibull distribution to identify unknown instances in the feature space of deep networks. OLTR (Liu et al. 2019) proposes to tackle the open-set recognition problem in a data imbalance setting by deep metric learning. Besides the open-set classification, Dhamija et al. (Dhamija et al.)
2020b) found that the object detectors exhibit a high error rate that misclassifies unknown classes to known classes with high confidence. To solve this problem, many works (Miller et al., 2021: 2018a| Dhamija et al., (0200b aims measure the uncertainty of the detectors' outputs to reject open-set error. Miller el al. (Miller et al. 2018a) uses Monte Carlo Dropout sampling to estimate the uncertainty in an SSD detector. After that, Miller et al. proposes to model a Gaussian Mixture distribution for each class in the detector's 's feature space to reject unknown classes. ORE (Joseph et al. 202T) uses an energy-based out-of-distribution recognition method (Liu et al.) 2020) to distinguish the known and unknown. However, such method needs to compute the energy score distribution among all instances, including known and unknown, during evaluation, making ORE non-end-to-ene method. We formulate the open-set recognition problem and the class incremental learning into a unified framework. Different from (Joseph et al.. 2021). our method doesn't ' access any unknown instances but achieves much superior performance.
Zero-shot Learning Aligning image and text into a common feature space has always been an
active research topic (Frome et al. 2013]] JJuulin al. 2016 Li et al. 2017 Desai & Johnson 2021
Yang et al.] 2019: 2021a), especially in zero-shot learning (Xian et al. 2017 Bansal et al. 2018
Zareian et al. 2021 [22]/un Gu al. 2021 Many researches in zero-shot learning leverage the information
in language models to assist zero-shot image classification (Xian et al.) 2017) or zero-shot object detection (Bansal et al. 2018 Zareian al. 202T Gu al. (021). However, this paper tackles a different problem and has different usage of language models. We identify that consistent feature manifold topology plays an essential role in open-world object detection, and the language model is used for generating growth-able and consistent semantic topology to constrain the feature space learning of an open-world detection. Also, this paper the first one to incorporate a language model, e.g., CL.IP, to assist open-world object detection, which results in simple training framework and strong empirical performance.
3 METHODOLOGGY
In this section, we introduce the Open World Object Detection problem definition in Section 3.1 the method overview in Section 3.2 and the details of the proposed method in Section 3.3 and
Section 3.4
3.1 PROBLEM DEFINITION
An open world object detector should detect all the previously seen object classes, and can also identify if a testing instance known unknown (belongs to previously seen classes or not). If unknown, the detector should gradually learn the unknown classes when their annotations progressively arrive without retraining from scratch. Formally, each time point object classes Ci. = 12..2.... 4c} and a set of unknown object classes and = flo+1, lo+2..... The detector D. at the time point has only trained classes in Cf while may encounter all classes
assume there exists set of known
4

---

Published as a conference paper at ICLR 2022
including Cin. and Cyk. during evaluation. Besides correctly classifying an object instance from the known classes Cap. the detector Dr should also label all instances from the unknown class set ween unknown. At the time point t 1, the unknown instances will be forwarded to a human can select n novel classes of interest to annotate and return them back to the model. The detector Dr should incrementally learn these novel classes and updates itself to D++1 without retraining from scratch on the whole dataset. The known class set Ckz in the time point + is also updated to C++11 Cha, + (1c+1,-.... -C+n). The data instances in the known classes Ckn assumed to be labeled in the form of {a, y), where r indicates the image and y indicates the annotations including class label l and object coordinates, i.c. y = [1, x. y, w, h] where 2, y, w, h denote the bounding box center coordinates, width and height respectively.
who
3.2 METHOD OVERVIEW
A Region Proposal Network (RPN) that can identify unknown and a discriminative and consistent feature space are two critical components for an open world detector. Here, we adopt the Unknown- Aware RPN proposed in (Joseph et al. (221)) and propose to constrain the detector's s feature space topology with a pre-defined Semantic Topology. Specifically, we create a unique and fixed centroid for each category in the feature space, named semantic anchor. The semantic anchors for all classes are derived by feeding forward their class names into a pre-trained language model. Our key idea is to manipulate the detector's feature space be consistent with the semantic topology constituted by semantic anchors during the whole life of the detector. Due to the feature dimension discrepancy, a fully connected layer (semantic projector) is used to align the dimension between Rol features and semantic anchors. At the training stage, the semantic features outputted by the semantic projector are forced to cluster around their corresponding semantic anchors by designed 'SA (semantic anchor) Head'. When incremental learning, the SA Head gradually registers new semantic anchors for novel classes and continually pulls close the novel class features and their semantic anchors. To mitigate 'catastrophic forgetting' caused by old class feature distortion, the SA Head also minimizes the distance between some stored old class exemplars and their semantic anchors when learning novel classes. To better leverage the well-constructed feature space, we attach an additional classification layer to classify the semantic features. Figure 3 shows the training pipeline of the proposed open- world object detector. At inference, we multiply the class posterior probabilities produced by the two classification heads and get the final prediction.
3.3 UNKNOWN-AWARS RPN
Open-world object detectors are required separate potential unknown objects from the background. Therefore, we need some specific designs for the Region Proposal Network (RPN). In this paper, we adopt the unknown-ware RPN proposed in (Joseph et al. 202T) which selects the top-k background region proposals, sorted by its objectness scores, as unknown objects. The unknown-aware RPN relies on the fact that Region Proposal Network is class agnostic. Given an input image, the RPN generates bounding box predictions for foreground and background instances, along with the corresponding objectness scores. The unknown-aware RPN labels those proposals with a high objectness score but do not overlap with any ground-trui -rrnthhhtthhhh object potential unknown object.
3.4 SEMANTIC TOPOLOGY
We propose to pre-define a semantic topology for detectors' feature space rather than learn from data. The semantic topology is constituted by semantic anchors. Each semantic anchor is a pre-defined feature centroid for an object class. The semantic anchors can be generated by embedding corre- sponding class names using a pre-trained language model. The semantic topology can dynamically grow as novel classes are introduced to the open-world detector.
3.4.1 SEMANTIC ANCHOR REGISTRATION
We generate semantic anchors for all classes by feeding the class names into an off-the-shelf pre- trained language model. Denote I; E C₁2, class name of the i-th known class at time & and M as an off-the-shelf pre-trained language model. The semantic anchor for class I is defined as A, = (l,). where A, E R", the dimension depends on the pre-trained language model. The semantic anchor registration is performed repeatedly as long as known class set update C - C!+11 when novel classes are introduced at time + 1. Note we also register semantic anchor for all instances labeled
5

---

Published as a conference paper at ICLR 2022
unknown by the unknown-aware RPN to better distinguish the unknown instances from the known instances. Follows the same strategy, the semantic anchor for class unknown is also generated by the word embedding of unknown text.
3.4.2 OBJECTIVE FUNCTION
The Rol features R2 are feature vectors generated by an intermediate layer of the object detector, which are used for category classification and bounding box regression. We manipulate the Rol features f to construct the detector's feature manifold topology. Denote f. as an Rol feature of the i-th known class, we first align the dimension of f; as the dimension of its corresponding semantic anchor A; using a fully connected layer with d n dimensional weights. The corresponding semantic feature is denoted as fi IR, We constrain the detector's feature manifold topology by clustering the semantic features around their corresponding semantic anchors, the learning objective is formalized
as
Le llfi Aill-
(1)
Minimizing this loss would ensure the desired feature space topology. better leverage the constructed feature space, we use an additional classification head to classify the semantic features ft. with the same label space Rol classification head. The total training objective is the combination of semantic anchor loss L.., semantic feature classification loss Lels.s> Rol feature classification loss and bounding box regression loss:
Liotal La + Lels.o + Lets freg-
At the inference stage, the classification results computed by multiplying the two class poste- rior probability vectors predicted by the Rol feature classification head and the semantic feature classification head.
3.4.3 TOPOLOGY STABILIZATION
To store the detection ability old classes, balanced set of exemplars are stored and used to fine-tune the model after each incremental learning session as in the previous open-world object detection upproach (Joseph et al., (02T). However, we argue that finetuning the detector ignoring the old knowledge topology as in (Joseph et al.) 2021) still suffers from the severe 'catastrophic forgetting' problem. Benefitting from pre-defining the feature space topology, our method can guarantee consistent feature space during the finetuning stage. The stored old class and new class instances are still forced to cluster around their pre-defined centroids to guarantec the feature space topology unchanged.
EXPERIMENTS
We introduce the evaluation protocol, including datasets and evaluation metries, implementation details, and experimental results in this section.
4.1 DATASETS
Following (Jsephh et al. 2021 the open- open-world detector is evaluated on all 80 object classes from Pascal VOC (Everingham et al. 2010) (20 classes) and MS- (Lin al., (014) (20+60 classes). All categories are grouped into set of tasks T (Ti.... , .T....... where all categories of T, will be introduced to the detector at a time point At the time point i, all categories from [T <=
will be treated as known and {T, |T > t} would be treated as unknown. As in (Joseph et al. 202 1). Ti consists of all VOC classes, and the remaining 60 classes from MS-COCO are grouped into three successive tasks with semantic drifts. The open-world object detector is trained the training set of all classes from Pascal VOC and MS-COCO, and evaluated on the Pascal VOC test split and MS-COCO val split. The validation set consists of Ik images from the training data of each task.
4.2 EVALUATION METRICS
We introduce three metrics to evaluate the detection performance of an open open-world object detector on known classes and unknown classes at each time point:

---

Published as a conference paper at ICLR 2022
(->)
Tisk
Tisk
Task
WI
|A-OSE
mAP (T)
WI
A-OSE]
mAP (1)
WI
A-OSE
MAP (1)
mAP
Curren
known
())
Previously
known
Curren
known
Both
(
Previously
known
Curent
known
Both
Proviously
known
Current
known
Both
Faster R-CNN 0.0666]
13286
55.95
[0.0492
9881
5.29
25.36
15.32
(0.023)
9294
6.09
13.53 8.570
1.98
13.95
4.97
Faster R-CNN
Finetuning
0.06161
13286
55.95
0.0523
11913
51.07
23.84
37.46
0.0288
9622
35.39
11.03
27.21
29.06
12.23
24.85
ORE
0,047
7995
56.02
[0,0297
7832
52.19
25.03
38.61
(0,0218
6900
37.23
12.02
28.82
29.53
13:09
25,42
Ours
0.0417
4889
56.20
[0.0215
2546
53.39
26.49
39.94 0.0146
2120
38.04
12.81
29.63
30.11
13.31
25.91
Table 1: Comparisons of different methods on Open World Object Detection. Wilderness Impact (WT) and Absolute Open Set Error (A-OSE) are both the less the better, which measure the ability on unknown identification. Our proposed method achieves consistent performance improvement during the whole life of the detector, surprisingly achieves much lower A-OSE compared with baselines and previous state-of-the-ar methods.
Mean Average Precision (mAP) [Everingham et al. 2010 Lin ct al. 2014). Following previous works (Joseph et al., 202T), , we use mAP IoU threshold of 0.5 to measure the performance on known classes. Since the object classes are continually involved, mAP is calculated on previously- known classes and currently- knnow classes.
Absolute Open-Set Error (A-OSE) (Joseph et al.) 2021 Miller et al. 2018b). A-OSE counts the absolute number of unknown instances that wrongly classified as any of the known classes. A-OSE evaluates the detector's ability to avoid misclassifying unknown instances as one of the known classes.
Wilderness Impact (WT) (Joseph et al., 2021 Dhamija et al., 2020ap. The definition WI
Pk_ 1, where Pk the precision of the detector when evaluated on known classes and Pkuu Pkuu
refers to the precision when evaluated on all classes including known and unknown. The recall level R is set to be 0.8. WI evaluates the detector's 's ability to successfully detect unknown objects and classify them as 'unknown' class.
4.3 IMPLEMENTATION DETAILS
We extend the traditional Faster R-CNN (Ren et al. 2013) object detector with ResNet-50 (He ee al. 2016) to be an open-world object detector. The Rol feature is extracted from the last residual block in the Rol head, which has 2048 dimension. The 2048-dim Rol feature is used for computing traditional bounding box regression loss and classification loss as the same as in many previous works (Joseph et al., 202 Ren et al.) 2015)). The semantic projector is fully-connected layer to align the dimension of Rol features with the semantic anchors. The dimension of semantic anchors depends on the choice of pre-trained language model, e.g., the semantic anchor is 512-dim when using CLIP text encoder (Radford et al., (021b) to embed categories names. The semantic feature outputted by the semantic projector is used for calculating semantic anchor loss L.o. and semantic feature classification Lete,, loss. The L.o and the Leta,e are added to the standard regression loss and Rol classification loss to jointly optimize the detector. For the topology stabilization, we store 100 instances per class as the same in (Joseph et al. (021I). To enable the classifier to handle variable number of classes at different time points, we assume there exists a maximum number of classes to be added and set the classification logits of unseen classes large negative value make their contribution to softmax output negligible, following (Joseph & Balasubramanian 2020 Rajasegaran al. 2020 Lopez-Paz & Ranzato 2017 Chaudhry al. 2018).
To make a fair comparison, we report experimental results obtained by re-running the ORE (Joseph) al. 2021) official codel] All the hyper-parameters and optimizers are also controlled be exactly the same for all methods.
4.4 OPEN-WORLD OBJECT DETECTION
Table [ shows the open-world object detection performance of our proposed method and other baselines. At task 1, all methods are trained over all 20 Pascal-VOC classes. Twenty novel object classes are introduced gradually at the following task. WI and A-OSE are used to quantify how unknown instances are confused with known class objects after training on each task. WT and A-OSE are both the less the better. ORE (Joseph et al. (021) used an extra clustering loss to separate class features, and an eaergy-based unknown identifier that learns shifted Weibull distributions from
Officiil repository: http:://jihhbbccoom//spp////oo/
7

---

Published as a conference paper at ICLR 2022
ORE (TT)
ORE (T4)
Ours (TI)
Ours (T4)
Figure 4: Visualization of ORE and our proposed method after training on Task 1 (T1) and Task (T4). At task 1, ORE frequently misclassify unknown object as one the known object classes. After training on task 4, ORE successfully detects novel classes but forgets objects learned on task 1. By explicitly introducing semantic prior into the detector to constrain the feature space topology, our proposed method performs favorably in open-world object detection.
all class data (including known and unknown) in the validation set to reject open-set error. Our method doesn't 't access any unknown data instances from the validation set and achieves much superior performance WI and A-OSE than Faster R-CNN and ORE. In task 3, proposed method achieves 1/3 A-OSE compared to ORE (2120 v.s. 6900) and 1/4 A-OSE compared to Faster R-CNN (2 (2120 v.s. 9622). Our method also reduces the Wilderness Impact (WI) by a large margin than baselines. This indicates our method has a much better ability on unknown detection. Furthermore, our method also achieves consistent mAP improvement over the whole life of the detector. At incremental learning session (task 2,3,4), 4), the performance 6.09). Faster R-CNN Finetuning, ORE, and our method store some old class instances and fine-tune on them after learning new classes to restore the old class performance. However, the Faster R-CNN Finetuning and ORE don't guarantee the feature manifold topology consistent. The superior performance on old classes also proves that guaranteeing the feature manifold topology is critical.
Faster R-CNN on old classes deteriorate severely (55.35
4.5 INCREMENTAL OBJECT DETECTION
To verify the effectiveness our proposed method, we also conduct experiments on class incremental object detection (iOD), where the detectors only need to continually recognize novel object classes without unknown identification. We use the standard iOD evaluation protocol as (Joseph et al. 2021 Shmelkov et al. 2017b) to evaluate all methods, where group of classes (10, 5, and the last class) from PASCAL VOC 2007 (Everingham et al.) 2010)) are incrementally Icarned by a detector trained on the remaining set of classes. Benefits from the well-constructed feature space using semantic anchors, our proposed method performs favorably well on the incremental object detection (iOD) task against several state-of-the -f-thhe-rt(2] Notably, our method exhibits strong ability to incrementally learn new classes while not forget old classes. This is because detector's feature space is constrained to be incrementally growth-able by assigning each object class a unique feature space location.
4.6 ABLATION STUDY
The effect of language model. We derive semantic topology from pre-trained language models. To explore the effect of the choice of semantic topology, we conduct ablation studies in this section. To be specific, text (Radford et al. 2021a) and BERT (Devlin et al.] 2019). The dimensions of semantic anchor generated by CL.t and BERT are 512 and 768, respectively. The comparison results shown in
generate semantic anchors using two pre-trained language models, í.e., CLIP-
8

---

Published as a conference paper at ICLR 2022
Task Settings (++)
10 + 10
+
19+1
(T)
(T)((T)
MAP (T)
clases
new classes
both
old classes
new classes
both
Did classes
new classes
both
Joint train
70.80
70.20
70.50
72.10
65.72
70.51
70.52
70.3
70.51
ILOD (Shnelkov 7017a
63.16
63.14
63.15
58.44
65.87
68.54
1L0D Faster R-CNN
67.33
54,93
61.14
69.24
57.56
66.35
67.81
65.1
67.72
Faster 11.01 (Peng et al. 2020
69.76
54.47
62.16
71.56
56.94
67.94
68.91
61.1
68.56
ORE Joseph al.
58.37
68.70
63.53
71.44
68.95
Ours
60.03
69.88
64.96
73.01
60.69
69.93
70.22
62.30
69.82
Table 2: Comparison of different methods on class incremental object detection. In the three task settings, 10, 5, and the last class from the Pascal VOC 2007 (Everingham et al. 2010) dataset are introduced to a detector trained on 10, 15, and 19 classes respectively.
T
WI |A-OSE] MAP (?)
WT
A-OSE]
(1)
WI A-OSE]
mAP (f)
mAP (f)
Current
(
(1)
Previously
Current known
Bath
Previously
Current
Both
Previously
Current
Both
known
known
known
known
known
known
ORE
(.047]]
7995
56.02
0.0297
7832
32,19
25.03
38.61 0,0218 6900
37.23
12,02
28.82
29.53
13.00 25:42
Random
0.0433|
5331
56.13
0.0246
2779
52.56
25,86
39.21
0.0183
3742
37.69
12.33
29.23
29.31
12.98
25.22
BERT
56.20
0.0222
2772
25.84
0.0153
37.92
29.52
30.07
13:29
25.87
CLIP
0.6417|
4889
56.20
0.0213
2546
33.39
26.49
39.94
0.0146
2120
38.04
12.11
29.63
30.11
13:31
25.91
Table 3: Ablation on semantic anchor generation. Semantic anchors derived from CLIP-text and BERT obtain the similar performance, both outperforming ORE, Surprisingly, semantic anchors from random vectors also achieve better result than ORE.
Task IDs (++)
Task I
Task 2
Tusk
Task 4
WI
O-ESE mAP
WI [A-OSE]
mAP
WI
O-OSE]
mAP (t)
mAP (f)
(1)
(1)
Carrent
known
(1)
(1)
PPrevionsly
known
Current
known
(1)
(3)
Previoudy
known
Camrent
known
Both
Provicesly
knowns
Current
known
Both
wlo C.s
0.0641
13097
55.98
[0.0317]
12564]
51.03
23.97
37.50
(0.000
9598
35.21
11.13
27.18
28.87
12.17 24.69
w/o unknown anchor 0,0476
6032
56.19
[0.0270]
3354
52.36
25.26
38,
|0.0193
4692
37.79
12.63
29.00
29.42
13.17
25.35
wdo Cat..
0.0179
12961
55.83
[0.0290]
11297
52.37
26.43
39.40
(0.0237
8713
35.46
11.33
27.41
28,91
12.34
24.76
wio Gav...
0,0428
$301
[0,0233]
2896
$3.40
39.92
J00162
2004
12.77
29.64
13.28
25.55
our full model
0.0417
4889
56.20
[0.0213]
2546
$3.39
26.49
39.9400.0166
2130
38.04
12.81
29.63
30.11
13:31
25.91
Table 4: Ablation on unknown anchor and loss components.
Table These two language models obtain similar results, which both outperform ORE, the previous state-of-the-arr method in open-world object detection. To further explore the importance of semantic priors, we also generate semantic anchors by randomly and uniformly sampling, which means we assume the total number of classes is known which not applicable in practice. Surprisingly, our experiment shows that semantic anchors as random vectors still surpass state-of thee--f-theerrtttt methods. This demonstrates that a consistent topology, even without semantic priors, could also largely benefit the task of open-world object detection, which strongly proves that the discriminative and the consistent are two key characteristics for open-world learning. However, the randomly assigned class centers would hinder the learning ability of networks. Instead, a pretrained language model can perfectly overcome these problem.
The effect of semantic anchor loss. In this section, explore the importance of unknown class anchor, semantic anchor clustering loss Lea, semantic anchor classification loss Lets,, and the Rol feature classification loss Lets.oi- The experiments in Table shows the semantic anchor clustering loss and the semantic anchor classification loss are both indispensable in identifying unknown objects.
CONCLUSION
Open world object detection raises two challenging problems on current detectors, unknown recogni- tion and incremental learning. Different from previous methods which combine domain-dependent technologies to respectively solve these two problems, our work provides a unified perspective, iee.emantic topology. In the framework of semantic topology, all object instances from the same category, including 'unknown' category, are assigned to their corresponding pre- pre-defined centroids. Therefore, a discriminative and consistent feature representations an open-world object detector. Aided with our proposed semantic topology, a huge improvement in open-world object detection performance across the board is achieved.
ensured during the whole life of

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENT
Ping Luo was supported by the General Research Fund of HK No.27208720 and the HKU-TCL Joint Research Center for Artificial Intelligence.
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. CoRR, Abs/1804.04344 2018.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In CVPR, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End object detection with transformers. In ECCV, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint
2018.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACI-HIT, 2019.
Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In WACV, 2020a.
Akshay Raj Dhamija, Manuel Günther, Jonathan Ventura, and Terrance E. Boult. The overlooked elephant of object detection: Open set. In WACV, 2020b.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. CenterNet: Keypoint triplets for object detection. In /CCV, 2019.
Mark Everingham, Luc. Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJC,, 88(2):303-338, 2010.
Pedro Felzenszwalb, Ross Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part based models. T-PAMI, 3209):627-11455, 2010.
Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semante embedding model. 2013.
Ross Girshick. Fast R-CNN. In ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Ben Goertzel. Artificial general intelligence: Concept, state of the art, and future prospects. J. Artif. Gen. Intell., 2014.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. CoRR, abs/2104.13921, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, 2018.
K J Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In CVPR, 2021.
10

---

Published as a conference paper at ICLR 2022
KJ Joseph and Vineeth N Balasubramanian. Meta-coonoliidttion for continual learning, arXiv preprint arXiv:2010.00352. 2020.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, European Conference on Computer Vision, pp- 67-84. Springer, 2016.
Ang Li, Allan Jabri, Armand Joulin, and Laurens der Maaten. Learning visual D-grams from web data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4183-4192, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting, 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar, Focal loss for dense object detection. In ICCV, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distributien detection. NeurIPS, 2020.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, volume 2. pp. 7, 2016b.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale tailed recognition in open world. In CVPR, pp. 2537-2546, 2019.
Mario Livio. Why?: What makes us curious. Simon and Schuster, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. NeurlPS, 2017.
John Meacham. Wisdom and the context of knowledge: Knowing that one doesn't 't know. On the development of developmental psychology, 8:111 1-114,,34, 1983.
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In ICRA, 2018a.
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. ICRA, pp. 3243-3249. IEEE, 2018b.
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In ICRA, 2018c.
Dimity Miller, Niko Sünderhauf, Michael Milford, and Feras Dayoub. Uncertainty for identifying open-set errors in visual object detection, 2021.
Can Peng, Kun Zhao, and Brian C. Lovell. Faster ILOD: incremental learning for object detectors based on faster RCNN, Pattern Recognit. Lett., 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00 103000000 2021b.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incrmeental task-agnostic meta-learning approach. In CVPR, pp. 13588-13597 2020.

---

Published as a conference paper at ICLR 2022
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning, 2017.
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. arXiv preprint
arXiv:1804.02700 02767, 2018.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: real-time object detection. In CVPR, 2016.
Unified,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In ICCV, 2017a.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In ICCV, Oct 2017b.
Peize Sun, Yi Jiang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Onenet: Towards end-to-end one-stage object detection. arXiv preprint arXiv:2012.05778, 2020a.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. Sparse r-cnn: End-to-end object detection with learnable proposals. arXiv preprint arXiv:2011.12455 2020b.
Mingxing Tan, Ruoming Pang, and Quoc V. Le. EfficienDet: Scalable and efficient object detection. In CVPR, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In ICCV, 2019.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning the good, the bad and the ugly. In CVPR, 2017.
Shuo Yang, Wei Yu, Ying Zheng, Hongxun Yao, and Tao Mei. Adaptive semantic-visual tree for hicrarchical embeddings. ACM MM, Oct 2019.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021a.
Shuo Yang, Min Xu, Haozhe Xie, Stuart Perry, and Jiahao Xia. Single-view 3d object reconstruction from shape priors in memory. In CVPR, 2021b.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open- object detection using captions. In CVPR, 2021.
Hongkai Zhang, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cascade RetinaNet: Maintaining consistency for single-stage object detection. BMVC, 2019a.
Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye. FreeAnchor: Learning to match anchors for visual object detection. In NeurIPS, 2019b.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850, 2019a.
Xingyi Zhou, Jiacheng Zhuo and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019b.
12


Reviews:
[{"review_id": "yRqIfqtEYxG", "reviewer": "Reviewer_pp7T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Marginal novelty. \nWhile the semantic topology scheme is somewhat new, this work is heavily built on the work of Joseph et al., CVPR’21. Many parts of this paper are recaps of Joseph et al.'s work. Furthermore, leveraging the embeddings of the pretrained language model is popular in the zero-shot learning literature, so I don't think the core idea is very original while it's interesting to see the effect of the simple method. \n\n- Predefined and rigid semantic topology.\nThe proposed semantic topology is rigid and not adaptive to the new object classes coming along. While it's surprising that it substantially improves the performance, this predefined and fixed topology is too simplistic and may not scale well to a larger number of classes. \n\n- Use of a pretrained language model.\nThe proposed semantic topology requires the use of a pretrained language model, which means an external knowledge source as well as the use of class 'word' information in training. In this aspect, the performance comparison to the previous work can be seen as not fair. While the authors provide the result with random anchors for semantic topology, the results are not complete and also make me intrigued. Since the random anchor assignment may organize the embedding space in a random manner, it may hinder the learning of a proper embedding space. It needs more careful investigation on this issue.  \n\n- Incomplete analysis.\nSome important studies on the effects of language models and semantic anchor losses are done only on task #2, which is strange. To see the consistency along with the incremental learning processes, the results need to be shown for all the tasks. In particular, the study on the random anchors needs more clarification and in-depth analyses. How exactly the random vectors are generated? If the random vectors are too close to each other, they may harm the performance, in particular, when the number of classes increases. Did the authors observe such effects? Did the author use a specific random assignment to make the anchors distant from each other?  \n\n- Missing details.\nUnknown-aware RPN in Section 3.3 is not clearly described. Is it trained or just borrowed from the work of Joseph et al.'s 2021? Since this looks like one of the core modules in this work, a more detailed description needs to be done for the paper to be self-contained. \n\n- Etc. \nThe title of this work is too general for readers to recognize what this work is about. I suggest changing the title and making it more informative, at least, including the term 'open-world detection'. \n", "summary_of_the_paper": "This paper proposes a semantic topology embedding for Open-World Object Detection (OWOD) where an object detector identifies objects of unknown classes and incrementally learns to classify them assuming that their annotations are progressively given by humans. To maintain discriminative and consistent relationships among object classes, the authors introduce a semantic topology for the feature space of the detector by constructing pre-deﬁned anchors for categories using a pretrained language model. During training, it enables the detector to distinguish unknown objects out of the known categories and also makes learned features of different classes undistorted during incremental learning. Eperimental results show that the semantic topology improves state-of-the-art open-world object detectors and help the open-world detectors preserve a discriminative and consistent feature space.", "summary_of_the_review": "While the proposed semantic topology is simple yet effective, outperforming the state of the art, this work has marginal novelty, clear limitations, and missing analyses. I'm on the borderline, slightly leaning toward rejection. I will make a final decision based on the authors' rebuttal responding to my critiques. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rmmshiFsqrX", "reviewer": "Reviewer_fqMP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n1. The idea to use semantic topology to enforce discriminative features that are consistent across incremental learning is an interesting direction for open set detection. The paper is well written and the ideas and experiments are presented with clarity.\n2. Experiments are thorough, including ablations that show the effectiveness of different model components. It is interesting to see that using an additional classification loss on a separate stream of clustered RoI object features leads to significant improvement in detection performance.\n\n#### Weaknesses:\n1. The proposed approach seems to show significant improvement in detecting unknown objects. However I have a few questions on Table 1 results:\n\n     (a) Wilderness impact values of ORE [A] in Table 1 is ~2 times that of the corresponding values in the [A], while all other results seem comparable. What is the reason for this? \n\n     (b) Task 2 Wilderness index for ORE [A] is shown as 0.2970. (Should it be 0.0297?) The number seems inconsistent, as it is an order of magnitude higher than other comparable results in the table(for task 1 and task3), and also the corresponding result from [A]. The same result is also shown in Table 3. \n     \n      (c) If (a) and (b) do point to some errors, it seems that there are significant gains in A-OSE but much less gain in WI when comparing with [A]. Could this be explained based on the data? \n\n2. The proposed approach is closely related to [A], which is also the main baseline. However, Fig 1 and Fig 2 show the learned representations using vanilla training strategies using Faster R-CNN vs. using semantic topology. Compactness of feature representation is also obtained in [A], so the figure tends to mislead the reader about prior work. It would be better to see a comparison of feature representations of semantic topology vs. that of ORE.\n\n#### Additional comments / Questions:\n\n1. Does semantic feature classification head also predict for the “unknown” class (unlike RoI feature classification)? It seems to be implied that way, but it is not made explicit in the paper. It would be useful to have a description on exactly how “unknown” objects are detected during inference. \n\n2. Formula of WI in subsection 4.2 is not correct (-1 should be a separate term, not in the denominator). Also, the citations for WI and A-OSE are swapped in that section.\n\n3. The model is designed such that features of unknown objects will change during incremental learning to cluster around predefined semantic anchors. It would be good to have more discussion on the nature and choice of unknown anchor in the semantic topology. For instance, what is a good choice for the unknown anchor? Does the choice of unknown anchor affect learning?\n\n[A] Towards Open World Object Detection, Joseph et al., CVPR 2021\n", "summary_of_the_paper": "The paper addresses the problem of open world object detection - a lifelong learning system where the object detector is required to detect objects belonging to all classes known so far and “unknown” objects. Unknown objects are annotated and available for training at a later stage. At a given time only a specific set of training data is available with annotations that involve all known classes so far, while all past training data are unavailable. \n\n\nThe proposed approach builds upon recent work [A], by incorporating a semantic topology in the object feature space (RoI features in Faster R-CNN). While [A] use contrastive clustering to group object features based on object category, and energy based out-of-distribution detection to detect “unknown” objects, the authors propose an end-to-end trainable approach. The proposed approach uses RoI features in two parallel streams, with one focused on clustering operation, while having object category prediction loss on both streams of features. Features are clustered around predetermined semantically meaningful object representations. Pre-trained word embeddings from a language model (CLIP) are used for this purpose. The use of pre-defined semantic anchors allows learned features of known objects to form consistent clusters during incremental learning. \n\n\nExperiments are conducted using 80 categories from PASCAL-VOC and MSCOCO datasets. The problem is designed as 4 sequential tasks each involving 20 object categories (same setup as in [A]). Detection performance is measured using mAP, as well as measures that focus on unknown object detection.\n\n[A] Towards Open World Object Detection, Joseph et al., CVPR 2021\n\n", "summary_of_the_review": "The proposed approach is significant and relevant in open world object detection, proposing an end-to-end approach that learns incrementally and can detect unknown objects. A key promise of the proposed approach over prior work seems to be the improved ability in detecting unknown objects. However, I have concern regarding the main results reported (see weakness 1), and some other questions. I am willing to raise my rating if the authors can address these.\n\n**[Edit after discussion period: ]** I thank the authors and the other reviewers for active engagement in the discussion. Important related work on zero shot learning was not cited in the preliminary version, which the authors included later on. While there is one point of concern in the paper claims (see below), I think this work provided a solid contribution to open-world detection problem. Hence I would like to raise my rating assuming the authors can make the necessary updates to fix the concern below.\n\n* It was brought to notice by a reviewer the discrepancy between the problem definition in Section 3.1 and the evaluation protocol. While the problem is defined to involve \"human user.. annotate (unknown instances) and return back to the model\" at each stage, the evaluation protocol involves a different training set involving the unknown instances at each stage. I believe both settings are equally relevant for open-world detection problem, and I would encourage the authors to update the claims to match with the actual evaluation protocol.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MlaYC04X5Ch", "reviewer": "Reviewer_JDQB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The paper deals with an interesting problem, which has clear practical relevance\n- Overall, the approach is well described and motivated, with some details that need further improvement (see below)\n- The proposed method seems technically sound\n- The results show a clear improvement over prior work\n\n\n### Weaknesses\n\n#### Clarity\n\n- Eq. 2 is not fully defined. What is the additional semantic feature classification loss exactly? What is its label space? Also the \"traditional detection\" losses need to be explained because this is a continual learning setup and not a \"traditional\" detection setup. Are boxes that RPN defines as \"unknown\" a separate class in the RoI classifier? Or are only the currently known categories used?\n\n- Is there any further discussion about why three classifiers (traditional detection classifier, semantic feature classifier and the clustering itself is essentially also a classifier) are needed, besides the empirical ablation study? Some form of justification would be good.\n\n- The last sentence of section 3.4.1 raises the question what the embedding is for the \"unknown\" class.\n\n\n#### Related work\n\n- How is the concept of \"semantic anchors\" different to approaches in zero-shot learning/detection that also encode the category names with a language model and use the embeddings as the classification/regression targets?  For example [A,B,C]\n\n- Another aspect to consider in the related work are methods that try to combine multiple object detection datasets with different label spaces to create a larger unified label space, like [D,E].\n\n\n#### Typos, minor comments and suggestions\n\n- \"has arisen\" in abstract is maybe the wrong word? What about \"has raised ... interest\"?\n- \"open-set error\" in abstract is unclear for the general reader I guess. I would express this in a different way or give a relative improvement.\n- Typo: \"making such detectors cannot handle\"\n- Is there a reference or a discussion to support the statement that the non-end-to-end ORE approach is \"far from optimal\"? This split into out-of-distribution detection and incremental learning seems natural.\n- 3.2: I would write out \"SA Head\", I assume it means \"Semantic Anchor Head\".\n- Why not use cosine similarity between RoI feature embedding and text embedding and do a cross-entropy loss over the similarities instead of Eq. 1? That's similar to how CLIP aligned images and text.\n- 4.1 What does \"semantic drifts\" mean?\n- 4.2 The recall level R is not described what it does\n- 4.4 Typo: \"at the following task\"\n- 4.5 Meaning of \"group of classes (10, 5 and the last class)\" is unclear\n- 4.6 Typo: \"To be specific, ...\"\n\n\n#### References mentioned above\n\n- [A] Zero-shot object detection. Bansal et al. ECCV'18\n- [B] Open-Vocabulary Object Detection Using Captions. Zareian et al. CVPR'21\n- [C] Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation. Gu et al. arXiv'21\n- [D] Object Detection with a Unified Label Space from Multiple Datasets. Zhao et al. ECCV'20\n- [E] Simple multi-dataset detection. Zhou et al. arXiv'21\n", "summary_of_the_paper": "The paper proposes a novel method for open-world object detection, where instances of unknown categories need to be identified and annotated data for such new categories need to be integrated into the model in an incremental fashion. Prior work typically separate this problem into two tasks, out-of-distribution detection and incremental/continuous learning. This paper proposes to use fixed semantic anchors for each category, which are embedding vectors from a language model (or randomly generated vectors) for each category.  Importantly, when new data arrives, new embeddings are added while previous ones do not change. This encourages the feature representation to be compact (discriminative) and consistent (over time).", "summary_of_the_review": "Overall, I think the paper presents a solid contribution to the task of open-world object detection. Some aspects of the paper need more clarification and some related work is missing that needs to be discussed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8UgaReNEZke", "reviewer": "Reviewer_8C8m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n* The core idea introduced in this paper is nice, simple, and intuitive: simply leverage large-scale language models for zero-shot transfer to initialize a large variety of semantic class prototypes in the embedding space. Such augmentation of ORE makes perfect sense, and as verified experimentally, is very effective! \n* This paper tackles and very challenging important problem and the methodology is presented clearly. \n* I like the experimental evaluation; besides showing that leveraging language models for this task improves performance on incremental object detector learning, this paper also nicely ablates the impact on the effect of different language models.\n \n\n**Cons**\n\n* I am confused about the claims that prior work does not encourage intra-class compactness; isn't that exactly what the contrastive loss (in ORE) (in this paper referred to as clustering loss) is supposed to do?\n* This paper adopts the task description from [A], claiming that novel objects are recognized as unknowns and should be labeled by annotators and used for re-training. This is not what the proposed method or method being built upon (ORE) does. Both are always only evaluated for the task of incremental learning, in terms of mAP and metrics, that evaluate confusion between known and unknown classes (e.g., Wilderness Impact measure): neither [A] or this paper provide any evidence that the learned detector is effective in detecting novel object instances (and separating them from the background *stuff* classes), and can be reliably be used in (for example) in conjunction with active learning to minimize the annotation effort. In summary, there is no evidence that any of the \"new\" classes in the new task sets are actually being detected before the network update.  \nTo justify the claims made in Sec. 3.1, the model would need to label instances as unknowns and transfer semantic labels to these before using them for the model re-training (and not simply usefully labeled images for the new task set). I know that this confusion was not introduced in this paper. Rather, this paper just iterates claims and statements made in [A]. Still, I would suggest not propagating it further and fairly acknowledge that this paper really tackles object detection in an incremental learning setting. \n* The basis of this paper is zero-shot learning: the knowledge about unseen classes is transferred from the language models. Such a paper absolutely must establish links with zero-shot learning and discuss prior work. I can understand that in the current rate of publications in our field one can miss a certain related paper, and I am usually happy to point this out and suggest updating the related work. However, this is not the case here; this paper entirely ignores the field of zero-shot learning, which is a basis for this work. \nI strongly suggest performing a literature review in this field. In terms of zero-shot recognition, a good source would be [D]. Closely related methods on zero-shot detection and segmentation are aforementioned [C], and [E].\n* A big part of the paper is 1-1 re-write of sections from [A], e.g., 3.1, 3.3, 4.1 (using own words). A fair thing to do would be to write a very short recap on problem definition and ORE detector, followed by introducing what is novel in this work. Reading this paper largely feels like re-reading [A].\n \n\n**Question:** which semantic classes are used to initialize the anchors in the embedding space?\n", "summary_of_the_paper": "This paper extends the recently-introduced object ORE object detector [A]. ORE is trained in an incremental fashion and was shown to minimize the confusion between classes presented in different task sets by adding a contrastive objective to the model training, that pushes features, representing different semantic classes, far away in the feature space. This paper presents the case that combining ORE detector with large-scale language models, trained by aligning textual queries and images, can significantly improve ORE detector results. This is not surprising: it was already shown that such multi-modal language models could be successfully used for detection of novel classes in a zero-shot setting (in which names of target classes are given, but image data for these classes is available during the model training, see [D]).\nTo the best of my knowledge, the core idea of semantic anchoring using language models was introduced in [C] (see Sec. 3.3 Densely Sampled Embedding Space). In this work, the semantic embedding space is extended with additional data from external sources that contain semantic information about the unseen classes via language embedding to image embedding alignment. This method suggests using CLIP [B] instead (which makes sense and simplifies the knowledge transfer!)\n\n[A] Joseph et al., CVPR’21\n[B] Radford et al., ICLR’21\n[C] Bansal et al., ECCV’18\n[D] Xian et al., CVPR'17\n[E] Zheng et al., CVPR’21", "summary_of_the_review": "This paper describes a nice way of leveraging knowledge about various visual concepts, distilled in modern language models, to improve over incremental learning method by [A]. \n\nHowever, overall I, unfortunately, need to point out that the proposed (valuable and effective!) extension of ORE is almost exactly what was proposed before in the scope of zero-shot object detection [C], just leveraging the most recent developments in natural language processing. In case my interpretation is wrong I would like to see a very thorough discussion in relation to the aforementioned (that should already have been in the original manuscript), with a focus on what exactly the difference is.  \n\n\nI still think this paper still carries an interesting message for the community, but (i) link to prior work on ZSL must be clearly established, and (ii) I do not think that ICLR is the right venue for this work in terms of novelty.  \n\n\n**Post rebuttal** \n\n\nI would like to thank the authors for their comments and colleagues for the discussion. First, I would like to clarify that the paper was updated to include a discussion on the closely related topic of ZSL. I take back my comments that the usage for semantic anchors reduces this problem from open-world detection to zero-shot detection. Those were based on my initial miss-conception on how semantic anchors were used. \n\nHowever, I still think that the evaluation protocol studies only incremental learning and provides no evidence whether the proposed method can be actually applied to the (significantly more challenging!) problem of open-world detection, and authors actually do acknowledge that in one of their responses. \n\nAs a reviewer, I cannot recommend accepting a paper that (in my view) has a mismatch between paper claims and premise (open world detection) and the actual delivery (incremental learning). My colleagues do not find this point as concerning (esp. given the fact that this issue was inherited from the prior work), therefore I will not strongly argue against acceptance, but I nevertheless wanted to bring this issue to AC's attention. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on open-world object detection.  The main idea is to use fixed per-category semantic anchors.  These can be incrementally added to when new data appear.  The reviewers engaged in significant discussion around the paper with many iterations of improvements to the paper.  Initial concerns regarding zero-shot learning were addressed, as were remarks on presentation and claims.

In the end the reviewers were split on this paper.  I recommend to accept the paper on the basis of the semantic topology ideas and the thorough experimental results.

The remaining concern centered around the evaluation protocol used in the paper, which follows that in the literature (e.g. Joseph et al. CVPR 21).  While this is not a fatal flaw, it is an issue with how this genre of methods is evaluated.  It would be good to add discussion to the final paper to highlight this as an opportunity for future work in the field to address.  Specifically, as a reviewer noted "after detecting "unknown" objects in T1, the (hypothetical) annotation process provides boxes for ALL objects of some new classes instead of only for those that have been correctly detected (localized and marked "unknown")."
Decision: Accept (Poster)