Objects in Semantic Topology

1 INTRODUCTION . Object detection , which aims at localizing and classifying objects in a given scene ( Felzenszwalb et al. , 2010 ; Everingham et al. , 2010 ; Lin et al. , 2014 ) , is one of the most iconic abilities of biological intelligence . It was introduced to the artificial intelligence field to endow an intelligence agent with the ability of scene understanding . Although significant advances have been made to improve the object detection system in recent years ( Girshick et al. , 2014 ; Ren et al. , 2015 ; Cai & Vasconcelos , 2018 ; Sun et al. , 2020b ; Redmon et al. , 2016 ; Lin et al. , 2017 ; Tian et al. , 2019 ; Zhou et al. , 2019a ; Carion et al. , 2020 ; Sun et al. , 2020a ) , a strong assumption that all the objects of interest have been annotated in the training set , i.e. , close-set learning , is always made but not holds well . All unknown objects are treated as background and ignored by current detectors , making such detectors can not handle corner cases in many real-world applications such as autonomous driving , where some obstacles are not available during training , but must be detected when intelligent cars are running on the road . However , creating a large-scale dataset that contains annotations for all objects of interest at once is extremely expensive , even impossible . Superior to current detectors , humans naturally have the ability to discover both known and unknown objects and gradually learn novel concepts motivated by their curiosity . Learning by discovering the unknown is crucial for human intelligence ( Livio , 2017 ; Meacham , 1983 ) , and has been considered as a key step to achieve artificial general intelligence ( AGI ) ( Goertzel , 2014 ) . Recently , a new object detection paradigm , named Open-World Object Detection , has been established ( Joseph et al. , 2021 ; Miller et al. , 2021 ; 2018c ; Liu et al. , 2020 ) to mimic this learning procedure . A qualified open-world object detector can not only identify known objects , but also discover object instances of unknown categories and gradually learn to recognize them when their annotations ∗corresponding author progressively arrive . The learning of novel categories is always in an incremental fashion , where the detector can not access all old data when training on new categories . This open-world learning setting is much more realistic but challenging than previous close-set object detection . The open-world object detection poses two challenges on current detectors , i.e. , recognition of unknown categories and incremental learning . On the one hand , previous close-set detectors do not explicitly encourage intra-class compactness ( Liu et al. , 2016b ; Yang et al. , 2021b ) . However , compact feature representation is highly required for unknown object recognition ( discovery ) . If the known categories occupy most of the feature space , the detector will probably classify the unknown object as one of the known categories . On the other hand , the vanilla training strategy of the object detector lacks the mechanism to prevent ‘ catastrophic forgetting ’ in incremental learning ( Joseph et al. , 2021 ; Shmelkov et al. , 2017a ; Peng et al. , 2020 ) , i.e. , previously-known objects features are severely distorted when learning new categories . Consequently , training on novel categories weakens the detector ’ s ability to detect previously-known objects . Previous works have made efforts to endow the object detector with the capacity of unknown recognition and incremental learning . One of the representative works , ORE ( Joseph et al. , 2021 ) , designs a clustering loss function to compact the object features , and involves an energy-based outof-distribution identification approach ( Liu et al. , 2020 ) to detect unknown objects . Combining these two independent technologies in a step-wise manner makes ORE a non-end-to-end framework . Such solutions for open-world object detection are far from the optimal status . Additionally , ORE ( Joseph et al. , 2021 ) doesn ’ t guarantee a feature space topology consistency , which is crucial for effective new class learning and avoidance of catastrophic forgetting , as shown in Figure 1 . This paper formalizes unknown recognition and incremental learning in a unified perspective and proposes a much simpler but more effective framework for open-world object detection than prior arts . We propose that an open-world object detector is desired to learn a feature space with such characteristics , including ( a ) discriminativeness : the unknown objects and the objects of new categories couldn ’ t be overlapped with any previously-known categories in the feature space , and ( b ) consistency : the feature topology of previously-known categories couldn ’ t be distorted severely when learning new categories . Our key idea is to pre-define a unique and fixed centroid in feature space for each category , including the ‘ unknown ’ category , and to push object instances close to their belonging centroids during the life-long learning of an open-world object detector . The pre-defined centroids are named as ‘ Semantic Anchors ’ , and all semantic anchors constitute the structure of ‘ Semantic Topology ’ . As shown in Figure 2 , all features are binded to their corresponding semantic anchors to satisfy the discriminativeness , previously-known objects feature topology is maintained when incrementally learning new categories to satisfy the consistency . We introduce an off-the-shelf pre-trained language model to set up the semantic topology . The semantic anchor for each category is derived from the language model by embedding the corresponding category name . By incrementally registering new semantic anchors when new classes are involved , the semantic topology gradually grows . In the experiments , we show that by combining a current detector with our proposed semantic anchor head , a huge improvement in open-world object detection performance across the board is achieved . In addition to the consistent mAP improvement over the whole life of the detector , the ability of unknown recognition of our proposed method outperforms the current state-of-the-art methods by a large margin , e.g. , the absolute open-set error is reduced by 2/3 , from 7832 to 2546 . More importantly , we conduct a comparison experiment by randomly generating semantic anchors instead of leveraging language models . Although randomly-generated anchors do not provide semantic priors , their performance still surpasses state-of-the-art methods . This strongly proves that the topology consistency is the most key characteristic for open-world learning while introducing semantic relationships can further boost the performance . 2 RELATED WORKS . Object detection Modern object detection frameworks ( Ren et al. , 2015 ; Redmon et al. , 2016 ; Liu et al. , 2016a ; Lin et al. , 2017 ; Zhang et al. , 2019a ; Redmon et al. , 2016 ; Redmon & Farhadi , 2018 ; Girshick , 2015 ; Duan et al. , 2019 ; Tian et al. , 2019 ; Tan et al. , 2020 ; Zhou et al. , 2019b ; Jiang et al. , 2018 ; Zhang et al. , 2019b ; Carion et al. , 2020 ; Sun et al. , 2020b ) take advantage of the high-capacity representation in deep neural networks to localize and classify the target class object in given images and videos . These well-established detectors can achieve excellent performance in the close-set dataset such as PascalVOC ( Everingham et al. , 2010 ) , MSCOCO ( Lin et al. , 2014 ) . However , the detectors can not handle open-world object detection , which is more common in the real world . To this end , ORE ( Joseph et al. , 2021 ) raises and formalizes the open-world object detection problem . Class incremental learning Class incremental learning aims to learn a classifier incrementally to recognize all encountered classes met so far , including previously-known classes and novel classes . Knowledge distillation is commonly adopted to mitigate forgetting old classes , which stores some old class exemplars to fine-tune the model or compute the distillation loss . iCaRL ( Rebuffi et al. , 2017 ) maintains an ‘ episodic memory ’ of the exemplars and incrementally adds novel class examples into the memory . Then the nearest neighbor classifier can be obtained incrementally . LwF ( Li & Hoiem , 2017 ) proposes a modified cross-entropy loss to preserve the knowledge in the previous task . BiC ( Wu et al. , 2019 ) points out the data imbalance between old classes and new classes causes the network ’ s prediction biased towards new classes . However , the existing class incremental methods can not handle the open-world problem , where the classifier should identify those ‘ unknown ’ classes and incrementally learns to recognize them , but existing methods would recognize unknown objects as background . Open-set Learning In open-set learning , the knowledge contained in the training set is incomplete , i.e. , the examples encountered during inference may belong to a category that does not appear in the training set . OpenMax ( Bendale & Boult , 2016 ) uses a Weibull distribution to identify unknown instances in the feature space of deep networks . OLTR ( Liu et al. , 2019 ) proposes to tackle the open-set recognition problem in a data imbalance setting by deep metric learning . Besides the open-set classification , Dhamija et al . ( Dhamija et al. , 2020b ) found that the object detectors exhibit a high error rate that misclassifies unknown classes to known classes with high confidence . To solve this problem , many works ( Miller et al. , 2021 ; 2018a ; Dhamija et al. , 2020b ) aims to measure the uncertainty of the detectors ’ outputs to reject open-set error . Miller et al . ( Miller et al. , 2018a ) uses Monte Carlo Dropout sampling to estimate the uncertainty in an SSD detector . After that , Miller et al . proposes to model a Gaussian Mixture distribution for each class in the detector ’ s feature space to reject unknown classes . ORE ( Joseph et al. , 2021 ) uses an energy-based out-of-distribution recognition method ( Liu et al. , 2020 ) to distinguish the known and unknown . However , such method needs to compute the energy score distribution among all instances , including known and unknown , during evaluation , making ORE a non-end-to-end method . We formulate the open-set recognition problem and the class incremental learning into a unified framework . Different from ( Joseph et al. , 2021 ) , our method doesn ’ t access any unknown instances but achieves much superior performance . Zero-shot Learning Aligning image and text into a common feature space has always been an active research topic ( Frome et al. , 2013 ; Joulin et al. , 2016 ; Li et al. , 2017 ; Desai & Johnson , 2021 ; Yang et al. , 2019 ; 2021a ) , especially in zero-shot learning ( Xian et al. , 2017 ; Bansal et al. , 2018 ; Zareian et al. , 2021 ; Gu et al. , 2021 ) . Many researches in zero-shot learning leverage the information in language models to assist zero-shot image classification ( Xian et al. , 2017 ) or zero-shot object detection ( Bansal et al. , 2018 ; Zareian et al. , 2021 ; Gu et al. , 2021 ) . However , this paper tackles a different problem and has a different usage of language models . We identify that a consistent feature manifold topology plays an essential role in open-world object detection , and the language model is used for generating growth-able and consistent semantic topology to constrain the feature space learning of an open-world detection . Also , this paper is the first one to incorporate a language model , e.g. , CLIP , to assist open-world object detection , which results in a simple training framework and strong empirical performance .

Summary: This paper extends the recently-introduced object ORE object detector [A]. ORE is trained in an incremental fashion and was shown to minimize the confusion between classes presented in different task sets by adding a contrastive objective to the model training, that pushes features, representing different semantic classes, far away in the feature space. This paper presents the case that combining ORE detector with large-scale language models, trained by aligning textual queries and images, can significantly improve ORE detector results. This is not surprising: it was already shown that such multi-modal language models could be successfully used for detection of novel classes in a zero-shot setting (in which names of target classes are given, but image data for these classes is available during the model training, see [D]). To the best of my knowledge, the core idea of semantic anchoring using language models was introduced in [C] (see Sec. 3.3 Densely Sampled Embedding Space). In this work, the semantic embedding space is extended with additional data from external sources that contain semantic information about the unseen classes via language embedding to image embedding alignment. This method suggests using CLIP [B] instead (which makes sense and simplifies the knowledge transfer!)  [A] Joseph et al., CVPR’21 [B] Radford et al., ICLR’21 [C] Bansal et al., ECCV’18 [D] Xian et al., CVPR'17 [E] Zheng et al., CVPR’21