Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection

1 Introduction . Deep learning have shown remarkable performance in a wide variety of tasks , and even surpass human experts in numerous tasks . However , humans are still better than machines in continually acquiring , fine-tuning and transferring knowledge throughout their lifetime . This is because deep networks suffer from catastrophic forgetting [ 18 , 21 ] , i.e . a phenomenon that causes a deep network to forget previously acquired knowledge on the old base classes when trained on new novel classes . As a result , this causes the performance of the deep networks to drop significantly on the base classes . In the task of image-based object detection , object detectors are trained on large-scale datasets and then deployed on real-world applications [ 9 , 11 , 8 , 24 , 5 , 15 , 23 , 17 , 16 , 1 , 2 , 39 , 33 , 34 , 38 , 36 , 37 , 35 , 32 ] . As these applications are often carried out in dynamic environments where novel object classes are continually presented , the ability for the deep networks to learn novel object classes without forgetting the base object classes becomes an imperative requirement . A naive approach to achieve this endeavor ∗Work fully done while first author is a visiting PhD student at the National University of Singapore . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is to retain all training data for the base classes and train the deep network concurrently with the base and novel training data from scratch . However , pragmatic issues such as privacy can limit the accessibility to the base class dataset previously used to train the base model . Recently , several incremental learning methods [ 40 , 27 , 4 , 10 , 20 , 31 ] are proposed to overcome catastrophic forgetting in the object detection task . Despite the impressive performance , these methods rely on the co-occurrence of unlabeled base classes in the training data of the novel classes . Due to the fact that the base and novel class datasets are obtained from the same dataset to simulate incremental learning , the base classes inevitably co-occur in the background of the novel class training data . Table 1 shows the co-occurrence statistics of the base classes in the novel datasets from PASCAL VOC under several commonly used experiment settings in the existing works . For example , class 1-19 are chosen as the base classes and class 20 ( “ tv '' ) as the novel class . We can see from the table that the “ areo '' , “ bicycle '' , “ bottle '' , etc . , objects in the base classes co-occur in the sample images of the novel “ tv '' object class . Despite the absence of ground truth labels of the base classes on the novel training data , knowledge of base classes can be replayed and distilled into the novel model . However , this reliance on the co-occurrence of unlabeled base classes in the training data of novel classes severely limits the practicality of most incremental learning approaches . This is due to the inherent non co-occurrence of the base and novel classes in most real-world data . In this paper , we propose the use of the abundance in-the-wild data to bridge the non co-occurrence of base classes in the training data of the novel classes . To this end , we first introduce a blind sampling strategy to select relevant data from the in-the-wild data that contains large amounts of irrelevant images with neither the base nor novel class information . Specifically , images with high probability response from the given base model and the novel model trained with the novel class training data are selected in the blind sampling step . We further design a dual-teacher distillation framework where the images selected from the blind sampling strategy are used to distill knowledge from the base and novel teacher models to the student model . Particularly , our dual-teacher distillation framework consists of : 1 ) A class remodeling step that remodels the irrelevant classes as background class in the base and novel model , respectively . This ensures the class probabilities of the disjoint set of classes in the base and novel models can be compared appropriately to the student model . 2 ) Image-level distillation with region of interest ( RoI ) masks from the pseudo ground truth of the bounding boxes obtained in the blind sampling step . These RoI masks are used to mask out the irrelevant regions of the feature maps in the distillation loss . 3 ) Instance-level distillation with the response heatmaps of the object detectors . This response heatmaps is essential in transferring both positive ( high response regions ) and negative ( low response regions ) information from the base and novel teacher models to the student model . Our main contributions are as follows : 1 . We tackle a more challenging and realistic scenario of incremental learning for object detection , where there is no co-occurrence of the base classes in the training data of the novel classes . This contrasts with previous approaches whose success depend largely on such co-occurrences . 2 . We propose a blind sampling strategy to effectively select useful data from large amounts of unlabeled in-the-wild data . We also design a dual-teacher distillation framework which utilizes the sampled data to distill knowledge from the base and novel teacher models to the student model . 3 . Extensive experiments conducted on two standard object detection datasets ( i.e . MS COCO and PASCAL VOC ) demonstrate the significant performance improvement of our approach over existing state-of-the-art . 2 Related Works . 2.1 Object Detection . Existing deep object detection models fall generally into two categories : 1 ) One-stage detectors and 2 ) Two-stage detectors . One-stage detectors such as YOLO [ 23 ] directly performed object classification and bounding box regression on the feature maps . SSD [ 17 ] uses feature pyramid with different anchor sizes to cover the possible object scales . RetinaNet [ 16 ] proposes the focal loss to mitigate the imbalanced positive and negative examples . Two-stage detectors such as R-CNN [ 9 ] apply a deep neural network to extract features from proposals generated by selective search [ 28 ] . Fast R-CNN [ 8 ] improves the speed and performance utilizing a differentiable RoI Pooling . Faster R-CNN [ 24 ] introduces Region Proposal Network ( RPN ) to generate proposals . FPN [ 15 ] builds a top-down architecture with lateral connections to extract features across multiple layers . Typically , both one-stage and two-stage object detectors require large amounts of training images per class and need to train the detectors over many training epochs . Unfortunately , it is unlikely that large amounts of data for the old classes are present in the new training data . Therefore , the extension of the capability of detectors to novel categories with no access to the original training data is imperative . 2.2 Class-incremental Learning . Most works on class-incremental learning focus on the image classification problems , and can be roughly divided into two major families : 1 ) regularization approaches and 2 ) rehearsal approaches . In the regularization approaches , learning capacity is assumed to be fixed and incremental learning is performed so that the change in parameters is controlled or reduced . Kirkpatrick et . al [ 13 ] propose the elastic weight consolidation ( EWC ) method in which Ωi is calculated as diagonal approximation of the empirical Fisher Information Matrix . The second type of regularization-based approaches is based on knowledge distillation [ 14 , 6 ] . Li et . al [ 14 ] propose to use Learning without Forgetting ( LwF ) to keep the representations of base data from drifting too much while learning the novel tasks . In the rehearsal approaches , the models strengthen memories learned in the past through replaying the past information periodically . They usually keep a small number of exemplars [ 22 , 29 , 3 ] , or generate synthetic images [ 26 , 19 ] or features [ 12 , 30 ] to achieve this purpose . The rehearsal method for class-incremental learning is first proposed in iCaRL [ 22 ] , and has been applied in majority of the class-incremental learning works . 2.3 Class-incremental Object Detection . Class-incremental object detection are relatively less explored than its image classification counterpart . All the proposed methods follow the regularization-based approach , in which knowledge distillation is used to address the catastrophic forgetting issue . Shmelkov et . al [ 27 ] proposes a Fast R-CNNbased class-incremental object detection model to address the catastrophic forgetting problem , where EdgeBoxes [ 41 ] is used to produce bounding box proposals . However , the proposals generation stage introduces an immense computational cost . Sub-optimal performance for the base and novel classes are obtained since a sub-optimal two-stage object detection model is used . Zhou et . al [ 40 ] proposes an incremental version of Faster R-CNN that distills selected anchors and proposals with the PseudoPositive-Aware Sampling strategy . However , it fails to improve the benchmark [ 27 ] on the standard evaluation criteria . The class-incremental object detection methods mentioned above are impractical in the real-world since the key success factor is the utilization of co-occurred unlabeled base objects with novel objects in the novel training data . Zhang et . al [ 31 ] presents a class-incremental learning paradigm called Deep Model Consolidation ( DMC ) for single-shot object detection architectures , which combines the base model and the novel model leveraging external unlabeled auxiliary data . In this work , we develop a novel mechanism for the task of class-incremental object detection based on the Faster R-CNN framework . Our framework is a dual-teacher that distill knowledge from the teacher base and novel models to the student model using sampled unlabeled in-the-wild data , without requiring the co-occurrence of base and novel object classes . 3 Our Approach . 3.1 Preliminaries . Let ( x , y ) ∈ D denotes a dataset D which contains images x and their corresponding object bounding box labels y . We further denote the training dataset of the base classes and the novel classes as Dbase and Dnovel , respectively . Following the definition of class-incremental learning , we only have access to the novel class data Dnovel , where ynovel ∈ { Cb+1 , . . . , Cb+n , Cbgn } . The base class data Dbase , where ybase ∈ { C1 , . . . , Cb , Cbgb } are no longer accessible . Cbgn and Cbgb are the background class of the novel and base data , respectively . Unlike other existing incremental learning methods for object detection where the base classes can still occur as unlabeled background classes in the novel training data , we enforce the more realistic strict non co-occurrence { Cb+1 , . . . , Cb+n , Cbgn } ∪ { C1 , . . . , Cb , Cbgb } = ∅ in our training data . We further assume that a large quantity of unlabeled in-the-wild data is accessible , from which we sample Dunlabel using our blind sampling strategy . The base model Mbase ( Dbase ; θbase ) is an object detector trained on the base class data , where θbase denotes the learnable parameters . Our goal is to train an object detection model Mstud ( Dunlabel ; θstud ) to detect the novel classes { Cb+1 , . . . , Cb+n , Cbgn } without catastrophic forgetting the base classes { C1 , . . . , Cb , Cbgb } . We use Faster R-CNN [ 24 ] as our object detector .

Summary: This paper investigates incremental object detection problem without making the assumption of base and novel classes occurring together, i.e. person (base) and TV(novel) objects being in the same image. The authors make the argument that this does not always happen in real-world scenarios, and they propose to sample in-the-wild data instead. They call this stage the "Blind sampling strategy", and this is achieved by applying the pre-trained Base and Novel models to the unlabeled data, and collecting those that have "positive" output. After the blind sampling strategy, a student model is learned by distilling the knowledge from the base and teacher models. Finally, the method is evaluated on PASCAL VOC and MS COCO datasets.