Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection

1 Introduction . Deep learning have shown remarkable performance in a wide variety of tasks , and even surpass human experts in numerous tasks . However , humans are still better than machines in continually acquiring , fine-tuning and transferring knowledge throughout their lifetime . This is because deep networks suffer from catastrophic forgetting [ 18 , 21 ] , i.e . a phenomenon that causes a deep network to forget previously acquired knowledge on the old base classes when trained on new novel classes . As a result , this causes the performance of the deep networks to drop significantly on the base classes . In the task of image-based object detection , object detectors are trained on large-scale datasets and then deployed on real-world applications [ 9 , 11 , 8 , 24 , 5 , 15 , 23 , 17 , 16 , 1 , 2 , 39 , 33 , 34 , 38 , 36 , 37 , 35 , 32 ] . As these applications are often carried out in dynamic environments where novel object classes are continually presented , the ability for the deep networks to learn novel object classes without forgetting the base object classes becomes an imperative requirement . A naive approach to achieve this endeavor ∗Work fully done while first author is a visiting PhD student at the National University of Singapore . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is to retain all training data for the base classes and train the deep network concurrently with the base and novel training data from scratch . However , pragmatic issues such as privacy can limit the accessibility to the base class dataset previously used to train the base model . Recently , several incremental learning methods [ 40 , 27 , 4 , 10 , 20 , 31 ] are proposed to overcome catastrophic forgetting in the object detection task . Despite the impressive performance , these methods rely on the co-occurrence of unlabeled base classes in the training data of the novel classes . Due to the fact that the base and novel class datasets are obtained from the same dataset to simulate incremental learning , the base classes inevitably co-occur in the background of the novel class training data . Table 1 shows the co-occurrence statistics of the base classes in the novel datasets from PASCAL VOC under several commonly used experiment settings in the existing works . For example , class 1-19 are chosen as the base classes and class 20 ( “ tv '' ) as the novel class . We can see from the table that the “ areo '' , “ bicycle '' , “ bottle '' , etc . , objects in the base classes co-occur in the sample images of the novel “ tv '' object class . Despite the absence of ground truth labels of the base classes on the novel training data , knowledge of base classes can be replayed and distilled into the novel model . However , this reliance on the co-occurrence of unlabeled base classes in the training data of novel classes severely limits the practicality of most incremental learning approaches . This is due to the inherent non co-occurrence of the base and novel classes in most real-world data . In this paper , we propose the use of the abundance in-the-wild data to bridge the non co-occurrence of base classes in the training data of the novel classes . To this end , we first introduce a blind sampling strategy to select relevant data from the in-the-wild data that contains large amounts of irrelevant images with neither the base nor novel class information . Specifically , images with high probability response from the given base model and the novel model trained with the novel class training data are selected in the blind sampling step . We further design a dual-teacher distillation framework where the images selected from the blind sampling strategy are used to distill knowledge from the base and novel teacher models to the student model . Particularly , our dual-teacher distillation framework consists of : 1 ) A class remodeling step that remodels the irrelevant classes as background class in the base and novel model , respectively . This ensures the class probabilities of the disjoint set of classes in the base and novel models can be compared appropriately to the student model . 2 ) Image-level distillation with region of interest ( RoI ) masks from the pseudo ground truth of the bounding boxes obtained in the blind sampling step . These RoI masks are used to mask out the irrelevant regions of the feature maps in the distillation loss . 3 ) Instance-level distillation with the response heatmaps of the object detectors . This response heatmaps is essential in transferring both positive ( high response regions ) and negative ( low response regions ) information from the base and novel teacher models to the student model . Our main contributions are as follows : 1 . We tackle a more challenging and realistic scenario of incremental learning for object detection , where there is no co-occurrence of the base classes in the training data of the novel classes . This contrasts with previous approaches whose success depend largely on such co-occurrences . 2 . We propose a blind sampling strategy to effectively select useful data from large amounts of unlabeled in-the-wild data . We also design a dual-teacher distillation framework which utilizes the sampled data to distill knowledge from the base and novel teacher models to the student model . 3 . Extensive experiments conducted on two standard object detection datasets ( i.e . MS COCO and PASCAL VOC ) demonstrate the significant performance improvement of our approach over existing state-of-the-art . 2 Related Works . 2.1 Object Detection . Existing deep object detection models fall generally into two categories : 1 ) One-stage detectors and 2 ) Two-stage detectors . One-stage detectors such as YOLO [ 23 ] directly performed object classification and bounding box regression on the feature maps . SSD [ 17 ] uses feature pyramid with different anchor sizes to cover the possible object scales . RetinaNet [ 16 ] proposes the focal loss to mitigate the imbalanced positive and negative examples . Two-stage detectors such as R-CNN [ 9 ] apply a deep neural network to extract features from proposals generated by selective search [ 28 ] . Fast R-CNN [ 8 ] improves the speed and performance utilizing a differentiable RoI Pooling . Faster R-CNN [ 24 ] introduces Region Proposal Network ( RPN ) to generate proposals . FPN [ 15 ] builds a top-down architecture with lateral connections to extract features across multiple layers . Typically , both one-stage and two-stage object detectors require large amounts of training images per class and need to train the detectors over many training epochs . Unfortunately , it is unlikely that large amounts of data for the old classes are present in the new training data . Therefore , the extension of the capability of detectors to novel categories with no access to the original training data is imperative . 2.2 Class-incremental Learning . Most works on class-incremental learning focus on the image classification problems , and can be roughly divided into two major families : 1 ) regularization approaches and 2 ) rehearsal approaches . In the regularization approaches , learning capacity is assumed to be fixed and incremental learning is performed so that the change in parameters is controlled or reduced . Kirkpatrick et . al [ 13 ] propose the elastic weight consolidation ( EWC ) method in which Ωi is calculated as diagonal approximation of the empirical Fisher Information Matrix . The second type of regularization-based approaches is based on knowledge distillation [ 14 , 6 ] . Li et . al [ 14 ] propose to use Learning without Forgetting ( LwF ) to keep the representations of base data from drifting too much while learning the novel tasks . In the rehearsal approaches , the models strengthen memories learned in the past through replaying the past information periodically . They usually keep a small number of exemplars [ 22 , 29 , 3 ] , or generate synthetic images [ 26 , 19 ] or features [ 12 , 30 ] to achieve this purpose . The rehearsal method for class-incremental learning is first proposed in iCaRL [ 22 ] , and has been applied in majority of the class-incremental learning works . 2.3 Class-incremental Object Detection . Class-incremental object detection are relatively less explored than its image classification counterpart . All the proposed methods follow the regularization-based approach , in which knowledge distillation is used to address the catastrophic forgetting issue . Shmelkov et . al [ 27 ] proposes a Fast R-CNNbased class-incremental object detection model to address the catastrophic forgetting problem , where EdgeBoxes [ 41 ] is used to produce bounding box proposals . However , the proposals generation stage introduces an immense computational cost . Sub-optimal performance for the base and novel classes are obtained since a sub-optimal two-stage object detection model is used . Zhou et . al [ 40 ] proposes an incremental version of Faster R-CNN that distills selected anchors and proposals with the PseudoPositive-Aware Sampling strategy . However , it fails to improve the benchmark [ 27 ] on the standard evaluation criteria . The class-incremental object detection methods mentioned above are impractical in the real-world since the key success factor is the utilization of co-occurred unlabeled base objects with novel objects in the novel training data . Zhang et . al [ 31 ] presents a class-incremental learning paradigm called Deep Model Consolidation ( DMC ) for single-shot object detection architectures , which combines the base model and the novel model leveraging external unlabeled auxiliary data . In this work , we develop a novel mechanism for the task of class-incremental object detection based on the Faster R-CNN framework . Our framework is a dual-teacher that distill knowledge from the teacher base and novel models to the student model using sampled unlabeled in-the-wild data , without requiring the co-occurrence of base and novel object classes . 3 Our Approach . 3.1 Preliminaries . Let ( x , y ) ∈ D denotes a dataset D which contains images x and their corresponding object bounding box labels y . We further denote the training dataset of the base classes and the novel classes as Dbase and Dnovel , respectively . Following the definition of class-incremental learning , we only have access to the novel class data Dnovel , where ynovel ∈ { Cb+1 , . . . , Cb+n , Cbgn } . The base class data Dbase , where ybase ∈ { C1 , . . . , Cb , Cbgb } are no longer accessible . Cbgn and Cbgb are the background class of the novel and base data , respectively . Unlike other existing incremental learning methods for object detection where the base classes can still occur as unlabeled background classes in the novel training data , we enforce the more realistic strict non co-occurrence { Cb+1 , . . . , Cb+n , Cbgn } ∪ { C1 , . . . , Cb , Cbgb } = ∅ in our training data . We further assume that a large quantity of unlabeled in-the-wild data is accessible , from which we sample Dunlabel using our blind sampling strategy . The base model Mbase ( Dbase ; θbase ) is an object detector trained on the base class data , where θbase denotes the learnable parameters . Our goal is to train an object detection model Mstud ( Dunlabel ; θstud ) to detect the novel classes { Cb+1 , . . . , Cb+n , Cbgn } without catastrophic forgetting the base classes { C1 , . . . , Cb , Cbgb } . We use Faster R-CNN [ 24 ] as our object detector .

Summary: The paper proposes a technique to address the learning-without-forgetting problem when base and novel classes do not co-occur. The approach relies on sampling unlabeled in-the-wild data from another domain, to bridge base and novel classes. Results against a few recent baselines, as well as ablations, demonstrate the effectiveness of the approach.