Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection
Authors: NA DONG
Yongqiang Zhang
Mingli Ding
Gim Hee Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: Incremental learning
Object detection
Non co-occurrence

Abstract: Deep networks have shown remarkable results in the task of object detection. However, their performance suffers critical drops when they are subsequently trained on novel classes without any sample from the base classes originally used to train the model. This phenomenon is known as catastrophic forgetting. Recently, several incremental learning methods are proposed to mitigate catastrophic forgetting for object detection. Despite the effectiveness, these methods require co-occurrence of the unlabeled base classes in the training data of the novel classes. This requirement is impractical in many real-world settings since the base classes do not necessarily co-occur with the novel classes. In view of this limitation, we consider a more practical setting of complete absence of co-occurrence of the base and novel classes for the object detection task. We propose the use of unlabeled in-the-wild data to bridge the non co-occurrence caused by the missing base classes during the training of additional novel classes. To this end, we introduce a blind sampling strategy based on the responses of the base-class model and pre-trained novel-class model to select a smaller relevant dataset from the large in-the-wild dataset for incremental learning. We then design a dual-teacher distillation framework to transfer the knowledge distilled from the base- and novel-class teacher models to the student model using the sampled in-the-wild data. Experimental results on the PASCAL VOC and MS COCO datasets show that our proposed method significantly outperforms other state-of-the-art class-incremental object detection methods when there is no co-occurrence between the base and novel classes during training. 

Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection
Gim Hee Lee! Department of Computer Science, National University of Singapore "chooo of Instrument Science and Engineering, Harbin Institute of Technology {dongna1994, zhangyongqiang, (angml)ahiit.ddu....
Na Dong 12-
Yongqiang Zhang?
Mingli Ding²
gimhee weeeeomee....pp...
mus mue edu.
Abstract
Deep networks have shown remarkable results in the task of object detection. However, their performance suffers critical drops when they are subsequently trained on novel classes without any sample from the base classes originally used to train the model. This phenomenon is known as catastrophic forgetting. Recently, several incremental learning methods are proposed mitigate catas- trophic forgetting for object detection. Despite the effectiveness, these methods require co-occurrence of the unlabeled base classes the training data of the novel classes. This requirement is impractical in many real-world settings since the base classes do not necessarily co-occur with the novel classes. In view of this limitation, we consider a more practical setting of complete absence of co- occurrence of the base and novel classes for the object detection task. We propose the use of unlabeled in-the-wild data to bridge the non co-occurrence caused by the missing base classes during the training of additional novel classes. this end, we introduce blind sampling strategy based on the responses of the base- class model and pre-trained novel-class model to select a smaller relevant dataset from the large in in-the-wild dataset for incremental learning. We then design a dual-teacher distillation framework to transfer the knowledge distilled from the base- and novel-class teacher models to the student model using the sampled in- the-wild data. Experimental results on the PASCAL VOC and MS COCO datasets show that our proposed method significantly outperforms other state-of -ot--h---rtrrrrr class-incremente object detection methods when there is no co-occurrence be- tween the base and novel classes during training. Our source code is available https:/ / //g thub. com/ homm//donnan777777m httf///rrdiingg Non-Co-occurre Co-occurrence.
Introduction
Deep learning have shown remarkable performance in wide variety of tasks, and even surpass human experts in numerous tasks. However, humans are still better than machines continually acquiring, fine-tuning and transferring knowledge throughout their lifetime. This is because deep networks suffer from catastrophic forgetting [18, 21], i.e. a phenomenon that causes a deep network to forget previously acquired knowledge the old base classes when trained on new novel classes. As a result, this causes the performance of the deep networks to drop significantly on the base classes. In the task of image-based object detection, object detectors deployed on real-world applications [9. 11, 8, 24, 5, 15, 23, 17, 16, 1, 2, 39, 33, 34, 38, 36, 37, 32]. As these applications are often carried out in dynamic environments where novel object classes continually presented, the ability for the deep networks to learn novel object classes without forgetting the base object classes becomes imperative requirement. A naive approach achieve this endeavor
trained on large-scale datasets and then
Work fully done while first author is visiting PhD student at the National University of Singapore.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table I: Co-occurrence statistics of the base classes in the novel datasets from PASCAL VOC under various splits of "base novel" classes.
wosder oigeets
Bebesable
witsbbll
Jusp
rovel classes
15:5
nevel clases
haso (loses
200
140
112
1034
164
156
20
1994
1250
10h
10w10
nnvel clases
621
3016
912
1084
1230
116
1901
106
734
to retain all training data for the base classes and train the deep network concurrently with the base and novel training data from scratch. However, pragmatic issues such as privacy can limit the accessibility to the base class dataset previously used to train the base model.
Recently, several incremental learning methods [40, 27, 4, 10, 20, 31] are proposed to overcome catastrophic forgetting in the object detection task. Despite the impressive performance, these methods rely on the co-occurrence of unlabeled base classes in the training data of the novel classes. Due to the fact that the base and novel class datasets obtained from the same dataset simulate incremental learning, the base classes inevitably co-occur in the background of the novel class training data. Table shows the co-occurrence statistics of the base classes in the novel datasets from PASCAL VOC under several commonly used experiment settings in the existing works. For example, class 1-19 are chosen as the base classes and class 20 ("tv") as the novel class. We can see from the table that the "areo". "bicycle", "bottle", etc. objects in the base classes co-occur in the sample images of the novel "tv" object class. Despite the absence of ground truth labels of the base classes on the novel training data, knowledge of base classes can be replayed and distilled into the novel model. However, this reliance on the co-occurrence of unlabeled base classes in the training data of novel classes severely limits the practicality of most incremental learning approaches. This is due to the inherent non co-occurrence the base and novel classes in most real-world data.
In this paper, we propose the use of the abundance in-the-wild data to bridge the non co-occurrence of base classes in the training data of the novel classes. To this end, we first introduce a blind sampling strategy to select relevant data from the in-the in-the-wild data that contains large amounts of irrelevant images with neither the base nor novel class information. Specifically, images with high probability response from the given base model and the novel model trained with the novel class training data are selected in the blind sampling step. We further design a dual -tachre distillation framework where the images selected from the blind sampling strategy are used to distill knowledge from the base and novel teacher models to the student model. Particularly, our dual-teacher distillation framework consists of: 1) A class remodeling step that remodels the irrelevant classes background class in the base and novel model, respectively. This ensures the class probabilities of the disjoint set of classes in the base and novel models can be compared uppropriately to the student model. 2) Image-level distillation with region of interest (Rol) masks from the pseudo ground truth of the bounding boxes obtained in the blind sampling step. These Rol masks used to mask out the irrelevant regions of the feature maps in the distillation loss. 3) Instance-level distillation with the response heatmaps of the object detectors. This response heatmaps is essential in transferring both positive (high response regions) and negative (low response regions) information from the base and novel teacher models to the student model.
Our main contributions are as follows:
1. We tackle a more challenging and realistic scenario of incremental learning for object detection, where there is no co-occurrence of the base classes in the training data of the novel classes. This contrasts with previous approaches whose success depend largely on such co-occurrences.
2. We propose a blind sampling strategy to effectively select useful data from large amounts of unlabeled in-the-wild data. We also design a dual-teacher distillation framework which utilizes the sampled data to distill knowledge from the base and novel teacher models to the student model.
3. Extensive experiments conducted on two standard object detection datasets (i.e. MS COCO and PASCAL VOC) demonstrate the significant performance improvement of our approach over existing state-of-the-art.

---

2 Related Works
2.1 Object Detection
Existing deep object detection models fall generally into two categories: 1) One-stage detectors and 2) Two-stage detectors. One-stage detectors such as YOLO [23] directly performed object
classification and bounding box regression on the feature maps. SSD [17] uses feature pyramid with different anchor sizes cover the possible object scales. RetinaNet [ [16] proposes the focal loss to mitigate the imbalanced positive and negative examples. Two-stage detectors such as R-CNN [9] apply deep neural network to extract features from proposals generated by selective search [28]. Fast R-CNN [8] improves the speed and performance utilizing a differentiable Rol Pooling. Faster R-CNN [24] introduces Region Proposal NetWork(RPPN) to generate proposals. FPN [15] builds a top-down architecture with lateral connections to extract features across multiple layers. Typically, both one-stage and two-stage object detectors require large amounts of training images per class and need to train the detectors over many training epochs. Unfortunately, it is unlikely that large amounts of data for the old classes are present in the new training data. Therefore, the extension the capability of detectors to novel categories with no access to the original training data is imperative.
2.2 Class-iccremental Learning
Most works on class-incrementie learning focus on the image classification problems, and can be roughly divided into two major families: 1) regularization approaches and 2) rehearsal approaches. In the regularization approaches, learning capacity is assumed to be fixed and incremental learning is performed so that the change in parameters is controlled reduced. Kirkpatrick et. al [13] propose the elastic weight consolidation (EWC) method in which S2 is calculated as diagonal approximation of the empirical Fisher Information Matrix. The second type of regularization-batt
approaches is based on knowledge distillation [14, 6]. Li et. al [14] propose to use Learning without Forgetting (LwF) keep the representations of base data from drifting too much while learning the novel tasks. In the rehearsal approaches, the models strengthen memories learned in the past through replaying the past information periodically. They usually keep small number of exemplars [22, 29, 3], or generate synthetic images [26, 19] or features [12, 30] to achieve this purpose. The rehearsal method for class-incremental learning is first proposed in iCaRL [22], and has been applied in majority of the class-incremental learning works.
2.3 Class-incremental Object Detection
Class-incremental object detection are relatively less explored than its image classification counterpart. All the proposed methods follow the regularization-base approach, in which knowledge distillation is used to address the catastrophic forgetting issue. Shmelkov al [27] proposes Fast R-CNN- based class- innse-inceeeenall object detection model to address the catastrophic forgetting problem, where EdgeBoxes [41] is used produce bounding box proposals. However, the proposals generation stage introduces an immense computational cost. Sub-optimal performance for the base and novel classes obtained since a sub-optimal two-stage object detection model is used. Zhou al [40] proposes an incremental version of Faster R-CNN that distills selected anchors and proposals with the Pseudo- Positive-Aware Sampling strategy. However, it fails to improve the benchmark [27] on the standard evaluation criteria. The Dlass-incremental object detection methods mentioned above are impractical in the real-world since the key success factor is the utilization of co-occurred unlabeled base objects with novel objects in the novel training data. Zhang et. al [31] presents plass-incrementae learning paradigm called Deep Model Consolidation (DMC) for single-shot object detection architectures. which combines the base model and the novel model leveraging external unlabeled auxiliary data. In this work, we develop a novel mechanism for the task of class-incremental object detection based on the Faster R-CNN framework. Our framework is dual-teacher that distill knowledge from the teacher base and novel models to the student model using sampled unlabeled in-the-wild data, without requiring the co-occurrence of base and novel object classes.
3

---

Masse
4g Moni
blox VotUR
Fiae
T
Sle. (G Eog)))
pip
Missse
Q-2 Pponsi
bbox voting
pscudn g Ye (Cassem Sai Goog)
Figure 1: Overview of proposed blind sampling strategy. Refer to the text for more details.
3 Our Approach
3.1 Preliminaries
Let (z, y) E 'D denotes dataset which contains images z and their corresponding object bounding box labels We further denote the training dataset of the base classes and the novel classes as Drase and novel, respectively. Following the definition of class-incremental learning, we only have access to the novel class data Dovel where Unovel E Co+1...., Cs+no Oog, The base class data
Dose, where Vhus E [CC,.... Cs, Cog. are no longer accessible. Cog, and Cog. are the background class of the novel and base data, respectively. Unlike other existing incremental learning methods for object detection where the base classes can still occur unlabeled background classes in the novel training data, we enforce the more realistic strict non co-occurrence { (Ci+1..... Co+ns Che., {CI..... Cb. Cog, 0 in our training data. We further assume that large quantity of unlabeled in-the-wild data is accessible, from which we sample Dunlabel using our blind sampling strategy. The base model Musse iDrose 0rose) ) is object detector trained on the base class data, where O denotes the learnable parameters. Our goal is to train an object detection model Matual ('a uuntaaell aadd) detect the novel classes [Cb+1,.... Cb+ns Cog, } without catastrophic forgetting the base classes {Cr,.... Co Cbg. J. We use Faster R-CNN [24] our object detector.
3.2 Blind Sampling Strategy
Although large amounts unlabeled mi-the-wild data are easily obtainable, most of them are not useful for alleviating the catastrophic forgetting problem. Naive training on these unlabeled in-the- wild data increases training time and can be detrimental to the network performance. We propose to circumvent this problem by sampling useful data from the large amounts of unlabeled in in-the-wild data to build the sampled unlabeled dataset Dunlabel. Furthermore, it important to note that not necessary for the in-the-wild data to contain any object in the base and novel classes. We postulate that false positives from the in-the-wild data with non-overlapping object classes from the base and novel classes can also serve useful training data in our dual-teacher framework. This is based on the intuition that the features maps and response heatmaps of false positives closely resemble the positive samples.
As shown in Fig. 1, we use do the blind sampling with the base Mbuse ('Dnsee Orase) novel Munwel DDoovll; (nnvel) models that are pre-trained on the base Dinee and novel D training data, respectively. Specifically, we first feed the input image into both Mbuse and Muovel- The image selected if the object class probability ge. V c {Ci...., Co. Obg, in the base model Mosse

---

Dunlabel
Moase
Rols Feat of Mstud
Rol Mask
BCNN
LMM
shared Rols
dist
Remodeling
M stud
Maaso Mnove
Remodeling
shared Rois
LM etse
Mask
dest
RCNN
Base classes
Maovel
M
Novel classes
Background
Figure Overview of our dual-teacher distillation framework for object detection incremental learning. Blue, orange and black branches denote the dual-teacher base and novel models, and the student model, respectively, Dash pink lines denote the knowledge distillation process with our proposed loss functions CRENN LIM and CIN Best viewed in color.
E (Co+1,.... Cotny Org } in the novel model Mnooel is greater than pre-defined threshold. We use two different thresholds: 1) Ohase for Mbase and 2) O'novel for Mnovel- We further enhance the accuracy and precision of the blind sampling strategy by random transformation consistency check. To this end, we augment the input image by applying random scaling Teate and horizontal flipping Thip- The original image, the randomly scaled and the randomly flipped images are then fed into the object detection network. We then apply the inverse scaling T and horizontal flipping Tmp on the respective outputs. The input image is selected into Dunlahel only the predicted outputs are consistent over the random transformations. We improve the precision of the estimated bounding boxes by bounding box voting [7], where the ensemble results of the regression bounding boxes from the augmented images are used as the pseudo ground truths in our subsequent dual-teacher distillation framework.
3.3 Dual-Teacher Distillation
As illustrated in Fig. 2, we use the base Mhose and novel M models as the dual teachers in our dual-teacher knowledge distillation framework. The objective is train a student model Matud that inherits the ability to detect all the foreground object classes in the base and novel classes, i.e. {CI..... Co, Cb+1,..., Cb+r. Cognn }. We denote the background class of the student model as Ohg.mn All Mosse Mnovel and Msuu are the same Faster R-CNN object detection network.
Remodeling prediction outputs. For knowledge distillation the object detection head of Faster R-CNN, we first select the informative foreground region of interests (Rols) proposals from the 256 candidates generated by the region proposal network (RPN) of the two teacher models, respectively. Subsequently, the combination of the selected RoIs from the base Musse and novel M models are shared to the student model Matud- These selected foreground Rols are passed through the R-CNN module the student model compute the prediction outputs (q...t r-=tnd), where is predicted probability and r is coordinates of the predicted bounding box. The output foreground Rols of the two teacher models (gaase, yrhase) and (g.ovel, y-hovel) serve as the targets. However, the student model a + n-class object detector, while the base model and the novel model are b-class and n-class object detector, respectively. This means that the output object class probabilities and the bounding boxes between the teacher and student models cannot be compared directly.
To alleviate this problem, we propose the remodel the prediction outputs of the student model in accordance the targeted teacher model. For the distillation of the base model, we remodel the

---

non-overlapping classes from the student model with the base model into the background class of the base model: 9. huudhase = (i
qou + Poot, + + ) ). qudd gutud ..... (udd J. Similarly for the
distillation of the novel model, we remodel the non-overlapping classes from the student model with the novel model into the background class of the novel model, i.e. get novel { + ge, + gea - stud ... Tos ); 4664) (ouud }. We do the same remodeling on the regression head of the object detectors. The regression outputs reme of the student model are remodeled into two parts pamud and F pstad,oovel according to the number of shared Rols with the base and novel models, respectively.
Finally, the output class probabilities and bounding boxes of the student and teacher models are be directly compared in the R-CNN distillation loss function given by:
CRCNN dist
bussy + SLmoooth,, (patud.base, ,lusse -LI div (log( Guuud.baso), ) 9e
+ Lai_iiv div (log( de @suddavvel) go + >.smoothz, (Sstudavvll : novel)
where La div is the KL-divergence loss between the class probabilities of the student and teacher models. Following [8], Lamooth,, is robust LI loss between the bounding box parameters of the student and teacher models. Ả is a hyperparameter to balance the KL-divergence and Smooth-L losses. Intuitively, dist supervises the student model Matud to produce outputs that are close the base Mbas and novel Maovel teacher models.
Image-level distillation with RoI masks. [25] shows that intermediate-level supervisions from the hidden layers of the teacher model can guide the student model towards better generalization than supervision on only the output predictions. However, the approach cannot be naively applied to our incremental learning setting. A direct knowledge distillation the full feature maps of the dual teachers causes conflicts and thus hurts the overall performance. Specifically, the base model teacher would wrongly instruct the student model suppress the novel classes as the background class, and vice versa. To mitigate these conflicts, we use the pseudo ground truth bounding boxes of the foreground classes from the blinding sampling step as Rol masks Mask hase and Mask."vel These Rol masks applied to the feature map distillation loss focus the attention the regions of interest. Concurrently, these masks prevent the confusion of the background classes from the two teacher models. We write the image-level distillation loss with the Rol masks on the feature maps as:
dist
ΣΣΣ
Maski
base stud
base Eylk
Mosk
novel
pstud 1igh
pnovell Eyk
2/Nbuse
2N
(2)
where busee
p.novel and gostud denote the feature of the teacher base and novel models, and the student model, respectively. W. H and are the width, height and channels of the feature map.
Mask base
! xi=l Ej=1 Ma ppuuse
novel
Instance-level distillation with heatmaps. Since the image-level distillation can lead to domination at instance-level, we introduce an instance-level distillation to balanced attentions on all instances. We define the instance-level distillation loss with the response heatmaps of the object detection models. Each location on the heatmap and its response represent the degree of influence on the model prediction outputs from a pixel on the input image. Let us denote the features of the object proposals generated by the base, novel and student models as fruse, fnovel and foud respectively. The heatmaps of the base, novel and student models are then given by channel-wise average pooling and fnovel), clement-wise Sigmoid activation S(.), i.e. Hbe t D f ), H = 5 D ijk
and M S De fijyel over the spatial locations 1,.... W and = 1,... H. To enforce consistency between the student model and the two teacher models instance level, design the instance-level distillation loss CIN as the mean squared error (MSE) between the student heatmap Hstdd and the union of the heatmaps Hbase U Haovel from the two teacher models, i.e.
C
IN dist
Lmse (Hchase H H Hald))
The overall loss Liotal to train the student model incrementally on the sampled unlabeled in-the-wild dataset Duntahel given by:
Overall loss
Lioral CRCNN + CRPN + a. CRCNN
RCNN + Cl + az
(4)
6

---

where CRCNN and C are the loss terms for the R-CNN and RPN module of the two-stage detector Faster R-CNN, supervised by the pseudo ground truths from the blind sampling step. ar o22 and ag are the hyperparameters to weigh the loss terms.
4 Experiments
4.1 Experimental Setup
Datasets and metrics. Following [27]. we evaluate our proposed method for class-incremental object detection on the PASCAL VOC 2007 and MS COCO 2014 datasets. PASCAL VOC 2007 consists of about 5K training and validation images and 5K test images over 20 object categories. Models are trained on the trainval set and tested on the test set. MS COCO 2014 contains objects from 80 different categories with 83K images in the training set and 41K images in the validation set. We train models on the training set and evaluate models on the first 5K images of the validation set. In the test stage, the mean average precision (mAP) is used as the evaluation metrics. We report the COCO style (mAP [0.5 : 0.95)) detection accuracy for MS COCO dataset, and PASCAL. style (mAP [0.5]) accuracy for PASCAL VOC dataset. Additional average precision and recall across scales are also reported, which is in line with standard evaluation protocol of MS COCO. To evaluate the compared methods under the setting with large amount of in-the-wild unlabeled data for the PASCAL VOC and MS COCO target datasets, we take the MS COCO and Open Images datasets as the unlabeled data. We also run the experiments under a more strict setting that excludes all MS COCO images that contain any object instance of 20 PASCAL VOC categories to avoid any potential advantage from the classes of PASCAL VOC appearing in MS COCO. We denote this setting as "w/o category".
Implementation details. We use ResNet-50 with frozen batch normalization layers as the backhone network. The training methodology is the same as standard Faster R-CNN. We use stochastic gradient descent with Nesterov momentum to train the models in all experiments. The initial learning rate is set to 1e-3 and subsequently reduced by 0. after every 5 epochs for the previous model and the current model. Each model is trained for 20 epochs for both PASCAL VOC and MS COCO datasets. The training is carried out on RTX 2080Ti GPU, and the batch size is set to 1.
4.2 Comparison of Methods
Addition of one class. Table shows the re- sults of one addition class incremental learning. We take the first 19 classes alphabetical order from PASCAL VOC as the base classes C1, Co and the remaining class is used as the novel class Cb+1- In addition to the disjoint base and novel classes following the definition of class- incremental learning, also exclude images that have co-occurrence of any base and novel ob- jects. We report the mean average precision (mAP) the base, novel and all classes, which we de- note as base novel all. The first three rows show the evaluation results of the "1-20", "1-19", "20" baselines without using incremental learn- ing. Furthermore, we compare to the state-of-the- art incremental object detection method [27] on training data with ("w co occur") and without ("w/o co occur") co-occurrence of the base and novel classes. In the absence of co-occurrence. we can see that [27] suffers drop in the performance of the novel class from 62.7% to 39.2% mAP, and base class from 68.5% 62.6% mAP. It should also be noted that the mAPs of [27] are significantly lower than the baselines without incremental learning. In contrast, our approach without co-occurrence and with class overlap in the in-the-wild data ("w calegory") achieves mAPs of 73.3% 50.7% 72.2% (base novel all). Our result is almost on par with the results of the baseline training with all classes "1-20" without incremental learning at 73.1%| 55.4% 72.3%. Additionally, also see that our method still achieves high mAP of 71.5%
Table 2: Results of "19+1" on VOC fest set. "1-19" and "20" ("tv") are base and novel classes. "base
novel all" is mAPs for base, novel and all classes.
Row 1-3 are baselines without incremental learning.
mAP(%)
Class
Method
(base novel all)
1-20 (wlo co-occur)
Ren [24]
73.1
55.4
1-19 (wio co-occur) 20 (w/o co-occur)
Ren [24]
73.4
Ren [24]
17.4
Shmelkov [27]
62.6
39.2
61.4
(1-19) + (20) (w/o co-occur)
Ours (w catleaor)
73.3
Ours (w/o caloggory
71.5
46.1
70.2
(1-19)+ + (20) co-occur)
Shmelkev [27]
68.5
62.7
08.3
7hou [40]
70.5
53.0
69.6
Ours (w calegory)
73.5
65.8
73.1
46.1% 70.2% even when
7

---

there is no class overlap in the in in-the-wild data ("w/o category"). This supports our postulation that false positives sampled from the in-the-wild data can also benefit our dual-teacher incremental learning framework. Interestingly, we can also run our dual-teacher distillation framework on the "w co occur" data. It can be seen that we significantly outperform [27] and [40].
Table 3: Results of "15+5" on VOC test set. "1-15"
Table 4: Results of "10+ 10" on VOC test set. "1-10"
and " "16-20" are the base and novel classes.
and "I 11-20" are the base and novel classes.
mAP(S)
mAP(S)
Class
Method
(bose
Nocel
all)
Class
Method
(base]
nozel
al/)
1-20 (wis co-necur)
Ren [24]
74.4
|
62.7
171.7
1-20 (wlo co-occur)
Ren [24]
66.5
69.0
67.7
1-15 (wlo co-occur) 16- 20 (wl co-oceur)
Ren [24]
72.0
1-10 (Wlo co-oecur]
Roo [24]
57.8
Ren [24)
|48.6
11 11-20 (wl co-occur)
Ren [24]
63.2
(1-15)+ (16-20) (wlo c0-occur)
Ours [W category)
Shmelkov [27]
67-2
70.5
46.1
49,4
62.00
65.3
(1-10) (11-20) (w/a occur)
Ours (w category)
Shmelkov [2?]
58.5
57.6
19.1
62.2
53.8
59.9
Ours (w/o categooy)
70.7
48.5
65.1
Outs (wlo catogory)
62.3
60.4
Class (1-15) + (16-20)
(w co-occur)
Ours: [w category)
Shmelkos [27]
72.7
08.4
58.4
58.4
665.9
69.1
(1-10) (11-20)
Shmelkov [27]
Zhou [40]
63.2
63.1
63.1
61.8
(W 00-OCCur)
Ours (W calegory)
69.2
68.3
688.7
Addition of a group of classes. Table shows the results on 5 novel classes. We can see that our proposed approach "w/o co occur" and "w & w/o calegory" achieves mAPs that are close to the baseline (without incremental learning) on the base (Ours "w & w/o calegory": : 70.7% & 70.5% 5% vs. "1-15": 72%) and novel (Ours "w & w/o category": 48.5% & 49.4% vs. "16-20": 48.6%) classes. Furthermore, we achieve higher performances compared to [27] (Ours: 70.7 48.5 65.1 vs. [27]: 67.2 46.1 62.0) when trained without co-occurrence of the base and novel classes. Table shows the results on 10 novel classes. We can see that our methods "w/o co occur" and "w & w/o category" achieve performances that are almost par with the baselines without incremental learning on the base (Ours "w & w/o category": 57.6% & 58.5% vs "1-10": 57.8%) and novel (Ours "w & w w/o category": 62.2% & 62.3% vs. "11-20": 63.2%) classes. Moreover, our method significantly outperforms [27] on the novel classes when trained without co-occurrence (Ours "w calegory": 62.2%: Ours "w/o calegory": 62.3% vs. [27]: 49.1%). In Tables 3 and 4, also show that our method outperforms [27] and [40] under the "wo co occur" setting. These results show the effectiveness of our proposed approach when group of novel classes are added.
Addition of classes sequentially. To prove that our method is also efficient for sequential incremental learning, we train the model by sequentially adding new groups of novel classes. In our experiments, we use "1-10" as the base classes, and "11-15" and "16- 20" are two groups of novel classes to be sequentially added. Tables 5 shows the results of sequential incremental learning. We can see that our method shows on par performance with [27] under "w o co occur" on the base classes when the novel class group of
Table 5: Results of "10+5+5" on VOC test set. "1-10" are the base classes, and "11-15" and "16-20" are the two groups of sequentially added novel classes.
mAP(%)
Class
Method
(buse)
novel
all)
1-20 (w/o co-occur)
Ren [24]
66.6
67.3 66.7
1-10 (wlo co-occur)
Ren [24]
57.8
11-15 (w/o co-occur)
Ren [24]
62.4
16-20 (w/o co-occur)
Ren [24]
48.6
Shmelkov [27]
59.8
52.4
57.3
(1-10)+ (11-15)
Ours (w category)
57.0
62.7
58.9
(wlo co-occur)
Ours (w/o category)
57.3
61.7
58.8
"11-15" (Ours "w & w/o category": 57.0% 57.3% vs. [27]: 59.8%) and "11-15"++ 16-220" (Ours w & w/o category: 56.7% & 56.9% vs. [27]:
Shmelkov [27]
59.0)
47.3
53.1
(1-10)+ (11-15)+ (16-20)
Ours (w category)
56.7
55.1
55.8
(wlo co-occur)
Ours (w/o category)
56.9
53.9
55.4
(1-10)+ (11-15)+ (16-20)
Zhou [40]
60,3
53.1
56.7
59.0%) are added. Under the same set- tings, our method significantly outper- forms [27] on the sequential addition of the two groups of novel classes "11-15" (Ours "w & w/o category" 62.7% & 61.7% vs. [27]: 52.4%) and 111-15"+"16-20 (Ours "w & w/o category": 55.1 & 53.9% vs. [27]: 47. 47.3%). We also outperform [40] significantly under the "w co - occur" setting.
(W co-occur)
Ours (w category)
68.1
64.8
66.55
To demonstrate the generality of our method, we conduct experiment using the validation set of Open Images as the unlabeled data. Specifically, we sample 1.5 x 10a images for D Duntabel from the validation set of Open Images with our blind sampling strategy in the "19+1" setting.
8

---

Table 7: Results of "40+40" COCO minival set. First 40 classes are the old classes, and the next 40 are the added classes.
Class
Method [24]
AP
AP50
AP75
APM
APL
(w/o co-occur)
27.7
45.8
29,4
30.9
42.5
(1-40) + (40-80)
Ours
22.5
40.9
23.0
8.3
25.9
34.6
(w/o co-occur)
Shmelkov [27]
21.3
37.4
(1-40) (40-80) (w en-occur)
Zhou [40]
22.7
36.8
Ours
23.7
42.5
24.3
8.6
26.6
37.5
The results shown in Table 6. We can that a remarkable performance improvement of 8.7% and 9.4% mAP is achieved by our method compared to [27] for the base and novel classes, respectively. In addition,
Table 6; Results of "19+1" on VOC test set. "1-19" and "20" are the base and added novel classe(s).
Class
Method
(base:
mAP(S)
novel
all)
significantly out- perform [27] by large margin of 8.7% mAP on all classes. These results demonstrate the su- perior ability of our proposed method for class- incremental object detection.
1-20 (wfo co-occur)
Ren [24]
73.1
35.4
72.3
1-19 (w/o co co-occur)
Ren [24]
73.4
16-20 (wlo co-oceur)
Ren [24)
47.4
(1-19)+ (20)
Shmelkov [27]
(w/o co-occur)
Ours
71.3
48.6
70.1
Table shows the results further experiments COCO with the test set of Open Images as the unlabeled data. In particular, we split the 80 classes into "40+40" setup. For fair parison, we report results on the minival dataset following [27]. Our method achieves better performance of 22.5% mAP without co-occurrence of base objects and novel objects compared to [27] with the unfair advantage of co-occurrence. Further- more, can see that our method significantly outperforms [27] and [40] under the "w co occur" setting.
(1-19) + (20)
Shmelkov [27]
68.5
62.7
68.3
(w co-occur)
Zhou [40]
70.5
53.0
4.3 Ablation Studies
Effect of the amount of unlabeled data. Fig. 3 illustrates the effect of the amount of unlabeled data used for training. For "19+1", we sam- ple 10 5 x 10 10 1.5 10 10a images using our blind sam- pling strategy for the "w category" setting. We sample 10 5 x 10 104 1.5 10ah images for the "w/o category" due to the limited amount of "w/o category" data in un- labeled MS COCO dataset. We re- port the mean average precision over all steps. Overall, our method can even outperform the previous state-of- the-art by only using 10 unlabeled "w/ category" images.
Figure 3: Results of "19+1" on VOC test set with varying amount of unlabeled data.
estegory
eesia category Shadkkavi27
74
mAP
Si0
1998)
1506
Amount unlabeled data
Effect of the main configurations. Table 8 shows the ablation studies to
understand the effectiveness of each component in our framework. The ablated components include: R-CNN distillation; Image-level distillation; 3) Instance-level distillation. The first row the result of only the standard Faster R-CNN loss, trained with pseudo ground-truth data generated by the two teacher models on the unlabeled in-the-wild dataset. We can see that this setting gives the lowest performance, which is evident on the importance of our three distillation losses. The subsequent inclusions of the three respective distillation losses in Row 2-5 show improvements over the standard RCNN baseline. These results further demonstrate the effectiveness of

---

Table 8: Ablation studies for the setting of "19+1" on VOC 2007 test set.
Blind sampling strategy
CRCNN + CRPN
L
CRNNN List
CIM dist
IN dist
mAP(%)
(base
novel
all)
64.4
35.3
62.9
68.0
44,9
66.9
68.5
39.6
67.1
67.6
39.2
66.2
70.0
44.3
68.7
71.5
46.1
70.2
the remodeling prediction outputs (istt Rol masks dist and heatmaps (Calg) on our method. Finally, the last row shows the best performance with our proposed blind sampling strategy and distillation losses.
Conclusion
In this paper, we present novel glass-incremental object detection for more challenging and realistic scenario when there is no co-occurrence of base and novel object classes in images of the novel training dataset. This contrasts with other existing methods whose key success factor the co-occurrence. We propose the use of unlabeled in-the-wild wiih data to bridge the non co-occurrence caused by the missing base classes during the training of additional novel classes. A blind sampling strategy proposed to select smaller set of relevant data for incremental learning. We then design a dual- dual-teacher knowledge distillation framework with three levels of distillation losses to transfer knowledge from the base- and novel-class teacher models to the student model using the sampled in-the-wit wild data. Extensive experimental results on benchmark datasets show the effectiveness of our proposed method.
6
Acknowledgement
The first author is funded by scholarship from the China Scholarship Council (CSC), This research supported in part by the National Research Foundation, Singapore under its Al Singapore Program (AISG Award No: HISG2-RP-2020--2 and the Tier 2 grant N0ET2EP20120-000 from the Singapore Ministry Education.
References
[1] Yancheng Bai, Yongqiang Zhang. Mingli Ding, and Bernard Ghanem. Finding tiny faces in the wild with generative adversarial network. In Proceedings the IEEE conference computer vision and pattern recogmition, pages 21-30, 2018.
[2] Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem. Sod-mtgan: Small object detection multi-task generative adversarial network. Proceedings of the European Conference on Computer
Vision (ECCV), pages 206-221, 2018.
[3] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-ene incremental learning. In Proceedings of the European conference computer vision (ECCV),
pages 233-248, 2018.
[4] Li Chen, Chunyan Yu, and Lvcai Chen. A new knowledge distillation for incremental object detection. In
2019 International Joint Conference on Neural Networky (IJCNN), pages 1-7. IEEE, 2019. [5] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region based fully convolutional networks. In Advances in neural information processing systems, pages 379-387, 2016.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learning without memorizing, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5138-5146, 2019.
[7] Spyros Gidaris and Nikos Komodakis, Object detection via multi regiti and semantic segmentation-awarr cnn model. In Proceedings of the IEEE international conference on computer vision, pages 1134-1122 2015.
[8] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-144 1448, 2015.
10

---

[9] Ross Girshick, Jeff' Donahue. Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 580-587, 2014.
[10] Yu Hao, Yanwei Fu, Yu-Gang Jiang, and Qi Tian. An end-to-end architecture for class-incremental object detection with knowledge distillation. In 2019 IEEE International Conference on Multimedia and Expo
(ICME), pages 1-6, IEEE, 2019.
[11] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis Machine Intelligence, 37(9))1904-
16, 2014,
[12] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv
preprint 1r1 10563, 2017.
[13] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Bariisska,a et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 14(13):3511-352 2017.
[14] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE transactions on pattern andlyii and
machine intelligence, 40( 11:229955-29477 2017.
[15] Tsung Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision Pattern Recognition, 2017.
[16] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEF international conference computer vision, pages 2980-2988,
2017.
[17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision,
2016.
[18] Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
[19] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11321-11322 1139,, 2019.
[20] Juan-Manuel Perez-Rua, Xiatian Zhu, Timothy M Hospedales, and Tao Xiang. Incremental few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13846-13855, 2020.
[21] Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by lcarning and forgetting functions. Psychological review, 97(2):285, 1990.
[22] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental
classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[23] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[24] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Sun Jian. Faster r-enn: Towards real-time object detection with region proposal networks. International Conference on Neural Information Processing Systems, 2015.
[25] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua
Bengio. Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.6550, 2014.
[26] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30:2990-2999, 2017.
[27] Konstantin Shmelkov. Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE International Conference on Computer Vision, pages 3400- 3409, 2017.
[28] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104(2):15-171, 2013.
[29] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu. Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
pages 374-382, 2019.
[30] Ye Xiang, Ying Fu, Pan Ji, and Hua Huang, Incremental learning using conditional adversarial networks.
In Proceedings of the IEEE International Conference on Computer Vision, pages 6619 6628, 2019.
[31] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In The IEFE Winter Conference Applications of Computer Vision, pages 1131-1 140, 2020.
[32] Yongqiang Zhang, Yancheng Bai, Mingli Ding, and Bernard Ghanem. Multi-task generative adversarial network for detecting small objects in the wild. International Journal of Computer Vision, pages 1-19, 2020.
[33] Yonggiang Zhang, Yancheng Bai, Mingli Ding, Yonggiang Li, and Bernard Ghanem. W2f: weakly- supervised to fully-supervised framework for object delection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 928-998888822 2018
11

---

[34] Yongqiang Zhang, Yaicheng Bai, Mingli Ding, Yongqiang Li, and Bernard Ghanem. Weakly-supervised object detection via mining pseudo ground truth bounding-boxes. Pattern Recognition. 84:68-81, 2018.
[35] Yonggiang Zhang, Yancheng Bai, Mingli Ding. Shibiao Xu, and Bernard Ghanem. Kgsnet: key-poing guided super-resolution network for pedestrian detection in the wild. IEEE transactions on neural networks and learning systems, 2020.
[36] Yongqiang Zhang, Mingli Ding, Yancheng Bai, and Bernard Ghanem. Detecting small faces in the wild based on generative adversarial network and contextual information. Pattern Recognition, 94.74-86, 2019.
[37] Yongqiang Zhang, Mingli Ding, Yancheng Bai, Dandan Liu, and Bernard Ghanem. Learning strong detector for action localization in videos. Pattern Recognition Letters, 128:407-413, 2019.
[38] Yongqiang Zhang, Mingli Ding. Yancheng Bai, Mengmeng Xu, and Bernard Ghanem. Beyond weakly supervised: Pseudo ground truths mining for missing bounding- bounding-boxes object detection. IEEE Transactions
Circuits and Systems for Video Technology. 30(4):983 997, 2019.
[39] Yongqiang Zhang, Mingli Ding, Wuyifang Fu, and Yongqiang Li. Reading recognition of pointer meter line. In Ninth International Conference based on pattern recognition and dynamic three-points
on Machine Vision (ICMV 2016), volume 10341, page 103410K. International Society for Optics and Photonics, 2017.
[40] Wang Zhou, Shiyu Chang, Norma Sosa, Hendrik Hamann, and David Cox. Lifelong object detection.
arXiv preprint arXiv:2009.00 129, 2020.
[41] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In European conference on computer vision, pages 391-405. Springer, 2014.
12


Reviews:
[{"review_id": "jH51drLGcTF", "reviewer": "Reviewer_m17V", "summary": "The paper proposes a more realistic setting for class-incremental object detection, removing the need for co-occurrence if base and novel object classes in images in the training dataset. They propose a slightly complex method to address this scenario by assuming co-occurrence of the base and novel classes in supplementary, unlabeled data and distilling a model which learns from that data along with the training set using three levels of distillation losses and separate teacher models for base classes and novel classes that are distilled into a single, joint student model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper points out the co-occurrence in training data assumption underlying much of the recent work in incremental object detection. The assumption that novel classes will co-occur with examples of the base classes is still central to this work, the main difference as I understand it based on this paper is the access to larger unlabeled datasets to have a larger pool that is likely to provide examples of base and novel classes.  This point would be made stronger with some additional data analysis and visualization. In the novel class data subsets in question, how frequently do novel classes occur with no co-occurrence to base ones? How many base classes are not represented at all? I know these datasets are not fully labeled, but your best model’s predictions could be used as an estimate of the level of co-occurrence/non-occurrence and the number of examples of each class in the training data (required by the SOA methods cited in related work) vs the co-occurrence and number of examples seen in the unlabeled data. \n\nThe paper only compares to a single benchmark, which it outperforms across the board. However, if I am understanding correctly this benchmark does not get to see the significant amount of unsupervised data that their model does. How much of the performance gain is just through seeing more and more varied examples of the classes? Is it possible to disambiguate the performance gain from your methodology/proposed loss structure vs the performance gain from the unsupervised data? Or is the design of this methodology inseparable from the access to novel data to distill from?\n\nSome minor grammatical errors throughout. Some additional editing for grammar is needed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have not addressed this as far as I can tell. One possible limitation is the assumption that all classes of interest will occur in the unsupervised data, allowing them to avoid forgetting. In real-world scenarios or applications there may not be a source of unlabeled data containing examples of all classes to distill from.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see ethical considerations beyond the potential biases in the underlying datasets they use.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MAvuF4-0qFk", "reviewer": "Reviewer_DzAY", "summary": "The paper proposes a technique to address the learning-without-forgetting problem when base and novel classes do not co-occur. The approach relies on sampling unlabeled in-the-wild data from another domain, to bridge base and novel classes. Results against a few recent baselines, as well as ablations, demonstrate the effectiveness of the approach.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is reasonable, but other than the novel setting (non-co-occurrence) the key novelty evades me. There are also some descriptions that were unclear. \n\n1. In particular, while it is clear why co-occurrence of base and novel classes shouldn't be assumed, it is not clear why strict non-co-occurrence is enforced (L135). \n\n2. From the description in L152-166, it is not clear to me where the data to be used in D_{unlabel} comes from. The previous text talks about D_{base} and D_{novel}. This is also not shown (perhaps just not labeled) in Fig. 1. \n\n3. The consistency check (L158) does not seem novel, and is in line with prior work on self-supervised learning (e.g. SimCLR). \n\n4. The right-hand side of Fig. 2 needs more explanation (e.g. textual labels for the different components). \n\n5. It was not clear to me why non-affection masks (where does the name come from?) alleviate confusion (L209). \n\n6. Experiments are strong and convincing, including ablations.\n\nMinor:\n(a) Why is the subscript \"s\" used for novel classes, and \"t\" for base?\n(b) Why not use D_{base}, D_{novel} as an argument to M_{stud}?\n(c) Typos, D_{basel} (L153) and \"are be\" (L193)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not discussed", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "Hm70-hlEnRt", "reviewer": "Reviewer_zfwt", "summary": "This paper investigates incremental object detection problem without making the assumption of base and novel classes occurring together, i.e. person (base) and TV(novel) objects being in the same image. The authors make the argument that this does not always happen in real-world scenarios, and they propose to sample in-the-wild data instead. They call this stage the \"Blind sampling strategy\", and this is achieved by applying the pre-trained Base and Novel models to the unlabeled data, and collecting those that have \"positive\" output. After the blind sampling strategy, a student model is learned by distilling the knowledge from the base and teacher models. Finally, the method is evaluated on PASCAL VOC and MS COCO datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is to use the in-the-wild data, rather than the data with co-occurence for incremental object detection. The authors do not make the assumption that novel and base classes will co-occur in the images, which makes the problem more challenging and the solution more general. However, other than this, the technical contribution of this paper is very limited, as dual-teacher distillation is simply applied to distill the information between multiple models. \n\nRegarding the blind sampling strategy, I would have found it beneficial if the authors had shown more ablations and experiments for this part since it is the main contribution. For example, what happens if the data is randomly selected rather than using pre-trained models? Can the pre-trained models minimize the impact of true negatives during the distillation in this case? Also what happens if different sources of unlabeled data are used? For example, does the performance change if Open Images dataset was used as the unlabeled data instead of MSCOCO for the PASCAL VOC experiments?\n\nThe dual-teacher distillation can be costly in terms of memory and complexity, as three models (two teachers and a student) need to be used during the training. Can the authors discuss why two teachers are required? For example, assuming that the novel data is available at the same time as the unlabeled data, can the student be learned with novel+unlabeled data while distilling information from a single teacher model (learned with base classes)?\n\nOverall, the paper is nicely written and offers a more generalized solution for incremental object detection. It would benefit from additional discussions/experiments as described above. My initial evaluation is leaning towards acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitation nor societal impacts in the submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "6tewrZ6To9", "reviewer": "Reviewer_SGgv", "summary": "Authors address the problem of catastrophic forgetting in the context of object detection. \nThey propose a novel class-incremental object detection method based on Faster R-CNN for when there is no co-occurrence of base and novel object classes.\nThey also introduce a blind sampling strategy to select data from the large in-the-wild dataset for incremental learning.\nTheir dual-teacher distillation framework trained on this data outperforms SOTA on COCO and VOC datasets.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Object detectors are an important class of models that are heavily used in practice and indeed one of the main challenges is adapting these models to learn to detect  new classes objects rather than training them from scratch. In this regard, it is a significant challenge that is addressed in this paper. \n\n\nL 52: “we first introduce a blind sampling strategy to select relevant data from the in-the-wild data that contains large amounts of irrelevant images with neither the base nor novel class information.”\nHow do you ensure that? Do you mean the objects from the base and novel class do not exist in this dataset?!\n\n\nL 69: Instead of assuming fully no co-occurrence of the base classes and the novel classes, wouldn’t it be more realistic to assume partial overlap as has been done in previous work?!\n\n\n\n\nEvaluation\n\nL 252: …  subsequently reduced by 0.1 after every 5 epochs? How? divided?\n\n\nIn table 1, It is odd that your method (last row) beats the baseline (row 1) where all data is seen at once? Why is that so?\nAlso, here the experiment is conducted in only one scenario (with tv being the novel class). It is important to repeat this experiment multiple times and also consider other permutations of base and novel classes to make sure results are statistically significant. Same argument applies to the results presented in other tables.\n\n\nResults in Table 6 over COCO dataset does not seem complete and convincing! I think this is important as being the only result over COCO. Other experiments are over VOC which is easier to overfit. COCO is much more challenging and is used widely. \n\n\nTypos: \nL 307: Tables 4 shows -> tabe \nL 363 Overall, our method can even outperform the previous state-of. (Move even to later in the sentence)\n\n\n\n\n\n\nAuthors have addressed an important and practical problem and results of their experiments are generally convincing although I have some concerns as raised above. I am leaning towards acceptance after considering the authors’ rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors admit not discussing the limitations\n\n(b) Did you describe the limitations of your work? [No]", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for incremental object detection, which removes the need for co-occurrence of base and novel object classes in images in the training dataset. The reviewers scored the paper with (5, 6, 6, 6). After discussion and rebuttal,
they lean towards accepting the paper as a poster. 
Decision: Accept (Poster)