Task Regularized Hybrid Knowledge Distillation For Continual Object Detection
Authors: Jinming Zhang
Mengxue Kang
Jinpeng Zhang
Zhe Ma
Shan Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge Distillation
Continual Learning
Continual Object Detection
Class Incremental Object Detection

Abstract: Knowledge distillation has been used to overcome catastrophic forgetting in Continual Object Detection(COD) task. Previous works mainly focus on combining different distillation methods, including feature, classification, location and relation, into a mixed scheme to solve this problem. In this paper, we propose a task regularized hybrid knowledge distillation method for COD task. First, we propose an image-level hybrid knowledge representation by combining instance-level hard and soft knowledge to use teacher knowledge critically. Second, we propose a task-based regularization distillation loss by taking account of loss and category differences to make continual learning more balance between old and new tasks. We find that, under appropriate knowledge selection and transfer strategies, using only classification distillation can also relieve knowledge forgetting effectively. Extensive experiments conducted on MS COCO2017 demonstrate that our method achieves state-of-the-art results under various scenarios. We get an absolute improvement of 27.98 at $RelGap$ under the most difficult five-task scenario. Code is in attachment and will be available on github.

Under review as a conference paper at ICLR 2023
TASK REGULARIZED HYBRID KNOWLEDGE DISTIL- LATION FOR CONTINUAL OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation has been used to overcome catastrophic forgetting in Con- tinual Object Detection(COD) task. Previous works mainly focus on combining different distillation methods, including feature, classification, location and rela- tion, into mixed scheme to solve this problem. In this paper, we propose a task regularized hybrid knowledge distillation method for COD task. First, we pro- pose an image-level hybrid knowledge representation by combining instance-level hard and soft knowledge to use teacher knowledge critically. Second, we propose a task-based regularization distillation loss by taking account of loss and cate- gory differences make continual learning more balance between old and new tasks. We find that, under appropriate knowledge selection and transfer strategies, using only classification distillation also relieve knowledge forgetting effec- tively. Extensive experiments conducted on MS COC02017 demonstrate that our method achieves state-of-the thett-rrtt results under various scenarios. We get an abso- lute improvement of 27.98 at RelGap under the most difficult five-task scenario. Code is in attachment and will be available on github.
INTRODUCTION
The existing object detection models (Ge al. 2021 mainly adopt overall learning paradigm, in which the annotations of all categories must be available before learning. It assumes that data dis- tribution fixed or stationary (Yuan et al. 2021). ). while data in real-world comes dynamically with non-stationary distribution. When model learns from incoming data continually, new knowledge interferes with the old one, leading to catastrophic forgetting (McCloskey & Cohen 1989 Goodfel- low et al. 2014). To solve this problem, continual learning is proposed in recent years and has made progresses in image classification (Zeng et al. 2019 Qu et al. 2021). On the other hand, continual object detection (COD) is rarely studied.
Knowledge distillation (Hinton et al., 2015) has been proved to be an effective method for COD task, in which the model trained on old classes performs as teacher to guide the training of student model on new classes. There are four kinds of distillation schemes: feature, classification, location and relation distillation. Most previous works combine feature distillation and classification distillation to construct their distillation methods (Li & Hoiem 2018 [i et al., 2019) Yang et al. (2022b). while the latest work (Feng et al.) (022) combines classification distillation and location distillation to construct a response-based distillation method. In addition, various distillation losses, based on KL diversity, cross entropy and mean square error, are proposed for knowledge transfer. In summary, the keys of knowledge distillation are what knowledge should be selected from teacher and how it is transferred student. The former question needs Knowledge Selection Strategy (KSS), while the latter needs Knowledge Transfer Strategy (KTS).
Continual object detection face two problems. (1) Teacher outputs probability distributions as logits and converts them into one-hot labels as final predictions. Logits and one-hot labels are regarded as soft and hard knowledge, respectively. Soft knowledge contains confidence relations among cate- gories, but brings knowledge fuzziness inevitably. While, hard knowledge has completely opposite effects. Therefore, how to design KSS to keep balance between accuracy and ambiguity of knowl- edge is key problem. (2) Continual learning should maintain old knowledge during the learning of new knowledge to overcome catastrophic forgetting, therefore how to design KTS to keep balance between stability of old knowledge and plasticity of new knowledge is key problem. This paper

---

Under review as a conference paper at ICLR 2023
focuses on how to design effective KSS and KTS for COD task. We demonstrate that long as KSS and KTS are good enough, using only classification distillation can also significantly alleviate catastrophic forgetting improve performance.
Firstly, the max confidence value of logits is always lower than its corresponding one-hot value (equal 1), which brings knowledge ambiguity and reduces supervise ability of teacher. This means soft knowledge is not completely reliable, which should be used critically. However, previous meth- ods ignore this keypoint. Motivated by this insight, we propose an image-level hybrid knowledge representation method, named as HKR, by combining instance-level soft and hard knowledge adap- tively to improve the exploration of teacher knowledge. Secondly, new coming data contains mas- sive labeled objects of new classes, while contains a few unlabeled objects of old classes, therefore student trends be dominated by new classes and falls into catastrophic knowledge. Thus it is very important to balance the learning of old and new classes. We propose a task regularized distillation method, named as TRD, by using losses difference between old and new classes to prevent student from task over-fitting effectively. We first explore imbalance learning problem explicitly for COD,
We propose a hybrid knowledge representation strategy by combing logits and one-hot predictions to make better trade-off and selection between soft knowledge and hard knowledge. (2) We propose a task regularized distillation method as an effective knowledge transfer strategy to overcome the imbalance learning between old and new tasks, which relieves catastrophic forgetting significantly. (3) We demonstrate that, compared with the composite distillation schemes, using only classification distillation with appropriate knowledge selection and transfer strategies can also reach up to the state-of-the-art performance of COD task.
Our contributions can summarized as follows: (1)
2 RELATED WORKS
Continual Object Detection. There are several schemes for COD task. Li & Hoiem (2018) first proposed a knowledge distillation scheme by applying LWF to Fast RCNN (Girshick] 2015). Zheng & Chen 2)22)) proposed a contrast learning scheme to strike balance between old and new knowl- edge. JJsephh et al. (2021b) proposed a meta-learning scheme to share optimal information across continual tasks. Joseph et al. 202 introduced the concept of Open World Object Detection, which integrates continual learning and open-set learning simultancously. In addition, ct al. (202T) first studied few-shot COD. Li et al. 2019) designed COD system on edge devices. Wang et al. (2021) presented an online continual object detection dataset. Recently, Wang et al. (2022) proposed a data compression strategy improve sample replay scheme of COD. Yang et al. (2022a) proposed pro- totypical correlation guiding mechanism to overcome knowledge forgetting. Cermelli et al. (2022)
proposed to model the missing annotations to improve COD performance.
Knowledge Distillation for Continual Object Detection. Knowledge distillation (Hinton et al., 2015) is an effective way to transfer knowledge between models with KL diversity, cross entropy or mean square error as the distillation loss. There are mainly four kinds of knowledge distillation used in COD task: feature, classification, location and relation distillation. LwF was the first to apply knowledge distillation to Fast RCNN detector (Li & Hoiem 2018). RILOD designed feature, classification and location distillation for RetinaNet detector on edge devices (Li et al. (019). SID combined feature and relation distillation for anchor-free detectors (Peng et al. 2021). Yang et al. (202bb) proposed a feature and classification distillation by treating channel and spatial feature dif- ferently. ERD is the latest ofate-of-the-a art method, combining classification and location distillation (Feng et al. 2022). Most of existing methods combine feature, classification and location distillation in composite and complex schemes to realize knowledge selection and transfer.
OUR METHOD
3.1
OVERALL ARCHITECTURE
We build our continual object detector the top of YOLOX (Ge et al. 2021). Fig shows its overall architecture. YOLOX designs two independent branches as its classification and location heads. Firstly, hybrid knowledge selection (HKS) module works after the classification head of teacher to discover and select the valuable predictions for old classes. Secondly, task regularized distillation (TRD) module works between the heads of teacher and student to transfer knowledge.
2

---

Under review as a conference paper at ICLR 2023
Knowledge Selection Strategy
Head
(0,22x3.44
Locatior Prediction
NMS
Teacher
Locottion Solillaiion
3,0.0.0
Cls Head
loght prediction
aneheo predictiion
knowledge
Hybrid Knowledge Representation
Backaone
we Featuree Dist llation
Old Tosk Loss
Cls Prediction
Old Taok
Cis Loss
LOC Loss
Task Rogularizd Distillation
New Taikk Cls & Loc Prediction
Student
Loc Head
Old
New Task Ground-Truth
New Task Loss
Prediction
Loc Feature
Knowledge Transfer Strategy
The overall architecture of our continual detector ilYOLOX. Cls and Loc refer classifi- cation and location respectively. Hybrid Knowledge Representation (HKR, Eq and Task Regular- ized Distillation (TRD, TO) refer to our proposed two components respectively. HKR and TRD play roles of knowledge selection and knowledge transfer strategies respectively. During learning of student on new task, teacher is frozen and outputs predictions of old task. ILYOLOX achieves SOTA performance using only classification distillation without any feature and location distillation.
Figure 1:
3.2 HYBRID KNOWLEDGE REPRESENTATION
Teacher outputs probability distribution as logits and converts them to one-hot labels as final predic- tions. Logits are regarded as soft knowledge, while one-hot labels are regarded as hard knowledge. Hinton et al. (2015) shows that soft knowledge better than hard knowledge distillation. However, although soft knowledge reflects more between-class information than hard knowledge, it also brings fuzziness to knowledge inevitably, which makes student confused during distillation learning. Meanwhile, teacher confidence reflects knowledge quality. If teacher has high confidence about its predictions, we should further strengthen this trend so that student can feel the certainty of this knowledge. Conversely, if teacher has low confidence, we should not do that.
classification
Therefore, the key problem is how to evaluate the quality of soft knowledge from teacher. We here propose evaluate soft knowledge according the confidence difference between the maximum value and the secondary maximum value of teacher logits. Given batch of images, teacher outputs a batch of logits for potential objects. For every logits the batch, if the difference between the maximum confidence and the secondary maximum confidence is larger than threshold, the knowl- edge quality of this logits will regarded as high, otherwise vanilla. High quality knowledge will be represented as one-hot prediction, while vanilla knowledge will be represented as soft prediction. We compute the mean value of the confidence differences across the entire batch as the threshold to judge knowledge quality adaptively. We formulate the description above as follows:
ConfDiff
Conf max Conf secondary
quality
ConfDiff > N Σ ConfDiff:
(2)
Hybrid quality Onehot (1 (1--qul -uulltty Soft
(3)
where, Nonfa Nxx refers to batch of logits predictions with batch size of N and categories of C. ConfDigf x1 refers to the confidence difference for every logits between its maximum confidence and secondary maximum confidence. N and i refers to the total number logits and the 4th logit. & EN ConfDiff, the threshold to judge knowledge quality. quality defined Eq is a Boolean vector indicate knowledge quality. Then, H Hbrid predictions be computed in E by com-
3

---

Under review as a conference paper at ICLR 2023
bining Onehot predictions and Soft predictions. Obviously, our method combines soft knowledge and hard knowledge dynamically to form a hybrid knowledge representation for every imput image.
3.3 TASK REGULARIZED DISTILLATION
The learning loss of student in COD task can be defined as following equation Eq New task loss (LosSnews 5) refers to the loss supervised by the ground-truth of new classes. Old task loss (Lossota. Eq 4 refers the loss supervised by one-hot or soft targets from teacher. The Losset and Los81oc are the same as the official YOLOX, which are cross entropy loss and IoU loss respectively with coefficients of a and 3 5. The task balance factor y is set to be by default.
Lossotd L088,1 + 3 L0881oc
(4)
Lossnew = a Loss, + B Los81oc
(5)
LosStotal Lossnew + 7 Lossold
(6)
Continual learning is easily affected by data proportion of old and new tasks. If the data proportion of new task is too large, student will be dominated by new task loss and forget old knowledge.
Conversely, student will obtain much more stability to old knowledge and lack of plasticity to accept
new knowledge. Therefore, the key problem of distillation learning is to keep balance between old and new tasks. Motivated by this insight, we propose a task regularized distillation method (TRD) to solve the imbalance learning problem. TRD method consists of two parts: task equal loss and task difference loss, which are formulated as follows:
2 Lossnew
Lossatd
Los8old
Lossold + Lossnew
2. Lo888ld
Loss new
-Lossnew
(8)
Lossold + Lossnew
Loss Nnew y2. (Lo8sold Lossnew ) Noid
(9)
Loss Loss + Loss'ad Loss
(10)
Where, Lossold and Lossnew are defined in Eq and Eq Notd and Nnew refer to the number of old and new classes, Jespectively_.oss" ald and Loss are the newly defined losses for old and new tasks, in which two dimensionless coefficients play role of cross balancing factor. Obviously, Loss and Loss will be always equal to each other during the entire continual learning, which ensures a completely dynamic balance between old and new tasks regardless of their data imbalance. Lossaiss measures the loss difference and categories proportion between old and new tasks, which can further contribute to their balance learning. Loss is the final formulation of TRD method. Compared with Eq 6 TRD emphasizes task balance explicitly by introducing task- bask-balancing penalty item (Eq D and prevents student from over-fitting to any task. Fig clearly compares their loss curves.
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We build our continual detector on the top of YOLOX (Ge et al. 2021 It is a typical one-stage anchor-free detector among famous and widely used YOLO series, which can contribute to the typical verification of our method. YOLOX uses CSPNet (Wang et al. 2020) as its backbone, which needs to be trained from scratch along with detection heads for 300 epochs In general COD settings et al. 2019). 1x training schedule with 12 epochs and frozen backbone are widely used. So we replace CSPNet with ResNet backbone (He et al. 2016) for 12 epochs training, which is pre- trained on ImageNet and frozen during continual learning. Official YOLOX adopts very strong data augmentation, including Mosaic, MI. Photo Metrie Distortion, EMA, Random Affine, etc.. to boost its performance, but we drop these tricks to reduce randomness. We keep the other components and hyper parameters of YOLOX unchanged. The modified YOLOX, denoted iIYOLOX, is used for continual object detection. The ilYOLOX trained on old task is used as teacher to guide the next step learning of student on new task.
seen YOLOX in https//diibbccmooggemmmmmmmeeti
4

---

M.P A.Aenntinuall Under review as a conference paper at ICLR 2023
4.2 DATASETS AND EVALUATION METRICS
MS COC02017 (Chen et al. 2015) is used to build benchmarks with the train set for training and the minival set for testing. We split the dataset of total 80 categories into several subsets by its alphabetic order. Each subset is continual learning task. For example, the scenario of 40+20+20 means the data is split into three subsets and each of them contains 40, 20 and 20 categories respectively. The standard COCO protocols, including AP, 5 APTs, APs. APas and APL, are used to evaluate the detection performance.
In order to better evaluate the performance of COD task, we use following metrics. (1) Abs and RelGap: the absolute gap and relative gap between the task-average mAP of continual learning and the mAP of overall learning, respectively. The mAP of overall learning from baseline detector is usually considered as Upper Bound mAP (denoted as Upper or Upper Bound) of continual learning. RelGap equals to the ratio of AbsGap to Upper Bound mAP. (2) F and F defined in Eq (a) and (b)) respectively, are proposed to evaluate the balance ability between old and new tasks. (3) Omega (S2), defined in Eq (Hayes al. 2018), evaluate the cumulative capability of continual learning task by task. Similar to COCO protocols, 52 can be extended as Putt, Lg0- 92y5- ag. Ds and Qt, RelGap and Ii can eliminate the influence of upper bound from baseline detectors, providing fair comparison among different continual learning methods.
(a)F) |mAPald MAPnewl
2/mAPP old X mAP new (b)
(11)
mAPod + mAP new
mAPol + mAP new
T
Ω = T m.Poverall.t overail.t
(12)
where T' and L is the number of total tasks and the jth task in continual learning. m.A.oontinual, and m.A means the task-average mAP and upper-bound mAP on all testing data containing learned categories after task t. respectively. The larger the metric is, the better the ability of reducing cumulative knowledge forgetting would be.
4.3 EXPERIMENT SETUP
We design the following scenarios as benchmarks. (i)Two-Task scenarios: 40+40, 50+30, 60+20 and 70+10 settings. (i))Three-aas) scenario: 40+20+20 setting with 20 new classes added to the previous learned classes at each learning step. (ii))Four-Task scenario: 20+20+20+20 setting with 20 new classes added to the previous learned classes at each learning step. Five scenario: 40+10+10+10+ setting with 10 new classes added to the previous learned classes at each learn- ing step. We denote A(a-b) as the first-step overall learning for classes a-b, while +B(c-d) as the successive continual learning steps for classes c-d.
Given a scenario, we continually train iIYOLOX task by task (step by step) under the following settings. Optimizer is SGD with warm-up iterations of 1500, a continual learning rate of 0.2 decayed by 10% at the 8th and 1h epochs respectively, a momentum of 0.9 and a weight decay of 0.0005. All experiments are performed on 8 NVIDIA 3090 GPUs with total batch size of 16x8. All images are randomly resized to [6400 x 320, 640 x 640) by their short sides with content shape ratios unchanged. Normalization and random horizontal flip with a probability of 50% also used.
4.4
OVERALL PERFORMANCE
4.4.1
CONTINUAL LEARNING ABILITY
T Table and Table report the results of Two-Task, Three- and Five-Tast, Task-Task respectively. Compared with previous works, including LwF (Li & Hoiem 2018), RILOD (Li et al., 2019), SID
(Peng et al. 2021) I) and the latest best method ERD (Feng et al. 2022). our method achieves best per- formance under all scenarios and all evaluation metrics of continual learning. For the most difficult scenario of Five-Task (40+10+ 10++110+++++0+++ 10+10+10 in Table 2 our method shows overwhelming advantages over ERD under final mAP (27.23% 20.70 20.70%). AbsGAP (7.03% 19.50%), RelGAP (20.53% vs 48.51%, 27.98 absolute improvement) and QuI (0.8893 vs 0.7961). Especially, though ERD has higher initial mAP for A(1-40) and higher upper bound mAP, but the final mAP (marked by
5

---

Under review as a conference paper at ICLR 2023
underline in Table and Table 2 reversed during continual learning. The results in Table also show the same trend. We further plot Qau curves in Fig to highlight our advantages. These results fully demonstrate much better continual learning capacity of methods. In order to analysis the influence of upper bound mAP, we make extra experiments by using -OLOOX-ediuu. Experiments in Table (seen in AppendixA) demonstrate that method get much better performance with very large improvement under higher upper bound mAP. This further demonstrates its potential.
Table 1: Continual learning results under Two-Task scenarios. Our experiments are implemented with YOLOX-smar while all the others are implemented with GFLvI.
Scenarios
Method
AbsGap/
ReIGapt
Pait
Aso f
Ω75
Qs f
ΩM f
Qr t
LwF
23.00
57.21%
0.7139
0.7178
0.7133
0.6703
0.7086
0.7328
RILOD
10.30
25.62%
0.8719
0.8859
0.8670
0.8405
0.8741
0.8879
40+40
SID
6.20
15.42%
0.9229
0.9408
0.9163
0.8966
0.9354
0.9301
ERD
3.30
8.21%
0.9590
0,9674
0.9541
0.9591
0.9580
0.9550
Ours
1.86
5.42%
0.9729
0.9688
0.9728
0.9659
0.9766
0.9697
LwF
35,20
87.56%
0,5622
0,5815
0,5528
0.6078
0.5760
0.5546
RILOD
11.70
29.10%
0.8545
0.8705
0.8463
0.8319
0.8583
0.8640
50+30
SID
6.40
15.92%
0,9204
0.9374
0.9140
0.8793
0.9320
0.9320
ERD
3.60
8.96%
0,9552
0.9631
0,9461
0.9181
0,9580
0.9598
Ours
2.03
5.92%
0.9704
0.9642
0,9741
0.9886
0.9805
0.9674
LwF
34.40
85.57%
0.5721
0.5926
0.5608
0.5862
0.5964
0.5738
RILOD
14.80
36.82%
0.8159
0.8328
0,8073
0.7996
0.8288
0.8228
60+20
SID
7.50
18.66%
0.9067
0.9271
0.8968
0.8707
0.9263
0.9167
ERD
4,40
10.95%
0.9453
0.9537
0.9404
0.9440
0.9467
0.9454
Ours
2.49
7.25 %
0.9637
0.9578
0.9687
0.9801
0.9688
0.9640
LwF
33.10
82.34%
0.5883
0.6063
0.5803
0.6034
0.6077
0.5958
RILOD
15.70
39.05%
0.8047
0.8250
0.7947
0.8060
0.8107
0.8209
70+10
SID
7.40
18.41%
0.9080
0.9202
0.9014
0.8685
0.9184
0.9262
ERD
5.30
13.18%
0.9341
0.9451
0.9289
0.9030
0.9399
0.9358
Ours
3.14
9.16%
0.9542
0.9532
0.9564
0.9517
0.9596
0.9539
1.00
Scenarios
Method
Old
New
F2J
F8t
1.00
ERD
41.60
32.10 0.1289 0.9917
40 + 40
0.96
Ours
30.70
34.12 0.0528 0.9986
00.90
ERD
38.00
34.30 0.0512 0.9987
50 + 30
0.80 85
ERD
Tasks
Ours
31.14
34.06 0.0447 0.9990
Ours
Tasks
ERD
39.80
37.10 0.0351 0.9994
0. .00
60 20
ERD
Tasks
Ours
31.02
34.07 0.0468 0.9989
0.75
10
20
30
ERD
35.70
28.80 0.1070 0.9943
Nunber Classuns
70 + 10
Ours 30.40 36.20 0.0871 0.9962
Figure 2: The performance multi-task continual learning on MS COCO2017
Figure 3: The balance performance between old and new tasks on MS COC02017.
4.4.2 BALANCE LEARNING ABILITY
For continual learning, the balancing ability between stability of old knowledge and plasticity of new knowledge is very important. Table compares the results of ERD method and method under the metrics of F and F.. Obviously, our method strikes a better balance between the mAP of old and tasks, reflecting its good balancing ability between stability of old knowledge and plasticity

---

Under review as a conference paper at ICLR 2023
Table 2: Continual learning results under Five-Task scenario of 40+10+10+10, A(a-b) is the first-
step normal training for categories a-b and +B(c-d) is the successive continual training for categories c-d. We A(I-40) as the first step.
A(1-40)
+B BB40-50)
+B(50-60)
mAP
AbsGapl
ReIGap!
Pan t
mAP
AbsGap..
RelGapl
Rau 1
CF
5.80
32.20
84.74%
0.5763
5.70
34.10
85.68%
0.4319
RILOD
25.40
12.60
33.16%
0.8342
11.20
28.60
71.86%
0,6499
SID
34.60
3.40
8,95%
0.9553
24.10
15.70
39.45%
0,8387
ERD
36.40
1.60
4.21 %
0.9789
30.80
9.00
22.61%
0,9106
Ours
32.47
1.69
4.94%
0.9753
29.56
4.10
12.17% %
0.9430
+B(60-70)
+B(70-80)
mAP
AbsGapl
RelGapl
Pai 1
mAP
AbsGap
RelGapl
Rau f
CF
6.30
29. 29.40
82.35%
0.3681
3.30
36,90
91.79%
0.3109
RILOD
10.50
25.20
70.59%
0.5610
8.40
31.80
79.10%
0,4906
SID
14.60
21.10
59.10%
0.7313
12.60
27.60
68.66%
0.6477
ERD
26.20
9.50
26.61%
0.8664
20.70
19.50
48.51%
0.7961
Ours
27.59
5.94
17.72%
0.9129
27.23
7.03
20.53%
0,8893
of new knowledge. The ablation experiments and supplementary material show more analyses about task- balanbnann learning.
4.4.3
KNOWLEDGE FORGETTING
Four-Tasls (20+204-2+ Continual Learring MS COCO2O17
Tesk3
Te
Togotgg
Upper Bound
36.0%
32%
33.02%
50.7%
26.0%
93%
21.04
10/09
5.0%
Continual Leaming Spochs
Figure 4: The continual learning curves under Four-Task scenario (20+00+00+2), which shows that knowledge forgetting of old tasks always controlled to a limited range during learning process.
Fig reports the results under Four-Task scenario, which clearly shows detailed continual learning process of iIYOLOX in successive data streams. The AbsGap (plotted as white fluctuation columns each learning epoch) shows that our method reduces the performance gap between continual learning and overall learning gradually and efffectively. The mAP curves of each task (denoted as Taski, = 1,2,3, 4) reflect their knowledge forgetting process. On the whole, the knowledge forgetting of old tasks is always controlled to limited range during successive continual learning. Meanwhile, the TaskAvg mAP after each learning step. marked on the corresponding light-blue curve, also shows very good stability (28.71%) %, 28.20 28.20%, 28. 18%, 28.1 11).. The results in Fig further demonstrate good effectiveness of our method.
5 ABLATION STUDY
The Independence and Compatibility of HKR and TRD. Table Asoow the results of ablation ex- periments. The two baseline methods, denoted Onehot and Soft, one-hot predictions and soft

---

Under review as a conference paper at ICLR 2023
Table 3: Continual learning results under Three-Task scenario of 40+20+20.
A(1-40)
+B(40-60)
+B(60-80)
MAP
AbsGap
RelGapt
Pai t
mAP
AbsGap
ReIGapl
Pait 1
CF
10.70
29.10
73.38%
0.6344
9.40
30.80
76.62%
0,5009
RILOD
27.80
12.00
30.85%
0.8492
15.80
24.40
60.70%
0.6972
SID
34.00
5.80
15.42%
0.9271
23.80
16.40
40.80%
0.8154
ERD
36,70
3.10
7.79%
0.9611
32.40
7.80
19.40%
0.9094
Ours
32.26
1.39
4.13%
0.9793
31.24
3.02
8.82 %
0,9568
Table 4: Continual learning results under Two-Task scenarios for ablation study. We equip YOLOX- small with two knowledge selection methods (Onchot and Soft) as our baselines.
Scenarios
60 classes 20 classes
70 classes + 10 classes
Methods
AbsGap 4 ReIGap Pau t Aoo f Ura
AbsGap
RelGap 4 Qau t Ωso
Onchot
3,86
11.27%
0.9437 0.9477 0.9387
4.91
14.33%
0.9284 0.9367 0.9264
Soft
3.29
9.60%
0,9520
0.9505 0.9537
4,49
13.10%
0.9345
0.9385 0.9332
Soft+HKR
3.10
9.04%
0,9548
0.9514 0.9578
4.19
12.22%
0.9389
0,9422 0.9414
Soft+TRD
2.95
8.62%
0,9569
0.9541 0.9578
4.06
11.83%
0,9408
0.9431 0,9441
Soft+Both
2.49
7.26%
0.9637 0.9578 0.9687
3.14
9.16% %
0.9542 0.9532 0.9564
predictions (logits) as teacher knowledge respectively. Then we add Hybrid Knowledge Selection module and Task Regularized Distillation module the Soft baseline respectively (seen F. whose results are denoted as Soft+HKR and Soft+TRD respectively. Finally, we add both HKR and TRD to the Soft baseline simultaneously, whose results are denoted as Soft+Both. The results under two scenarios all show that soft knowledge is better than hard knowledge, but both are inferior to hybrid knowledge. Compared with the Soft baseline, TRD shows higher performance improvement than HKR. This demonstrates that both HKR and TRD have their own effects two independent components. Mcanwhile, the results of 'SoftBBoth' (means Soft+HKR+TRD get further significant improvement, demonstrating that HKR and TRD have good additivity and compatibility.
The HKR under Three-Task Scenario. In order to further analyze hybrid knowledge representa- tion, we make additional ablation studies under Three-Task scenario. The results shown in Table 5 demonstrate the effectiveness of HKR clearly. It further reflects that hybrid knowledge can fully utilize the advantages of both soft knowledge and hard knowledge in adaptive manner.
The Task Balance During Continual Learning. In order to further analyze the influence of task make experiments by changing the task balancing factor (ry in Eq from 0.2 to 3.0. The results of Table show that our TRD method gets a medium mAp for new task, but gets the highest performance under all other metrics. Brown-marked digits show the sub-optimal results. When changes from 0.2 to 3.0, the mAp values in Table shows noticeable changes. similar experiment made on another small dataset (seen appendix) and shown in Fig It shows that even a very small change of from I to 0.9 can lead to dramatically descending of the old task mAP curve (blue dotted line) and bring catastrophic knowledge forgetting. Obviously, task balance factor (7) has a significant influence on continual learning by controlling knowledge
balance continual learning.
Table 5: Continual learning results under Three-Task scenario (40+20+20) for ablation study.
Scenarios
+B(40-60)
+B(60-80)
methods
mAP
AbsGap ReiGapl Qas 1 Aso 1 175 t
mAP
AbsGap RelGapl Quu
Oso
A75 1
Onehot
31.62
2.04
6.05%
0.9698 0.9721 0.9694
30.44
3.83
11.16%
0.9426 0.9453 0.9424
Soft
31.84
1.81
5.38%
0.9731 0.9712 0.9750
30.87
3.39
9.91%
0.9491
0,9453 0.9488
H Hybrid
32.03
1.62
4.81%
0,9759
0.9703 0.9819
31.19
3.08
8.97%
0.9540
0,9478 0.9580
8

---

Under review as a conference paper at ICLR 2023
transfer from teacher to student. Compared with different y values, the mAP of old 70 classes reflects that TRD relieves knowledge forgetting to the greatest extent. Meanwhile, TRD shows much more balance learning ability between old and new tasks with the minimum Fj and the maxmum F Overall speaking, our TRD realizes dynamic balance knowledge transfer by introducing task-based regularization. More over, TRD is statistically adaptive method without hyper parameters.
Table 6: Continual learning results under Two-Tas scenario for ablation study. Hyper parameter the task balance factor (seen Eq TRD is task regularized distillation method (seen Eq [0]
m.P f
Methods
Ral
Qo0 f
F5 J
Old 70 Classes
New 10 Classes
Final
0.2
27.00
37.41
28.30
0.9129
0.9257
0.9101
0.1617
0.9868
7 0.5
28.41
37.70
29.57
0.9315
0.9367
0.9305
0.1406
0.9901
y 0.6
28.37
38.02
29.58
0.9316
0.9358
0.9292
0.1453
0.9894
y 0.7 0.7
28.58
37.69
29.72
0.9336
0.9385
0.9319
0.1375
0,9905
0.8.
28.92
37.48
29.99
0,9376
0,9394
0.9360
0.1289
0.9917
=
28.59
37.53
29.71
0.9335
0,9358
0.9332
0.1352
0.9908
7= 1.0
28.72
37.14
29.78
0.9345
0.9385
0.9332
0.1278
0.9918
7= 1.5
28.61
36,96
29.66
0.9328
0.9358
0.9292
0.1273
0.9919
2.0
28.37
36.04
29.33
0.9280
0,9312
0.9292
0.1191
0.9929
3.0
27.11
35.14
28.12
0.9103
0.9174
0.9074
0.1289
0.9917
TRD
29.24
36,97
30.21
0.9408
0.9431
0.9441
0.1167
0.9932
3.5
loss_ new" _TRD - loss_old" _RD
loss, dift" TRD
Task, classes)
loss new y=0.8 loss y=0.8
(New Task, classes)
--- 4-0.9 (Old Task, classes)
v-0.9 (New Task desses) ---
0.5
0.10
0.0
0.00
111111111771111 12 Continual Training Epochs
Continual Training Epochs
Figure 5: The training loss of old and new tasks Figure 6: The catastrophic forgetting caused by imbalance learning of old and new tasks.
at = 0.8 and TRD for Table respectivell.
CONCLUSION
In order improve the performance of continual object detection, we propose knowledge distilla- tion method that combines knowledge selection strategy and knowledge transfer strategy effectively. For the first strategy, hard knowledge and soft knowledge are dynamically and adaptively combined to construct kind of hybrid knowledge representation to use teacher knowledge critically and ef- fectively. For the second strategy, loss difference and category proportion are combined construct task regularized distillation loss to enhance task balance learning. Extensive experiments under different scenarios validate the effectiveness of our method. Most existing methods in COD task adopt mixed distillation scheme including feature, classification, location and relation to relieve catastrophic forgetting. However, we demonstrate that as long as knowledge selection and transfer strategies appropriate, even single classification distillation can also achieve state-of-the-ar per- formance. In addition, our method has good prospect to work together with feature and location distillation for further improvements. More analyses and discussions provided in Appendix

---

Under review as a conference paper at ICLR 2023
REFERENCES
Fabio Cermelli, Antonino Geraci, Dario Fontanel, and Barbara Caputo. Modeling missing annota- tions for incremental learning in object detection. ArXiv, hbss2204.087666 2022.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and
C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/ Jbbs/5544.00355, 2015.
Tao Feng. Mang Wang, and Hangjie Yuan. Overcoming catastrophic forgetting in incremental object detection via elastic response distillation. ArXiv, abs/2204.02136, 2022.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: exceeding YOLO series in 2021. CoRR, abs/2107.08433 08430, 2021. BKKLtttps://axxvv.rrg wwt./oogggaaaaaoooo.ooeoe 08430
Ross B. Girshick. Fast r-cnn. 2015 IEEE International Conference Computer Vision (ICCV), pp- 1440-1448, 2015.
lan J. Goodfellow, Mehdi Mirza, Xia Da. Aaron C. Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient-based neural networks. CoRR, abs/1312.6211, 2014.
Tyler L. Hayes, Ronald Kemker, Nathan D. Cahill, and Christopher Kanan. New metrics and exper- imental paradigms for continual learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Pp. 2112-21123, 2018.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.0253/1 2015.
K. J. Joseph, Salman Hameed Khan, Fahad Shahbaz. Khan, and Vineeth N. Balasubramanian. To- wards open world object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5826-5836, 2021a.
K. J. Joseph, Jathushan Rajasegaran, Salman Hameed Khan, Fahad Shahbaz Khan, Vineeth N. Bal- asubramanian, and Ling Shao. Incremental object detection via meta-learning. IEEE transactions paltern analysis and machine intelligence, PP, 2021b.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trrabska-Barwinskee Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2017.
Dawei Li, Serafettin Tasci, Shalini Ghosh, Jingwen Zhu, Junting Zhang, and Larry Heck. Rilod: near real-time incremental learning for object detection at the edge. Proceedings of the 4th ACM/IEEE Symposium Edge Computing, 2019.
Pengyang Li. Yanan Li, and Donghui Wang. Class-incremental few-shot object detection. ArXiv, abs/2105.076377, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935-222 2947, 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The
sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.
Steven J. Nowlan and Geoffrey E. Hinton. Simplifying neural networks by soft weight-sharing Neural Computation, 4:473 493, 1992.
Can Peng, Kun Zhao, Sam Maksoud, Meng Li, and Brian C. Lovell. Sid: Incremental learning for anchor-free object detection via selective and inter-related distillation. ArXiv, abs/2012.154399 2021.
10

---

Under review as a conference paper at ICLR 2023
Haoxuan Qu, Hossein Rahmani, Li Xu, Bryan M. Williams, and Jun Liu. Recent advances of continual learning in computer vision: An overview. ArXiv, abs/2109.11369, 2021.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15: 1929 1958, 2014.
Chien- Yao Wang, Hong-Yurr Yuan Mark Liao, I-Hau Yeh, Yueh-Hua Wu, Ping- Yang Chen, and Jun- Wei Hsich. Cspnet: A new backbone that can enhance learning capability of cnn. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1571-1580, 2020.
Jianren Wang, Xin Wang, Yue Chang-Guan and Abhinav Kumar Gupta. Wanderlust: Online con- tinual object detection in the real world. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10809-10818, 2021.
Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. ArXiv, hiss22222.6592.. 2022.
Binbin Yang. Xinchi Deng. Han Shi, Changlin Li, Gengwei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiaodan Liang. Continual object detection via prototypical task correlation guided gating mechanism. ArXiv, abs/2205.0300 03005,, 2022a.
Dongbao Yang. Y. Zhou, and Weiping Wang. Multi-view correlation distillation for incremental object detection. ArXiv, abs/2107.01787, 2022b.
Zhendong Yang, Zhe Li, Yuan Gong, Tianke Zhang, Shanshan Lao, Chun Yuan, and Yu Li. Re- thinking knowledge distillation via cross-entropy. ArXiv, abs/2208. 10139, 2022c.
Hangjee Yuan, Dong Ni, and Mang Wang. Spatio-temporal dynamic inference network for group activity recognition. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7456-7465, 2021.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context-dependent processing in neural networks. Nat. Mach. Intell., 1:364-37 372, 2019.
Kai Zheng and Cen Chen. Contrast r-cnn for continual learning in object detection. ArXiv, abs/2108.04224, 2021.
Zhaohui Zheng, Rongguang Ye, Ping Wang, Dongwei Ren, Wangmeng Zuo, Qibin Hou, and Ming- Ming Cheng. Localization distillation for dense object detection. 2021.
A APPENDIX
A.I COMPARED WITH KL DIVERGENCE LOSS
Kullback-Leibler Divergence loss (denoted as KLD loss) is used for knowledge distillation of image classification (Hinton et al.) 2015). YOLOX (Ge et al. 2021) uses cross entropy loss (denoted as CE loss) for its classification head. Table shows the comparison results of this two losses on continual object detection. The experiments adopt the same loss weight setting with a 1 and 8 5 (seen in an/ E for the two losses. T is temperature factor, a hyper parameter of KLD loss. When the temperature T changes from 5, iIYOLOX gets its best performance (marked by brown color) under the medium temperature of 3. However, CE loss has much better performance than KLD loss (marked by underline). Our TRD loss exceeds both KLD and CE loss. The use KLD loss usually requires careful adjustment of temperature factor T and loss weight a. However, the change of a in Lossold will destroy the loss consistency about classification and location between old task (Lossold. 4) and new task (Lossnew- Eq which will influence task balance during continual learning. Based on this consideration and the experiment results in Table we use cross entropy loss our fundamental knowledge transfer strategy.

---

Under review as a conference paper at ICLR 2023
Table 7: The comparison with KI. Divergence Loss
Scenarios
70 classes
classes
mAP1 1
Methods
AbsGap +
RelGap
Qau 1
Old 70 Classes
New 10 Classes
Final
KI.D T=1
25.55
35.19
26.75
7.51
21.92%
0.8904
KLD T-2
26.36
35.87
27.55
6.71
19.59%
0,9021
KLD T=3
26.86
36.29
28.04
6.22
18. 16%
0,9092
KLD T=4
26.84
35.96
27.98
6.29
18.35%
0,9083
KLD T=5
26.49
35.50
27.62
6.65
19.41 %
0.9030
CE Loss
28.72
37.14
29.78
4.49
13.10%
0.9345
TRD Loss
29.24
36.97
30.02
4.24
12.38%
0.9381
A.2 EXPERIMENT DETAILS FIG
We deseribe the experiments in Fig in detail as follows. It is made small dataset that consists of 3800 images, classes (commonly seen toys including car, truck, train, person and so on) and have 400 500 instances for every class with relative balanced category distribution. We build a Two-Task scenario of 5 classes + classes as our continual object detection benchmark. The ex- periment results are illustrated Table When the task balance factor (7 defined in Eq changes from 1.3 to 0.7, the mAP of old task (old 5 classes) drops from 54.24% to 5. 5.10% quickly. It fully demonstrates that the loss imbalance between old and new tasks can bring significant catastrophic forgetting during continual leaning.
Table 8: Task balance experiment small dataset
Scenarios
5 classes classes
mAP
Methods
AbsGapl
ReIGapt
Rall t
Old Classes
New Classes
Final
0.7
5.16
58.73
28.97
32.31
52.73%
0.7364
0.9
5.10
59.20
29.14
32.14
52.44%
0.7378
54.24
60.18
56.88
4.40
7.18%
0.9641
= 1.1
54.34
59.98
56.84
4.44
7.25%
0.9638
== 1.3
53.58
59.05
56.01
5.27
8.60%
0,9570
TRD
54.46
64.18
58.78
2.50
4.08%
0.9796
TRD+HKR
54.66
64.63
59.09
2.19
3.58%
0.9821
A.3 DISCUSSION
Hybrid Knowledge Representation. Teacher outputs logits and one-hot representation as its pre- dictions. Soft logits contains more information about between-clar class confidences and is regarded as a kind of soft knowledge (Hinton et al. 2015 Yang et al. 2022c). Knowledge distillation meth- ods are developed prosperously under this background in image classification and object detection tasks. But there are significant difference between the two tasks. For image classification, since an imput image contains only one instance, the knowledge of the image just the knowledge of its instance. However, for object detection, one input image contains several instances, the knowledge on an image collection of the knowledge of all instances on it. Instances on an image can be regarded as entity nodes, so the overall image knowledge should be constructed based on the entity nodes and their relations. other words, image-level knowledge is the combination level knowledge. Logits and One-hot predictions provide two kinds of instance-level knowledge representation as soft and hard knowledge, respectively. Soft knowledge contains confidence re- lations among categories, but brings knowledge fuzziness inevitably. While, hard knowledge has completely opposite effects. By combining their advantages, we construct an image-level hybrid knowledge representation (Eq showing better performance COD task in Table and Table
instance-
12

---

Under review as a conference paper at ICLR 2023
Task Regularized Distillation. Regularization method is very import for statistical machine learn- ing, which can prevent model from over fitting to some part of data. Classical regularization methods introduce weight constraint in terms of p-Norm as model penalties. Other regularization methods include stopping the training as soon as performance on a validation set starts to get worse and soft weight sharing (Nowlan & Hinton 1992). Dropout works by dropping some connections randomly to prevent over-fitting of deep neural networks (Srivastava et al. 2014). In continual learning, model needs to be trained task by task in the continuous data flow, therefore, bring task-based imbalance learning and lead to catastrophic forgetting. A few previous works propose regularization- based methods on continual image classification (Kirkpatrick et al., 2017: Li & Hoiem| 2018). We give solution by proposing task-based regularized distillation loss [0] for COD tasks. It explicitly uses the loss and categories difference between old and new tasks as a model penalty to constraint op- timization process. Experiments in Table Table Table Fig.5] and Fig 6 all show its strong effectiveness to prevent continual model from over-fitting to some task.
A.4 MORE DETAILS OF IMPLEMENTATION AND EXPERIMENTS
All the previous works, including Catastrophic Forgetting, RILOD (Li al. 2019). SID (Peng al., 2021] and ERD (Feng et al. 2022) models, are implemented based on GFLvI (Zheng et al. 2021] with original image size of 1333 800. Since they the same base detector, the upper bound mAP of these previous models are all 40.20 for A(1-80), the overall training of total 80 categories. On the other hand, our experiments are implemented based on YOLOX-small detector (Ge al. 202T). YOLOX is typical one-stage mchor-free detector among famous and widely used YOLO series, which can contribute to the typical verification of our method. The official YOLOX implementation in MMDetection Fl adopts 300 epochs training schedule with strong data augmentation, including Mosaic, MixUP, Photo Metric Distortion, EMA, Random Affine, Random Horizontal Flip, etc, to boost its performance. We drop these tricks to reduce randomness for continual learning and for better reproducibility. The modified YOLOX, denoted as ilYOLOX, is used for continual object detection. Since our experiments are conducted for demonstration of model effectiveness, ran- domly resize all the images to [640 x 320, 640 x 640) by their short sides with content shape ratios unchanged. The upper bound mAP of our model is therefore 37.13 for A(1-80).
Although different models have different baseline detectors, the evaluation metrics, like RelGap and Q, are used fairly compare their continual learning capability. We also analyze the influence of upper bound mAP by comparing different base detectors: YOLOX-small versus YOLOX-mediing Experiments in Table shows that higher upper bound mAP leads better continual learning per- formance, which further demonstrates the potential of our method. Here, we bring the continual object detection research on widely used lightweight model YOLOX to reduce computing loads and resource consumption, accelerating the researches of this topic and protecting our environment.
Table 9: Results on COCO benchmark with different upper bound mAP.
Scenarios
Method
AbsGap
ReIGapl
Pau f Rso 1 Prs t Is t
Pxx t Oz 1 mAP Upper
iIYOLXX-S
1.86
5,42%
0.9729 0.9688 0.9728 0.9659 0.9766 0.9697 32.41 34.26
40 + 40
iIYOLOX-M
1.11
2.99%
0.9849
0.9850 0.9887
0.9922 0.9915
0.9698 36.02 37.13
iYOLOX-S
3.14
9.16%
0,9542
0.9532 0.9564
0.9517 0,9596
0.9539
31.13 34.26
70 + 10
iIYOLOX-M
2.62
7.04%
0.9648
0.9575 0.9699 0.9560
0.9732
0.9435
34.52 37.13
Algorithm 1 Algorithm of Hybrid Knowledge Representation
Input: Unlabeled image 1, teacher detector
Output:
Hybrid predictions H ybrid of response
1: Inference 1 with 0' yields the logits predictions Soft and one-hot predictions Onehot
2: Compute ConfDiff = Confmax Conf seconderg.max
3: Compute quality- = ConfDiff > N ΣN ConfDiff;
4: Compute Hybrid = quality Onehot + (1 - quality) Soft
seen YOLOX in http://gsttal.cmmmmoommmmmmmmmit
13


Reviews:
[{"review_id": "dGzQcKLrbc", "reviewer": "Reviewer_jG8i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to alleviate the problem of catastrophic forgetting in continual object detection from the perspective of knowledge distillation (KD). Different from other methods that adopt complex KD supervision, such as feature, location and relation, the proposed method only relies on classification supervision and can also achieve promising results with the help of a combination of soft-hard label assignment and class-adaptive loss coefficient.", "summary_of_the_review": "This paper should be rejected for the aforementioned reasons.", "strength_and_weaknesses": "1. The fusion of soft and hard labels has been explored in [1], what is the difference between the proposed “HKS” and the fusion method of [1]? Can the authors additionally compare the fusion method of [1] in the experiment part?\n\n2. Class-balanced loss is a common practice, such as the balanced CE loss in [2]. The idea behind the loss design of \"TRD\" seems not strong enough to be accepted.\n\n3. Although the authors have repeatedly emphasized that different detection models do not affect some metrics like “RelGap” and “Omega”. I still think that the authors should consider experiments based on the same model as other methods.\n\n4. I would like to know if the training settings of this paper is consistent with the settings of other papers used as comparisons. If consistent, some citations can be added to the part of experimental setup.\n\n5. In Figure 4, the continual learning curves of other methods are expected to be plotted for better comparison.\n\n6. The importance of “$Loss_{diff}$” term in “TRD” is not considered in ablations.\n\n7. Can the observed phenomenon be reproduced when experiments w.r.t. Fig. 6 being carried in a common dataset, such as MS-COCO, rather than the designed small dataset?\n\nThings to improve the paper that do not impact the score:\n\n1. The loss formulation for “TRD” seems to be redundant, for example, only one of “$Loss_{new}^{\\*}$” and “$Loss_{old}^{\\*}$” is required in Eq.10.\n\n2. Illustrations in the paper had better be vector graphs.\n\n3. In the caption of Table 2, an additional “+10” is missing after “Five-Task scenario of 40+10+10+10”.\n\n4. In Page 8, “Knowledge Selection” in “Then we add Hybrid Knowledge Selection module……” should be “Knowledge Representation” for consistency.\n\n5. Some explanations in A.3, such as the instance-level discussions, seem to be unrelated to the main ideas of this paper and can be simply removed in my view.\n\n[1]: Kim, Kyungyul, et al. \"Self-knowledge distillation with progressive refinement of targets.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[2]: Homayounfar, Namdar, et al. \"VideoClick: Video Object Segmentation with a Single Click.\" arXiv preprint arXiv:2101.06545 (2021).\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Z1ln1VPaRi", "reviewer": "Reviewer_i3PC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a knowledge selection module between soft and hard labels. It also proposed a task-based regularization distillation loss to balance multiple task losses.\n\nAs a new detector, YOLOX, is employed in this paper, the performance of this work is higher than previous ones. Extensive results on multiple datasets have been shown.", "summary_of_the_review": "I tend to reject the paper. The main contribution is to employ a new powerful detector YOLOX for continuous learning. This can not bring too many insights. I might improve my rating if the author could analysis some interesting insights related to their proposed methods.\n\n", "strength_and_weaknesses": "++ The paper pays attention to the regularization of various tasks.\n++The experiment results on multiple settings demonstrate the effectiveness of the proposed detectors.\n\n-- The comparison happens between different detectors and training schemes, which is debatable. Although we should not be limited to old benchmarks for algorithm design, in order to make a fair comparison, experiments on the same detectors should first be shown. For example, the author could re-implement other continuous learning algorithms with the same detector, YOLOX. Or, the proposed method can be first applied to original benchmarks to make a fair comparison, and then a new powerful detector could be proposed for future comparison.\n\n--Ablation studies can not fully demonstrate the effectiveness of the proposed TRD method. If there are more task stages in the training set, will the proposed TRD be more powerful? Or it performs the same. It will be better if more weight-balancing algorithms could be compared with TRD.\n\n-- The representation is not that clear, including the terms and the figures. Details could be referred to in the next blank.\n\n-- For the 60+20 setting in Table1 and Table 4. The one-hot performance of the YOLOX already outperforms the previous work ERD. The improvement of the proposed two methods is not that significant.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LabjY1UiAw6", "reviewer": "Reviewer_qEVh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper address the continual object detection problem, which is a very recent and challenging vision task. It proposes an image-level hybrid knowledge distillation method. The proposed method exhibits a noticeable gain in COCO 2017 dataset.", "summary_of_the_review": "Given that the paper has only merit in empirical results but poorly motivates the proposed method, the paper should be heavily revised and encouraged to resubmit.", "strength_and_weaknesses": "**Strengths**\n- S1: In under explored COD task, on COCO dataset, the gain by the proposed method is noticeable (Table 1)\n\n**Weaknesses**\n- W1: The hybrid equation (Eq. (1)-(3)) is not well motivated. Why $ConfDiff$ should be defined in this manner? Why should the hybrid loss mix up the onehot and soft in this manner (necessariliy)?\n- W2: The eq (7) and (8) are the same. Explanation is required.\n- W3: No justification of the proposed evaluation metric of $F_b^1$ and $F_b^2$ in Eq. (11).\n- W4: Choice of hyperparameter also seems arbitrary or based on empirical results which may be specific to COCO dataset only. This limits the applicability of the proposed method to other datasets or tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IkAbiyrkOKQ", "reviewer": "Reviewer_rDxx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a knowledge distillation approach for continual object detection. The paper introduces two contributions: an image-level hybrid knowledge representation method, named HKR, that distill knowledge from the teacher by combining soft and hard pseudo-labels. Secondly, it proposes task regularized distilliation (TRD), that is a strategy to balance the knowledge distillation and classification loss, to prevent the student model to overfit the new tasks. \nOverall, the method performance are assessed on MS-COCO using the YOLOX architecture.", "summary_of_the_review": "The paper proposes a method for Continual Object Detection introducing two slight modifications to the standard knowledge distillation framework. Moreover, the paper lacks important ablation studies and comparisons with state-of-the-art results. \n", "strength_and_weaknesses": "Weaknesses:\n1. The TRD is not clear from the paper's description. First, Eq.7 and Eq.8 are not only always equal, but without other explanations, they are exactly the same equation and have the exact same effect on the network gradients, actually being a single loss with value Loss*_old + Loss*_new = 4*Loss_new*Loss_old / (Loss_old + Loss_new). Seen in this perspective, TRD does not bring any contribution but just computes the harmonic mean between the two losses. Furthermore, very few intuitions are provided for Eq. 9. Why should minimizing the quadratic difference between the old and new loss be beneficial for the training? Finally, the paper should experimentally demonstrate the benefits of adding Loss_diff through an ablation study.\n2. Mixing soft and hard pseudo-labeling is an interesting concept, however, to better exploit the model confidence, previous works propose to introduce a temperature value in the softmax operation to better transfer the model confidence. This is similar in spirit to the HKR and should be carefully demonstrated that the paper technique is superior. Furthermore, the paper introduces a threshold to decide whether use one-hot or soft-pseudo labeling. How is the threshold fixed? Is there any ablation study demonstrating the robustness of this choice? \n3. Regarding the experimental section, the paper reports experiments on MS-COCO. However, the standard and mainly used benchmark for continual object detection is the Pascal-VOC dataset but it is not reported. The paper should report results on this benchmark and should compare with all the state-of-the-art methods and not only a subset of them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject