PseDet: Revisiting the Power of Pseudo Label in Incremental Object Detection
Authors: Qiuchen Wang
Zehui Chen
Chenhongyi Yang
Jiaming Liu
Zhenyu Li
Feng Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Incremental Object Detection
Catastrophic Forgetting
Pseudo Labeling
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Incremental Objection Detection (IOD) facilitates the expansion of the usage scope of object detectors without forgetting previously acquired knowledge. Current approaches mostly adopt response-level knowledge distillation to overcome forgetting issues, by conducting implicit memory replay from the teacher model on new training data. However, this indirect learning paradigm does not fully leverage the knowledge generated by the teacher model. In this paper, we dive deeper into the mechanism of pseudo-labeling in incremental object detection by investigating three critical problems: (a) the upper bound quality of the pseudo labels is greatly limited by the previous model, (b) fixed score thresholds for label filtering, without considering the distribution across categories, and (c) the confidence score generated by the model does not well reflect the quality of the localization. Based on these observations, we propose a simple yet effective pseudo-labeling continual object detection framework, namely PseDet. Specifically, we introduce the spatio-temporal enhancement module to alleviate the negative effects when learning noisy data from the previous model. Considering the score distribution divergence across different classes, we propose the Categorical Adaptive Label Selector with a simple mathematical prior and fast K-Means pre-computation to dynamically determine the class-wise filtering threshold. In order to align the label score with the localization quality of the pseudo labels, we project the score through non-linear mapping to calibrate the distribution and integrate it into the new-step supervision. Extensive experiments on the competitive COCO benchmarks demonstrate the effectiveness and generalization of PseDet. Notably, it achieves 43.5+/41.2+ mAP under the 1/4-step incremental settings, achieving new state-of-the-art performance.

Published as a conference paper at ICLR 2025
PSEDET: REVISITING THE POWER OF PSEUDO LABEL IN INCREMENTAL OBJECT DETECTION
Qiuchen Wang', Zehui Chen', Chenhongyi Jiaming Liu2, Zhenyu Li", Feng Zhao Moo Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC Universitt Edinburgh Peking University 4 *King Abdullah University of Science and Technology qiuchenwang@mal www.usst.e.. edu. cr. fzhao 6:ustc. edu. on
ABSTRACT
Incremental Objection Detection (IOD) facilitates the expansion of the usage scope of object detectors without forgetting previously acquired knowledge. Current approaches mostly adopt response-level knowledge distillation to overcome for- getting issues, by conducting implicit memory replay from the teacher model on new training data. However, this indirect learning paradigm does not fully leverage the knowledge generated by the teacher model. In this paper, dive deeper into the mechanism of pseudo-labeling in incremental object detection by investigating three critical problems: (a) the upper bound quality of the pseudo labels is greatly limited by the previous model, (b) fixed score thresholds for label filtering, without considering the distribution across categories, and (c) the confidence score gener- ated by the model does not well reflect the quality of the localization. Based on these observations, we propose a simple yet effective pseudo-labeling continual object detection framework, namely PseDet. Specifically, we introduce the spatio- temporal enhancement module to alleviate the negative effects when learning noisy data from the previous model. Considering the score distribution divergence across different classes, we propose the Categorical Adaptive Label Selector with a simple mathematical prior and fast K-Means pre-computation to dynamically determine the class-wise filtering threshold. In order to align the label score with the localiza- tion quality of the pscudo labels, we project the score through non-linear mapping to calibrate the distribution and integrate it into the new-step supervision. Extensive experiments on the competitive COCO benchmarks demonstrate the effectiveness and generalization of PseDet. Notably, it achieves 43.5+/41.22 .2+ mAP under the 1/4-step incremental settings, achieving new state-of-the-ar performance. Code is available at
INTRODUCTION
Object detection is a fundamental and important task with various applications (Li ct al. 2020 Tran et al. 2019 Zhang al.) 2020bb Chen et al. 2021). However, static detectors cannot well fit the needs of real-world scenarios since the changing enviromments and extra demands require the model to seamlessly adapt to new class detection with updated training data (Joseph et al. 2021 Caccia jet al. 2021). Such a problem, which refers to incremental object detection (IOD), denotes the ability of object detectors to acquire new knowledge without forgetting (Liu al. 2023b a). Concretely, training samples for different categories are observed at varied training steps, and the detectors are restricted from accessing the ground truth labels in the past steps.
Previous approaches mainly adopt knowledge distillation (KD) CCen et al. 2017 Hinton et al. 2015) to overcome the catastrophic forgetting problem. By distilling from the intermediate products generated by the teacher model, it provides direct supervision to enforce the new model to be close to its former one. This imposes an implicit regularization to the update of the training model. Such a distillation framework can also be viewed as special case of pseudo-labeling, where the distillation targets are actually treated as pseudo-labels during new-step training.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Despite being effective, it does not fully anticipate the potential benefits of pseudo-labels. Specifically, we observe three critical issues in current paradigms: (1) the upper bound quality of distillation supervision is greatly limited by the previous (teacher) model. Due to the learning ability of different detectors, the localization quality generated by the previous model is far from the ground truth. Directly enforeing the target model mimic the knowledge from these noisy (mixed-quality) data can confuse the learning process of the model and even deteriorate the performance. (2) object detectors tend to exhibit score biases across different categories (Figure due to the data distribution and learning difficulty of the training labels. However, current works mostly set hard thresholds for all classes for label filtering, e.g., 0.3, which overlooks the score divergence at the category level, and (3) in order to fully leverage the information generated from the pseudo labels, we carefully examine the relationship between confidence score and localization quality. Though the confidence score is viewed as the metric to reflect the quality of detected bounding boxes, we empirically find these two variables do not demonstrate linear correlations. Therefore, simply taking the confidence score as the label quality for new-step training may introduce extra noise for the model.
Based on the above observations, we propose a simple yet effective pseudo-labeling continual object detection framework, namely PseDet, which fully leverages the intermediate labels for effective model regularization. To eliminate the noisy information from the immature previous model, we introduce the Temporll Enhancement Module, which enhances the label quality along the spatial and temporal dimensions. By augmenting inputs at different input scales and different steps, we seamlessly improve the quality of the generated labels with the multi- multi-group ensemble. In terms of the score distribution divergence across different classes, we propose the Categorical Adaptive Label Selector to dynamically determine the threshold on different categories with a simple mathematical prior and fast K-Means crepcopuutaiin. It removes the massive human efforts to manually select the filtering threshold for each class and greatly ameliorates the final label quality. Besides, to align the confidence score and localization quality of the generated labels, we project the score through non-linear mapping function to calibrate the distribution and seamlessly integrate it into the new-step training. Thanks to these enhancements the pseudo-labeling, PseDet achieves 43.5 3..5+/41.2+ mAP on the challenging COCO dataset under 1/4 steps training settings, surpassing previous satte-of-thhaatt art by large margin.
Our contributions are summarized in three-fold:
We revisit the strategy of pseudo-labeling on incremental object detection and identify three critical problems that hinder it from achieving competitive performance.
Based on the above findings, we propose PseDet, a simple yet effective pseudo-labeling framework for incremental object detection, which consists of three key components: spa- tiotemporal enhancement module, categorical adaptive label selector, and confidence score calibration supervision.
Extensive experiments conducted on the MS COCO dataset with various incremental settings validate the effectiveness and generalization of our approach. Notably, PseDet outperforms previous methods by 4-~17 mAP on different learning settings, achieving new state-of-the-art incremental object detection.
RELATED WORKS
2.1 INCREMENTAL LEARNING
Incremental learning has been studied by the machine learning community for a long time (Schlim- mer& & Granger, 1986, Masana et al., 2022 Joshi & Kulkarni 2012). with the main goal of alle-
viating the catastrophic forgetting problem (Kmmker et al.) 2018 8 Goodfellow et al./ 2013 Hayes et al. 2020). Generally speaking, there are four classes approaches. Firstly, memory- based et al. 2020a] Lopez- & Ranzato 2017] Caccia et al.)
approaches (Chaudhry et al. 202I) use a small-sized cache to store information from previous tasks, which are replayed when training the model on new tasks. Secondly, during the current task training, regularization-ben based methods (Kirkpatrick et al., 20177: Zenke et al. 2017 Li & Hoiem| 2017) regularize the updating of those model parameters that important previous tasks. For example, EWC (Kirkpatrick) jet al. (207)) measures the parameter importance using Fisher information matrices, whose compu- tation efficiency is improved in the subsequent work SI (Zenke al. 2017). Thirdly, parameter
2

---

Published as a conference paper at ICLR 2025
isolation-based methods (Rusu al. 2016 Yoon et al. 2019 Fernando et al. 2017 Kang al. 2022), e.g., PathNet (Fernando et al. (017) aims to separate the model parameters of each task by that they will not be able to affect each other. Finally, the approaches (Li & Hoiem 2017 Rebutti et al., 2017 (W et al. 2019). e.g., LwF ([i & Hoiem 2017), explicitly inject task-dependent information of previous tasks through knowledge distillation. For example, in iCaRL (Rebuffi et al. 2017), knowledge distillation is used alleviate the excessive deterioration problem.
learning task-dependent sub-networks
knowledge distillation-haan
2.2 INCREMENTAL OBJECT DETECTION
While there is a significant amount of research focusing on incremental learning for image classi-
fication, solving the incremental learning for object detection (Peng al. 2021 Perez-Ree Ruvezz-Raaue al.)
2020 Li et al. 2019 Joseph et al., 2021]: Feng et al. 2022) is a non-trivial task. (Liu et al.) 2020
first proposes weight consolidation approach by applying EWC to two-stage detectors like Faster
R-CNN. MVCD (Yang et al. 2022) introduces feature and response distillation by splitting channels
and spatial features for model regularization. Elastic Response Distillation is proposed in (Feng ctt al. 2022) to conduct knowledge distillation on detector classification and regression branches.
CL-DETR (Liu et al. 2023a) explores the incremental detection setting in the scope of detection transformer. SDDGR (Kim et al. (024) leverages generative model based on stable diffusion to mitigate catastrophic forgetting. Besides, (Joseph al. (202I) leverages open- approaches to integrate incremental learning and open-set object detection.
PSEDET
In this section, we first provide the problem formulation on incremental object detection ($ 3.T. Then we briefly introduce the overall framework of PseDet ($ (.2p. which consists of three key components,
Spatio- Temporal Enhancement Module ($ 3.3). Categorical Adaptive Label Selector ($ 3.4). and
Confidence Score Calibration Supervision ($ 3.5
3.1 PROBLEM FORMULATION
Incremental object detection (IOD) aims to detect interested objects across class domains in multiple steps, where we assume that there are N steps. Let DD denotes complete dataset comprising images I and their corresponding annotations y. which can be represented as {z, y/ (z, y) E D}. Mcanwhile, let C represent the category of the annotated objects in D, and partition the dataset into N subsets within the class domain C = C* .... UCN In step i, the model can only access the labels belonging to category Ct, but it cannot access previously learned categories Cla-11 even if objects of those categories exist in the images, The model needs to maintain the ability to detect class Clai-1 while learning to detect new ones C each class is only learned once.
3.2 OVERALL FRAMEWORK
The overall framework of our approach is shown in Figure During each step training, PseDet first prompts the previous model to generate improved pseudo labels with the proposed Enhancement module, by attenuating noise at its source through label fusion both the spatial and temporal domains. Then, the Categorical Adaptive Label Selector dynamically adjusts thresholds for each class within every mini-batch, automating the whole filtering process to assure the quality of pseudo labels across different categories. Finally, when computing the loss Lote the target model, we introduce the Confidence Score Calibration Supervision to align the confidence score with the localization quality through non-linear mapping and seamlessly integrate it into the supervision process.
3.3 SPATIO-TEMPORA ENHANCEMENT MODULE
In most incremental object detection frameworks, the detection performance of learned categories gradually declines as the learning progresses step by step. This is possibly due to the reorganization of the parameter space, which is caused by the limited model capacity knowledge is learned.
3

---

Published as a conference paper at ICLR 2025
STE Module
Categorical Adaptive Label Selector
Spatial
Temponal
Num Boxes
Detector Stage
Cotegary
Detector Stage T-1
Pseudo Baxes
Score
Filtered Bowes
Weight Inheritunce
Callbrated
(ol) x sigmookla G 50)
positivns
Detection Loss
Detector
Stage
Regetivc
Input Image
Confidence
Score Calibration Supervision
Figure 1: The overall framework of PseDet for incremental object detection. We introduce three critical components maximize the power of pseudo-labels in IOD: Peatio-Temporal Enhancement module, Categorical Adaptive Label Selector, and Confidence Score Calibration Supervision.
However, assume that such decline also be attributed to noise interference during the knowl- edge transfer process. Maximizing the signal-to-noise ratio of the information source and minimizing noise in the transmission process is crucial for improving the accuracy of information transmission. Inspired by this, we enhance pseudo-labels at the spatial domain to reduce the discrepancy when inferring pseudo-labels from the old model. Moreover, in multi-step scenarios, conduct fusion in the temporal domain to prevent the accumulation and magnification of noise across steps.
Current approaches enhance supervision by leveraging zero-shot learning methods generate labels or by simply replaying labels. To avoid introducing more complex networks and operations, we employ straightforward strategy by applying series of spatial transformations the inputs. Inputs processed through different augmentation techniques can aid the model in detecting objects of varying types and sizes, thus improving the robustness of the model. Formally, we define an augmentation transformation set A. which encompasses transformations applied to image tensors in the spatial domain. For input image x E DD transformed via A,, we use the old model Potd to predict objects that belong to the old categories:
ota (A.())),
(1)
where y, is the prediction of the model oot after transformation A, Note that all these augmentation operators are invertible. Once the predictions are obtained, we simply reverse them back to the original space and perform unified fusion operations:
p = F=A
where P is the set of pseudo-labels passed the next step, A,- denotes the reverse transformation of A,, for inputs that are horizontally flipped, A re-flips the model's output horizontally; for scaled inputs, it scales them back to their original size. represents spatial fusion, typically using NMS with an loU threshold of 0.4.
The quality labudo-labels generated by the teacher model decreases the incremental learning progresses. In the context of multi-step incremental learning, the knowledge acquired earlier is contaminated by accumulated noise more severely as the training continues. This contamination can potentially deteriorate the model's ability to effectively learn new classes. Therefore, we introduce the temporal domain enhancement minimize the model's susceptibility to noise interference. Suppose the model can access ground truth labels of categories C and pscudo-labels of categories C²-1 at step i, we collect the optimal pseudo-labels P. the temporal domain:
P = pl U... ....!!
(3)
where p 1 represents the pseudo-labels generated by the model P1.-11 containing the set of categories Cla By combining the spatial and temporal enhancement, our PseDet greatly alleviates the noise influence from the old label and greatly improves the label quality.
4

---

Published as a conference paper at ICLR 2025
43
Stage
Stage
Threshold Upper Bound
0.7
Ver
Confidenze Score
0.-
23
204
18
18
033
13
8
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Confidence Score Threshold
Category
Figure 2: Experimental results with different Figure 3: The class-wiss confidence score distribu- hard thresholds, showing the impact filter on tions in the 40-40 experiment, show its variability across different classes and steps.
performance.
3.4 CATEGORICAL ADAPTIVE LABEL SELECTOR
In IOD tasks, the key reason for catastrophic forgetting by the detector the shift between the new labels and the previously learned ones (Liu et al. 2023b). While images contain both old and new classes, annotations only contain labels for objects the current class, causing the model to consider the old classes as background during the fitting process to the current dataset of this step.
Pseudo-labeling provides an effective solution to transfer such knowledge across different steps. Simply applying different thresholds to filter labels based on confidence score leads to fluctuated performance depending on the threshold selection. As shown in Figure 2 a common issue in pseudo- labeling methods is that the performance of the detector is highly sensitive to the threshold. When the threshold is too small, a large amount of noise can mislead the detector's training. Conversely, when the threshold too large, many valid pseudo-labels may be discarded, leading to significant forgetting. Not only that, as shown in Figure 3 we observe that the score distribution of pseudo-labels changes with each step in continual learning. It is unrealistic manually determine the threshold for each class, especially under the incremental learning setting. Therefore, it is non-trivial to automatically determine the score threshold for label quality filtering.
Interestingly, we observe a distinct clustering of confidence scores for noisy and high-quality boxes, primarily in the low-confidence region, with evident class-specific characteristics. Based on the varying distributions of true positives (TP) and false positives (FP), we employ K-Means to fit these distributions for different classes, dynamically determining class-wise thresholds.
Algorithm 1 Pseudo label selection in stage
Input: candidate pseudo label set Pinputi k-means input queue length N.
Output: selected pseudo label P Poutput
Initialize the confidence score queue Q" +- { for each category c E (1:--1
for (S. C) Pinput do
Add the confidence score to the queue S - Q
end for for e E Cl:i-1 do K-means Q9) + DDr Dr. end for
P Poutput
D). .... .....UD
Algorithm|] I provides detailed demonstration dynamic class-wise pseudo-label filtering process. As the distribution of confidence varies across classes, we initialize queue @ for each category C.. We first enqueue the samples S of this batch, while the sample at the front the queue will be dequeued. For each queue, we apply K-Means fit the true and false distributions. The final output the union of positive samples across all categories. Our method effectively filters noise by combining short-term memory with statistical methods, achieving high performance.
5

---

Published as a conference paper at ICLR 2025
3.5 CONFIDENCE SCORE CALIBRATION SUPERVISION
Different from the carefully human-annotated ground truth obtained at the current stage, pseudo- labels inferred from the previous detector, containing significant amount of unexpected noise. not only leads to forgetting but also interferes with the learning of new classes at the current step. Previous works tend to filter out low-confidence noise directly retain accurate pseudo-labels, which leads to significant knowledge forgetting. As shown in Figure 2 using a higher threshold directly to filter noise P {(s", c)ls > Sthres } results in a significant performance drop.
Orsgir
Colbbrttd: 0.77
0.9
0.9
0.7
0.7
0.5
0.6
0.5
0.5
0.5
0.4
0.5
0.5
0.7
0.8
0.9
1.0
0
0.5
0.5
0.7
0.8
1.0
Tol: with ST
Toll with S7
(a) Original Distribution
(b) Distribution After Mapping
(c) Mapping Function Curve
Figure 4: Distribution of IoU-Score for pseudo-labels before and after mapping, along with the plot of the mapping function. In the top left corner of the distribution plot is the Pearson Correlation Coefficient (higher is better), which measures the strength of the correlation between IOU and score. From (a) and (b), it can be observed that after mapping. labels with high IoU are concentrated in the high-score region compared to before mapping, benefiting the model's attention to high- pseudo-labels.
Recent work (Jiang et al. 2018) suggests that confidence scores can indicate label accuracy, implying higher attention should be given to samples with higher scores during model training. However, our findings show a nin-linear relationship between label quality and confidence scores. Therefore, we map the confidence score s to a quality coefficient q. which better represents label accuracy. As shown in Figure 4al the scatter plot of IoU versus is not linear around the line y = In the interval s [0.5, 1]. bounding box quality improves significantly, while s = [0, 0.5). accuracy is generally lower. The sigmoid function is suitable for mapping q and with translation and scaling to ensure it operates within [0.1]:
((s) = sigmoid(a (s- 1+ a=(s--)))
(4)
where is the scaling coefficient, d is the translation distance. We set = 10 and d = 0.5 in the implementation. Figure 44 illustrates the graph of the mapping function, and Figure 4b illustrates the distribution after mapping.
Instead of directly assigning pseudo labels and ground truth labels to each prediction as 0 or 1, we use the qpt and the IoU r between predictions and their matched labels to soften the categorical labels. Unlike GFL (Li et al. 2020), which assigns labels based on the IoU between network predictions and labels, consider the quality of labels an important factor in approach, where transform Quality Focal Loss into a form suitable for continuous label representations IOD:
Úpt T 4pl-
where dnt is the joint quality, q is the quality of the pseudo label, and 7 is the IoU between the label and its assigned prediction. The class loss for the pseudo labels is:
Ced = y | ((1 - 3) log(1 - únt) + log(ūp)))
(6)
where y is the predictions of the current step model, / is set 2. For the ground truth, which is universally recognized as high quality 9gt = 1. allowing us to uniformly represent 0pt and Øge with Φ:
1gt
ggt

---

Published as a conference paper at ICLR 2025
Table 1: Incremental results on COCO benchmark under the one-step setting. Most experimental results are borrowed from SDDGR (Kim et al. 2024). AP, APso, and AP7s reflect the overall performance (higher is better) of the model after one step of learning. AbsGap (lower better) and
RelGap (lower is better) represents the absolute gap and the relative gap toward upper bound. The best performance highlighted in bold.
Scenarios
Method
Detector
APT
APsot
APas
AbsGapl RelGapl
Upper Bound (Full data)
GFL
40.2
58.3
43.6
Deformable DETR
47.0
66.1
50.9
LwF (Li & Hoiem.
2017
GFL
17.2
25.4
18.6
23.0
57.2%
RILOD Li al. 2019
GFL
29.9
45.0
32.0
10.3
25.6%
SID
Peng al. 2021
GFL
34.0
51.4
36.3
6.2
15.4%
ERD Feng ct 2022
GFL
36.9
54.5
39.6
3.3
8.2%
40 + 40
PseDet (Ours)
GFL.
38.5
54.9
41.9
1.7
4.2%
CL-DETR (Luu et al.
2023a
Deformahle DETR
42.0
60.1
45.9
5.0
10.6%
SDDGR Kim
2024
Deformable DETR
43.0
62.1
47.1
4.0
8.5%
PseDet (Ours)
Deformable DETR
43.5
61.5
47.2
3.5
7.4%
LwF (Li & Hoiem.
2017
GFL
7.1
12.4
7.0
33.1
82.3
RILOD L
2019
GFL
24.5
37.9
25.7
15.7
SID Peng ct al. 202
GFL
32.8
49.0
35.0
7.4
18.4
ERD Feng ct al.
2022
GFL
34.9
51.9
37.4
5.3
13.2
70 10
PseDel (Ours)
GFL
39.2
55.6
42.8
1.0
2,5%
CL-DETR [Liu et al. 2023a
Deformable DETR
40.4
58.0
43.9
6.6
14.0%
SDDGR (Kim et al. 2024
Deformable DETR
40.9
59.5
44.8
6.1
13.0%
PseDet (Ours)
Deformable DETR
44.7
62.9
48.6
2.3
4.9%
For the ground truth, we QFL the class loss without any alterations. Integrating the above formulas, we can unify the class loss for both pseudo labels and ground truth as Cets:
Cots = Cole
Cat =- y- 21 ((1 y) log(1 9) log(
(8)
It's worth noting that method can be utilized with other variations of continual versions of cross-entropy loss. Moreover, it can be easily extended to other domains that leverage pseudo-labels, such semi-supervised object detection (Wang et al. 2023).
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets and Evaluation Metric. MS COCO 2017 (Lin et al. 2014) is object detection dataset with 80 categories. These categories will be divided into different mutually exclusive sets based on the scenario of the experiment different steps. The evaluation metries include standard COCO protocols: AP, AP APsu, AP APM and APL; absolute gap (AbsGap) and relative gap (RelGap) between final mAP of incremental learning and mAP of upper-bound; forgetting percentage points (FPP), which is used to evaluate the degree of forgetting for trained categories.
Experimental Setup. our experiments, we apply the method different scenarios by partitioning the sets of categories into different collections C'. At each step, we start by training the detector normally using the initial set categories. subsequent steps, the model can only access the ground truth of the current step's 's categories. We mainly focus the following two scenarios: (a) One-step: 40 40, 50 30, 60 20, 70 + 10; (b) Multi-step: 40 + 20 2, 40 + 10 x 4.
Implementation Details. We implemented method GFL (Li et al. 2020 and Deformable DETR et al., 2020) with ResNet-50 image backbone. All experiments are performed on NVIDIA Tesla V100 GPUs, and the basic settings follow the official implementation(Chen al. 2019). For GFL (Deformable DETR), we set the batch size to (4) per GPU, trained for 12 (50) epochs, and used SGD (AdamW) the optimizer.

---

Published as a conference paper at ICLR 2025
Table Incremental results (AP, %) on COCO benchmark under the multi-step setting, In the first step, normal training is conducted with 40 categories, followed by the addition of 20 and 10 new categories the 2-step and 4-step settings each time, respectively.
40+10+10+ 10+ 10
40+20+20
Method
Detector
(40-50)
(50-60)
(60-70)
(70-80)
(40-60)
(60-80)
RILOD (Li al.
2019
GFL
25.4
11.2
10.5
8.4
27.8
15.8
SID Peng
2021
GFL
34.6
24.1
14.6
12.6
34.0
23.8
ERD Feng 2022
GFL
30.8
26.2
36.7
32.4
PseDel (Ours)
GFL
37,9
37.5
38.4
38.1
CL-DETR (Liu et al. 2023a
Deformable DETR
28.1
35.3
SDDGR (Kim al. 2024
Deformable DETR
42.3
40.6
40.0
36.8
42.5
41.1
PseDet (Ours)
Deformable DETR
42.7
41.1
41.5
41.2
42.3
42.8
Table 3: Ablation study using the COCO benchmark of 40 classes 40 classes. All categories and Old categories represent the performance (AP/ PPPPPP///RR 7ss higher is better) evaluated by the model after completing one-step of learning all 80 categories and the old 40 categories, respectively. The Forgetting Percentage Point (FPP) reflects the performance gap on the initial 40 categories between the start and completion of training, indicating the degree forgetting with lower values preferred.
All categories
Old categories
FPP
Method
AP
Afso
APrs
AP
APsa
AP3
AP
APso
APss
Fine-tuning
17.9
26.9
19.3
0.0
0.0
0.0
40.6
59.0
44.1
+ Normal Pseudo Labeling
22.8
33.1
24.8
26.6
37.9
29.4
14.0
21.1
14.7
Spatial Enhancement
29.9
42.8
32.6
31.9
44.5
35.3
8.7
14.5
8.8
+++ Categorical Adaptive Label Selector
34.1
49.2
37.2
35.9
51.2
39.5
4.7
7.8
4.6
+++ Confidence Score Calibration
38.5
54.8
41.9
40.8
57.5
45.1
-0.2
1.5
-1.0
4.2 OVERALL PERFORMANCE
One-step. Table shows the incremental performance of PseDet and other approaches COCO in the one-step scenarios of 40 + 40 and 70 + 10. In each scenario, our method demonstrates substantial improvements state-of-the-arr, is worth mentioning that in the 70+10 scenario, we achieve 3.8% increase in mAP compared to the previous best method (SDDGR), with AbsGap between the upper bound narrowing to just 2.3%. Similarly, in 40 40 scenarios, PseDet also maintains leading performance, with improvements of 0.5% in mAP. reducing the AbsGap with the upper bound 3.5%.
Multi-step. Table shows the incremental performance of our method and others on COCO in the more challenging multi-step scenarios of 40 20 and 40 10 x 4, which pays more attention to the capability in long-term incremental learning. Unlike other methods that suffer from severe memory decay, our approach exhibits relatively slow forgetting. In the challenging 4-step setting, upon completion of all steps, achieved 16.4% improvement in mAP compared to SDDGR. Similarly, in the 3-step setting, we observe an improvement of 1.7%, which validates the effectiveness and the anti-forgetting of our approach.
4.3 ABLATION STUDIES
In this section, we conduct ablation experiments using the GFL detector in the 40+40 scenario, and the experimental results are shown Table Fine-tuning refers to training on new categories without additional operations, resulting in catastrophic forgetting. Normal Pseudo Labeling refers to the utilization of pseudo labels generated by an old detector without any filtering or processing, and mixing them directly with ground truth for training. For Spatial Enhancement, since it is a one-step setting, the experiments improve the quality of pseudo labels solely through spatial enhancement. After implementing the Categorical Adaptive Label Selector and confidence Score Calibration, the model's 's performance improved by 4.2% and 38.5% respectively in mAP, outperforming previous oftt-of-thh-art methods.
8

---

Published as a conference paper at ICLR 2025
Table 4: The performance of detectors with different queue length of Categorical Adaptive Label Selector under the scenario of 40+40.
All categories
Old categories
New categories
Queue Length
AP
APso
AP1s
APso
APts
AP
APs
AP.75
38.5
54.8
41.9
40.8
57.5
45.1
36.2
52.1
38.7
150
38.5
54.9
41.8
40.9
58.0
45.6
36.1
51.8
38.1
200
38.5
54.9
41.8
40.9
57.7
45.2
36.1
52.0
38.5
Table 5: The performance of detectors with varying and ó of the mapping function sigmoid(a
(s 6)) under the scenario 40+40.
Params
categories
categories
New categories
AP
APso
APrs
AP
APso
APrs
AP
APs
AP75
10
0.6
37.1
53.1
40.3
39.2
55.9
43.2
34.9
50.3
37.4
10
0.5
38.5
54.8
41.9
40.8
57.5
45.1
36.2
52.1
38.7
10
0.4
37.9
54.2
40.9
40.3
57.2
44.4
35.4
51.3
37.5
5
0.5
38.3
54.5
41.5
40.7
57.3
44.9
35.9
51.7
38.1
20
0.5
38.1
54.6
41.6
40.6
57.4
44.7
35.9
51.8
38.4
4.4 ANALYSIS
Queue length in Adaptive Label Selector. Table summarizes model performance lengths. Despite minor fluctuations, overall performance remained steady at 38.5% 5% AP, with slight deviations of only 0.1% in A 5 and A With a queue length of 100, old category performance decreased slightly while new categories showed a slight improvement, differing by just 0.1% in AP. However, longer queues increase computational demands, impacting efficiency, Thus, opting for smaller value, N 100, which is more practical.
queue
Coefficients for Score Calibration. presents the performance across various a and å settings for the mapping function, with the best performance achieved at 10 and 8 = 0.5. As 6 changes, model performance affected: increasing å reduces attention pseudo-labels and decreases AP by 1. 1.4%, while decreasing 8 increases g, enhancing pseudo-label supervision but introducing noise and causing performance decline. a scales the mapping function, and increasing a widens the quality gap between high and low scores. Compared to adjusting ô has smaller impact performance.
aati-forgetting Performance in Multi-step. Figure illustrates the model's performance on specific categories under multi-step scenarios. Each line depicts the model's performance on the same subset of categories C at different steps. Starting from accessing ground truth at step i, the model exhibits a relatively low forgetting rate in each subset. More detailed analysis can be found in the appendix.
42
40
40
38
1-40
36
1-40
34
50-60
40-50
34
-040-60
66-80
32
60-70
70 80
Overall
Overall
Initia (1-40)
1st-step (+20) 2nd-step (-20)
Inihial (L-40)
1st-step (+10)
2nd-step (-10)
3rd-atep (-10)
4th-atep (-10)
Incremental Learning Process
Incremental Leamnnng Process
(a) Two-step performance
(b) Four-step performance
Figure The performance of our model the multi-step scenario. Overall indicates an evaluation of the entire set in the current step, while others refer to evaluations of subsets C:.

---

Published as a conference paper at ICLR 2025
5 CONCLUSION
In this paper, we revisit the power of pseudo-labeling in continual object detection. Specifically, we identify three critical issues that arise when applying pseudo-labeling. Based on these observations, we propose simple yet effective framework, namely PseDet. It introduces the spatio-temporal enhancement module to effectively improve the quality of the labels. With the proposed categorical adaptive label selector, PseDet can dynamically decide label filtering thresholds based the score distributions of different categories. To fully leverage the information from pscudo-labels, we align the confidence scores with the localization quality and integrate them into the supervision. Through extensive experiments conducted on the MS COCO dataset with various settings, we validate the effectiveness of method, achieving state-of- of-the--rtttttt in this era.
ACKNOWLEDGEMENT
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
REFERENCES
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. arXiv preprint 00005 2021.
Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:18/2.00422 2018.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: / 906.0000..555 2019.
Zehui Chen, Chenhongyi Yang. Qiaofei Li, Feng Zhao, Zheng-Jun Zha, and Feng Wu. Disentangle your dense object detector. In Proceedings of the 29th ACM international conference on multimedia, pp. 4939 4948, 2021.
Tao Feng, Mang Wang, and Hangjie Yuan. Overcoming catastrophic forgetting in incremental object detection via clastic response distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9427-9436, 2022.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprini arXiv: nee:::111000444 2017.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient-bart based neural networks. arXiv preprint arXiv: arXiv:1312.00 1312.621,, 2013.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In Proceedings of the European conference on computer vision, pp. 466-483. Springer, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: /503. 05.. 2015.
10

---

Published as a conference paper at ICLR 2025
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In Proceedings of the European conference on computer viston, pp. 784-799, 2018.
KJ Joseph. Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5830-5840, 2021.
Prachi Joshi and Parag Kulkarni. Incremental learning; Areas and methods-a survey. International
Journal of Data Mining & Knowledge Management Process, 2(5):43, 2012.
Haeyong Kang, Rusty John Lloyd Mina. Sultan Rizky Hikmawan Madjid, Jachong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang. and Chang D Yoo. Forget-free continual learning with winning subnetworks. In Proceedings of the International Conference on Machine Learning, pp. 10734-10750. PMLR, 2022.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tirunch, and Seungryul Back. Sddgr: Stable diffusion-based deep generative replay for class incremental object detection. arXiv preprint arXiv:2402. 17323, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Dawei Li, Serafettin Tasci, Shalini Ghosh, Jingwen Zhu, Junting Zhang, and Larry Heck. Rilod: Near real-time incremental learning for object detection at the edge. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 113-126, 2019.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Gen- eralized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33:2 33:21002-21012, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40 40(12))2935-2947, 2017
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European conference on computer vision, pp. 740-755. Springer, 2014.
Liyang Liu, Zhanghui Kuang, Yimin Chen, Jing-Hap Hao Xue, Wenming Yang, and Wayne Zhang. Incdet: In defense of elastic weight consolidation for incremental object detection. IEEE transactions neural networks and learning systems, 32(6):2306-23199 2020.
Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, and Christian Rupprecht. Continual detection transformer for incremental object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23799-23880 2023a.
Yuyang Liu, Yang Cong, Dipam Goswami, Xialei Liu, and Joost van de Weijer. Augmented box replay: Overcoming foreground shift for incremental object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 11367-11377, 2023b.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning.
Advances in neural information processing systems, 30, 2017.
Mare Masama, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-ines incremental learning: survey and performance evaluation on image classification. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
11

---

Published as a conference paper at ICLR 2025
Can Peng, Kun Zhao, Sam Maksoud, Meng Li, and Brian C Lovell. Sid: Incremental learning for anchor-free object detection via selective and inter-related distillation. Computer vision and image understanding, 210:103229, 2021.
Juan-Manuel Perez-Rua, Xiatian Zhu, Timothy M Hospedales, and Tao Xiang. Incremental few-shot object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13846-1333 13855, 2020.
Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampeet. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Puttern Recognition, pp. 2001-2010, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 2016.
Jeffrey C Schlimmer and Richard H Granger. Incremental learning from noisy data. Machine learning, 1:317-354, 1986.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9627-9636, 2019.
Xinjiang Wang, Xingyi Yang, Shilong Zhang, Yijiang Li, Litong Feng, Shijie Fang, Chengqi Lyu, Kai Chen, and Wayne Zhang. Consistent-teacher: Towards reducing inconsistent pseudo-targets in semi-supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3240-3249, 2023.
Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 374-382, 2019.
2019.
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances neural information processing systems, 34: 10077-1090000 2021.
Dongbao Yang, Yu Zhou, Aoting Zhang. Xurui Sun, Dayan Wu, Weiping Wang, and Qixiang Ye. Multi-view correlation distillation for incremental object detection. Pattern Recognition, 131: 108863, 2022.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXi:: 09432, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131-1140, 2020a.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 9759-9768, 2020b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint urXiv22010.04159, 2020.
12

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Dataset Preparation and Preprocessing. There are 80 actual valid categories in COCO 2017, and the category numbers are not consecutive. We employ cocopytool to read annotations and randomly assign category labels to label indices ranging from O to 79. In each consecutive experiment, maintain a fixed random seed and divide them into mutually exclusive sets based on scenarios.
Training and Evaluation. During model training, due the defined category sets, small portion of images do not contain any annotations. We filter out images that contain neither ground truth nor pseudo-labels. During evaluation, assess all images in the eval-set but only calculate the metrics for the currently learned categories.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 NOT NESSE BUT REINFORCEMENT: EMPOWERING MODELS TO MASTER PRIOR KNOWLEDGE
Figure Al depicts the performance of various methods across different scenarios. Each line represents the per-
40
formance of the detector on the evaluation set under the one-step scenario. The number of categories in C varies from 40 to 70, while the number of categories in C2 changes from 40 10. The trend of the curves indicates that the previous method exhibits significant forgetting characteristics, whereas our method demon- strates better performance the amount of knowledge to remember increases. Based on the experimental re- sults above, have effectively addressed the issue of knowledge forgetting that occurs during the transfer of new knowledge in continual learning, well bal- anced the utilization of model capacity between new and old knowledge.
-B-Ours
-+-ERD
-.-SID
->.BOLOD
22
40
50
70
Initial Number of Categories
Figure Al: The performance under the one-step scenario. With other methods, the larger the initial set, the faster the for- getting; method performs better and is suitable for continual learning.
It's quite surprising that despite the expected increase in the absolute value of knowledge forgetting the number of categories in the first step increases, the performance of the final detector actually improves with our method. We believe this could be attributed the effect of attention, where we introduced in the loss calculation, which played significant role.
Table AT shows the experimental results of our method based on GFL under the 50+30 and 60+20 scenarios as supplement to the main paper. All experimental results, including those in Table 3 demonstrate the scalability and robustness our method, providing further support for our viewpoint:
our method not only alleviates forgetting but also reinforces previous knowledge.
Table AI: Additional results on COCO benchmark under the one-step setting based on GFL detector.
Most experimental results are borrowed from ERD (Feng et al. 2022 paper.
Detector
Scenarios
Method
AP
APs
APrst
AbsGapl
ReIGapl
Full (Upper Bound)
40.2
58.3
43.6
LwF LL & Hoiem
2017
S.0
9.5
4.6
35.2
87.6%
RIL.OD [T.
2019
28.5
43.2
30.2
29.1%
50 classes classes
SID
Peng et al.
2021
33.8
51.0
36.1
6.4
15.9%
ERD Fong ct al.
2022
36.6
54.0
38.9
3.6
9,0%
GFL
PscDet (Ours)
39.0
55.6
42.1
1.2
3.0%
LwF Li Hoiem 2017
5.8
10.8
5.3
34.4
85.6%
RILOD LLi al 2019
25.4
38.8
26.8
14.8
36.8%
60 classes 20 classes
SID Peng
2021
32.7
49.8
34.6
7.5
18.7%
[reng et al.
2025
35.8
52.9
38.4
11.0%
PseDel (Ours)
39.1
55.7
42.5
2.7%
13

---

Published as a conference paper at ICLR 2025
B.2 ROBUST AND HIGH- PERFORMEFE TACKLING MULTI- SILTI-STEP CHALLENGES WITH PSEDET
Evaluating the untieftrgg performance of the model requires only comparing the performance of the detector the entire dataset across different steps but, more importantly tracking the perfor- mance of the detector on the same subset. Each line Figure and Figure A2 tracks the performance of the detector subset C2 starting from when the detector learns the ground truth of C*.
In addition what we observed in Figure we also noticed a similar phenomenon in multi-step settings, where the more knowledge to be retained, the stronger the anti-forgetting ability. From the Table A4 our method's implementation on Deformable DETR achieved final performance 42.8% in AP on the 2-step setting, which is 0.5% higher than the intermediate stage. Similarly, the 4-step setting, the final performance is 0.15 % higher than the intermediate stage.
48
46
44
42
mAP
36
33
70-80
60-70
34
Overall
32
Initial (1-40)
Initil (1-40)
Ist-step (-10) 2nd-step (+10) ard-step (-10) 4th-step (+10)
Incremental Learning Process
Inenemental Learning Process
Two-step performance
(b) Four-step performance
The performance of the Deformable DETR, based on our method, in the multi-step scenario serves supplement to Figure Overall indicates evaluation of the entire in the current step, while others refer to evaluations of subsets C The stability of each line represents the excellent anti-forgetting ability of method.
Figure A2:
B.3 LESS IS MORE: NOISE PRESENTS GREATER DISRUPTION TO LEARNING
In object detection, AP, as critical evaluation metric, intuitively, is sometimes used to evaluate the quality of pseudo-labels. As shown in Table A2, when we use filters to remove noise, there is significant decrease in the metrics. However, after the second stage of learning, the performance of the group of detectors with lower AP noticeably improves.
useful information, through experimental observation, we believe that appropriately screening out uncertain annotations is helpful for learning and will not result in a loss of knowledge.
Although noise contains
Table A2:
The evaluation results of the pseudo-labels. the first step of 40+40 scenario, the validation set is inferred based on Deformable DETR with or without our pseudo-label selector.
Setting
AP
AP.,
AP77
APs
APM
AP,
w/o Pseudo-label filtering
46.0
66.6
49.4
22.9
44.7
61.6
w/ Pseudo-label filtering
33.5
48.1
36.2
17.8
33.2
43.7
C PROOF OF ALGORITHM
To start with, we infer pseudo-labels represented by (c, b) using old model Oold, where e denotes the class, indicates the confidence score, and represents the position coordinates of the bbox.
Based on careful observation, we have noted that, for each class the model's prediction confidence exhibits distinct bimodal distribution, comprising both high-confidence regions (positive samples) and lww-confidence regions (noise). The data in these regions respectively display a "Gaussian bell" shape centered around specific means Jr:
14

---

Published as a conference paper at ICLR 2025
High confi-enfieeece peak: This peak indicates that the model is highly confident in its detection results, representing high-quality pseudo-labels. These samples are characterized by clear features and high accuracy, clustering around jrr.
Low-confidence peak: Typically, this peak reflects the model's uncertain predictions or samples that difficult to distinguish. In this region of distributions, the pseudo-labels are mostly inaccurate noise, clustering around μ.p.
The confidence score follows bimodal distribution, and its probability density can be expressed as:
p( ()) = (s |r, of) + Tr.N(s 1up, x
(A1)
Let S; E In represents the set of confidence scores for the i - th class. In E-step of the Expectation- Maximization (EM) algorithm, we assign each data point the cluster in which it has the highest posterior. The posterior is:
TT HT, (TT,O))))
plc; = k 83) TT Mr. of. Ty ur, up,oz)
(A2)
For the convenience of analysis, we can obtain the reciprocal of the posterior:
T TFN(85 |r, plcj True 8z)= 1
TT pr,, of. oF exp(
(A3)
1
=1+ TT
(s; -r 207
(s; - pF)?)
20
Here, K-means aims to partition data into two clusters by minimizing the Squares:
Sum of
J= Σ uT|? + ||8; 1=l1²)
(A4)
iECL
In this distribution, samples can be considered have soft assignments. However, K-means which is type of hard assignment to assign sample i to the cluster k (k E {True, False)):
arg min ||>}
m
(A5)
By combining Equations A3 and A5
we obtain:
||s: Mk 112 x p-
(A6)
Maximizing the posterior probability p(c, = k {;) is equivalent to minimizing the ||sg 1 1|?
K-Means can be considered special case of the Dnpectation-Maxiiiaaan hard assignments, aimed at modeling the bimodal distribution of scores for true and false clusters. In addition, K-means clustering is often highly sensitive to the choice of initial centroids. Therefore, we repeat the clustering process multiple times.
(EM) algorithm with
D ADDITIONAL ANALYSIS ON MEMORY AND TIME OVERHEAD.
(1) GPU Memory. No additional GPU Memory is needed. The current methods, such as ERD (Fng al. 2022). require loading two models during each training iteration: one is the original model used to infer the logistic of a sample, and the other is the model from the current new stage. Our method generates pseudo labels in an offline manner. This means that during training, our method the same as the regular training and does not require any additional GPU memory. For example, based on GFL, our method occupies approximately 4.5 GB GPU memory per GPU, whereas ERD requires more than 10 GB per GPU, nearly double the GPU memory compared to our method. (2) Time Efficiency. Additional time cost is negligible. Diffusion-baast samples, while distillation sasti-lattoo- methods (Feng
methods (Kim et al., 2024) require additional
time train diffusion model and generate ct al. 2022) necessitate an extra inference step in each training iteration. In contrast, our approach infers pseudo labels once in offline manner. This pseudo-label set can then be repeatedly used in
15

---

Published as a conference paper at ICLR 2025
subsequent training sessions. If training is conducted only once, the additional time overhead for this offline inference is approximately limited to 5%. As the number of training steps increases, the time efficiency advantage of our method becomes even more significant. (3) Disk Usage: The additional disk usage required by our method is negligible. After inferring pseudo labels with the previous weights, the labels can be stored. It takes only few bytes per sample. For the COCO dataset, the additional pscudo label .kl files occupy approximately 10 MB of disk space.
Table A3: Comparison about Memory and Time.
Method
Core
GPU Memory
Time Efficiency
Disk Usage
ERD
KD-Based
Two Models loaded
An additional inference for one training iteration
Last step checkpoint.
SDDGR
Diffusion-hased
Normal Train
Additional time to gen- erate new samples with diffusion model.
Additional samples generated by diffusion model.
(One Model)
PseDet (Ours)
Pseudo-Label
Normal Train
Additional time to infer- ence Pseudo-I Labels.
Every step checkpoint hut only few hundred megahytes are required for each.
(One Model)
E IMPLEMENTATION IN SEGMENTATION TASK
We implemented our method based on SegFormer Xie et al. 2021) and the basic settings follow the official implementation. We selected the ADE20K (Zhou al. 2017) experimental dataset. This dataset presents a significant challenge due to its extensive category count of 150. It offers significant reference value for the implementation of class incremental learning.
In our experiment, we design two scenarios: 100+50 as the one-step setting and 100+25+25 as the multi-step setting. The evaluation metrics include mloU (higher better); absolute gap (AbsGap, lower is better) and relative gap (RelGap, lower is better) between final mloU of incremental learning and mIoU of upper-bound; forgetting percentage points (FPP, lower better), which used to evaluate the degrec of forgetting for trained categories.
Table A4: Incremental results on ADE20K benchmark based on SegFormer.
Method
100+150
Scenario
100+25+25 Scenario
Overrall 1-100
101- 101-150
AbsGopl ReiGapi FPP| Overall
1-100
101-125
126-150
AbsGaap ReIGap/ [PP]
Upper Bound
37.41
41.23
29.77
37.41
41.23
32.94
26.60
Catastrophic Forgetting]
0.88
2.63
36.53
97.64
43.11
0.23
0
1.36
37.18
99.38
43.11
PseSeg (Ours)
36.18
41.43
25,69
1,23
3.29
1668
35,00
40.63
25.44
22,04
2,11
6,44
0.60
Considering that segmentation tasks involve dense predictions and that, for SegFormer, there no classification and regression branches as seen in detection tasks, it is unnecessary to consider decoupling classification and regression tasks within the context of continual learning. Consequently, pseudo-label-baa methods may yield better performance when applied to segmentation tasks. Consistent with the experimental results, the model trained through continual learning exhibits an absolute gap of only 1.23 mloU in the one-step scenario and 2.41 mloU in the multi-step scenario between our method and the upper bound. FPP gets 1.68 mloU and 0.60 mloU in the one-step and multi-step settings respectively, reflecting the excellent resistance to forgetting of our method.
F ADDITIONAL VISUALIZATION RESULTS
Figure A3 displays the visualization of pseudo-labels at the final stage in the 70+10 scenario. can be observed that the detector still maintains good detection performance for objections with challenging difficulty levels. The advantage of the pseudo-label strategy also lies in the detector's ability to discover and annotate objects that were not manually labeled. As can be seen from the last two images in Figure A3 flowers that are occluded or smaller horses that were not annotated were still detected by the detector. This, to some extent, mitigates the loss in memory.
16

---

Published as a conference paper at ICLR 2025
1%
96.5%
66.
93.1%
27.7%
37
17.%
Figure A3: The visualization results of pseudo-labels. The ground truth is depicted with green bounding boxes, while pseudo labels are indicated by blue bounding boxes.
17


Reviews:
[{"review_id": "ATKP8OF5vr", "reviewer": "Reviewer_aL8R", "summary": "This paper presents PseDet, a framework for Incremental Object Detection (IOD) designed to address limitations in existing methods, such as low-quality pseudo-labels, fixed filtering thresholds, and misaligned confidence scores. PseDet introduces a spatio-temporal enhancement module to handle noise from previous models, a Categorical Adaptive Label Selector to adjust thresholds by class, and a non-linear score mapping for better localization accuracy. Experiments on COCO benchmarks show that PseDet achieves state-of-the-art performance.", "questions": "1.Equation 2 is unclear in its practical application. It’s not evident how the augmentation transformation set A for the image x can be reversed for the label y to obtain pseudo-labels after fusion. A more detailed explanation of this process would be helpful.\n\n2.It would be interesting to see a discussion on applying the proposed methods to other tasks, such as image classification and semantic segmentation, where pseudo-labels are also used.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-structured and easy to follow, with a clear motivation. The three main components—Spatio-Temporal Enhancement, Categorical Adaptive Label Selector, and Confidence Score Calibration Supervision—are thoughtfully designed to address specific limitations in existing methods.\n\n2. The observations presented, such as the effect of varying hard thresholds (Figure 2) and the IoU-score distribution for pseudo-labels before and after mapping (Figure 4), are straightforward yet insightful. These findings offer valuable perspectives that could inspire future research directions.", "weaknesses": "1.The proposed three components each address distinct issues, but the relationships between them are not well explored in the paper. A more thorough discussion of their interconnections would strengthen the overall framework.\n\n2.In the Spatio-Temporal Enhancement module, typically only the t−1 model can be stored in memory. Feeding varied types and sizes into all previous models would incur substantial memory costs and may create an unfair comparison with other methods. Including an analysis of training time and memory costs would be valuable for a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGIAwVxlo1", "reviewer": "Reviewer_T1Rk", "summary": "The paper addresses the challenge of Incremental Object Detection (IOD), where object detectors need to learn new classes without forgetting previously learned ones. The authors propose PseDet, a framework that improves pseudo-labeling in three key ways:\n\n1. Spatio-Temporal Enhancement Module: Improves pseudo-label quality by combining predictions from different spatial transformations and temporal steps.\n2. Categorical Adaptive Label Selector: Automatically determines class-specific thresholds for filtering pseudo-labels using K-means clustering, replacing fixed thresholds.\n3. Confidence Score Calibration: Aligns confidence scores with localization quality through non-linear mapping for better supervision.\n\nThe approach achieves state-of-the-art performance on COCO dataset, with 43.5+/41.2+ mAP under 1/4-step incremental settings.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies and addresses three critical issues in current pseudo-labeling approaches\n2. The proposed solutions are practical and well-justified\n3. The framework is modular and can potentially be integrated with other detection systems\n4. PseDet achieves large performance improvements on several benchmarks.", "weaknesses": "1. Algorithm 1 is not clear. What does D^c_T and D^c_F mean? Why the output of K-means is D^c_T and D^c_F? I suggest the authors to clarify these.\n2. The non-linear mapping function for confidence calibration seems empirically determined without strong theoretical justification. Can PseDet generalize well to other datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GY4NB4dUab", "reviewer": "Reviewer_qjDZ", "summary": "Incremental Object Detection (IOD) expands object detectors without forgetting prior knowledge. Existing methods rely on knowledge distillation but underutilize the teacher model's insights. This paper identifies three problems in pseudo-labeling: limited pseudo label quality, fixed label filtering thresholds, and weak alignment of confidence scores with localization quality. To address these problems, this paper proposes PseDet, which includes a spatiotemporal enhancement module to handle noisy data, a Categorical Adaptive Label Selector for dynamic class-specific thresholds, and confidence score calibration improves label confidence scores alignment with localization.", "questions": "see the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed modules look useful for incremental object detection, including spatio-temporal enhancement module, categorical adaptive label selector, confidence score calibration supervision.\n2. The motivation of categorical adaptive label selector is clear.\n3. The experimental performance of the proposed PseDet looks good compared to the SOTAs.\n4. According to the ablation study in Table 3, the proposed modules are effective.", "weaknesses": "1. It is unclear why the proposed spatio-temporal enhancement module is effective. With the empirical good results, please provide more inside explanation.\n2. The algorithm of categorical adaptive label selector is provided. Please provide some caption and formula to further describe it for clearer presentation.\n3. In categorical adaptive label selector, why k-means of the confidence scores can be a good threshold?\n4. The difference between the proposed confidence score calibration supervision and the IoU-aware classification score in VFNet [A].\n\nOverall, the proposed modules are effective, but lack inside explanations. I will give a rating of 5. If more inside explanations are provided, I will give a higher rating.\n\n[A] VarifocalNet: An IoU-aware Dense Object Detector.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)