PseDet: Revisiting the Power of Pseudo Label in Incremental Object Detection

Authors: Qiuchen Wang, Zehui Chen, Chenhongyi Yang, Jiaming Liu, Zhenyu Li, Feng Zhao
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Incremental Object Detection, Catastrophic Forgetting, Pseudo Labeling

Abstract:
Incremental Objection Detection (IOD) facilitates the expansion of the usage scope of object detectors without forgetting previously acquired knowledge. Current approaches mostly adopt response-level knowledge distillation to overcome forgetting issues, by conducting implicit memory replay from the teacher model on new training data. However, this indirect learning paradigm does not fully leverage the knowledge generated by the teacher model. In this paper, we dive deeper into the mechanism of pseudo-labeling in incremental object detection by investigating three critical problems: (a) the upper bound quality of the pseudo labels is greatly limited by the previous model, (b) fixed score thresholds for label filtering, without considering the distribution across categories, and (c) the confidence score generated by the model does not well reflect the quality of the localization. Based on these observations, we propose a simple yet effective pseudo-labeling continual object detection framework, namely PseDet. Specifically, we introduce the spatio-temporal enhancement module to alleviate the negative effects when learning noisy data from the previous model. Considering the score distribution divergence across different classes, we propose the Categorical Adaptive Label Selector with a simple mathematical prior and fast K-Means pre-computation to dynamically determine the class-wise filtering threshold. In order to align the label score with the localization quality of the pseudo labels, we project the score through non-linear mapping to calibrate the distribution and integrate it into the new-step supervision. Extensive experiments on the competitive COCO benchmarks demonstrate the effectiveness and generalization of PseDet. Notably, it achieves 43.5+/41.2+ mAP under the 1/4-step incremental settings, achieving new state-of-the-art performance.

Decision: accept (poster)
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730514604121

thread_type:
official_review

<thread_object>
creator: reviewer_qjdz

<content>
summary:
Incremental Object Detection (IOD) expands object detectors without forgetting prior knowledge. Existing methods rely on knowledge distillation but underutilize the teacher model's insights. This paper identifies three problems in pseudo-labeling: limited pseudo label quality, fixed label filtering thresholds, and weak alignment of confidence scores with localization quality. To address these problems, this paper proposes PseDet, which includes a spatiotemporal enhancement module to handle noisy data, a Categorical Adaptive Label Selector for dynamic class-specific thresholds, and confidence score calibration improves label confidence scores alignment with localization.

strengths:
1. The proposed modules look useful for incremental object detection, including spatio-temporal enhancement module, categorical adaptive label selector, confidence score calibration supervision.
2. The motivation of categorical adaptive label selector is clear.
3. The experimental performance of the proposed PseDet looks good compared to the SOTAs.
4. According to the ablation study in Table 3, the proposed modules are effective.

weaknesses:
1. It is unclear why the proposed spatio-temporal enhancement module is effective. With the empirical good results, please provide more inside explanation.
2. The algorithm of categorical adaptive label selector is provided. Please provide some caption and formula to further describe it for clearer presentation.
3. In categorical adaptive label selector, why k-means of the confidence scores can be a good threshold?
4. The difference between the proposed confidence score calibration supervision and the IoU-aware classification score in VFNet [A].

Overall, the proposed modules are effective, but lack inside explanations. I will give a rating of 5. If more inside explanations are provided, I will give a higher rating.

[A] VarifocalNet: An IoU-aware Dense Object Detector.

questions:
see the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688104409

thread_type:
official_review

<thread_object>
creator: reviewer_t1rk

<content>
summary:
The paper addresses the challenge of Incremental Object Detection (IOD), where object detectors need to learn new classes without forgetting previously learned ones. The authors propose PseDet, a framework that improves pseudo-labeling in three key ways:

1. Spatio-Temporal Enhancement Module: Improves pseudo-label quality by combining predictions from different spatial transformations and temporal steps.
2. Categorical Adaptive Label Selector: Automatically determines class-specific thresholds for filtering pseudo-labels using K-means clustering, replacing fixed thresholds.
3. Confidence Score Calibration: Aligns confidence scores with localization quality through non-linear mapping for better supervision.

The approach achieves state-of-the-art performance on COCO dataset, with 43.5+/41.2+ mAP under 1/4-step incremental settings.

strengths:
1. The paper identifies and addresses three critical issues in current pseudo-labeling approaches
2. The proposed solutions are practical and well-justified
3. The framework is modular and can potentially be integrated with other detection systems
4. PseDet achieves large performance improvements on several benchmarks.

weaknesses:
1. Algorithm 1 is not clear. What does D^c_T and D^c_F mean? Why the output of K-means is D^c_T and D^c_F? I suggest the authors to clarify these.
2. The non-linear mapping function for confidence calibration seems empirically determined without strong theoretical justification. Can PseDet generalize well to other datasets?

questions:
See weakness above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730900627236

thread_type:
official_review

<thread_object>
creator: reviewer_al8r

<content>
summary:
This paper presents PseDet, a framework for Incremental Object Detection (IOD) designed to address limitations in existing methods, such as low-quality pseudo-labels, fixed filtering thresholds, and misaligned confidence scores. PseDet introduces a spatio-temporal enhancement module to handle noise from previous models, a Categorical Adaptive Label Selector to adjust thresholds by class, and a non-linear score mapping for better localization accuracy. Experiments on COCO benchmarks show that PseDet achieves state-of-the-art performance.

strengths:
1. This paper is well-structured and easy to follow, with a clear motivation. The three main components—Spatio-Temporal Enhancement, Categorical Adaptive Label Selector, and Confidence Score Calibration Supervision—are thoughtfully designed to address specific limitations in existing methods.

2. The observations presented, such as the effect of varying hard thresholds (Figure 2) and the IoU-score distribution for pseudo-labels before and after mapping (Figure 4), are straightforward yet insightful. These findings offer valuable perspectives that could inspire future research directions.

weaknesses:
1.The proposed three components each address distinct issues, but the relationships between them are not well explored in the paper. A more thorough discussion of their interconnections would strengthen the overall framework.

2.In the Spatio-Temporal Enhancement module, typically only the t−1 model can be stored in memory. Feeding varied types and sizes into all previous models would incur substantial memory costs and may create an unfair comparison with other methods. Including an analysis of training time and memory costs would be valuable for a more comprehensive evaluation.

questions:
1.Equation 2 is unclear in its practical application. It’s not evident how the augmentation transformation set A for the image x can be reversed for the label y to obtain pseudo-labels after fusion. A more detailed explanation of this process would be helpful.

2.It would be interesting to see a discussion on applying the proposed methods to other tasks, such as image classification and semantic segmentation, where pseudo-labels are also used.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel framework addressing key limitations in pseudo-labeling for incremental object detection (IOD). The paper identifies three critical challenges: noisy pseudo-labels, fixed thresholding for label filtering, and misaligned confidence scores. The proposed framework, PseDet, tackles these through a Spatio-Temporal Enhancement module, a Categorical Adaptive Label Selector, and Confidence Score Calibration. These components collectively aim to enhance pseudo-label quality and alignment, thereby reducing catastrophic forgetting and improving detection performance. The work demonstrates state-of-the-art results on the COCO dataset across various settings. 

During the author-reviewer discussion period, reviewers raised concerns about the novelty of the core idea, the training time and memory costs of the Spatio-Temporal Enhancement module, etc. The authors responded by providing additional results. They clarified implementation details for Equation 2 and addressed memory concerns by highlighting the method's offline pseudo-label generation, which avoids excessive resource overhead. They also extended their approach to segmentation tasks, showing its generalizability. After the rebuttal period, two of the three reviewers are positive towards this paper. Reviewer aL8R still has remaining concerns about the memory assumption, which seems impractical and unrealistic in many scenarios.

Given that most reviewers are positive about the paper and there is no strong objection to accepting it, the paper is accepted. The authors are required to include the additional results and discussions in the final version. Besides, they should also discuss the limitations raised by Reviewer aL8R in their camera-ready version.