Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Authors: Qirui Wu
Shizhou Zhang
De Cheng
Yinghui Xing
Di Xu
PENG WANG
Yanning Zhang
Venue: ICML 2025 poster (2025)
Keywords: Object Detection
Continual Learning
Primary area: applications->computer_vision
TL;DR: We identify the RoI head classifier as the main source of forgetting in two-stage detectors and propose NSGP-RePRE, which uses Regional Prototype Replay and Null Space Gradient Projection to address it.

Abstract: Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD). Most existing methods treat the detector monolithically, relying on instance replay or knowledge distillation without analyzing component-specific forgetting. Through dissection of Faster R-CNN, we reveal a key insight: Catastrophic forgetting is predominantly localized to the RoI Head classifier, while regressors retain robustness across incremental stages. This finding challenges conventional assumptions, motivating us to develop a framework termed NSGP-RePRE. Regional Prototype Replay (RePRE) mitigates classifier forgetting via replay of two types of prototypes: coarse prototypes represent class-wise semantic centers of RoI features, while fine-grained prototypes model intra-class variations. Null Space Gradient Projection (NSGP) is further introduced to eliminate prototype-feature misalignment by updating the feature extractor in directions orthogonal to subspace of old inputs via gradient projection, aligning RePRE with incremental learning dynamics. Our simple yet effective design allows NSGP-RePRE to achieve state-of-the-art performance on the Pascal VOC and MS COCO datasets under various settings. Our work not only advances IOD methodology but also provide pivotal insights for catastrophic forgetting mitigation in IOD. Code will be available soon.

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Qirui Wu Shizhou Zhang'
De Cheng?
Yinghui Xing'
Di Xu²
Peng Wang' Yanning Zhang'
Abstract
detection (Ren al., 2016; Khanam & Hussain, 2024; Li et al., 2023). Traditional methods mostly solve the object detection task under a static closed- setting, where all to-be to-be-detected object classes and annotations are fully available before training. Nevertheless, real-world applica- tions frequently encompass dynamic environments where new object categories appear progressively over time. De- tectors should possess the capability to adjust to novel tasks through sequential learning, while simultaneously preserv- ing the knowledge gained from detecting previous classes.
Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD), Most existing methods treat the detector monolithically, rely- ing on instance replay or knowledge distillation without analyzing component-spect Through dissection of Faster R-CNN, we reveal key insight: Catastrophic forgetring is predomi- nantly localized to the Rol Head classifier, while regressors retain robustness across incremental stages. This finding challenges conventional as- sumptions. motivating us to develop framework termed NSGP-
forgetting.
Conventional object detectors (Ren et al., 2016; Carion et al., 2020; Li et al., 2020) often suffer from catastrophic forget- ting during incremental learning, which significantly ham- pers their performance previously learned classes when new tasks are introduced. Unlike incremental learning in classification tasks, incremental object detection (IOD) is more challenging than classification it requires the si- multaneous classification and location of a set of objects in the image. To obtaín an incremental object detector with excellent performance, many research efforts have been de- voted by introducing knowledge distillation or data replay techniques (Cermelli et al., 2022; Yuyang et al., 2023; Mo et al., 2024) into popular detection frameworks.
Regional Prototype Re- play (RePRE) mitigates classifier forgetting via replay of two types of prototypes: coarse pro- totypes represent class-wise semantic centers of Rol features, while finegrrined prototypes model intra-class variations. Null Space Gradient Pro- jection (NSGP) further introduced to eliminate pretotype-feature misalignment by updating the feature extractor in directions orthogonal sub- space of old inputs via gradient projection, align- ing RePRE with incremental learning dynamics. Our simple yet effective design allows NSGP- RePRE to achieve state-of-the-an performance on the Pascal VOC and MS COCO datasets under various settings. Our work not only advances IOD methodology but also provide pivotal insights for catastrophic forgetting mitigation in IOD. Code is available at RePRE.
Current research in the IOD field usually treats the detector as a whole, and few works pay attention to whether catas- trophic forgetting mainly comes from a certain component or whether all modules contribute roughly the same. Demys- tifying catastrophic forgetting in a sophisticated detector necessary and helpful not only establish bridge between incremental learning in classification and object detection, but also to provide principled guidance for designing sim- pler and more effective IOD methods. In this study, we chose the widely adopted two-stage Faster R-CNN (Ren et al., 2016) detector as pioneering research object.
1. Introduction
As one of the most fundamental tasks in computer vision, significant progress has been made in the field of object
Faster R-CNN is composed of backbone, neck, region pro- posal network (RPN) and region of interest head (Rol Head), each of which is crucial to the detector's performance. Our primary focus is on the RPN and Rol Head, known for their key roles in object detection. Through systematic analy- sis of its core components, we uncover several important insights. 1) RPN's recall ability remains consistent when transitioning to new tasks. 2) RPN's forgetting has an in- significant impact on overall performance. 3) Forgetting
'School of Computer Science, Northwestern Polytechnical Un- viersity, Xi' an, China "Scoool of Telecoommuniatooon and Engineer- ing, Xidian University, Xi' an, China Huawei Technologies Ltd. Correspondence to: Shizhou Zhang <szzhang@@un @wwwewwwwwe.m. www.ppp..ducc...... De Cheng <dcheng @ xwdiii.edu.cc.m edu.cn..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
mainly occurs in the Rol Head's classifier, while the re- gressor component efficiently retains its knowledge. These findings challenge conventional assumptions and inspire us to propose novel simple yet effective IOD framework.
2.1. Incremental Learning for Classification
Most influential incremental learning studies have focused classification tasks. Some regularization-batt based methods enforee the stability of logits (Li & Hoiem, 2017: Zhang et al., 2023; Yan et al., 2025) or intermediate features (Simon et al., 2021) to preserve the learned knowledge, while others apply restrictions on the weight of the model (Kirkpatrick et al., 2017) or on gradients during optimization (Lopez-Paz & Ranzato, 2017; Wang et al., 2021). Structure- based meth- ods are dedicated learning specific parameters for differ- ent tasks, with dynamically expanding architecture (Rusu et al., 2016) or grouped parameters in a static model (Fer- nando et al., 2017). For replay-based methods, they can be divided into experience replay (Buzzega et al., 2020; Zhu et al., 2021; Kong et al., 2023) and generative replay meth- ods (Zhai et al., 2020; Kemker & Kanan, 2017), depending the examples stored in buffer or generated with a model. Recently, incremental learning based foundation mod- els such as CLIP (Radford et al.. 2021) has also attracted attention. Research works such as L2P (Wang et al., 2022), O-LoRA (Wang et al., 2023a), and VPT-NSP" (Lu et al.) at- tempt to learn continuously based on the parameter-efficient transfer learning technique (Zhou et al., 2022; Jia et al., 2022; Xing et al., 2023; Zhang et al., 2025) have achieved superior performance.
In this paper, we propose NSGP-RePRES which is composed of two components: Regional Prototype REplay (RePRE) and Null Space Gradient Projection (NSGP). To address catastrophic forgetting in the Rol Head classifier, RePRE alleviates forgetting by replaying stored regional prototypes, including coarse regional prototypes and fine-grained re- gional prototypes of each class. Coarse prototypes act as stable semantic centers, representing the core structure of the Rol feature space. Fine-grained prototypes complement these as a semantic augmentation by capturing intra-class di- versity, ensuring a more holistic modeling of the feature dis- tribution. By jointly leveraging these components, RePRE strengthens the capacity of the Rol Head classifier to retain learned knowledge across tasks while accommodating new knowledge, significantly improving incremental learning performance. To prevent toxic replay with misaligned pro- totypes due to the drift of Rol features caused by updates to the feature extractor, we introduce NSGP to manipulate the gradient in the feature extractor. By projecting gradients into the null space of previous examples, Rol feature distortion is greatly minimized, ensuring prototype and Rol feature alignment. Our approach achieves state-of-the-art results on the PASCAL VOC and COCO datasets under various single and multi-step settings.
2.2. Incremental Learning for Object Detection
Incremental object detection presents unique challenges compared with the classification task. IOD are required to locate and classify the visual objects in images. It also faces distinctive missing annotation problem where potential instances not belonging to the classes of the current learning stage are labeled as background. Most existing IOD works can be summarized into two categories. One knowledge distillation based methods (Mo et al., 2024; Cermelli et al., 2022). BPF (Mo et al., 2024) bridges past and future with pseudo-labeling and potential object estimation to align models across stages, ensuring a consistent optimization direction. MMA (Cermelli et al., 2022) consolidates the background and all old classes into one entity to minimize the conflict between optimization objects between previ- ous and current tasks. The other is to preserve knowledge through a replay of previous data stored in images (Liu et al., 2023), instances (Yuyang et al., 2023), or features (Acharya et al., 2020). ABR (Yuyang et al., 2023) replayed fore- ground objects from previous tasks stored in a buffer to reinforce the learned knowledge. RODEO (Acharya et al., 2020) stored compressed representations in a fixed-capacity memory buffer to incrementally perform object detection in streaming fashion. Unlike existing methods, delve into analyzing where the forgetting originated for the two-stage incremental object detector. Then we tailor method specif- ically designed to combat the crux forgetting module, i.e.
Our main contributions are three-fold:
We comprehensively studied the key components of Faster R-CNN and identified Rol Head classifier as the primary cause of catastrophic forgetting, providing principled guidance for IOD method design.
Based on our finding, we propose NSGP-RePRE. to alleviate forgetting of Rol Head classifier by Regional Prototype Replay complemented with Null Space Gra- dient Projection for Rol feature anti-driftin.
Our method not only achieves -fate-o--the-an perfor- mance across multiple datasets under various single and multi-step settings, but also provides pivotal in- sights for mitigating forgetting in IOD.
2. Related Work
Incremental learning, or continual learning, progressively learn new knowledge while retaining previous information. It is categorized into task-incremental, class- maassinncrem,nal,, and
challenges. The most challenging class-incremental learning the primary focus of this paper.
2

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
135
Corrert
10 04
06 2.4
90 0.0
02 04
(a) RPN's recall on Diest.
(b) RPN's recall on Diest (c) RPN's recall on Di (d) RPN's recall on Diast
Figure 1: Rbeaal-Objeetness curve of RPN's prediction. IoU threshold is set to 0.5. Blue: Mj has been trained with training
images of Dj in earlier stages. Green: M, is just fine-tuned on D, Red: Mj has not seen the training set of D; before. Gray: M joint is trained jointly on all training images of DD
RoI Head classifier, by replaying Rol features from previ- ously seen tasks to preserve the classification performance.
question driving this work is: Which component predomi- nantly leads to forgetttni or do all components have a contributing role? To systematically address this, we decompose the ultimate question into three interconnected sub-questions: 1. Can RPN retain its recall ability in incre- mental learning? RPN functions an initial object localizer and its recall rate plays a critical role in the overall perfor- mance of the detector. 2. How much does RPN's forgetting affect the final performance of the detector? RPN doesn't t produce final predictions on classification nor localization, it is crucial to investigate the actual impact caused by its degradation. 3. Which branch of the Rol Head predomi- nantly accounts for forgetting? The Rol Head is responsible for the ultimate prediction of the detector, its dual role in classification and modifying the bounding box potentially makes it sensitive to task-specific changes. To answer these questions, we conduct a series of analytical experiments in the following section from statistical perspective, as the detector learns sequentially.
3. Anatomy of Faster R-CNN
3.1. Preliminary
Problem Formulation of Incremental Object Detection. In Incremental Object Detection, training is structured across n sequential learning stages, with each stage in- corporating a new set of classes to be detected. Let C C= CE..C,,...CC Ct....,C,, C,. represent the entire class set that the detector M incrementally acquires, with C, = 0 for all * The dataset D.- = (2, comprises images and annotations for the 1-th learning stage. Each image in X2 could feature multiple objects of various classes from C, though only those in C, are annotated. The main challenge in IOD is to update the detector from M to Mr using D, solely, without access to earlier datasets (Di,..., : D.-1), while preserving or enhancing the detector's performance on previously learned classes [CI,...,C,-]]
Our analytical experiments are conducted on the PASCAL VOC dataset, starting with five classes and incrementally adding five classes across three additional stages. We em- ploy pseudo-labeling as a basic strategy to mitigate the missing annotation issue and top 1,000 proposals are se- lected to provide sufficient number for our investigation. In the following sections, we evaluate the RPN and Rol Head within the detectors learned on all stages and jointly trained detector, i.e. Mi to M1 and M Mjoints on the test set of each learning stage, i.e. Djest to Dieet..
Faster R-CNN Architecture. Our study utilizes the two- stage object detector Faster R-CNN, which involves four primary components: a backbone network fo. neck fn. a Region Proposal Network (RPN) SRPN, and a Region of Interest (Rol) Head fROI- The backbone and neck modules are responsible for feature extraction, and their combination represented as frs = fn o fo. The RPN generates object proposal boxes accompanied by objectness scores, which express the likelihood of each box containing a target object. Following this, proposals with higher objectness scores P are chosen for Rol feature extraction using Rol Align. The Rol Head is divided into two branches: the classification branch Jas and the regression branch fitos- The obtained Rol features P are fed into these branches to classify and adjust the positions of the bounding boxes.
To clarify, we use various colors to depict the performance of Model MM, on test set of Drest across different training stages. Green indicates M, when just fine-tuned in its corresponding stage with the training set of D; (1 i). displaying peak performance represents M, that has encountered training set of DD in earlier stages (t > i), demonstrating the phenomenon of forgetting after multiple training stages. Red illustrates M, that has not been fed with training images of D, before (t < i). highlighting the model's 's generalization capability.
test set of Djest, Blue
3.2. Rationale for Anatomy
When adapting Faster R-CNN to sequential learning tasks, catastrophic forgetting the primary limitation. The central
3

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Eval
Eval Aee
Eval
Eval
als generated by M, As depicted in Figure 2 (a), M1 is evaluated on test set of Diest with varying sets of propos- als. Pi demonstrates the optimal performance of Mi since P1 is generated with the RPN of M1. i.e. zero forgetting, Although P2 to P4 exhibit some forgetting compared to PI. the performance deterioration of M with P2 to Pa is minimal, with only a 1.3% reduction in mAP observed on Di, from P1 (77.4%) to P1 (76.1%). This minor decrease suggests that RPN's forgetting has minor impact on the de- tector's 's final performance. Consistent conclusion can be obtained from Figure 2 (b) and (c).
26-
P4
A
Pa
P5
A Propsoal
Troposal
Prosooal
Propocal
(d) On Diest Figure 2: Results of M, on Dj with different proposals. Pj
(a) On Drest (b) On Drest (c) On Djest
are produced by corresponding M,.
Eval with fixed Pl
Eval with fixed P2
Eval with fixed P3
80
80
MAP (96)
80
mAP
60
70
75
The Rol Head classifier exhibits severe catastrophic for- getting. As discussed previously. RPN contributes mini- mally to the detector's forgetting. To investigate the crux of forgetting, we fixed the proposals P, generated with M, and fed them into Rol Heads of detectors in subsequent stages M,+1 to Mn- By designating the classification re- sults of each proposal with M,'s 's results, we can isolate the forgetting caused by the regression branch and the classi- fication branch. As shown in Figure 3 (a), the dashed line represents the mAP of models designated with the M1 's classification results, while the solid line represents the clas- sification results produced by the corresponding models the x-axis. In Figure 3 (a), the dashed line remains al- most unchanged, suggesting the forgetting caused by the regressor is minor. The solid line deteriorates rapidly as more stages have been trained on the detector, indicating that the classification head primarily causes the forgetting. The same trend in Figure 3 (b) and (c) further confirms the conclusion.
40
AM MM Abb Mr
.M43 .A5 Mr
Abb Ads Model
Model
Model
(a) On Diest
(b) On D2
(c) On D
Figure 3: Results of .M, on various D. by using a fixed set of proposals. " indicates the classification results of each proposal is designated by Model freshly trained on the corresponding D. "_" indicates the predicted classification results for the corresponding model in the x-axis.
3.3. Anatomy of Faster R-CNN
RPN's recall ability remains consistent across sequen- tial tasks. RPN allows the detector to generate possible Rols and deterioration in proposal quality will hamper the detector's final performance. It is essential to examine the RPN's recall rate from statistical view, as it can reflect the knowledge-prie pereeergggggeiiggggggg. ability of RPN. We perform experi- ments on the RPNs within the detectors learned on all the four stages and the jointly learned detector, i.e. M1 to MM and Mjontt by using Recall- Objjcttss curves on all the four test sets of each training stage Dieat to Dieat, Note that the threshold of IoU between the proposals and GTs is set to 0.5. As shown in Figure 1 (a), the blue curves represent the recall ability of RPNs within M2 to M4 which have been previously tuned on training set of Dj. The green curve shows the recall of M₁. which has just been fine- tuned on Dj. It can be clearly seen that the blue curves show a slight reduction compared the green curve. As the objectness score approaches O, the recall rates of various models improve to similar outcomes close to 100%. (Note that only top 1,000 proposals are selected in our experi- ments.) The slight reduction between the blue and green curves highlights that the RPN experiences little forgetting after multiple sequential learning stages. This trend is also observed in Figure 1 (b) and (c).
Interestingly, our findings also reveal that RPN effectively generalizes to previously unseen classes as can be seen from the red curves shown in Figure 2 and Figure 3. More detailed analysis are presented in the appendix. We note that the reason for minimal forgetting in regression and large forgetting in classifier is unclear. We infer that may be due to the absence of task conflicts in the detector regression branches, while it is severe in the classification task (Wang et al., 2023b; Huang et al., 2024) for incremental learning.
3.4. Key Findings
Through statistical evaluation and systematic analysis, we demonstrate three key findings:
RPN Recall Stability: sequential tasks, the stability of the RPN recall ability is largely maintained.
RPN's minimal forgetting negligibly affects overall per- formance. To assess the actual impact of RPN's forgetting on the detector's 's final performance, we adopt proposals gen- erated from models of subsequent training stages (M++1 to M2) to the current model on the current stage M,, to test the final performance of M; P, denotes the propos-
RPN's Impact on Performance: RPN's minimal forget- ting has a negligible impact on overall performance.
Rol Head Classifier Vulnerability: The Rol Head classi- fier suffers severely from catastrophic forgetting, while the regressor can efficiently retain its knowledge.
4

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Prototype Generation
Recs
Rol Head
Hypeepphoees
Coorso Rogirnd Procaypo
Cls
Aire Cas
Rol fcolures
Plame
Finegpireed Reogh Perotyyp
Oy O-11
NSGP
RCs
Bird
Feature
RPN
Reg
Labbxx
Extractor
Ae dallaset clos
D2
(a) ASGP-RePRE
(b) Prototype Generation
Figure 4: The overall architecture of our NSGP-RePRE framework. This framework incorporates RePRE to mitigate forgetting within the Rol Head's classifier. NSGP is introduced to counteract the shifts induced by the evolving feature extractor.
Our analysis reveals that Catastrophic forgetting in Faster R-CM R-CNN stems predominantly from the instability of the Rol Head's 's classifier, rather than degradation in RPN's s recall capability or the regression branch of Rol Head. Our analy- ses demonstrate minimal forgetting in regression, building a bridge between classical incremental classification and two-stage incremental object detector. This offers funda- mental insights for developing simpler and more efficient IOD methods. Consequently, we present straightforward and effective approach to address forgetting in the Rol Head classifier, thereby reducing forgetting in the detector.
4.1. RePRE
RePRE retains the previously learned classification knowl- edge by replaying regional prototypes from the past. Specif- ically, to obtain coarse regional prototypes for the RePRE in the next training stage + 1, we extract Rol features O = {of E N. c E N. 1 S i I ne No-1 S S No} from the feature maps as
of RoLAlign fub (xx)),
(I)
where P are the proposals covering class c, xf is the image containing objects of and ne is the number of proposals that cover object from class N₂ represents the total class number of Cotd = {C1, ... ,,,,,,,,,,,,,, The generation of P. can be expressed as
4. Method
C.}.
Overview of Framework. Earlier discussions have pin- pointed that the crux of catastrophic forgetting in Faster R-CNN mainly stems from the classification branch of the Rol Head, establishing a bridge between incremental clas- sification and incremental object detection. Based on our previous analytical results, we propose a simple yet effec- tive Regional Prototype Replay (RePRF) incorporated with Null Space Gradient Projection (NSGP) framework termed NSGP-RePRE specifically targeting the forgetting in Rol Head classifier.
p = Srpp (fn)(x()).
(2)
Next, we compute and store a single prototype for each class given by
Hc
o
(3)
nh
The resulting p4c will be appended to R.-1 {u& l: N, $ h I N.-1} to form Rr, which stores prototypes from past stages.
As depicted in Figure 4, NSGP-RePRE employs NSGP for regulating the backbone and neck, while RePRE manages the Rol Head. RcPRE creates coarse regional prototypes from Rol features of each class, along with fine-gre mmeigra re- gional prototypes to enhance semantic diversity. These pro- totypes are replayed via the Rol Head's 's classification branch. Unlike previous works (Yuyang et al., 2023; Mo et al., 2024), RePRE provides consistent guidance with minimal proto- type storage per class to prevent forgetting specifically on RoI Head classifier. Addressing the issue of prototype- feature misalignment identified in prior research (Yu al., 2020; Gomez-Villy Villa al., 2025), NSGP is introduced to restrict changes in Rol features, ensuring prototype's 's align- ment with the evolving Rol feature distributions.
To capture the entire spectrum of useful information on the distribution of Rol features. We introduce complementary fine-grained regional prototypes chosen through a density- aware prototype selection strategy. Specifically, we first cal- culate the cosine similarity between Rol features extracted via Rol Align:
To O
1
S S n.
(4)
For each Rol feature. we define a hypersphere with ra- dius r, centered at of. that contains neighboring features S. = {of st, > r, S i S ne)- The importance of hypersphere quantified by its cardinality (i.e. the
5

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
number of Rol features it contains). To ensure diversity and avoid redundancy, we greedily select the top-K hy- perspheres {S; | 1 S j S }} in descending order of importance. During selection, any candidate hypersphere whose center lies within the radius r of a previously selected (more important) hypersphere is excluded. The fine-grained regional prototypes are computed by averaging all features in their corresponding hypersphere:
the null space of the previous samples (Wang et al., 2021), to obtain A W. This projection ensures that AW remains orthogonal to the inputs of the old tasks X. Consequently, the update can be formulated as
W = W - W
in the time step 1, where a is learning rate. The orthogonality condition between X' and A WW ensures
Poa =
Σ oES;
o,
(5)
X(W, - aá ,,, =,W,
(10)
5
is satisfied, effectively preventing drifts in the feature extrac- tor's updates. We adjusted the projection matrix in NSGP to enhance its compatibility with Faster R-CNN. Additional details can be found in the Appendix.
and these prototypes are added to the fine-grained prototype buffer R1-1= = (44 k,j E N, 1 S k S N is K}.
To replay these prototypes, at stage + 1, a regional proto- type μg is fed into the classification branch of the Rol Head to predict the class probabilities
In general, the NSGP will control the G of the backbone and neck, ensuring that they are projected into the null space corresponding to input from previous examples. This approach stabilizes the Rol features, thus improving not only the classification accuracy but also the minimal forgetting in regression.
y& = (oftmax(fas(Hxx)))
(6)
The replay loss Cre is computed as
K Σ
5. Experiments
=re=- -
Σ y2ECold
Yk Tog - Σ Y4ECold 1=1
yu log (7)
5.1. Experimental Settings
Datasets and Evaluation Metries. Following the same protocols as in previous works (Yuyang et al., 2023; Mo et al., 2024), we evaluate method on the PASCAL VOC 2007 (Everingham et al., 2010) and MS COCO 2017 (Lin et al., 2014) datasets. PASCAL VOC 2007 contains 20 different classes, including 9,963 annotated images. MS COCO 2017 dataset comprises 80 classes, with around 1 118k images for training and 5,000 images for validation. The mean average precision at the 0.5 IoU threshold (mAP@0.5) is used as the primary evaluation metric for VOC dataset, and the mean average precision ranging from 0.5 to 0.95 is the main evaluation metric for the COCO dataset. For each incremental setting (A-B), the first number A denotes the number of classes in the first task, while the second number B represents the number of classes in the subsequent tasks.
where yι represents the ground-truth label associated with the coarse prototype pa and fine-grained prototype Mk,-- The overall loss function for the detector is then formulated as:
L = Lets + Liboa + Lre:
(8)
where Lets and Laboz correspond to the classification and bounding box regression losses for the current stage t.
4.2. NSGP for Rol Features Anti-drifting
When using regional prototype replays that stabilize the Rol Head, the continuously updating feature extractor may cause the features of previous classes to drift. The drift results in misalignment between the stored prototypes and the Rol features in the current training stage, which will hamper the model to retain its knowledge. To reduce distortions in the Rol feature space during learning new tasks, we introduce a Null-Space Gradient Projection (NSGP) strategy to prevent updating of the backbone and neck from interfering with the features of previously seen tasks. RePRE and NSGP work together form an exquisite incremental object detector, with RePRE managing the Rol Head and NSGP governing the backbone and neck.
Implementation Details. works (Yuyang al., 2023; Mo et al., 2024), we build our incremental Faster R-CNN (Ren et al., 2016) with R50 (He et al., 2016). In our method, we incorporate a pseudo-labeling strategy to solve the missing annotation problem as in BPF (Mo et al., 2024). More implementation details can be found in the appendix.
Similar
previous
5.2. Quantitative Evaluation
Denote the parameters of the Convolution/FC layer in the backbone and neck as W, and the gradient G is calculated by the backward pass. To ensure that updating based on G will not change previous tasks, NSGP projects into
Following previous works (Yuyang et al., 2023; Mo et al., 2024), our method is evaluated on various settings includ- ing single-step and multi-step increments. We compare
6

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Table I: m. results on single incremental step on PASCAL VOC 2007. The best performance in each is presented with bold, and the second best is presented with underline.
19-1
15-5
10-10
5-15
Method
1-19
20
1-20
Avg
1-15
16-20
1-20
Avg
1-10
11-20
1-20
Avg
1-5
5-15
1-20
Avg
Joint
76.4
76.4
76.4
76.4
78.3
70.7
76.4
74.5
76.9
76.0
76.4
76.4
73.6
77.4
76.4
75.5
Fine-tuning
12.0
62.8
14.5
37.4
14.2
59.2
25.4
36.7
9.5
62.5
36.0
36.0
6.9
63.1
49.1
35.0
ORE (Joseph et al., 2021a)
69.4
60.1
68.9
64.7
71.8
58.7
68.5
65.2
60.4
68.8
64.6
64.6
OW-DETR (Gupta al., 2022)
70.2
62.0
69.8
66.1
72.2
59.8
69.1
66.0
63.5
67.9
65.7
65.7
ILOD ILtD (Joseph et al., b)
70.9
57.6
70.2
64.2
71.7
55,9
67.8
63.8
68.4
64.3
66.3
66.3
ABR (Yuyang al., 2023)
69.7
70.9
70.4
73.0
65.1
71.0
69.1
71.2
72.8
72.0
72.0
64.7
71.0
69.4
67.9
FasterlLOO (Ren
2016)
68.9
61.1
68.5
65.0
71.6
56.9
67.9
64.3
69.8
54.5
62.1
62.1
62.0
37.1
43.3
49.6
PPAS (Zhou et ul., 2020)
70.5
53.0
69.2
61.8
63.5
60.0
61.8
61.8
(Yang 2022)
70.2
60.6
69.7
65.4
69.4
57.9
66.5
63.7
66.2
66.0
66.1
66.1
PROB (Zohar et al,, 2023)
73.9
48.5
72.6
61.5
73.5
60.8
70.1
67.0
66.0
67.2
66.5
66.5
PseudoRM (Yang et al., 2023)
72.9
67.3
72.6
70.1
73.4
60.9
70,3
66.9
69.1
68.6
68.9
68.9
MMA (Cermelli al., 2022)
71.1
63.4
70.7
73.0
60.5
69.9
66.7
69.3
66.6
66.6
66.88
57.2
59.6
BPF (Mo et 2024)
74.5
65,3
74.1
69.9
75.9
63.0
72.7
69.5
71.7
74.0
72.9
72.9
66.4
75.3
73.0
70.9
NSGP-RePRE
76.3
69.0
76.0
72.7
77.5
61.8
73.6
69.7
75.3
72.7
74.0
74.0
68.5
74.5
73.0
71.5
Table 2: mAP@0.3 0.5 results on multiple incremental steps on PASCAL VOC 2007. The best performance in each is presented with bold, and the second best is presented with underline.
10-5(3tasks)
5-5(4tasks)
10-2(6tasks)
15-1(6tasks)
10-I(1ttasks)
Method
1-10
11-20
1-20
1-5
6-20
1-20
1-10
11-20
1-20
1-15
16-20
1-20
1-10
11-20
1-20
Joint
76.9
76.0
76.4
73.6
77.4
76.4
76.9
76.0
76.4
78.3
70.7
76.4
76.9
76.0
76.4
Fine-tuning
5.3
30.6
18.0
0.5
18.3
13.8
3.8
13.6
8.7
0.0
10.5
5.3
0.0
5.1
2.6
ABR (Yayang et al., 2023)
68.7
67.1
67.9
64.7
56.4
58.4
67.0
58.1
62.6
68.7
56.7
65.7
62.0
55.7
58.9
FasterlLOD (Ren et al., 2016)
68.3
57,9
63.1
55.7
16.0
25.9
64.2
48.6
56.4
66.9
44.5
61.3
52.9
41.5
47.2
MMA (Cermelli al. 2022)
66.7
61.8
64.2
62.3
31.2
38.9
65.0
53.1
59.1
68.3
54.3
64.1
59.2
48.3
53.8
BPF (Mo 2024)
69.1
68.2
68.7
60.6
63.1
62.5
68.7
56.3
62.5
71.5
53.1
66.9
62.2
48.3
55.2
NSGP-RePRE
72.4
67.6
70.0
64.6
66.1
65.7
70.1
58.8
64.4
77.7
55.0
72.0
69.9
55.1
62.5
Table 3: mAP results on MS COCO 2017 at different IoU. The best performance in each is presented with bold, and the second best is presented with underline.
a multi-step incremental task setting, covering settings such as 10-5, 5-5, 10-2, 15-1, and 10-1.
Single-step Increments. In Table 1, we make a comparison between our proposed method and existing approaches. Our method frequently surpasses others in range of settings, particularly in the base classes in the initial learning stage, demonstrating its superior ability to mitigate catastrophic forgetting. Specifically, NSGP-RePPR exceeds the previous leading replay- approach ABR, by an average of 4.4% in the initial class set. It also exceeds the previous SOTA method BPF by 2.3% in the initial class set, bolstering our assertion regarding the superior forgeetigg capability of our approach. JSGP-RePREE exceeds ABR by 3.3% and BPF by 1% in all 20 classes, underscoring the effectiveness of our method. The Avg metric equally average base and new classes mAP, showing stability and plasticity balance without the influence of the number of classes. In Avg, our method surpasses ABR and BPF by 2.1 % and 1.2%, respec- tively. demonstrating that our method prevails in balance between stability and plasticity in all methods.
Method
40-40
70-10
AP
AP50
AP75
AP
AP50
AP75
Joint
36.7
57.8
39.8
36.7
57.8
39.8
Fine-tuning
31.2
Meta (Joseph et al., 2021b)
23.8
40.5
24.4
ABR (Yuyang al., 2023)
34.1
57.8
35.2
31.1
52.9
32.7
FasterILOD (Ren 2016)
20.6
40.1
21.3
39.9
PreudoRM (Yang al., 2023)
25.3
44.4
MMA (Cermelli al., 2022)
33.0
56.6
34.6
30.2
52.1
31.5
BPF (Mo et al., 2024)
34.4
54.3
37.3
36.2
56.8
38.9
NSGi-RePRE
35.4
55.3
38.6
36.5
56.0
39.8
our method against two basclines: Joint Training, which involves training the model on the complete dataset using all annotations, and Fine-Tunning where the model is in- crementally trained on new data without any regularization strategy or data replay.
5.2.1. 1. PASCAL VOC 2007
Multi-step Increments. The issue of catastrophic forgetting becomes more challenging in longer incremental settings. As demonstrated in Table 2, fine-tuning nearly completely
On the PASCAL VOC 2007 dataset, we assess our ap- proaches using a single-step incremental task setting, which
includes 19-1, 15-5, 10-10, and 5-15 tasks. We also examine
7

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Table 4: Ablation study on each component. Where "Coarse" indicates coarse prototypes replay only, "Fine" indicates fine-grained regional prototypes are also incorporated.
mAP Di with Fixed P1
75
70
VOC(5-5)
NSGP
Course
Fine
Model
1.5
11-15
16-20
1-20
(n)
46.6
56.5
71.1
59.6
58.4
m000
(b)
62.3
60.4
73.1
57.4
63.3
60
49.8
61.0
73.5
60.5
61.2
(d)
65.9
61.6
73.8
56.0
64.3
NSGP-RePRE SEEG--RePE--ixeeee
(c)
64.6
66.2
73.1
59.0
65.7
Deseline+ fixed ds
50
Baseline
My
Mr
Mx
Mr
Model
forgets the initial classes. NSGP-RePRE shows 4.7% improvement over ABR in initial classes across all 5 set- tings, and 4.2% improvement in all 1-20 classes. Our method exceeds the performance of BPF by 4.5% in the base classes and 2.3% in the 1-20 classes. In the particularly demanding 10-1 settings, our method is 3.6% better than ABR, highlighting the efficacy of our proposed approaches. The improvements observed in more complex multi-step step increment settings further validate the effectiveness of our proposed methods.
Figure 5: mAP of different model on Diast in VOC(5-5) settings. To better demonstrate the impact of our method on the classifier, P1 is fixed to all models. Fixed cls indicates the models classification results is designated by Mi.
achieves the highest performance among all models, ex- ceeding d by 1.4% in the 1-20 division, underscoring the effectiveness of our method. As shown in Table 4, each adopted component independently reduces forgetting and reaches peak performance when used together.
5.2.2. MS COCO 2017
On MS COCO 2017 dataset, we performed experiments on 40-40 and 70-10 settings using the same protocol in com- parison methods. As shown in Table 3, fine-tuning suffers from catastrophic forgetting in both settings. While previous approaches have been enhanced with fine-tuning, NSGP- RePRE increased the average AP by 1.0% over the previous state-of-te of-thheart in the 40-40 configuration. In the 70-10 10 scenario, the performance is close to that of joint training, with our method yielding 0.3% improvements over the pre- vious SOTA BPF, These experimental results demonstrate the efficacy of our approach.
Anti-forgetting in Rol Head's classifier, As we intend to minimize the classification error caused by the forgetting in Rol Head's classifier, we demonstrate that our method can effectively solve the problem. As shown in Figure 5, we fixed set of proposals predicted by the M, as the proposals for all M. Fixed cls indicates that the model classification results are designated by MM. The baseline is the detector only applied with a pseudo-labeling strategy. Mi is the ideal upper bound in Diest as it is freshly trained D1. From Figure 5, we can draw some conclusions: 1. By com- paring the red curves, we can see that our method has a better classification performance, suggesting the effectiveness of our proposed method. 2. Suggested by the light blue area, NSGD can further reduce the already minimal forgetting in regression. Though the classifier specifically focuses reducing classification error, the extra components intro- duced by the method will not disrupt the observation that the regressor exhibits minimal forgetting.
5.3. Further Analysis
Effectiveness of Each Component. In Table 4, we analyze the effectivvness NSGP, Coarse, and Fine under the VOC 5-5 setting, where "Coarse" indicates that only coarse pro- totypes are adopted during replay while "Fine" shows the results incorporated with fine-grained regional prototypes. Variant a denotes our baseline model using pseudo-labeling. Variant b denotes that NSGP is employed to solve the feature drift based upon a, which significantly reduces the catas- trophic forgetting of old classes, thus markedly improving old class detection over a. Variant incorporates RePRE with coarse prototypes only to mitigate catastrophic forget- ting. However, performance remains suboptimal due to the feature shift from the updating of the feature extractor. The variant denotes our NSGP-RePRE with coarse prototypes only, which substantially reduces catastrophic forgetting and demonstrates the efficacy of the method. NSGP-RePRE.
6. Conclusion
This study investigates Faster R-CNN as the representa- tive two-stage incremental object detector and demonstrates that catastrophic forgetting primarily originates from the Rol Head's 's classifier while regressor exhibits minimal for- getting. The finding can provide principled guidelines for designing simple yet effective IOD method. Consequently, we introduce the NSGP-RePRE framework to mitigate for- getting the Rol Head classifier complemented with NSGP
8

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
on the feature extractor. Our extensive experimental results demonstrate the efficacy of the proposed methods. We hope that our research will offer significant insights into IOD, facilitating progress in this area.
continual representation learning via learnable drift com- pensation. In European Conference on Computer Vision, pp. 473-490. Springer, 2025.
Gupta, A., Narayan, S., Joseph, K.,, Khan, S., Khan, F. S., and Shah, M. Ow-detr: Open Open-world detection transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9235-9244, 2022.
Acknowledgements
This work was supported in part by the National Natural Sci- ence Foundation of China (NSFC) under Grant 62476223, 62176198, 62201467, the Key Research and Development Program of Shaanxi Province under Grant 2024GX- YBXM- 135, in part by the Young Talent Fund of Xi' an Association for Science and Technology under Grant 959202313088, Innovation Capability Support Program of Shaanxi (No.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Huang, W.-C., Chen, C.-F., and Hsu, H. Ovor: Oneprompt with virtual outlier regularization for rehearsal-free class- incremental learning. arXiv preprint arXiv:2402.04129, 2024.
2024ZC-KIXX-045)
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Jia, M., Tang, L.,, Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
References
Joseph, K., Khan, S., Khan, F. S., and Balasubramanian, V. N. Towards open world object detection. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5830-5840 2021a.
Acharya, M., Hayes, L., and Kanan, C. Rodeo: Replay for online object detection, arXiv preprint arXiv:2008.066 06439, 2020.
Joseph, K., Rajasegaran, J., Khan. S., Khan, F. S., and Balasubramanian, V. N. Incremental object detection via meta- meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12);900992116 2021b.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn- ing: strong, simple baseline. Advances in neural infor- mation processing systems, 01215920-15933 2020.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Kemker, R. and Kanan, C. Fearnet: Brain- iniin-inppieedee model for incremental learning. arXiv preprint arXiv: 1711. 10563, 2017.
Khanam, R. and Hussain, M. Yolov11: An overview of the key architectural enhancements, arXiv preprint arXiv:2410.17725, 2024.
Cermelli, F., Geraci, A., Fontanel, D., and Caputo, B. Model- ing missing annotations for incremental learning in object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3700- 3710, 2022.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-333) 2017.
Crabska-Barwinee
Everingham, M., Van Gool, L., Williams, C. K., Winn, J.,
and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Kong, J., Zong, Z., Zhou, T., and Shao, H. Condensed prototype replay for class incremental learning, arXiv preprint arXiv:2305.161 16143, 2023.
Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A. A., Pritzel, A., and Wierstra, D. Pathnet: Evolu- tion channels gradient descent in super neural networks. arXiv preprint arXiv: /0X.iv:7701.8800 701.08834, 2017.
Li, R., He, C., Li, S., Zhang, Y., and Zhang, L. Dynamask: dynamic mask selection for instance segmentation. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 11279-11288 11288 2023.
Gomez- Villa, A., Goswami, D., Wang. K., Bagdanov, A. D., Twardowski, B., and van de Weijer, J. Exemplar-free

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Li, X., Wang, W., Wu, L.,, Chen, S., Hu, X., J., Tang, J., and Yang, J. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33: 21 21002-21012, 2020.
Simon, C., Koniusz, P., and Harandi, M. On learning the geodesic path for incremental learning. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition. pp. 1591-1600, 2021.
Wang. S., Li, X., Sun, J., and Xu, Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 194-193, 2021.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Wang. X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. Orthogonal subspace learning for language model continual learning. arXiv preprint 23.. 14152, 2023a.
Wang, Y., Ma, Z.,, Huang. Z., Wang, Y., Su, 7., and Hong, X. Isolation and impartial aggregation: A paradigm of incremental learning without interference. In Proceed- ings of the AAAI Conference on Artificial Intellignnce, volume 37, pp. 10209-10217,202333
Liu, Y., Schiele, B., Vedaldi, A., and Rupprecht, C. Contin- ual detection transformer for incremental object detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23799-23808, 2023.
Wang. Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Lu, Y., Zhang, S., Cheng, D., Xing, Y., Wang, N., WANG, and Zhang, Y. Visual prompt tuning in null space for con- tinual learning. In The Tiihty-eighth Annual Conference on Neural Information Processing Systems.
Xing, Y., Wu, Q., Cheng, D., Zhang, S., Liang, G., Wang, P., and Zhang, Y. Dual modality prompt tuning for vision- language pre-trained model. IEEE Transactions on Mul- timedia, 2023.
Mo, Q.. Gao, Y., Fu, S., Yan, J., Wu, A., and Zheng, W.-S. Bridge past and future: Overcoming information asymme- try in incremental object detection. In European Confer- ence on Computer Vision, pp. 463-480. Springer, 2024.
Yan, Q. Yang, Y., Dai, Y., Zhang. X., Wiltos, K., Wožniak, M., Dong, W., and Zhang, Y. Clip-guided continual novel class discovery. Know-BBased Syst., 310(C), Febru- ary 2025, ISSN 0950-7051. doi: 10. 10.1016/j.knosys. 2024. 1024.1129200 URL https: k.knosys.2024 112920.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Yang, D., Zhou, Y., Zhang, A., Sun, X., Wu, D., Wang, W., and Ye, Q. Multi-view correlation distillation for incremental object detection. Pattern Recognition, 131: 108863, 2022.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): 137--149, 2016.
Yang. D., Zhou, Y., Hong, X., Zhang, A., Wei, X., Zeng, L., Qiao, Z., and Wang, W. Pseudo object replay and mining for incremental object detection. In Proceedings of the s ACM International Conference on Multimedia, pp. 153-162, 2023.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252, 2015.
Yu, L., Twardowski, B., Liu, X., Herranz, L.,, Wang, K., Cheng, Y., Jui, S., and Weijer, J. v. d. Semantic drift compensation for tlass-incrementee learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6982-6991, 2020.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint 1r0iv:1606.0 1601666661,,, 2016.
10

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Yuyang, L., Yang. C., Dipam, G., Xialei, L.., and van de Wei- jer, J. Augmented box replay: Overcoming foreground shift for incremental object detection. arXiv preprint arXiv:2307.12427, 2023.
Zhai, M., Chen, L., He, Nawhal, M., Tung, F., and Mori, G. Piggyback gan: Efficient lifelong learning for image conditioned generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 397-413, Springer, 2020.
Zhang, G., Wang, L., Kang. G., Chen, L., and Wei, Y. Slea: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19148-19158, 2023.
Zhang, S., Kong, D., Xing, Y., Lu, Y., Ran, L., Liang, G., Wang, H., and Zhang, Y. Frequency-guided spa- tial adaptation for camouflaged object detection. IEEE Transactions on Multimedia, 27:72-83, 2025. doi: 2.1109/TMM.2024.3300
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Learning to prompt for lision-language models. International Jour- nal of Computer Vision, 130(9))2337-2348. 2022.
Zhou, W., Chang, S., Sosa, N., Hamann, H., and Cox, D. Lifelong object detection. arXiv preprint arXiv:2009.01 129, 2020.
Zhu, F., Zhang, X.-Y,, Wang, C., Yin, F., and Liu, C.-L. Pro- totype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5871- 5880, 2021.
Zohar, O., Wang, K.-C., and Yeung. S. Prob: Probabilistic objectness for open world object detection. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11444-1 1453, 2023.

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
A. Implementation Details.
Similar to previous works, we use the Faster R-CNN architecture with a Resnet-50 (He et al., 2016) backbone pre-trained in ImageNet (Russakovsky al., 2015). On PASCAL VOC dataset, we train the network with SGD optimizer. momentum of 0.9 and weight decay of 10e-4. We use a learning rate of 0.02 for all tasks. For MS COCO, we adopt AdamW as the optimizer, weight deacy of 0.01 and learning rate of 5e-5. Batch size is set to 16 for both datasets. In NSGP, follow the adaptive selecting stategy proposed in (Lu et al.) keep the singular vaules. We sample 9 extra fine fine-grained prototypes to complement the coarse prototype, 10 prototypes are used in total. The radius r is set to 0,6. In our method, we incorporate a pseudo-labeling strategy to solve the foreground shift problem as implemented in BPF (Mo et al., 2024). All of our experiments were conducted on 2 RTX 3090 GPU.
B. Generalization on unseen classes of RPN.
Interestingly, findings reveal that RPN effectively generalizes to previously unseen classes. As depicted in Figure 1, the red lines represent the RPN's recall for objects belonging to unseen classes. Figure (d) illustrates how RPNs of MM to M3 successfully recall certain objects belonging to classes in the 4-th stage. It can be clearly seen that the recall ability of Mi to M3 on test set of Dinst can be consistently improved after sequential learning. similar trend is seen in Figure (b) and (c), suggesting RPN's potential to enhance zero-shot detection with sufficient training data. In Figure 2 (d), the red dots represent the outcomes of testing M4 on the test set of Dtstt employing proposals P1 to Ps. which were generated by models that have not encountered the classes within C4. Despite this, M4 still able identify unseen objects with high mAP, showcasing the impressive zero-shot ability of the RPN.
C. Is the Rol Head robust to low-quality proposals?
411366
35%
70
Distribution of Progosals
30%
315522
25%
50
(50
20%
34741
40
30
10%
114577
20
56685
10
5%
22119
7049
879
0
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Threshald of loU
Figure 6: Plot: Results of Mjoint after removing high-quality proposals with varying IoU threshold. Bar: The distribution of the proposals generated with Mjoint over IoU. The number the bar indicates the count of proposals.
robust Rol head capable of effectively offsetting RPN's forgetting. To evaluate the robustness of the RoI Head, we manually removed high-quality proposals during inference, i.e. high IoU with GTs, assess the mAP result of the detector. As shown in Figure 6, when removing the proposals with IoU above 0.7, comparable final results can still be obtained (74.5% to 76.4%). In particular, the detector still manages to detect some instances and achieves noticeable results when removing proposals with IoU above 0.5, showing the strong robustness of the Rol Head. The robustness of the Rol Head can be attributed to the training process, where the Rol Head is trained to refine coarse proposals which have a very broad loU range from a given value, 0.7 for example, to 1. The training with coarse proposals enables the Rol Head to refine rather low-quality proposals, leading robust performance of the Rol Head.
12

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
D. Null Space Gradient Projection Details.
We introduced NSGP to alleviate the Rol feature shift caused by the evolution of the feature extractor. crucial for the NSGP to obtain projection matrix that can project the gradient G into the null space of the old example X; {Ir i N, 1 S S Mr where Mr is the total number of inputs in the t-th training stages, An overview of NSGP are provided in Figure 7.
Forward
FC/Convl, n
Null Space
FC/ConvL n
Old Input
Projection
AW
Null Space
Figure 7: An overview of NSGP.
To obtain the projection matrix of an FC layer or a convolution layer with parameters W, we first compute the uncentered covariance of Xg Specifically, we can accumulate uncentered covariance matrix in t-th stage Ti as:
T N₂
Tt.
(11)
After obtaining the uncentered covariance in l. The uncentered covariance of all previous training stages can updated as
T= MI-17,11 + Mj M₁
T,.
(12)
Me
Here, Mr = Mi-1 + Mc. Then SVD is performed to obtain Uz, Ars (U+)* T as
U2, Ac, (U+) SV (T+-1)
(13)
Following (Lu et al.), we adaptively determine the nullity R and retain U/ correspond to R smallest diagonal singular vaules in A, Finally, the projection matrix for (t + 1)-th training stage obtained by
B U
(14)
and the gradient G' is projected to the null space of X; as
AW GB.
(15)
It is common practice in previous works (Wang et al., 2021; Lu et al.) normalize the B as
B
B'
(16)
|
Unlike previous works, we adopt B as normalized B' will decrease the update stride of the model, leading to a slow and difficult optimization. The slow learner is beneficial to the classification task, as shown in SLCA (Zhang et al., 2023), but not applicable to components in Faster R-CNN except backbone. Thus we only apply to the backbone, adopting B to the rest of the components in the detector. EWC (Kirkpatrick al., 2017) adopted regulate the update of parameterized normalization layers.
To justify our choice of only applying NSGP to the backbone and neck, we conduct experiments on all components of the detector, as show in Table 5. B or 13' indicates that the gradient of component, except the backbone, is projected by B or B'. Our experiments suggest that adopting NSGP different components leads to results without significant fluctuations, suggesting the detector is not sensitive to the NSGP. Comparing B and suggests that lower scale for the update stride in neck, RPN and Rol Head leads a significant decrease in performance. These results justify our choice of B instead of B'.
13

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Table 5: The experimental results of NSGP in different components with projection matrix B or B'. The dataset we adopted
VOC (5-5).
Backbone
+Neck
+RPN
+Rol Head
B'
62.6
60.3
59.0
41.9
B
62.6
63.3
63.0
63.2
E. Different strategy generating fine-grained prototypes.
To evaluate the effectiveness of the proposed fine-grained prototype generation process, we compared our method with clustering algorithms: K-Means and DBSCAN. To justify our choice of prototype instead of instances, we selected the center of the hypersphere instead of the averaging of the Rol features included in the hypersphere and named this method Instance. As shown in Table 6, our results outperform K-Means and DBSCAN by 0.7% on average, suggesting the effectiveness of the proposed method. Our prototypes surpass Instance by 1.4%, justifying the choice of the prototype instead of instances.
Table 6: Different strategy generating complementary prototypes of our method.
VOC(5-5)
Method
1-5
6-20
1-20
K-Means
62.7
65.6
64.9
DBSCAN
63.6
65.5
65.1
Instance
63.2
64.6
64.3
Ours 64.6
66.1
65.7
F. RePRE Performance with Coarse Regional Prototype Only.
Our RePRE can surpass previous works even with only coarse prototype being replayed. We name NeGR-RePRE incorporated with coarse prototype only as the SSGP-RePRE-Coaree.
PASCAL VOC Single-step Increments. In Table approaches. SSGP-RePRE- Coaase surpasses previous state-of-the-art BPF by 1.7% in base classes and by 0.7% in all 20 classes, underscoring the effectiveness of approach.
make comparison between our proposed method and existing
PASCAL VOC Multi-step Increments. The increases in initial classes indicate reduced forgetting with only coarse prototypes, while the improvements in 1-20 and the average reflect that our method achieves the optimal balance between stability and plasticity compared with previous methods. In Table 8, NSGG-
shows a 4.7% improvement over ABR in the initial classes in all settings and 8% improvement in the 1-20 classes. Our method exceeds the performance of BPF by 4.5% in the base classes and 2.3% in the 1-20 classes.
MS COCO Single Increments. In Table 9, CeEE----PRE-Caaa increased the average AP by 0.8% over the previous state-of-fe oftthh--tth-ttttt in the 40-40 configuration. In the 70-10 scenario, the performance is close to that of joint training, with our method yielding results comparable to the previous SOTA BPF. These experimental results demonstrate the efficacy of NSGP-RePRE-Coarse.
14

---

Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector
Table 7: m.P@0.5 results on single incremental step on PASCAL VOC 2007. The best performance in each is presented with bold, and the second best is presented with underline.
19-1
15-5
10-10
5-15
Method
1-19
20
1-20
Avg
1-15
16-20
1-20
Avg
1-10
11-20
1-20
Avg
1-5
5-15
1-20
Avg
Joint
76.4
76.4
76.4
76.4
78.3
70.7
76,4
74.5
76.9
76.0
76.4
76.4
73.6
76.4
75.5
Vine-tuning
12.0
62.8
14.5
37.4
14.2
59.2
25.4
36.7
9.5
62.5
36.0
36.0
6.9
63.1
49.1
35.0
ORE (Joseph al., 2021a)
69.4
60.1
68.9
64.7
71.8
58.7
68.5
65.2
60.4
68.8
64.6
64.6
OW-DETR (Gupta 2023)
69.8
59.8
ILtD-Meta (Joseph et al., b)
70.9
57.6
70.2
64.2
71.7
55,9
67.8
63.8
68.4
64.3
66.3
66.3
ABR (Yayang et al., 2023)
71.0
69.7
70.9
70.4
73.0
65.1
71.0
69.1
71.2
72.8
72.0
72.0
64.7
71.0
69.4
67.9
FasterLLOD (Ren al. 2016)
68.9
61.1
68.5
71.6
56.9
67.9
64.3
69.8
62.1
62.0
37.1
43.3
49.6
PPAS (Zhou
2020)
70.5
53.0
69.2
61.8
63.5
60.0
61.8
61.8
MVC (Yang et al., 2022)
70.2
60.6
69.7
65.4
69.4
57.9
66.5
63.7
66.2
66.0
66.1
66.1
PROB (Zohar et al.,
48.5
72.6
73.5
70.1
67.0
66.5
66.5
PscudoRM (Yang al., 2023)
72.9
67.3
72.6
70.1
73.4
60.9
70,3
66.9
69.1
68.6
68.9
68.9
MMA (Cermelli al., 2022)
71.1
63.4
70.7
67.2
73.0
60.5
69.9
66.7
69.3
63.9
66.6
66.6
66.8
57.2
59.6
62.0
BPF (Mo et al., 2024)
74.5
65.3
74.1
69.9
75.9
63.0
72.7
69.5
71.7
74.0
72.9
72.9
66.4
75.3
73.0
70.9
76.2
66.5
75.8
71.4
22.1
62.0
73.4
69.6
73.7
73.2
73.4
73.5
68.4
74.5
73.0
71.5
NSGP-RePRE
76.3
69.0
76.0
72.7
77.5
61.8
73.6
69.7
75.3
72.7
74.0
74.0
68.5
74.5
73.0
71.5
Table 8: mAP@0.5 results on multiple incremental steps on PASCAL VOC 2007. The best performance in each is presented with bold, and the second best presented with underline.
10-5(3tasks)
5-5(4tasks)
10-2(6tasks)
15-1(6tasks)
10-I(1Itasks)
Method
1-10
11-20
1-20
1-5
6-20
1-20
1-10
11-20
1-20
1-15
16-20
1-20
1-10
11-20
1-20
Joint
76.9
76.0
76.4
73.6
77.4
76.4
76.9
76.0
76.4
78.3
70.7
76.4
76.9
76.0
76.4
Fine-tuning
30.6
18.3
5.3
2.6
ABR (Yayang et al., 2023)
68.7
67.1
67.9
64.7
56.4
58.4
67.0
58.1
62.6
68.7
56.7
65.7
62.0
58.9
FasterlLOD (Ren al., 2016)
57.9
63.1
55.7
16.0
25.9
64.2
48.6
66.9
44.5
61.3
52.9
41.5
47.2
MMA (Cermelli 2022)
61.8
62.3
31.2
38.9
65.0
53.1
54.3
64.1
59.2
$3.8
BPF (Mo al. 2024)
69.1
68.2
68.7
60.6
63.1
62.5
68.7
56.3
62.5
71.5
53.1
66.9
62.2
48.3
55.2
ASGP-R<PRE-Come
71.9
66.2
69.1
65.9
63.8
64.3
68.7
54,8
61.8
77.0
53.9
71.2
71.2
50.6
60.9
RePP
72.4
67.6
70.0
64.6
66.1
65.7
70.1
58.8
64.4
77.7
55.0
72.0
69.9
55.1
62.5
Table 9: mAP results MS COCO 2017 at different loU. The best performance in each is presented with bold, and the second best is presented with underline.
Method
40-40
70-10
AP
AP50
AP75
AP
AP50
AP75
Joint
36.7
57.8
39.8
36.7
57.8
39.8
Fine-tuning
19.0
31.2
20.4
5.6
8.6
6.2
ILOD-Meta (Joseph et al., 2021b)
23.8
40.5
24.4
ABR (Yuyang et al., 2023)
34.5
57.8
35.2
31.1
52.9
32.7
FasterlLOD (Ren et al., 2016)
20.6
40.1
21.3
39.9
PseudoRM (Yang et al., 2023)
25.3
44.4
MMA (Cermelli et al., 2022)
33.0
56.6
34.6
30.2
52.1
31.5
BPF (Mo et al., 2024)
34.4
54.3
37.3
36.2
56.8
38.9
NSGP-RePRE-Coarse NSGP-RePRE
35.2
55.3
38.1
36.3
55.8
39.6
35.4
55.3
38.6
36.5
56.0
39.8
15


Reviews:
[{"review_id": "IZ5bAF6wBS", "reviewer": "Reviewer_2Xwn", "summary": "This paper addresses the challenge of catastrophic forgetting in incremental object detection, particularly in two-stage detectors like Faster R-CNN. The authors identify that catastrophic forgetting predominantly occurs in the RoI Head classifier, while the RPN and regression branches remain robust across incremental stages. Based on these findings, they propose NSGP-RePRE, a framework combining Regional Prototype Replay (RePRE) to mitigate classifier forgetting via coarse and fine-grained prototypes and Null Space Gradient Projection (NSGP) to counteract feature extractor drift by projecting gradients orthogonally to the subspace of old task inputs. This approach ensures alignment between prototypes and updated feature distributions. The experiments are conducted on the PASCAL VOC and MS COCO datasets under various incremental learning settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positvie feedback and insightful comment. The Questions and Responses are as follows.\n\n**Q1**: Using NSGP has been explored in the literature for continual learning.\n\n**R1**:Although NSGP has been explored in incremental classification, it is non-trivial to apply NSGP in IOD. The following table demonstrates the performance applying NSGP from Backbone to RoI Head accumulatively under VOC 5-5 setting.\n\n|          | Backbone | +FPN   | +RPN  |  +RoIHead | Ours |\n|----------|----------|--------|-------|-----------|------|\n| NSGP only| 62.6     | 63.3   | 63    | 63.2      | 65.7 |\n\nThe catastrophic forgetting in two-stage detectors is mainly caused by the severe classifier instability in the RoI Head. However, directly appling NSGP on these two-stage object detector (i.e., FPN, RPN,RoIHead)  shows limited performance improvements and can not well address the classifier instability issue, as illustrated in the above Table. \n\nInstead, the proposed Regional Prototype Replay (RePRE) module addresses  this issue by replaying coarse and fine-grained region prototypes in the RoI Head's classification branch. NSGP serves as an assistant component in our framework by mitigating semantic drift caused by parameter updates, thereby preventing toxic replay in RePRE. As shown in the table, our RePRE achieves a +2.4% gain compared with +FPN, underscoring the effectiveness of our proposed framework.\n\n**Q2**: Although the specific form of replying regional prototype has not been explored, the idea of prototype based continual learning has been proposed in the literature. Online Prototype Learning for Online Continual Learning, ICCV 2023 Prototype-Guided Memory Replay for Continual Learning, IEEE TNNLS 2024\n\n**R2**: While naively applying prototype-based methods on the classifier does achieves performance gain for the detector, our work makes distinct contributions in the context of IOD as we considered RoI Head's uniqueness in pre-processing MLPs before classifier. Further, we extend our method to fine-grained regional prototype replay to capture the distribution of regional object features, which is crucial for preserving old knowledge in continual learning.\n\nTo validate our design, we compare NSGP-RePRE against a baseline (\"Classifier\") that applies prototype replay only to the classifier (mimicking classification-focused approaches):\n\n|                  | 5-5  |      |      |\\|| 10-10 |       |      |\n|------------------|------|------|------|-------|-------|------|----|\n|                  | 1-5  | 6-20 | 1-20 |\\|| 1-10  | 11-20 | 1-20 |\n| w/o Prototype    | 62.3 | 63.6 | 63.3 |\\|| 71.8  | 73.2  | 72.5 |\n| Classifier       | 63.6 | 63.3 | 63.4 |\\|| 73.2  | 73.3  | 73.2 |\n| NSGP-RePRE       | 64.6 | 66.1 | 65.7 |\\|| 75.3  | 72.7  | 74.0 |\n\nAs shown in the table, our method outperforms \"w/o Prototype\" by +2.4%(5-5) and +1.5%(10-10). Though Classifier achieves performance gain compared with w/o Prototype, our method outperforms \"Classifier\" by +2.3% (5-5) and +0.8% (10-10). These results demonstrating that regulating only the classifier (as in classification-based works) is insufficient for IOD. \n\n**In addition, we also find our finding still holds across two extra two-stage detectors and a remote sensing dataset, which further highlights the reliability of our findings and the effectiveness of our framework. The reviewer may find them in our response to Reviewer huSY(Q3, architecture) and Reviewer K17L(Q1,dataset).**\n\nWe emphasize that our goal is not to propose an overly complex method, but to offer a simple yet effective solution grounded in our analysis of catastrophic forgetting in two-stage detectors. We hope this work contributes to bridging the gap between continual learning for classification and detection.\n\nIn the end, we like to thank the reviewer for their valuable suggestions and questions.", "claims_and_evidence": "Yes, the two components, RePRE and NSGP are well described, and the motivation is clear.", "methods_and_evaluation_criteria": "Yes, the paper adopted standard evaluation criteria as in the literature.", "theoretical_claims": "There is no fundamental theoretical claim.", "experimental_designs_or_analyses": "The experimental design follows previous works and the analysis is sound.", "relation_to_broader_scientific_literature": "By addressing the unique challenges of IOD, RePRE contributes to the broader literature on replay-based incremental learning. NSGP focuses on null-space projections for incremental learning, whose core idea has been explored in “Training networks in null space of feature covariance for continual learning, CVPR 2021”", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Insights into Catastrophic Forgetting in IOD\n2. Extensive experiments and good results\n3. Clear writing.\n\nWeaknesses:\n1. Using NSGP has been explored in the literature for continual learning.\n2. Although the specific form of replying regional prototype has not been explored, the idea of prototype based continual learning has been proposed in the literature. \nOnline Prototype Learning for Online Continual Learning, ICCV 2023\nPrototype-Guided Memory Replay for Continual Learning, IEEE TNNLS 2024\n\nThe above two points make the paper's contribution is not so significant.\n\n\n=================  After rebuttal ================\nBased on the authors' feedback and other reviews, I would like to upgrade my score from weak accept to accept.", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxmbpfLDj8", "reviewer": "Reviewer_huSY", "summary": "The paper investigates catastrophic forgetting in incremental object detection using standard Faster-RCNN architecture. The authors show that catastrophic forgetting mainly happens in the RoI part of the model, and the regressor behaves more robustly while learning subsequent tasks. Based on their observations, the authors propose Regional Prototype Replay (RePRE) method, which mitigates classifier forgetting via replay of coarse and fine-grained prototypes and Null Space Gradient Projection (NSGP). NSGPRePRE is evaluated on the Pascal VOC and MS COCO datasets, where it demonstrates improved stability compared to the other IOD methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments.\n\n**Q0: On the Avg metric.**\n\n**R0:** We show the Avg performance at every step.\n\n|10-10|Step1|\\||Step 2|Base|New|Avg|All|\n|-|-|-|-|-|-|-|-|\n|Baseline|77.8|\\||Baseline|69.3|73.3|71.3|71.3|\n|BPF*|77.8|\\||BPF*|71.8|73.4|72.6|72.6|\n|NSGP-RePRE|77.8|\\||NSGP-RePRE|75.3|72.7|74|74|\n\n|5-5|Step1|\\||Step2|Base|New|Avg|All|\\||Step3|Base|New|Avg|All|\\||Step4|Base|New|Avg|All|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Baseline|77.4|\\||Baseline|63.1|79.9|71.5|71.5|\\||Baseline|64.7|76.1|70.4|68.5|\\||Baseline|58.0|59.6|58.8|58.4|\n|NSGP-RePRE|77.4|\\||NSGP-RePRE|72.6|78.1|75.3|75.3|\\||NSGP-RePRE|70.8|76.3|73.6|72.6|\\||NSGP-RePRE|67.9|59.0|63.5|65.7|\n\n\nAll represents the mAP of all seen classes, e.g. 1-15 in 5-5 setting step 3. Our method achieves the best Avg and All mAP at every time step, showing its superior balanced stability-plasticity properties.\n\n\n**Q1 & Q6:On the plasticity-stability trade-off.** \n\n**R1:** The plasticity-stability trade-off is explicitly controlled by the nullity of the uncentered feature covariance. In our experiments, we adopt an adaptive approach VPT-NSP^2(Lu e al.,2024)\nand achieved better trade-off.\n\n**Q2: On the scaling of memory requirement and prototype interference risk.**\n\n**R2:** The memory footprint of our RePRE scales linearly with increasing number of classes. Each class consumes approximately 3.8Mb, and each prototype consumes 0.38Mb. \nThe comparison between our method and previous exampler-based SOTA method ABR is shown below.\n\n|VOC(10-10)|||||\n|-|-|-|-|-|\n|Type|Memory↓|1-10|11-20|1-20|\n|ABR|15.5Mb|71.2|72.3|72.0|\n|RePRE-Coarse|3.8Mb|70.5|73.8|72.1|\n|NSGP-RePRE|38Mb|75.3|72.7|74.0|\n\nWe address redundancy by setting a distance between prototypes to capture the whole distribution of the feature space. Our results show consistent gains, suggesting effective handling of similar classes.\n\n**Q3: conclusion on other two-stage architectures.**\n\n**R3:** To show the generalizability of our key findings, we also conducted experiments with two popular  two-stage detectors, i.e. Cascade-RCNN and Vanilla Faster RCNN without FPN and RoI Align. **The three key findings still hold with different two-stage IODs,** as shown in these links [CascadeRCNN](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/c.pdf) and [VanillaFasterRCNN](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/v.pdf). We conducted the same anotomy to these detectors in our paper, all curves are aligned with Faster RCNN using Pascal VOC 5-5. These two detectors' RPN Recall curve show that seen models even can surpass current model, suggesting that our findings are universal in most two-stage IODs.\n\nWe also evaluated our method on these detectors. \n\n\n|CascadeRCNN|5-5|||\\||10-10|||\n|-|-|-|-|-|-|-|-|\n||1-5|6-20|1-20|\\||1-10|11-20|1-20|\n|Baseline|57.2|65.4|63.4|\\||69.7|74.3|72.0|\n|NSGP-RePRE|66.8|66.6|66.7|\\||74.1|74.6|74.4|\n\n\n|VanillaFasterRCNN|5-5|||\\||10-10|||\n|-|-|-|-|-|-|-|-|\n||1-5|6-20|1-20|\\||1-10|11-20|1-20|\n|Baseline|13.3|27.5|23.9|\\||27.2|32.3|29.8|\n|NSGP-RePRE|19.2|26.9|25.0|\\||29.8|32.6|31.2|\n\nOur method achieved noticeable performance improvement.\n\n**Q4: Concerns about pseudo-labeling.**\n\n**R4:** One major problem in IOD is that objects from past tasks can be included in the subsequent tasks yet their labels are not annotated. Optimizing with wrong label will lead to a drastic performance drop. Pseudo labeling(Mo et al.,2024; Liu et al.,2023;) are widely adopted to alleviate such performance drop. Thus, we also choose pseudo labeling as our baseline.\n\nWe also conducted experiments with pseudo label been ignored(neither foreground nor background). As shown in\n\n|W/o Pseudo Label|\\||5-5|||\\||10-10|||\n|-|-|-|-|-|-|-|-|-|\n||\\||1-5|6-20|1-20|\\||1-10|11-20|1-20|\n|Baseline (w/o pseudo label)|\\||0|28.2|21.2|\\||14.5|66.9|40.7|\n|NSGP-RePRE (w/o pseudo label) |\\||50.5|47.8|48.5|\\||66.2|58.9|64.3|\n\nOur NSGP-RePRE achieves +20% performance gain compared with baseline under the settings of without pseudo label module, demonstrating the effectiveness of our framework.\n\n**Q5: How NSGP affect training time.**\n\n**R5:** Our NSGP-RePRE maintains high efficiency.\nNSGP introduces additional training time by SVD decomposition and null-space projection. Both SVD and projection's required time only increases with the size of the model. For SVD, it only computes once at every incremental step and only took around 30 seconds in our experiments, much less than the whole  training time. As for null-space projection, it also adds only ~1% overhead per iteration compared to baseline training as shown in \n\n||Baseline|NSGP|NSGP+RePRE|\n|-|-|-|-|\n|Time/iter|0.714s|0.719s|0.720s|\n\nNSGP-RePRE does not introduces extra overhead during inference, which makes it practical for real-time application.", "claims_and_evidence": "The claims made in Section 3 are consistent with the experimental results.\n\nHowever, the claim that the Avg metric reflects a better trade-off between stability and plasticity (Line 371) is problematic, and the authors do not provide a good justification for it. For example, separately evaluating the accuracy of base and novel classes after each incremental learning step would offer clearer insights into how the method balances stability and plasticity over time.", "methods_and_evaluation_criteria": "The proposed evaluation is well-suited for the problem.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "See **Questions**.", "relation_to_broader_scientific_literature": "The key contributions of this paper are rooted in and extend the broader literature on incremental learning and object detection, addressing critical gaps in prior work. Previous IOD methods do not dissect component-level contributions to forgetting, and this paper reveals classifier instability as a key source of forgetting in IOD.", "essential_references_not_discussed": "See **Questions**.", "other_strengths_and_weaknesses": "See **Questions**.", "other_comments_or_suggestions": "See **Questions**.", "questions_for_authors": "1. NSGP projects gradients into the null space of old tasks to prevent feature drift. Could this restrict the model's plasticity, especially when new tasks require significant feature adaptation? How does NSGP balance stability and plasticity?\n\n2. RePRE requires storing multiple prototypes per class. How does the memory footprint scale with the number of incremental stages? Is there a risk of prototype redundancy or interference when handling highly similar classes?\n\n3. The conclusion about catastrophic forgetting being localized to the RoI Head classifier is based solely on Faster R-CNN. Have the authors validated this finding on other two-stage architectures? If not, how can we ensure this is a generalizable insight rather than architecture-specific?\n\n4. How critical is pseudo-labeling to NSGP-RePRE’s performance?\n\n5. How does the NSGP affect training time compared to baseline methods? Is the method practical for real-time applications?\n\n6. As shown in the results, the proposed method performs well on base classes. However, I noticed that in Tables 1, 2, and 3, the proposed method underperforms its counterparts on incremental tasks. Does this imply that the method overly prioritizes stability while exhibiting weaker plasticity for learning new tasks?\n\nWhile the authors present a novel finding, their validation is limited to Faster R-CNN, with little evidence of broader applicability. Additionally, critical experimental validations are still missing. I am temporarily giving this paper a weak reject, but I will continue to follow the author's response and the comments from other reviewers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XLafSsRWQQ", "reviewer": "Reviewer_5Yg2", "summary": "The paper addresses the critical challenge of catastrophic forgetting in incremental object detection (IOD). The authors focus on the Faster R-CNN architecture and identify that catastrophic forgetting predominantly occurs in the RoI Head classifier, while the regressor remains robust across incremental stages. Based on this insight, they propose NSGP-RePRE, which combines Regional Prototype Replay (RePRE) and Null Space Gradient Projection (NSGP) to mitigate forgetting in the RoI Head classifier. The method achieves state-of-the-art performance on the Pascal VOC and MS COCO datasets under various incremental learning settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positvie feedback and insightful comment. The Questions and Responses are as follows.\n\n**Q1:** More theorical analysis, such as complexity analysis can be provided.\n\n**R1:** Thanks for the suggestions. We provide a comprehensive analysis of the parameter, computational, and memory complexity of NSGP-RePRE:\n\n**Parameter Analysis:**\n\nThe origional detectors contains a total of 96.89 M trainable parameters. Our approach maintains the same parameter count, ensuring no increase in model complexity.\n\n**Computational Complexity:**\n\nTo assess the computational complexity during training, we present the FLOPs for key components of the model, as shown in the table below:\n\n|       | forward | +RePRE     | backward | +NSGP    |\n|-------|:---------:|:-----------------:|:----------:|:----------:|\n| GFLOPs | 551.35   | +2.78 | 1102.7    | +118.1 |\n\nIt is important to note that the computational complexities of RePRE and NSGP do not scale with batch size, whereas the forward and backward passes of the detector do. As shown in the table, the majority of the computational cost during training arises from the detector's forward and backward passes. In contrast, RePRE and NSGP contribute only about 1% of the additional computation compared to the forward and backward operations when scaling batch size to commonly used 8.\n\nWe also present the actual training time for different components on 1 RTX 3090 GPUs in the following table:\n\n|           | Baseline | NSGP   | NSGP+RePRE |\n|-----------|----------|--------|------------|\n| Time/iter | 0.714s   | 0.719s | 0.720s     |\n\nThis table suggests that the additional computational cost of NSGP-RePRE is minimal, adding only ~1% overhead per iteration compared to baseline training, which is aligned with our complexity analysis. \n\n\n**Memory cost of RePRE:**\n\nThe memory footprint of our RePRE scales linearly with the increase of the number of classes. In our implementation, each class consumes approximately 3.8Mb of memory in float32 with each prototype consuming 0.38Mb. Our method achieves performance comparable to the previous example-based SOTA method, ABR, using only one coarse prototype per class and without relying on NSGP in the 10-10 setting. Additionally, it consumes about 25% of the memory required by ABR. As shown in\n\n| VOC(10-10)   |         |       |       |       |\n|--------------|---------|-------|-------|-------|\n| Type         | Memory↓ | 1-10  | 11-20 | 1-20  |\n| ABR          | 15.5Mb  | 71.2  | 72.3  | 72.0  |\n| RePRE-Coarse | 3.8Mb   | 70.5  | 73.8  | 72.1  |\n| NSGP-RePRE   | 38Mb    | 75.3  | 72.7  | 74.0  |", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Sound", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a significant insight into the nature of catastrophic forgetting in two-stage object detectors, specifically identifying the RoI Head classifier as the primary source of forgetting. This challenges conventional assumptions and offers a new direction for addressing forgetting in IOD.\n\n2. Comprehensive experimental results.\n\n3. Topic is interesting\n\nWeaknesses:\n\n1. More theorical analysis, such as complexity analysis can be provided.", "other_comments_or_suggestions": "See the Strengths And Weaknesses", "questions_for_authors": "See the Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3hNRWfUeS", "reviewer": "Reviewer_K17L", "summary": "The paper focuses on catastrophic forgetting in the two-stage object detector. The authors first analyse the forgetting in component-level and reveal that RoI Head classifier is the primary cause of catastrophic forgetting. Then the authors propose Regional Prototype Replay (RePRE) to mitigate forgetting via replay of coarse prototypes and fine-grained prototypes. The authors also propose using Null Space Gradient Projection (NSGP) to eliminate prototype-feature misalignment. Experiments on VOC and COCO show that the proposed method, NSGP-RePRE, can significantly improve the performance of Faster-RCNN in IOD.\n\n## update after rebuttal\n\nAfter reviewing all the reviews and responses, my primary concerns have been addressed. At this point, I am inclined to accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments.\n\n**Q1:** Dataset concern.\n\n**R1:** To show the generalizability of our key findings, we conducted experiments on a wildly used remote sensing detection datasets DIOR. **The three key findings still hold with different two-stage IODs.** As shown in this link [DIOR](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/d.pdf), all curves are aligned with VOC dataset. Our results show that the detector's RPN struggles with unseen data, as indicated by the red curves. However, the detector performs well on seen data, suggesting the RPN and the RoI Head's regressor remain resilient and do not forget previously learned knowledge on DIOR.\n\nWe also conducted experiments on DIOR with our framework, as shown in\n\n|DIOR|5-5|||\\||10-10|||\n|-|-|-|-|-|-|-|-|\n||1-5|6-20|1-20|\\||1-10|11-20|1-20|\n|Baseline|43.6|56.9|53.6|\\||61.03|63.9|62.5|\n|NSGP-RePRE|54.8|57.0|56.5|\\||66.4|62.25|64.3|\n\nOur framework surpass the baseline by 2.9\\% in 5-5 setting and 1.8\\% in 10-10 setting, further showing the effectiveness of framework.\n\n**Q2:** NSGP's novelty concern.\n\n**R2:**  Although NSGP has been explored in incremental classification, it is non-trivial to apply NSGP in IOD. The following table demonstrates the performance applying NSGP from Backbone to RoI Head accumulatively under VOC 5-5 setting.\n\n||Backbone|+FPN|+RPN|+RoIHead|Ours|\n|-|-|-|-|-|-|\n|NSGPonly|62.6|63.3|63|63.2|65.7|\n\nThe catastrophic forgetting in two-stage detectors is mainly caused by the severe classifier instability in the RoI Head. However, directly appling NSGP on these two-stage object detector (i.e., FPN, RPN,RoIHead)  shows limited performance improvements and can not well address the classifier instability issue, as illustrated in the above Table. \n\nInstead, the proposed Regional Prototype Replay (RePRE) module addresses  this issue by replaying coarse and fine-grained region prototypes in the RoI Head's classification branch. NSGP serves as an assistant component in our framework by mitigating semantic drift caused by parameter updates, thereby preventing toxic replay in RePRE. As shown in the table, our RePRE achieves a +2.4% gain compared with +FPN, underscoring the effectiveness of our proposed framework.\n\n**Q3:** Architecutre-specific concern.\n\n**R3:** To show the generalizability of our key findings, we also conducted experiments with two popular  two-stage detectors, i.e. Cascade-RCNN and Vanilla Faster RCNN. **The three key findings still hold with different two-stage IODs as shown in these links [CascadeRCNN](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/c.pdf) and [VanillaFasterRCNN](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/v.pdf).** Detailed discussion can also be found in our response to reviewer huSY R3.\n\n**Q4:** Insufficient ablation concern.\n\n**R4:**  Thanks for the suggestion. We conducted ablation study in VOC 10-10 setting as shown in\n\n||||VOC|(10-10)||\n|-|-|-|-|-|-|\n|NSGP|Coarse|Fine|1-10|11-20|1-20|\n||||69.3|73.3|71.3|\n|x|||71.8|73.2|72.5|\n||x||70.5|73.8|72.1|\n|x|x||73.7|73.2|73.4|\n|x|x|x|75.3|72.7|74.0|\n\nThe same conclusion can be drawn from this table as in ablation study in 5-5 setting.\n\n**Q5:** Visualization of the results.\n\n**R5:** As shown in this link [visualization](https://anonymous.4open.science/r/aosdhoaihfoiahsodjasjiohf/vis.pdf), we visualized images from VOC2007 test set under 10-10 setting. Task 1 represents visualization with model from time step 1 while baseline and NSGP-RePRE represents the visualization results are from time step 2 trained with corresponding strategy. In (a) baseline forgets \"boat\". In (b) and (c), baseline forgets \"cat\" and \"car\" due to the interference of new classes \"dog\" and \"motorbike\". Our NSGP-RePRE successfully remembers old classes while learning new classes effectively, suggesting that our methods achieves better stability while retaining comparable plasticity compared with baseline.\n\n**Q6:** The pseudo labeling concerns\n\n**R6:** One major problem in IOD is that objects from past tasks can be included in the subsequent tasks yet their labels are not annotated. For example, airplanes will be labeled as \"airplane\" in the first task but considered as background in the subsequent tasks. Optimizing with wrong label will leads to a drastic performance drop. Pseudo labeling(Mo et al.,2024; Liu et al.,2023;) are wildly adopted to alleviate such performance drop. Thus, we also choose pseudo labeling as our baseline to avoid this problem.\n\nWe also conducted experiments under the setting of without pseudo label. As shown in\n\n|W/o Pseudo Label|\\||5-5|||\\||10-10|||\n|-|-|-|-|-|-|-|-|-|\n||\\||1-5|6-20|1-20|\\||1-10|11-20|1-20|\n|Baseline (w/o pseudo label)|\\||0|28.2|21.2|\\||14.5|66.9|40.7|\n|NSGP-RePRE (w/o pseudo label) |\\||50.5|47.8|48.5|\\||66.2|58.9|64.3|\n\nOur NSGP-RePRE achieves +20% performance gain compared with baseline, demonstrating the effectiveness of our framework.", "claims_and_evidence": "No.\nThe authors claim that 'In sequential tasks, the stability of the RPN recall ability is largely maintained.'. However, all experiments are conducted on VOC and COCO, lacking experiments on drastic variations in target scale or target aspect ratio between different tasks.", "methods_and_evaluation_criteria": "No.", "theoretical_claims": "There is no Theoretical Claim in the paper.", "experimental_designs_or_analyses": "No. The paper lacks experiments on more datasets. Conclusions drawn from COCO and VOC may not be universally applicable.", "relation_to_broader_scientific_literature": "Previous works in the IOD field usually treats the detector as a whole, lacking a fine-grained analysis. This paper decouples localization and classification and analyzes catastrophic forgetting with experimental results in two-stage detectors at the component level.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "## Strengths\n\n1. The paper identifies a meaningful perspective in IOD with two-stage detectors and propose a solution to mitigate forgetting in classification.\n\n2. Experiments on two widely-used datasets are adequate. The proposed method reaches state-of-the-art on multiple datasets and settings.\n\n3. The writing of the papre is easy to follow and it is clearly structured.\n\n## Weakness\n\n**Major**\n\n1. The paper lacks experiments on more datasets with significant variations in target scale and aspect ratio, which makes the conclusions of the paper more limited.\n\n2. The NSGP seems to be a simple application of previous method, which is less innotative.\n\n**Minor**\n\n1. The authors only use Faster R-CNN in experiments. It is suggested to try more two-stage detectors like Cascade-RCNN to verify the generalizability of the conclusions to two-stage detectors.\n\n2. The ablation in Tab 4 is insufficient. The authors are suggested to conduct more experiments on other settings, such as VOC 10-10.\n\n3. The paper lacks visualization results. Including visualization results that demonstrate how the proposed method corrects misclassifications would provide readers with a more concrete understanding of its performance.", "other_comments_or_suggestions": "1. A Grammatical error in line 258~259 should be corrected: \"To capture the entire spectrum of useful information on the distribution of RoI features.\"\n\n2. The citation of BPF (ECCV'24) should be a conference version.\n\n3. The result of ABR in the last column of Tab 2 is incorrectly bolded.", "questions_for_authors": "1. All experiments are conducted on VOC and COCO. Do the findings about catastrophic forgetting in Faster R-CNN still hold on datasets with significant variations in target scale and aspect ratio (such as remote sensing detection datasets)?\n2. The author mentions that the baseline employs a pseudo-labeling strategy, and the reviewer is interested in the performance of the proposed method after removing the pseudo-labels.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers highlighted this paper's novel setting, good writing, and convincing experiments. The authors also provided reasonable answers to the doubts about the details of the proposed dataset and method. The reviewers’ concerns were well-addressed. Therefore, the decision is to recommend Acceptance.
Decision: Accept (poster)