Purified Distillation: Bridging Domain Shift and Category Gap in Incremental Object Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 09:55:59

Abstract: Incremental Object Detection (IOD) simulates the dynamic data flow in real-world applications, which require detectors to learn new classes or adapt to domain shifts while retaining knowledge from previous tasks. Most existing IOD methods focus only on class incremental learning, assuming all data comes from the same domain. However, this is hardly achievable in practical applications, as images collected under different conditions often exhibit completely different characteristics, such as lighting, weather, style, etc. 
Class IOD methods suffer from severe performance degradation in these scenarios with domain shifts. To bridge domain shifts and category gaps in IOD, we propose Purified Distillation (PD), where we use a set of trainable queries to transfer the teacher's attention on old tasks to the student and adopt the gradient reversal layer to guide the student to learn the teacher's feature space structure from a micro perspective. This strategy further explores the features extracted by the teacher during incremental learning, which has not been extensively studied in previous works. Meanwhile, PD combines classification confidence with localization confidence to purify the most meaningful output nodes, so that the student model inherits a more comprehensive teacher knowledge.
Extensive experiments across various IOD settings on six widely used datasets show that PD significantly outperforms state-of-the-art methods. Even after five steps of incremental learning, our method can preserve 60.6\% mAP on the first task, while compared methods can only maintain up to 55.9\%.

Main paper: 
 introduction: 
 Benefiting from the development of deep learning, object detection has achieved significant progress [10, 15, 37] . However, with the constant expansion of datasets and increasing number of categories, traditional training paradigms for object detection are facing new challenges [21] , primarily in adapting to new domains [25] and categories [5, 11] . The simplest solution is to collect new data and retrain the model with both new and old data, but retraining a model from scratch can be costly or impractical if the old data is no longer accessible. This necessitates engineers to finetune models exclusively with new data, while maintaining the knowledge from previous data, i.e., to avoid catastrophic forgetting [12] . Incremental Object Detection (IOD) [24, 26, 33] formalizes this problem, where the model continually accumulates new knowledge without forgetting previous knowledge. Existing works [5, 11, 13, 26, 27, 31] predominantly focused on class incremental learning, assuming that data for both new and old tasks comes from the same domain, and new categories are introduced in the new task. Nevertheless, this assumption proves overly restrictive in real-world applications, where models are commonly deployed across diverse contexts [21] . This implies that data for both new and old tasks often originates from different domains, thereby presenting a more challenging and less explored problem, incremental object detection with domain shift and category gap. As shown in Fig. 1 , the detector encounters a series of tasks, each with potentially different categories and domains. After incremental learning, the detector should be able to remember all previously seen classes and domains. Liu et al. [21] initiated exploration into this problem and proposed Attentive Feature Distillation (AFD) as an attempt to alleviate forgetting. The efficacy of AFD largely relies on exemplars preserved from old data, but accessing old data can be impractical, and an excessive number of exemplars increases training costs. Therefore, further exploring the informative knowledge held by teacher models, we propose a novel distillation approach that requires no exemplars from previous tasks to bridge domain shifts and category gaps in IOD.IOD methods typically utilize features and outputs distillation [26, 27] to prevent catastrophic forgetting, where the model trained on the old task serves as a teacher. Features extracted by the teacher prove to focus on areas beneficial for old classes and domains, hence some works [13, 21, 27] minimize the differences (𝐿 2 or 𝐿 1 loss) between the teacher's and student's features to help preserve the student's perception of old tasks. However the teacher focuses exclusively on the old task, the response to the new task is limited. Such feature distillation approaches can confuse the student model during the learning of new tasks. On the other hand, guiding the student model using the teacher's outputs is also a common practice [5, 27, 30] . The outputs are often noisy and exhibit a significant imbalance between positive and negative samples [5] , so it is necessary to select the most meaningful nodes to contribute distillation, typically based on a top-𝑘 strategy [30] or by setting a threshold [5] . Nevertheless, we notice that these selection strategies introduce some low-quality nodes, such as those with high classification scores but low confidence in bounding box positions, as detailed in the appendix.To address the above issues, we delve into the analysis of features and outputs extracted by the teacher model, proposing Purified Distillation (PD). PD employs three distillation strategies: Multi-scale Cross Attention Distillation (MCAD), Feature Space Distillation (FSD), and Entropy Guided Output Distillation (EGOD). MCAD: We use trainable parameters as queries to transform the multi-scale feature maps extracted from both the teacher and student into attention maps [35] . Subsequently, by minimizing differences between the attention maps of the teacher and student, these queries adaptively purify regions preserving previous knowledge, providing flexibility for balanced learning between current and previous tasks. FSD: We notice that the trained model possesses a unique feature space, which reflects the abstract characteristics of training data but has been overlooked by previous methods. Based on this observation, we propose to transfer the structural properties of the teacher's feature space from a microscopic perspective. Specifically, we design a multi-scale feature discriminator and adopt the Gradient Reversal Layer (GRL) [6] to encourage the student to learn a feature space that simultaneously exhibits characteristics of both new and old tasks. In this way, the student's feature space can be seen as an approximation of full dataset training. EGOD: Inspired by modeling a bounding box as a discrete distribution [16] , we propose entropy guided output distillation, which employs an entropy guided selection strategy to choose the most important output nodes. This strategy treats the entropy of the distribution as the location confidence and integrates classification confidence to select the top-𝑘 bounding boxes to teach the student model. Extensive qualitative and quantitative experiments demonstrate that PD can preserve more knowledge of previous tasks and avoid catastrophic forgetting. Since PD does not require any modifications to the detector's structure, the detector can still maintain its efficiency during inference.In summary, our contributions are as follows:• To our knowledge, the proposed purified distillation is the first work to achieve incremental object detection with both domain shift and category gap, and eliminates dependence on exemplars from old tasks. • By leveraging learnable parameters to query feature maps, we propose multi-scale cross attention distillation to enable the model to adaptively query information that is more meaningful for old knowledge.• We devise a multi-scale feature discriminator and utilize a gradient reversal layer to distill the structure of feature space, so that the student can learn feature characteristics of both new and old tasks. • We develop the entropy guided selection strategy to purify more representative nodes from outputs, which considers both the location confidence and classification scores to reduce redundant information. • We construct multiple incremental object detection scenarios across five widely used datasets, and extensive experiments demonstrated the advantages of our method in mitigating catastrophic forgetting. 
related works 2.1 object detection: 
 Object detection has been extensively studied. Transformer-based detectors [1, 14, 19, 37, 40] commonly train a set of queries to locate objects within images, whereas suffering from significant computation. CNN-based detectors can be categorized into two-stage and one-stage detectors. As the most classic two-stage detector, Faster R-CNN [30] utilizes a region proposal network to generate proposals, followed by separate branches for computing classification scores and location offsets. Due to the complex forward procedure, two-stage detectors are not applicable for real-time scenarios. While one-stage detectors [20, 32, 38] directly feed multi-level feature maps into detection heads to predict classification scores and locations. Furthermore, Li et al. [16] proposed Generalized Focal Loss (GFL), modeling the location output in an integral form to reduce model ambiguity and uncertainty. The latest work in the YOLO series [28, 29, 36] , YOLOv8 [10] , incorporates GFL and task-aligned sample assigner [4] , achieving state-of-the-art in both efficiency and accuracy.Recently, incremental learning [24, 26, 33] has received much attention, which requires the model trained on the new task to retain knowledge from the old task. To avoid catastrophic forgetting, knowledge distillation [17] is widely used to transfer the teacher's knowledge to the student. Shmelkov et al. [31] pioneered incremental object detection by leveraging distillation losses, effectively maintaining the memory of old tasks. Afterwards, Li et al. [13] and Peng et al. [26] further proposed distillation methods for Reti-naNet [18] and Faster R-CNN [30] , respectively. SID [27] characterized the relation between instances, proposing selective and inter-related distillation for anchor-free detectors [32, 38] . While Liu et al. [22] proposed CL-DETR, a transformer-based IOD, by utilizing knowledge distillation and exemplar replay. Considering the inadequacy of existing methods in exploring responses from detection heads, Feng et al. [5] introduced elastic response distillation, selectively distilling nodes from output responses. Despite their success in class incremental object detection, the problem with both domain shift and category gap has been overlooked. Liu et al. [21] made the first work to explore this problem and proposed Attentive Feature Distillation (AFD) composed of top-down and bottom-up feature distillations. AFD relies heavily on a set of exemplars from old tasks, but this could be impractical in real-world applications. Thus, we propose a more feasible and realistic distillation method that does not require any exemplars, alleviating catastrophic forgetting in incremental object detection with both domain shift and category gap. 
method 3.1 overall structure: 
 We construct PD on the single-stage detector which can be divided into two key components: the backbone that extracts features, and the detection head that predicts the category probability and location offsets. Considering the two components, we propose multi-scale cross attention distillation and feature space distillation to preserve knowledge from old tasks in the latent space, and the entropy guided selection strategy to transfer the classification and location information of the valuable output nodes. The architecture of our method is depicted in Fig. 2 . The teacher detector trained on the old task provides initial weights of the student model, which will be incrementally trained on the new task. During incremental training, the teacher is frozen, and only the student is trainable. The backbones for the teacher and student, denoted as B T and B S , extract multi-scale features F T and F S respectively. We fuse the multi-scale feature maps, and construct trainable parameters 𝑄 to adaptively query features that are crucial for maintaining previous knowledge. Moreover, a feature discriminator is responsible for distinguishing whether feature maps are extracted by the student. We employ a gradient reversal layer to reverse the gradient backward from the discriminator, ensuring the transfer of the structural characteristic of feature space. Feature map F is subsequently passed through the detection head H to predict classification scores F 𝑐𝑙𝑠 and location offsets F 𝑙𝑜𝑐 . To purify output nodes, we propose the Entropy Guided Selection (EGS) strategy, which selectively transfers semantic and location knowledge from teacher to student.Overall, the loss function of our incremental learning method is as follows:EQUATIONwhere L 𝑜𝑟𝑔 is the standard loss of the single-stage detector for learning the new task, typically composed of classification and localization losses, and L 𝑝𝑑 is the distillation loss of the proposed PD. The parameter 𝜆 balances the distillation and standard training, empirically set to 10.To transfer the knowledge to the student in the latent space and avoid adversely affecting the current task, we utilize a group of trainable parameters as queries to adaptively focus the feature most crucial to the old task. The features F extracted by the backbone include 𝐿 levels (𝐿 = 3 typically) of multi-scale features, and the features at the 𝑙-th level can be denoted as F 𝑙 ∈ R 𝑤 𝑙 ×ℎ 𝑙 ×𝑑 𝑙 , where 𝑤 𝑙 , ℎ 𝑙 , and 𝑑 𝑙 are the width, height, and number of channels of the feature map, respectively. The queries can be considered as a matrix 𝑄 ∈ R 𝑛 𝑞 ×𝑑 , where 𝑛 𝑞 and 𝑑 are the number of queries and dimensions, respectively. Without loss of generality, we set 𝑑 to the dimension of the feature map at the deepest level.Taking into account that the channels of feature maps vary across levels, and larger feature maps can lead to significant computational costs, we devise Multi-scale Cross Attention (MCA) to query the attention map by down-sampling and integrating feature maps at different levels. As illustrated in the right of Fig. 2 , using channelwise average pooling, the MCA first aligns the channels of feature maps to 𝑑 on each level. Then the features are concentrated along the horizontal and vertical directions, so that the key and value matrices are obtained by concatenating the pooled features:EQUATIONEQUATIONwhere P 𝑐 , P 𝑥 , and P 𝑦 are the average pooling layer over the channel, x-axis, and y-axis, respectively; 𝐶𝑎𝑡 is the concatenation operation; and 𝑛 𝑝 = 𝐿 𝑙=1 𝑤 𝑙 + ℎ 𝑙 . Finally, given the softmax function 𝜏, the attention map is:EQUATIONAs trainable parameters, queries 𝑄 have considerable flexibility to adapt and learn the intricate dependencies among features, thus the attention maps 𝐴 queried by 𝑄 can gradually capture knowledge critical to old tasks during training. By compelling the student's attention map 𝐴 S to mimic the teacher's 𝐴 T , the student can retain memory about old tasks. Hence, the attention distillation loss L 𝑎𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 is defined as follows:EQUATIONwhere L 𝑀𝑆𝐸 is the mean square error loss.We observed that models trained on different datasets possess unique feature spaces (detailed in Sec. 4.1), meaning that the feature vectors extracted by a model are distributed in a specific region of the latent space. Therefore, we argue that the teacher's feature space can reflect its knowledge about the abstract style of data in the old task, and introduce Feature Space Distillation (FSD) to transfer this property to the student model. Because of the challenge of quantitatively representing the feature space, we investigate utilizing individual feature maps to transfer the structure of the feature space. Directly applying an 𝐿 2 loss, however, would lead to conflicts between new and old knowledge, thus we innovatively introduce a Gradient Reversal Layer (GRL) [6] into incremental object detection to encourage the student to extract more "old-task-style" feature maps, instead of requiring strict consistency between student and teacher. By trying to confuse the discriminator, the student model can learn a feature space that can represent both new and old task characteristics, closer to that of the full dataset training.Existing discriminators are mostly based on the patch discriminator [39] , which accepts a single image as input and outputs the probability of whether each patch is synthesized, but is unsuitable for evaluating multi-scale feature maps. Feature maps at different scales represent objects and backgrounds from different perspectives, and isolated processing of different scales of feature maps overlooks their intrinsic connections. Therefore, we construct an efficient multiscale feature discriminator, which comprehensively leverages the semantic information of feature maps at different levels, as depicted in Fig. 3 .We use convolutional layers with different strides to align the feature maps to the same dimension and fully connected layers to reweight feature maps at different scales. The reweighted feature maps are then added together and fed into two convolutional layers to predict the discrimination results. The output of the discriminator is a logits matrix O 𝑙 ∈ R 𝑤 𝑙 ×ℎ 𝑙 , which determines whether a patch belongs to the student or teacher. The teacher feature maps F T 𝑙 are directly fed into the discriminator, while the student features F S 𝑙 require passing through the GRL before inputting the discriminator D, detailed as follows:EQUATIONEQUATIONwhere R is the GRL. During the forward propagation, R does not modify its input, whereas during the backward propagation, it reverses the sign of the gradient. [𝑃 𝑙 ] 𝑖,𝑗 represents the confidence that the patch belongs to the student, and the corresponding target is defined as [𝑌 S 𝑙 ] 𝑖,𝑗 = 1 and [𝑌 T 𝑙 ] 𝑖,𝑗 = 0, thus the feature space distillation loss L 𝑠𝑝𝑎𝑐𝑒 is defined as:EQUATIONwhere L 𝐵𝐶𝐸 is the binary cross entropy loss. As training progresses, the discriminator tries to discern whether the input feature is extracted from the teacher or student model, while the student aims to extract features that confuse the discriminator, aligning its features more similar to those of the teacher.The outputs from the detection head represent the semantic and location knowledge held by the model but contain a considerable amount of redundant information, therefore, we define the Entropy Guided Selection (EGS) to transfer classification and location information from the teacher to the student selectively. Specifically, the 𝑙-th detection head H 𝑙 takes the feature maps F 𝑙 as inputs, and outputs two sets of predictions: classification scores F 𝑐𝑙𝑠,𝑙 ∈ R 𝑤 𝑙 ×ℎ 𝑙 ×𝑛 𝑐 and location offsets F 𝑙𝑜𝑐,𝑙 ∈ R 𝑤 𝑙 ×ℎ 𝑙 ×4×𝑛 𝑙𝑜𝑐 , where 𝑛 𝑙𝑜𝑐 is the number of sampling points for the distribution, typically set to 16; 𝑛 𝑐 is the number of classes, and 𝑛 T 𝑐 ≤ 𝑛 S 𝑐 .For the teacher's classification scores F T 𝑐𝑙𝑠,𝑙 , we select the maximum over the class dimension to obtain the class selection scores 𝑆 𝑐𝑙𝑠,𝑙 ∈ R 𝑤 𝑙 ×ℎ 𝑙 :EQUATIONwhere 𝜎 is the sigmoid function.For the teacher's location offsets F T 𝑙𝑜𝑐,𝑙 , we first apply the softmax function to transform the location of any edge into a discrete probability distribution. Then, we can calculate the entropy of each edge to measure the detector's location confidence as follows:EQUATIONwhere [𝐶 T 𝑙 ] 𝑖,𝑗,𝑘 ∈ R 𝑛 𝑙𝑜𝑐 and [𝐸 T 𝑙 ] 𝑖,𝑗,𝑘 ∈ R is the distribution and the entropy of the 𝑘-th edge of bounding box (𝑖, 𝑗) respectively.In a discrete probability distribution, the maximum entropy occurs when all possible outcomes have equal probability, yielding an entropy of log 𝑛 𝑙𝑜𝑐 ; the minimum entropy occurs when one outcome has a probability of 1 while others are 0, resulting in zero entropy. Thus we can normalize the entropy to the range of [0, 1]:EQUATIONA larger normalized entropy 𝐸 ′ 𝑙 indicates the detector has higher confidence in localizing that edge, and vice versa. For bounding box (𝑖, 𝑗), we take the minimum normalized entropy of the four edges as the location selection score [𝑆 𝑙𝑜𝑐,𝑙 ] 𝑖,𝑗 :EQUATIONBy multiplying the class and location selection scores, we obtain the selection score for the 𝑙-th level:EQUATIONwhere ⊗ is the Hadamard product. We select the top-𝑘 nodes with the highest selection scores, denoted as G 𝑐𝑙𝑠,𝑙 and G 𝑙𝑜𝑐,𝑙 . The output distillation loss L 𝑜𝑢𝑡𝑝𝑢𝑡 is defined as:EQUATIONwhere L 𝐾𝐿 is the KL divergence loss; 𝜎 and 𝜏 are sigmoid and softmax function respectively; 𝑡 is the temperature, typically set to 2.Given the attention distillation loss, the feature space distillation loss, and the entropy guided output distillation loss, we formalize the total distillation loss L 𝑝𝑑 as follows:EQUATIONBy purifying the teacher's knowledge from various perspectives, L 𝑝𝑑 facilitates the transfer of the most meaningful knowledge from the teacher to the student, and prevents interference from previous knowledge during learning new tasks.  
experiments: 
 Datasets and Metrics. To validate the generalizability of our method, we employ six commonly used datasets: PASCAL VOC [3] , Watercolor [9] , Comic [9] , KITTI [7] , Kitchen [8] , and Parasites [2] , detailed in Tab. 1. To quantify the detection performance, we use the mean Average Precision at the Intersection over Union (IoU) threshold of 0.5 (mAP@0.5).Based on YOLOv8, we implement our PD and other methods, including vanilla finetuning, LwF [17] , Faster ILOD [26] , SID [27] , and ERD [5] . We train the model on the old task from the official weights [10] , subsequently, this model is treated as the common teacher of different IOD methods. In each task, the model is trained for 20 epochs with a batch size of 16, a learning rate of 1e-4, and the AdamW optimizer [23] , while other training parameters are the same as the original YOLOv8. All experiments are conducted on a single RTX 3090 GPU. For hyperparameters, we set 𝑘 = 10% and 𝑛 𝑞 = 512 whose sensitive analyses are detailed in the appendix.To visualize feature spaces, we train a model on each dataset, then extract feature maps of samples from all six datasets and plot them in Fig. 5a using t-SNE [34] for dimensionality reduction. We can see that even samples not in the training dataset have feature maps distributed near the training samples, forming a unique region. This suggests that the structure of feature space represents the model's certain knowledge about the characteristics of the training dataset, which has not been explored by existing methods. To verify whether Feature Space Distillation (FSD) can transfer knowledge of feature space structure, we remove FSD from our method and compare it with different methods in the scenario (Kitchen → KITTI), as shown in Fig. 5b . Compared to other methods (including PD without FSD), PD has the highest overlap with joint training in feature space, indicating that FSD effectively guides the student to extract features that exhibit both new and old task characteristics. (d) SID [27] , (e) ERD [5] , and (f) Ours. We provide the predictions on old tasks after incremental learning on new tasks. We can see that competitive methods exhibit significant catastrophic forgetting, manifested as the inability to detect objects from old tasks or misclassification into other categories. In contrast, our method excels at preserving knowledge of old tasks. The models are incrementally learned with different methods under the scenario (Kitchen → KITTI) and visualize feature maps in all datasets. "Joint" is training with both new and old datasets, and "PD w/o FSD" is our method without FSD.One-step incremental object detection can be categorized into three scenarios: "Domain Shift", "Category Gap", and "Domain Shift + Category Gap". In detail, the dataset pairs (Kitchen, KITTI), (Comic, Parasites), (Kitchen, Parasites), and (VOC, Comic) differ in both domains and categories across old and new tasks; the dataset pair (Watercolor, Comic) only differs in domains with the same categories; and the dataset pair (VOC[:10], VOC[10:]) only differs in categories with the same domain, where VOC[:10] and VOC [10: ] denote the sub-dataset that contain the first and last ten classes of VOC. Tab. 2, Tab. 3, and Tab. 4 illustrate the experimental results of our method compared to others under scenarios with both category gap and domain shift, only category gap, and only domain shift, respectively. It can be observed that models finetuned achieve high accuracy on new tasks but fail to retain the knowledge of old tasks, resulting in catastrophic forgetting. Although LwF is a classic method in incremental classification, it proves inadequate in IOD and tends to forget old tasks. SID shows some improvement but maintaining memory of old tasks has caused significant interference with learning new tasks. As the state-of-the-art method in class incremental object detection, ERD outperforms other methods in the "Category Gap" scenario but falls short in scenarios with domain shift. We also notice that despite being a relatively outdated method, Faster ILOD performs comparably to ERD in many scenarios, and even surpasses ERD. In comparison, our PD achieves up to 4.65% higher mAP compared to the state-of-the-art method in maintaining knowledge of old tasks. Moreover, we visualize the qualitative results of incremental learning in different scenarios in Fig. 4 . It can be observed that even the most competitive method, ERD, inevitably exhibits forgetting of old tasks, incorrectly identifying objects as background or other categories, while our PD effectively detects objects from both new and old tasks.In real-world applications, detectors are often expected to incrementally learn multiple tasks. Therefore we construct a task sequence with six datasets (VOC → KITTI → Watercolor → Comic → Kitchen → Parasites), and report the detection performance on all previously learned datasets after each incremental step. The results of multi-step incremental object detection are visualized in mAP matrices, as shown in Fig. 6 . We can tell that PD effectively preserves the knowledge of old tasks while adapting to new ones. After five steps of incremental learning, our method retains 60.6% mAP for the first task, whereas the least effective comparative method, LwF, manages to preserve only 10.1%. Even the best-performing comparative method, Faster ILOD, is only able to maintain 55.9% detection performance for the first task.In this section, we select the two most challenging scenarios: (Kitchen → KITTI) and (Comic → Parasites), and conduct detailed ablation experiments on three distillation losses. As shown in Tab. 5, exclusive reliance on either attention distillation or feature space distillation is insufficient to preserve comprehensive old knowledge. Student models trained with these individual approaches tend to overlook objects that appear only in old tasks, while the introduction of output distillation can further mitigate catastrophic forgetting. Achieving Pareto optimality for both new and old tasks is observed when combining all three distillation losses.We have demonstrated that PD effectively preserves old task knowledge on the one-stage detector YOLOv8 [10] . In this section, we Table 6 : Results of implementation on other detectors. To validate the scalability of our method, we implement PD and competitive methods on the two-stage detector (Faster R-CNN) and the Transformer-based detector (DINO). Since CL-DETR is designed specifically for DETR and is hard to apply to other models, we only re-implemented it on DINO. Here CL-DETR* denotes we maintain a reservoir of exemplars as the original paper [22] . aim to explore the scalability of PD when applied to other detectors, the most classic two-stage detector Faster R-CNN [30] and a novel transformer-based detector DINO [37] . After modifying the detection heads, we re-implemented our method and compared it with Faster ILOD and ERD. For DINO, we additionally implemented the latest incremental learning method for transformer detectors, CL-DETR [22] . The results of incremental object detection are shown in Tab. 6. It is evident that despite not being specifically tailored for these models, PD also performs well on other types of detectors. CL-DETR, however, shows significant performance degradation. This occurs due to its use of pseudo-labeling on the current dataset to avoid catastrophic forgetting, which is not suitable when there is a significant disparity between the old and new datasets. We also observed a more severe forgetting when applying these CNN-based methods to DINO, with the drop of mAP on old tasks being nearly eight times higher compared to Faster R-CNN. We speculate this is due to the instability of the Hungarian matching [1] in the transformer detector, and we will focus on this issue in future work. 
conclusion: 
 In this paper, we proposed Purified Distillation (PD) to mitigate catastrophic forgetting in incremental object detection with both domain shift and category gap. PD leveraged multi-scale cross attention distillation to adaptively focus on features relevant to old tasks and employed feature space distillation to guide students in learning the characteristics of both new and old tasks. Additionally, an entropyguided selection strategy is introduced, integrating classification and location confidence to purify the most representative output nodes for output distillation. We conducted experiments across multiple scenarios using six datasets to simulate the complex demands of real-world applications. Extensive experiments underscored the generalizability and robustness of PD. 
 
 Appendix:  


Decision: This paper proposes a distillation-based incremental object detection method for handling both domain and class incremental learning. All reviewers agreed to accept this paper after the rebuttal period. Therefore, my final recommendation is accept (poster).

Reviews: review: paper_summary: paper_summary: This work propose a novel knowledge distillation method to mitigate the catastrophic forgetting in incremental object detection. Three different distillation losses are proposed to address the domain shift and category gap. Extensive experiments demonstrate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: - The experiments are solid.
- The idea of introducing the discriminator for distillation seems interesting
limitations: limitations: - In this paper, three losses are proposed. However, the motivation for using these three losses is unclear, and the authors are encouraged to give an explanation to show that the proposed method is not a simple combination of three losses.
- The latest baseline in Table2 is up to 2022, the authors are advised to compare with recent methods.
- The writing needs further improvement, and there are some grammatical errors.
- In Equation 3, the channel dimension has been global average pooled, why there still exists the channel dimension $d$?
- In the Multi-scale Cross Attention Distillation, the authors stated the query can obtain domain-specific feature by cross-attention, it needs more evidence to support this statement. Moreover, this Multi-scale Cross Attention Distillation is very similar to existing promptKD[1], with both using learnable parameters to query interesting region, except the additional multi-scale strategy used in this work. The author should clarify the specific difference.
 
> [1] PromptKD: Unsupervised Prompt Distillation for Vision-Language Models
paper_summary: paper_summary: The paper addresses the task of  incremental object detection with both domain shift and category gap. 
The authors propose a novel purified distillation including multi-scale cross attention distillation, feature space distillation, and entropy guided output distillation. Extensive experiments across various Incremental Object Detection(IOD) settings on six datasets show that PD significantly outperforms state-of-the-art methods in preserving knowledge from previous tasks.
strength_weakness: strength_weakness: 1. This paper is well-written and easy to follow with figure of overall framework.

2. The authors produce a novel purified distillation methods with state-of-the-art results.

   2.1 The authors accordingly bring discriminator to solve the gap between source domain and target domain in latent space

   2.2 They provide detailed explanations of the equations with figures as **fig. 3.**

3. These methods do not need extra data from source domain.(e.g. exemplars, expanded network)

4. This paper thoroughly demonstrates the effectiveness of the proposed method through experiments.
limitations: limitations: The paper demonstrate the effectiveness of the proposed **multi-scale cross attention distillation** method. Multi-scale distillation techniques are widely used in many computer vision tasks, however, also there are already existing papers that address **catastrophic forgetting** in incremental learning utilizing **additional learnable parameters(branch)**. (e.g. "FDCNet: Feature Drift Compensation Network for Class-Incremental Weakly Supervised Object Localization" and "Class-Incremental Learning with Strong Pre-trained Models")

**novelty and comparison:** 

The authors need to describe the differences in **Sec. 2.** or **Sec. 3.**.

**Additional experiments:** 

The ablation study on the three proposed losses is well conducted. However, if possible, I would like to see the experimental results varying with different $\lambda$ values.
paper_summary: paper_summary: This paper proposes a distillation-based incremental object detection method for handling both domain and class incremental learning. Multi-scale Cross Attention Distillation (MCAD), Feature Space Distillation (FSD), and Entropy Guided Output Distillation (EGOD) are designed to transfer knowledge from the teacher to the student. Single-step and multiple-step incremental learning experimental results on five datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. It is a practical problem to handle both domain and class incremental learning simultaneously.
2. The proposed three types of distillation are interesting and effective.
3. Experiments are sufficient. The results are superior to the compared methods.
limitations: limitations: 1. The authors’ insights into the unique problems of domain-incremental object detection should be presented in the introduction. Why previous incremental object detection methods can not be directly applied to this task? Why the proposed method can bridge domain shift and category gap simultaneously?
2. The reason why this method can eliminate dependence on exemplars from old tasks, as the author claimed, should be given theoretically. From the results of the implemented methods, these methods can also achieve satisfactory results without storing exemplars, such as Faster ILOD. This may conflict with the statement in the abstract: “Class IOD methods suffer from severe performance degradation in these scenarios with domain shifts.”
3. Several operations are designed for multi-scale features. Are there any experiments to verify the effectiveness of transferring multi-scale features rather than single-scale features? Previous methods are primarily designed for single-scale features. 
4. How about the computational cost? The attention module for Multi-scale Cross Attention Distillation and convolutional layers for aligning feature maps both increase the parameters.
5. What’s the meaning of “We train the model on the old task from the official weights [10]” in Line 553?
6. The related works are not sufficient. Many recent works in 2023 and 2024 are not cited, such as:

a) Non-exemplar Domain Incremental Object Detection via Learning Domain Bias. AAAI 2024.

b) Domain Incremental Object Detection Based on Feature Space Topology Preserving Strategy. TCSVT 2024.