# Predictive Distillation Method of Anchor-Free Object Detection Model for Continual Learning
## 2 Related Work  
### Continual Learning and Knowledge Distillation  
In the process of accepting new knowledge, the re-learning of a pre-trained model forgets a large portion of the existing knowledge, and it is difficult to accept new knowledge if the existing knowledge is excessively fixed. This problem is called the stability-plasticity dilemma. Furthermore, the above-described problem of forgetting substantial amounts of existing knowledge is called catastrophic forgetting. The objective of CL is to minimize catastrophic forgetting and determine ideal decision boundaries in continual learning [1; 2; 4; 9; 10; 11; 12]. The part covered in this study is learning without forgetting (LwF) in the green part, and this algorithm is a type that uses the KD method among the various CL fields. The KD-based CL method, a data normalization method, began with the model compression technique proposed by [PERSON] et al. [5], and its application was extended to CL. The first study to use this method for CL was LwF [2; 6]. The application of CL using LwF as an example is a proposed case for classification problems, and most CL studies are still focused on them. However, research subjects are gradually expanding into object detection and semantic segmentation. As one of the object detection cases to which KD was applied, [PERSON] et al. [13] proposed a KD-based CL model using Fast R-CNN, an object detection model. This study involved class IL, and the LwF technique described above was applied so that the model learning the next time received the information learned previously. In addition, [PERSON] et al. [14] also proposed a continual learning model with Fast R-CNN with KD, which applied RoI and RPN between the t-model and s-model. The data scenario in both studies were used as a sequence style, and the use of the data in continual learning is described in detail in Section 2.2 [15]. [PERSON] et al. [8] proposed an object detection model that applies distillation techniques to CenterNet and full convolutional one-stage object detection (FCOS) [16], which are object detection models without anchor boxes. This paper has proposed, for the first time, a CL model in an anchor-free model, and the data scenario is disjoint. KD was performed for the output layer rather than the regression type in the two anchor-free models, and inter-relation, a method of distilling the relationship between images, has also been proposed. For the inter-relation, the feature map from the uppermost layer of the decoder was used as  
Figure 1: Conceptual diagram of continual learning. the distillation layer. After one batch training, the weight of the heatmap output physically returns to its original state (i.e., the t-model state) by restoring to the next model, among the final output loads of the existing learned model, the output layer related to the previous class. In other words, the purpose is to restore information that has been forgotten in the previous learning process. [PERSON] et al. [17] used the KD method in GFLV2 [18], a one-stage model, to generate a CL object detection model. Scenarios for the study have been mentioned, but it is not known specifically what style of data utilization methods were used. [PERSON] and [PERSON] [15] proposed several methods of CL models in the semantic segmentation model DeepLabv3+ [19], utilized distillation techniques, and verified various data scenarios [15; 20; 21]. Ref. [15] presented good results compared to previous studies by adding three loss functions in addition to the distillation method. That is, to minimize the occurrence of catastrophic aggregation in the CL process, both of the predictions between the t-model and the s-model were distilled using the KD technique.