## 2 Related Works
### Knowledge Distillation Methods in Computer Vision  
While large-scale deep models have demonstrated remarkable achievements, their substantial computational complexity and extensive storage demands pose significant challenges for real-time deployment, particularly on resource-constrained devices [32]. To tackle this issue, [PERSON] [33] introduced model compression, a technique that transfers information from a large or ensemble model to train a smaller model without a substantial loss in accuracy. This method was later widely recognized as knowledge distillation. There are different forms of knowledge in the following categories: response-based knowledge, feature-based knowledge, and relation-based knowledge.  
**Response-Based Knowledge**. Response-based knowledge distillation typically involves the neural response of the teacher model's final output layer. The concept revolves around directly emulating the teacher model's ultimate prediction. Soft targets are widely recognized as the most popular form of response-based knowledge for image classification. They represent the class probabilities of the input and can be estimated using a softmax function [34]. In recent studies, further exploration has been conducted on response-based knowledge, focusing on utilizing ground-truth label information as conditional targets [35]. The idea of response-based knowledge is straightforward and effective, particularly in the context of \"dark knowledge\". Alternatively, the role of soft targets can be likened to the effects of label smoothing [36] or the use of regularizers in model training [37]. However, it primarily relies on the last layer's output, limiting its ability to capture important intermediate-level supervision. Given that soft logits directly impact the class probability distribution, it is important to note that the application of response-based knowledge distillation is primarily constrained to supervised learning scenarios.  
**Feature-Based Knowledge**. Deep neural networks excel in representation learning [38], gradually capturing more abstract features. Both the last layer's output and intermediate layer feature maps can supervise the training of the student model. Utilizing feature-based knowledge, especially for thinner and deeper networks, extends response-based knowledge effectively. [PERSON] [39] pioneered the use of intermediate representationsto enhance student model training. The core concept involves directly aligning the feature activations between the teacher and student models, providing valuable guidance for improved learning. Inspired by this idea, [PERSON] [40] transferred knowledge by matching the probability distribution in feature space to make it easier to transfer the teacher's knowledge. While feature-based knowledge transfer offers beneficial information for student model learning, there is still a need for further investigation on effectively selecting hint layers from the teacher model and guided layers from the student model [41].  
**Relation-Based Knowledge.** Response-based and feature-based knowledge distillation leverage specific layers' outputs in the teacher model. In contrast, relation-based knowledge distillation delves into the interdependencies between different layers or data samples. [PERSON] [42] introduced a robust and effective knowledge distillation approach by utilizing an instance relationship graph. The transferred knowledge within the instance relationship graph encompasses instance features, instance relationships, and feature space transformations across layers. [PERSON] [43] presented a knowledge distillation method focused on preserving similarities. Specifically, the method transfers similarity-preserving knowledge from teacher networks, which is derived from similar activations of input pairs. The student network then receives this knowledge while maintaining pairwise similarities.