# Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
## 1 Introduction  
Deep neural networks (DNNs) have come to have a very wide range of applications in computer vision in recent decades. However, powerful networks typically benefit from large model capacities, introducing high computational and storage costs [1, 2, 3]. To remedy this problem, one potential method is knowledge distillation (KD). The concept of KD was first proposed by [PERSON] et al. [4], which represents a range of methods for transferring knowledge from a heavy model (teacher) to a light model (student), which can improve the performance of the light model without introducing additional costs. Generally, KD can be classified into two categories: logit-based distillation [5, 6, 7, 8, 9, 10, 11, 12] and feature-based distillation [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. Logit-based distillation obtains the student model by minimizing the KL divergence between logit layers of student networks and teacher networks based on training data. In general, most of the methods on logit-based distillation methods focus on regularization or optimization methods. [PERSON] [7] proposed a mutual learning approach that trains both the student and the teacher simultaneously. TAKD [8] was proposed by [PERSON] to introduce a medium-sized network called \"teacher assistant\" that serves as a bridge between the teacher and the student. [PERSON] [9] proposed a correlation-based loss to explicitly capture the intrinsic inter-class relationships of the teacher. BookKD [10] was proposed by [PERSON] et al. By decoupling the process of knowledgedistillation into two parts, \"book-making\" and \"book-learning\", it significantly reduces the resource consumption of knowledge distillation. Additionally, there are some works that focus on explaining logit distillation methods and that propose DKD [11] to decouple the KL loss into separate meaningful parts. logit-based distillation is simple and easy to use, and it significantly improves performance. However, it may not fully exploit the potential of the teacher model, thereby conveying limited information through the logits. For instance, features from other layers of the teacher network remain underutilized. Compared to logit-based methods, feature-based methods align features in the intermediate layers, extracting knowledge from the intermediate feature maps with more diversity, flexibility, and selectivity. Most feature-based distillation methods [13; 14; 16; 17] train the student model by encouraging similarity between intermediate-layer features and teacher features, such as minimizing the \(L_{2}\) distance between features. [PERSON] [13] investigated the cross-level factors of the connection paths between the teacher and student networks and reveals their importance. KD++ [18] was proposed by [PERSON] to introduce a correlation loss that encourages the student to generate features with large norms and directions consistent with the teacher's class means. [PERSON] proposed a novel adaptive multi-teacher multi-level knowledge distillation learning framework (AMTML-KD) [19] that allows students to learn multi-level knowledge from multiple teachers. MulKD [20] was proposed by [PERSON] to use sequential learning and parallel learning to gradually generate more suitable teacher models, thereby promoting the transfer of knowledge to student models. Most feature-based methods can achieve better performance (significantly higher than logit-based methods), but forcing the student to generate similar features to the teacher may not correctly extract the teacher's knowledge. If the student simply imitates the teacher, generating similar prediction results or features, they might make mistakes that are similar to the those of the teacher. To reduce the issue of incorrect supervision caused by teacher prediction errors, we propose a Logitwise Distillation Network (LDN). The ranking of ground truth in logits reflects the confidence of the teacher model's intermediate-layer features on that sample.