# Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
## 3 Method
### Logitwise Distillation Network  
In KD and KD-based approaches, the student model is trained under the predictions of the teacher model, regardless of whether they are correct or not. Clearly, when the teacher makes mistakes, it is difficult for the student to correct the erroneous information on their own. Although [PERSON]'s method is simple and effective, it does not differentiate the confidence levels of different samples. Instead, it simply divides them into trustworthy and untrustworthy categories. However, we find that the ranking of ground truth in teacher logits accurately reflects the credibility of the teacher model's predictions for each sample. A higher ranking indicates greater credibility of the teacher model's predictions. Based on this observation, Logitwise Distillation Network (LDN) is proposed. As shown in Figure 1, in LDN, we introduce logit-ranking information from three aspects, weighted class mean, feature selection, and PNSA, to help the student model avoid incorrect supervision from the teacher's erroneous predictions. The class mean can be used as a strong classifier, but misclassified samples will obviously reduce the reliability of the class mean. Therefore, when calculating the class mean, logit information is introduced, and different weights are assigned to each sample based on the ranking of the ground truth in the logit information. This is called weighted class mean. The weighted class mean of teacher feature vectors is represented as Equations (6) and (7):  
\[{c_{k}^{t}}^{\prime}=\frac{1}{N}\sum_{i=1}^{N}bf_{i}^{t} \tag{6}\]  
Figure 1: General framework of the LDN. \[b=\text{softmax}\bigg{(}\frac{\text{rank}(y)}{\tau}\bigg{)} \tag{7}\]  
where \(b\) represents the weight, which is related to the ranking of the teacher logits for that sample. The confidence weight \(b\) for each sample is derived from the ranking of its ground truth label in the teacher's logits. Higher rankings (e.g., Top-1) yield larger \(b\), reflecting stronger confidence. To avoid extreme weighting, we normalize \(b\) using a temperature-scaled softmax over all samples within a class. The ranking of the ground truth label in teacher logits correlates with feature discriminability. Higher rankings indicate that the teacher's features for this sample are closer to the class centroid in the embedding space. Thus, weighting by \(b\) effectively emphasizes samples with well-clustered features, reducing noise from ambiguous or misclassified instances. In LDN, weighted class means are used to replace teacher features to regularize student features. This helps the student model achieve higher classification accuracy. For the majority of samples, using weighted class mean to regularize student features instead of the original teacher features will have a positive impact on the student model. However, for some samples with significant deviations in teacher predictions, i.e., samples with ground truth ranked lower in logits, opting for a weighted class mean may have a negative effect instead. Therefore, in LDN, a flexible approach is employed. This approach regulates student features by utilizing either the weighted class mean of teacher features or the original features. The choice is based on the ranking of samples' ground truth in the teacher logit information. When the ground truth in logits is ranked higher, using weighted class mean instead of teacher features will be beneficial for the student to learn typical features of that category. Conversely, if the ground truth in logits is ranked lower, using the original teacher features will yield better results. We refer to this part as feature selection. In general feature-based distillation methods, minimizing the \(L_{2}\) distance between the student and teacher features in the same layer allows the student to generate features that are close to the teacher's. In ideal cases, the student model can achieve performance similar to that of the teacher. However, when faced with different samples, different learning strategies should be adopted for the teacher features.