# Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
## 4 Experiments
### Ablation Study  
In this section, we detail how we first conducted ablation experiments on the FD loss to validate its effectiveness as an independent module added to existing KD methods. Then, we investigated the impact of weighted class means (WCMs), feature selection (FS), and positive-negative sample adjustment (PNSA) on the FD loss, demonstrating the effectiveness of each component in the FD loss. Finally, we tested the impact of different training sets on FD loss.  
We trained teacher (ResNet-56) and student (ResNet-20) models on the CIFAR100 dataset and reported the accuracy (%) on their respective test sets. We used KD as the baseline, which is a logit distillation method.  
Table 4 and Figure 3 show comparisons of the performance of FD loss under different configurations. The baseline (CE+KL) achieves 70.66% accuracy, while adding FD loss as a standalone module (CE+KL+FD) boosts accuracy to 72.36%. This 2.4% improvement highlights FD loss's ability to complement existing distillation frameworks. Notably, when FD loss is used to replace the original module (CE+FD), the gain is smaller (71.09%), suggesting that FD loss works best as an auxiliary component rather than a complete replacement.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Methods** & **Top-1** & **Top-5** \\ \hline teacher & 73.31 & 91.42 \\ student & 69.75 & 89.07 \\ CRD [27] & 71.17 & 90.13 \\ ReviewKD * [13] & 71.21 & 90.21 \\ KD [4] & 70.66 & 89.88 \\ DKD * [11] & 71.24 & 90.23 \\ DKD++ * [18] & 71.42 & 90.12 \\ KD + FD (ours) & 71.03 & 90.18 \\ ReviewKD + FD (ours) & 71.48 & 90.32 \\ DKD + FD (ours) & **71.56** & **90.38** \\ \hline \hline \end{tabular}  
* represents the results we reproduced. Bold numbers indicate the best performance under this structure.  
\end{table}
Table 2: Top-1 and top-5 accuracy (%) on the ImageNet validation. We set ResNet-34 as the teacher and ResNet-18 as the student.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Methods** & **Top-1** & **Top-5** \\ \hline teacher & 76.16 & 92.86 \\ student & 68.87 & 88.76 \\ CRD [27] & 71.37 & 90.41 \\ ReviewKD * [13] & 72.13 & 90.86 \\ KD [4] & 68.58 & 88.98 \\ DKD [11] & 72.05 & 91.05 \\ ReviewKD++ * [18] & 72.44 & 90.83 \\ KD + FD (ours) & 72.13 & 90.18 \\ DKD + FD (ours) & 72.23 & **91.32** \\ ReviewKD + FD (ours) & **72.49** & 91.11 \\ \hline \hline \end{tabular}  
* represents the results we reproduced. Bold numbers indicate the best performance under this structure.  
\end{table}
Table 3: Top-1 and top-5 accuracy (%) on the ImageNet validation. We set [PERSON] as the teacher and MobileNet-V2 as the student.  
Table 5 and Figure 4 demonstrate the impact of weighted class means (WCMs), feature selection (FS), and positive-negative sample adjustment (PNSA) on the FD loss. It can be seen that using all three components in the FD loss yields the best performance (72.35%), while removing any one of them leads to varying degrees of performance degradation (71.97%, 71.71%, and 71.77%). Table 5 reveals synergies among LDN's three core modules, WCMs prepare denoised prototypes for FS to be selectively enforced. FS ensures that PNSA operates on contextually appropriate features (prototypes or raw), maximizing error correction precision, and PNSA's inversion mechanism amplifies the contrast between reliable and erroneous supervision, guided by WCM/FS filtering.  
Table 6 evaluates FD loss on a subset of CIFAR-100 where the teacher model predicts correctly (CIFAR-100-1). The minimal improvement (68.09% vs. 67.87%) indicates that FD loss primarily enhances performance by correcting erroneous supervision. This is further supported by the significant gains on the full dataset (72.36% vs. 70.66%), where teacher errors are prevalent. These findings underscore FD loss's role in mitigating error propagation rather than merely amplifying correct predictions.