# Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
## 4 Experiments
### Experimental Results
#### 4.2.2 ImageNet Classification  
We conducted further in-depth research on the effectiveness of the FD loss on the larger ImageNet dataset. Tables 2 and 3 present the performance of knowledge distillation on the ImageNet dataset. +FD indicates the integration of our new FD loss into existing methods. \(*\) represents the results we reproduced. From Tables 2 and 3, it is evident that incorporating the FD loss into existing knowledge distillation methods leads to performance improvements, regardless of whether it is a homogeneous or heterogeneous network. Furthermore, the performance enhancement is more pronounced in homogeneous networks, which aligns with the results observed on the CIFAR-100 dataset.  
Table 2 presents the top-1 and top-5 accuracy of LDN on ImageNet with ResNet-34 as the teacher and ResNet-18 as the student. The baseline KD method achieves 70.66% top-1 accuracy, while LDN + KD improves this to 71.03%. More notably, when integrated with DKD (a state-of-the-art logit distillation method), LDN achieves 71.56% top-1 accuracy--a 0.9% gain over the original DKD (70.66%). This improvement is consistent across both homogeneous (ResNet-34\(\rightarrow\)18) and heterogeneous (ResNet-50\(\rightarrow\)MobileNet-V2) architectures, demonstrating LDN's versatility. The top-5 accuracy also sees a significant boost, with LDN+DKD reaching 90.38%, outperforming the baseline by 0.5%. These results validate LDN's ability to enhance existing distillation frameworks without additional computational overhead.  
Table 3 extends the evaluation to a more challenging setting: distilling from ResNet-50 to MobileNet-V2. Here, LDN + ReviewKD achieves 72.49% top-1 accuracy, surpassing the baseline ReviewKD by 0.44%. The top-5 accuracy also improves from 90.83% to 91.11%. Notably, the performance gap between LDN and non-LDN methods is smaller in this heterogeneous setting (0.44% vs. 0.9% in homogeneous pairs), aligning with our earlier observation that architectural alignment enhances error correction. Despite this, LDN still delivers consistent gains, underscoring its robustness across diverse teacher-student pairs. Combining the results from Tables 2 and 3, we observe two key trends: (1) Homogeneous vs. heterogeneous gains: LDN achieves larger improvements in homogeneous architectures  
\begin{table}
\begin{tabular}{c c c c} \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c}{**Homogeneous**} & \multicolumn{2}{c}{**Heterogeneous**} \\  & **ResNet-56** & **ResNet-32\(\times\)4** & **ResNet-50** \\  & **ResNet-20** & **ResNet-8\(\times\)4** & **MobileNet-V2** \\ \hline teacher (T) & 72.34 & 79.42 & 79.34 \\ student (S) & 69.06 & 72.50 & 64.60 \\ \hline \multicolumn{4}{c}{Feature distillation methods} \\ \hline FitNet [24] & 69.21 & 73.50 & 63.16 \\ RKD [29] & 69.61 & 71.90 & 64.43 \\ PKT [30] & 70.34 & 73.64 & 66.52 \\ ReviewKD * [13] & 71.89 & 75.63 & 69.73 \\ \hline \multicolumn{4}{c}{Logit distillation methods} \\ \hline KD [4] & 70.66 & 73.33 & 67.65 \\ DIST [9] & 71.78 & 75.79 & 69.17 \\ DKD * [11] & 71.97 & 75.44 & 69.78 \\ \hline KD++ * [18] & 72.05 & 74.65 & 68.87 \\ DKD++ * [18] & 71.72 & 75.92 & 69.86 \\ KD + FD (Ours) & **72.36** & 75.27 & 69.32 \\ DKD + FD (Ours) & 71.98 & **76.33** & **70.02** \\ ReviewKD + FD (Ours) & 72.08 & 76.05 & 69.83 \\ \hline \end{tabular}  
* represents the results we reproduced. Bold numbers indicate the best performance under this structure.  
\end{table}
Table 1: Top-1 accuracy (%) on the CIFAR-100 validation.  
(e.g., ResNet-34\(\rightarrow\)18: +0.9%) compared to heterogeneous pairs (e.g., ResNet-50\(\rightarrow\)MobileNet-V2: +0.44%). This is attributed to the shared feature hierarchies in homogeneous networks, which facilitate more effective error correction through PNSA. (2) Compatibility with SOTA methods: LDN consistently boosts the performance of existing distillation frameworks (e.g., DKD, ReviewKD) without requiring architectural modifications or additional training costs. This plug-and-play compatibility makes LDN a practical solution for real-world applications.