# Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
## 4 Experiments and Analysis
### Discussion on Results  
Despite the improvement in performance compared to LSH, our algorithm still performs slightly worse than SSKD in heterogeneous model distillation. This can be attributed to two reasons. First, there are differences in the penultimate dimension between teachers and students in heterogeneous model distillation. The feature output of a heterogeneous teacher is less helpful to the student than that of a homogeneous teacher. Secondly, because the SSKD self-supervised image enhancement method increases the amount of data and requires pre-training of the teacher self-supervision module, the SSKD training time is longer. In contrast, as shown in Table 8, our algorithms have a shorter training time and lower memory usage. \begin{table}
\begin{tabular}{c c c c c c c} \hline Teacher & vgg13 & resnet50 & resnet50 & resnet32\(\times\)4 & resnet32\(\times\)4 & WRN-40-2 \\ Student & MobileNetV2 & MobileNetV2 & vgg8 & ShuffleNetV1 & ShuffleNetV2 & ShuffleNetV1 \\ Teacher & 74.64 & 73.34 & 79.42 & 79.42 & 79.42 & 75.61 \\ Student & 64.60 & 64.60 & 70.36 & 70.50 & 71.82 & 70.50 \\ \hline KD [10] & 67.37 & 67.35 & 73.81 & 74.07 & 74.45 & 74.83 \\ FitNet [11] & 64.14 & 63.16 & 70.69 & 73.59 & 73.54 & 73.73 \\ AT [17] & 59.40 & 58.58 & 71.84 & 71.73 & 72.73 & 73.32 \\ SP [19] & 66.30 & 68.08 & 73.34 & 73.48 & 74.56 & 74.52 \\ AB [18] & 66.06 & 67.20 & 70.65 & 73.55 & 74.31 & 73.34 \\ FT [21] & 61.78 & 60.99 & 70.29 & 71.75 & 72.50 & 72.03 \\ CRD [12] & 69.73 & 69.11 & 74.30 & 75.11 & 75.65 & 76.05 \\ CRD+KD [12] & 69.94 & 69.54 & 74.58 & 75.12 & 76.05 & 76.27 \\ SSKD [22] & 71.24 & 71.81 & 75.71 & 78.18 & 78.75 & 77.30 \\ \hline LSH * [13] & 68.38 & 68.34 & 74.48 & 75.63 & 76.56 & 76.54 \\ LSH-T & 68.71 & 68.74 & 74.45 & 76.23 & 76.69 & 76.89 \\ \(\blacktriangle\) & +0.33 & +0.40 & \(-\)0.03 & +0.60 & +0.13 & +0.35 \\ \hline \end{tabular}
\end{table} 
Table 6: Test accuracy (%) of the student network on CIFAR-100. The teacher and the student share different architectures. We use \({}^{*}\) to denote methods we re-ran five times using author-provided codes. The results of our method were run five times. Red represents the best, while blue represents the second best. \(\blacktriangle\) indicates the performance improvement effect of LSH-T (ours) over LSH. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline  & **Teacher** & **Student** & **KD [10]** & **AT [17]** & **CRD [12]** & **CRD+KD [12]** & **SSKD [22]** & **LSH [13]** & **LSH-T** \\ \hline Top-1 & 26.70 & 30.25 & 29.34 & 29.30 & 28.83 & 28.62 & 28.38 & 28.78 & 28.72 \\ Top-5 & 8.58 & 10.93 & 10.12 & 10.00 & 9.87 & 9.51 & 9.33 & 9.76 & 9.73 \\ \hline \end{tabular}
\end{table} 
Table 7: Top-1 and Top-5 error rates (%) on the ImageNet-2012 validation set. The teacher and student are resnet34 and resnet18. In Sections 4.2 and 4.3, we conducted ablation experiments on LSH-TL and LSH-T, respectively. Although these two algorithms cannot be utilized together, the experiments indicate that they have distinct application scenarios. At the same time, we summarize the underlying reasons for the excellent performance of LSH-TL and LSH-T:  
The superior performance of LSH-TL and LSH-T compared to other algorithms in homogeneous and heterogeneous model distillation is the effectiveness of the penultimate layer of features as knowledge. The model penultimate layer features are derived from the output of the backbone network features after pooling. Compared with other knowledge distillation algorithms based on intermediate layer features, the advantage is that more layers of neural networks process the penultimate layer features, and the amount of information contained is richer. As opposed to logits distillation, students prioritize learning \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Time** & **Memory Usage** \\ \hline SSKD & 7564.332 s & 3.3 GB \\ LSH-TL & 4047.144 s & 2.4 GB \\ LSH-T & 3870.336 s & 2.4 GB \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparison of SSKD, LSH-TL, and LSH-T on training time and memory usage.