# Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
## 4 Experiments and Analysis
### Experiment on Fitting the Ground-Truth Distributions of Features (LSH-T)  
In Table 5, LSH-T outperforms LSH [13] in terms of distillation performance in three out of seven groups of homogenous model distillation. One group achieves the best performance, while three groups achieve second-best performance. We have anticipated that the inclusion of the temperature parameter in resnet's homogeneous model \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **Teacher** & **Student** & **KD [10]** & **AT [17]** & **CRD [12]** & **CRD+KD [12]** & **SSKD [22]** & **LSH [13]** & **LSH-TL** \\ \hline Top-1 & 26.70 & 30.25 & 29.34 & 29.30 & 28.83 & 28.62 & 28.38 & 28.78 & 28.64 \\ Top-5 & 8.58 & 10.93 & 10.12 & 10.00 & 9.87 & 9.51 & 9.33 & 9.76 & 9.70 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Top-1 and Top-5 error rates (%) on the ImageNet-2012 validation set. The teacher and student are resnet34 and resenet18. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline Teacher & WRN-40-2 & WRN-40-2 & resnet56 & resnet110 & resnet110 & resnet32\(\times\)4 & vgg13 \\ Student & WRN-16-2 & WRN-40-1 & resnet20 & resnet20 & resnet32 & resnet8\(\times\)4 & vgg8 \\ Teacher & 75.61 & 75.61 & 72.34 & 74.31 & 74.31 & 79.42 & 74.64 \\ Student & 73.26 & 71.98 & 69.06 & 69.06 & 71.14 & 72.50 & 70.36 \\ \hline KD [10] & 74.92 & 73.54 & 70.66 & 70.67 & 73.08 & 73.33 & 72.98 \\ FitNet [11] & 73.58 & 72.24 & 69.21 & 68.99 & 71.06 & 73.50 & 71.02 \\ AT [17] & 74.08 & 72.77 & 70.55 & 70.22 & 72.31 & 73.44 & 71.43 \\ SP [19] & 73.83 & 72.43 & 69.67 & 70.04 & 72.69 & 72.94 & 72.68 \\ AB [18] & 72.50 & 72.38 & 69.47 & 69.53 & 70.98 & 73.17 & 70.94 \\ FT [21] & 73.25 & 71.59 & 69.84 & 70.22 & 72.37 & 72.86 & 70.58 \\ FSP [20] & 72.91 & n/a & 69.65 & 70.11 & 71.89 & 72.62 & 70.23 \\ CRD [12] & 75.48 & 74.14 & 71.16 & 71.46 & 73.48 & 75.51 & 73.94 \\ CRD+KD [12] & 75.64 & 74.38 & 71.63 & 71.56 & 73.75 & 75.46 & 74.29 \\ SSKD [22] & 75.55 & 75.50 & 71.00 & 71.27 & 73.60 & 76.13 & 74.90 \\ \hline LSH * [13] & 76.30 & 74.51 & 71.34 & 71.59 & 73.97 & 76.71 & 74.50 \\ LSH-TL & 76.42 & 74.50 & 71.50 & 71.81 & 74.10 & 76.73 & 74.11 \\ (ours) & +0.12 & \(-\)0.01 & +0.16 & +0.12 & +0.13 & +0.02 & \(-\)0.39 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Test accuracy (%) of the student network on CIFAR-100. The teacher and the student share similar architectures. We use * to denote methods we re-ran five times using author-provided codes. The results of our method were run five times. Red represents the best, while blue represents the second best. \(\blacktriangle\) indicates the performance improvement effect of LSH-TL (ours) over LSH. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Teacher & vgg13 & resnet50 & resnet50 & resnet32\(\times\)4 & resnet32\(\times\)4 & WRN-40-2 \\ Student & MobileNetV2 & MobileNetV2 & vgg8 & ShuffleNetV1 & ShuffleNetV2 & ShuffleNetV1 \\ Teacher & 74.64 & 79.34 & 79.34 & 79.42 & 79.42 & 79.42 & 75.61 \\ Student & 64.60 & 64.60 & 70.36 & 70.50 & 71.82 & 70.50 \\ \hline KD [10] & 67.37 & 67.35 & 73.81 & 74.07 & 74.45 & 74.83 \\ FitNet [11] & 64.14 & 63.16 & 70.69 & 73.59 & 73.54 & 73.73 \\ AT [17] & 59.40 & 58.  
distillation would degrade the distillation performance. The data from Figure 4 indicate that when resnet is used as the teacher model, the feature loss values between teacher and student are within a reasonable range. The feature distribution of the teacher model exhibits sufficient feature discriminability. However, introducing feature temperature will disrupt the original feature output. The finding that the addition of feature temperature to the distillation process of the resnet homogeneous model leads to a decrease in distillation performance further supports the argument made in this study that the feature loss of the resnet as a teacher model is within the normal range. In Table 6, LSH-T outperforms LSH [13] in five out of six groups of heterogeneous model distillation. Three groups achieve second-best performance.