# Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
## 5 Conclusions  
In this paper, we argued that the existing work on penultimate layer feature distillation overlooked the efforts focused on fitting the ground-truth distribution of the image. To address this issue, we proposed two penultimate layer feature distillation algorithms LSH-TL and LSH-T.  
LSH-TL focuses on fitting the ground-truth distribution based on classification. It supervises the classification output by replacing the traditional dataset labels with teacher classification labels that are more consistent with the ground-truth distribution of classes; LSH-T corrects the low-feature output distribution of the teacher by designing the feature temperature. Make the teacher feature output distribution more consistent with the ground-truth distribution of features. On the Cifar-100 dataset, LSH-TL outperforms the base LSH in nine out of a total of thirteen homogeneous and heterogeneous distillation model groups, providing a maximum performance gain of 0.17% for LSH. LSH-T outperforms the base LSH in eight out of a total of thirteen homogeneous and heterogeneous distillation model groups, providing a maximum performance gain of 0.60% for LSH. The two methods have their application areas and compensate for each other's shortcomings. LSH-TL can provide a performance gain effect in the three-group homogeneous model of resnet where LSH-T distillation performance is poor. LSH-T can also obtain good distillation performance results on distillation teacher-student groups such as vgg13-vgg8 where LSH-TL performance is not satisfactory. Experimental results on the large dataset ImageNet-2012 show that LSH-TL and LSH-T are lower than the original LSH algorithm in both Top-1 and Top-5 error rates. These experimental results validate the effectiveness of our ground-truth distribution fitting approach.  
Our work still has some limitations. In LSH-TL, it is hard to eliminate the effect of teacher output mislabeling on the distillation performance of the algorithm. In LSH-T, the characteristic temperature parameters need to be set manually, which makes it difficult to select the optimal parameters for all distillation model groups. And, the distillation performance is not good after the fusion of the two methods.  
Teacher output mislabeling affects the performance of the LSH-TL algorithm because teacher labels are used in LSH-TL to supervise student classification output. Students' judgment is poor when their performance is low, and students will receive all of the teacher's output mislabels, causing them to learn the wrong knowledge and affecting the algorithm's distillation performance. To address this issue, future work will explore how to incorporate more efficient supervised approaches such as self-supervised learning to compensate for the problem of teachers outputting incorrect labels. In the following works, we will explore more ways to improve our algorithms, such as incorporating a dynamic search for optimal intervals for feature temperature parameters, and combining the two algorithms into one to become a more adaptable algorithm.  
Conceptualization, J.L. and Z.T.; methodology, J.L.; software, J.L.; validation, J.L., K.C. and Z.C.; formal analysis, J.L.; investigation, J.L.; resources, Z.T.; data curation, J.L.; writing--review and editing, J.L. and Z.T.; visualization, J.L. and K.C.; supervision, Z.T.; project administration, Z.T.; funding acquisition, Z.T. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the National Natural Science Foundation of China under grant 62171145 and by the Guangxi Natural Science Foundation under Grant 2021 GXNSFAA220058.  
Not applicable.  
The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.  
The authors declare no conflicts of interest.