# Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
## 3 Method
### Fitting the Ground-Truth Distributions of Features (LSH-T)  
When the distillation training stage is at epoch 0, the penultimate layer feature losses between some distillation teacher-student groups are quite low, which is almost equal to the feature loss value of some distillation teacher-student groups that have completed the training. We show some examples in Figure 4. The penultimate layer feature losses between the pre-trained teachers (vgg13, resnet32\(\times\)4, and WRN-40-2) and untrained students are significantly lower. On the contrary, the penultimate layer feature losses are relatively normal when resnet110 and resnet56 are used as pre-training teachers. We believe that overly low feature loss is harmful to feature distillation in the penultimate layer, which limits the potential of feature mimicking. This phenomenon can be explained using Figure 5. In normal circumstances, features that are more relevant to the image tend to receive higher feature values. In the real feature distribution of images, feature A is the most important feature and feature B is the least important feature. The pre-trained resnet110 includes feature recognition ability and can assign appropriate feature values to each feature. Untrained resnet20's feature recognition ability is relatively low, and it is impossible to determine which features are important. It assigns relatively average and lower feature values to all features. Therefore, when resnet110 and resnet56 serve as teachers in Figure 5 they maintain a relatively high feature loss when facing all students. In the penultimate layer of feature distillation, the student gradually reduces feature loss by learning the feature recognition ability from the teacher. In abnormal circumstances, the pre-trained vgg13 assigns low feature values to all features, even though the feature value of the most important feature (A) remains the highest compared to other features (B, C). However, the feature loss between teacher and student remains low. This makes feature mimicking more difficult during the distillation process. In order to solve the above problems, we introduce the feature temperature (\(\tau\)) as a corrective factor to improve the teacher's feature output:  
\[L_{mse}(\tau f_{t},f_{s})=\frac{1}{nD}\sum_{i=1}^{n}\bigg{\|}\tau f_{t}(x_{i} )-f_{s}(x_{i})\bigg{\|}_{2}^{2}. \tag{12}\]  
As shown in Figure 5. By introducing the feature temperature parameter, we can enhance the distinguishability between important features and other features. Make the overall feature distribution of the teacher closer to the ground-truth distribution of features and release the potential of feature mimicking. The loss function of LSH-T is as follows:  
\[L_{LSH-\tau}=\alpha(L_{lsh}(\tau f_{t},f_{s})+L_{mse}(\tau f_{t},f_{s}))+\beta _{1}L_{ce}. \tag{13}\]  
By introducing the feature temperature parameter, we widen the gap between the feature values of important and non-important features. However, the relative importance of features remains unchanged. Both the KD [10] temperature parameter and our feature temperature parameter are designed to make the transmitted knowledge better match  
Figure 4: The MSE loss values (\(y\)-axis) between teachers (different colored lines) and students (\(x\)-axis) when the distillation training stage is at epoch 0 epoch. the ground-truth distribution of images. The reason for using distillation temperature is that the teacher's classified output is too absolute after Softmax. Low-probability classes receive fewer weights, making it challenging to provide similar information for low-probability classes during distillation. In contrast, our design of feature temperature aims to address the issue of the teacher's feature distribution being too flat, resulting in minimal differences between the features represented by feature values. During the distillation process, the teacher cannot provide distinguishing information between important and non-important features.