# Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
## 3 Method  
The overall framework is shown in Figure 2. For a given image data \(x\), all images are cropped to the same size via image preprocessing operations and fed into the teacher backbone network and student backbone network respectively. Teachers usually refer to complex networks that have been pre-trained and have a certain feature recognition capability. Students refer to untrained, smaller, as well as lighter networks relative to teachers. In the teacher and student backbone networks, the data undergo a layer-by-layer feature extraction that captures the main information in the data and outputs a data matrix in each layer. The data are processed through the backbone network and changed into a one-dimensional matrix through the average pool (avgpool) layer, which outputs the features of the penultimate layer. The teacher backbone network can extract features \(f_{t}\) which are the penultimate layer feature outputs of the teacher network. The student backbone network also extracts the penultimate features \(f_{s}\) and fits \(f_{t}\) for knowledge distillation. When conducting knowledge distillation using the model's penultimate layer outputs, especially in the case of heterogeneous model distillation, we encounter the issue of mismatched dimensions between the student's feature dimension \(D_{s}\) and the teacher's feature dimension \(D_{t}\). Therefore, a linear embedding layer \(fcl_{s}\) is added to complete the dimension matching between teacher and student:  
\[fcl_{s}=wf+b. \tag{1}\]  
\(w\in\mathbb{R}^{D_{s}\times D_{t}}\) and \(b\in\mathbb{R}^{D_{t}}\) are the weight and bias parameters of the embedded layer which are updated via loss backward in training. After the student backbone network output enters \(fcl_{s}\), it will output the student feature with dimension \(D_{t}\). The loss function used is as follows. \(L_{mse}\) is defined as the mean squared error loss:  
\[L_{mse}(f_{t},f_{s})=\frac{1}{nD}\sum_{i=1}^{n}\lVert f_{t}(x_{i})-f_{s}(x_{i} )\rVert_{2}^{2} \tag{2}\]  
where \(n\) represents the number of samples. \(D\) denotes the feature dimension. \(x\) represents input image data. \(f_{t}(x_{i})\) and \(f_{s}(x_{i})\), respectively, represent the penultimate layer feature outputs of the teacher model and the student model for the \(ith\) image in the dataset. \(L_{LSH}\) is locality-sensitive hashing knowledge distillation:  
\[L_{LSH}=\alpha(L_{lsh}(f_{t},f_{s})+L_{mse}(f_{t},f_{s})+\beta_{1}L_{ce}. \tag{3}\]  
\(L_{LSH}\) builds upon \(L_{mse}\) by introducing locality-sensitive hashing loss (\(L_{lsh}\)). The cross-entropy loss \(L_{ce}\) is used as the classification loss. \(\alpha\) and \(\beta_{1}\) are balancing weights. \(L_{lsh}\) is locality-sensitive hashing loss:  
\[L_{lsh}=-\frac{1}{nN}\sum_{i=1}^{n}\sum_{j=1}^{N}\left[h_{j}\log p_{j}+(1-h_{j })\log(1-p_{j})\right] \tag{4}\]  
where \(h\) represents the teacher's hash and \(p\) represents the student's hash:  
\[h=sign(W^{T}f_{t}+B), \tag{5}\]  
\[p=\sigma(W^{T}f_{s}+B). \tag{6}\]  
\(W\) is the weight sampled from the Gaussian distribution and \(B\) is the bias. Equation (5) generates \(N\) hashes for each feature. \[sign(x)=\left\{ \begin{array}{ll}1,&\quad if\ x>0\\ 0,&\text{otherwise}\end{array} \right. \tag{7}\]\[\sigma(x)=\frac{1}{1+\exp(-x)} \tag{8}\]  
LSH extracts the correctly classified teacher features through the _sign_ function. The calculation of the hash value can be implemented via the linear layer in the neural network. The purpose is to match the hash of the student features with the hash of the teacher's correct classification features. When the teacher and the student have completed their respective feature extraction, the penultimate layers of features are mapped into a classifier equal in size to the number of categories in the dataset. The distribution of image categories judged by the network is the output in the classifier. The class with the largest category value obtained is the class to which the network has judged this image to belong.