# Few-Shot Federated Learning: A Federated Learning Model for Small-Sample Scenarios
## 3 Preliminary  
### Knowledge Distillation  
The concept of knowledge distillation was first introduced by [PERSON] et al. in [22], comprising a system of teacher and student models [23; 24; 25; 26]. The so-called teacher model refers to a large, structurally complex, and redundantly trained network that acts as the \"teacher\" during training, instructing a simpler, under-trained \"student\" model to achieve a performance as close as possible to that of the teacher model.  
This process is realized through the use of soft and hard labels along with corresponding loss functions. Specifically, it begins by calculating the soft output of the teacher model, which is the raw prediction values, or logits, from the last fully connected layer of the neural network--values that have not been activated by the softmax function. Using the probability distribution after applying the softmax function would limit the knowledge transferred to the student model and might even have a negative effect by amplifying errors. These prediction values represent the teacher model's confidence level for each class.  
\[q_{i}=\frac{exp\bigg{(}\frac{z_{i}^{T}}{T}\bigg{)}}{\sum_{j}exp\bigg{(}\frac{ z_{i}^{T}}{T}\bigg{)}} \tag{1}\]  
Here, \(z_{i}^{T}\) denotes the logit output of the teacher model for the \(i\)th class, and \(T\) is the temperature parameter that controls the smoothness of the output probability distribution. It is evident that a smaller \(T\) value, which brings the function closer to the softmax function, makes \(q_{i}\) more similar to the probability distribution after activation. Conversely, a larger \(T\) value results in a smoother probability distribution curve from the softmax output, allowing the student model to acquire knowledge with richer information entropy.  
After obtaining the logits from the teacher model, the student model's logits are calculated in a similar manner. Here, \(z_{i}^{S}\) represents the logit output of the student model for the \(i\)th class.  
\[p_{i}=\frac{exp\bigg{(}\frac{z_{i}^{S}}{T}\bigg{)}}{\sum_{j}exp\bigg{(}\frac{ z_{i}^{S}}{T}\bigg{)}} \tag{2}\]  
Subsequently, a loss function is employed to enable the student model to absorb and compress the knowledge from the teacher model.  
\[L_{soft}=KL(Q_{i},P_{i})\times T^{2} \tag{3}\]  
Here, \(Q_{i}\) and \(P_{i}\) represent the probability distributions of the teacher and student models, respectively. The difference between these probability distributions is calculated using the Kullback-Leibler divergence, with the temperature parameter used to adjust it, ensuring that the gradients do not become too small as the temperature value increases.  
Throughout the knowledge distillation process, in addition to computing the soft target loss, it is also necessary to calculate the hard target loss. The hard target loss, also known as hard label loss, enables the student model to learn not only the decision boundaries and related knowledge from the teacher model through soft labels, but also to make accurate predictions from the true labels. Typically, the cross-entropy loss function is used to calculate the hard label loss between the true labels \(y\) and the probability distribution \(P_{i}\) of the student model:  
\[L_{hard}=CrossEntropy(y,P_{i}) \tag{4}\]  
Therefore, by introducing a parameter \(\alpha\) to control the weights, a total loss function for knowledge distillation can be obtained:  
\[L_{total}=\alpha L_{hard}+(1-\alpha)L_{soft} \tag{5}\]