# Few-Shot Federated Learning: A Federated Learning Model for Small-Sample Scenarios
## 4 Model System
### Model Learning Process
\[\underset{\theta}{min}L(f_{\theta})=\sum\limits_{(x_{i},y_{i})\in D_{query}} L(f_{\theta}(x_{i}),y_{i}) \tag{9}\]

If \(C\) represents the total number of categories, substituting Equation (8) results in the expanded expression below. \[ \begin{array}{ll}L(f_{\theta}(x_{i}),y_{i})&=-y_{i}\text{log}\frac{\exp \left(-d\big{(}f_{\theta}(x_{i}),\,p_{y_{i}}\big{)}\right)}{\sum\limits_{c=-1 }^{C}\exp\left(-d(f_{\theta}(x_{i}),\,p_{c})\right)}\\ &=-y_{i}\big{(}-d\big{(}f_{\theta}(x_{i}),p_{y_{i}}\big{)}\big{)}+\text{log} \sum\limits_{c=1}^{C}\exp\left(-d(f_{\theta}(x_{i}),p_{c})\right)\end{array}  \tag{10}\]

In the System Model section, the server-side aggregation of \(w_{c}\) is performed as follows:

\[w_{g}=\sum\limits_{c=1}^{K}\frac{w_{c}}{K} \tag{11}\]

The server aggregates the parameters \(w_{c}\) received from the \(K\) clients by summing them and then taking the average, resulting in the global model parameters \(w_{g}\). ### Personalized Knowledge Distillation Based on Student Model Classification Tasks

In the text on Personalized Knowledge Distillation, a scenario is considered where each client possesses different data samples with varying data distributions. Consequently, the final parameters obtained from model training, tailored to each client's specific classification tasks, also differ. If clients directly extract parameters from the global model provided by the server, they will inevitably acquire parameters related to other clients' classification tasks, which are irrelevant to their own. This directly impacts the clients' ability to conduct subsequent rounds of local training. To address this issue, when the server sends a non-initial global model to the clients, personalized knowledge distillation is employed on the client model. This allows clients to selectively extract parameters relevant to their local classification tasks, thereby avoiding the interference of unnecessary parameters in modeltraining. Initially, the classification task of the client model focuses on a specific set of categories \(C_{m}\), while the server focuses on a broader set of categories \(C_{b}\). The distillation target is for the client to distill knowledge from the server on the categories \(C_{m}\). Here, the output probabilities of the client model and the server model for a sample \(C_{x}(C_{x}\in C_{m})\) are defined as \(P_{c}(C_{x})\) and \(P_{s}(C_{x})\), respectively. The objective is to minimize the prediction probability difference between \(P_{c}(C_{x})\) and \(P_{s}(C_{x})\) as much as possible to achieve the effect of personalized distillation. The loss function is defined as follows [10; 23; 24; 25; 26; 27; 28]:

\[L_{KD}=-\sum_{C_{x}}P_{c}(C_{x})\log P_{s}(C_{x}) \tag{12}\]

where the output probability \(P_{c}(C_{x})\) for the client model is given by the following formula:

\[P_{c}(C_{x})=\frac{\exp\left(\frac{z(C_{x})}{T}\right)}{\sum_{k\in C_{m}}\exp \left(\frac{z(C_{k})}{T}\right)} \tag{13}\]

and the output probability \(P_{s}(C_{x})\) for the server model is defined as follows:

\[P_{s}(C_{x})=\frac{\exp\left(\frac{z^{\prime}(C_{x})}{T}\right)}{\sum_{k\in C_ {m}}\exp\left(\frac{z^{\prime}(C_{k})}{T}\right)} \tag{14}\]

Furthermore, the model's prediction probabilities are softened by utilizing the temperature parameter \(T\). Initially, the difference between the maximum logit output by the server model and the logit of the true category, known as the confidence discrepancy \(\Delta z_{x}\), is calculated. \[\Delta z_{x}=\max\left(C_{x}^{t}\right)-C_{x}^{t}*y_{x} \tag{15}\]

\(C_{x}^{t}\) represents the logit values for all categories by the server model on sample \(x\), and \(y_{x}\) is the true label of the sample.