Parameterized Knowledge Transfer for Personalized Federated Learning
Year: 2021

Abstract:
In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others’ local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others’ personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

heading: 1 Introduction
text: Federated Learning (FL) [1] has emerged as an efficient paradigm to collaboratively train a shared machine learning model among multiple clients without directly accessing their private data. By periodically aggregating parameters from the clients for global model updating, it can converge to high accuracy and strong generalization. FL has shown its capability to protect user privacy while there remains a crucial challenge that significantly degrades the learning performance, i.e., statistic heterogeneity in users’ local datasets. Given the Non-Independent and Identically Distributed (Non-IID) user data, the trained global model often cannot be generalized well over each client [2–5].
To deal with the above issues, employing personalized models appears to be an effective solution in FL, i.e., personalized federated learning (pFL). Recent works regarding pFL include regularizationbased methods [6–8] (i.e., pFedMe [6], L2SGD [7], FedAMP [8]), meta-learning-based Per-FedAvg
∗Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
[9] and cluster-based IFCA [10, 11]. However, in order to aggregate the parameters from all clients, it is inevitable for them to have identical model structure and size. Such constraints would prevent status quo pFL methods from further application in practical scenarios, where clients are often willing to own unique models, i.e., with customized neural architectures to adapt to heterogeneous capacities in computation, communication and storage space, etc. Motivated by the paradigm of Knowledge Distillation (KD) [12–16] that knowledge can be transferred from a neural network to another via exchanging soft predictions instead of using the whole model parameters, KD-based FL training methods have been studied [12, 14–19] to collaboratively train heterogeneous models in a privacy-preserving way. However, these works have neglected the further personalization requirement of FL clients, which can be well satisfied via the personalized knowledge transfer for heterogeneous FL users.
In this paper, we seek to develop a novel training framework that can accommodate heterogeneous model structures for each client and achieve personalized knowledge transfer in each FL training round. To this end, we formulate the aggregation phase in FL to a personalized group knowledge transfer training algorithm dubbed KT-pFL, whose main idea is to allow each client to maintain a personalized soft prediction at the server that can be updated by a linear combination of all clients’ local soft predictions using a knowledge coefficient matrix. The principle of doing so is to reinforce the collaboration between clients with similar data distributions. Furthermore, to quantify the contribution of each client to other’s personalized soft prediction, we parameterize the knowledge coefficient matrix so that it can be trained simultaneously with the models following an alternating way in each iteration round.
We show that KT-pFL not only breaks down the barriers of homogeneous model restriction, which requires to transfer the entire parameters set in each round, whose data volume is much larger than that of the soft prediction, but also improves the training efficiency by using a parameterized update mechanism. Experimental results on different datasets and models show that our method can significantly improve the training efficiency and reduce the communication overhead. Our contributions are:
• To the best of our knowledge, this paper is the first to study the personalized knowledge transfer in FL. We propose a novel training framework, namely, KT-pFL, that maintains a personalized soft prediction for each client in the server to transfer knowledge among all clients. • To encourage clients with similar data distribution to collaborate with each other during the training process, we propose the ‘knowledge coefficient matrix’ to identify the contribution from one client to others’ local training. To show the efficiency of the parameterized method, we compared KT-pFL with two non-parameterized learning methods, i.e., TopK-pFL and Sim-pFL, which calculate the knowledge coefficient matrix on the cosine similarity between different model parameters. •We provide theoretical performance guarantee for KT-pFL and conduct extensive experiments over various deep learning models and datasets. The efficiency superiority of KT-pFL is demonstrated by comparing our proposed training framework with traditional pFL methods.

heading: 2 Related Work
text: Personalized FL with Homogeneous Models. Recently, various approaches have been proposed to realize personalized FL with homogeneous local model structure, which can be categorized into three types according to the number of global models applied in the server, i.e., single global model, multiple global models and no global model.
single global model type is a close variety of conventional FL, e.g., FedAvg [1], that combine global model optimization process with additional local model customization, and consist of four different kinds of approaches: local fine-tuning [20–23], regularization (e.g., pFedMe [6], L2SGD [7, 24], Ditto [25]), hybrid local and global models [11, 26, 27] and meta learning [9, 28]. All of these pFL methods apply a single global model, and thus limit the customized level of the local model at the client side. Therefore, some researchers [8, 10, 11] propose to train multiple global models at the server, where clients are clustered into several groups according to their similarity and different models are trained for each group. FedAMP [8] and FedFomo [29] can be regarded as special cases of the clustered-based method that each client owns a personalized global model at the server side. As a contrast, some literature waive the global model to deal with the heterogeneity problem [30, 31], such as multi-task learning based (i.e., MOCHA [31]) and hypernetwork-based framework (i.e.,
FedHN [30]). However, all these methods require aggregating the model parameters from the clients, who have to apply identical model structure and size, which hinders further personalization, e.g., employing personalized model architectures for heterogeneous clients is not feasible.
Heterogeneous FL and Knowledge Distillation. To enable heterogeneous model architectures in FL, Diao et al. [32] propose to upload a different subset of global model to the server for aggregation with the objective to produce a single global inference model. Another way of personalization is to use Knowledge Distillation (KD) in heterogeneous FL systems ([12, 14, 15, 17–19, 33, 34]). The principle is to aggregate local soft-predictions instead of local model parameters in the server, whereby each client can update the local model to approach the averaged global predictions. As KD is independent with model structure, some literature [13, 16] are proposed to take advantage of such independence to implement personalized FL with heterogeneous models at client sides. For example, Li et al. [13] propose FedMD to perform ensemble distillation for each client to learn well-personalized models. Different from FedMD that exchanging soft-predictions between the clients and the server, FedDF [16] first aggregates local model parameters for model averaging at the server side. Then, the averaged global models can be updated by performing knowledge transfer from all received (heterogeneous) client models.
To sum up, most of these schemes construct an ensembling teacher by simply averaging the teachers’ soft predictions, or by heuristically combining the output of the teacher models, which are far away from producing optimal combination of teachers. In our framework, KD is used in a more efficient way that the weights of the clients’ soft predictions are updated together with the model parameters during every FL training iteration.

heading: 3 Problem Formulation
text: We aim to collaboratively train personalized models for a set of clients applying different model structures in FL. Consider supervised learning whose goal is to learn a function that maps every input data to the correct class out of C possible options. We assume that there are N clients, and each client n can only access to his private dataset Dn := {xni , yi}, where xi is the i-th input data sample, yi is the corresponding label of xi, yi ∈ {1, 2, · · · , C}. The number of data samples in dataset Dn is denoted by Dn. D = {D1,D2, · · · ,DN}, D = ∑N n=1Dn. In conventional FL, the goal of the learning system is to learn a global model w that minimizes the total empirical loss over the entire dataset D:
min w L(w) := N∑ n=1 Dn D Ln(w), where Ln(w) = 1 Dn Dn∑ i=1 LCE(w;xi, yi), (1)
where Ln(w) is the n-th client’s local loss function that measures the local empirical risk over the private dataset Dn and LCE is the cross-entropy loss function that measures the difference between the predicted values and the ground truth labels.
However, this formulation requires all local clients to have a unified model structure, which cannot be extended to more general cases where each client applies a unique model. Therefore, we need to reformulate the above optimization problem to break the barrier of homogeneous model structure. Besides, given the Non-IID clients’ datasets, it is inappropriate to just minimize the total empirical loss (i.e., minw1,··· ,wN L(w1, · · · ,wN ) := ∑N n=1 Dn D Ln(w
n)). To that end, we propose the following training framework: Definition 3.1. Let s(wn, x̂) denote the collaborative knowledge from client n, and x̂ denote a data sample from a public dataset Dr that all clients can access to. Define the personalized loss function of client n as
Lper,n(wn) := Ln(wn) + λ ∑ x̂∈Dr LKL
( N∑
m=1
cmn · s(wm, x̂), s(wn, x̂) ) , (2)
where λ > 0 is a hyper-parameters, LKL stands for Kullback–Leibler (KL) Divergence function and is added to the loss function to transfer personalized knowledge from a teacher to another. cmn is the knowledge coefficient which is used to estimate the contribution from client m to n. The second term in (2) allows each client to build his own personalized aggregated knowledge in the server and enhance the collaboration effect between clients with large c. The concept of collaborative
knowledge can either refer to soft predictions or model parameters depending on the definition in different situations. For example, s(wn, x̂) can be deemed to be a soft prediction of the client n, which are calculated with the softmax of logits zn, i.e., s(wn, x̂) = exp(z n c /T )∑C
c=1 exp(z n c /T ) , logits zn is the output of the last fully connected layer on client n’s model, T is the temperature hyperparameter of the softmax function.
Definition 3.2. We define c ∈ RN×N as the knowledge coefficient matrix:
c =  c11 c12 · · · c1N c21 c22 · · · c2N ... ... . . . ...
cN1 cN2 · · · cNN  . (3) Our objective is to minimize
min w,c L(w, c) := N∑ n=1 Dn D Lper,n(wn) + ρ‖c− 1 N ‖2, (4)
where w = [w1, · · · ,wN ] ∈ R ∑N
n=1 dn is the concatenated vector of all weights, dn represents the dimensions of model parameter wn. 1 ∈ Rn2 is the identity matrix whose elements are all equal to 1. The second term in (4) is a regularization term that ensures generalization ability of the whole learning system. Without the regularization term, a client with a completely different data distribution tends to set large values of the knowledge coefficient (i.e., equal to 1), and no collaboration will be conducted during training in this case. ρ is a regularization parameter that is larger than 0.

heading: 4 KT-pFL Algorithm
text: In this section, we introduce the proposed KT-pFL algorithm, where the local model parameters and knowledge coefficient matrix are updated alternatively. To enable personalized knowledge transfer in FL, we train personalized models locally according to the related collaborative knowledge. Insert (2) into (4), and we can design an alternating optimization approach to solve (4), that in each round we fix either w or c by turns, and optimize the unfixed one following an alternating way until a convergence point is reached.
Update w: In each communication round, we first fix c and optimize (train) w for several epochs locally. In this case, updating w depends on both the private data (i.e., LCE on Dn, n ∈ [1, · · · , N ]), that can only be accessed by the corresponding client, and the public data (i.e., LKL on Dr), which is accessible for all clients. We propose a two-stage updating framework for w:
Algorithm 1 KT-pFL Algorithm Input: D, Dr, η1, η2, η3 and T Output: w = [w1, · · · ,wN ]
1: Initialize w0 and c0 2: procedure SERVER-SIDE OPTIMIZATION 3: Distribute w0 and c0 to each client 4: for each communication round t ∈ {1, 2, ..., T} do 5: for each client n in parallel do 6: wnt+1 ← ClientLocalUpdate(n,wnt , ct,n) 7: Update knowledge coefficient matrix c via (7) 8: Distribute ct+1 to all clients 9: procedure CLIENTLOCALUPDATE(n,wnt , ct,n)
10: Client n receives wnt and cn from the server 11: for each local epoch i from 1 to E do 12: for mini-batch ξt ⊆ Dn do 13: Local Training: update model parameters on private data via (5) 14: for each distillation step j from 1 to R do 15: for mini-batch ξr,t ⊆ Dr do 16: Distillation: update model parameters on public data via (6)
return local parameters wnt+1
• Local Training: Train w on each client’s private data by applying a gradient descent step:
wn ← wn − η1∇wnLn(wn; ξn), (5) where ξn denotes the mini-batch of data Dn used in local training, η1 is the learning rate.
• Distillation: Transfer knowledge from personalized soft prediction to each local client based on public dataset:
wn ← wn − η2∇wnLKL
( N∑
m=1
c∗,Tm · s(wm, ξr), s(wn, ξr)
) , (6)
where ξr denotes the mini-batch of public data Dr, and η2 is the learning rate. c∗m = [cm1, cm2, · · · , cmN ] is the knowledge coefficient vector for client m, which can be found in m-th row of c. Note that all collaborative knowledge and knowledge coefficient matrix are required to obtain the personalized soft prediction in this stage, which can be collected in the server.
Update c: After updating w locally for several epochs, we turn to fix w and update c in the server.
c← c− η3λ N∑
n=1
Dn D ∇cLKL
( N∑
m=1
cm · s(wm,∗, ξr), s(wn,∗, ξr) ) − 2η3ρ(c− 1
N ), (7)
where η3 is the learning rate for updating c.
Algorithm 1 demonstrates the proposed KT-pFL algorithm and the idea behind it is shown in Figure 1. In every communication round of training, the clients use local SGD to train several epochs based on the private data and then send the collaborative knowledge (e.g., soft predictions on public data) to the server. When the server receives the collaborative knowledge from each client, it aggregates them to form the personalized soft predictions according to the knowledge coefficient matrix. The server then sends back the personalized soft prediction to each client to perform local distillation. The clients then iterate for multiple steps over public dataset 2. After that, the knowledge coefficient matrix is updated in the server while fixing the model parameters w.
Performance Guarantee. Theorem 4.1 provides the performance analysis of the personalized model when each client owns Non-IID data. Detailed description and derivations are deferred to Appendix A. Theorem 4.1. Denote the n-th local distribution and its empirical distribution by Dn and D̂n respectively, and the hypothesis h ∈ H trained on D̂n by hD̂n . There always exist c ∗ m,n,m = 1, . . . , N ,
2Note that our method can work over both labeled and unlabeled public datasets.
such that the expected loss of the personalized ensemble model for the data distribution Dn of client n is not larger than that of the single model only trained with local data: LDn( ∑N m=1 c ∗ m,nhD̂m) ≤ LDn(hD̂n). Besides, there exist some problems where the personalized ensemble model is strictly better, i.e., LDn( ∑N m=1 c ∗ m,nhD̂m) < LDn(hD̂n).
Theorem 4.1 indicates that the performance of the personalized ensemble model under some suitable coefficient matrices is better than that of the model only trained on its local private data, which theoretically demonstrates the necessity of the parameterized personalization. However, it is challenging to find such matrices due to the complexity and diversity of machine learning models and data distributions. In this paper, our designed algorithm KT-pFL can find the desired coefficient matrix in a gradient descent manner, hence achieving the performance boost. Besides, we have the similar claim for the relationship between the personalized ensemble model and average ensemble model.
Remark 4.1. There always exist c∗m,n,m = 1, . . . , N , such that the expected loss of the personalized ensemble model for the data distribution Dn of client n is not larger than that of the average ensemble model: LDn( ∑N m=1 c ∗ m,nhD̂m) ≤ LDn( 1 N ∑N m=1 hD̂m). Besides, there exist some
problems where the personalized ensemble model is strictly better, i.e., LDn( ∑N m=1 c ∗ m,nhD̂m) <
LDn( 1N ∑N m=1 hD̂m).

heading: 5 Evaluations

heading: 5.1 Experimental Setup
text: Task and Datasets We evaluate our proposed training framework on three different image classification tasks: EMNIST [35], Fashion_MNIST [36] and CIFAR-10 [37]. For each dataset, we apply two different Non-IID data settings: 1) each client only contains two classes of samples; 2) each client contains all classes of samples, while the number of samples for each class is different from that of a different client. All datasets are split randomly with 75% and 25% for training and testing, respectively. The testing data on each client has the same distribution with its training data. For all methods, we record the average test accuracy of all local models for evaluation.
Model Structure: Four different lightweight model structures including LeNet [38], AlexNet [39], ResNet-18 [40], and ShuffleNetV2 [41] are adopted in our experiments. Our pFL system has 20 clients, who are assigned with four different model structures, i.e., five clients per model.
Baselines: Although KT-pFL is designed for effective personalized federated learning, it can be applied to both heterogeneous and homogeneous model cases. We first conduct experiments on heterogeneous systems where the neural architecture of the local models are different among clients. We compare the performance of KT-pFL to the non-personalized distillation-based methods: FedMD [13], FedDF [16] and the personalized distillation-based method pFedDF3 and other simple versions of KT-pFL including Sim-pFL and TopK-pFL. Sim-pFL calculates the knowledge coefficient by using cosine similarity between two local soft predictions. Instead of combining the knowledge from all clients, TopK-pFL obtains the personalized soft predictions only from K clients4 who have higher value of cosine similarity.
To demonstrate the generalization and effectiveness of our proposed training framework, we further compare KT-pFL to FedAvg [1] and state-of-the-art pFL methods including Per-Fedavg [9], FedavgLocal FT[23], pFedMe [6], FedAMP [8], FedFomo[29] and FedHN[30] under the homogeneous model setting. Note that Per-FedAvg is a MAML-based method which aims to optimize the one-step gradient update for its personalized model. pFedMe and FedAMP are two regularized-based methods. The former one obtains personalized models by controlling the distance between local model and global model, while the later one facilitates the collaboration of clients with similar data distribution.
Implementation The experiments are implemented in PyTorch. We simulate a set of clients and a centralized server on one deep learning workstation (i.e., Intel(R) Core(TM) i9-9900KF CPU@3.6GHz with one NVIDIA GeForce RTX 2080Ti GPU).
3We use the fused prototype models at the last round of FedDF as the pFedD’s initial models. Each client fine-tunes it on the local private dataset with several epochs.
4In our experiments, we set K as 5.

heading: 5.2 Results
text: Subject to space constraints, we only report the most important experimental results in this section. Please refer to Appendix B for the details of different Non-IID data settings on EMNIST, Fashion_MNIST and CIFAR10, the implementation details and the hyperparameter settings of all the methods, and also extra results about the convergence and robustness analysis. In our experiments, we run each experiment multiple times and record the average results.

heading: 5.2.1 Performance Comparison
text: Heterogeneous FL. For all the methods and all the data settings, the batch size on private data and public data are 128 and 256, respectively, the number of local epochs is 20 and the distillation steps is 1 in each communication round of pFL training. Unless mentioned, otherwise the public data used for EMNIST and Fashion_MNIST is MNIST, and the public data used for CIFAR-10 is CIFAR-100. the size of public data used in each communication round is 3000, the learning rate is set to 0.01 for EMNIST and Fashion_MNIST, and 0.02 for CIFAR-10.
Figure 2 and 3 show the curves of the average test accuracy during the training process on four different models with three datasets, which include the results of FedMD, FedDF, pFedDF, Sim-pFL,
TopK-pFL and KT-pFL. We also summarize the final average test accuracy in Table 1. In two cases of Non-IID settings, KT-pFL obtains comparable or even better accuracy performance than others. The performances of FedMD, FedDF, pFedDF are worse than the other three methods. The reason is that taking the global aggregation of all local soft predictions trained on the Non-IID data from different clients produces only one global soft prediction, which cannot be well adapted to each client. The other two personalized methods, i.e., Sim-pFL and TopK-pFL, achieve comparably performance on most of cases with pFedDF. Our proposed method KT-pFL has the best performance, because each client can adaptively aggregate all local soft predictions to form a personalized one instead of being restricted to a global soft prediction.
Homogeneous FL. Apart from heterogeneous system, we further extend KT-pFL to homogeneous model setting. To conduct fair comparison with baselines, we exchange model parameters with the server to perform knowledge transfer. Specifically, each client maintains a personalized global model at the server side and each personalized global model is aggregated by a linear combination of local model parameters and knowledge coefficient matrix (Appendix C). In this case, we compare the performance of homogeneous-version of KT-pFL with FedAvg, Per-Fedavg, Fedavg-Local FT, pFedMe, FedAMP, FedFomo and FedHN. Figure 4 and Table 2 show the curve of the average test accuracy during training and the final average test accuracy on three different datasets, respectively. For all datasets, KT-pFL dominates the other seven methods on average test accuracy with less variance. Besides, the concept of FedAMP and FedFomo is similar with our proposed method KT-pFL. The difference is that the weights for each local model during personalized aggregation phase are updated in a gradient descent manner in KT-pFL.
To verify the efficiency of KT-pFL on large-scale FL system, we conduct experiments on 100 clients and apply the same client selection mechanism as other baselines. Specifically, in KT-pFL, only partial of elements in the knowledge coefficient matrix c will be updated in each communication round on large-scale FL systems. From the results in Table 2, it is obvious that our proposed method can work well both on small-scale and large-scale FL systems.

heading: 5.2.2 Effect of hyperparameters
text: To understand how different hyperparameters such as E, R, ρ, and |Dr| can affect the training performance of KT-pFL in different settings, we conduct various experiments on three datasets with η1 = η2 = η3 = 0.01.
Effect of Local Epochs E. To reduce communication overhead, the server tends to allow clients to have more local computation steps, which can lead to less global updates and thus faster convergence. Therefore, we monitor the behavior of KT-pFL using a number of values ofE, whose results are given in Table 3. The results show that larger values of E can benefit the convergence of the personalized models. There is, nevertheless, a trade-off between the computations and communications, i.e., while larger E requires more computations at local clients, smaller E needs more global communication rounds to converge. To do such trade-off, we fix E = 20 and evaluate the effect of other hyperparameters accordingly.
Effect of Distillation Steps R. As the performance of the knowledge transfer is directly related to the number of distillation steps R, we compare the performance of KT-pFL under different setting of distillation steps (e.g., R = 1, 2, 3, 5). The number of local epochs is set to 20. The results show that larger value of R cannot always lead to better performance, which means a moderate number of the distillation steps is necessary to approach to the optimal performance.
Effect of ρ. As mentioned in Eq (4), ρ is the regularization term to do the trade-off between personalization ability and generalization ability. For example, larger value of ρ means that the knowledge coefficient of each local soft prediction should approach to 1N , and thus more generalization ability should be guaranteed. Table 4 shows the results of KT-pFL with different value of ρ. In most settings, a significantly large value of ρ will hurt the performance of KT-pFL.
Effect of different public dataset. Besides, we compare the performance of our proposed method on different public datasets (Table 5). From the experimental results, we can observe that different settings of the public dataset have little effect on the training performance.
Effect of |Dr|. Table 6 demonstrates the effect of public data size |Dr| used in local distillation phase. When the size of the public data is increased, KT-pFL has higher average test accuracy. However,
very large |Dr| will not only slow down the convergence of KT-pFL but also incur higher computation time at the clients. During the experiments, the value of |Dr| is configured to 3000.

heading: 5.2.3 Efficiency Evaluation
text: To evaluate the communication overhead, we record three aspects information: Data (batch size used in distillation phase), Param (model parameters) and Soft Prediction without using data compression techniques. The results are shown in Figure 5. Compared with conventional parameter-based pFL methods, the communication overhead on soft prediction-based KT-pFL is far less than Conv-pFL.
Broader Impact
FL has been emerged as new paradigm to collaboratively train models among multiple clients in a privacy-preserving manner. Due to the diversity of users (e.g., statistical and systematic heterogeneity, etc.), applying personalization in FL is essential for future trend. Our method KT-pFL not only breaks the barriers of homogeneous model constraint, which can significantly reduce the communication overhead during training, but also improves the training efficiency via a parameterized update mechanism without additional computation overhead at the client side. This research has the potential to enable various devices to cooperatively train ML tasks based on customized neural network architectures.

heading: Acknowledgements
text: This research was supported by the funding from Hong Kong RGC Research Impact Fund (RIF) with the Project No. R5060-19, General Research Fund (GRF) with the Project No. 152221/19E and 15220320/20E, the National Natural Science Foundation of China (61872310), and Shenzhen Science and Technology Innovation Commission (R2020A045).

Review summaries:
The paper studies personalized federated learning (pFL), where the local models have different structures and sizes. The proposed method enables each client to maintain a personalized soft prediction at the server-side. Empirical results show the strength of the proposed solution.

This paper works on personalized federated learning (FL). The authors specifically studied the situation that clients train different model architectures. The authors leveraged additionally, public unlabeled data and knowledge distillation (KD) to aggregate these heterogenous clients' models specifically for each client, so that each client can learn their personalized model with the help from other clients. Experimental results on several datasets show the effectiveness of the proposed method.

The work aims to address the problem of creating adequate personalized models in a federated learning regime. The problems stems from a general observation that global models do no generalize well in the Federated Learning (FL) scenarios. The authors propose to use the idea of knowledge distillation to properly transfer knowledge to different clients via soft prediction mechanics. It is based on the knowledge about predictions that can be obtained on a public dataset (also referred as collaborative knowledge). At the server side, the proposed model learns the corresponding knowledge transfer coefficients that align inputs from multiple clients and hence distill the knowledge. The training is performed in an alternating manner by switching between knowledge coefficients (updated at the server side) and the task-specific parameters of the model (updated at the client side). Advantages of the proposed approach are empirically demonstrated in comparison against other FL algorithms w.r.t. both accuracy and communication efficiency on several public datasets.

This paper proposes a personalized group knowledge transfer training algorithm (KT-pFL) based on knowledge distillation that deals with model architecture heterogeneity in FL systems.  The authors claim the following contributions: - This paper is the first to study personalized knowledge transfer in FL. - Proposing the ‘knowledge coefficient matrix’ to identify the contribution from one client to others’ local training. - Providing theoretical performance guarantee for KT-pFL and conduct extensive experiments.