Parameterized Knowledge Transfer for Personalized Federated Learning
Authors: Jie Zhang
Song Guo
Xiaosong Ma
Haozhao Wang
Wenchao Xu
Feijie Wu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Personalized Federated Learning
Knowledge Transfer
Parameterized Distillation

Abstract: In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training.  KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models.  The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion\_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

Parameterized Knowledge Transfer for Personalized Federated Learning
Jie Zhang's Song Guo!.", Xiaosong Ma', Haozhao Wang", Wencao Xu', and Feijie Wu
Department of Computing. The Hong Kong Polytechnic University
DDparrment of Computer Science and Technology. HUST he_ wangehust.. edu...edd... cn,
{j fieeaa zhang, harli wu]@eonnctt. pol.loennnettttyo......o.o hk, hz {song. guo, xiaosma, wenchao xpoolll edu. hk
Abstract
recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, propose a novel training framework to employ personalized models for different clients. Specifically, formulate the aggregation procedure in original pFL into personalized group knowledge transfer training algorithm, namely, KTTpFLL, which enables each client maintain personalized soft prediction at the server side guide the others' local training. KT-pFL updates the personalized soft prediction of each client by linear combination of all local soft predictions using knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that can trained simultancously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Biasion_MNNISS different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with oftte-of-theaar algorithms.
exploit the potentials of heterogeneous model settings and
CIFAR-10) are conducted under
Introduction
Federated Learning (FL) has emerged as an efficient paradigm to collaboratively train a shared machine learning model among multiple clients without directly accessing their private data. By periodically aggregating parameters from the clients for global model updating, it can converge high accuracy and strong generalization. FL has shown its capability protect user privacy while there remains a crucial challenge that significantly degrades the learning performance, i.e., statistic heterogeneity in users' local datasets. Given the -ndeeendennd and Identically Distributed (Non-IID) user data, the trained global model often cannot be generalized well over each client (215).
To deal with the above issues, employing personalized models appears to be effective solution in FL., i.e., personalized federated learning (pFL). Recent works regarding pFL include regularization- based methods 8 (i.e., PFedMe (6). L2SGD 7 FedAMP (8), mneaa-larringgbaar based Per-FedAAg
Conresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

[9] and cluster-based IFCA [T. However, in order to aggregate the parameters from all clients, is inevitable for them to have identical model structure and size. Such constraints would prevent status quo pFL. methods from further application in practical scenarios, where clients are often willing to own unique models, i.e., with customized neural architectures to adapt to heterogeneous capacities in computation, communication and storage space, etc. Motivated by the paradigm of Knowledge Distillation (KD) [TH[6] that knowledge can be transferred from a neural network to another via exchanging soft predictions instead of using the whole model parameters, KD-based FL training methods have been studied [2119 to collaboratively train heterogeneous models in a rrivacy-preservic way. However, these works have neglected the further personalization requirement of FL clients, which can be well satisfied via the personalized knowledge transfer for heterogeneous FL users.
In this paper, we seek to develop novel training framework that can accommodate heterogeneous model structures for each client and achieve personalized knowlede transfer in each FL. training round. To this end, we formulate the aggregation phase in FL. to a personalized group knowledge transfer training algorithm dubbed KT-pFL, whose main idea is allow each client to maintain a personalized soft prediction at the server that can be updated by a linear combination of all clients' local soft predictions using knowledge coefficient matrix. The principle of doing so is to reinforce the collaboration between clients with similar data distributions. Furthermore, to quantify the contribution of each client to other's 's personalized soft prediction, we parameterize the knowledge coefficient matrix so that can be trained simultancously with the models following an alternating way in each iteration round.
We show that KT-pFLL. not only breaks down the barriers of homogeneous model restriction, which requires to transfer the entire parameters set in each round, whose data volume is much larger than that of the soft prediction, but also improves the training efficiency by using a parameterized update mechanism. Experimental results on different datasets and models show that our method can significantly improve the training efficiency and reduce the communication overhead. Our contributions are:
To the best of our knowledge. this paper is the first to study the personalized knowledge transfer in FL. We propose a novel training framework, namely, KT-pFL, that maintains a personalized soft prediction for each client in the server to transfer knowledge among all clients.
To encourage clients with similar data distribution to collaborate with each other during the training process, we propose the 'knowledge coefficient matrix' to identify the contribution from one client to others' local training. To show the efficiency of the parameterized method, we compared KT-pFL with two non-parameterized learning methods, i.e., TopK-pFL and Sim-pFL, which calculate the knowledge coefficient matrix the cosine similarity between different model parameters.
We provide theoretical performance guarantee for KT-pFL and conduct extensive experiments over various deep learning models and datasets. The efficiency superiority of KT-pFL is demonstrated by comparing our proposed training framework with traditional pFL methods.
2 Related Work
Personalized FL with Homogeneous Models. Recently, various approaches have been proposed to realize personalized FL with homogeneous local model structure, which can be categorized into three types according to the number of global models applied in the server, i.e., single global model, multiple global models and no global model.
single global model type is a close variety of conventional FI., e.g., FedAvg [] that combine global model optimization process with additional local model customization, and consist of four different kinds of approaches: local fine-tuning [20/23), regularization (e.g., pFedMe [6]. L2SGD 7 24], Ditto (25).. hybrid local and global models 26 27] and meta learning 28 All of these pFL methods apply single global model, and thus limit the customized level of the local model at the propose to train multiple global models at the server, where clients are clustered into several groups according to their similarity and different models are trained for each group. FedAMP S and FedFomo
client side. Therefore, some researchers [8] 10
29 can be regarded as special cases of the clustered-basee method that each client owns personalized global model at the server side. As a contrast, some literature waive the global model deal with the heterogeneity problem RD) 31 such as multi-task learning based (i.c., MOCHA (31] and dypernetwork-basee framework (i.c..

---

However, all these methods require aggregating the model parameters from the clients, who have to apply identical model structure and size, which hinders further personalization, e.g..
FedHN (30).
employing personalized model architectures for heterogeneous clients is not feasible.
Heterogeneous FL and Knowledge Distillation. To enable heterogeneous model architectures in
FL. Diao et al. 32 propose to upload a different subset of global model to the server for aggregation with the objective to produce single global inference model. Another way of personalization is
to use Knowledge Distillation (KD) in heterogeneous FL systems ( 15 17-19 33 34]).
The principle is to aggregate local soft ooft-rreiictios instead of local model parameters in the server, whereby each client can update the local model to approach the averaged global predictions. As KD is independent with model structure, some literature [6] are proposed take advantage of such independence to implement personalized FL with heterogeneous models at client sides. For example, Li al. [13 propose FedMD) to perform ensemble distillation for each client learn personeesddd models. Different from FedMI) that exchanging soft-predictions between the clients and the server, FedDF [16] first aggregates local model parameters for model averaging at the server side. Then, the averaged global models can be updated by performing knowledge transfer from all received (heterogeneous) client models.
To sum most of these schemes construct an ensembling teacher by simply averaging the teachers' soft predictions, or by heuristically combining the output of the teacher models, which are far away from producing optimal combination of teachers. In our framework, KD is used in a more efficient way that the weights of the clients' soft predictions are updated together with the model parameters during every FL. training iteration.
3 Problem Formulation
We aim collaboratively train personalized models for a set of clients applying different model structures in FI.. Consider supervised learning whose goal is to learn function that maps every input data to the correct class out of possible options. We assume that there are N clients, and each client n can only access to his private dataset DD, == {xr:"' ). where x; is the i-th input data sample, y/ is the corresponding label of Ts, y/ E (1, 2........ ,C}. The number of data samples in dataset ID, is denoted by Da- D = {D1, D2, Dr == =Ex= Dn. In conventional FL. the goal of the learning system is to learn a global model w that minimizes the total empirical loss over the entire dataset D:
Dn min L(w) :- Σ DD Ln(w, where (w) Lcr(wwe...)))
N
Dr
where L, (w) is the n-th client's 's local loss function that measures the local empirical risk over the private dataset ID, and LCE is the cross-entropy loss function that measures the difference between the predicted values and the ground truth labels.
However, this formulation requires all local clients to have unified model structure, which cannot be extended to more general cases where each client applies a unique model. Therefore, we need to reformulate the above optimization problem to break the barrier of homogeneous model structure. Besides, given the Non-IID elients' datasets, it is inappropriate to just minimize the total empirical loss (i.e., min w........" L(wl,... w Σ D. L,, (w")). To that end, we propose the following training framework:
Definition 3.1. Let s(w", .t) denote the collaborative knowledge from client and å denote data sample from a public dataset D, that all clients can access to. Define the personalized loss function of client n as
Eper, (pern(w"")= := Cn(w") +
A Σ CKL
Σ Cmn s(w") ".w", s(w", s(w", .2) m==1
SED,
where À > O is and is added to the loss function to transfer personalized knowledge from a teacher to another. Cmn is the knowledge coefficient which is used to estimate the contribution from client n. The second term in 2 allows each client to build his own personalized aggregated knowledge in the server and enhance the collaboration effect between clients with large e. The concept of collaborative
LKL stands for Kullback-Leibe (KL) Divergence function
3

---

Server
Private
Labrls
Labrls
Labalg
Public
Clicnt 1
Solt Hrddeioon
Clictt 2
Solt Ptedkdiin
Clinnt Solt Redddnn
Figure 1: Illustration of the KT-pFL framework. The workflow includes 6 steps: 1 local training on private data; @, @ each client outputs the local soft prediction on public data and sends it to the server; @ the server calculates each client's personalized soft prediction via a linear combination of local soft predictions and knowledge coefficient matrix; @ each client downloads the personalized soft prediction to perform distillation phase; @ the server updates the knowledge coefficient matrix.
knowledge either refer soft predictions model parameters depending the definition in different situations. For example, s(w", .:) can be deemed to be a soft prediction of the client n, which are calculated with the softmax of logits 2", i.e., s(w" output of the last fully connected layer on client n's model, T is the temperature hyperparameter of the softmax function.
Ee exp(=ø /T)" logits 2" is the
exp(=") /T)
Definition 3.2. We define c IR. N as the knowledge coefficient matrix:
C11
C12
C1N
C21
C22
C2N
CN1
CN2
CNN
Our objective is to minimize
Dn D Cpern( + pllc - N 11
min L(w, c) = Σ n=1
(4)
w.c
where = [w ..... w E R 22n du is the concatenated vector of all weights, d, represents the dimensions of model parameter w". g=" is the identity matrix whose elements are all equal to 1. The second term in (4) is a regularization term that ensures generalization ability of the whole learning system. Without the regularization term, a client with a completely different data distribution tends to set large values of the knowledge coefficient (i.e., equal to 1), and no collaboration will be conducted during training in this case. p is regularization parameter that is larger than 0.
KT-pFL Algorithm
In this section, we introduce the proposed KT-pFL algorithm, where the local model parameters and knowledge coefficient matrix are updated alternatively. To enable personalized knowledge transfer in FL, we train personalized models locally according to the related collaborative knowledge. Insert into and can design an alternating optimization approach to solve 4 that in each round we fix either w or с by turns, and optimize the unfixed one following an alternating way until convergence point is reached.
Update w: each communication round, we first fix and optimize (train) for several epochs locally. In this case, updating w depends on both the private data (i.e., LCF on ID, n E [1..... N)), that can only be accessed by the corresponding client, and the public data (i.e., Lxt. on Dz), which is accessible for all clients. We propose two-stage updating framework for w:

---

Algorithm KTT-pLL Algorithm
Input: D. D, 72. 7Js and T
Output: w w= [- ,.., w
I: Initialize Wo and Co
2: procedure SERVER-SIDE OPTIMIZATION
3: Distribute wo and Co each client
4: for each communication round / E 21, do
5: for each client n. in parallel do
6:
W & Clieett pdate(n, w nr
7:
Update knowledge coefficient matrix e via Distribute C++1 to all clients
8:
9:
procedure
w Cr.n)
10:
Client n receives w and Cn from the server for each local epoch from E do for mini-batch & C Dr, do
12:
13:
Local Training: update model parameters private data via (5)
14:
for each distillation step. from I to do for mini-batch E,, C D, do
15:
16:
Distillation: update model parameters on public data via (6) return local parameters w
Local Training: Train on each client's private data by applying gradient descent step:
w" f- w" 7) w (w";
(5)
where &. denotes the mini-batch of data D, used in local training, is the learning rate. Distillation: Transfer knowledge from personalized soft prediction to each local client based on public dataset:
N
W f w T2 LKL
Σ m
c
s(w" s(w"),))),,Ww" (w),,((",,)))
(6)
where &, denotes the mini-batch of public data ID, and 7/2 is the learning rate. CT [Cm1, Cm2, ... CmN] is the knowledge coefficient vector for client m, which can be found in m-th row of Note that all collaborative knowledge and knowledge coefficient matrix are required to obtain the personalized soft prediction in this stage, which can be collected in the server. Update c: After updating w locally for several epochs, we turn to fix and update in the server.
D. c- Σ DuV
Σ Cm s(w""* s(""" m=1
.5,) -nsp(c-
1 1 N
(7)
where 7/a is the learning rate for updating c.
Algorithm demmonstatts the proposed KT-pFL. algorithm and the idea behind it is shown in Figurel In every communication round of training, the clients local SGD train several epochs based on the private data and then send the collaborative knowledge (e.g., soft predictions on public data) the server. When the server receives the collaborative knowledge from each client, it aggregates them to form the personalized soft predictions according to the knowledge coefficient matrix. The server then sends back the personalized soft prediction each client to perform local distillation. The clients then iterate for multiple steps over public dataset After that, the knowledge coefficient matrix is updated in the server while fixing the model parameters w.
Performance Guarantee. Theorem 4.1 provides the performance analysis of the personalized model when each client owns Non-IID data. Detailed description and derivations are deferred to Appendix A.
Theorem 4.1. Denote the n-th local distribution and its empirical distribution by D,, and D, re- spectively, and the hypothesis H trained on DD,, n, by h.p. There always exist cm. m. 1,..., N,
Note that our method can work over both labeled and unlabeled public datasets.
5

---

such that the expected loss of the personalized ensemble model for the data distribution D,, of client is not larger than that of the single model only trained with local data: LD, L B... S DD (h-,)) Besides, there exist some problems where the personalized ensemble model is strictly better, i.e., LD. (Em=1 hn < LD., (hD.))
Theorem 4. indicates that the performance of the personalized ensemble model under some suitable coefficient matrices better than that of the model only trained on its local private data, which theoretically demonstrates the necessity of the parameterized personalization. However, it is challeng- ing to find such matrices due to the complexity and diversity of machine learning models and data distributions. In this paper, our designed algorithm KT-pFL can find the desired coefficient matrix in gradient descent manner, hence achieving the performance boost. Besides, we have the similar claim for the relationship between the personalized ensemble model and average ensemble model.
Remark 4.1. There always exist Cm.n m alized ensemble model for the data distribution D,, of client n is not larger than that of the aver- age ensemble model: LDD, (Em=1 Cm,n D.D... S LDD, (N Em=1 h.m)) Besides, there exist some problems where the personalized ensemble model is strictly better, ie., LDD LD (Em=i Sn. "D <
,,..., N, such that the expected loss of the person-
CD. (1 N Em=1 l.p...
5
Evaluations
5.1 Experimental Setup
Task and Datasets We evaluate our proposed training framework on three different image classifi- cation tasks: EMNIST [35,, Fashion __NIST (36) and CIFAR-IO [37]. For each dataset, we apply two different IID data settings: 1) each client only contains two classes of samples; 2) each client contains all classes of samples, while the number of samples for each class is different from that of different client. All datasets are split randomly with 75% and 25% for training and testing, respectively. The testing data on each client has the same distribution with its training data. For all methods, we record the average test accuracy of all local models for evaluation.
Model Structure: Four different lightweight model structures including LeNet 138 AlexNet [39], ResNet- 18 40. and ShuffleNetV2 [4] are adopted in our experiments. Our pFL system has 20 clients, who are assigned with four different model structures, i.e., five clients per model.
Baselines: Although KT-pFL is designed for effective personalized federated learning, it can be applied both heterogeneous and homogeneous model cases. We first conduct experiments on heterogeneous systems where the neural architecture of the local models are different among clients. We compare the performance of KT-pFL to the non- non-personalized distillation- based methods: FedMD [13], FedDF [16] and the personalized distillation based method pFedDF and other simple versions of KT-pFL including Sim-pFL and TopK-pFL. Sim-pFL calculates the knowledge coefficient by using cosine similarity between two local soft predictions. Instead of combining the knowledge from all clients, TFp obtains the personalized soft predictions only from K clients who have higher value of cosine similarity.
To demonstrate the generalization and effectiveness of our proposed training framework, we further compare KT-pFL to FedAvg [1] and state-of-the-arr pFL methods including Per-Fedavg [9), Fedavg- Local FT 23 PFedMe 66. FedAMP [8], FedFomo 29] and FedHN under the homogeneous model setting. Note that Per-FedAvg is MAML-based method which aims to optimize the one-step gradient update for its personalized model. pFedMe and FedAMM are two regularized-bue based methods. The former one obtains personalized models by controlling the distance between local model and global model, while the later one facilitates the collaboration of clients with similar data distribution.
Implementation The experiments are implemented in Py PoTocc. We simulate a set of clients and a cen- tralized server on one deep learning workstation (i.e., Intel(R) Core(TM) 19-9900KF FPU@3.6GHz with one NVIDIA GeForce RTX 2080Ti GPU).
2 We use the fused prototype models at the last round FedDF as the pFedD's initial models. Each client fine-tunes on the local private dataset with several epochs. In our experiments, set K as 5.

---

Averago Test Accuracy
0.90
Average Test Accuracy
Average Test Accurage
0..49
0.81
0.72
0.42
0.72
TedMi
0.68
Feb
PeeMD
0.63
FeED
Felbs
FeeDF
0.64
0.28
0.54
0.45
Topi--ptL
SimppLL
0.60
Sim
Topk-ptL
0.21
Sin
topk-atL
0.36
10
KT-sFt
25
10
KT.FFL
25
30
0.14
10
KTyFL
25 30
Communication Rounds
Communication Rounds
Communication Rounds
(a) EMNIST
(b) Fashion MNIST
(c) CIFAR-10
Figure 2: Performance comparison of FedMD, FedDF, pFedDF, TopK-pFD, Sim-pFD, and KT-pFLL
in average test accuracy on three datasets (Non-IID case 1: each client contains all labels).
Average Test Accurncy e
Average Test Accurncy
0.80
Average Test Accuracy
0.80
30.44
0.75
0.75
0.40
0.70
0.36
0.70
0.65
Tott
Falte
Pettir
0.60
pladtt
Topi--pr1
0.65
NedD
Topk-po
0.28
tspk-ant
0.55
0.60
0.24
0.50
5
10
15
20
25
30
0.55
5
10
15
20
25
30
0.20
5
10
15
20
25 30
Communication Rounds
Communication Rounds
Communication Rounds
(a) EMNIST
(b) Fashion MNNSTT
(c) CIFAR-10
Figure 3: Performance comparison of FedMD, FedDF, pFedDF, -PDD, Sim-pFD, and KT-pFL average test accuracy three datasets (Non-IID case 2: each client contains only two labels).
5.2 Results
Subject to space constraints, we only report the most important experimental results in this section. Please refer Appendix B for the details of different Non-IID data settings on EMNIST, Fash- ion_ M and CIFARIO, the implementation details and the hyperparameter settings of all the methods, and also extra results about the convergence and robustness analysis. In our experiments, we run each experiment multiple times and record the average results.
Table The comparison of final test accuracy (%) on different datasets with heterogeneous models (i.e., Lenet, AlexNet, ResNet- 18 and ShuffleNetV22.
EMNIST
Fashion MNIST
CIFAR-10
Method
Non-IID
Non-IID
Non-IID
Non-IID 2
Non-IID
Non-IID 2
FedMD 13
71.37
70.15
68.02
66.10
39.14
35.56
FedDF 16
71.92
72.79
70.32
70.83
40.19
37.81
PFedDF
78.59
75.72
72.69
73.45
42.83
39.54
Sim-pFD
82.72
74.42
72.82
72.71
42.50
39.90
TopK-pFD
83.00
73.91
72.24
73.39
42.58
39.34
KT-pFI.
84.65
76.30
75.48
75.60
43.82
41.04
5.2.1 Performance Comparison
Heterogeneous FL. For all the methods and all the data settings, the batch size on private data and public data are 128 and 256, respectively, the number of local epochs is 20 and the distillation steps in each communication round of pFL training. Unless mentioned, otherwise the public data used is M MNIST, and the public data used for CIFAR-10 10 is CIFAR-100. the size public data used in each communication round is 3000, the learning rate is set to 0.01 for EMNIST and Pashion_MNISS and 0.02 for CIFAR-10.
for EMNIST and Fashion MNIST
Figure and show the curves of the average test accuracy during the training process on four different models with three datascts, which include the results of FedMD, FedDF, pFedDF, Sim-pFL,

---

Ayerage Accuracy
1.0
11.
0.7
0.9
0.9
0.8
0.8
0.7
- Pa Feibie
Palling
0.7
- Pe hoblig
0.4
Pe hodoe
0.3
0.55
PedAM!
Pobshono
0.5
-6- Fetdiino hans
REAMP
Averag
00.2
Podalors
TRSAM??
0.4
0.4
0.1
0.30
200
400
600
800
0.30
200
400
600
800
0.0
200
400
800
Communication Rounds
Communication Rounds
Communication Rounds
(a) EMNIST
(b) Bishion_MNIST
(c) CIFAR 10
Figure 4: Performance comparison of FedAvg, Per-Fedavg, Fedavg-Local FT, pFedMe, FedAMP, FedFomo, FedHN and KT-ppLL average test accuracy on three datasets. The Non-IID data setting: each client contains all labels. 20 clients with homogeneous models: CNN [I]. Learning rate: 0.005.
Table 2: The comparison of final test accuracy (%) on different datasets with different number of homogeneous models (i.e., the same CNN architecture as [I]. For large-scale FL system with 100 clients, set the client sampling rate 0.1.
EMNIST
Fashion MNIST
CIFAR-10
Method
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
Fedavg
85.86
76.84
80.69
72.36
$1.94
43.56
Per- Fer-Fedavg
86.16
82.87
85.91
77.93
55.61
49.82
Fodavg-Local FT 23
86.59
84.55
90.17
80.93
42.88
50.91
PFedMe [
84.26
86.14
90.97
84.02
62.79
52.46
FedAMP 8
88.51
85.63
90.79
84.72
60.75
51.26
FedFomo
29
92.58
90.08
88.33
87.76
63.11
55.73
pFedHN
94.27
92.21
92.22
89.14
65.31
57.38
KT-pFL
94.40
92.50
93,93
90.37
66.96
58.29
TopK-pFL and KT-pFL. We also summarize the final average accuracy in Table !1 of Non-lID settings, KT-pFL obtains comparable or even better accuracy performance than others. The performances of FedMD, FedDF, pFedDF are worse than the other three methods. The reason is that taking the global aggregation of all local soft predictions trained on the Non-IID data from different clients produces only one global soft prediction, which cannot well adapted to each client. The other two personalized methods, i.e., Sim-pFL and TopK-pFL, achieve comparably performance on most cases with pFedDF. Our proposed method KT-pFI. the best performance, because each client adaptively aggregate all local soft predictions form personalized one instead of being restricted to a global soft prediction.
In cases
Homogeneous FL. Apart from heterogeneous system, we further extend KT-pFL to homogeneous model setting. To conduct fair comparison with baselines, we exchange model parameters with the server to perform knowledge transfer. Specifically, each client maintains personalized global model the server side and each personalized global model aggregated by linear combination of local model parameters and knowledge coefficient matrix (Appendix C). In this case, compare the performance of homogeneous-verte versionn of KT-pFL with FedAvg, Per-Fedavg, Fedavg-Loocs FT, pFedMe, FedAMP, FedFomo and FedHN. Figure Seuureaan and Thow the curve of the average test accuracy during training and the final average test accuracy on three different datasets, respectively. For all datasets, KT-pFL dominates the other seven methods average test accuracy with less variance. Besides, the concept of FedAMP and FedFomo similar with our proposed method KT-pFL. The difference is that the weights for each local model during personalized aggregation phase are updated a gradient descent manner in KT--pF.
To verify the efficiency of KT-pFL on large-scale FL system, we conduct experiments on 100 clients and apply the same client selection mechanism as other baselines. Specifically, in KT-pFL,, only partial of elements in the knowledge coefficient matrix will be updated in each communication round on large-scale FL systems. From the results Table is obvious that our proposed method work well both small-scale and large-scale FL systems.

---

Table 3:
The comparison of final test accuracy on different setting of local epochs E and distillation
steps R.
Local epochs (E)
# Distillation steps (R)
Dataset
10
15
20
5
EMNIST (%)
90.15
92.76
91.03
90.15
91.76
91.40
91.18
91.54
Fashion MNIST (%)
88.73
89.14
88.58
89.42
89.14
89.90
89.07
88.08
CIFAR-10 (%)
58.30
59.38
59.34
59.24
59.24
57.99
59.22
58.91
Table 4: The comparison of final test accuracy on different setting of regularization parameter
Regularization parameter
Dataset
0.1
0.3
0.5
0.7
EMNIST (5)
90.96
90.66
90.88
91.76
Fashion MNIST (%)
89.49
89.30
89.56
89.14
CIFAR-10 (%)
59.08
58.52
59.56
58.40
5.2.2 Effect of hyperparameters
To understand how different hyperparameters such as E, R, p. and [D,,| can affect the training performance of KT-pFL in different settings, conduct various experiments on three datasets with 7 = 1/2 = 773 = 0.01.
Effect of Local Epochs F. To reduce communication overhead, the server tends to allow clients to have more local computation steps, which can lead less global updates and thus faster convergence. Therefore, we monitor the behavior of KTT-pFL using number of values of E, whose results are given in Table The results show that larger values of E' can benefit the convergence of the personalized models. There is, nevertheless, trade-off between the computations and communications, i.e., while larger E' requires more computations at local clients, smaller E needs more global commu- nication rounds to converge. To do such trade-off, fix E 20 and evaluate the effect of other hyperparameters accordingly.
Effect of Distillation Steps R. As the performance of the knowledge transfer is directly related to the number of distillation steps R. we compare the performance of KTT-pFI. under different setting of distillation steps (e.g., R = 2, 3, 5). The number of local epochs is set to 20. The results show that larger value of R cannot always lead better performance, which means a moderate number of the distillation steps is necessary to approach to the optimal performance.
Effect of p. As mentioned in Eq (4) p is the regularization ization ability and generalization ability. For example, larger value of means that the knowledge coefficient of each local soft prediction should approach to 1 and thus more generalization ability should be guaranteed. [/bles]shows the results of KT-pFL with different value of p. In most settings, significantly large value p will hurt the performance of KT-pFL.
do the trade-off between personal-
Table 5: Performance of KT-pFL. under different setting of public dataset (Task: Dashion_MNIST Unlabeled Open Dataset: divided from Bashion_MNIST)
Public dataset
Test Accuracy
MNIST
89.14 (=: 0.15) 88.76 (= 0.21)
EMNIST
Unlabeled Open Dataset
89.03 (= 0.11)
Effect of different public dataset. Besides, compare the performance of our proposed method on different public datasets (Table 5). From the experimental results, we can observe that different settings of the public dataset have little effect on the training performance.
Effeet of [ Table Jabbee/demoostraccee the effect of public data size [D,| used in local distillation phase.
When the size of the public data is increased, KT-pFL has higher average test accuracy. However,

---

Table 6: The comparison of final test accuracy different setting of the public data size [IDy].
Size of public data (D,,
Dataset
10
100
500
1000
3000
5000
EMNIST (%)
56.19
73.44
91.25
91.47
91.66
91.32
Fashion MNIST (%)
50.80
67.43
88.08
89.40
89.50
89.14
CIFAR-I0 (%)
39.08
47.70
58.93
58.79
58.91
58.40
very large [D,|] will only slow down the convergence of KT-pFL but also incur higher computation time at the clients. During the experiments, the value of [D; is configured to 3000.
5.2.3 Efficiency Evaluation
To evaluate the communication overhead, we record three aspects information: Data (batch size used in distillation phase), Param (model parameters) and Soft Prediction without using data compression techniques. The results are shown in Figure [] Compared with conventional parameter- based pFL methods, the communication overhead soft prediction-based KT-pFLL is far less than Conv-pFl..
Soft prediction
XT-pFL
48.1 14.6
48.1
14.6
300.0 14.6
Comv-pPL
4,777.4
EMNIST
Fashion, MNIST
CIFAR-10
Figure 5: Communication Efficiency (X-axis units: MBytes) three datasets. KL-pFI.: soft prediction-based personalized federated learning; Conv-pFL: conventional parameter-bases personal- ized federated learning. (20 models, 30 communication rounds for all datasets. In this experiment, the public dataset is stored on the server.)
Broader Impact
has been emerged paradigm to collaboratively train models among multiple clients in privacy-preserving manner. Due to the diversity users (e.g., statistical and systematic heterogeneity. etc.), applying personalization in FL is essential for future trend. Our method KT-pFL. not only breaks the barriers homogeneous model constraint, which can significantly reduce the communication overhead during training. but also improves the training efficiency via parameterized update mechanism without additional computation overhead at the client side. This research has the potential enable various devices cooperatively train ML tasks based customized neural network architectures.
Acknowledgements
This research was supported by the funding from Hong Kong RGC Research Impact Fund (RIF) with the Project No. R5060-19, General Research Fund (GRF) with the Project No. 152221/19E and 1,220320/20F, the National Natural Science Foundation of China (61872310), and Shenzhen Science and Technology Innovation Commission (R2020A045).
References
[1] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-eeficcenn learning of deep networks from decentralized data. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, AISTATS, 2017.
10

---

[2] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, Mys,,, 2020.
[3] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Proceedings of International Conference on Machine Learning, ICML, pages 4615-4625, 2019.
[4] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of International Conference on Machine Learning, JCML, pages 5132-5143, 2020.
[5] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In Proceedings of 8th International Conference Learning Representations, ICLR, 2020.
[6] Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurlPS, 2020.
[7] Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.053: 06,606, 2020.
[8] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2021.
[9] Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Proceedings of Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems. NeurlPS, 2020.
[10] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, 2020.
[11] Yishay Mansour, Mchryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020.
[12] Eunjeong Jeong. Scungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong Beong-Lyum Kim. Conmunication-efficcte on-device machine learning: Federated distillation and augmen- tation under non-iid private data. arXiv preprint arXiv: / I. 1479, 2018.
[13] Daliang Li and Junpu Wang. FedMD: Heterogenous federated learning via model distillation. arXiv, oct 2019.
[14] Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint 1rXiv:1912.11279, 2019.
[15] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-ben based semi-supervised federated learning for communication-eefccceentt collaborative training with non-iid private data. arXiv preprint arXiv:2008.06118 2020.
[16] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, 2020.
[17] Hong-Yo You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[18) Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. CoRR, 0ttp//2099079999 2020.

---

[19] Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, pages 1563-1570. ijcai.org. 2021.
[20] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint 19Xiv:1910.1000 10252, 2019.
[21] Johannes Schneider and Michail Vlachos. Personalization of deep learning. arXiv preprint arXiv: 090000000 2019.
[22] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint 19120089,,,08818, 2019.
[23] Tao Yu, Hugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. CoRR, abs/2002.04758, 2020.
[24] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, NeurlPS, 2020.
[25] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
[26] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.
[27] Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture of experts. arXiv preprint arXiv:2107.06724 2021.
(28) Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning, arXiv preprint arXiv: 12488, 2019.
[29] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In 9th International Conference on Learning Representations, ICLR 202/. Openkeview.net, 2021.
[30] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139, pages 9489-9502. PMLR, 2021.
[31] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi- task learning. In Proceedings of Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems, NeurIPS, 2017.
[32] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
[33] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. CoRR, abs/1902.1
1175, 11175,
2019.
[34] Chaoyang He, Salman Avestimehr, and Murali Annavaram. Group knowledge transfer: Collab- orative training of large enns on the edge. arXiv preprint arXiv:2007.14513. 2020.
[35] Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. EMNIST: extending MNIST to handwritten letters. In Proceedings of 2017 International Joint Conference on Neural Networks, IJCNN, 2017.
[36] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.07747, 2017.
12

---

[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324, 1998.
[39] Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of Advances in Neural Information Processing Systems 25: 26th Annual Conference Neural Information Processing Systems, NeurIPS, pages 1106-11 1114 2012.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 770-778, 2016.
[41] Ningning Ma, Xiangyu Zhang, Hai-TTo Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. Proceedings of the European conference on computer vision, ECCV, pages 116-131, 2018.
13


Reviews:
[{"review_id": "xaA0nSy2qhA", "reviewer": "Reviewer_21U2", "summary": "This paper works on personalized federated learning (FL). The authors specifically studied the situation that clients train different model architectures. The authors leveraged additionally, public unlabeled data and knowledge distillation (KD) to aggregate these heterogenous clients' models specifically for each client, so that each client can learn their personalized model with the help from other clients. Experimental results on several datasets show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I number my questions or the paper's weaknesses as follows. If possible, I would like the authors to at least address 1-4 and 7-8.\n\n======= *Originality*: The proposed method is somewhat a novel combination of (1) KD for FL and (2) deriving a personalized global model/regularizer for FL. However, the authors did not explicitly state these relationships to existing works clearly. Instead, the authors mentioned some of the relationships distributedly in several places of the paper. Besides, many related works are not cited, discussed, or empirically compared. All these factors largely limit the papers’ originality.\n\n1. There have been a few FL works on heterogeneous systems, e.g., [a, b]. Specifically, [b] has a title similar to the proposed method. Both papers are not cited. Also, there is no description in the related work on existing solutions to heterogeneous systems (e.g., [16]), giving a wrong impression that the proposed method is the first to handle this situation.\n\n[a] HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients, ICLR 2021\n\n[b] Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge, NeurIPS 2020  \n\n2. Knowledge distillation (KD) is one promising way to handle system heterogeneity. However, instead of discussing how KD has been used in FL in the related work (e.g., [12-16]), the authors just provided a general overview of KD. Specifically, [13] is on personalized FL but without any discussion. Moreover, several other FL works using KD are not cited and discussed [c-f].\n\n[c] FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning, ICLR 2021\n\n[d] Distilled one-shot federated learning, arXiv 2020\n\n[e] Federated Model Distillation with Noise-Free Differential Privacy, arXiv 2020\n\n[f] One-Shot Federated Learning, arXiv 2019\n\n3. Another important component is personalized knowledge transfer. It seems that [8, 24] share similar ideas to the proposed method, but the authors didn’t provide a detailed discussion. Some other similar works also derive personalized global models or regularizers [g, h].\n\n[g] Personalized federated learning with first order model optimization, ICLR 2021\n\n[h] Federated Mixture of Experts, OpenReview 2020\n\nWithout a clear discussion of these related works and a discussion on the challenge of a naïve combination of them, it is hard for me to assess the paper's originality.\n\n======= *Quality*: The proposed method is technically sound. The optimization procedure is very clean and should be easily reproducible. The authors also conduct extensive experiments to justify the effectiveness of the proposed algorithm. However, there is no clear description and motivation on why it is designed. Also, some important baselines and experimental details are missing; some experimental designs can be improved. See the following.\n\n4. More specifically, the authors suddenly gave a formulation in Eq (2) without giving the design principle. I would suggest that the authors first describe an objective function for personalized FL, and then gradually lead to Eq (2) for the heterogeneous case. Also, I don’t fully understand the description in Line 146-147 for the regularization term in Eq (4).\n\n5. There seems to be no clear description of how the public data is constructed. Does each communication round use the same public data? Also, there is no study on the size of the public data (not merely the batch size).\n\n6. Experimental designs. I checked the design of \"each client contains all classes of samples\" in the supplementary. It does not lead to a strong non-IID case. Please take a look at [c, i] and consider re-design the experiments\n\n[i] Measuring the effects of non-identical data distribution for federated visual classification, arXiv 2019\n\n7.\tExperiments on heterogeneous systems. Can the authors provide more details on FedDF, which was not designed for personalized FL? Specifically, do the authors compare to the distilled model at the server or the local models after local update/fine-tuning? It is very important to compare to the local models. Can the authors compare to the original FedDF (and with local updates before evaluating on personalized data), instead of the modified version in footnote 2? Note that, FedDF does not add any regularized term like the one in Eq (2) in its local training. Also, [13] should be compared. These experiments are very important to ensure a fair and comprehensive comparison. Besides, can the authors further compare to a version of Eq (2) with identical c_m,n? That is, there is no personalized knowledge transfer. \n\n8.\tExperiments on homogeneous systems. First, I don’t see a reason why the authors have to modify their algorithm for this setup (Line 249-252). Is this a limitation of the work? Second, the baselines in Table 2 are not very strong. Please consider comparing to some of [i, j, g]. Moreover, please compare to FedAvg after local fine-tuning [17, k]. Third, the proposed method has an obvious advantage --- the public data. Thus, I would suggest that the authors further compare methods based on KD (e.g., [13, 16, c]). For those not designed for personalized FL, the authors can simply fine-tune the global models to get personalized models. Finally, the accuracy of CIFAR-10 in Table 2 is fairly low.  Do the authors know why?\n\n[i] Think locally, act globally: Federated learning with local and global representations, arXiv 2020\n\n[j] Federated learning with personalization layers, arXiv 2019\n\n[k] Salvaging Federated Learning by Local Adaptation, arXiv 2020\n\n======= *Clarity*: The paper is well-written and organized.\nSignificance: The authors work on an important problem, federated learning under the heterogeneous system setting. I think the proposed method could serve as a strong baseline to be compared or built upon in the future. However, the significance of the paper is limited by the aforementioned questions.\n\n**Post-rebuttal update**\nI thank the authors for their tremendous efforts in addressing my concerns for multiple rounds. Many of my concerns are addressed so I increased the score from 4 to 6. I encourage the authors to follow my latest response to incorporate the feedback and their rebuttal into the final version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't discuss the limitation. However, this does not affect my score. I don't think the paper will have potential negative societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "qDH4v4EJ3ha", "reviewer": "Reviewer_7FPU", "summary": "The work aims to address the problem of creating adequate personalized models in a federated learning regime. The problems stems from a general observation that global models do no generalize well in the Federated Learning (FL) scenarios. The authors propose to use the idea of knowledge distillation to properly transfer knowledge to different clients via soft prediction mechanics. It is based on the knowledge about predictions that can be obtained on a public dataset (also referred as collaborative knowledge). At the server side, the proposed model learns the corresponding knowledge transfer coefficients that align inputs from multiple clients and hence distill the knowledge. The training is performed in an alternating manner by switching between knowledge coefficients (updated at the server side) and the task-specific parameters of the model (updated at the client side). Advantages of the proposed approach are empirically demonstrated in comparison against other FL algorithms w.r.t. both accuracy and communication efficiency on several public datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The text is well written and easy to read. It follows logical structure and is transparently organized. The related literature section is relevant and gives a good overview of the state of the field in this direction. The problem formulation and the corresponding derivations of the solution are mathematically sound and non-contradicting. The key concepts are properly introduced and clearly described. I find the way knowledge distillation utilized in this work very creative and novel. Considering that the FL topic draws a lot of attention from industry, the proposed approach seems to offer a very promising solution with high practical importance. It also open perspectives for further research in this direction, e.g. for finding better ways of aligning client data on a server side. I enjoyed reading this paper very much.\n\nHowever, I also have several concerns and questions. \n\n1) In the very beginning, the authors mention privacy-related issues in the FL context. The proposed scheme promises to resolve any issues as it doesn't directly transfer any private user data in any way. On the other hand, it seems that in order for this model to work properly, the distribution of data on public and private datasets should be close (ideally the same). If so, then knowing distribution on the public dataset $\\mathbb{D}_r$ may potentially help revealing distribution on the private data $\\mathbb{D}_n$, which in turn would allow for some better-then-random-guess attempts to deanonimize user data. I'm not sure what are the risks of that and it would be good to have authors add more details on this issue and the general risks of loosing privacy in the proposed scheme.\n\n2) Why is the total number of samples summed over all $D_n$ is equal to the number of clients $N$. It basically means that there's only 1 data sample per client. But clients may generate/use many data samples. Am I missing something?\n\n*Typos*:  \n- line 126: \"to breaks the barrier\"\n- line 136: the $c$ variable should be in bold font, I guess\n- line 147: \"that larger\", missing \"is\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "FL methods are traditionally connected with privacy issues. I have outlined my privacy-related questions in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "oeEG34CrlK", "reviewer": "Reviewer_PX5C", "summary": "The paper studies personalized federated learning (pFL), where the local models have different structures and sizes. The proposed method enables each client to maintain a personalized soft prediction at the server-side. Empirical results show the strength of the proposed solution.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a knowledge coefficient matrix at the server side to guide the local training. The pFL is a timely topic and the proposed approach is also interesting. \n\nOriginality: This work is a novel combination of well-known techniques.\nQuality: The submission is technically sound, and all claims are well supported. \nClarity: The submission is clearly written and well organized.\nSignificance: The others (researchers or practitioners) are likely to use the ideas or build on them.\n\nStrengths: \n1.\tWriting is clear. Details of each component are explained and reproducible from the appendix and the code.\n2.\tThe methodology is reasonable. The proposed knowledge coefficient matrix is well-designed with intuitive justifications. \n3.\tThis paper presents non-trivial performance improvement against SOTA pFL methods on 3 popular benchmark datasets.\n\nWeaknesses: \n1.\tThe introduction of the knowledge coefficient matrix may raise the concern of privacy leaks and the communication cost. \n2.\tThe use of the public data in step 2 in Figure 1 conflicts with the general FL setting.\n3.\tTo my best knowledge, there are some other personalized FL works (Hierarchical Personalized Federated Learning for User Modeling, WWW'21; Exploiting Shared Representations for Personalized Federated, arXiv:2102.07078), which need to be listed as baselines.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "nKTlxh5Yi5", "reviewer": "Reviewer_eENH", "summary": "This paper proposes a personalized group knowledge transfer training algorithm (KT-pFL) based on knowledge distillation that deals with model architecture heterogeneity in FL systems.\n\nThe authors claim the following contributions:\n- This paper is the first to study personalized knowledge transfer in FL.\n- Proposing the ‘knowledge coefficient matrix’ to identify the contribution from one client to others’ local training.\n- Providing theoretical performance guarantee for KT-pFL and conduct extensive experiments.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros/Cons**:\n\n[+] The problem this paper aims to solve is novel,  to the best of my knowledge this is the first work to address model architecture heterogeneity in FL.\n\n[+] The paper is well written and structured. \n\n[+] Good theoretical analysis on KT-pFL's performance.\n\n[-] The usage of knowledge coefficient matrix $\\textbf{c}$ makes this method unfeasible for large-scale FL systems. $\\textbf{c} \\in \\mathbb{R}^{N \\times N}$ where $N$ denotes the number of clients. Previous works in P/FL the Hub/Server transmits the model parameters which can be at extreme case billions of parameters. If we take a small federation size of $10K$ clients we will transmit $0.1$ billion parameters in each COM. The proposed approach will be effective (in terms of communication costs) only on small-scale FL systems. \n\n[-] A major limitation is the usage of public data and its size. In the experiments, the authors used $3000$ samples for their public data which for CIFAR-10 is $5%$ of the data. In Healthcare FL systems it would be hard to obtain such public data, especially in this size. The authors did not provide any discussion related to the public data and did not explain how it was partitioned. Did the baseline methods use these $3000$ samples or they are held out?\nI also think that an important addition for this paper should be an experiment that investigates the effect of the public data size on performance.\n\n[-] The authors did not address more recent approaches in PFL [1,2] and did not compare to them.\n\n[-] The update rule of KT-pFL involves three update loops: (i) local training, (ii) distillation, (iii) c update.  This may lead to asynchronous between clients, in FL we aim to mitigate it as it costs time and resources. In addition, it will be interesting to check what is the clock time of KT-pFL compare to FedAvg.\n\n[-] All the clients in the federation take part in each communication round - this is a big limitation for large-scale federations, in real life FL systems federations can contain millions of clients such that training this federation will cost lots of computational power and money. Other PFL methods sample a subset of clients at each COM round (most of them sample 5 clients per round). This limitation raises another question, have the authors run all methods on the same number of communications? each COM round in KT-pFL performs $N$ communications as $N$ denotes the number of clients, while for pFedMe and [1,2] less amount of communications occur per COM round. In FL we care about the number of communications it takes the model to converge.\n\n[-] \"the learning rate is set to $0.01$ for EMNIST and Fashion MNIST and $0.02$ for CIFAR-10.\" It is not clear if the mentioned learning rate values were fixed after HPs search or not and if these values were used for the baselines as well. From the text \"All datasets are split randomly with 75% and 25% for training and testing, respectively\" I understand that no HPs search was done and these values were used for the baselines as well.\n\n[-] Intuitive baseline for the Hetro FL experiment would be to assign each group of clients (that have different model architecture) with a server and train them separately.\n\n[-] Experiments were conducted on a limited number of clients (20), in real-life FL systems, we have thousands and millions of clients I encourage the authors to run experiments on larger federation sizes e.g. 500 clients.\n\n[-] Missing information on homogenous FL experiment - it is not what \"homogenous\" refers to if it is only for the model architecture or to the data distribution as well. \n\n[-] The authors should provide STD analysis (i.e. running the experiments on multiple seeds).\n\n**References**:\n\n[1] Shamsian, Aviv, Aviv Navon, Ethan Fetaya, and Gal Chechik. \"Personalized Federated Learning using Hypernetworks.\" arXiv preprint arXiv:2103.04628 (2021).\n\n[2] Zhang, Michael, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. \"Personalized Federated Learning with First Order Model Optimization.\" arXiv preprint arXiv:2012.08565 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should discuss the limitations of their approach in addition to potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "at least 5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach for handling model architecture heterogeneity in federated learning. Theoretical analysis and numerical experiments demonstrate the efficacy of the new method. Good writing and clear structure also make the paper stand out. Based on the reviews, I recommend acceptance. Meanwhile, please make sure to incorporate the reviewers' comments and the rebuttal into the final version.
Decision: Accept (Poster)