Personalized Heterogeneous Federated Learning with Gradient Similarity
Authors: Jing Xie
Xiang Yin
Xiyi Zhang
Juan Chen
Quan Wen
Qiang Yang
Xuan Mo
Venue: ICLR 2022 Submitted (2022)

Abstract: In the conventional federated learning  (FL), the local models of multiple clients are trained independently by their privacy data, and the center server generates the shared global model by aggregating local models. However, the global model often fails to adapt to each client due to statistical and systems heterogeneities, such as non-IID data and inconsistencies in clients' hardware and bandwidth. To address these problems, we propose the Subclass Personalized FL  (SPFL) algorithm for non-IID data in synchronous FL and the Personalized Leap Gradient Approximation  (PLGA) algorithm for the asynchronous FL. In SPFL, the server uses the Softmax Normalized Gradient Similarity  (SNGS) to weight the relationship between clients, and sends the personalized global model to each client. In PLGA, the server also applies the SNGS to weight the relationship between client and itself, and uses the first-order Taylor expansion of gradient to approximate the model of the delayed clients. To the best of our knowledge, this is one of the few studies investigating explicitly on personalization in asynchronous FL. The stage strategy of ResNet is further applied to improve the performance of FL. The experimental results show that  (1) in synchronous FL, the SPFL algorithm used on non-IID data outperforms the vanilla FedAvg, PerFedAvg, and FedUpdate algorithms, improving the accuracy by $1.81\!\sim\!18.46\%$ on four datasets  (CIFAR10, CIFAR100, MNIST, EMNIST), while still maintaining the state of the art performance on IID data;  (2) in asynchronous FL, compared with the vanilla FedAvg, PerFedAvg, and FedAsync algorithms, the PLGA algorithm improves the accuracy by $0.23\!\sim\!12.63\%$ on the same four datasets of non-IID data.

PERSONALIZED HETEROGENEOUS FL WITH GRADI- ENT SIMILARITY
Anonymous authors Paper under double-blind review
ABSTRACT
In the conventional federated learning (FL), the local models of multiple clients are trained independently by their privacy data, and the center server generates the shared global model by aggregating local models. However, the global model often fails to adapt to each client due to statistical and systems heterogeneities, such as non-IID data and inconsistencies in clients' hardware and bandwidth. To address these problems, we propose the Subclass Personalized FL (SPFL) algo- rithm for non-IID data in synchronous FL and the Personalized Leap Gradient Approximation (PLGA) algorithm for the asynchronous FL. In SPFL, the server uses the Softmax Normalized Gradient Similarity (SNGS) weight the relation- ship between clients, and sends the personalized global model to each client. In PI.GA, the server also applies the SNGS to weight the relationship between client and itself, and uses the first-order Taylor expansion of gradient to approximate the model of the delayed clients. To the best of our knowledge, this is one of the few studies investigating explicitly stage strategy of ResNet is further applied to improve the performance of FL. The experimental results show that (1) in synchronous FL, the SPFL algorithm used on non-lID data outperforms the vanilla FedAvg, Per-FedAvg. FedUpdate, and pFedMe algorithms, improving the accuracy by 1.81 18.46% four datasets (CIFAR 10, CIFAR 100, MNIST, EMNIST), while still maintaining the state of the art performance IID data; (2) in asynchronous FI., compared with the vanilla FedAvg, Per-FedAvg, and FedAsyne algorithms, the PLGA algorithm improves the accuracy by 0.23 12.63% the same four datasets of non-IID data.
personalization in asynchronous FL. The
INTRODUCTION
With the popularity of smartphones, personal computers, and other devices, the data stored them has increased dramatically. These data are related to each other but exist independently on different devices. Also, their owners often not willing share their private data, which prompted Fed- erated Learning (FL.) (McMahan et al., 2017). Conventional FI. is a distributed machine learning framework in which multiple clients train a global model together under central server's coordina- tion without sharing their local data. It's common process is as follows: (1) At each global iteration, the central server broadcasts its current global model to the client; (2) With the help of the global model, the client updates its local model which trained on its local data, and sends it back to the central server; (3) The central server aggregates the local models to get a new global model. The FL process will repeat these three steps until convergence. Under the premise of privacy protection, it can obtain a shared global model with higher generality. Nonetheless, the wide application of FL. still is facing the following two major challenges:
Statistical heterogeneity. As the user preference of each device may be different, the data distribution on each device may also be inconsistent. Consequently, the data as a whole on all the clients may unbalanced and non-lID. This may cause undesirable performance of the FL participants.
Systems heterogeneity. Due the differences of users' hardware and network bandwidth, conventional FL models are prone to generate stragglers whose models are easy to be dis- carded after server's time-out.

---

Firstly, address the statistical heterogeneity, the personalization of FL is widely studied. The ma- jority of the available personalized FL (PFL) methods focus on one shared global model's 's parameters updating by client's 's fine-tune. FedAvg (McMahan et al., 2017) is treated by (Jiang et al., 2019a) as a client's meta-learning- FedMeta (Chen et al., 2018), also personalized FL. based on meta-learning, generates a local and personalized model for each client using a common global model. Different from the aforemen- tioned methods, we propose Subclass Personalized Federated Learning (SPFL) algorithm, which utilizes the Softmax Normalized Gradient Similarity (SNGS) to generate global models with each specific to one client for its parameter fine-tune.
process fine-tuning on one shared global model with higher accuracy.
Secondly, to tackle the systems heterogeneity, the server of an asynchronous FL algorithm named FedAsyne (Xie et al., 2019), can immediately perform aggregation after receiving a local model, re- ducing the waiting time for stragglers. In TWAFL (Chen et al., 2019), time-weighted aggregation strategy is used by server to handle stragglers according to the staleness of the received model pa- rameters. Although these asynchronous FL algorithms can overcome some difficulties encountered in synchronous aggregation, such as waiting for the response from client, they are still inefficient for directly aggregating the received local model the server, without taking the iteration gap of stragglers' local models at global communication round into consideration. Therefore, we pro- pose the Leap Gradient Approximation (LGA) algorithm using SNGS predict the local model of the stragglers in each global communication round. These local models estimated by server are also aggregated with non-stragglers' local models on the server to attained one shared global model. Further, to consider the real scenarios where system heterogeneity and statistical heterogeneity often co-exist, personalization is employed in our asynchronous FL. Before each individual personalized model for client is dispatched from server, it is computed by using the shared global model and the SNGS value between server and that client. We refer this personalized asynchronous FL approach as Personalized Leap Gradient Approximation (PLGA) algorithm.
Contribution The main contributions of this paper towards personalized heterogeneous FL can be summarized as follows: (1) We propose the SPFL algorithm for both IID and non-III data in personalized synchronous FL. where the server uses the SNGS to weight the relationship between clients, and delivers the personalized global model to each client; (2) We propose the PLGA algo- rithm for the personalized asynchronous FL., in which the server also applies the SNGS to weight the relationship between client and itself, and uses the first-order Taylor expansion of gradient to approximate the model of the delayed clients; (3) The stage strategy of ResNet is applied to improve the performance of both SPFL and PLGA algorithms.
2 RELATED WORK
In general, the personalized process in FL can be achieved by variety of ways. The work (Hard et al., 2018) shows that by adding user-related features into the local inference process to get a personalized model may be better than the global one. The completely decentralized algorithms for each client by smoothing some similar tasks or model parameters between similar data distributions studied in (Bellet et al., 2018; Vanhaesebrouck et al., 2017). Algorithms are also proposed to achieve personalization by jointly learning the model on a similar graph structure (Lalitha et al., 2019; He et al., 2021). FedFOMO (Zhang al., 2020) another approach PFL, where each client only federates with other relevant clients for model aggregation. The pFedHN (Shamsian al., 2021) algorithm is proposed using hypernetwork to generate different personalized model for each client.
Furthermore, PFL can also be studied by its relationship with other machine learning paradigms. The close relationship between FL and meta-lcarning is analyzed by (Khodak et al., 2019; Jiang et al., 2019b). Following the classic meta-learnigg algorithm named MAML (Finn et al., 2017), Reptile algorithm (Nichol et al., 2018) is proposed, which has a close relationship with the Federated Averaging algorithm. When each individualized model learning is treated as a task, such a scenario can be viewed as multi-task learning(Zhang & Yang, 2017), in which each task learns task- related model. MOCHA (Smith et al., 2017) is designed by regarding FL as a multi-task, which solves communication efficiency and fault tolerance. To provide a good global model, there are PFL applying transfer learning and domain transfer (Mansour et al., 2009; Cortes & Mohri, 2014; Ben- David et al., 2010). Model mixing is another strategy approach PFL (Deng et al., 2020; Hanzely

---

& Richtárik, 2020; Liang al., 2020). Alternatively, Other PFL methods apply clustering (Mansour et al., 2020: Ghosh et al., 2020; Duan et al., 2020). Finally, pFedMe (Dinh et al., 2020) applies Moreau Envelopes clients' regularized loss functions to achieve personalization.
There are just few studies on asynchronous FL. TWAFL (Chen et al., 2019) aggregates a temporally weighted local models the server. FedAsyne (Xie et al., 2019) introduced an algorithm balance the parameters of the updated model and the last global model. Meanwhile, the distillation technique has also been applied in (Li & Wang, 2019; Bistritz al., 2020).
3 METHOD
Our method generates a personalized model for each client during the server aggregation process. Therefore, the initialization model of each client is personalized. the same time, we also consider the contribution of different neural network layers to the personalized model. We will introduce the application of this personalized algorithm to synchronous FL and asynchronous FL in the following subsections.
Algorithm Subclass Personalized Federated Learning (SPFL) Algorithm
Input: initialized global model parameters w"; initialized SNGS matrix S, E RN*', initialized personalized local model of each client's {we] = w"; update frequency T and the number S of SNGS matrix; client learning rate 83; server learning rate a; local epoch E; local batch
size B of each client
Output: Model Parameters {we}
Def MainLoop:
for t 0, 1.... do
perform the following steps in parallel for each client C3 E C if t%r
then w gi N
1
Σ w
{g = ClientUpdate (w'y (,,9)
3, Update ((9)))
{gf} = ClientUpdate C1,3)
{w ++1+1 ServerUpdate (ww ( { o)
Def ClientUpdatelwgg.ccc
w- w
for 0 E;e + + do
for batch size in B do
/ + i 2 (f
return {wys w Def UpdateSMarrix ( (gi)):
for s E S do perform (3) and (6) return (i.
Def ServerUpdate (fwta (g.
for s E S do perform (7) to get 1+1
return [w++]]
++1
Definition of personalized FL algorithm FL, there is set N clients C (C i 1,.., N.
Define the dataset all clients D {D,), 1, 1,..., N. with D, as the corresponding dataset of C, The sample size of each datasets D, is D2 (I,, Vú) is a sample of D,, with x; E Rd as the corresponding input feature and yj R2 the corresponding input label. The prediction model of

---

client C2 defined ú; f(w,, 2)), where w, R is the model parameter. The dataset on each client C. is assumed to obey distribution of P.. And the loss function on each client C2 is (w,w). The optimization goal for each client is:
ID.
argmin (w; BBD:::=BLL (...,,, R
Σ
L(((ww,))).)))
Dj
3.1 SUBCLASS PERSONALIZED FL (SPFL) ALGORITIIM
We compared the performance of each client's independent training (i.e., the model in each client performs SGD only on local data available, and model averaging is not performed) and conventional global FL algorithms, such as FedAvg. The results Appendix B. show that FL usually outperforms each client's independent training on IID data, demonstrating the global model trained with multiple client data improves the model generalizability ability of each client. On the other hand, for some clients, independent training has better performance on non-IID data. Our experiment Appendix B illustrates that the global FL model cannot adapt well to the heterogeneous data. To tackle this problem, we propose personalized FL. algorithm that improved the performance of conventional global FL algorithms on non-lll) data. At the same time, it still maintains high performance on IID data.
Firstly, we introduce similarity matrix S E RNNN to model the relationship between different clients, with S(i, represents the similarity between C, and Cj. If the data distributions P. and P3 similar, then the value of Sii.j) is relatively high, and vice versa.
For any two clients, C2 and C,, supposing they receive the global model parameters w at the same time t. After the local update of both clients using w', their local model parameters are wf and wf, respectively. Then the SGD updates of the corresponding gradients of w and w/ are:
gi = w² wi 9] = w² w/y
The correlation between C. and Cj based on the global model w is measured by the cosine simi- larity of the gradient updates of and gj. as calculated in (3):
of T o
((i
llo
||
where || is the vector L22 normal. And the aggregation strategy on the server for C2 is as follows:
N
[D D ((1,))))
w 2+1 = : a Σ
where a is the learning rate of the server. As the gradient itself is extremely directionality sensitive, when S(i, j) is negative, multiplying it with the original gradient 9i will reverse the gradient direc- tion. Therefore, softmax function is applied to make Sii,j) nonnegative and normalized as Sii,j) (), the softmax version of similarity relationship between C₁ and C,,, is defined as:
((i.))
(ii
(5)
Σ e((.))
The experimental results Appendix B show that different layers influence personalization differ- ently in the convolutional neural network (CNN), Inspired by ResNet, we use stage strategy that contains a set of network layers to measure the similarity between clients. Therefore, the similarity matrix Š is calculated as S, based on a stage as in (6), where its subscript s indicates a stage:
(.,(3)
$.(1.)) ΣN eS.(1.))
(6)

---

Finally, the aggregation strategy the server for C2 base stage is defined as:
N
Dyl
++1 W 82 w/g
Σ
(1.
a
[D
where a is the learning rate the server, Pzj and wt, are the updated gradient and model parameters of the stage on the client at iteration round t, respectively.
As the model changes with iteration round i, so does the gradient of the model. Consequently, the similarity matrix $ obtained based on the gradient will also change. Therefore, the similarity matrix is updated regularly during the training process. During each update, the model parameters of all clients are aggregated with equal weight to get Wi of (2), and the similarity matrix is calculated based w/y In the view of cach aggregation, it is still similar to the federated average algorithm. However, each client obtains a personalized model based on its similarity matrix in the training rounds in each aggregation. In this process, each client is individually regarded subclass, and the algorithm is called Subclass Personalized Federated Algorithm (SPFL). The algorithm is shown in Alg.
3.2 PERSONALIZED LEAP GRADIENT APPROXIMATION (PLGA) ALGORITHM
This section introduces an asynchronous updating strategy that addresses the system heterogeneity problem mentioned in Section The server can perform model parameters aggregation immedi- ately after receiving few local models from clients within a small time window, overcoming the inefficiency of the conventional synchronous updating strategy, which either updates the model pa- rameters after all the clients uploads their local model or discards the straggers.
Algorithm Personalized Leap Gradient Approximation (PLGA) Algorithm
Input: Initialized global model parameters w"; local step E; local batch size B of each client; client learning rate f3: server iteration round T; client number
Output: Model parameters (w and (w*)
Def MainLoop ( ([w)) w
for t 0,t < T + do ClientUpdate(u""
(Ck,,) ,Ck.8)
The server receives multiple local models {wf) > the same time within small time window
for the straggers w/, do (8), (13), (11), (14) in sequence;
the server aggregates all clinets" model:w = No E wf
1
Def ClientUpdate (w², (Cg,B):
w/ w²
for c 0;e < E;e + + do
for be
/ w w (f (2))))))))
return mf.
Definition of asynchronous FL Assuming that the FL process consists of T global iterations that the number of the server aggregation. During each global iteration s, after receiving a global model parameters wx*,, S T, from the server, the client C, uses the local data D; for a local training of E epochs. After finishing the local training, each client uploads its local model w to the server for aggregation. Then the next global iteration performed. On the other hand, as shown in Fig. I for the conventional asynchronous FL process, the model parameters w and wi"" uploaded by different clients C₂ and C, respectively. have equal importance in the server aggregation round + 1, ignoring the fact that model w performs several more SGD (local updates) than model wi that is called straggers. Intuitively, as the server iteration round increase,the gap between server and stragglers get bigger, the adverse impact server aggregation increases.

---

to
t2
tr
r+1
whi
1
wfr
w w = (Wff wfe))
server
f
((wfo))
F(w w(,
F F(w(
ofo
W o
w
Wfr
Ci
Dj - W.
Dj - w¹"
Dj w
W
w 5
Cj
Dj w
C
Dg w
W k
Figure 1: The illustration of asynchronous FL. The straggler Cj need more computation time, and the straggler Cs has poor network bandwidth. The server aggregates models immediately and does not need to wait for all clients in asynchronous FL. F is a specific aggregation strategy, t, is time of the 7-th server aggregation round.
Leap Gradient Approximation (L.GA) Here, we let the server aggregation round 8 = to sim- plify introduction. First, the client Cg receives the global model w delivered by the server the global iteration round O for initialization, and the wi is the local model of client Cß that is trained by local data Dz. w is the global model obtained right after the server performing an aggregation at iteration round 1. So the gradient similarity S(k) of the server and the client C2 can be calculated as follows:
((k = (w" = (ux w())
(8)
| w 11 woo
We also apply the softmax function to normalize 5(k).
(A()
5(k) el e(())
(9)
As shown in as in Fig. 1, w is directly used compute the global model w/++1 in conventional asynchronous FL. To avoid this, assuming that the training loss function is Li-Lipschitz, inspired by DC-ASGD (Zheng al., 2017), alleviate the update gap between stragglers and the server, we use the approximation of the first-order Taylor expansion of the gradient to estimate the model of straggler Ck in the current server iteration round by (10):
wh w + 5(k) (wt w()) @ (w - w0)) @ (w*r - w'1) + (wf - w
(10)
where w is the local model of straggler client C₁ the server aggregation round т, and @ is the element-wise product. The last two terms on the RHS of (10) is the prediction of the changes of who which we called as gradient leap; especially, Diag (ww w')) @ (wh - -wfo)) ) is the approximation of the gradient of the gradient of w The first term on the RHS is the initial value of w Furthermore, we also proposes another view on calculating wf: as in (11)
w = - + S(k) (w =0) @ (w - w'o) @ (w- w²1) + (w - w0))
(1I)
where w is the global model at the server aggregation round 7. The only difference between (10) and (11) is their first terms on their RHS. In (11), it says that we want to predict w based on current global model w', yet still use the prediction of the changes of wh in the previous FL iteration round.
Personalized Leap Gradient Approximation (PLGA) The personalized model for each client C₁ the server aggregation round T is as follows:
f. whp = f0 + (1 3(k)) (wt - w00) + +5(k) (10' - -w
(12)

---

where S and w derived from (9) and (10) (11), respectively. Similar to (6), we can also apply stage strategy to St)), and obtain:
e e8a(M)
(k)
(13)
E e eS.(k))
and
whhp = v0 + S. (1- (weh - w + (k) (w - -w
(14)
The personalized leap gradient approximation (PLGA) algorithm is introduced in Alg. 2.
4 EXPERIMENTS
In this section, we first evaluate the performances of SPFL compared with Fed/Agg(MeMahan et al., 2017), Per- FedAvg(Faalla et al., 2020), FedUpdate(Jiang et al., 2019a) and pFedMe(Dinh et al., 2020) when the data distributions are IID and non-IID, we also explore the effect of hyperparameters N, E, We then compare PLGG with FedAvg (both synchronous and asynchronous settings), Per-FedAAgg(Falla))) et al., 2020), FedAsync(Xic et al., 2019) when the data distributions non-IlD. The source code is available at https:://// /ttp:::://ttttti github hamm//ooddaya//eee.
Experimental Settings For both of SPFL and PLGA experiments, use four datasets, namely, CIFARIO (Krizhevsky et al., 2009), CIFARIO0 (Krizhevsky et al., 2009), MNIST (Lecun et al., 1998), EMNIST (Cohen et al., 2017) datasets. These datasets are suitable for testing different real- world scenarios in the proposed FL. framework, such as non-IID, balanceless datasets. We run the experiments with 10 clients and server, where each client has access 6,60,6,32 classes of the above datasets, respectively. The detailed introduction and construction of the datasets can be found in Appendix B. All datasets are split randomly with 80% and 20% for training and testing, respectively. All experiments were conducted using Py Torch (Paszke et al., 2019) version 1.1.0. All the baselines have the same parameters and network structures our model. We use 10 seeds get a average result for each method.
4.1 SPFL EVALUATION
In the synchronous FL, we compare the proposed SPFL with the FedAvg, FedUpdate, Per-FedAvg respectively. Although the FedAvg algorithm is not personalized learning algorithm, its exper- imental results can still be helpful as a referances. FedUpdate and Fed-FedAvg are FL algorithms dedicated to personalization based on the meta-learning And PFedMe is using Moreau Envelopes to get a PFL model. The common parameters are fixed as B = 128, a = 3 = 0.01, = 10, E 1, S 2. We provide an extensive ablation study on design parameters' choices in Appendix C. We also test the personalized learning based on weight fusion, and the aggregation strategy changes from w; w; + J7 E3 Sag; to w; = J E, S.wy. The weight-based personalized SPFL algorithm is defined as SPFL-W. Noting that twice local updates, which mimic MAML, are used in SPFL, to explore the influence of the number of local updates the experiment, also add two models using once local update on the clients as ablation studies: SPFL-w (1-step) and SPFL (1-step).
As shown Table 1, the accuracies on the four datasets are quite different, and decreasing in the order as MNIST, EMNIST, CIFARIO, CIFARI00 It also reflects the difficulty of each dataset. On a whole, the handwritten number recognition relatively simpler than the image classification. And the two datasets within the same category are also different, such as MNIST and EMNIST, or CIFAR10 and CIFAR100. For all methods evaluated, the accuracy on IID dataset is always higher than that of non-IID. Therefore, the evaluation results on the four datasets are relatively objective and generalized, and not overfitting to one dataset.
SPFL performs much better on four non-lll) datasets, compared to FedAvg, FedUpdate and Per- FedAvg. For IID datasets, Per-FedAvg has the best results except on EMNIST dataset. Nonetheless, the performance gap between SPFL and the best methods is still acceptable reason that SPFL performs a little bit worse on IID datasets due to the delay of the gradient similarity calculation. Considering the actual application scenarios of personalization, in which the possibility of IID data on the client is relatively small, our algorithm can bring more benefit in the real environment.
IID datasets. The

---

Table Comparison of clinet average accuracy (%) SPFL. and baseline algorithms on III datasets and non-IID datasets. The best and second to the best performance are bolded and underlined, respectively.
MNIST
CIFARIO
EMNIST
CIFAR100
Methods
IID
non-
IID
non non-IID
IID
DOn: ILD
IID
DO0-
FedAvg
98.46:1:0221
96.52:00.21
67.00:1:.38
18.76+
90.41:1.88
84.42:0.35
27.39+
18.79:1104
FedUppaae
98.46.10..2
97.211:0.35
66.45:1:.37
59.45:1.22
19.95.1..88
26.6111:1008
21.13:1:.86
Per Ber-FedAvg
08.89-10.115
97.19:10.42
74.00:1.304
15.65.1141
91.27 1.84
85.32.10.41
35.03.1..221
24.79:1:.009
pledMe
98.80-1:016
97.18:20.22
72.22:3.15
55.47433339
91.03-+1.83
85.30:10.68
33.93:40.76
24.20:::133
SPFL-W (1-x)
98.46.10.18
96.52.10.33
67.03.13.44
48.96.11.07
90.32.1..13
83,23
27.49.10.89
18.87:10.93
SPFL-w
98, 98.90:1.0116
91.22:10.36
74.044...18
1..36:t..14
90.9112.11
85.25
34.90:t1.110
24.73:1:.99
(1-s.
98.35
97.71110.16
61.26.10.44
89.94.12.85
SPFI.
98.82 18.82:10.16
98.33-10.16
71.9143.55
67.22:11.15
89.53-22.30
87.49:10.72
28.52:::.01
24.97:1100
Compared to FedAvg and independent training of each client in Appendix B, SPFL outperforms both no matter on IID datasets or non-IID datasets. To differentiate the influence of gradient and weight aggregation strategy, we observed that SPFL-w is slightly better in the case of IID data, but SPFL much better than SPFL-w on non-IID data. Nonetheless, SPFL-w outperforms the FedAvg and the independent training model both on IID and non-llI datasets.
The expermental results also show that the twice local updates dominate its counterpart, which indicate that MAML alike updates is helpful in personalization than once local update.
4.2 PLGA EVALUATION
In the asynchronous FL, the proposed PLPA is compared with the FedAvg (in the synchronous and asynchronous conditions), FedAsync and Per-FedAvg. Although the FedAvg and Per-FedAvg are synchronous federated algorithms, their experimental results can still be helpful as references here. FedAvg(Sync) means that discard the straggers when server aggregate, while FedAvg(Asyno) means the straggers participate directly in server aggregation. FedAsync is an FL algorithm dedicated to asynchronous FI.. The hyperparameters are fixed as 64, 8 0.001, 20, 7 = 800. Based on ablation experiments, we find that these parameters have best performance. We use N = 10 clients evaluate the performance of the PL.A algorithm. To make the simulation of the asynchronous environment simple and controllable, we fix the set of straggler of each asynchronous FL experiment as S, and the rest clients in set C S\S participate in each server iteration round synchronously. For stragglers in S, we let them missing server aggregation round periodically with different fixed miss- ing period as 1, .,. (S], where the values correspond to the number of server iteration rounds. Suppose that the fixed missing period of the straggler in S is 2, the straggler will participate one server iteration and miss two server iterations, and then participate one server iteration and miss two server iterations again, and so on. Here, we use non-IID dataset (see Appendix B) for the evaluation as it is the most common data case in asynchronous FL. The experimental results in Table 2, where the total number of the stragglers is |S| = 5, show that the clients' performances of PLGA on the four non-IID datasets have notable advantage over the other baseline algorithms. Note that when [S] = 0, it is equivalent to synchronous FL. We also notice that PLGA is better than LPA as a result of personalization, and that w/: computed by (11) is better than by (10) for PFGA.
Table Comparison test accuracy (%) of the PLGA and baseline algorithms on non-IID datasets. The best and second to the best performance bolded and underlined, respectively.
MNIST
CIFARIO
EMNIST
CIFARI00
Methods
client
client
client
client
FodAvg (Async)
97.24 0.32
97.61=0.12
42.09-+2.07
46,5242.63
18.1003.68
74.05+22.53
1.15
20.98:10.93
FedAvg (Sync)
97.31-1:.39
97.72.10.18
1..96.111.6
46.6612..4
61.05
74.22.42.97
19.38.1..00
20.98.1-1.15
Per-FodAvg
97.27=:0.35
97.68-00.15
41.79++2.3
45.48422.59
55.55-23.43
73.74:22.61
1..1
20.71+0.94
FedAsyne
97.18.10.44
97.5910.116
43.59.1263
46.51
61.09.1..39
74.74.12.59
18.63100.98
20.53:11.01
LGA
97.48:10.28
97.82-10.15
44.06:1:3.11
45.73413.68
62.02-22.73
73.93:42.711
19.41
10.74:21.15
PLGA
97.83:10.16
97.82=20.16
46.11::3.22
45.96+3.59
68.18=2.31
73.91:t2.72
10.73::1.15
10.84::1.12

---

5 CONCLUSION AND FUTURE WORK
This paper studied the personalized heterogeneous FL with gradient similarity. We tackled the sta- tistical and system heterogeneity in FL in two scenarios, namely synchronous FL and asynchronous FL with SPFL and PLGA algorithms, respectively. In both SPFL and PLGA algorithms, SNGS is proposed capture the similarity between the participants of FL, ether among clients between sever and client. Stage concept is also studied in the computing of SNGS. The experiment results demonstrate that the proposed algorithms have excellent performances maintaining the state of the art performance on IID data. We are going to further work on the theo- retical proof of the convergence of the proposed algorithms, and also the sampling strategies in the evaluation of SNGS between clients to alleviate the communication and computation overheads.
non-IlI data, while still
0.20
Reokegsasyncc
Fodlegläsynd
TechegiSynct
0.18
TeskvpsSyecl
Pertedting
Putredkog
USAWW"!
0.15
LOM
0,350
PLGA0W
The total number the stragiles
The total number of the stragpiers
(a) CIFARIO result
(b) CIFARI00 result
EMNIST
0.70
5.400
Clannt Awerage Acrentry
3.00
0.55
Porfedleg
Teckeppe
(GRAW!
C.965
tedieyne
5.900
mumber strggless
The istal strogplers
(c) EMNIST result
(d) MNIST result
Figure 2: The average accuracy results of the 10 clients of PLGA algorithm the four datasets
REFERENCES
Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, and Mare Tommasi. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statis- fics, pp. 473-48711111 PMLR, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 7911)1151-175 2010.
Mai Bistritz, Ariana Mann, and Nicholas Bambos. Distributed distillation for on-device learning. Advances in Neural Information Processing Systems, 33, 2020.
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv: arXivv/802.000 880200887777 2018.

---

Yang Chen, Xiaoyan Sun, and Yaochu Jin. Communication-efficiie federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE transactions neural networks and learning systems, (2(10))42294425533 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN, pp. 2921-2926, 2017. doi: 10.1 1011///NN..001.7996666
Corinna Cortes and Mchryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126, 2014.
Y. Deng, M. M. Kamani, and M. Mahdavi. Adaptive personalized federated learning. 2020.
C. T. Dinh, N. H. Tran, and T. D. Nguyen. Personalized federated learning with moreau envelopes. 2020.
Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficient clustered federated learning via decomposed data-driven measure. arXiv preprint arXiv:2010.06870. 2020.
A. Fallah, A. Mokhtari, and Ozdaglar. Personalized federated learning: A meta-learning ap- proach. 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp- 1126-1135. PMLR, 2017.
A. Ghosh, J. Chung, Y. Dong, and K. Ramchandran. An efficient framework for clustered federated learning. 2020.
Filip Hanzely and Peter Richtárik. Federated learning a mixture of global and local models. CoRR, abs/2002.05516, 2020. URL https: /ttpiv.avvv//v.//////.e../...ava org wttp:/rg//g/........oo 05516.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint / 2018
Chaoyang He, Keshav Balasubramanian, Emir Ceyani, Yu Rong, Peilin Zhao, Junzhou Huang, Mu- rali Annavaram, and Salman Avestimehr. Fedgraphnn: federated learning system and bench- mark for graph neural networks. 2021.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning. arXiv preprint arXiv: 12488, 2019a.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning. arXiv preprint arXiv:1909 12488. 2019b.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient- based meta- learning methods. arXiv preprint arXiv:1906.02717, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anusha Lalitha, Osman Cihan Kiline, Tara Javidi, and Farinaz Koushanfar. Peer-to-peer federated learning graphs. arXiv preprini 1rXiv:1901.11191 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-bass based learning applied to document recog- nition. Proceedings of the IEEE, 86(11))2278-2324, 1998. doi: 10.1109/5.726711.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.088
2019.
Paul Pu Liang, Terrance Liu, Ziyin Liu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. CoRR, abs/2001 hos/2001.01523, 2020. URL http::/ttaareeeeeeee wwwpw/wwwwww///////////ooese/ 52333.

---

Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. Computer Science, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicatonn-efficii learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02900 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high -eg-pperformance deep learning library, 2019.
A. Shamsian, A. Navon, Fetaya, and G. Chechik. Personalized federated learning using hyper- networks. 2021.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv: 2017.
Paul Vanhaesebrouck, Aurélien Bellet, and Marc Tommasi. Decentralized collaborative learning of personalized models over networks. In Artificial Intelligence and Statistics, pp. 509-517, PMLR,
2017.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934, 2019.
M. Zhang, K. Sapra, S. Fidler, S. Yeung, and M. Alvarez. Personalized federated learning with first order model optimization. 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint 1rXiv:1707.08114 2017.
Shuxin Zheng, Qi Meng. Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie- Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning, pp. 4120-4129. PMLR, 2017.
A APPENDIX
B PREPARATION IID AND NON-IID DATASETS
Here we choose four public benchmark datasets MNIST, CIFARIO, CIFARI00, and EMNIST, as the original data source to construct the IID and Non-IID datasets. A brief introduction to the four datasets is shown in table 3. It should be noted that all non-III situations in this paper are feature distribution skewed, that is, while the distribution P(X) of feature X on each client may be different, the distribution of conditional probability P(Y of label Y the same. Specifically, the sampling steps designed in this paper the total number of classes in the dataset is K, and the dataset class is D* Supposing the total number of clients participating in personalized FL is N, for client the actual number of classses it has kz, 2 S h < K. The steps of preparing IID and non-IID datasets are as follows:
follows: Assuming that the original dataset is D,
1. For any class k. its dataset D is randomly and equally partitioned into N parts, where N is the total number of clients. Dataset D denotes the n-th partition portion of D* For the generation of IID dataset goto step 2, otherwise goto step

---

MNIST is handwriting recognition dataset, including 0-9, 10 kinds of handwritten characters. The training set contains 60,000 images, and the test set contains 10,000 images. Each image is standardized, the size of each image is 28*28, and the image is grayscale image.
MNIST:
The EMNIST dataset can be regarded as an extended version of the MNIST dataset. The dataset includes 671,56 0,15585 numbers and letters in upper and lower case images in 62 categories. There are 47 categories except for the case.
EMNIST:
The CIFARIO dataset includes 60,000 32*32 color images 6000 images in each category. The training set is 50,000 sheets, and the test set is 10,000 sheets uniformly distributed for each category of the training set and the test set.
categories, with
CIFARIO: 10:
The format of this dataset is the same CIFARIO; the difference that it contains a total of 100 classes of images, and each class has a capacity of 600 images. The total number of datasets is also 60,000, and the training data 50,000 sets, and the test data is 10,000. Unlike the datasets of CIFARIO 10, because CIFAR 100 has more types of data, it can have higher degree of freedom in constructing simulated FL data distribution.
CIFARIO0:
Table 3: Overview of the original four datasets
1
3
4
3
IID
D2
3
C1
C
C3
C4
Non-lID
3
3
2
Figure 3: Non-IID and IID datasets generation example
For IID dataset, determine the number of classes each client as k.. Sample kt classes from the K classes, and denote the sampled class sets K sample: K samptel k For each i, í K Kammple> assign each data partition D
I, to each client without overlapping. Goto step 4.
For non-lID dataset, P. (XX).. the distribution feature X, on each client may not be the same. Here the difference from step 2 is that in non-lII sampling, each client will independently sample number k2 classes. The rest operations are the same as step 2.
4. Dataset each client is partitioned randomly into training and testing sets. And two sets are IID for each client.
When the number of clients is N 4. the total number of classes is K 3, the total number of classes for each client is k, = 2, the generation of IID and non-IID is shown Fig. (Note for non-IIID dataset generataion, C2 and C4 are IID by chance).

---

C ANALYSIS OF PERSONALIZED ALGORITHM
We chose the CNN model architecture in the following table 4. In the experiment, the accuracy rate
Table Convolutional network model structure
Conv (5 x 5 x 20)
MaxPool (2 x 2)
Conv (5 x 5 x 50)
MaxPool (2 x 2)
Full Connect (512)
Full Connect (Number of categories in the data set)
is used as the evaluation index on each client, shown in the formula (15), where TP, TN, FP, and FN represent positive, true negative, false positive, and false negative.
TP TNN
acc TP+TN + FP+PN
(15)
After confirming the experimental parameters settings, we will conduct comparative experiments with the parameters settings in Table 5. The method used in the experiment is the federated average learning algorithm (FedAvg). Table 6 the corresponding experimental result.
Table 5: Federated average algorithm exploration experiment parameters table
Number
datasets
Data distribution Training way Total client train client categories train test
3-1-1
MNIST
IID
Independent
10
2800 700
3-1-2
MNIST
IID
Federated
10
10
6
2800 700
3-1-3
MNIST
non-lID Independent
10
2800 700
3-1-4
MNIST
non-lID
Federated
10
10
6
2800 700
3-1-5
EMNIST
IID
Independent
10
10
32
6100 1500
3-1-6
EMNIST
IID Federated
10
10
32
6100 1500
3-1-7
EMNIST
non-lID non-lID Federated
Independent
10
32
6100 1500
3-1-8
EMNIST
10
10
32
6100 1500
3-1-9
CIFARI0
IID Independent
10
10
6
2400 600
3-1-10
CI
IID Federated
2400 600
3-1-11
CIFARIO
non-lID non-lID Federated
Independent
10
10
6
2400 600
3-1-12
CIFARIO
10
2400 600
3-1-13 CIFARIO0
IID Independent
10
10
60
2400 600
3-1-14
CIFAR100
IID Federated
10
60
2400 600
3-1-15
CIFAR100
non-lID non-IID
Independent
10
10
60
2400 600
3-1-16 CIFAR100
Federated
10
2400 600
The experimental results show that the results of the FL algorithm are often better than the inde- pendent training method when the data is IID. The result of the FL will be worse than that of the independent training when the data is non-IlID. This experimental conclusion shows that the fed- erated average learning algorithm (FedAvg), which executes the average aggregation step of the model parameters, is more suitable for the case where the datasets on the client are IID. has the opposite effect when the data is non-IID. To further explore the influence of the federated averaging algorithm on the model parameters under non-IID, we calculated the average value of each client's upload gradient similarity in different distribution scenarios. The results shown in Fig. 4 and Fig. 5. Based on the above experimental results, we can find that: (1) In independent and synchro- nized steps, the mean of the gradient similarity is close 1. In non-III scenario, the mean of the

---

Table 6:
Federal average experimental results (percentage%)
Number
cl
c2
c3
c4
c5
c6
c7
c8
c9
c10
Average accuracy
3-1-1
95.57
97.92 97.23
97.64 97.09 97.23 97.23
97.51 97.51 98.20
97.313
3-1-2
97.09
98.47 98.47 98.75 98.78 98.34 98.75 98.61 98.34 98.47
98.307
3-1-3
94.60
97.45 98.23 97.10
96.99 97.49 98.03 97.30 96.95 97.72
97.186
3-1-4
95.57
97.87 97.55
96.41 97.40 97.49 96.76 96.74 96.95 98.57
97.131
3-1-5
87.04
85.61 84.76 84.30
84.40 85.54 84.96 84.50 84.71 85.67
85,095
3-1-6
90.59
90.95 90.36 90.23 89.12 90.69 90.55 89.64 89.58 91.47
90.318
3-1-7
86.71
82.22 87.17 84.11 84.76 82.68 84.17 82.16 84.89 82.16
84.103
3-1-8
85.80
82.87
89.51
82.35
85.09 81.96 83.85 83.65 87.63 79.68
84.239
3-1-9
60.50
64.50 69.50
63.66
63.33 63,50 62.83 65. 16 63.00 59.66
63.564
3-1-10
67.00
69.83 74.66 72.66 69.50 71.33 69.83 71.00 69.50 68.16
70.347
3-1-11
60.50
58.33 56. 16
49.83
62.16 62.50 63.33
59.66 57.00 64.83
59.43
3-1-12
56.83
46.83
59.16
41.83
61.16 62.16 61.50
46.33
57.50
57.00
55.03
3-1-13
19.66
22.5
20.5
20.66
21.16
20.83 21.5
18.83
23.83
21
21.047
3-1-14
32.33
34.16
33.5
30.66
32.83 32.66
30.33
27.5
33.16
28
31.513
3-1-15
20
18.5
17.83
16.66
20
19.16 18.16
17
18.16
17
18.247
3-1-16
22.66
20.66
20.16
18.16
20.33
19.5 19.83
20
21
19
20.13
gradient similarity on the client differs. (2) In the non-III, compared with the deeper layer features, the mean of the SNGS will be higher for the shallow layer features.
IID
(b) non-lID
Figure 4: Gradient similarity statistics under MNIST, EMNIST, CIFAR1O, CIFARI00 100 for each client category number of 4, 32, 4, 40
D ANALYSIS OF HYPERPARAMETERS ON PERSONALIZED EXPERIMENTS
We explored the impact of update frequency of the hyperparameters similarity matrix T, the number of classes A; the number of clients N, and the different local learning rounds E on the experimental results, which are shown in Table. 7.
E FINE-TUNE EXPERIMENTAL RESULTS
When the personalized FL algorithm is completed, the server delivers the final personalized model to each client. Like the meta-learning algorithm, after each client receives a proprietary personalized model, it can still make fine-tuning on itself datasets based on the model to achieve better results. The primary considerations for fine-tuning are the learning rate (3), the training batch size (B), and the number of steps the fine-tuning takes (step).

---

rear
HAIT
(a) IID
(b) non-IID
Figure 5: Gradient similarity statistics under MNIST, EMNIST, CIFARIO, CIFARI00 for each client of class number of 6, 40, 6, 60
Table 7: The influence of hyperparameters the experiment of personalized federated algorithm
MNIST
CIFARIO
EMNIST
CIFAR100
Methods
IID
non-IID
IID
non-IID
IID
non-III
IID
non-IID
T 5
98.86
98.32
76.06
68.18
90.67
88.58
38.20
31.36
T = 10
98.82
98.44
75.76
68.91
90.58
88.76
37.06
30.94
T = 20
98.65
98.30
75.15
68.48
90.13
88.62
35.4
29.38
98.96
98.32
82.05
73.3
91.23
90.14
42,2
31.2
k2
98.82
98.44
75.76
68.91
90.58
88.76
37.06
30.94
k3
98.75
98.30
68.03
66.55
87.88
87.21
31.65
30.41
N 6 N 10
98.70
98.01
75.25
66.88
89.74
88.65
33.41
28.08
98.87
98.24
75.64
67.99
90.33
88.53
35.99
30.10
98.82
98.44
75.76
68.91
90.58
88.76
37.06
30.94
E E =
98.82
98.44
75.76
68.91
90.58
88.76
37.06
30.94
99.05
98.79
77.16
72.23
90.71
88.98
37.84
30.63
E =
99.18
98.76
77.11
69.83
90.66
88.56
36.68
28.99
Table. 8 shows the model based on SPFI. training, according to consideration factors, the results of multiple of different experimental parameters First, we compared the accuracy of the model after fine-tuning with the previous model. It can be found that fine-tuning can indeed improve the performance of the model. However, compared with the improvement of SPFL in the federated average algorithm, the benefits of fine-tuning are not very good. It may because that the model after learning has original personalized attributes.

---

99
99-
98
98-
97
97-
96
96
95
95
C8
C10
MID
(a) MNIST result
80
75
75
70
70
65
50-
50-
45
45
Mean
C7
NII
(b) CIFARIO result
90
88
85
86-
84
84-
82
82
80
80
Mean
NID
(c) EMNIST result
40
20
20-
C4
C6
C7
C8
C9
C10
Mean
C1
C2
C4
C3
C6
C10
Meon
IID
NID
(d) CIFAR100 result
Figure The results of each client of the SPFL algorithm on the four datasets

---

Table 8: SPFL algorithm fine-tuning experimental results
B=128
B=128
B=128
B=128
B=128
B=64
datasets
distributed
B==-2
3=e-3
3=c-4
3=e-3
3==-3
step= 10
step=10
step=10
step=20
step=100
step=10
IID 98.57
98.85
98.82
98.86
98.89
98.87
MNIST
non-IID
98.20
98.44
98.45
98.48
98.47
98.49
73.75
75.91
75.8
76.08
76.01
75.98
CIFARIO
non-lII
65.66
69.14
69.21
69.14
69.41
69.31
IID
89.62
90.40
90.53
90.42
90.43
90.35
EMNIST
non-lID
87.56
88.82
88.80
88.76
88.85
88.77
IID
32.93
37.55
37.29
37.30
37.30
37.33
CIFAR100
non-lID
26.16
31.68
31.13
31.18
31.18
31.21


Reviews:
[{"review_id": "zaxp6MHIhfM", "reviewer": "Reviewer_HSpu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n1. Interesting idea of weighted averaging of client gradients to do personalized FL\n2. Potentially useful experimental results.\n\n*Weaknesses*\n1. Table 1: Factoring in standard error, it is not clear if there is a clear gain for their algorithm except in CIFAR-10 with non-IID setting. So the bolding does not make sense. However, I am not saying that only positive results should be published. Negative results are also good, as long as the experiments and methods are well thought out.\n\n2. Since this is a experimental paper, author could augment their contributions with more comprehensive and useful experiments.\n\n3. Can the authors please add error bars to the table? \n\n4. What is “stage strategy?” This is never defined nor discussed.\n\n5. Table 2: What is client and server? \n\n6. Typo: straggers\n\n7. Typo: Where is “Diag” used? There seems to be some mistake in eq (10). I don’t understand why (10) or (11) has this form. Please put more effort in explaining your strategy and the intuition behind it. Currently it is hard to follow.\n\n\n*Comments*\n1. Eqn (3): Could enhance the similarity metric S(i,j) by a weighting hyperparamter \\gamma. S(i,j) = \\gamma * <g_i, g_j>/(\\|g_i\\| \\|g_j\\|).\n\n2. Are the authors reporting standard errors or standard deviations in the table?\n\n3. Sec 4.1: What was the reasoning behind fixing the common parameters at these particular values?\n\n4. It is not fair that authors provide a non-anonymous github link.", "summary_of_the_paper": "This paper proposes gradient based weighting strategies for synchronous (SPFL) and asynch (PLGA) personalized Federated Learning (FL). Authors also provide experimental results comparing their methods to baselines on many benchmark datasets.\n", "summary_of_the_review": "Although the main idea seems promising, there needs to be a lot work done on presentation and evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "tg5Dv-Hd2xP", "reviewer": "Reviewer_1psQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The idea behind the paper is intuitive, and the numerical results also show its advantage compared to other methods.\n\nWeaknesses: \n\n- My first concern is regarding the computational cost of the method. Note that classic FedAvg, let's say with full participation, requires $\\mathcal{O}(Nd)$ computation at each iteration, where $N$ is the number of clients and $d$ is dimension. However, for this method, the computation would be $\\mathcal{O}(N^2 d)$, since all cross similarities are being computed. \n\n- My second concern is on the memory budget that this method requires on the server-side (especially for the synchronous part). Does the server need to store all local gradients to compute the update? And how feasible is this in practice? \n\n- My third concern is on the sensitivity of this method to gradient accuracy. In many cases, especially in real-world applications, the local gradients might be very noisy. For instance, they are computed over small batches or because noise is added intentionally to satisfy privacy constraints. It seems to me that this method might not work properly in such settings. I wonder whether the authors have considered this matter or not.\n", "summary_of_the_paper": "This paper proposed a personalized federated learning algorithm which takes into account the similarity of gradient of different users to update the model. More formally, the authors define $\\tilde{S}(i,j)$ as a measure of similarity between the gradients of two user $i$ and $j$, and then update the model of user $i$by weighting the gradient of user $j$ by $\\tilde{S}(i,j)$. The authors study their method in various numerical settings. ", "summary_of_the_review": "I find the idea of the paper intuitive and interesting. However, I have a number of concerns about the computation and memory cost of the method and the sensitivity of the proposed approach to the exactness of gradients. \nI am open to raising my score, given that my concerns are (at least partially) addressed. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XuYUVsHxPmK", "reviewer": "Reviewer_3fbQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed method requires modeling the pair-wise similarity between each client. This requires maintaining a potential large matrix, maintains the identity of each client, and cannot be applied to new clients. \n2. The algorithms are all tested on synthetic (label-heterogeneous) Federated dataset. Could you test on natural heterogeneous dataset?\n\nThe paper is overall hard to read due to typos, grammar errors, and influent sentences. Here are some examples I found on page 4 and 5:\n\n- \"Then the SGD updates of the corresponding gradients of $w_i^t$ and $w_j^t$ are:\"\n- \"where ∥ · ∥ is the vector L 2normal. And the aggregation strategy on the server for C iis as follows:\"\n- This section introduces an asynchronous updating strategy that addresses the system heterogeneity problem mentioned in Section [missing].\n\nThese aspects make the current version of this paper look incomplete.", "summary_of_the_paper": "This paper proposed two personalized federated learning algorithms: SPFL, PLGA.", "summary_of_the_review": "The paper is overall not well-written and I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JgZRsfafkkf", "reviewer": "Reviewer_7GZS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is no question that personalized federated learning is an important and practical problem. The proposed approaches and initial results presented seem promising, but they are very far from solving the problem.\n\nFor a paper proposing new methods for Federated Learning, one generally expects either that theoretical convergence analysis is provided, or a very thorough experimental evaluation is provided. This paper doesn't provide any theoretical guarantees (and the motivation for the proposed approaches is not always clear). There are a lot of questions about the experiments, which reduce my confidence in the overall conclusions. Moreover, the proposed schemes appear to be fundamentally incompatible with privacy, which is a critical factor for Federated Learning methods. Consequently, I don't believe that this paper is acceptable. Substantial additional work would need to be done (experiments and/or theory) to improve it to an acceptable state. Detailed comments follow.\n\nThe paper is missing references to other relevant and more recent work on asynchronous FL:\n* Chen et al., \"Asynchronous Online Federated Learning for Edge Devices with Non-IID Data,\"  arXiv:1911.02134\n* van Dijk et al., \"Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise,\" arXiv:2007.09208\n* Chai et al., \"A Communication-Efficient Federated Learning Method with Asynchronous Tiers Under Non-IID Data,\" arXiv:2010.05958\n* Nguyen et al., \"Federated Learning with Buffered Asynchronous Aggregation,\" arXiv:2106.06639\n\nI'm confused by the description between equations (3) and (4). Does the server update the client parameters $w_i^{t+1}$ and $w_j^{t+1}$?\n\nThe proposed scheme either requires clients to directly share model updates $g_i^t$ with each other, or they need to share their individual model updates directly with the server. This violates the general principle in Federated Learning that the server should only receive/process updates aggregated from many clients, in order to better preserve the privacy of client models. Hence, the proposed scheme seems fundamentally incompatible with current practices in Federated Learning systems. Is there a way to preserve privacy while also performing these types of updates?\n\nAdditional explanation should be included to justify how (10) alleviates the update gap between stragglers and the server. In addition, it would be useful to illustrate the impact of using (10) vs. another approach experimentally. In order to implement the similarity calculation (8) at the server, the server must maintain a long history of the past model versions (up to some maximum staleness). Is this additional memory overhead practical? And again, it does not appear to be possible to perform such similarity-weighted updates in a privacy-preserving manner.\n\nThe methods, such as PLGA, are described without very much motivation and no formal justification (e.g., no theoretical convergence guarantees, even under strong assumptions).\n\nThe experiments appear to fix some of the hyperparameters to be the same for all of the methods compared (e.g., local and server learning rates). Normally each method will benefit from having these tuned separately per-method. Because of this, I'm not convinced that the comparison is a fair one.\n\nHow many clients are there in the SPFL evaluation?", "summary_of_the_paper": "This paper proposes two methods for personalized federated learning, one synchronous and one asynchronous. The general approach taken in both cases is to adapt the weights when averaging information from different clients, so that clients with more similar gradients are given more weight in the update for each client. The two approaches are called SPFL (synchronous) and PLGA (asynchronous). The approaches are illustrated on small image classification tasks.", "summary_of_the_review": "For a paper proposing new methods for Federated Learning, one generally expects either that theoretical convergence analysis is provided, or a very thorough experimental evaluation is provided. This paper doesn't provide any theoretical guarantees (and the motivation for the proposed approaches is not always clear). There are a lot of questions about the experiments, which reduce my confidence in the overall conclusions. Moreover, the proposed schemes appear to be fundamentally incompatible with privacy, which is a critical factor for Federated Learning methods. Consequently, I don't believe that this paper is acceptable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a personalized federated learning algorithm which takes into account the similarity of gradient of different users to update the model. Although the ideas presented are intuitive, the algorithms have fundamental limitations, for example, they may cause large overhead of memory, communication and computation, and are unsuitable for privacy-preserving machine learning. In addition, there are no rigorous analysis and the experiments are not convincing. This is a clear rejection.
Decision: Reject