FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning
Authors: Longfei Zheng
Yingting Liu
Xiaolong Xu
Chaochao Chen
Weipeng Sun
Xiaolong Hu
Lei Wang
Li Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Non-IID
Communication Efficiency

Abstract: Federated learning (FL) is a popular distributed machine learning framework in which clients aggregate models' parameters instead of sharing their individual data. In FL, clients communicate with the server under limited network bandwidth frequently, which arises the communication challenge. To resolve this challenge, multiple compression methods have been proposed to reduce the transmitted parameters. However, these techniques show that the federated performance degrades significantly with Non-IID (non-identically independently distributed) datasets. To address this issue, we propose an effective method, called FedPSE, which solves the efficiency challenge of FL with heterogeneous data. FedPSE compresses the local updates on clients using Top-K sparsification and aggregates these updates on the server by element-wise average.  Then clients download the personalized sparse updates from the server to update their individual local models.  We then theoretically analyze the convergence of FedPSE under the non-convex setting.  Moreover, extensive experiments on four benchmark tasks demonstrate that our FedPSE outperforms the state-of-the-art methods on Non-IID datasets in terms of both efficiency and accuracy.

Under review as a conference paper at ICLR 2023
FEDPSE: PERSONALIZED SPARSIFICATION WITH WLEMENT-WISE AGGREGATION FOR FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) popular distributed machine learning framework in which clients aggregate models' parameters instead of sharing their individual data. In FL, clients communicate with the server under limited network band- width frequently, which arises the communication challenge. To resolve this chal- lenge, multiple compression methods have been proposed to reduce the transmit- ted parameters. However, these techniques show that the federated performance degrades significantly with Non-IID (non-identically independently distributed) datasets. To address this issue, propose effective method, called FedPSE, which solves the efficiency challenge of FL with heterogeneous data. FedPSE compresses the local updates clients using Top-K sparsification and aggre- gates these updates on the server by element-wise average. Then clients down- load the personalized sparse updates from the server update their individual local models. We then theoretically analyze the convergence of FedPSE under the non-convex setting. Moreover, extensive experiments on four benchmark tasks demonstrate that our FedPSE outperforms the state-of-the-art methods on Non- IID datasets in terms of both efficiency and accuracy.
INTRODUCTION
Federated learning (FL) prevailing distributed framework that can prevent sensitive data of clients from being disclosed (Kairouz et al. 2021 McMahan et al. 2017b). The naive FL in- cludes three steps: uploading clients' models to the server after local training, global aggre- gation, and downloading the aggregated model from the server. In practice, weight updates AW W new Word can be communicated instead of model weights W (Asad et al.) 2021] Li al.) 2021a). Recently, FL is increasingly applied in multiple tasks, such computer vision, ommender systems, and medical diagnosis (Bibikar et al. 2021 Kairouz et al. 2021 Qayyum et al.) 2020
Xu al. 2021).
1.1
EXISTING PROBLEM
Despite the aforementioned advantage, the communication cost of FL is overburdened by the fact that the server and clients exchange massive parameters frequently (Asad et al.) 2021 Kairouz et al.) 2021 Furthermore, there usually is a limited upstream/ddonsstrem bandwidth between the server and clients, such as wireless connection in the cross-device (ToC) FL and dedicated network in the cross-silo (ToB) setting. which further decreases the communication efficiency (Li et al.) [2011a] Sattler et al. 2019). FL is much more time-consuming than traditional centralized machine learning, especially when the model parameters are massive under the cross-silo FL scenarios (Qayyum al., 2020 Shii et al.] 2020). Therefore, it is necessary to optimize the bidirectional communication cost to minimize the training time of FL (Bernstein et al. 2018 Philippenko & Dieuleveut, |021 Sattler et al.) 2019: Wen et al. 2017). In order resolve the aforementioned challenge, various methods have been proposed, such as matrix decomposition (Li et al. 2021c| McMahan et al, 2017b), quantization (Li et al.) 2021a Sattler al., 2019), and sparsification (Gao al.) [021] Mostafa & Wang 2019: et al. 2020; Yang et al. (0216). Although these novel algorithms can reduce the quantity of communicated information significantly, most of them only work well

---

Under review as a conference paper at ICLR 2023
Sarver
Element-wete Aggregation
Percreladd
Upioad
Downiaac
Dus
Server
Clint
Clem
Clten No
(a) Upstream personalized sparsifica- tion
(b) Element- wiie aggregation
(c) Downstream personalized sparsi- fication
Figure The proposed framework FedPSE.
under the ideal condition with IID (identically and independently distributed) datasets (Li et al, 2021c; (Satter et al.]] 2019) Wen et al. 2017). In fact, the isolated datasets in clients are usually heterogeneous, due to the reason that each dataset belongs to a particular client with specific geographic location and time window of data collection Kairouz al. (2021); Kulkarni al. (2020); Xu & Huang (2022); Yang et al. (2021a). Hence, the current compression techniques, ignoring the personalization of clients, face significant performance degradation Non-IID datasets Liu al. 2022); ; Sattler et al. (2019): Wu et al. (2020).
1.2 SOLUTION
To bridge this gap, we propose a Personalized Sparsification with Element-wise aggregation for the cross-silo federated learning (FedPSH) paradigm, as shown in Figure [ For the first step of FedPSE, under the concern of efficiency and personalization, clients train their models with local datasets and upload the sparse updates to the server, as shown in Figure [(a) The kept indices of these compressed updates are probably different from each other due to the heterogeneity of clients' datasets. Secondly, we leverage element- wise averaging to aggregate the collected sparse updates on the server, which can relieve the bias of the traditional aggregation method, as shown in Figure [(b)) Lastly, the server sparsifies the downstream parameters for each client a personalized manner, as shown in Figure I(c) Especially, the downstream updates, transferred from the server to each client, also possess individual elements to keep the overall compression ratio. Please see Scction 4 for more details. To this end, FedPSE compresses both upstream and downstream communication overhead with personalization concerns.
1.3 CONTRIBUTION
We summarize our main contributions as follows:
We propose novel personalized sparsification with an element- weiee aggregation framework for FL. which resolves the bidirectional communication challenge on Non-IID datasets.
We propose element-wise aggregation method, which can promote the performance of FL with sparse aggregated matrices.
We propose a downstream selection mechanism to personalize the clients' models, which adapts various distributions and significantly increases the performance in the Non-IID setting.
We provide convergence analysis of our method well extensive experiments four bench- mark datasets, and the results demonstrate that our proposed FedPSE outperforms the existing state-of-the-ee FL framework on Non-IID datasets in terms of both efficiency and accuracy.
RELATED WORK
In this section, we briefly review optimization methods that focus on the core challenges in FL.
COMMUNICATION EFFICIENCY
Although FedAVG (McMahan et al. 2017a), the naive federated algorithm, decrease the com- munication cost by allowing multiple local steps, the massive transmitted parameters in one com- munication step still critical bottleneck. In general, there three kinds of compression
2

---

Under review as a conference paper at ICLR 2023
approaches to tackle this problem, i.e. matrix decomposition (Huang et al. 2022, Li et al.
2021c>
quantization (Bernstein et al., 2018 Li et al. 2021a). and sparsification (ao et al. 2021
Sattler
et al. 2019 Wu et al. (020).
Firstly, the decomposition method, decomposing the transmitted matrix, is unpractical with a smaller compression ratio and more computation complexity. Secondly, the quantization-basel methods, limiting the number bits, have the upper bound of the compression ratio and slow down the convergence speed in terms training iterations (Chai al. 2020). Thirdly, the sparsification--asse methods mask some elements of the transmitted matrices, among which the Top-K sparsification with residual error is most widely used due its promising performance and convergence guarantee Dan Alistarh et al. 2018) Gao et al.] [202 Wu et al. [2020; Yang et al.) (2221). Therefore we leverage the error-compensated Top-K compressor to reduce the communication overhead in this article.
2.2 PERSONALIZATION
Since FL relies the stochastic gradient descent (SGD) algorithm to train neural networks, the performance of FL is easily biased by the Non-IID datasets, e.g. feature skew and label skew, etc (Kairouz et al. 2021 Zhu et al. 2021). Different strategies have been applied to FL with heteroge-
neous concerns (e.g. data shuffling. multi-task learning, and local model optimization) (Achituve)
al. 2021, Chen & Chao 2021; Huang al., [022] Li et al. 2b21b Ma al. 2022 Oh al.)
2021
Dinh et al. [2020] Xu & Huang [2022] [2a2g] et al. (021). Although these state-of-the-an
methods perform well under the Non-IID condition, the communication efficiency is often ignored.
2.3
COMPRESSION WITH PERSONALIZATION
In a word, few algorithms take both of the above two challenges into consideration. To our best knowledge, FedSTC (Sattler et al., 2019) and FedSCR (Wu et al. 2020) declare that they resolve the communication overburden with Non-IID data in one shot. Specifically, FedSTC compresses the clients' updates by combining Top-K sparsification and ternary quantization. Similarly, Fed- SCR compresses the transferred information by removing the redundant updates which are less than the adaptive threshold. However, the clients in FedSTC and FedSCR share the same model, which decreases the federated performance significantly in the Non-IID setting. In this paper, we pro- pose a personalized compression paradigm for FL., which has competitive efficiency and promoted performance for heterogeneous data.
PRELIMINARY
In this section, we present some preliminary techniques of proposal, including FedAVG and Top-K sparsification.
3.1 FEDAVG
FedAVG (MMMaan et al., 201 basic FL algorithm, builds distributed machine learning models via model aggregation rather than data aggregation. We suppose that there are N clients with their datasets (Di, D2.... Dx) in FL. Client i, E (1..... N). trains the local model W using samples individually and uploads the weight updates AWT to the server for aggregation during the r-th federated round. Then the server leverage the averaging function to generate the global matrix:
AW" M EN R4 AWT. In the end, the global matrix AWT is sent back each client to update the local model WT +- WT AAW..
3.2 TOP-K K SPARSIFICATION
Top-K Sparsification is the most widely used compression method in FL, which remains a stable
performance with high proportion of ignored parameters (Gao al. 2021 Wu et al.) 2020: Sattler et al, 2019, Yang et al. 20216). The Top-K compressor selects K elements of the input matrix with the largest absolute values.
3

---

Under review as a conference paper at ICLR 2023
4 METHOD
In this section, we first give an overview of the proposed FedPSE paradigm. We then present its three main components, ie.. Upstream Personalized Sparsification (UPS), Element- Wise Aggrega- tion (EWA), and Downstream Personalized Sparsification (DPS), Finally, we summarize the whole algorithm.
4.1 OVERVIEW
Our purpose of FedPSE is to design an FL paradigm that reduces the bidirectional communication cost while improving the performance on Non-IID datasets. We optimize the steps of FL. according to its training process, as shown in Figure [ Firstly, order to reduce the upstream communica- tion burden, we should compress the information from clients to the server. Secondly, as the global updates may be biased due to the compression, it is worth developing new aggregation method that is suitable for sparse matrices. Thirdly, since clients with Non-IID datasets should have individ- ual local models, the server needs to personalize the downstream information to each client while keeping the compression sparsity. Then we propose the motivation of our methods in the following paragraphs.
The first step, upstream personalized sparsifi- cation, compresses clients' updates after local training, which is inspired by the current work (attter et al. 2019, Wu et al.) (020). Ac- cording to the prior research Wu et al. (2020), the matrices of weights updates are structure- sparse, in which most elements are redundant in the Non-III) setting. As result, we leverage the Top-K sparsification operator with residual error compress the weight updates, whose convergence rate has been theoretically proved (Gao al.) 202T, Haddadpour et al.) [202]). Apparently, the reserved indices of compressed updates have its personalized distributions on heterogeneous datasets. Then each client trans-
Algorithm 1: UPS
Input: downstream personalized sparse updates AWT;, sparsity ratio p upstream sparse updates A AW,, samples number n
Output:
W = W. AW 8,
Reset n. = for each local training step 1, ,,..., do
Sample a batch: B ~ Dj
g. Vi(W, 1r
W Opt(W W... - (x)
mf ni + Biar1|
mits its individual updates to the server for the following aggregation.
end
A AW Wri.. W + +cp-
The second step is proposing an appropriate ag- gregation method for the sparse updates. We demonstrate the bias of FedAVG with two examples under IID/Non-IID settings in Ap- pendix A which is caused by the sparsification. Then the effectiveness of the EWA method is proved, which is more suitable for sparse aggregation. Please see more details in Appendix
10 AWT = Top-K(AWT,
of AWT AWT
12 return AW; T, m
Thirdly, the server should sparsify the downstream updates for each client under the concern of personalization. In order to quantitatively measure the divergence between the global distribution and the local distribution, we compress the global updates, possessing i elements, and compute its correlation distances with clients' upstream updates. Apparently, the correlation distances are likely different from each other in the Non-III setting. Finally, the personalized downstream updates are selected by the corrllation distances. Further details can be found in Section 4.4
4.2 UPSTREAM PERSONALIZED SPARSFICATION (UPS)
We suppose that there are N clients with their own training datasets (Di, D2, ..., Dx and a server participated in FL. At the beginning of FL, client (Vi E P,P 21 2, 2.., N)) initializes its weights
W; with the same parameters Wo (Zhao et al.) 2018). Algorithm [ presents the process of local training on client i during r-th federated round, where the inputs, i.c. weight updates AWT-I s.i and sparsity ratio p. are transmitted from the server. The first step of local training updating the client model with the received AW (line 1). Secondly, client trains its individual model with n!
4

---

Under review as a conference paper at ICLR 2023
samples (lines 3-8), in which fi indicates the loss function and Opt (e.g., SGD or Adam) indicates the model's optimizer with the learning rate a.
After that, the client compresses the updates via the crror-compensated Top-K compressor (lines 9-11). Finally, the sparse updates AW, and the number of local training samples nł are uploaded to the server for aggregation.
4.3 ELEMENT-WISE AGGREGATION (EWA)
From the perspective of the selection mecha- nism in federated learning, the kept non-zero value in AW; indicates that the correspond- ing client i is selected by the server at this ele- ment location. Apparently, each element of the aggregated matrix has its individual participat- ing samples. We introduce our Element- Wise Aggregation (EWA) method in Algorithm 2 Firstly, the server receives the sparse updates AWT from client i during r-th federated round, Vi P. Then the server computes the indices of nonzero values M egg.i for A AW,, which is used calculate the matrix .V' of training samples for each location (lines 1-5). Finally, the server aggregates the global updates AWT, in which @ indicates the element-wise division operator (line 6).
Algorithm 2: EWA
Input: sparse updates A AW", samples number ni, E P
Output: aggregated updates: AWT for i 1,...: .N do
Computes the index matrices of client updates:
M Magg. Sigm (A
end
Sum matrix of training samples:
N = Σ nf: M Mega.
AW? Eming
nf -.W:) ON
return AW?
4.4 DOWNSTREAM PERSONALIZED SPARSIFICATION (DPS)
This process can be separated into three sub-steps: measurement of the heterogeneity between the global distribution and local distribution, selection of the downstream indices, and computation of the downstream personalized updates.
First of all, during the r-th federated round, the server quantifies the heterogeneity of data distribu- tions via the distance between AWT and AWT. Vi E P (lines 1-4). The server flattens the updates followed by normalization (lines 2-3). Then we compute the distance via the cosine function, which can be replaced by the other correlation functions (line 4). Secondly, we get the index matrix My of the global sparse updates AWT via the sign function, In similarity, the index matrix M. of client i is calculated (line 6). The intersection matrix M of M and M. can be regarded as the common information owned by both the server and client i, which will be wholly kept during downstream transmission (line 7). As M.inn has kj.in elements, we choose the rest k - kū elements from the compensation matrices M., and M In order to enhance the generalization of training, we leverage the random mechanism to choose d',,, (k - ky,,n) elements from Mi.., while selecting (1.0 - d",;) (Ic ks.sn) from My.c (lines 8-13). Finally, we combine the personalized index matrix My. of client (line 14) and calculate the downstream sparse updates AWT A from the server to client é (lines 15-16).
Apparently, each client uploads its unique sparse updates AWT, which means that the correlation distance d=,, and the downstream sparse updates AWT, are different from each other.
4.5 PUTTING ALL TOGETHER
To sum up, we conclude the FedPSE framework in the Algorithm 4 which executes the federated process, named PSE, for R times. Before the training process, we initialize the model weights of clients as Wo. the received updates AWO s,i as zero and the sparsity ratio p for communication compression concern. First of all. we get the personalized sparse updates AWT of each client i using UPS method (Algorithm I] with the last AW and the sparsity ratio p (line 5). Then we leverage the EWA operator (Algorithm compute the global updates (AWT) during r-th federated round
5

---

Under review as a conference paper at ICLR 2023
Algorithm 3: DPS
Input: Upstream sparse updates AW,, global updates AW", global sparse updates AW?
Output: Downstream sparse updates Á AW",, s, to the client
# Measure the heterogeneity of distributions via AW; and AWT
Normalize the global sparse updates: AW
Flatten(
Normalize the client i sparse updates: AW?
(atlee(((([]])))) AW)) Compute the correlation distance: d=,, = 0.5 0.5 Cosine AW? AW;) Select the reserved indices of downstream updates
Get the index matrices: M Sign (Awwa) and MT = Sign (Aww:))
The intersection index matrices: M. MM @ M,
The number of non-zero value Mº: ||
The number of non-zero value Mi kg.in M
10 The compensations index matrices: Mr. = M Mf and My = M - M
I Combine the indices:
12 My Random (M di.;;" (k- ki,sn)
M = Random(Mg, (1 d=.)) (k- ki,in))
MY M + M + My.
is #Compute the downstream personalized updates: AW,, = MG, @ AWT 16 return AWT,
(line 8). Furthermore, the server sparsifies the global updates AWT and generates the AWT with the compression rate p (line 9). Finally, we get the personalized updates of each client via DPS method (Algorithm 3 with the inputs sparse upstream updates AWT, global updates AW? and sparse global updates AW (line 11).
4.6 THEORETICAL ANALYSIS
Algorithm FedPSE framework
In this subsection, we analyze the convergence results of the FedPSE framework theoretically. We suppose that the loss function (Ji R
Initialization: W -W.
ΔW 0 (vi E P). sparsity ratio p for each federated round 1, ..., ... do
R) of client (Vi P)) differentiable, where is the dimension of parameters. We consider the general setting deep learning where fü is non-convex function. Our convergence results are proved under the following assumptions:
# At clients: for each client
do
AWT,n n, < AUS ,p)
end
Assumption 1. Lipschitz Smoothness:
# At Server: AWT < EWA(AW,, m(), Vi EP AWT + Top-K(AW,,
The loss function ft (Vi P) is Lipschitz smooth (L-smooth), ie.., (VT) ((WW))l S TIIW"" Will, VW T W E R
10
for E P do
AWT. + DPS(AWT AWT.WW"
n
Assumption 2. Bounded Gradient:
end
The second moment of stochastic gradient G; calculated by single sample within client
end
i bounded, i.e., Elll EN G.(W.) < return R.T VIEP o², VW, E R
Elll B[I R
We aims to prove that min re(1.....
0, which is a normal convergence guar-
antee in the non-convex problem Liu & Wright 2015). Similar to Dan Alistarh et al. (0188)

---

Under review as a conference paper at ICLR 2023
WT to denote the auxiliary random variable during r-th federated round client and
W++1= W a' G(W"), where G(WT) = Σ G,(WT) and w W E P).
Lemma 1. For any federated round 2 1: [T WT 1+2e (1=((((+)))((00"
where p 1 k: 0 < k - and p 0.
We propose the bound of the difference between W and W during r-th federated round in Lemma Consequently, the convergence of FedPSE is guaranteed as shown below.
Theorem 1. Choose the learning rate schedule a, s.1. Eg=1(p(1 1))o then our proposed FedPSE satisfies:
S <C(r > 0),
R
( Σr=1 o
20' N
2L(1+20)C
D (a))²
Σ F(EEIIIT
P
E==1
2r=1 a"
where constant C'> 0 and W denotes the optimal solution auxiliary variable.
Theorem implies that client model W/¹ converges federated round R is large enough when a' Σ (a"² satisfies the following conditions: lim E a" = oo, lim 2r= a" = 0. Finally, we derive the convergence speed of framework, please see more details and proofs in Appendix B
EXPERIMENT
In this section, empirically compare the performance of our proposed FedPSE framework with other federated learning paradigms for personalized compression. We aim to answer the following questions.
Q1: whether FedPSE outperforms other optimized algorithms on the Non-IID data?
Q2: whether the convergence speed of FedPSE is acceptable the Non-III) setting?
Q3: whether the correlation distance can be adopted to quantitatively measure the divergence between the local distribution and the glohal distribution?
Q4: whether the EWA method can promote the performance of FedPSE?
Q5: how does the sparsity ratio influence the performance of FedPSE?
5.1 EXPERIMENTAL SETTINGS
Datasets. To the effectiveness of proposed model, we choose four widely used benchmark datasets: MNIST (Deng) 2012), Meshion-MNiS"" (FMNIST) (Xiao et al. 2017), IMDB (Maas et al.) 2011] and Cifar-10 10 (Krizhevsky al.) 2009). We load these datasets using the Keras package in Tensorflow2.8, keeping their original train/test samples (Abadi et al. 2016).
Non-IID Setting, As most empirical work on synthetic Non-lID datasets partitions a "flat" existing dataset based on the labels (Kairouz et al. (02I). we also use the label distribution skew our Non-IID setting. Then we set a variable Non-IID ratio X, ranging from 0.0 to 1.0, to simulate the Non-IIInnes of clients' datasets, which is the same with Beutel et al. (202)) this way, the distribution of clients' datasets is becoming more heterogeneous with a larger X. Please see more details in Appendix C.I
Metries. Following the existing work (Sattler et al., 2019: Wu et al. 2020 we use accuracy on the test dataset as the evaluation metric. To compare the performance of different strategies in the decentralized scenario, we optimize their
and choose the average metrics in all clients as the optimization target. Please see more details in Appendix C.2 and C.3
5.2 PERFORMANCE COMPARISON OF ALGORITHMS WITH DIFFERENT NON- RATIOS
To answer the proposed question Q1, we firstly set different Non- ratios X, i.. 0.0, 0.5 and 1.0, to partition the original datasets for clients. As shown in Table and Appendix we set the different

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparison of algorithms on datasets with different Non-IID ratios À.
Dataset Clients
[FedAVG
FedSTC
FedSCR
FedPSE
Dataset FedAVG
FedSTC
FedSCR
FedPSE
1.0 0.9785
0.9927
0.9904
0.9967
0,8406
0,8522
0.8961
0.9999
0.5
0,9901
0.9913
0,9851
0.9910
0,8804
0,9010
0.8811
0.9013
0.0
0.9905
0.9915
0.9867
0.9910
0.8810
0.8815
0.8815
0.8869
MNIST
1.0
0.9694
0.9907
0.9741
0.9939
IMDB
0.8494
0.9099
0.9375
0.9989
0.5
0.9899
0,9902
0,9816
0.9909
0.8864
0.8941
0.8862
0.8977
0.0
0.9904
0.9910
0.9851
0.9908
0.8827
0.8862
0.8841
0.8792
1.0|
0.9530
0.9876
0.9791
0.9940
0,9080
0,9591
0,9633
0.9991
0.5
0.9725
0,9899
0.9801
0.9901
0.8900
0.8922
0.8901
0.8951
0.0
0.9904
0.9905
0.9819
0.9865
0.8908
0.8846
0.8770
0.8602
1.0
0.8781
0.8844
0,9031
0.9466
0.7508
0.7611
0.7723
0.8576
0.5
0.8804
0.8926
0.8624
0.8953
0.7712
0.7732
0.7731
0.7743
0.0
0.9003
0.8906
0.8523
0.8911
0.7736
0.7711
0.7724
0.7721
FMNIST
1.01
0.8068
0.8388
0.8403
0.9243
Cifar10
0.7654
0.7786
0.7841
0.8602
5
0.5
0.8835
0.8952
0.8512
0.8967
0.7760
0.7801
0.7822
0.7843
0.0
0.8995
0.8839
0.8641
0.8760
0.7805
0.7805
0.7804
0.7780
1.0
0.7909
0.8040
0.8305
0.9334
0.7630
0.7721
0.7734
0.8741
10
0.5
0,8697
0,8850
0.8655
0.8910
0.7800
0.7811
0.7832
0.7856
0.0
0.8993
0.8960
0.8644
0.8725
0.7815
0.7822
0.7788
0.7770
numbers of clients, ranging from 2 to 100, for the cross-silo federated learning, which is similar to previous studies (Gao et al. 2021 Wu et al., 2020). Furthermore, we compare the accuracy of our proposed FedPSE and the three aforementioned algorithms. i.e. FedAVG McMahan al. (2)17a), FedSTC Sattler et al. (2019), and FedSCR (Wu et al. 2020). Among them, FedAVG is the baseline method without communication compression. In addition, FedSTC and FedSCR are state-of-the-art methods to solve the communication efficiency problem in the Non-IlI) setting. Especially, we set the same sparsity parameter p = 0.9 for FedSTC, FedSCR, and FedPSE, which means that only transmitted between clients and the server compared with FedAVG. Then
10% parameters summarize the results in Table T and Appendix D.I which compare the performance of FedAVG, FedSTC, FedSCR, and FedPSE paradigms these benchmark datasets.
From the results of the experiments, we conclude that our proposed FedPSE almost achieves the best performance of all algorithms in the Non-IID setting (A = 1.0 or 0.5),
0.8
Tss Acuuray
0..
which is also robust the number of clients. For exam-
plc, the average metric of FedPSE outperforms 6.85% for
0.5
FedAVG
0.4
FedSTC and 5.57% for FedSCR on FMNIST dataset as shown in Table 2 of Appendix D.I Besides, FedPSE is
FedSTC
0.2
FedSCR
FedPSE
also robust the Non-III ratios, which can get very sim- ilar performance compared with FedAVG and FedSTC, while FedSCR has weakness when the data distribu- tion is more symmetrical (A= 0.0). Furthermore, our method also outperforms other models with partial clients participation for aggregation as shown in Appendix D.2
50
100
200
250
Federated Round
Figure 2: Test accuracy regarding rounds.
5.3 CONVERGENCE SPEED OF FEDPSE IN THE NON-IID SETTING
To answer the proposed question Q2, we plot the metric
over the federated rounds the Non-IID (A- 1.0) FM-
NIST dataset as shown in Figure 2 in which the com-
EMN/ST Distumce
0.38
FMNIST IMDB
0.30
pression ratio p is 0.9 with participated clients. We
observe that the performance of FedAVG is unstable on Non-IIID datasets, which also proposed in the previ- ous result (Zhao et al., 2018), while other methods suffer the least from Non-IID data. is essential that our pro- posed FedPSE, compared with FedSTC and FedSCR, im- proves the performance with a faster convergence speed. Furthermore, FedSCR performs best in the Non-IID set- ting, which is consistent with the former result (Wu et al.) 2020
0.36
0.28
0.34
0.26
0.32
0.24
0.30
0.22
Non-IID Ratio
Figure Correlation distance on 5 clients.
8

---

Under review as a conference paper at ICLR 2023
5.4 CORRELATION DISTANCE FEDPSE WITH DIFFERENT NON-IID RATIOS
To answer the proposed question Q3, we plot the correlation distance of participated clients with different Non-IID ratios (i.e. À 0.0, 0.5, 1.0) as shown in Figure
Figure3 that the average correlation distance increases over the Non-IID ratios, which demon- strates that the correlation distance able to represent the heterogeneity between local distribution and global distribution. Furthermore, we compute the variance of correlation distance under diverse Non-IID settings as shown in Appendix D.3] Apparently, the variance is becoming larger with the growth of heterogeneity, which means that the correlation distance also has a relationship with the distribution divergence among each client.
0.945
F 0.50
FedPSE wihout FWA
6,50
FedPSE whbout EWA
FeiPSE witlout EWA
8.835
FedPSE winh KWA
FodPSR with EWA
Fad with EWA
0.5
6.7
Sparsity radls
Spansily ratie
Spaniity ratls
Comparison clients
Comparison 5 clients
Comparison 10 clients
Figure 4: Accuracy comparison of EWA in FedPSE over compression sparsity ratios on Non-IID datasets.
5.5 PERFORMANCE COMPARISON OF FEDPSE WITH DIFFERENT AGGREGATION METHODS
To answer the proposed question Q4, we vary sparsity ratio from 0.1 to 0.9 on FMNIST dataset and report the average test accuracy on Non-IID datasets (A 1.0)
FedPSE without EWA
Test ccuracy
FedPSE with EWA
in Figure 4 in which FedPSE without EWA indicates FedPSE using naive aggregation method of FedAVG. Figure @ that FedPSE with EWA consistently out- performs FedPSE without EWA under different compres- sion rates, which demonstrates the effecctivnness of our proposed EWA method. Besides, the average promotion of EWA is 0.002
0.90
0.886
Non-IID Ratio
2 clients, while the promotion is 0.0071 on 5 clients and 0.0143 on 10 clients. We can indicate that the promotion of EWA increases with the number of clients.
Figure 5: Accuracy with Non-IID ratios.
Furthermore, we construct experiments on the FMNIST datasets over Non-IID ratios À (i.e. from 0.0 to 1.0) with a fixed p 0.9 as shown in Figure The test accuracy of FedPSE with EWA exceeds the algorithm without EWA through different distributions. The performance of FedPSE increases rapidly with the rising A, which means that our proposed paradigm performs better under the extreme Non-IID condition. In conclusion, the EWA method is useful for the sparse aggregation process in the server.
5.6 PERFORMANCE COMPARISON OF FEDPSE WITH DIFFERENT SPARSITY RATIOS
Figure 4 can also answer the proposed question Q5. We can find that the accuracy of FedPSE over sparsity ratios changes individually on the different number of clients. For instance, the accuracy keeps steady with most sparsity ratios on two clients, while the metric increases rapidly with the rising p from 0.1 to 0.7 on more clients. We can optimize the sparsity hyper- aapee-rrreeter p of FedPSE to balance the performance and efficiency of FedPSE.
CONCLUSION
We propose a personalized sparsification with element-wise aggregation for federated learning to solve the Non-IID isolated scenario. We first sparsify the upstream updates of clients via the Top-K operator. Then we propose the EWA aggregation method to promote the federated performance for sparse matrices. Finally, leverage the DPS method to keep the personalization and sparsification for the downstream information. Experiments on real-world datasets demonstrate that our model significantly outperforms the current methods the isolated Non-IID data.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Tensorflow: Large-scale machine learning heterogeneous distributed systems. arXiv preprint arXiv: 163.v1660300447, 2016.
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34, 2021.
Muhammad Asad, Ahmed Moustafa, Takayuki Ito, and Muhammad Aslam. Evaluating the commu- nication efficiency in federated learning algorithms. In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 552-557. IEEE, 2021.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. International Conference on Machine Learning, pp. 560-569. PMLR, 2018.
Daniel Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390, 2020.
Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. arXiv preprint arXiv:21 12t.22222..9824.. 2021.
Zheng Chai. Yujing Chen, Liang Zhao, Yue Cheng, and Huzefa Rangwala. Fedat: A communication-efficeen federated learning method with asynchronous tiers under non-iid data. ArXivorg, 2020.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning. arXiv preprint arXiv:2107.00778. 2021.
Torsten Hoefler Dan Alistarh, Sarit Khirirat Mikael Johansson, and Cedric Renggli Nikola Kon- stantinov. The convergence of sparsified gradient methods. NeurlPS, 2018.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 19(6):1411142 2012.
Hongchang Gao, An Xu, and Heng Huang. On the convergence of communication-efficii local sgd for federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, pp. 18-19, 2021.
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Feder- ated learning with compression: Unified analysis and sharp guarantees. In International Confer- ence on Artificial Intelligence and Statistics, pp. 2350-2358. PMLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jiwei Huang, Zeyu Tong, and Zihan Feng. Geographical poi recommendation for internet of things: federated learning approach using matrix factorization. International Journal of Communication Systems, pp. e5161, 2022.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7865-7873, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-200, 2021.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Viraj Kulkamni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794-797. IEEE, 2020.
Chengxi Li, Gang Li, and Pramod K Varshney. Communication-effiien federated learning based on compressed sensing. IEEE Internet of Things Journal, 2021a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021b.
Xiaoli Li, Yuzheng Li, Shixuan Li, Yuren Zhou, Chuan Chen, and Zibin Zheng. A unified federated dnns framework for heterogeneous mobile devices, IEEE Internet of Things Journal, 93); 1748, 2021c
Liu and Stephen J Wright. Asynchronous stochastic coordinate descent: Parallelism and conver- gence properties. SIAM Journal Optimization, 25(1):351-376, 2015.
Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. From dis- tributed machine learning to federated learning: A survey. Knowledge and Information Systems, pp. 1-33, 2022.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for person alized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10092-10101, 2022.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 201 Association for Computational Linguistics. URL http://www. sctteb.og//atthooy//P1--0111
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017a.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiennn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282 PMLR, 2017b.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp- 4646 4655. PMLR, 2019.
Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint arXiv: 106.0642, 2021.
Constantin Philippenko and Aymeric Dieuleveut. Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems, 34, 2021.
Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha. Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering, 14:156-180, 2020.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and coamuuncaaiion-ffficin federated learning from non-iid data. IEEE transactions on neural net- works and learning systems, 31(9):3400-3 3413,, 2019.
Martin Jaggi Sebastian U. Stich, Jean-Bapa Baptiste Cordonnier. Sparsified sgd with memory sebastian.
arXiv preprint arXiv: 000. 2018.
11

---

Under review as a conference paper at ICLR 2023
Yuanming Shi, Kai Yang, Tao Jiang, Jun Zhang, and Khaled B Letaief. Communicatton--ffiiiienn edge ai: Algorithms and systems. IEEE Communications Surveys & Tutorials, 22(4):2167-2191, 2020.
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. Enabling fast differentially private sgd via just-in-time compilation and vectorization. Advances in Neural Information Processing Systems, 34, 2021.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Systems, 33:21394-21405. 2020.
Wei Wen, Cong Xu, Feng Yan, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Temgrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems, 30, 2017.
Xueyu Wu, Xin Yao, and Cho-Li Wang. Fedscr: Structure- communication reduction for federated learning. IEEE Transactions on Parallel and Distributed Systems, 32(7):155-11577, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.
An Xu and Heng Huang. Coordinating momenta for cross-silo federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 2022.
Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5(1):1-19, 2021.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participa- tion in non-iid federated learning. Proceedings of ICLR, 2021a.
Haibo Yang, Jia Liu, and Elizabeth S Bentley. Cfedavg: achieving efficient communication and fast convergence in non-iid federated learning. In 202/ 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt), pp. 1-8. IEEE, 2021b.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang. Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Process- ing Systems, 34, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806. 00582, 2018.
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning non-iid data: A survey. Neurocomputing, 465:371-390, 2021.
12

---

Under review as a conference paper at ICLR 2023
A COMPARISON OF DIFFERENT AGGREGATION METHODS
In this section, we give two examples to com- pare the results of different aggregation meth- ods under the IID and Non-IID settings.
For the IID setting, we suppose that there are three clients with their updates, i.c. AW1 (1,1), AW2 (2 and AW; (3.3). It is evident that the original aggregation of these
Dunt
EWA
(a) IID
(b) Non-IID
dense updates (2. 2) (red arrow) as shown in Figure 6(a] which is regarded as the accu- rate result. We then generate the correspond-
Figure 6: Aggregation results of different methods.
ing sparse updates, i.c. AW; = (0, 1). AW, = (2 0) and AW; = (0, (0,3), whose biased result
(2, {)) (green arrow) is easily aggregated via the naive averaging method (McMahan et al. 2017b). From microcosmic perspective, if we compute the aggregated vector via the Element- Wise Ag- gregation (EWA) method as shown in Section 4.3, we would get the precise aggregation (2, 2) (blue vector) with the above sparse updates.
In similarity, we assume that the clients' updates are AW1 = (1, AW2 = (3, 2) and AW3 = (3, 4) in the Non-IID setting. Then we sparsify these updates via Top-K operator, i.e. AW, 20, AW; = (3, 0) and AW;- (0, 4). It's easy to calculate the accurate aggregation, (3 3 (red arrow), the naive result, (1, 2) (green arrow), and the EWA vector, (3, 3) (blue arrow). Apparently, the EWA result more close the accurate dense aggregation than the naive sparse aggregation shown in Figure 6(b) when it refers to the direction and magnitude.
Overall, the original averaging approach is easily biased by the sparse matrices, while our proposed EWA method is more suitable for the sparse aggregation.
B CONVERGENCE ANALYSIS
B. ANALYSIS PRELIMINARIES AND ASSUMPTIONS
The framework FedPSE used minimize the differentiable loss function ft R" + R (vi P) individually, where n indicates the dimension of parameters. We consider the general setting in deep learning where fi is non-convex function. Our convergence results are proved under the following assumptions:
Assumption 3. Smoothness: The loss function f (Vi P) is L-Lipschitz smooth (L-smooth), i.e., [V - -V((W))) s L| - WTll, VW W.' E IR".
Then we assume that client i trains its model W; with unbiased stochastic gradient, i.e., (W:) = (W,,), in which G; denotes the stochastic gradient with a batch of M samples.
Assumption 4. Bounded Gradient: The second moment of local gradient G2 is bounded, i.e., ΣN EW.) < o², VW, E R", where ||-I] is normm..
As shown in Algorithm FedPSE includes three steps, i.e., upstream personalized sparsification (UPS), element-wise aggregation (EWA), and downstream personalized sparsification (DPS). The training steps of each federated round are termed as PSE collectively in the following sections. The local model is updated during the (r + 1)-th federated round by the following expression:
WT+! WT (a"G +c(²)
(1)
where a" is the learning rate.
13

---

Under review as a conference paper at ICLR 2023
According Algorithm 2, e a-- GW(""")) " expand the PSE with the previous parameters and get
(G: 2)
We then
PSE (WT) +
-M,, + R(M dz.s ko) + R(M (1 (2 Ke) @ AW"
-(M, + Mr.. + MM,e) o AW?
=W" @ (M + Mf.e) o Top-k Σ Top-k(a" + +((
+ Mr O Top-k(a (W")) + +
where Top-k(-) is the compress operator which is described in Algorithm R() denotes the random selection function Algorithm and W In 2 Σ Sign (T))-)( + +e(" element weight matrix.
the
Similar to Dan Alistarh al. 2018),
W to denote the auxiliary model weight with conver- gence guarantee during r-th federated round on client i, and get
W W G'G(W"
(3)
where G(WT) #/- Σ GW())) and 7 wP(Vi P). The difference between the auxiliary model W and the local model W7' can be represented by
W Wr
W
PSE E + + 1 QGG(W")
(4)
Σ PSE N I (a + +
0 GWW))
N 1=
Then we define commonly used k-contraction operator (Gao et al. 2021 shown below:
Sebastian U. Stich 2018
Definition 1. For any vector W R and 0 < k S n, operator (R IR") a k-contraction operator if it satisfies the following property;
BlJW- Q(w) 2 S (1 WW
(5)
Apparently, the Top-K compressor is k-contraction operator.
Lemma 2. VW, E R" and 0 < k S n, we have
Ell aa ((W)) + ET) - (a
(6)
|| Σ aa +(()) <(1 - n
Proof. Combining equation 2 and Definition I we obtain
Elll
(a"G.(WT)++ +() No (o=,-
N
1 + -
<Ell|
N
N
<E 1
(a' - Σ (oGG
N
N n N
k k)ll
<(1
Σ
(a + +
14

---

Under review as a conference paper at ICLR 2023
B.2
MAIN PROCESS
As a standard situation non-convex settings following convergence:
Liu & Wright 2015). we want to guarantee the
min BIIVs. R=300 0 ref1,-, RY
that is, the algorithm converges ergodically the optimal point where gradients are zero. Our purpose is to minimize the difference between the "real" model WT and the viewed WT observed at federated round r, which means decreasing the value of loss function fi(W?). Furthermore, we need bound the following expression:
a
Σ
Then We get Lemma 3 Lemma 3. For any federated round 1:
E B|| - WT]11]
1 2.0
Σ (0(1 + 9))
(7)
P
N
where :p - -4 < t < and p 0.
Proof. We derive the difference between W and Wr+1
B|| W1 =][ PSE (a GWTT) +f) Wr a'G(WT)] |
W" PSE | + + a" Σ G((WT)
=
N N
N
- Wr
c
(a' + o.)) + N Σ (W)())++))ll
N
ト Σ $
<BI|
(a GWW)) + +
Blllw
N
From Lemma
we can obtain
B||lW++ W 119
Spll N
Σ aa (WW))
+ + wr 1
N
N
S:pl N 1=1
Σ a G.T)I
Σ Eلل + B[[ - w
Σ
N
From equation
4 and Lemma
we have
BillW]
wr+1 1|2]
<pl a G(WT) 2 + PBIIWT Wr11
2
+ (1 E
Top- (oW) + - Σ (a9G +
SpBl opEllo"G(11 G(W || + WE : (1 E||++ 11
<(1 + (1) + 1 1 " G(
w(1 D W - 1
15

---

Under review as a conference paper at ICLR 2023
Iterate the above inequality by r to get:
1 2
W W
P
From Assumption and Lemma 3
have
1 24
W[[
Σ (s(1 (0)++)))(""""W""""""""))0)")""01
P
1 + 24
(50( + (a)
P
N
-=1
Theorem 2. Assume that our proposed FedPSE is applied to minimize the objective loss function fi that satisfies the assumptions in A.1. If we choose learning rate schedule that satisfies:
Σ (s0(1 + p))a (a 9Q-
SC
(8)
or
then for some constant C have the following result after R federated rounds:
(((W))))((W))
20
N (1 + 2L(1+2x)C
Σ
Σ a C'ELIIAAIEIII S
p
2/m1 a
D/1 a
where W is the optimal solution fc-
Proof. Under the Assumption
3 we have
(IWT+1)-A(WW)) & (VS(())) Wit1 -WT) L ||r+1 W/|1
! "la=GG(TT)I
--(VA("".
and
-
= -V
< <WW) - - -
(10)
22.
Taking the expectation federated round r, bonud
E|A( - -(((m)))))))))))
( + El l G(WT)
2
(AΛ(#))) (()))")) L
Bl G(W")
(a B[IGGWTOI)))
2
a" 2
(a Bl|
I W/|1
La")
< 2 (IV + - -w
a I'
W112
2N
16

---

Under review as a conference paper at ICLR 2023
Taking the expectation before yields
E [[f((W++]] - [W]]
a" (BllI (BIlVf.( | - W W (a")²To² 2N
s a' [ 1,2 W - W1]33
+ 2N + a a' 2p)
Lo))LO?
(0(1 + pa)) N
a'
(BIIV)((W"))) |2 1 W - [T]]]
Lo))
2N + (a + 24)
Σ (p(1 + p)) 0))o 9a- 0²
a" N
Apply 8 to the above inequality, we get
E(i(w++1)) - [((((W)))
[B [B]] + L3I - Ww + (a")
(a")2 2xp)
+
2
2N
PN
(a
Po")
2L(1 + 2p)C
[B]] +3I,, (/)))
(1
2N
Combine with inequality 10 can obtain
a <'BIII < 2a 20 (B]I)V) 1311 -
<4 -(((((W))) 2(a
2L + +
N
P
Summing up the above inequality for r = 1,2, ... R. we have
R
20 (1+ 21(1 2
- s((i)) -
Σ
(a') (11)
By dividing the summation of learning rates and therefore:
R
) )(W))) a (EWII
20 (1+ 21+1+2))
DR - (a")²
p
Σ
Σ a"
Σ Σ a"
(12)
The condition
8 keeps right if (pl p) 1. To derive the bound of p, we have
e + p) = (1- (1+
Therefore, one should choose / म to satisfy the above inequality. Theorem 2 implies that each client's model in the FedPSE framework will converge if federated round R is large enough when a" satisfies the following conditions:
and lim Σ (ar)²
lim 700
Σa a
k0000
R Σ ar
Corollary 1. Under the assumptions in Theorem
if 4(1 + p) and a" = 0
MN
Vr
0
R
where 0 > O is a constant, we have the convergence speed of FedPSE:
R
VA(WW))) A(f(W?) - f((W))
20L /
4o2 222 + 20p
R
0/MNR
VNR
Rp(1 -
17

---

Under review as a conference paper at ICLR 2023
Proof. First we prove that a" = /
MN R
constant step size, satisfies the inequality 8
we set
a" for simplification:
Σ (0(1 + p)) 9) 9-1
9)²
7(1 -T)
0
Σ
79
0
a²
a²
1-5
q=1
q=1
Since 0 S r < L, we then obtain
7 - T²) OT
lim
--T 1
Therefore, inequality ( holds under the condition: C or inequality of the expected average-squared gradients of Sú i.e.
From Theorem
obtaín the
I-r
[VA(W))))) 4lf:(W:)- -(WW))
20
2L +24)C )ar
E
Σ
N (1
aR 4(A(W)) - I(WI)) 8VMNR
P
20Lo² VM
40² 12(L + 20p
VNR
Rp(1 -
From Corollary we can conclude that the framework FedPSE has convergence rate of O with proper learning rate,, also indicates that the hyper-parameter (in Top-K) has minor impacts on the convergence rate R large enough.
vT
C DETAILS OF EXPERIMENTS SETTINGS
C.1 SEPARATION OF CLIENTS' DATASETS
In this section, we propose the details of the splitting method with a "flat" existing dataset in the Non-III setting. Firstly, the whole training/test samples are separated into two parts, including the heterogencous subset with À percent and the homogeneous subset with (1.0 - A) percent. On one hand, the homogeneous subset is averaged partitioned into clients randomly. On the other hand, the heterogeneous subset is sorted by labels and split into N clients by order, which indicates that each client has its own distribution. Finally, we combine the above- menteneet homogeneous part and heterogeneous part into the individual training/test dataset of each client.
C.2 MODELS FOR DIFFERENT DATASETS
We establish an ConvNet2 model (Beutel et al. 2020) for MNIST and FMNIST datasets (Subramani et al. 2021). In similarity, construct model with one Embedding layer, which is prebuilt in Keras, followed by fully-connected layer on the IMDB dataset. Finally, we train the Resnet18 model (He et al. 2016) the Cifar10.
C.3 HYPER-PARAMETER OPTIMIZATION
We fix the architectures of models and the random seed as 1. Then we optimize the hyper- parameters(e.g. batch size ranging from 64 to 512 and learning rate ranging from 1e-3 le-2) for different strategies to compare their best metrics in the following experiments. Particularly. in order accelerate the training process of Resnet18 on Cifar10, we use pre-trained model initialize the clients' weights. The experiments are conducted in a stand-alone PC to simulate the communication federated learning.
D SUPPLEMENTARY EXPERIMENTS
D.1 PERFORMANCE COMPARISON OF ALGORITHMS WITH DIFFERENT NUMBERS OF CLIENTS
In order to prove the effectiveness of our method, we vary the number clients from 2 to 100 and report their accuracy in Table where perform the experiments the FMNIST and IMDB
18

---

Under review as a conference paper at ICLR 2023
0.99
Test Accuracy
0.91
Accuracy
0.95
0.87
0.91
0.83
0.87
0.79
0.83
FolPSS
Sampling Rate
Sampling Rate
(a) Comparison FMNIST
(b) Comparison IMDB
Figure 7: Accuracy comparison over sampling ratios on Non-IID datasets.
datasets with 1.0. Then we compute the promotion of our method compared to the SOTA methods via the following function:
Promotion AccrearsE - AcccoOTA
AccSOTA
Then we abbreviate the promotion of FedPSE compared to FedSTC as P1 and the promotion of FedPSE compared to FedSCR as P2 shown Table 2 From Table 2 we have the conclusion that our proposed FedPSE achieves the best performance among all algorithms under the different numbers of clients, which consistent with Section 5.2.
Table Performance comparison algorithms with the different number of clients on Non-IID datascts.
Datasets
FMNIST
IMDB
clients
FedAVG FedSTC FedSCR FedPSE
P1
P2
FedAVG FedSTC FedSCR FedPSE
P1
P2
2 0.8781
0.8844
0.9031
0.9466
7.03%
4.82%
0.8406
0.8409
0.8438
0.9953
17.33%
11.65%
5
0.8068
0.8756
0.8821
0.9267
5.84%
5.06%
0.8494
0.9066
0.9366
0.9966
9.78%
6.55%
10
0.7909
0.8614
0.8603
0.9371
8.79%
8.93%
0.9080
0.9582
0.9620
0.9989
4.17%
3.72%
25
0.8045
0,8553
0.8612
0.9021
5.47%
4.75%
0.8570
0.9104
0.9112
0.9923
10.30%
8.99%
50
0.8071
0.8343
0.8531
0.8939
7.14%
4.78%
0.8356
0.9003
0.9085
0.9986
10.92%
9.92%
100
0.7921
0.8211
0.8344
0.8772
6.83%
5.13%
0.8401
0.9036
0.9101
0.9975
10.41%
9.63%
Average
0,8133
0.8553
0.8657
0.9139
6.85% 5.57%
0.8551
0.9017
0.9120
0.9965
10.68%
9,43%
D.2
PERFORMANCE COMPARISON OF ALGORIHHMS IN THE PARTIAL CLIENT PARTICIPATION SCENARIO
We take experiments with partial clients participation for aggregation on the FMNIST and IMDB datasets in the Non-III setting (A = 1.0). Specifically, the clients are randomly sampled by the server at different rates. As shown in Figure our method (red line) significantly outperforms other models, which is align with the conclusion in Section 5.2.
D.3 VARIANCE OF CORRELATION DISTANCE
Table Variance of correlation distances in FedPSE with different Non-IID ratios.
Datasets
FMNIST
IMDB
clients
2 clients
5 clients
10 clients
2 clients
5 clients
10 clients
À 1.0
5.8e-3
6.3e-3
2.0e-3
2.8e-3
1.2e-2
1.7e-3
À 0.5
3.6e-5
4.7e-5
4.0e-4
2.3e-4
4.5e-4
1.3e-3
A 0.0
1.0e-8
3.3e-6
2.9c-4
1.7c-4
2.0c-4
4.1e-4
19

---

Under review as a conference paper at ICLR 2023
In this section, compute the variance of correlation distances with diverse Non-IID ratios À as shown in Table 3 in which we perform the experiments on the FMNIST and IMDB datasets with variable clients' number. Then deduce our conclusion in section 5.4.
D.4 CONTRIBUTION OF THE DPS ALGORITHM
0.93
In order to prove the effectiveness of the DPS algorithm, we construct experiments of FedPSE with different downstream updates on the FM- NIST datasets over Non-IID ratios À (i.e. from 0.0 to 1.0) shown in Figure [ The FedPSE without the DPS algorithm indicates that the server sparsifies the global updates with the TopK method and broadcasts the same sparse information to all clients. Apparently, the curacy of FedPSE with DPS exceeds the algo- rithm without DPS through different distribu- tions in the Non-IID setting (i.e. À 2 0.2). In conclusion, the DPS method, personalizing the clients' models, is useful for our paradigm with heterogeneous datasets.
FedPSE without DPS
Test Accuracy
0.91
FedPSE with DPS
0.89
0.87
0.85
0.0
0.2
0.4
0.6
0.8
1.0
Non-IID Ratio
Figure 8: Accuracy with different downstream updates.
E EXAMPLE CODE
We provide the example code for our framework in this section. Figure 9 shows an example code of the UPS algorithm, while Figure an example implementation of the EWA algorithm and Figure 12 for the DPS algorithm. Furthermore, we present the training process of FedPSE in Figure [3 in which we assume that there are two clients and b) with two-layers neural networks.
20

---

Under review as a conference paper at ICLR 2023
from abc import ABC import numpy as np
import tensorflow as tf
4
define the client class with the ups algorithn class Client def __init__
self, model, ds_train, ds_valid
10
):
self.modee model
12
self deltain_ dsan self.ds_valfi ds_valid self. self.model_we weigh_ self.res_ert []
13
14
15
16
17
def ups (self, updates, sparsity):
18
update local model with downstream updates self.model model weights np.add (w, u) for w, u ]
19
updates)
22
self. seadel.set_wettee wet_ (self. model_ model_weights)) # train the local model using local dataset hist melf.model. model.fit(
23
24
self. ds_train,
26
validation waw.iionnddassel..dss.s..... ds_elf.dddvalid. epochs=1
27
28
)
n. namsaeles self. ds_train cardinality(). numpy
sparsify the upstream updates topk with error residual self. clienttuapetss =
31
32
np. subtract (new_w, old_w)
33
for new_w, old_w in zip(self model. get weights)) melf.model_weigee
]
35
merged_updan updates
36
np. add (new_u, err) for new_u, err in zip client_updates, self res_err)
37
38
]
39
upstream_updates = topk_algorin thm thm (merggged__pddtees
40
self. res_erssrre err =
41
np. subtract (lu, su) for lu, su in (pddaeeee__uddaees. upstrean_updates)
42
]
44
45
return ppstream_updates. n_samples
Figure 9: Example code of the UPS algorithm.
21

---

Under review as a conference paper at ICLR 2023
from functools import reduce import numpy as np
4
define the function of the EWA algorithm def ewa(clients secliient__updats, clients_ sanntsssmmles): get the participated samples of each element
weighted_num =
sign (np- absplute(layer)) num for layer in updates] for updates,
10
in zipcels clients_samples)
12
num jum_samples = reduce (np. add, layer_num) for layer_nus num in zip
13
14
]
15
16
get the weighted sum of updates on each index wei geightee_updates =
17
18
[layer num for layer in updates]
19
for updates, sum_up reduce (np. add, layer_updates) for layer updates in zip ("weighted_updatee )
in zip(clients__pdaat
clients_samplest
21
22
]
23
24
get the element-wised updates el element_wise wiintttwwse_updatesetstte =
26
np divide (u, n)
27
for u, in cip(su_upddates, num_samples)
28
]
30
return elieeent.vie__updaees
Figure Example code the EWA algorithm.
22

---

Under review as a conference paper at ICLR 2023
import numpy np
def merge_ (servverns merged_indices []
client_indices, dist):
compute the merged indices of each layer
for s_ind, c_ind, in zip(serveriindices client_indices, dist):
# get the intersection of index matrices inter_ind np app.mltipllys__ind, c_ind)
10
i_nonzero_num np. (ooounttoonzeerrinttee_.. inter_inder np. (laatonnerroinnte___iie compute corresponding compensation s $_monzero_num copuottt_nnozeeeee (s
11
12
13
14
server_c s_ind inter_ind
15
client_c c_ind inter_ind server_c_inde np. (latoonzerosservvr.cc client_c_indes np. caanoonero(clleente
16
17
18
select the indices from the compensations client_num round (s_nonzero_num i_nonzero_num)
19
20
server_num round d) (s_nonzero_nmg i_nonzero_num) server_inder np. sort(
22
choice (server_c_indes)
server_num replace=False)
23
)
24
cl client_index np. sort ap.random.choice
25
choice ChoiceecClintt_ccinner, client_num, replace=False)
)
27
# get the merged indices merge_index = np. concatenate (anentee((iin (Cinter_index, server_index, client_index)
28
29
indices_ array np. zeros lind_nddd.. facaapee..latte. andices_arrayymerge..ee merged_indices... cwiiiieeeaapnediiiiisecccno (indices_array reshape (s_ind, shape)
30
32
return merged_indices
Figure 11: Example cade of the utils.
23

---

Under review as a conference paper at ICLR 2023
import numpy as np from utils import berge_indices
define the function of the DPS algorithm def dps(c_updates, updattes, -.spatre_uddates):
[Lients_down_updal
[]
get the index matrices of the server server_indices =
10
np. nignign absolute (layer))
11
for layer in s_updates
12
13
s_norm [w / np. linall.l........ (w for и in s_updates]
14
get the downstream updates of each client for client_updates in c_updates:
15
16
get the index matrices of the client client_indices np. sign (np absolute
18
19
for layer in client_updates
20
] compute the correlation distance client_norm dist =
21
np www.linall.... norm (w) for in c_updates]
23
24
0.5 0.5
c_layer)
25
for s_layer, c_layer in zirms_norm client_norm)
26
]
get combined indices merged_indices merge_indices( server server_indices. ALient_indices:
28
29
30
dist
32
)
33
# compute the downstream personalized updates dlient_dow__updaeer =
34
35
np. mp.multiply (layer, index) for layer, index zip(s_updates, merged_indices)
36
37
]
38
clients (ww.aaaaeeeeeeeeeeeeeeee......
39
return
Figure 12: Example code of the DPS algorithm.
24

---

import numpy np import tensorflow as tf Under review as a conference paper at ICLR 2023
4 from ewa import ewa from dps import dps from ups import Client
define neural network model 10
def build_model (inild__meellinut___1e__ (input_shape): model ke.ks Sequential(
12
13
keras Input (shape=input (nppt(sstpi-iiptttsspp)))
14
tf. Dense (50, activation="relu"), tf.keras layers pense(10, activation="relu")
15
16
]
17
)
18
model. compile(
19
optimizerstf. keras. oprrassetiiizers... Adam
loss=tf. keras. losses. categorical (trrics=("cccurayy"
21
22
)
23
return model
main process
26 initial byper-parameter
27 sparsity = 0.9
28
max_round 100
30 clients load data
31 a_ds_train, vali_, feature_dim = tfa__tf_attsets ("data_a_
32
b_di_train, b_ds_valid, feature_dim load tfddttfdatteete
define model
35
model = build_model (feature_dimg
36
client_a = Client (model, a_ds_train, a_ds_valid)
37
client_b Client (model, b_ds_train, b v_lid
a d_down_updaa updates
[] d_dow__pdates updates []
39
down
40
for in range (max_round):
41
a_up_updates, a_samples
siient_a.ussa__eee (adddnnuutttd sparsity)
b_up_updates, b_samples client_b. ups db_ddww_updaeess sparsity)
43
clients_updates = [a_up_updates. b_up_updates]
44
clients siients_sample = [a_ [a_samples, b_samples]
45
s_updates ewa (cclients__udates. el clients_samples)
46
s_sparse_updates thpk_aalgrrttmms_upppppp
48
_doow__udates b_dowm_updates = dps(
49
Wlients_updated s_updates, updates
50
51
client_a monttaa.modll... model. save sondel..aveecllientttt. cclient_a_mode) .h5
53
client_b model, save
Figure 13: Example code of the main process.
25


Reviews:
[{"review_id": "lQmlrqgX3oO", "reviewer": "Reviewer_7ra3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a personalized sparsification strategy for federated learning that compresses the updates in both directions, i.e., from clients to server and from server to clients. The proposed method, FedPSE, achieves that by (1) sparsification of model updates using Top-k algorithm before communication from clients to server, (2) element-wise aggregation at the server, and (3) sparsifying the global model updates in a personalized way before communication from the server to the clients. The experimental results show that FedPSE outperforms FedAVG consistently, and FedSTC and FedSCR occasionally on MNIST and FMNIST datasets. ", "summary_of_the_review": "Please see my comments above. ", "strength_and_weaknesses": "Strengths:\n- The paper cares about bidirectional communication cost, which is nice as server-to-client communication cost is mostly neglected in the literature. \n- The authors provide theoretical convergence guarantees. \n\nWeaknesses:\n- I am confused about the Sign(.) operation in Algorithms 2 and 3. What is the point of taking the sign of a nonnegative matrix $|\\Delta \\hat{W}^r_i|$?\n\n- The upstream sparsification and element-wise aggregation are part of many existing frameworks. The only novel block seems to be the downstream personalized sparsification. \n\n- In Section 2.3, it says FedSTC and FedSCR share the uniform model. What does that mean? What I understand from a uniform model is a model with uniformly distributed parameters. But I think the authors meant that each client has the same model. I think changing the terminology here would avoid some potential confusion about the prior work. \n\n- Since each client have a different dataset distribution and a task, wouldn't it make more sense to choose different sparsity ratios for each client? By forcing the same sparsity ratio on all clients, FedPSE performs worse for clients whose datasets require denser models. So I think this approach brings a fairness problem. Do the authors have a solution for this?\n\n- In Section 5.1, the authors said they used the same learning rate for all compression methods. The learning rate and most other hyperparameters should be tuned for each baseline for the best performance. Otherwise, the comparisons are not valid and they would favor the method the authors tuned the learning rate for. \n\n- It would make reading easier if the references for FedAVG, FedSTC, and FedSCR were included in the Table captions, or at least somewhere in the experiments section. \n\n- What is the strategy to communicate the sparse models between the server and the clients? Once the model update is sparsified, it should be encoded in a way that the sparsity helps reduce the bitrate. What is the proposed strategy there?  \n\n- It seems from Table 1 that FedPSE is losing the performance gain for $\\lambda < 1$? While it outperforms the baselines consistently for $\\lambda=1$, it either underperforms or performs almost the same as the baselines for $\\lambda < 1$. Does this imply that FedPSE does not provide much advantage for not purely heterogeneous datasets? A comment on this in the experiments section would be nice.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QDGXWWj-Qb", "reviewer": "Reviewer_E2iq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new communication compression method for federated learning with heterogeneous data. In particular, the compression scheme is personalized to each client. The authors show that several new techniques, such as downstream compression personalization, and element-wise aggregation, are critical to achieve a good performance in the presence of non-IID data.", "summary_of_the_review": "Currently I am leaning towards a reject. The main reasons are (1) the motivation of this paper is in conflict with their own experimental observations. So it requires some rewriting; (2) The effectiveness of DPS is unclear. More ablation studies are needed.", "strength_and_weaknesses": "Strength\n1. The proposed element-wise aggregation method seems to be very promising. It can significantly increase the performance of compression algorithms in FL.\n2. Combining all new techniques, the proposed algorithm can achieve significant improves over existing algorithms, especially on CIFAR-10 dataset.\n\n\nWeakness\n1. Weak motivation: The first key problem I found is that the problem this paper aims to address may not even exist. In the introduction, the authors claimed that \"the current compression techniques, ignoring the personalization of clients, face a significant performance degradation on Non-IID datasets\" However, if we look at table 1, previous FedSTC and FedSCR algorithms seems to be even more robust to the data heterogeneity than FedAvg. In many cases, the performance of FedSCR and FedSTC even increases along with the increase of data heterogeneity. These experimental observations make the motivation of this paper invalid. The authors may need significant efforts to change the wording in the whole paper.\n2. Misleading name: The name of \"element-wise aggregation\" is confusing. Because in FedAvg and many other basic FL algorithms, we also conduct element-wise averaging over clients. What the authors really does here is element-wise normalization. I suggest the authors to change the method name.\n3. Compared with previous communication compression works, I think one of the key differences in this paper is downstream personalization. Clients will download different parts of the global model to begin the next round. So a key question here is that how effective is this technique? The authors did not provide ablation study on this. It is possible that this technique is unnecessarily complex. For example, how about transferring the same compressed global model to all clients?\n4. The theoretical analysis is a bit weak. As there is no local updates at clients, which is the basic component in FL algorithms. Also, it is unclear what the exact algorithm the authors analyzed. They remove the local updates part. So are there any components removed from the algorithm in order to conduct the analysis? It would be nice if the authors can write down the exact update rule of the proposed algorithm.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FVYhIz-IMcK", "reviewer": "Reviewer_DP2J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on enhance the performance of non-iid FL on the basis of reducing the communication cost.\nThe author proposes a method that uses the personalized Top-K sparsification and element-wise average techniques.\nThe main contribution of the paper is that it proposes a downstream selection mechanism to personalize the clients' models, which adapts to various distributions and can increase the performance in the non-iid setting.", "summary_of_the_review": "In short, the motivation of the paper is good to me. The author(s) may consider doing more experiments as mentioned above to show the strengths of the proposed algorithm. Besides, it would be more convincing if pseudocode/code is provided.", "strength_and_weaknesses": "1. This paper is not the first one to utilize element-wise aggregation in FL to enhance the performance in the non-iid setting. For example, pFedLA (Layer-wised Model Aggregation for Personalized Federated Learning), which was published in CVPR 2022, also utilizes element-wise aggregation in FL. So the comparison of the proposed method and pFedLA is required to be included in the paper's analysis as well as experiments.\n\n2. The details of the experiments are missed, such as learning rate, random seed, batch size.\n\n3. The experiments have not show the advantage of the communication efficiency. It only compare with the algorithms that also have improve the communication efficiency. However, it's necessary to compare with the SOTA personalized federated learning approach that didn't consider the communication efficiency, to show the necessity of communication efficiency. For example, if your approach performs much worse than those general personalized FL, then the FL operators may insist to use general SOTA personalized FL. Besides, it's necessary to do a experiment to compare with a method that skipping the communication efficiency enhanced strategy of your proposed algorithm to show the contribution to reducing the communication cost; on the other hand, it's necessary to compare with that skipping the \nelement-wise aggregation to show it's useful to set up such a component.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject