Personalized Federated Learning with Communication Compression
Authors: El houcine Bergou
Konstantin Pavlovich Burlachenko
Aritra Dutta
Peter Richtárik
Venue: Accepted by TMLR ()

Abstract: In contrast to training traditional machine learning~(ML) models in data centers, federated learning~(FL) trains ML models over local datasets contained on resource-constrained heterogeneous edge devices. Existing FL algorithms aim to learn a single global model for all participating devices, which may not be helpful to all devices participating in the training due to the heterogeneity of the data across the devices. Recently, Hanzely and Richt\'{a}rik (2020) proposed a new formulation for training personalized FL models aimed at balancing the trade-off between the traditional global model and the local models that could be trained by individual devices using their private data only. They derived a new algorithm, called {\em loopless gradient descent}~(L2GD), to solve it and showed that this algorithms leads to improved communication complexity guarantees in regimes when more personalization is required. In this paper, we equip their L2GD algorithm with a {\em bidirectional} compression mechanism to further reduce the communication bottleneck between the local devices and the server. Unlike other compression-based algorithms used in the FL-setting, our compressed L2GD algorithm operates on a probabilistic communication protocol, where communication does not happen on a fixed schedule. Moreover, our compressed L2GD algorithm maintains a similar convergence rate as vanilla SGD without compression. To empirically validate the efficiency of our algorithm, we perform diverse numerical experiments on both convex and non-convex problems and using various compression techniques.

Published in Transactions on Machine Learning Research (11/2023)
Personalized Compression
Federated Learning
with Communication
EI Houcine Bergou"
Elhoucine www.::inne.errroooumm......
College of Computing
Mohanamed Polytechnic University
Ben Guerir, Morocco
Konstantin Burlachenko
konstantin wwwp.a/aaaaaaacnoookkk.........
King Abdullah University of Science end Technology, KSA
Aritra Duttal
aritra.dutta@uce
Artificial Intelligence Initiative University of Central Florida Orlando, Florida-32816
Peter Richtárik
peter cisterrichttrikkkkuut.cco edu.
King Abdullah Universily of Science and Technology, KSA
Reviewed on OpenReview: https: // openreview. formm? id= dZugyhbNFF
Abstract
In contrast training traditional machine learning (ML) models in data centers, federated learning (FT.) trains ML models local datasets resource-constrained heterogeneous edge devices. Existing FL algorithms aim to learn single global model for all participating devices, which may not be helpful to all devices participating in the training due to the heterogeneity of the data across the devices. Recently, Hanzely and Richtárik (2020) proposed a new formulation for training personalized FI. models aimed at balancing the trade-off between the traditional global model and the local models that could be trained by individual devices using their private data only. They derived a new algorithm, called loopless gradient descent (L2GD), to solve it and showed that this algorithm leads improved communication complexity guarantees in regimes when more personalization is required. In this paper, equip their L2GD algorithı with bidirectional compression mechanism to further reduce the communication bottleneck between the local devices and the server. Unlike other compression- based algorithms used in the FL setting, our compressed L2GD algorithm operates on probabilistic communication protocol, where communication does not happen on a fixed schedule. Moreover, our compressed T.2GD algorithm maintains similar convergence rate as vanilla SGD without compression. To empirically validate the efficiency of our algorithm, we perform diverse numerical experiments on both convex and non-convex problems and use various compression techniques
1
Introduction
We live in the era of big data, and edge devices have become part of our daily lives. While the training of ML models using the diverse data stored on these devices is becoming increasingly popular, the traditional data benter-based approach to training them faces serious privacy issues and has to deal with high commanication
"Equal contribution
Equal contribution
"Our repository available online: http:///ghtth..coooooaaaaaeeeees

---

Published in Transactions on Machine Learning Research (11/2023)
Central server
D1
Time >
D2
D1
coin
1)2
h(x) = h4 xx)
Local model
n
up
up
Duta
Local model
Wn
RA
W1
Loss
Local
modes
Loss
A
x
RP
(()))
(x()
Datal
W
Local Loss
DI
D2
DI
D2
(x;))
(b)
(a)
Figure 1: (a)rrainng n local devices, } on the loss, Ji of their local model, 2; with central server server/maste /master node, where h2 penalizes for dissimilarity between the local model, and the average of all local models, a. Fed (MeMahan et al. 2017) and L2GD (Hanzely & Richtárik] 2020) algorithm on devices (D1 and D2). Unlike Avd,vg L2GD does not communicate after fixed T local steps, it communicates based probabilistic protocol.
and eneryy cost associated with the transfer of data from users to the data center (Dean et al. 2012). Federated learning (FL) provides an attractive alternative to the traditional approach as it aims to train the models directly on resource constrained heterogencous devices without any need for the data to leave them (Konečný ett al. 2016 Kairouz et al. 2019).
The prevalent paradigm for training FL models is empirical risk minimization train single global model using the aggregate of all the training data stored across all participating devices. Among the popular algorithms for training FT. models for this formulation belong FedAvg (McMahan et al.
2017), Local GD
[Kaledd et al. 2019 2020), local SGD (Stich 2019 Khaled et al. 2020: Gorbunov et al. 2021 and Shifted Local SVRG (Gorbunov et al.] 2021 All these methods require the participating devices to perform a local training procedure (e.g., by taking multiple steps some optimization algorithm) and subsequently
communicate the resulting model to an orchestrating server for aggregation; see Figure la This process is repeated until a model of suitable qualities is found. For more variants of local methods and further pointers to the literature, refer the reader to Gorbunov et al. 2021).
1.1 Personalized FL
In contrast, Hanzely & Richtárik (2020) recently introduced formulation of FL alternative to the existing singll-model-suuts-alll approach embodied by empirical risk minimization. Their formulation explicitly aims to find personalized model for every device; see Figure la In particular, Hanzely & Richtárik (2020) considered the formulation
min [F(x) = f(x) h(x)]
EERod
for simultaneous training of n personalized FL models 21, In E Rd for n participating devices. They chose
= =
+5 M(x;), and Mxx = Σ lx(x).
where represents the loss of model I; over the local data stored device i. Function h; penalizes for dissimilarity wetn the local model and the average of all local models F = Σ Tz, and is defined to be
hx(x) 311 =
where A > 0 controls for the strength of this penalization At one extreme, - forces the local models to be equal to their average, and hence, mutually identical. Therefore, equation reduces to the classical
Zhang 2015) considered similar model different context and with different, motivations.
2

---

Published in Transactions on Machine Learning Research (11/2023)
empirical risk minimization formulation of FL
n Σ f((2). ZER
min
On the other hand, for À = 0 problem equation is equivalent to each client (node) training independently using their data only. In particular, the ith client solves
min f((z)).
By choosing À to a value in between these two extremes, i.e., <<<<< 00, we control for the level of similarity we want the personalized models {ziJi=1 to possess.
We remark that local methods such as FedAvg by McMahan et al. (0177) (also see similar methods in (Haddadpour et al.) 2019b; Stich, 2019 Wang & Joshi 2019 Zhhou & Cong 2018 Lin et al. 2020 ) are
popular for training FL models. Nevertheless, their main drawback in the heterogeneous setting with data and device heterogeneity is inefficient communication. Hanzely & Richtárik (2020) proposed this new personalization to tackle heterogeneous data, and we are using their model to build our compressed, personalized FL.
proposed a probabilistic gradient descent algorithm for which they coined the name loopless local gradient descent (L2GD). Hanzely Richtárik (2020) shows how L2GD can be interpreted as a simple variant of FodAvg, typically presented as method for solving the standard empirical risk minimization (ERM) formulation of FL. However, alongside Hanzely & Richtárik (2020) argue, L2GD is better seen as algorithm for solving the personalized FL formulation equation By doing so, they interpret the nature of local steps in classical FL: the role of local steps in classical FL methods is to provide personalization and not communication efficiency as was widely believed- Feddvy can diverge on highly non-identical data partitions (McMahan et al.) 2017). Instead, communication efficiency in local methods comes from their tendency to gear towards personalization, and personalized models are provably easier to train.
To solve equation Hanzely & Richtárik 2020
Communication compression. We observe that the L2GD algorithm does not support any compression mechanism for the master-worker and worker-master communication that needs to happen- This is the starting point of our work. We believe that equipping personalized FL with fast and theoretically tractable communication compression mechanisms an important open problem.
In distributed training of deep neural network (DNN) models, synchronous data-parallelism (Dean al. 2012) is most widely used and adopted by mainstream deep learning toolkits (such as PyTorch, TensorFlow) However, exchanging the stochastic gradients in the network for aggregation creates communication bottleneck, and this results in slower training (Xu et al. 2021a). One way to save on communication costs is to use compression operators (Alistarh et al.] 2017 7 Horváth et al. 2019: Xu et al.
2021a) Gradient
compression techniques, such as quantization (Alistarh et al. 2017 Bernstein et al.
2018 Horváth et al.,
2019
Beznosikov et al. 22020] [Safaryan et al.) 2020), sparsification (uurehh et al. 2017
Konečný & Richtárik
2018
Aji & Heafield 2017 Sahu et al. 2021 Stich al. 2018 Dutta et al. 2020
Safaryan al. 2020
hybrid compressors (Strom 2015 Basu et al. 2019 and low-rank methods (ogels ct al. 2019 have been proposed to overcome this issue.
Although recent works have introduced compression in traditional FL formulation (Konečný et al. 2016 Reisizadch et al. 2020 Philippenko & Diculeveut] 2020 SSlezinggrr et al. 2020: Amiri ot al. 2020 Xu et al.
2021b); except (Horváth et al., 2019 Philippenko & Dieuleveut, [0200] Gorbunov el al.] 2020 Amiri et al. 2020), others use compression only for the throughput limited uplink channel, that is, to upload the local models from the devices to the central server. But limited bandwidth in the downlink channel may pose communication latency between the server and the devices and consequently, slow down the training: see detailed discussion in 2 As of now, no study combines bidirectional compression techniques with a probabilistic communication protocol in the FL set-up by using a mixture of a local and global model as in equation D In this work, we combine these aspects and make subsequent contributions.
Model compression Guo 2018 Chraibi
2019 is orthogonal to gradient compression and not the scope this work.
3

---

Published in Transactions on Machine Learning Research (11/2023)
1.2 Contributions
(1) L2GD algorithm with bidirectional compression. Communication compression prevalent in recent local FT, training algorithms, but these algorithms are not robust data and device heterogeneity. T,2GD by Hanzely Richtárik (2020) remedics this issue by introducing personalization in FL. However, integrating compression with the L2GD algorithm is a nontrivial task -ulike other FL algorithms, L2GD does not communicate after fixed local steps, it communicates based probabilistic protocol; see $ and Figure 1b Additionally, duc to this probabilistic protocol, the communication involves local model updates, as well as gradients; see S To reduce the communication bottleneck in L2GD, we use compression techniques on top of its probabilistic communication at both master and the participating local devices; see $ To the best of our knowledge, we are the first to integrate bidirectional compression techniques with a probabilistic communication the FL set-up, and we call our algorithm compressed L2GD; see Algorithm
(i) Convergence analysis. Tn $5 we prove the convergence of our compressed L2GD algorithm based on the most recent theoretical development, such as expected smoothness as in Gower et al. 2019 Admittedly, convergence analysis of first-order optimization algorithms with bidirectional compression exists in the literature, see (Tang et al. 2019 Horváth et al. 2019 Amiri et al. 2020 Dutta et al.
2020), integrating
arbitrary unbiased compressors with a probabilistic communication protocol into personalized FL, and showing convergence are nontrivial and the first one in its class. Our compressed L2GD algorithu maintains a similar asymptotic convergence rate as the baseline vanilla SGD without compression in both strongly
convex and smooth nonconvex cases; see Theorem 1 and 2 in $5
(wi) Optimal rate and communication. We optimized the complexity bounds of our algorithm a function of the parameters involved in the algorithm. This leads us to the optimal setting of our algorithm. Mainly, we derived the optimal expected number of local steps to get the optimal iteration complexity and communication rounds; see SG Although our analysis is based on some hard-to- co-mppute constants in real life, e.g., the Lipchitz constant, this may help the practitioners to get an insight into the iteration complexity and communication trade-off; see Theorem and in 6
(iv) Empirical study. We perform diverse numerical experiments on synthetic and real datasets by using both convex and non-convex problems (using 4 DNN models) and invoking various compression techniques; see details in $7 Table 1 In training larger DNN models, to obtain the same global Top-1 test accuracy, compressed L2GD reduces the communicated data volume (bits normalized by the number of local devices or clients, #bits/n), from 15t5 to 10"h, rendering approximately 10 times improvement compared to FedAvg; see $7.2 Moreover, L2GD with natural compressor (that by design has smaller variance) empirically behaves the best and converges approximately times faster, and reaches the best accuracy on both train and the test sets compared to no-compression FedOpt (Reddi al.] 2020) baseline; see 77.2 and AA2 These experiments validate the effect of the parameters used and, the effect of compressors, and show the efficiency of our algorithm in practice.
2 Related Work
Numerous studies are proposed to reduce communication but not all of them are in the FL setting. In this scope, for completeness, we quote a few representatives from each class of communication-elfliien SGD.
Smith et al. (2017) proposed a communicttion-eficieen primal-dual optimization that learns separate but related models for each participating device. FedAvg by McMahan et al. (2017) performs local steps subset of participating devices in an FL setting. Similar to FedAvg, but without data and device
heterogeneity, (Haddadpour et al.] 2019b: Stich 2019 Wang & Joshi 2019 Zhou & Cong 2018 Lin et al.
2020) independently proposed local SGD, where several local steps are laken on the participating devices before periodic communication and averaging the local models. While FedProx by Li et al. (2000) is a generalization of Fed,vg, SCAFFOLD uses variance reduction to correct local updates occurring from non-i.id data in FodAvg. From the system's perspective, on TensorFlow, Bonawitz et al. (2019) built a FL system on mobile devices.
4

---

Published in Transactions on Machine Learning Research (11/2023)
Compression has also been introduced in the FL setup. Shlezinger et al. 2020) combined universal vector quantization with FT, for throughput limited uplink channel. In FedPAQ by Reisizadeh et. al. (2020) each local device sends a compressed difference between its input and output model to the central server, after computing the local updates for a fixed number of itcrations. While Amiri et al.) (2020) used a bidirectional compression in FL set-up, Philippenko & Dieuleveul (2020) combined with a memory mechanism or error foedback (Stich et al.] 2018). For a unified analysis of compression in FL we refer to Haddadpour al. 2021
Among other proposed commmicatiinn-eficite SGDs, parallel restarted SGD Yu et al. 2019a reduces the number of communication rounds compared to the bascline SGD. Haddadpour et al. (2019a) showed that redundancy reduces residual error as compared with the baseline SGD where all nodes can sample from the complete data and this leads to lower communication overheads. CoCoA by (Jaggi et al.) 2014), and Dane by (Shamir et al. (2014) perform several local steps and hence fewer communication rounds before communicating with the other workers. Lazily aggregated gradient (LAG) algorithm by Chen et al.) (2018a) selects subgroup of workers and uses their gradients, instead of obtaining fresh gradient. from each worker in each iteration. For communcationn-efiieet local SGD see (Gao et al. 2021
In decentralized training, where the nodes only communicate with their neighbors, Koloskova et al. 2019 implemented an average consensus where the nodes can communicate to their neighbors via a fixed communication graph. Li et al. (2018) proposed Pipe-SGD a framework with decentralized pipelined training and balanced communication.
Personalizatton in FL. is growing research area. AAriazhaaan al. 2019) proposed FedPer to mitigate statistical hetcrogeneity of data; also, see adaptive personalized FL algorithm in (Deng ct al.
2020 Mei et al.
(2021) proposed to obtain personalization in FL by using layer-wise parameters, and two-stage training; also, see (Ma et al. 2022) and model personalization in (Shen et al.) 2022). Shamsian et al. (2021) trained a central hypernetwork model to generate a set of personalized models for the local devices. Li et al. 2021 proposed Hermes a communication--ffiiee
personalized FL, where each local device identifies a small subnetwork by applying the structured pruring, communicates these subnetworks to the server and the devices, the server performs the aggregation on only overlapped parameters across each subnetwork; also, see Pillutla al. (2022) for partial model personalization in FL. DispFL is another commuicattinn-effcian personalized FL a algorithm proposed by Dai el al. (022).. In recent work, Zhang et al. (2021) introduces personalization by calculating optimal weighted model combinations for each client without assuming any data distribution. (MAML), see (Fallah) et al. 2020 Additionally, we refer to the surveys (Kulkarni et al.] 2020 Tan et al. 2021 for an overview of
For a connection between personalization in FL and moodel-ggnosti--mmtt-eeeee
personalization FI.
3 Background and Preliminaries
For given vector, x E Rnn, by xa we denote the 7th subvector x, and write x = (sI, ...., x where x; E Idd. We denote the 7th component of x by T()) and ||ж|| represents its Euclidean norm. By [n] we denote the set of indexes, (1, nl. By E+(-) define the expectation over the randomness of g conditional to n).
Notation.
all the other potential random variables. The operator, C(.) (....) c.))) Rnnd + Rad denotes a compression operator with each (-)- being compatible with the size of x; Denote Q=
= 11 I, ,.....,]" I E IR nnddd where denotes the identity matrix of Rddd With our Assumptions that we will introduce later in the paper, the problem in equation has a unique solution, which we denote by x" and we define x² as x 1 E x By [S] we denote the cardinality of a set, S.
Loopless local gradient descent (L2GD). We give a brief overview of the loopless local gradient descent (L2GD) algorithm by Hanzely & Richtárik (020) to solve equation [ as a two sum problem. At each iteration, to estimate the gradient of F, T.2GD samples either the gradient of f or the gradient of h and updates the local models via:
x -GG((**), i= 1,...,,,
5

---

Published in Transactions on Machine Learning Research (11/2023)
where Gz(z*) for i= 1,.... n, is the tt block of the vector
V v((x")
with probability 1- p,
1-p
(Local gradient step)
G(*+))
vh(=")
with probability p. (Aggregation step)
where < < V=(x*) is the gradient of f at at, and v.h(a**) A (1 - -x*) is the 7th block of the gradient of h at zf.
In this approach, there is a hidden communcattin between the local devices because in aggregation steps they need the average of the local models. That is, the communication occurs when the devices switch from a local gradient step to an aggregation step. Note that there no need for communication between the local devices when they switch from an aggregation slep to a local gradient step. There is also no need for comnunication after two consecutive aggregation steps since the average of the local models does not change in this case. If kt and i + are both aggregation steps, we have _b-1 Ei 24 R++
- ad
aan Ei=1 (a F²)) =²
4 Compressed L2GD
Now; we are all set describe the compressed L2GD algorithm for solving compression operates in our set-up.
1 We start by defining how the
4.1 Compressed communication
Recall that the original L2GD algorithm has a probabilistic communication protocol the devices do not communicate after every fixed number of local steps. The communication occurs when the devices switch from a local gradient step to an aggregation. Therefore, instead of using the compressors a fixed time stamp (after every > 0 iterations, say), each device requires compress local model x; when neods to communicate it to the master, based on the probabilistic protocol. We assume that device uses the compression operator, C.()) : Rd + Rd. Moreover, another compression happens when the master needs to communicate with the devices. We assume that the master uses the compression operator, Cm(()) Rd Rd Therefore, the compression is used in uplink and downlink channels similar to Dutta et al. (2020): Horváth et al. (2019), but occurs in a probabilistic fashion. There exists another subtlety although the model parameters (either from the local devices or the global aggregated model) are communicated in the network for training the FL model via compressed L2GD, the compressors that we use in this work are the compressors used for gradient compression in distributed DNN training; see (Xu al. 2021a).
4.2 The algorithm
Note that, in each iteration 1: 2 0, there exists a random variable, & E 10 with P(Lx = 1)= and P(E: = 0) =1 p. If & = 0, all local devices at iteration k perform one local gradient step. Otherwise (if & = 1), all local devices perform an aggregation step. However, to perform an aggregation step, the local devices need to know the average of the local models. If the previous iteration (i.e., k - 1 iteration) was the same average
aggregation step (i.c., ε 1) then at the current iteration the local devices the one at iteration k - (recall, the average of the local models does not change after two consecutive aggregation steps). Otherwise, communication happens with the master to compute the average. In this case, each local device compresses its local model of to C,(2)) and communicates the result to the master. The master computes the average based on the compressed values of local models:
J サー n C,
then it compresses g² CMr (g²) by using compression operator at the master's end and communicates it back to the local devices. The local devices further perform aggregation step by using Cx instead of

---

Published in Transactions on Machine Learning Research (11/2023)
the exact average. This process continues until convergence. From Algorithm 1
have, for i == .....,
1 = 2 - -n)G((**
k+I
where
(=((
if fΙ 0
RI p)
G((x*)=
x
Cm (T*))
&l & _k = 0, if &N & &k-1 1.
A
np
We give the pseudo code in Algorithm
A Algorithm Compressed L2GD)
Input: 10011...... stepsize > 0, probability p. {-1 = 1, x 1 +2= 2
for = 0, 1, 2,,...... do
Draw: & with probability p
& = 0 then
on all devices: 2; r at m[I-p) Vf(()t) for i [n]
else
if fk-1= 0 then
all devices: Compress of Cx(x)) and communicate (x to the master
master: receive (xxx)) from the device i, for all [n] compute 8² = C
compress ÿ² to (T²) and communicate it to all devices
on all devices: Perform aggregation slep x; at
nd
(1 Car Car(r²))
up
else
all devices: Æ = zk-1, Perform aggregation step oit1 = xt end if
nn (1 B*)
up
end if end for
The initialization, E-1 not important, as it does not impact the Algorithm. In Algorithm 1 we start the for loop (for k ===0, 0, l....) by drawing &k- ileration k, we perform an aggregation step if &a 1. That is why to be consistent in the algorithm, initialized & at iteration 5.1" by 1 and initialized x-1 by the average of the initial models, x.-
Remark
All local devices have access to the same value of &. We assumed that is drawn at random at the server side and broadcasted to the local devices.
Remark 2
5
Convergence Analysis
With the above setup, we now prove the convergence of Algorithm ! see detailed proofs in $
5.1 Assumptions
We make the following general assumptions this paper.
Assumption For 1,...,
The compression operator, C,()) Rd + Rd is unbiased,
Ec, [C(x)) = x,
Yo Rd
There cxists constant, W; such that the variance of C2 bounded follows:
Ee, [1C,(x) x 2;|p] will [a,l|l,,te Var e Rd..

---

Published in Transactions on Machine Learning Research (11/2023)
The operators, (C:()== are independent from each other, and independent from Ex, for all k 2 0. The compression operator, C-r(-) unbiased, independent from {c,)i=1 and has compression factor, Wм.
From the above assumption we conclude that for all x E R we have
Ec, [[[[(())]]] < (1 + wx)|[=
The following lemma characterizes the compression factor, of the joint compression operator, c(.) ((,))),...,C((()))) as function of wi...,,m..
Lemma 1 Let x E Rnd, then
Ec [|[(())] s
where w = max
For the convergence of strongly convex functions, we require an additional assumption on the function f as follows.
Assumption assume that f Ly-smooth and u-strongly convex.
5.2 Auxiliary results
Before we state our main convergence theorem, we state several intermediate results nceded for the convergence. In the following two lemmas, we show that based on the randomness of the compression operators, expectation, we recover the exact average of the local models and the exact gradients for all iterations.
Lemma 2 Let Assumption hold, then for all l 2 0, Ec,c. [CM((T .k
Lemma 3 Let Assumplions hold. Then for all k 2 0, knowing wt, G(**) is an unbiased estimalor of the
gradient of function F at x
Our next lemma gives an upper bound on the iterate at each itcration. This bound is composed of two terms the optimality gap, F(x²) F(x"), and the norm at the optimal point, x².
Lemma 4 Let Assumption hold, then
4
+ 1
1
Lemma 5 helps to prove the expected smoothness property (Gower et al. 2019 The bound in Lemma 5 is composed of the optimality gap, the difference between the gradients of h at x* and x", and an extra constant, B that depends on the used compressors.
Lemma 5 Let Assumptions
and 2 hold. Then
== &er.ch I|a - (T - x² QC < In Vh(**) - vh(x*)]1² (F(x*) - F(z*)) + B,
where V' = 1 E C( a
4(4w+)ws((L+w)) and 3 = 2(w + 4w (1 + w)) ||
1
Al [|QCas(T*) -
Lemma 6 is the final result that (together with Lemma 5 shows the expected smoothness property and gives an upper bound on the stochastic gradient. This bound composod of three terms-the optimality gap, F(**) F(=*), the expected norm of the stochastic gradient at the optimal point, E]|G(=*)[[3 and some other quantity that involves interplay between the parameters used in Algorithm and the used compressor.
8

---

Published in Transactions on Machine Learning Research (11/2023)
0.650
f(p), train
f(A), train
0.60
0.625
f(p), test
f(A), test
0.600
0.58
0.575
0.56
0.550
0.54
0.2
0.4
0.8
20
40
100
P
(a) aia
(b) aia
0.650
f(p), train
f(A), train
0.60
0.625
f(p), test
[(A), lest
0.600
0.58
0.575
0.56
0.550
0.2
0.6
20
40
100
p
(c) a2a
(d) a2a
Figure 2: Uncompressed L2GD 5 workers for fi(æ) to be local empirical risk minimization for logistic regression loss with la regularization for local data D.. We show the loss, I as a function of p and À obtained after K 100 iterations of Algorithm 1 with C identity compressor (no compression). (a) ala dataset, = 124, À = 10, (b) aia dataset, d = 124, = 0.65, (c) a2a dataset, = 124, À 10, (d) a2a dataset, 124, p 0.65.
Lemma 6 (Expected Smoothness) Let Assumplions 7 and hold, then
E [[GG(>))[[1]]] < 47 (F(**) - F F(x*)) + 6,
(2)
where
a1-(1-p) max 2n-p
1/ [I-p]'
(1 1(1-21
7 ==
and
23 -)) + "(x")(("")""""""
n2p
Remark 3 If there is no compression, the operators, Ci(+), for E [n], and Cm(-) are equal to identily. The compression constants, Wa, for i [m], and wM are equal to zero. Therefore, a = 3 O, and the factor in the formula of y can be replaced by and thus
=
(1P nap
(= 2 2E|[G(²*)|1+
with
7 max I1 2 (T-p)) n (1+ max [n(I-p)
up
where L = nLy. Same constants arise in the expected smoothness property in Hanzely & Richtárik 2020
For nonconvex convergence of our algorithm, we cannot use the expected smoothness of Lemma as it requires p-strong convexity the loss function. Therefore, similar to (Sahu et al. 2021 Stich & Karimireddy 2020 we require a new assumption to bound the stochastic gradient, G(x*). Let G(x²) of the form: G(* = F(**) + Ck, where ČK is the stochastic noise on the gradient.
Assumption 3 There exist M, 2 2 0, such that E[[[G(>*>) b+ < MI 2
9

---

Published in Transactions on Machine Learning Research (11/2023)
0.675
f(p), train
f(x), train
0.650
f(p),te test
0.65
f(),, test
0.625
0.60
0.600
0.575
0.55
0.2
0.8
1.0
20
40
100
X
(a) aia
(b) aia
0.68
1000000000000
0.66
f(p), train
0.675
f()), train
S(p),ess Lest
0.650
[(A), test
0.64
0.625
0.62
0.600
0.60
0.575
0.58
0.2
0.8 1.0
40
80
100
(c) a2a
(d) a2a
Figure Compressed L2GD logistic regression loss with & regularization for local data D.. We show the loss, I as a function of the workers
= workers for fi(2)) to be local empirical risk minimization for
and obtained after == 100 iterations of Algorithm [ with with natural compressor and identity compressor (no compression) at the master. (a) ala dataset, d = 124, À = 10, (b) ala dataset, d 124, 0.65, (c) a2a dataset, d 124, À 10, (d) a2a dataset, d= 124, 0.65.
5.3
Main result
We now state the convergence result for Algorithm ] for both strongly convex and nonconvex cases.
Thcorem (Strongly convex case) Let Assumptions and 2 hold. If 24 then
E B||a*
21- ) 1|.00 1=1 nn8
Proof 1 The proof follows directly from Lemma 6 and Theorem 3.1 from Gower et al. (2019).
Theorem (Non conver case) Let Assumptions and hold. Assume also that F Ly-smooth, bounded from below by F(x*). Then to reach a precision, e 0, set the stepsize, min 1 4L, ((F()) F(w")) we have min E VF S..
Ty M 2T. }. such that for
Remark 4 For smooth non-conver problems, we recover the optimal OEea) classical rate as vanilla SGD.
6 Optimal Rate and Communication
In this section, we provide the "optimal' setting of our algorithm that obtained by optimizing the complexity bounds of our algorithm function of the parameters involved. The analysis this section is based
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1 Gradient compression methods used in this work. Note that llãloo and lloilo are the number of elements
in the compressed and uncompressed gradient, respectively; nature of operator C is random deterministic. We implement that mechanisms for FedML.ai framework.
Compression
Ref.
Similar Methods
I@ālo
Nature of C
Horváth
(0199:
Wang al. (018 Wen el.
2017
QSGD
Alistarh et al.
2017
l
Rand, unbiased
al. 2018
20196
Zhang
2017
Natural
Horväth et al.
2019
Alistarh al.
(2017):
Yu et al. (2019b) Zhang et al 2017
Illoo
Rand, unbiased
TernGrad
Wen et al. 2017
Alistarh et al,
(2017:
Wang et al.
2018
Yu et al. 2019b
lllo
Rand, unbiased
Bernoulli
Khirirat al.
2018)
Rand, unblased
Top--
Aji Heafield
(017
Alistarh
2018
Stich al. 20188
k
Det, Biased
the following upper bound of y. We recall that
a1²(1-2) -
Ly
A
1 4(1 - p)
7
2n²p
+ max
(1 p) n p
ax²(1 p)
4X
+ max
Yu-
2n²p
(1 p) np
Note that the number of iterations linearly dependent on y. Therefore, to minimize the total number of iterations, it suffices to minimize 7. Define L := nLy.
Theorem 3 (Optimal rate) The probability 8' minimizing is equal to max PA}, where a>² + m[1-p] in (0,
7A+ T +12 and PA the optimizer of the function A(p) 2n'p
6λ
Remark 5 If we marimize the upper bound, Tue instead of then Pe simplifies to T.44X
14
Lemma 7 The optimizer probability PA of the function A(p) = 2n²p n(I
xd2
L. in (0,1) is equal to
2nL ax²
2
222
2f ax² otherwise.
2(2nL
caa
2 2Av22vnl
2(2nL
Note that the number of communication rounds is linearly proportional to C = p(1 p)7- Therefore, minimizing the total number of communication rounds suffices to minimize C' or nC.
Theorem (Optimal communication) The probability p" oplimizing C is equal to max max PA), where
Pe TA+L-VXe 1A-L-VAP+14x--- and Pa
Im ax
6A
Remark 6 As in Remark 5 we note that, if we use the upper bound, Tu instead of n then Pe simplifies to
L+4
We note that A implies p* + This means that the optimal strategy, in this case, no communication at all This result is intuitive since for X deal with pure local models which can be computed without any communication. As X implies p + denoting that the optimal strategy is to communicate often to find the global model.
7 Empirical Study
We conducted diverse numerical experiments with L2GD algorithm that includes: (1) Analysis of algorithm meta-pares peta-parameters (with and without compression) for logistic regression in strongly convex setting; see
(i) analysis of compressed T.2DD algorithm on image classification with DNNs; see 7.2
11

---

Published in Transactions on Machine Learning Research (11/2023)
(2 U.1, Minbeed
PB0AM 01, 1.10))
PB 0.1)
120000 2.0, 9.80, Natsrell
FEIDAVGu0 01, Maturai)
PRD:4HGud 0.1 TayAAe 0n5og
(e 20, 0.5, pm Bemoalije=
LaGDala 0.255 p 0.3. QSGD) -
0.8
Accural)
3155
3:..5
0.5
2530
7903 12500 15300
5303
10000 Rounds
15003
1.0
1.
#bits/n
16l6
1652
(a)
(b)
(d)
0.0
Acquraty
0.6
Accuracy
100
31.5
0.4
Rounds
#bitssn
Rounds
#bitshh
3620
(0)
(1)
(h)
Figure Training ResNet-18 on CIFAR-10 with n 10 workers. The top row represents the Top-1 accuracy vs. rounds in (a), loss functional value vs. communicated bits in (b), loss functional value vs. rounds in (c); and Top-1 accuracy vs. communicated bits in (d) on the train set. The bottom row presents the similar plots on the Test set in (e)-(h).
PBDAAWa0g-011. 0.1,
FBGAAW)lg +01 Topoopt(k 12000 2.0, A 0.55 TopAT (.00))
12G00el 2.0 0.0.5pp00 Natwol! 125Dsig-20 2:03:-0--0.111-- Q8CD)=B)
11108M0oy-01111ppp 0100))
FECOPT
13C7Daly 0.55 0.3, Berswalill 0.55)
0.8
Accuracy
Accurssc
2.5
0.2
0.5%
Rounds
#blts/n
1A15
Rounds
#bissin
1415
(b)
(c)
(d)
2.5
0.6
Accurazy
0.6
2002 4000 esto. 2002 10002 32500
2502 4200 8000
8930 Rounds
20000
12050
Rounds
bisin
#bitsih
(0)
(f)
(g)
(h)
Figure Training DenseNet-121 CIFAR-10 with 10 workers. The top row represents the Top-1 accuracy vs. rounds in (a), loss functional value vs. communicated bits in (b), loss functional value vs. rounds in (c), and Top-1 accuracy vs. communicated bits in (d) on the train set. The bottom row presents the similar quantities on the Test set in (h).
Computing environment. We performed experiments on server-grade machines rumning Ubuntu 18.04 and Linux Kernel v5.4.0, equipped with 8-cores 3.3 GHz Intel Xeon and a single NVIDIA GeForce RTX 2080 Ti. Tesla--V10--XMM GPU with 32GB of GPU memory. The computation backend for Logistics Regression experiments was NumPy library with leveraging MP14PY for inter-node communication. For DNNs we used recent version of FedML He et al. (2020) benchmark and patched it with: (f) distributed and standalone version of Algorithm 1 (8) serializing and plotting mechanism; (Mi) modifications in standalone, distributed version of FedAvg McMahan et al. (2017) and FedOpt Reddi et al. (2020) to be consistent with equation 1 (iv) not to drop the last batch while processing the dataset.
12

---

Published in Transactions on Machine Learning Research (11/2023)
1o ywi'
Sartiute Super
2000//1 Layport
0.8
0.3
Accurang
0.6
Accura Aogura
0.6
9.4
0.2
3006
10000 13660
20302
03
20
30000
13000
23026
Hounds
#bity/n
Hounds
*bityn
(a)
(b)
(c)
(d)
0.3
Accural
2.5
Accuracy 1
0.4
0.2
5200
30000
20003
01
10000
35383
lels
Rounds
#bits/n
1854
(0)
(f)
(g)
(h)
Figure 6: Training MobileNee on CIFAR-10 with n = 10 workers. The top row represents the Top-1 accuracy vs. rounds in (a), loss functional value vs. communicated bits (b), loss functional value vs. rounds in (c), and Top-1 accuracy vs. communicated bits in (d) on the train set. The bottom row presents the similar plots on the Test set in (e)-(h).
7.1 Meta- peta-parameter study
The purpose of these experiments to study the meta-past pastapeasseeerss involved in uncompressed and compressed L2GD algorithm. We used L2GD algorithm with and without compression for solving regularized logistic regression on LIBSVM ala and a2a datasets Chang & Lin (011)) Both datasets contain shuflled examples in the train set, and we did not perform any extra shuffling. To simulate the FL. settings, we divided both datasets into 5 parts. After splitting, each worker has 321 and 453 records for ala, and a2a, respectively.
Sctup and results. We define fi(z) to be local empirical risk minimization for logistic loss with additive regularization term for local data D; and of the form:
n
log(1 + exp(- TT d())) + L2 2 ||
f(()) = n2d j=1
where al) E 122 ((2) E {+1,-1} = Di). We set L2 = 1, and varied meta meta-parameters p and X. For each parameter, we performed 100 iterations of Algorithm ! Note that, as meta--araaeter À decreases the models will fit more its local data, while p provides stochastic balance between local gradicnt stops with probability, 1 1-p p and aggregation with probability, p. For compressed L2GD, we used natural compressor al. the workers and identity compressor at the master.
Takeaway message. The results in Figure support the theoretical finding there exists an optimal choice of (p, A); where the loss function, f achieves the least value. Nevertheless, this choice problem dependent. Additionally, we find small p is not good due to lack of samples in single node compared to samples available at other nodes. There is trade-off for each node in learning from the other nodes' data and spending time to learn from its own data. For uncompressed L,2GD, the optimal setting of our algorithm is attained for p p= 0.4 and À in [0, 25). The same obscrvations hold for compressed L2GD; see Figure 3 For compressed L2GD with natural compressor, the optimal setting of our algorithm is attained for = 0.8 and in [0, 5). Finally, we observe that to get the smallest errors on the training and validation sets, it is better not to perform the averaging step too often.
7.2 Training DNN models
We choose four practically important DNN models used for image classification, and other down-streaming tasks, such as feature extractions for image segmentation, object detection, image embedding, image captioning,
to name few.
13

---

Published in Transactions on Machine Learning Research (11/2023)
ResNet (He et al. 2016). The overwhelmingly popular ResNet architecture exploits residual connections to remedy vanishing gradients. The network supported the trend toward smaller filters and deeper architectures, more curated towards FL training. We use ResNet-18 and ResNet-56 architectures (He et al. 2016).
DenseNet (Huang et al., 2017 contains short connection between layers via connecting each layer to every other layer in a leed-forward fashion. Dense connection allows propagating information to the final classifier via concatenating all feature maps. Each layer in DenseNet is narrow and contains only 12 filters another practical model for FL training.
MobileNet (Howard et al.] 2017 DNN architecture has a trade off between computational complexity and accuracy (Bianco et al. 2018 p.3, Fig.1). For mobile devices that appear in cross-device FL, the computation cost and energy consumption are both important. The energy consumption is mostly driven by memory movement (Chen et al. 2018b Horowitz 2014). In MobileNet architecture standard convolution blocks performs depth-wise convolution followed by 1 convolution. This is computationally less expensive in flops during inference time (see (Bianco et al. 2018 Fig.1, p.3)) and is ~ 3.5x more power efficient compare to DenseNet (Garcia-Martin et al. 2019 p.85, Table 7).
This makes Mobileliet an attractive model for FL training.
Datasct and sctup. We consider CIFAR-10 dataset (Krizhvssky & Hinton 2009 for image classification. It contains color images of resolution 28 x 28 from 10 classes. The training and the test set are of size, 5 10a and 104, respectively. The training set is partitioned heterogeneously across 10 clients. The proportion of samples of each class stored at each local node is drawn by using the Dirichlet distribution (a 0.5). In our experiments, all clients are involved in each communication round. Additionally, we added a linear head in all CNN models for CIFAR-10, as they are originally designed for classification task with 1000 output classes.
Loss function. Denote f((x) = w; T ElanbeeD, Ma b, x)) to be weighted local empirical risk associated with the local data, Dj stored in node, i. We note that (a,, b,, x) is a standard unweighted cross-entropy loss, a, RR 28x3 b, E {0, 1) 10 with only one component equal 1, the ground truth value, and the weight is set to w. 1 U U...UU... Dzl.
Metrics. To measure the performance, we examine the loss function value, f'xx), and the Top-1 accuracy on both train and the test set. We use the weighted average the local models, where the weight, wy for each client defined above. In our experiments, we wanted to assess the efficiency of both models the local models and the global model (or the average model), 1 n Σ x4. To do this, we used two metrics:
Loss. We compute the average loss over all the losses of the local models, that is, (",,,".,,, Ei(r)) This allows us to assess the efficiency of the local models.
Accuracy. We compute the accuracy of the average model, that is, we compute the accuracy using the model. As the experiments demonstrate both local models and the average model perform well.
We use stale-of-the-art FedML benchmarking for our experiments, and it does not support personalization. Due to this limitation, although wanted to present the average accuracy over all the accuracies of the local models, we were unable to do so changing the FedML benchmarking for personalization is involved task. Neverthcless, we compared with compressed commmication approaches in the FedML framework that supports a single global model to see how despite personalization, our compressed L2GD performs. Additionally, we measure the number of rounds, and bits/n communicated bits normalized by the mumber of local clients, n. The intuition behind using the last metric is to measure the communicated data-volume; it is widely hypothesized that the reduced data-volume translates to faster training in a constant speed network in distributed setup (Gajjala et al. 2020 Xu et al. 2021a).
Compressors used. The theoretical results of compressed L2GD are attributed to unbiased compressors. We used 4 different unbiasod compressors at the clients: Bernoulli (Khirirat et al., 20018), natural compressor Horvááth al. 2211),, random dithering a.k.a. QSGD (Alistarh al. | (017), and Terngrad (Wen al. 2017):
see Table 1 for details. Additionally, we note that. biased compressors (mostly sparsifiers) are popular in DNN
14

---

Published in Transactions on Machine Learning Research (11/2023)
2.4
Model
Training parameters
L2GD bits/n
Baseline bits//n
0.22
2.3
Accuracy
2.2
Lasiks a B00 derwedy
DenseNet-121
79 10a
10
4 4-1015
0.18
2.2
MobileNet
32 10
1.7 x 10t
1 1046
RosNet-18
11 x 10
1.1 x 10
1.5 x 10t6
0.14
2.1
21
0.1
Table 2: Summary of the benchmarks. The measured quantity is bits/n to achieve 0.7 Top-1 test accuracy, with 10 clients. For DenseNet-121, MobileNet, Resnet-18 the baseline Aed with natural compressor with local epoch; L2GD also uses natural compressor.
#bits/n
1:9
# bits//
Figure FedAvg as particular case L2GD; Test and train accuracy for ResNet-56 on CIFAR-10.
Figure 8: FedAvg particular case L2GD: Test and train loss for ResNet-56 on CIFAR-10.
training. Therefore, out of scientific curiosity, we used popular sparsifier: Top-k (Aji & Heafield 2017 Sahu et al., 2021) a proof of concept. We note that extending the compressed L2GD theory for biased compressors (with or without crror-feedback (Xu et al. 2021a)) is nontrivial and mathematically involved, and left for future work.
Algorithms used for comparison. We used state-of-the-art FL training algorithms, FodAvg (McMahan et al., 2017) and FedOpt (Reddi al., 2020) as no compression baseline compare against our L2GD. However, the performance of FedAvg is not stable but improves with the compression mechanism. The original FedAvg algorithun does not contain any compression mechanism, but for comparison, we incorporated compressors into FedAvg via the following schema which is similar to the classic error feedback (Xu et al., 2021a): (i) After local steps, client estimates change of current iterate from the previous round and formulates direction, Ic.computed': (iz) client sends compressed difference between previous gradient estimator from previous round and currently computed gradient estimator, C(gee.ompatedd ' -- "1 to the master; (Mi) both master and client updating ge via the following schema: ge = ge (-1 C(gc.computed go the details about step size and batch size in Appendix.
-eoe--9---- 11) We provide
7.2.1 Results
We show the results for training ResNet-18, 18, DenseNet-121, and MobileNet with compressed L22G and other state-of-the-art FL algorithms in Figure 4 6 For these experiments, the communication rounds are set to 12 x 103, 25 x 103, and 20 x 10a, respectively. For the FedAvg algorithm, each client performs one epoch over the local data. We empirically tried 1, 2, 3, and 4 epochs over the local data as local steps, but one epoch is cmpirically the best choice.
For training ResNet-18, from Figure 4 we observe that FedAvg with compression has albeit better convergence than no compression FedAvg Fl At the same time, compressed FedAvg affects the convergence as a function of communicated rounds only negligibly (sce Figure | (b). Thorefore, for training other DNN models we use FedAvg with compression and FedOpt without. any compressors enjoy the best of both baselines.
Take away message. Compressed L2GD with natural compressor sends the least data and drives the loss down the most in these experiments. At the same time, L2GD with natural compressor (by design it has smaller variance) reaches the best accuracy for both train and test sets. Compressed L.2G outperforms FedAvg by huge margin For all DNN experiments, to reach the desired Top-1 test accuracy, compressed L2GD reduces the communicated data-volume, # #bits/n, from 10 to 10 rendering approximately 10a times improvement compared to FedAvg; see Table 2
5 We have observed that batch normalization Toffe & Szegedy 2013 in ResNet sensitive aggregation; see discussion
15

---

Published in Transactions on Machine Learning Research (11/2023)
Interestingly, in training MobileNet, the performance of biased Top-k compressor degrades only about 10% compared to natural compressor, while approximately degrades 35% in training DenseNet. Additionally, see discussion in 4A.2 Figures 9-11. This phenomena may lead the researchers to design unbiased compressors with smaller variance to empirically harvest the best behavior of compressed L2GD in personalized FL training.
Nevertheless, we also observe that compressed L2GD converges slower compared to other FL algorithms without compression in all cases. What follows, it can be argued, is that when compare the communicated data volume for all DNN models, the convergence of compressed L2GD is much better. Additionally, the gain in terms of lowering the loss function value is significant-by by sending the same amount of data, L2GD lowers the loss the most compared to the other no-compression FL baseline algorithms. These experiments also demonstrate that when communication is a bottleneck, FedAvg is not comparable with L2GD, The only comparable baseline for L2GD is FedOpt; see Table II, also, see discussion in A. Figures 9-11. A similar obscrvation holds for the Top-1 test and train accuracy. Taken together, these indicate that for training larger DNN models in a personalized FL settings, with resource constrained and geographically remote devices, compressed L2GD could be the preferred algorithm because its probabilistic communication protocol sends less data but obtains better test accuracy than no compression FedAvg and FedOpt.
Additionally, we observe that when nd E [0.5, 0.95], compressed L2GD incurs significant variance in objective function during training. Empirically, the best behavior was observed for nd
= 1 or
nd E (0, 00.7.
np
Aed as a particular case of L2GD. We note that if nA/mp = 1, then the aggregation step of Algorithm reduces to x; + I², for all devices. Thus, in this regime L2GD works similarly as FedAvg with random number of local steps. E.g..if p 0.5, then Algorithm [ reduccs to randomized version of FedAvg with an average of 3 local steps. Figures 7 and S confirm this observation mumerically, where
that both
algorithms exhibit similar performance. For that experiment, we trained ResNet-56 on CIFAR10 with n = 100 workers, and for 600 rounds. For L2GD, we set IA
=1
pm
8 Conclusion and Future Direction
In this paper, we equipped the loopless gradient descent (L2GD) algorithm with compression mechanism to reduce the communication bottleneck between local devices and the server in an FL context. We showed that the new algorithm enjoys similar convergence properties as the uncompressed L2GD with a natural increase in the stochastic gradient variance due compression. This phenomenon similar classical convergence bounds for compressed SGD algorithms. We also show that in a personalized FL setting, there a trade-off that must be considered by devices between learning from other devices' data and spending time learning from their own data. However, particular parameterization of our algorithı recovers the well-known FedAvg Algorithm. We assessed the performance of the new algorithm compared to the state-of-the-art and validated our theoretical insights through a large set of experiments.
Several questions remain open and merit further investigation in the future. For example, we plan on including compression when devices calculate their local updates, especially in an FL setting, as the devices might not be powerful, and the computing energy is limited, and examine how the algorithm behaves. Additionally, we observed the efficacy of compressed L2GD with biased compressor, such as Top-k. Nevertheless, extending the compressed L22D theory for biased compressors (with or without error-feedback (Xu et al. 2021a)) is nontrivial and challenging. In the future, we plan to prove a more general theory for compressed L2GD that include both biased and unbiased compressor operating in a bidirectional fashion. A more detailed meta- parameter study covering different network bandwidths, diverse ML tasks with different DNN architectures, and deploying the models on real-life, geographically remote servers will be our future empirical quest.
Acknowledgments
Aritra Dutta acknowledges being an affiliated rescarcher at the Pioncer Centre for AI, Denmark. The authors acknowledge many fruitful discussions with Md. Patel on this project while he was a remote undergraduate intern at. KAUST.
16

---

Published in Transactions on Machine Learning Research (11/2023)
References
Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017 URL https
pp. 440 445. Association for Computational Linguistics, 2017. doi: 7/doi otg/1o..g11... 10. 18653/v1/617-1045
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: communicttion-efficenn SGD via gradient quantization and encoding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Arnual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1709 1720, 2017. URL https::/prroceedingg ietireeeedinnn.n.epppsp neurips html
Dan Alistarh, Torsten Hoefter, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (cds.), Advances in Newral Information Processing Systems 31: Annual Conference on Newral Informalion Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp.
2018. URL
Abstract.
Mohammad Mohammadi Amiri, Deniz Gündüz, Sanjeev R. Kulkarni, and H. Vincent Poor. Federated learning with quantized global model updates. CoRR, abs/2006.10677 2020. URL https://aaxxii:i http:::://xxxvor//
0bs/2006.1665
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, abs/ 0bs/1912.00816 2019. URL http::/arxxw.org/abbs// 00818
Debraj Basu, Deepesh Data, Can Karakus, and Suhas N. Diggavi. Qsparse-locallsgd: Distributed SGD with quantization, sparsification and local computations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14668 14679, 2019. URL https; //proceedings ceeurips.cc/papppe httpp//////p/p.aaaaaaaaaaaaaaaaa att.::/aaaae.attaattatttttt....a
Jeremy Bernstein, Yu-Xiang Wang, Kanyar Arizzadenesheli, and Animashree Anandkumar. SIGNSGD: compressed optimisation for non-convex problems. In Jemnifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, TCML, 2018, Slookholmsmüsseen, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 559 568. PMLR, 2018. URL http::t///ooceeeiiiggg/////////e mml.iiinnnmm........mmmm...
Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. CoRR, abs/2002.122410, 2020. URL
Simone Bianco, Rémi Cadène, Luigi Celona, and Paolo Napoletano. Benchmark analysis of representative deep neural network architectures. IEEE Access, 6:64270-64277, 2018. doi: /ttp:/AACESSSS21.2877777 URL http::://doo...r////// 1109/ACCESS. 2018. 2877890
Kallista A. Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In Amoct Talwalkar, Virginia Smith, and Matei Zaharia (eds.), Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 April 2, 2019. mlsys.org. 2019. URL httpss//proocedinng..t http:/eeedd.......lsss.rrrr///ea https///sssss.ooooo//////////// org/book//11..ptt
17

---

Published in Transactions on Machine Learning Research (11/2023)
Chih- Chih-Chung& Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. (0:):27:122:221 2011. doi: 10.1145/1961 19. 19.19911999 9961189.196119
URL httpps////// http///do..rgg////// org/10. 1145/
Tianyi Chen, Georgios B. Giannakis, Tao Sun, and Wotao Yin. LAG: lazily aggregated gradient for communicationn-lliiieet distributed learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biancle and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Newral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 5055-5065, 2018a. URL htttps://proceedingss wwwp:/ewddinnn......s........... cc/paper/
https///eeeee7707773//-tttttt..a html
Y Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sae. Understanding the limilations of existing energy-eflicient design approaches for deep neural networks. Energy, 2(T.1):L33, 2018b. URL http::////poopleeslll... heth//enet//media//ppper////////
S6lim Chraibi, Ahmed Khaled, Dmitry Kovalev, Peter Richtárik, Adil Salim, and Martin Takác. Distributed fixed point methods with compressed iterates. CoRR, abs/ abs/1912.09925, 2019. URL http://arxiv.org
abs 1bbs/191.0099 09925
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication--ffiiiee personalized federated learning via decentralized sparse training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (cds.), ) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Ballimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 4587 4604. PMLR, 2022. URL htt.ttts://pocceddiiggeeo mlr. www.m.rrreess..e html
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoe V. Le, Mark Z. Mao, Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25: 26th Anmual Conference on Newral Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 1232 1240 2012. URL htttps://proceedingg hwt.iins...n.ipp.pppppprrrr 001/hhaa//accc777666666222228882 obt. html
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. CoRR, abs/2003. 13461, 2020. URL pestps://arxxr wwwpr/iiv.oo//bb/////eoa/ 13461
Aritra Dutta, El Houcine Bergou, Ahmed M. Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis. On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IA 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3817 3824. AAAI Press, 2020. doi: 10. 10.1609/ V341045554 URL https://doi. org/ 10. 1609/ aaai , v34404..599
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Jeuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 39: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL httpss:/prroceeding. www.n/ps..c/aaapee//eeeaa.t. Abstract html
Rishikesh R. Gajjala, Shashwat Banchhor, Ahundd M. Abdelmoniem, Aritra Dutta, Marco Canini, and Panos Kalnis. Huffman coding based encoding techniques for fast distributed deep learning. In DnstributedM&&CoNEEEE 2020: Proceedings of the 1st Workshop on Distributed Machine Learning, Barcelono, Spain, December 1, 2020, pp. 21-27. ACM, 2020. doi: (0.1445/44664453448333 URL
18

---

Published in Transactions on Machine Learning Research (11/2023)
Hongchang Gao, An Xu, and Heng Huang. On the convergence of communication-effiiieee local SGD for federated learning. In Thirty-Fifth AAAT Conference Artificial Intelligence, A AATT 2021, Thirly-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 7510 7518. AAAT Press, 2021. doi: N0.1609/AAALV3550 16920. URL https://doi. org/10. 1609/ aaai v3519. V6929
Eva Garcfa-Martín, Crefeda Faviola Rodrigues, Graham D. Riley, and Hákan Grahn. Estimation of energy consumption in machine learning. J. Parallel Distributed Comput. 134:75 88, 2019. doi: 10.1016/J.JPDC.
2019.07.007. URL https:///oi. http::://do..org////// org/10 1016/ jpdc.. 201..07....... 007
Eduard Gorbunov, Dmitry Kovalev, Dmitry Makarenko, and Peter Richtárik. Linearly converging error compensated SGD. In Hugo Larochelle, Marc'Aurelee Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anrual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL 1E220010BB3777770884444111177171 hwar:51--bbtrtttt.cccc html
http://roceeding.nneu.rpp wc/paper/e/000/ saneer//220//aaas
Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local SGD: unified theory and new efficient methods. In Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of
Muchine Learning Research, pp. 3556 3564. PMLR, 2021. URL http://prooceedings mlr press/v130/ gorbunov2la.
Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: general analysis and improved rates. CoRR, abs/1901.09400. 2019. URL http:://axiii org/abs/ 1901.09401
Yunhui Guo. A survey on methods and theories of quantized neural networks. CoRR, abs/1808.0477 2018. URL
1808. 04752
Farzin Haddadpour, Mohammad Mahdi Kamani, Mchrdad Mahdavi, and Viveck R. Cadambe. Trading redundancy for communication: Speeding up distributed SGD for non-convex optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference
Machine Learning, TCML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2545-2554. PMLR, 2019a. URL http:: mlr. wtwpiiiis.........ss.eeemmmm press/v97/ haddadpour 19a. html
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck R. Cadambe. Local SGD with periodic averaging: Tighter analysis and adaptive synchronization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' D'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances én Newral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11080 1 11092, 2019b. URL hhttps
Abstract.
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Evend, volume 130 of Proceedings of Machine Learning Research, pp. 2350 2358. PMLR, 2021. URL http::/pproceeingg..ot
abs/2002.05516, 2020. URL pttpa//....e.////////aaoorrr... html
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. CoRR, http/aawwwww.aa/////////...//
Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishck Singh, Hang Qiu, Li Shen, Poilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr. Fodml: A research library and benchmark for foderated machine learning. CoRR, abs/2007.13511 2020. URL https: htttss///axxiv htt////xvv.orggaab..//a org/abs/2007. 13518
19

---

Published in Transactions on Machine Learning Research (11/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770 778. IEEE Computer Society, 2016. doi: D0.1109/CVPR.206.000 URL https://doi org/ 10. 1109/CVPR. 2016.90 90
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Conference on Solid-State Circuits Conference, ISSCC 2014, Digest of Technical Papers, San Francisco, CA, USA, February 9-13, 2014, pp. 10-14. IEEE, 2014. doi:
URL https:
//doi.org/10. 11w111199SSSSSS 2014. 6757323
Samuel Horváth, Chen- Yu Ho, Ludovit. Horvath, Atal Narayan Sahu, Marco Canini, and Peter Richtárik. Natural compression for distributed deep learning. CoRR, abs/1905.10988, 2019. URL http://arxiv
org/ ataaabs/110... 10988
Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Scbastian Stich, and Peter Richtárik. Stochastic Distributed Learning with Gradient Quantization and Variance Reduction. arXix:1904.05115 2019. URL
https://arxiiv httpp:://arsivorr//bb//////// 1904. 05115
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/ abs/1704.04861,, 2017. URL http:://aaxxv.rorr///////......0 04861
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honoluh,, HI, USA, July 21-26, 2017, pp. 2261-2269. IEEE Computer Sociely, 2017. doi: 10.1109/CVPR.220 2VPP.2017.22...... URL https:://ti...oo org/ 1109/CVPR. .037.243
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMr.R.org, 2015. URL http:::/proceedddiiine mlr. press/ /v37 1offe15.hte .htll.
Martin Jaggi, Virginia Smith, Martin Takác, Jonathan Terhorst, Sanjay Krishnan, Thomas Hofmann, and Michael I. Jordan. Communication-effiiies distributed dual coordinate ascent. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (ods.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3068-3076, 2014. URL htttps://proceedings Abstract. html
cwwrips.cc/ppeeee. 201 4/ htt./////abaaaaeeeeee
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang Hc, Lie Hc, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song. Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramer, Pranccth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. CoRR, abs/1912.0445 2019. URL http://arxiv. org/abs/ 1912.04977/
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local GD on heterogeneous data. CoRR, abs/ abs/1909.04713, 2019. URL http:://aaxxv..rr/abbb/// 04715
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local SGD on identical and heterogencous data. In Silvia Chiappa and Roberto Calandra (eds.). The 23rd International Conference on Artificial Intelligence and Statislies, A ATSTATS 2020, 26-28 August 2020, Online [Palermo, Sicily,
20

---

Published in Transactions on Machine Learning Research (11/2023)
Italy), volume 108 of Proceedings of Machine Learning Reseorch, pp. 4519-4529. PMLR, 2020. URL http:///rroceedings mlr. wttppp/sww///saooooaoara hatrr///gaaaa.eeee/////////aoo// 20a. html
Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson, Distributed learning with compressed gradients. arXiv preprint 1xXiu:180606677) 2018. URL httpps///aaxxiv.ogggbb////////// 06573
Anastasia Koloskova, Sebastian U. Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 8478-3485 PMLR, 2019. URL http: http::/prroceeding... mlr. press/ paress/v97/Kee k97/koooskovaa19 html
Jakub Konečný and Peter Richtárik. Randomized distributed mean estimation: Accuracy vs. communication. Frontiers Appl. Math. Stat., 4:62, 2018. doi: D..3389/FMMS.2016
htttps://doi..rr//t/
fams. 0asss2018.00000
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárii, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency, CoRR, abs/1610.05499 2016. URL htttp://axiiv.oot/ wwtg//a////1/.05///
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Onlario, 2009.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. CoRR, abs/2003.000 22000000037 2020. URL
Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. Hermes: an efficient federated learning framework for heterogencous mobile clients. In ACM MobiCom '21: The 27th Annual International Conference on Mobile Computing and Networking, New Orleans, Louisiina, USA, Oclober 25-29, 2021, pp. 3483278
420-437. ACM, 2021. doi: (001145/344999.336
URL http://///001..0110.1.1....11
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze (eds.), Proceedings of Machine Learning and Systems 2020, MLSys 2020, Auslin, TX, USA, March 2-4, 2020. mlsys.org, 2020. URL https::/
mlsys. org/booo/////oooooooooooo bbtb///o//////////// /11. pdf
Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander G. Schwing. Pipe-sgd: A decentralized pipelined SGD framework for distributed deep net training. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 8056-8067, 2018. URL https; 7/proceedings
http:////a///e.aa.a..00000000.00 Abttreatt.tttttatttaaaaaa html
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local SGD. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.not, 2020. URL
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In TEEF/CVF Conference on Computer Vision and Pattern Recognation, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10082 10091. IEEE, 2022. doi: 10.1109 (LLL00/CCVPR22288.2200 00985. URL https: org/ 10. 110.9///VPPP2688....
Brendan McMahan, Eider Moore, Daniel Ramage, Set. Hampson, and Blaise Agüera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.). ), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20- April 2017, Fort Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pp. PMI.R, 2017. URL http://prroceedings. mlr. htaes////////mmmaaaa...a.a.
21

---

Published in Transactions on Machine Learning Research (11/2023)
Yuan Mei, Binbin Guo, Danyang Xiao, and Weigang WI.. Fedvf: Personalized federated learning based layer-wise parameter updates with variable frequency. In IEEE International Performance, Compuling, and Communications Conference, IPCCC 2021, Austin, TX, USA, October 29-31, 2021, pp. 1 9. IEEE, 2021. doi:
483.2021.96694111 URL https://doi.co. org/ 1tt...1119//P1115111000
Constantin Philippenko and Aymeric Dieuleveut. Artemis: tight convergence guarantoes for bidirectional compression in federated learning. CoRR, abs/2006. 145991 2020. URL httpp://arzziv.or//aabb./
14591
Krishna Pillutla, Kshitiz Malik, Abdelrahman Mohamed, Michael G. Rabbat, Maziar Sanjabi, and Lin Xiao. Fedorated learning with partial model personalization. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Leurning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17716 17758. PMLR, 2022. URL Ittts:///proceddinngg .mlr 1asss//v166//llluuaaaaa .html
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization. CoRR, abs/ 2es/20003.0225, 2020. URL
has///bb//000 200300
Amirhossoin Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication--ffiieee federated learning method with periodic averaging and quantization. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, ATSTATS 2020, 26-28 August 2020, Online (Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 2021 2031. PMLR, 2020. URL http://proceedding: mlr press/v108/ reisizadeh20a.tt h
Mher Safaryan, Egor Shulgin, and Peter Richtárik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. CoRR, abs/2200..88988 2020. URL https:://arxv..ott org/ abs abs/2002.000000 08958
Atal Narayan Sahu, Aritra Dutta, Ahmed M. Abdelmoniem, Trambak Banerjee, Marco Canini, and Panos Kalnis. Rethinking gradient sparsification as total error minimization. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vanghan (eds.), Advonces in Neural Information Processing Systems 94: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 8133 8146, 2021. URL https://prooceddings neurips cc/ cc/paper//e 2021://8444440000080077888115330 Abstraa html
Ohad Shamir, Nathan Srebro, and Tong Zhang. Commmicatiion-eficcenn distributed optimization using an approximate newton-type method. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 1000 1008. JMLR.org, 2014. URL http:://rroceednng/ mlr. /ttess//3///saaarrrrr... shamir14 14. html
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of
Machine Learning Reseurch, pp. 9489 9502. PMLR, 2021. URL http:://rooeeddings mlr htt..ppress/v////
shamsian21a.1 html
Yiqing Shen, Yuyin Zhou, and Lequan Yu. Cd2-pfed: Cyclic distillation-guided channel decoupling for model personalization in federated lcarning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10031 10040. IEEE, 2022. doi: 2022.0092 URL /tdos: 1tte://..1119///11 1109/CVPR52688 2022. 009800
Nir Shlezinger, Mingzhe Chen, Yonina C. Eldar, H. Vincent Poor, and Shuguang Cui. Federated learning with quantization constraints. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pp. 8851-8855 IEEE, 2020. doi: 10.1109/ S.ASSPP40776.2220006642 URL inttps://ddn wtt.:///..10///////////777700 2020 20249999777 9054168
22

---

Published in Transactions on Machine Learning Research (11/2023)
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In Tsabelle Guyon, Ulike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4424 4434, 2017. URL https://proceedings hetpendiiinnnn.uuppspe cc/paper/2011111 /
621 22100019990111111005556600000000
Sebastian U. Stich. Local SGD converges fast and communicates little. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Sebastian U. Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rales for sgd with delayed gradients and compressed updates. The Journal of Machine Learning Research, 21(1):9613996)) 9648, 2020. URL https://aaxii orggabs/19///// 05350
Sebastian U. Stich, Jean-Bapliste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (cds.), Aduances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NearIPS 2018, December 3-8, 2018, Montréal, Canada, pp- 4452-4463, 2018. URL https::/proceeddngs patpoceedinngs.eeuipp..... pccpppp///01/////// 20sh//hh////h//////// 04000000000500002eeeaaaaaaa----0
html
Nikko Strom. Scalable distributed DNN training using commodity GPU cloud computing. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015, pp. 1492. ISCA, 2015. doi: NU2118//ITERRPPEECHH20---39 URL
sttps://doi.org/ 10. (14437/IIterspeeec 2015-354
Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and H. Brendan McMahan. Distributed mean estimation with limited communication. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3329 3337. PMLR, 2017. URL http://ppooceding: mml. prwps/ http://///sureshaaaaaaa
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. CoRR, abs/ abs/2103.007700 2021. URL attps://axxxv.orrr/aa//22222 o2w./////////.....oooo 00710
Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass cooor-comeensated compression. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 96th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6155 6165. PMLR, 2019. URL http::/proceedding/ mlr. garess/v97/tang
html
Thijs Vogels, Sai Pranceth Karimireddy, and Martin Jaggi. Powersgd:
Practical low-rank gradient compression for distributed oplinization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Acché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annuul Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14236-14245, 2019. URL httptst//procceddinngg wewpreddiinnnneuiisss.....p
neurips cauriipp.cc/ppe//201)
Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris S. Papailiopoulos, and Stephen J. Wright. ATOMO: communication--lliiees learning via alomic sparsification. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9879 9882-9883 2018. URL https:
wtt.eo/i..g..oooooooo///////aaan htt.::/ec......a..c.aaaarrc..... html
23

---

Published in Transactions on Machine Learning Research (11/2023)
Jinnyu Wang and Gauri Joshi. Adaptive communicatio strategies to achieve the best error-runtime trade-off in local-update SGD. In Ameet. Talwalkar, Virginia Smith, and Matei Zaharia (eds.), Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31- April 2, 2019. misys.org, 2019. URL
mlsys org/book/2577 bttp///oook/.ppp.fff..
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terugrad: Ternary gradients to reduce commuication in distributed deep learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Syslems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1509 1519, 2017. URL https: : http:ips..ccpppppe///11111111111 cct httpp/ppeeee2///aa////ffe.oo http:hhhhh/.t00//22000 hhaaan/999..00000000766223366600 hww.:/ecc......---aaaaaart.aaa.. html
//proceedings..
Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang. Error compensated quantized SGT) and its applications to large-scale distributed optimization. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 13299932.. PMLR, 2018. URL http://proceeedings mlr. www.alr.reeeeevv.. wu18d. html
Hang Xu, Chen- Yu Ho, Ahmed M. Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. GRACE: A compressed communication framework for distributed machine learning. In 41sl IEEE International Conference on Distributed Compuling Systems, TCDCS 2021, Washington DC, USA, July 7-10, 2021, pp. 561-572. IEEE, 2021a. doi: 0021.00060
URL https://doi. org/10 10. 1119/I1DC9591111111
Hang Xu, Kelly Kostopoulou, Aritra Dutta, Xin Li, Alexandros Ntoulas, and Panos Kalnis. Deepreduce: A sparse-tensor communication framework for federated deep learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 21150 21163, 2021b. URL httpp://prooeeddings 1200- neurips wewrips.cccpaper/DDD0... httppp////000.aaaaao/o/////////2 htt.:/52221aaa11110000-o-ttaaaaa html
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7184-7192 PMLR, 2019a. URL http://
mlr. www...pw...m.........m.. wwt..peees/9/yy11ee000 html
Yue Yu, Jiaxiang Wu, and Junzhou Huang. Exploring fast and communication-effeeen algorithms in large- scale distributed networks. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 674 683. PMLR, 2019b. URL http://proceedinns mlr. www.m.rrreeeeeee -pppss//v8 v89 hw1/99/yyl9..aaaam
Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, and Ce Zhang. Zipml: Training linear models with end-to-end low precision, and little bit of deep learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Leurning, ICML, 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 4035 4043. PMLR, 2017. URL http:://pocceedigg
mlr. peress/v70/zhangee
html
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and José M. Álvarez. Personalized federated learning with first order model optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpss://openrevier het//forumm.d-ehhjjDps..
Sixin Zhang, Anna Choromanska, and Yann LeCun Deep learning with clastic averaging SGD. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 28: Annual Conference on Neural Information Processing Systems
24

---

Published in Transactions on Machine Learning Research (11/2023)
2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 685 693, 2015. URL https://proceeddings neurips.co/
Fan Zhou and Guojing Cong. On the convergence properties of k-step averaging stochastic gradient descent algorithm for nonconvex optimization. In Jérôme Lang (ed.); Proceedings of the Twenty-Seventh International Joint Conference Artificial Intellignnce, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 3219 3227. ijcai.org, 2018. doi: 10.24963/ 10.24963/IJCAI22000 20147 URL https://dde wtw....
2018/447
25

---

Published in Transactions on Machine Learning Research (11/2023)
A
Appendix
Contents
Introduction
1
Personalized FL
2
2 Contributions
4
Related Work
4
Background and Preliminaries
5
4
Compressed L2GD
6
4.1
Compressed communication
6
4.2
The algorithm
Convergence Analysis
7
Assumptions
7
Auxiliary results
Main result
6
Optimal Rate and Communication
10
Empirical Study
11
Meta- parameter study
13
7.2
Training DNN models
13
7.2.1
Results
8
Conclusion and Future Direction
16
A Appendix
26
A.1 Convergence Analysis
Proofs of the Lemmas and the Theorems
26
................................
A.1.1
Technical results used for convergence
27
A.1.2
Main convergence results
28
A.1.3 Nonconvex convergence
................................
A.1.4 Optimal rate and communication
................................
31
A.2
Addendum to the Experimental Results
32
A.1
Convergence
of the Lemmas and the Theorems
In this section, we provide the proofs of convex and non-convex convergence results of the compressod L2GD algorithm.
26

---

Published in Transactions on Machine Learning Research (11/2023)
Overview of results. In |A.1.1 provide the technical lemmas necessary for the analyses. A.112 contains the auxiliary results pertaining to both convex and nonconvex convergence. In AA..3 we provide the non-convex convergence results, and $.114 contains the proofs for optimal rate and communication.
A.1.1 Technical results used for convergence
The following two Lemmas are instrumental in proving other compression related results.
Lemma 1 Let x & Rnd, then
Ec [[[(())]]]] (1 + ||||?
where max [we...
Proof 2 By using Assumplion 1, we have
n
Ee [[[[2))]]]] = Ec
Σ |[c(xx))1?
Σ Be, Be, (c.,)() au+ (1+w)) ||0||
=
Hence the result.
Lemma 2 Let Assumption hold, then for all & 2 0, Ec,cs [CM(T*]] = #*
Proof 3 We have
Reccss [Cas(T")] IEC. [Rss (Cm((T)))
Fec
Re, [c,(a])] zh. n Σ
Hence the result.
In the following Lemma, we show that based on the randomness of the compression operators, in expectation, we recover the exact average the local models and the exact gradient for all iterations.
Lemma 3 Let Assumptions hold. Then for all le 2 0, knowing xt, G(x*) is an unbiased estimator of the gradient of function F al !
Proof 4 We have
(.(()))) mi1 p)
if Ex
Ec,es [G](
Ecc,
[CM (f+)])
if &. & & 0,
(2
x*)
& & E 1,
np
By Lemmn
f+ 0.
mI--p]
(o
x²)
if Ex
Therefore,
Eeg [BCes [G](d*)]]
vf((x))) +p- -
A
(1 p) n(1 - p)
np np
Vo.f(x*) + V., (I**) = F(.
Hence the result.
27

---

Published in Transactions on Machine Learning Research (11/2023)
A.1.2 Main convergence results
Based on the results given in the previous section, we are now set to quote our key convergence results. Our next lemma gives an upper bound on the iterate at each iteration. This bound is composed of two terms the optimality gap, F(x²) F(x"), and the norm of the optimal point, ||>*||
Lemma 8 For p-strongly convez function F, we have || - x'1l -
Proof 5 For p-strongly conver function, F, for all x. y we have
F(a) 2 F(y) VF(y) y) + 2 vl1
At the optimal point y x², have VF(x*) = 0, and we obtain the desired result.
Lemma
Let Assumplion
hold, then
b 12 s 4 p
(F(x*) F(x))) +
Proof 6 We have
11 2311l=231
2 211a*
==1 + 2 1|
Lemm $
4 (F(x*) - F(xx)) + + 11
P
Hence the result.
Recall that, inspired by the expected smoothness property Gower (2019). we use similar idea in our convergence proofs. The next lemma is technical Lemma that helps prove the expected smoothness properly Gower el al. (201)). The bound in Lemina is composed of the optimality gap, F(x*) F(x*), the difference between the gradients of at x² and x", that is, Vh(a**) - Vh(=*) and an extra constant, B, which depends on the used compressors.
Lemma 5 Let Assumptions and hold, then
Bexic 11aa QCM((T*)) x² + QCas(T))11 < x
Vh(a*) - vh(x>)12 + a (F(x*) F(x*)) + B,
where T" = : Co(3)) a= (ww and
3 == 2(w 4war(1 w)) 112 + 4F e (QCr((T") Q="1²
28

---

Published in Transactions on Machine Learning Research (11/2023)
Proof 7 We have
A
Ec 1|+t - -Q + Q+ - Q1)((")) - + +0 - QF + QCas (vn)((")) (1))
.C C 11(a -Q -r + Qū")==== + (QF - QQ7") + (QT* ((xr("") + - Q²*)|²
4 1|1²* Q -x¹ + Q=1l || + ABc 1Q - Q7 1| + 4.c c 11 - QCM((T"))
S
C 11)) - QF" ||
+
4 1|1=* Qt - x² Q= ||² An ||5 5²1 + An e 11tt Cx
+
C 11 (Q*)(0)) - QQ*11
A|1=* Qt -x" Q="1 ||² + Σ
Ee || (=(xa + Anaws Beas 11/5/1
CECM. (T*) - QF*11
"
4 4|1=* Q -x" + Q5"1 1| + Σ Wi ||0 + 1ww Σ (1 + wi) || + 4Bcs c IIQCM("") Q="1|
W
n2
1
4 x Vh(a*t) /1(2")11 + (4w + 4w + w)) || + 4E 11QCM(0")) Q=*11²
By Lemma
0
/h(ct)) - VM(x")]2 + (4w + 4 + +W) ( (F(ct)) - F(x*)) + 2 12-1)3)
5
X2
+ +. c [100xr00)) 1I"122
n²
(h(a*) - vh(x*)] a +| X2
+ B.
Hence the result.
Now we are all set to prove the expected smoothness property in our setup.
Lemma 6 (Expected Smoothness) Let Assumptions 1 and 2 hold, then
E [IGG(>>)A11]] S 47 (F(x*) - F(x*)) + 8,
where
(1+ )(1-2) } 7 y:= a2P(--2) max { by A
2n-p
(I-p)' n
and
8: 8:= 2. p)
+ +2E|
n2p
Proof 8 We have
[VV[(* -1)())11
ff = 0
(1-p)²
1** (F) - + QCM(
if Eu = 1 & E = 0,
[h(*** - Vh(z*)12
if E = 1 & Ex-1 = 1.
p²
Finally,
E RE - G(x*)|12 (1-p) [V - (2))
+ 12 1
[V - V1(21)12
(1 - p)2
+ p(1 p) 1²
11- -Q" - -x" ==+QQCM(T)]
nzp²
IV (x))
(f(x))
+ +IVh((*) vh(2*)1²
(1-p) p)
2 - p)
n2p 1100
-QCx(0*) - + 20)x(
29

---

Published in Transactions on Machine Learning Research (11/2023)
Therefore, by using Lemma b we get
IV. (1+)
(+()) 112
[Vh(2)) vh(2))] AP 12(1-P) A
B|G(x*) G(x")|=*'
(1 1-P) P)
n²p
IV (x
V/
+ [/((**) - VM("")1
P)
22 p)
4 X2
n?
|Vh(a*>) - vh(>')/' )12 + (F(x*) - F(x")) +
n2p
IVS (x**
Vf (z())) 11?
+ 1+ 4(1 -11-p1) /Vh(zt) Vh(z')12
(1 (1-P)
P
a1 P) (F(x*)- -F(x))) + 3X2 -p)
n2pp
1221
(1-P) (s(x²) -s(x') 21
2L
1 4(1 - p)
(h(x*)) 1(x²1)
n
p
a1²(1 p) (F(x*) - F(xx)) + 3x2 p)
n²p
n22p
s 27 -F(x*))F(x*)) F(x))) BA p) nºpp
Finally, we obtain
E [(
2BI|G(a*) G(a*)|=* 2E||G(=))||
s 47 (F(z*) - F(x*)) + 228 - + 2EI|G(x
n²p
s 4y (F(x)) - F(x*)) + 5.
Hence the result.
Based on the above results, the convergence of Algorithm [ for strongly convex functions follows directly from Lemmas and Theorem 3.1 from Gower al. 2019
A.1.3
Nonconvex convergence
Theorem (Non convex case) Let Assumptions and 3 hold. Assume also that F is L Ly-smooth, bounded from below by F(x*). Then to reach precision, 0, set the stepsize, n minf M 21. ) such that for
41, M F(a')) we have min K V Se.
K 2
2
Proof 9 From smoothesss of F we have
Lg ná IG T G(z²))+ 2
F(ghtl) < F(**) - m V
By taking the expectation in the above inequality, conditional on aht we gel
F(I*) - TH 1 1 + By Lemmal
L
E [[F(a*+))
x*]
(IG())))P1)))
2
which by using Assumplion reduces to
E ((x+tt) 1 S F(a*) - 7 [nF|\VF>>>>1 L
[1]
02)
x F(a*) - 77k (1 ( M
||VF(+
2
2
30

---

Published in Transactions on Machine Learning Research (11/2023)
After rearranging, we have
Mi T] 2
|IVF(* S F(x*) E F(((tt 1x) + T.
Setting Th = 1 > in the above, taking expectation, using the tower property of expectation, and finally summing over the ilerates k 0.1. K we have
(1 Mn
K-1
113 $ (F((2)) (F(z9)=((x*) + Kl
Σ k=
E [IVF
2
2
If 11 S L then
K-1
2 F(x)))(xx)) LyKno²
E
|1
"
Dividing throughout by K, we get
2 (F(x") - -F(x)) T.mo?
K Σ
n
Finally, setting 1) T, M we have
M
(F(z") - + o²
min
E VF(+*)]
(4)
A= 1...l.KKK K-
K
M
2L, (F(a
-- For given precision, >> 0, to make min LE| V VF S e we require and I, < & resulting in
K > (F(x(=(2)) - -F(x)) and S e
2
20
Hence the result.
A.1.4 Optimal rate and communication
The following proofs are related to optimal rate and communication given in $G
Theorem (Optimal rate) The probability p² minimizing is equal to max PA), where Pe. = ax² + m((pp] in (0, 1).
TX+L VX+IALL and PA is the optimizer of the function A(p) 2n
Proof 10 We can rewrile y as follows
a>2
+ max (A(p), B(p))
=
2n
where A(p) 2npp + .i-ツ and B(p)
ax²
2n0p + 41 31 The function B is monotonically decreasing one, and it has stationary point
ad²
a function of p. The function A goes to as goes to between zero and one hence il is conver in the interval (0, 1). Thas il admils an oplimizer pa in (0, 1). Note that Pe -N-L-V1+11A is the point for which A(p) is equal B(p). Note also that near to zero B(p) 2 A(p). Therefore if Pe I PA then the optimizer of y is PA otherwise is equal Pe- Thus the probability p" optimizing & is equal to max (ex((pe Pa)-
GÅ
31

---

Published in Transactions on Machine Learning Research (11/2023)
12G0g19=200 20, 0.28.p Dernouliigs
LaGDyla=244 A== 0.d0p.p Nataral)
L8GDgly= 24,, p00.. -0)D)
Accurayy
25.5
#bilson
1.2
63
#bils/n
20
25
03
*b1sshn
(b)
(c)
(d)
Figure Training ReaNet-18 CIFAR-10, with 10 workers. Loss and Top-1 accuracy on train (a) (b) and
test data (c) (d).
in (0, 1) is equal to ad² L Lemma 7 The optimizer probability Pa of the function A(p) 2n²p n(1-p)
if 2nL ax2
2a / 212nL-ax' 2n 22A
if 2n > ax otherwise
PA
2(2n 2(2nI--a27)
Proof 11 If 2nL + aX2, then the function A has the following two slalionary points -2a 22u2-22NZanL 2nL = ax?, then the function has one stationary point equal to 1
-2n 22n22+222200 and 2[2mL ax")
2121
Theorem 2 (Optimal communication) The probability p" optimizing C is equal to max PAJ, where Pe T 14X--112 and Pa ax
Im
6A
Proof 12 We can rewrite nC as follows
nc max (A(p), B(p)}
where A(p) ax'r(1-p) (1=(1=p) Lp and B(p) ad p1pp(1=p) + (1-0 + 4A(1 - p) -3pp(1 -p). The 2n
2n
2n
2n
function B is monotonically decreasing function of p in [0, 1]. Note that B(0)
ad² 4A and B(1) 0.
2n
The function admils a minimizer equal PA axz Of course Pa a probabilily under the conditioo that Ln ax2 Thus we consider the following 2 scenarios
1. If Ln > ax² (PA < <0) then p² Pe
2. else p"
We conclude in both cases that p² = max PA.,
A.2 Addendum to the Experimental Results
Batch Normalization. Beside the trainable parameters, the ResNet models contain batch normalization lofte & Szegedy (2015) layers that are crucial for training. The logic of batch normalization depends on the estimation of running mean and variance, and these statistics can pretty personalized for each client in a heterogeneous data regime. The implementation FedAvg and FedOpt in FedML considers the averaging of these statisties during the aggregation phase. In our implementation, the batch normalization statistics are included into aggregation.
Step-size. The siep-sizes for FedAvg and FedOpt tuned via selecting step sizes from the following set (0.01, 0.2, 0.5, 1.0, 2.0, 4.0). We consider the stop size for both algorithms be 0.1. Starting with step size 0.2 algorithms diverge; we also did not use step size schedulers. Additionally, we have tuned number of local epochs for FedAvg from the following set 21, 4), and pick local epoch. The batch size is set to 256.
32

---

Published in Transactions on Machine Learning Research (11/2023)
70
Ea00 A00 Strondss -032
fati 24 3-133 Kainnd)
10070 2.A $5, gaorp-
25
Accyras
Accuracy
52.5
0.2
#bitsin
2611
#btsh
1el1
#bitsm
16l1
#bits/n
(a)
(b)
(c)
(d)
Figure 10: Training Densetiet-121 on CIFAR-10, with workers. Loss and Top-1 accuracy on train (a) (b), and test data (c) (d).
1300r7a- a.)
2200 20..0.25 0..5 D.J. Beeeooitlyy........ DAS)
1870 A- A.5.p-t.. qsanp---
Accuracyy
Accuracy
0.2
#bitsm
#btsh
#bisin
#bitsn
(a)
(b)
(c)
(d)
Figure 11: Training Mobilellet on CIFAR-10, with workers. Loss and Top-1 accuracy on train (a) (b), test data (c) (d).
Compressed L2Gd vs. FedOpt. From the experiments in Section
7.2 Figures we realized that FedAvg is not competitive no-compression baseline for L2GD; see Table 2 FedOpt, on the other hand, remains a competitive no-compression baseline comparable to compressed L2GD. Therefore, we separately measure the performance of compressed T22D and non-compression FedOpt for training ResNet-18, DenseNet- and MobileNet. Figurcs 10 demonstrate that L2GD with natural compressor (that by design has small variance) empirically behaves the best and converges approximately times faster compare to FedOpt. They also show that compressed L2GD with natural compressor sends the least data and drives the loss down the most. At the same time, L2GD with natural compressor reaches the best accuracy for both train and test. sets.
Reproducible research. compressed-fl-12ee code Our source codes have been constructed on top of the popular federated learning repository: FedML.ai; see http//ttttttuttte//// http:////ggaaaaooooooaaaaaaaaaaa
See our repository online: httpss//ggihuuu.cmmmburlaaeeeee
33


Reviews:
[{"review_id": "p5QVVhqiSv", "reviewer": "Reviewer_uMx1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n* The paper is quite well written (modulo some clarity questions, see below).\n* The authors do a good job of surveying the surrounding literature. \n* as such the theoretical results (related to the convergence of compressed L2GD) seem sound.\n* The experiments make a consistent case that the method works well.\n\n*Weaknesses*\n\n* The proposed algorithm seemed like a fairly incremental change to L2GD, and therefore was not very interesting to me. The main contribution is an concatenation of a known technique (probabilistic compression in both uplink and downlink) with a fairly niche way to do personalized FL (L2GD). I do agree with the authors that personalized FL is an important problem of practical interest, but there are a ton of ways of doing personalization and L2GD does not seem to rank among the methods of choice at the moment. However, I'm not very up tp date with the latest trends in FL, so perhaps there is even now a push towards \"exhaustively characterizing\" the space of all possible FL approaches, in which case I suppose this paper fills a specific gap in the literature.\n* Some clarification questions seem unaddressed; see below for requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a modification to the Loopless Gradient Descent (L2GD) algorithm of Hanzely and Richtarik [HZ20]. This is an algorithm to solve a variation of federated learning (FL) which allows for some degree of \"personalization\" and is relevant in settings where there is heterogeneity among the clients' local datasets. The L2GD algorithm essentially switches between (local) gradient update steps and (global) model aggregation steps --- just like regular FL --- but these switches happen probabilistically. It is not too hard to algebraically show that for many model aggregation steps, some amount of communication needs to happen between the central server and the clients.\n\nThe algorithmic contribution of the paper is an updated version of this algorithm where both the \"uplink\" and \"downlink\" communication is implemented via compression algorithms. This (obviously) reduces communication costs, and the main theoretical contribution of the paper is to show the convergence is not (essentially) affected by such a compression step. Variants of this approach have appeared in the FL literature before; the key novelty seems to be its application to L2GD and therefore personalized FL.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Requested changes*\n\n* Consider pre-emptively clarifying answering some of the questions below in the main text.\n* Consider pre-emptively addressing the \"incrementality\" question up front in the paper.\n\n\n*Questions*\n\n* How do you initialize $\\xi_{-1}$ in Algorithm 1? I understand that for subsequent iterations, this parameter is based on random coin tosses.\n* I assume that all the local agents have shared randomness (or at least that the probabilistic bit is broadcast to all agents so that they know what to do in each iteration)? It may be helpful to clarify this in the pseudocode.\n* I'm somewhat confused about one point in the algorithm. It seems that $\\bar{x}^{k}$ -- the \"pristine\"/non-compressed average of all client models -- is never made available to all clients, except at initialization. The only information received by the clients are the *noisy* / compressed averages of the model parameters. What if we had a series of coin tosses where $\\xi$ was set to 0 (i.e., all the model weights move far away from their original inits), which was then followed by consecutive 1's? In this case, the operation $\\bar{x}^{k} = \\bar{x}^{k-1}$ would throw an error since $\\bar{x}^{k-1}$ is undefined?\nI think this discrepancy doesn't affect any convergence analysis (due to Lemma 2) but please check if this intuition is correct.\n* It may be helpful to provide a bit more background/context for Assumption 1 -- why this is necessary, whether standard, etc. Also, $C_i$ is at the moment defined as a deterministic compression operator at the start of Section 3, in which case it probably does not make sense to talk about unbiasedness, independence, write down $E_{C_i}$, etc.\n* Consider making Fig 2 bigger and more readable. Also, specify the model (logistic regression) in the caption.\n* I am not quite sure why \"optimal\" is in quotes in Section 6 and on Page 12. Consider being more precise.\n", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GjYgJbO7ai", "reviewer": "Reviewer_HbRJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n-- The paper is well-written and easy to follow. \n\n-- The Algorithm and the statements of the analysis are clear and interpretable.\n\n--- Extensive Empirical analysis for various compression techniques for L2GD under logistic regression and compressed L2GD on image classifications tasks using DNN.\n\n--- The authors claim that this is the first bidirectional compression technique coupled with a probabilistic communication in the federated learning setting.\n\nWeaknesses:\n\n1. The setting explored in the paper is very specific to the L2GD algorithm.\n\n2. Lack of comparison with other communication efficient FL algorithms. For ex. QUAFL: FEDERATED AVERAGING MADE ASYNCHRONOUS AND COMMUNICATION-EFFICIENT, SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learning.\n\n3. More details on the compression operators used and their properties could be incorporated to make the paper more complete.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper follows the “Personalized FL” framework introduced by Hanzely and Richtarik which involves optimizing a regularized cost function that seeks a personalized FL model for every device. In order to solve such a cost function, Hanzely and Richard proposed the loop less local gradient descent (L2GD). However, the L2GD algorithm lacks a compression mechanism between for both master-worker and worker-master communication. The authors propose the compressed L2GD algorithm which utilizes a bidirectional compression technique for the L2GD algorithm. They also provide convergence analysis illustrating their compressed L2GD algorithm has a comparable asymptotic convergence rate as vanilla SGD without compression (in both strongly convex and smooth non-convex case) albeit with an increase in the stochastic gradient variance due to the additional compression. The authors provide empirical studies to illustrate the practicality of their algorithm both within their theoretical framework and beyond (e.g. biased compressors).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. How does the optimal rate affect the performance in experiments, i.e. results similar to Figure 3 for the compressed method?\n\n2. If the goal of L2GD is to achieve personalisation, why do the experimental results compare the performance of the global model?\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hiWu3hEo2J", "reviewer": "Reviewer_RSnY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed method is well-motivated, and the writing is easy to follow. \n\n2. This paper provides extensive experimental results to demonstrate the performance of the proposed approach. \n\n\nWeakness\n\n1. The communication-efficient algorithms for federated learning have been extensively studied. However, this paper missed many of them, e.g., [1-4]\n\n2. The proof is not correct. When proving theorem 5, the third step uses Lemma 6, and lemma 6 uses lemma 4. But Lemma 4 uses the strong convexity of the loss function, which does not hold for nonconvex problems. \n\n[1] Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations\n\n[2] FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization\n\n[3] Federated Learning with Compression: Unified Analysis and Sharp Guarantees\n\n[4] On the convergence of communication-efficient local sgd for federated learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper developed a new approach for reducing communication costs and perusing personalization in federated learning. However, it missed some important related works and the proof has some flaws. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The communication-efficient algorithms for federated learning have been extensively studied. However, this paper missed many of them, e.g., [1-4]\n\n2. The proof is not correct. When proving theorem 5, the third step uses Lemma 6, and lemma 6 uses lemma 4. But Lemma 4 uses the strong convexity of the loss function, which does not hold for nonconvex problems. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers' comments have been addressed and all recommended acceptance.
Decision: Accept (poster)