Linear Convergence of Decentralized FedAvg for PL Objectives: The Interpolation Regime
Authors: Shruti P Maralappanavar
Prashant Khanduri
Bharath B N
Venue: Accepted by TMLR ()

Abstract: Federated Learning (FL) is a distributed learning paradigm where multiple clients each having access to a local dataset collaborate to solve a joint problem. Federated Averaging (FedAvg) the algorithm of choice has been widely explored in the classical {\em server} setting where the server coordinates the information sharing among clients. However, the performance of FedAvg in the {\em decentralized} setting where only the neighboring clients communicate with each other depending on the network topology is not well understood, especially in the interpolation regime, a common phenomenon observed in modern overparameterized neural networks. In this work, we address this challenge and perform a thorough theoretical performance analysis of FedAvg in the interpolation regime under {\em decentralized} setting.  We consider a class of non-convex functions satisfying the Polyak-{\L}ojasiewicz (PL) inequality, a condition satisfied by overparameterized neural networks. For the first time, we establish that {\em Decentralized} FedAvg achieves linear convergence rates of $\mathcal{O}({T^2} \log ({1}/{\epsilon}))$, where $\epsilon$ is the solution accuracy, and $T$ is the number of local updates at each client. We also extend our analysis to the classical {\em Server} FedAvg and establish a convergence rate of $\mathcal{O}(\log ({1}/{\epsilon}))$ which significantly improves upon the best-known rates for the simpler strongly-convex setting. In contrast to the standard FedAvg analyses, our work does not require bounded heterogeneity and gradient assumptions. Instead, we show that sample-wise (and local) smoothness of the local objectives suffice to capture the effect of heterogeneity. Experiments on multiple real datasets corroborate our theoretical findings.

Published in Transactions on Machine Learning Research (MM/YYY)
Linear Convergence of Decentralized FedAvg for PL Objec- tives: The Interpolation Regime
Shruti Maralappanavar
mshruti32@gmail..c com
Deportment of Electrical, Electronics and Communication
II Dherwad Dharwad, India
Prashant Khanduri
khanduri, www.srrrirrrraaattroooo Grwayne. edu
Department of Computer Science
Woyne State University
Detroit, MI, USA
B. N. Bharath
bharathbu @iildh ac. in
Department of Electricnl, Electronics and Communication ITT Dharwad Dharwad, India
Reviewed on OpenReview:
https: // openreview. pdf/ (oddffdd=0ppp D003aBBhhh
Abstract
Federated Learning (FL) is a distributed learning paradigm where multiple clients cach having access to local dataset collaborate solve a joint problem. Federated Averaging (FedAvg) the algorithm of choice has been widely explored in the classical server setting where the server coordinates the information sharing among clients. However, the perfor- mance of FedAvg in the decentralized setting where only the neighboring clients communicate with each other depending on the network lopology is not well understood, especially in the interpolation regime, a common phenomenon observed in modern overparameterized neu- ral networks. In this work, we address this challenge and perform thorough thcorctical performance analysis of FedAvg in the interpolation regime under decentralized setting. We consider class of non-convex functions satisfying the Polyak--.ojsiewiczz (PI.) inequality, condition satisfied by overparameterized neural networks. For the first time, establish that Decentralized FedAvg achieves linear convergence rates of O log (og(1/c)), where is the solution accuracy, and 7 is the number of local updates at each client. We also extend our analysis to the classical Server FedAvg and establish a convergence rate of (log(1/e)) which significantly improves upon the best-known rates for the simpler strongly-convex setting. In contrast to the standard FedAvg analyses, our work does not require bounded heterogeneity and gradient assumptions. Instead, show that sample-wise (and local) smoothness of the local objectives suffice to capture the effect of heterogeneity. Experiments multiple real datasets corroborate our theoretical findings.
1 Introduction
In the age of Bigdata, Federated Learning (FL) provides machine learning (ML) practitioners with an indispensable tool for solving large-scale learning problems. FL is a distributed machine learning scenario that allows the edge devices to learn a shared model while maintaining the training data at the edge devices (Konečný al. 2016 McMahan al.] 2017). This avoids the need to share the data with a central server and preserves the privacy of individual clients (edge devices). Assuming a supervised learning setting, where each of the N distinct clients have access to some local data (ær, y) ~ Dr. from distribution Dj for k E [N)

---

Published in Transactions on Machine Learning Research (M//YYY)
aim to solve:
N Σ
1 N Lx(w),
FL Problem: min (w) wERd
where (x(w)) == ((a.))
y) is the expected loss at client E [N] for the input feature x E X, and the corresponding label y E V. Here, (x)) is the model's output with parameters w E IRd. In this work, we focus on solving the FL Problem in the interpolation regime (cf. Assumption 1 an assumption usually
satisfied by overparametterieed models (Bassily el al. 2018).
The de-facto standard for solving the above FL Problem is the simple Federated Averaging (FedAvg) algo- rithm (McMahan al., 2017). In recent years, many works have attempted to characterize the convergence
of Fed Avg under different settings (Stich 2018 Li et al.
2019a: Woodworth et al. 2020a Ma et al. 2018 Yu
et al. 2019b). For example, the authors in Stich (2018) show a convergence rate of O (1 Ne) for minimizing
strongly convex functions while Haddadpour & Mahdavi (2019) establishes similar rates for minimizing fune-
tions satisfying
(PL) condition. For minimizing non-convex smooth functions, FedAvg
achieves a convergence rate of ((1/Ne2) (Karimiredyy et al. 2020 Woodworth et al. 2020b), where, 6 refers
to the desired solution accuracy.
Note that the majority of studies have focused on FedArg in the classical server-based setting (referred to as Server FedAvg), where a central server aggregates information. In this approach, clients compute model updates and send them to the server, potentially leading to communication bottlenecks and delays. More- over, an attack on the server could compromise the privacy of the aggregated model. Additionally, in many practical learning scenarios, access to a central server may not be feasible. For such cases, the alternative is Decentralized FedAvg. In this docentralized sctting, global aggregation is replaced by local aggregation, where each client updates its model based on connections with neighboring clients. While the Decentralized FedAvg algorithm has been examined in earlier works (Koloskova et al. 2019 Li et al. 2019b), its linear (fast) conver- gence has only been established for the strongly convex setting in the interpolation reginatttiirrnmeeeee (cf. Assumption I In practice, it has been observed that even for non-convex settings Decentral- ized FedAvg converges at a much faster rate compared to the rates demonstratod in these works. To illustrate this fact, in Fig. D we plot the behavior of the train- ing loss (on log scale) as a function of communication rounds for Decentral- ized FedAvg to solve classification task on MNIST data set utilizing overparam- terized models, i.e., in the interpolation regime (for experimental details please see Section 5) It is clear from the plot that in the interpolation regime the loss decreases linearly as function of communication rounds. This sug- gests that the standard analyses of Decentralized FedAvg lack a theoretical foundation to explain its linear convergence.
1.3
Overparameterized
Log Training Loss
-1.4
-1.5
-1.8
-1.9
50
100
150
200
250
300
350
Communication Rounds
Figure 1: log-training loss vs communication rounds for overparameterized
Deep Neural Networks (DNNs) and a simple regression model.
In this work, we attempt to fill these gaps and perform a thorough theoretical analysis of Decentralized FedAvg in the interpolation regime where the local nodes communicate over an undirected graph. Under this setting, we establish the linear convergence of FedAvg for minimizing class of non-convex functions satisfying the PL inequality. We also extend our analysis to the classical Server FedAvg and establish improved guarantees compared to the state-of-the-art. We note that PL inequality plays a key role in the training of overparancterized systems. Specifically, many works have shown that the loss function of an overparameterized neural network satisfies the PL inequality (Bassily et al. 2018 Liu et al., 2020). Furthermore, our analysis reveals that the standard but restrictive assumptions of bounded gradients (Yu
ct al.
2019b Stich 2018 Li et al.) [2019a] Koloskova et al.) (2020), heterogencity (Yu et al., 2019a] Woodworth
et al.
2020b Yu et al. [0199]] Wang et al. [202]] Sery et al.) 2021). and variance (Woodworth et al. 2020bb
Qu et al. 2020) be avoided while guaranteeing this linear convergence of Decentrulized FedAvg
2

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 1: Comparison with existing works. Here, SC, and NC represent Strongly convex, Convex and
Non-convex settings, respoctively.
ALGORITHM
CONVERGENCE
EXTRA ASSUMPTIONS
SETTING
Local SGD (Stch 2018
(1/Ne))
Bounded gradient
Local SGD Yu
O (1/Ne)
Bounded variance, smoolluess
NC
Local SGD (Haddadpour dl al.
(5)
(1/NNe)
Bounded variance, smoothness
NC
FodAvg Qu al.
18
O7 (1))))
Bounded Gradieut, Bounded Variance
Overparameterized SC
Local SGD Woodworth
2020b
Bounded Variance
Local SGD
Woodworth
ai.
2022
(8)
Bounded Variance
PR-SGD
Nu
201
(1/Nee)
Bounded Variance
NC
FodAvg
Karimireddy et al
2020
(8)
(1/Ne))
Bounded gradient dissimilarity
NC
Bounded heterogeneity
SGD Koloskova
2020
(7 (77c)
Smoothness
Overparameterized SC
OUR WORK
(log
Smoothness
Overparamelerrizel NC
NFSGD [Iaddadpour & Mahdavi 2019
(1I/Ne")
Bounded local variance
NC
DECENTRALIZED SCD [Koloskova et al
2020
OT (T/e))
Smoothness
Overpatamecerizzed SC
DECENT RALIZED (DnSong et al.
DECENTRALIZED
(log (1/4))
Smoothness
Overparameterizet NC
OUR WORK
O(T (og(1
Smoothness
Overparameterizet NC
Contributions. Here, we list the major contributions of our work:
First, we consider the decentralized setting where N distributed clients communicate over an undirected graph. We show that to achieve an e-accurate solution Decentrulized FedAvg requires O (T² log (1/e)) rounds of communication. We also characterize the effect of the network topology on the performance of Decentrolized FedAvg.
For the classical Server FedAvg also we establish linear convergence rates of R~ O (log (1/e)). Compared to the best-known result Koloskova et al. (2020) where simpler strongly-convex problem is considered (see section
C for details), we get improved rates independent of the number of local updates T (cf. Table
I and Section 1.1 for comparison).
Our theoretical results do not require assumptions on the boundedness of heterogeneity, gradients, and stochastic variance. We show that sample-wise smoothness of the stochastic loss functions suffices capture the effect of data heterogeneily among different clients while avoiding the need to impose the restrictive bounded gradient and variance assumptions.
Finally, we present experimental results on various data sets such as CIFAR-10, Shakespeare, MNIST, and FMNIST, and corroborate our theoretical findings under the decentralized settings. Through our experiments, we show that an optimal number of local updates Tia exists and increasing T beyond Ta hurts the performance of FedAvg in both server and decentralized settings.
1.1 Related Work
After the introduction of the FedAvg (McMahan et al. 2017), multiple works have analyzed the convergence of FedAvg in the server setting with homogencous data, i.c., when the data i.id across clients (see Stich
2018 Wang Joshi 2018 Khaled al. 2019 Yu et al. 2019b Wang et al., 2019 Yang al. 2021 The
authors {Stich 2018) were the first to obtain a rate of (1/Ncc) for strongly convex and smooth problems
for achieving an e-accurate solution (ef. Definition 2).. Later (Haddadpour et al. 2019 [Haddadpour &
Mahdavi, 2019) proved similar result but for non-convex functions satisfying PL inequality. The analysis of FedAvg for the general non-convex settings was first performed in (Yu et al. 2019b) where the authors establish a rate of () to reach an e-stationary point where the stationarity is measured with respect to the gradient norm of the loss funciton.
There are few works that have analyzed the performance of Fedvg in decentralized settings as well. One of the initial works, (Lian et al., (017)) considered decentralized parallel SGD (D-PSGD) and provided a convergence rate of ((1NNe) to c-stationary point to minimize smooth non-convex functions. Later,
3

---

Published in Transactions on Machine Learning Research (MM/YYY)
Haddadpour & Mahdavi 2019 analyzed the convergence of FedAvg under both server and decentralized settings with bounded gradient dissimilarity assumption. The authors also established convergence rate of O()/Ne") to an e-stationary point to minimize nonconvex functions in both the server and decentralized settings. The authors in (Yu et al., 2019a) also extended the analysis of Momentum SGD to decentralized networks and established convergence of ((1/Ne2) to an e-slationary point to minimize non-convex func- tions. Recently, the authors in (Song et al.) established a. linear convergence rate of ((log(1/e) to an e-accurate solution for a decentralized gradient descent algorithm in the overparameterized regime. But, it is well known that in the deterministic setting distributed algorithms for minimizing PL loss functions are capable of achieving linear convergence (even in the non-interpolated regime). Moreover, for data-centric problems, deterministic algorithms involve computations of large batch gradients during each update, mak- ing them impractical for large-scale problems. Further, the local nodes conduct. only a single local update. However, in this work, we consider a stochastic problem where each agent conducts maltiple local updates within cach communication round. Morcover, it is an open problem to establish if (stochastic) Decentralized FedAvg can guaranlee linear convergence in the interpolation regime for non-strongly convex losses. All the above works provide a sublinear rate of convergence for Decentralized FedAvg, however, as illustrated in Fig. [ Decentrulized FedAvg converges at much faster rate in practice. To understand this behavior of Decentralized Fed.vvg, in this work, we analyze the performance of Decentralized FedAvg for minimizing a special class of non-convex functions satisfying PI. inequality under the interpolation regime. We note that overparameterized neural networks /systorms//stemsss usually operate in the interpolation regime while their loss functions have been shown to satisfy the PL inequality.
Similarly, the authors in (Koloskova et al. 2020] have also established the linear convergence of Fed in the decentralized sctting for minimizing strongly-convex losses in overparameterizze setting. The above works ouly focus on the analysis of FedAvg for the strongly-convex objectives in the overparameterized regime while we focus on the more general class of non-convex functions satisfying the PL inequality. Note that for strongly-conver objectives the local clients will all share a unique minimum, this implics that even if there is no communication, FedAvg will converge with the same rate to the local optimal. Importantly, our analysis improves the analyses of (Qu el al., 2020) and (Koloskova et al. 2020), and establishes better dependence on the local updates on the performance of FedAvg (Please see Table [ Moreover, compared to other works that assume restrictive bounded gradient, heterogencity, and variance assumptions, we show that such assumptions can be avoided by using a sample-wise smoothness assumption. The authors in (Qui ett al. 2020) show linear speedup for strongly convex and convex functions. Table [ presents summary of the above discussion.
The Decentralized SGD algorithum is extensively studied under various conditions which consists of time- varying network graphs (Koloskova, et al., 2020), momentum updates (Lin et al.) 2021), asynchronous model updating (Nadiradze al.) 2021). To handle the heterogeneity in data across the clients, various tracking algorithms such as gradient tracking, model tracking (Yue Liu & Stich 2024 Aketi et al., 2024) and mo- mentum tracking (Takezawa et al. 2023) have also been proposed. In separate line of works, the authors in (Zhu et al., 2023b a) prove generalization guarantees for Decentralized SGD algorithm. The authors in NNadiradze et al. p along with asynchronous updates consider quantization and multiple local steps. Some recent works such as (Beznosikov et al. 2022) have looked at the stochastic extragradient method with time varying networks for the decentralized methods.
Notations: We use bold small letters to denote vectors and capital bold letters for matrices. We denote the expected value of a random variable X by LE [X]. We denote Ig-norm by I1-1|2, Frobenius norm by II-I1 and operator norm by I1-Ilop- (,-) denotes the inner product. The cardinality of any set B is represented by [B]. We use the standard notation ((n) to denote the order of n. For a vector-valued function ((w), the gradient is denoted by (un) E IRd, and the Hessian is denoted by (2 E Rdxd We use 1 to represent a column vector with all ones. We use [N] to denote the set [I....,N]
4

---

Published in Transactions on Machine Learning Research (MM/YYY)
2 The Decentralized FedAvg Algorithm
In many practical scttings, the central server is absent, and the clients are required to communicate and update the model weights in a decentralized manner while communicating with only the neighboring nodes. In this work, we consider such a setting and study Decentralized FedAvg, an extension of FedAvg to the decentralized sctting. The decentralized sctting consists of N distributed edge devices which are represented using a connectivity graph & E (V,E). Here, V [N] is the vertex set or clients, and E {V V} represents the edges of the graph. Any edge (i,j) E represents a connection between node and j. Further, the connections are represented using mixing matrix P [PL,]] E RNXN, where Pi, = 0 if there is no edge between node i and k i.c., (i, & E, clse Pk,i > 0. Unlike many existing works on decentralized settings (Lian et al.] 2017), here we consider a general framework where each client performs T rounds of local updates. Decentralized FedAvg is presented in Algorithm 1 while in the following, we provide an outline:
1. Initialization: Each client & E [N] initializes the model parameters denoted by wn- See Step 1 of Algorithm !
2. Local updates: Each client performs T steps of SGD starting from the model parameters obtained by the aggregation of the updates from neighbouring clients. Towards computing the stochastic gradient, each client & E [N] uniformly randomly samples a batch of data of size b denoted by B% and then computes the gradient. The resulting model parameters after T local rounds in the r-th global round are denoted by wf." which is sent to all the neighbouring clients of k. See Steps 6 and 7 of Algorithm Note that this procedure similar to the local update step in the FedAvg case.
3. Aggregation: In the r-th global communication round, each client E [N] computes a local average of the model parameters received by its neighbors. The aggregate model is denoted by wt. The steps (2) and (3) above are repeated for R rounds. See Steps 11, 12 of Algorithm 1
2.1 Assumptions
In this subsection, we present the assumptions and definitions used in the analysis of the Decentralized FedAvg algorithm.
Definition 1. (L-Smoothness)): The function & (u) is said to be L smooth if there erists a constant L > O such that V/ (u1) 4 (u2)ll2 S L lll1 -uzllz for any u1 U2 Rdd Note that this further implies that @ (ui) 2 & (u2) + (V2( u1 u2) + 2 ||u1- u2ll for any un, u1 E IR..
Definition 2. (e-accurate solution): A stochastic algorithm is said to achieve an e-accurote solution in r rounds if (ww) - -b(w") ) s, where the expectation is taken over the stochasticity of the algorithm and w" E arg min we ((r)).
Assumption 1. (Interpolation); We say that the model parameters is operating in the interpolation regime if there erists a w IRd such that the per sample loss æu.j (w) = for all samples j E [b].
Assumption 2. (PL inequality): The joint objective & (v) satisfies the PL inequality, i.e., | (v)) > pa (v) for some μ > 0 and for all u E Rd. Further, the local loss functions Pp (v) for all k = 1,2,..... ... N are also assumed to satisfy the PL inequality, henceforth referved to as local PL inequality, i.e., / (v)]² > p& 2 (v) for some μi > and for all v E Idd.
Assumption 3. (Sample-wise, Local and Global smoothness): The functions Pug()) for allj E [b) E [N] are assumed to be -mooth.. The local functions xx(-) for all k E [N] are assumed to be Lm-shhoth. The above assumptions imply |/V (1)ll² < 2 (v) and V (v)1 < (v) for all l E [N] and j E [b). We also assume the global loss (()) to be L-smooth.
A ssumption 4. (Urbiasednesss): We assume that the stochastic samples of the gradient and the loss function at each client & E [N] are unbiased, i.e., E [V (w)] = 742 (w) and E [2x.s (w)] = 2b (w) for any j E [b] and w E dd.
Most of the above assumptions, including interpolation, PL inequality, and smoothness are standard as-
sumptions made in various works in the past (Bassily et al. 2018 Karimi et al. 2016 Ma. et al.] 2018
5

---

Published in Transactions on Machine Learning Research
Algorithm Decentralized FedAvg
1: Initialize ( 0.0 = wx , we E Rd for & [M]
2: for r= 0, 0,1,..., R do
Initialize wa at device & [N] do for devices k E [M] do
for w= 1.......
6: Sample batch " and [8%] =
SGD step on w for & [M]: w
r.+ - &E Vana (w)
8: end for
9: end for
10: Receive w/ from clients E [N]
11: Aggregation step : w = EieN, Pk, for k E [N]
12:
end for
Nguyen & Mondelli 2020 For example, the authors in (Bassily et al. 2018 Liu et al. 2022a. Karimi et al.
2019) assume PL inequality along with sample-wise wise smoothness to prove linear convergence of FodAvg in the interpolation regime. It is also important to note that the overparameterized systems satisfy PL inequality (Liu al. 2020 Nguyen & Mondelli 2020; Nguyen et al.) 2021 Allen-Zhu ct al. 2019 Liu et al.) 2022a and hence plays crucial role in the analysis of overparameterized systems [Liu et al. 2022a). Moreover, note that the assumption of sample-wise smoothness is not very stringent since any neural network with a smooth activation function satisfies this assumption.
2016 Haddadpour et al.,
3 Convergence of Decentralized FedAvg
In this section, we prove that the Decentralized Fed algorithm converges linearly to the global optimum for any smooth non-convex function satisfying PL inequality in the interpolation regime. Compared to the classical Server FedAvg analyses this problem poses several challenges. In particular, unlike Server FedAvg in the decentralized setting each client has access to only parameters from its neighbours. This implies that for Decentralized FodAvg we need to handle two drift terms, namely local drift and the global drift. Local drift refers to the update at each client drifting away from the average obtained from the neighboring clients while the global drift refers to the average obtained from the neighboring clients drifting away from the global average. These two equations are coupled, and hence we use the Lyapunov based approach to show that both drift as well as the loss down lincarly. In addition to the Assumptions
14 our analysis also
relies on the following assumption on the mixing matrix (Koloskova et al., 2020).
Assumption 5. P is symmetric, i.e., P=p². and doubly stochastic, i.e., P1 = 1, 1'P=",,
The above assumption covers all networks that are symmetric, for example, fully connected graph, ring graph, (Hua et al. 2022), etc. In the following, we provide the main result for the Decentralized FedAvg. The details of the proof are presented in Sec. of the Appendix.
Thcorem 1 (Convergence of Algorithm 2). Under Assumptions after T local iterations, choosing 1 O))TT*,, we gel
E (w" +1)
0D 0D,41,u S
17p1 16
E (w")
+ +DD,
Proof: See Sec. B.2 in Appendix and Sec. 3.1 for a proof sketch.
Theorem 1 establishes linear convergence of FedAvg in the decentralized setting. A somewhat related work is Koloskova et al. (2020), where they consider a class of jt-strongly convex functions in the decentralized setting, and showed that a linear convergence rate of (llg(E)) can be achieved by the docentralized FodAvg algorithm in the interpolation regime. Note that in strongly convex setting, all clients share the unique minima x* =0 due the interpolation assumption. In this case, clients do not need to communicate since

---

Published in Transactions on Machine Learning Research (MM/YYYY)
each client can run multiple local rounds to reach the global minima that minimize the average, and hence making ralined or collaborative learning vacuous! On the other hand, for the PT, setting, multiple local rounds may lead to different optimal points, making the problem more challenging. In order to prove linear convergence, the authors in Koloskova et al. (2020) consider bounding a part of the global drift term. More specifically, they consider ||x-
(Lemma 8), where x the average of the local updales and x" the global optimal. In our setting, we cannot consider x* as there is no unique local or global optimal point. Furthermore, they use the property of strong convexity to bound ||x++1 x=1; this cannot be done in our setting. Hence, our analysis is quite different from the existing work. See Sec. C for more details. Next, we characterize the sample complexity of Decentralized FedAvg.
Corollary 1. Under Assumptions + to achieve an e-accurate solution, Algorithm o OTT [log (E($ [)))) number of communiication rounds.
requires R =
Proof: It is clear from equation 48 of Theorem I that 17 scales as देí Thus, from a scaling point-of-view; using 1) = # in Theorem and the fact that (1 x) S E we get
(- C3 (-G,T²
R2
El (w" (1+ 0D,+1.0l S exp
E ( (w²)).
From above we see that to obtain accuracy, we want exp
E ( (w²)) S e. Now rearranging the (w++)) gives the result in the corollary. 7
Ry
above, and using the fact that B[ (w*t1) +D 2 E/
(w"
Corollary 1 shows that even in the decentralized setting, FedAvg can achieve lincar convergence. More importantly the sum of the drift and the loss goes to zero linearly with R, as opposed to most of the existing work {Sun et al. 2021). Observe from Theorem 1 Tand the corollary above that an e-accurate solution can be achieved if the number of global communications rounds R scales as O(T²). For the strongly convex setting since the clients share a unique minima the impact of the local rounds T on the convergence performance is less severe shown in (Koloskova, et al.] 2020). Since the local clients share unique minimum, which allows the algorithm to converge at the same rate to the local optimal irrespective of the local updates and communication protocol used. We believe that the slightly worse convergence of our analysis is due to the non-convex functions satisfying the PL inequality. Importantly, as a special case of our analysis, for the classical FodAvg algorithm (with a server), we establish a convergence rate of O(log(1/c)) which is significantly better than that existing results (see Section 4 Note that one can optimize the number of local rounds that lead to faster convergence. However, this optimization is cumbersome in the decentralized setting, and hence the convergence depends on T.
Effect of Network Topology. The effect of decentrulized clients is captured through the term involving A2 In order to explain the dependency of X2 on convergence, consider the case of T = 1, i.e., FedSGD. Tn this case, if A2 * 0 but closer to 1, then the learning rate is dominated by the term (1 - (1-A2)/ /oonstnnt Thus, the learning rate is small in the decentralized setting (as opposed to x = 0). As a consoquence, (1 - n-///8 is closer to 1 leading to a slower convergence. In the extreme case of X2 = 1, ie., fully disconnected graph, 7 = 0, which leads to divergence, as expected. Later, we perform experiments to show the effect of network topology on the convergence for different network settings.
3.1 Proof Sketch of Theorem
In contrast to the strongly convex setting of (Koloskova et al., 2020), our setting results in each client sharing different optimal points. As consequence, the execution of multiple local updates within each communication round leads to the following drifts: (i) local drift: multiple local updates (see line 4 to line 8 of Algorithm [ differ from the average obtained by the neighboring clients, and (ii) global drift: the local average obtained by using updates from the neighboring clients difler from the global average, i.e., the average obtained from all the clients. This implies that we need to control both global and local drifts, We handle this challenge by bounding the loss in terms of the drift term that captures both local and global drifts as mentioned in the Lemma below. We start by proving an upper bound on the average loss E. T [1]]+ in terms of the loss o (w") in the r-th communication round, and the drift D, D,,,, as shown in
7

---

Published in Transactions on Machine Learning Research (MM/YYN
the following Lemma.
Lemma 1. The average loss is bounded in terms of the drift as follows
1n 2
- (1
1/4
Dr.0.
(1)
8
N
where the drift Dr,u= = E
k=
E
r.
w
000
and n is chosen according to equation 48
Proof: See Sec. B.2
It is easy to see from Lemma 1 that obtain the convergence result established in Theorem provided the drift term on the right hand side of equation is bounded in terms of loss. Towards this, first, we bound the drift. term which depends on the average loss, leading to two coupled equations (see equation and equation 2 We construct a single equation that linear combination of the two coupled equations and show that the linear combination goes to zero exponentially, leading to linear convergence of both drift as well as the loss function. In the following lemma, we provide a recursion of the drift in terms of the average loss and the past drift.
Lemma 2. The drift is bounded in terms of db 1w) as follows
((++)) 22 + 12 L max
11
Dr.0 s
Dr-1,0 + 2n7² 2 man NE [ (w" (w'-1,)],
(2)
where 3 = 46moe(1+a)]
Pmin
Proof: See Sec. B.3
Next, our task to show that the recursion in equation D and equation satisfy bound of the form Φ [ww") 0Dr,0 S o" x ( (w0") 06 (u="), for some 0 > 0 and (0, 1), which is the desired result. Tn summary, prove the bounds loss) in Lemma 2 Note that proving the above bound involves carcfully handling the drift due to multiple local rounds. Using the above results, we constructed a Lypunov function in terms of drift and loss for some 0, and prove the following bound by appropriately choosing the learning rate 77 and constants:
the loss (in terms of drift) in Lemma and the drift (in terms of the
Lemma 3. By choosing 17 as in Theorem 1 for some > 0, we obtain the following
w +1)
0D,
s (1
nP 16
(Φ (ww) + ODr,.)
Proof: See Sec. B.4
The above Lemma shows that the drift and the loss go down to zero exponentially fast (linear convergence) The above approach is quite different from the existing work (Koloskova et al.
and results in Theorem 1 2020
4 Server Setting: FedAvg with Improved Rates
In this section, we show that our analysis specialized to the server setting, i.e., X2 = enables us to show that there exists an optimal number local rounds that maximizes the rate of convergence. In particular, we provide a closed form solution to the optimum number of local rounds, and show that an improved rate of (loo(11/e) can be achieved as opposed the existing work (Koloskova et al. (020) where they show a convergence rate of O (og(T/{)). Further, the same observation made empirically in both server and
8

---

Published in Transactions on Machine Learning Research (MM/YYX)
the decentralized settings (sce Sec. 5
Optimal local rounds: Consider the equation for drift from Lemma 2
Dr.0
Lmaz Dr-1.0 + 2nj² XT L Lm [ (w-=1.))}
(4)
and a bound on the loss function from equation 35 of Sec. B.2
[(1 nuy 64n)
" I I.
4,/T1,
E T w
tt]
E
4
o (wx)
N -o)--)
Pmin
2nT
[(
[16mmam Hmmin
T T
Lmaz > Lmast? Dr.o + 4n2 2yT 11000 (w || (5)
N
It is evident from the above equations that the drift increases with T, as expected. However, a part of the expression in the average loss decreases with T (more specifically, the term (1- )") while the other terms increase with T. In principle, one should be able to characterize the optimal T. However, the above is a complicated expression to optimize with respect to To get more insights into the effect of T, in the following, we look at the server setting, which is special case of our framework. The server setting consists of the central server which coordinates the information sharing among participating clients. We obtain the server sctting by making the second largest eigenvaluc of the mixing matrix, i.c., d2 0 in the decentralized case. Now using the fact that A2 = 0 in equation and equation lead to Dr.0 = 0 and
(11
7/4 T
64 I L2 Limax
E o w t1)]
E p
(w=)
(6)
Pmin
respectively. Choosing equation result in
S G4lmar 12 Lmaz, and utilizing the upper bound e Uninn T
x +
2 for all x 2 in
Tut + 1
(1e (1= E 1 (w=)).
w w1 <
Now we choose such that the right hand side above is minimized, i.e.
nuT + 212 ( _ (5+)]]
inf
Note that the above is convex function. Hence, diffcrentiating the above w.r.t T and equating it to 0, we get.
1 T= Te (n2 + 8)
The above analysis leads to the following "faster' convergence rate for the decentralized setting:
Corollary By choosing the number of local updates such that T Teb
the iterates gen-
7(4488.
crated by Algorithm 1 achieve accurate point after R O (8(2+8) rounds.
log
commanication
#
In the above analysis, we capture the effect of local updates on the performance of the decentralized FedAvg. Specifically, we show that there exists an optimal number of local updates T' beyond which the convergence of the algorithm slows down and one needs to choose T carefully to achieve the optimal convergence guarantees. Again, this is the first result establishing linear convergence of FedAvg in the decentralized setting when minimizing non-convex functions satisfying PL-inequality in the interpolation regime. In the next section, we present the experimental results.
9

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Underparameterized Overparameterized
1.3
Underparameterized Overparameterized
1.6
Underparameterized
14
Overparameterized
Training Loss
1.2
1.0
Training
0.2
Training
0.5
0.6
300
Communication Rounds
50
Communication Rounds
100
150
230
250
300
300
Communication
150
250
(a) CIFAR-10
(b) FMNIST
(c) MNIST
Figure 2: Training loss different datasets versus the communication rounds for FedAvg in the decentralized setting.
1.0
Testing accuracy
0.6
Testing accuracy.
0.9
0.5
Testing
0.7
Underparameterizee
Underparamettrried
Underparameterized
Overparameterized
Overparameterized
0.5
Overparameterized
150
200
300
400
500
50
180
150
200
250
300
350
100
150
200
300 350
# Communication Rounds
Communication Rounds
# Communication Rounds
(a) CIFAR-10
(b) FMNIST
(c) MNIST
Figure Testing accuracy on different datascts versus the communication rounds for FodAvg in the decen- tralized setting.
5
Experimental Evaluation
In this section, we experimentally validate our thcoretical findings for the decentralized versions of FedAvg. First, we present the experimental setup for various settings.
5.1 Setup: Decentralized FedAvg
We use 60 edge devices to run the Decentralized Fe- dAvg algorithm with multiple local SGD steps and then broadcast the updated model with the nodes connected to it. We consider the image classification tasks CIFAR-10, MNIST, and FMNIST datasets using overparameterized simple regression and Deep Neural Network (DNN) models. We have im- plemented all our experiments on NVIDIA DGX A100. The experimental setup consists of the fol- lowing model and data set:
Topologies
centailied
tonus
Training Loss
stachastk
1.0
0.5
0.0
Overparameteriizd regression. Here we con- sider simple regression model with linear lay- ers. There are 231490 trainable parameters with no activation function. We evaluate the performance of Decendralized FedAvg algorithm on the MNIST dataset for an image classification task.
Communication Rounds
Figure 5: Training loss versus the communication rounds for FodAvg in the decentralized sctting. Here, random doubly stochastic case has 5 clients while for others we have used 60 clients.
Deep neural network. We consider an image clas- sification task under two different settings: underparameterizell and overparameterrized settings. In this case,
10

---

Published in Transactions on Machine Learning Research (YYY)
Local iterations
2.2
Local iterations
2.0
Training Loss
10
Training Loss
2.0
10
1.5
50
1.8
1.66
1.4
1.0
0.8
50
100 Communication Rounds
150
200
250
500
#
# Communication Rounds
Simple regression model
CNN model
Figure 4: Effect of T on the convergence of Decentralized FedAvg for simple regression and CNN model.
each device implements a convolutional neural network (CNN) model. We consider the CIFAR-10, MNIST and FMNIST datasets. In the overparameterized setting, each edge device implements a three hidden layer convolutional neural network (CNN) with 256, 128, and 64 filters followed by three linear layers having 1642849 trainable parameters for CIFAR-10 and two linear layers for MNIST and FMNIST with 1046426 trainable parameters. On the contrary, the indepaarameterized setting considers relatively smaller neural network. In this setting, each device implements two hidden layer CNN having 25 and 52 filters followed by two lincar layers for CIFAR-10 and one linear layer for MNIST and FMNIST datascts. We set the number of local updates T = 10 and pick the lunable learning rale in the range E [0.001 : 0.01] for CIFAR-10, MNIST, and FMNIST datasets. We consider that each device has 490 training samples and 90 test samples for the CIFAR-10 dataset. On the other hand, for MNIST and FMNIST datasets, 540 samples are used for training and 80 samples are used for testing.
In this setting, run Algorithm for the following networks (i) ring, (ii) random doubly stochastic, and (iii) torus topologies. For the Decentralized FedAvg, we compare (a) the performance of Decentralized FedAvg with both underparametrized and overparameterized neural network models, (b) effect of topology on the convergence, and (c) effect of local updates on the convergence. In the following, we provide a detailed experimental results.
2.25
Centralized FedAvg
2.00
Decentralized FedAvg
Testing accuracy
1.75
0.7
1.50
0.6
Training
1.25
1.00
0.4
0.75
0.3
0.50
Centralized FedAvg
0.25
0.2
Decentralized FedAvg
# Communication Rounds
100
Communication Rounds
300
400
CIFAR-10
Figure 6: Training loss and Testing accuracy for centralized (A2 = 0) and Decentralized FedAvg algorithm with ring topology (A2 = 0.33) on CIFAR-10 dataset versus communication rounds.
5.2 Experimental Results for Different Settings
Using the settings described above, here we present experimental results for Decentrulized FedAvg's, and corroborate various theoretical findings made in this paper:
11

---

Published in Transactions on Machine Learning Research
DSGDmn
Decentralized FedAvg
(tn")
Training Loss
1.5
db (w d
0.5
0.5
100
200
300
400
500
50
100
150
200
250
300
Communication Rounds
Communication Rounds
Figure 7: Optimality gap for Decentralized FedAvg algorithm with ring topology MNIST dataset versus communication rounds. (see (a)) and Comparison of the DSGDm and Decentralized FedAvg, (see (b)).
Underparameterized versus overparametrized. Fig. 2 shows the plots of training loss of FedAvg in the decentralized setting for mndepparameterizeed and overparamcterized models on MNIST, FMNIST, CIFAR-10 datasets. As established in Theorem 1 the loss of FedAvg in the decentralized setting diminishes rapidly for the overparameterized models compared to the underparameterized models. This is due to the fact that the PL inequality is satisfied for overparametterized systems which helps to reach the global optimum at linear rate as demonstrated by Theorem Fig. show plots for testing accuracy for FedAvg the decentralized setting. As expected the convergence speed of underparameterizeel case is slower than the overparameterizeed case.
Effect of local updates T. Fig. shows plots of the training losses on MNIST dataset for the Fed FedAvg under the decentralized setting on the overparameterized regression model and the CNN, From equation 6 we see that as T incrcases, the convergence speed either decrcases or incrcases depending on the cocfficient of T in the second term. We capture this phenomenon in Fig. 4 In particular, as T' increases, the rate of convergence increases for simple regression model while it decreases saturates for the CNN based DNN model. One plausible explanation is that the smoothness constants of simple regression is small, and hence results in smaller second term in equation 6 However, for CNN based DNN, the second term dominates, and hence results in slower convergence with T.
3. Effect of optimality gap. Fig. (a) shows the plot of the optimality gap, i.c., $ (w=) $ (w") versus R. Here w" is approxinate optimal solution obtained by rumning a centralized algorithm for sufficient number of rounds. As expected the optimality gap decreases exponentially with R.
4. Comparison with the existing work. Fig. 7 (b) shows the comparison of Decentralized Stochastic Gradient Descent with momentum (DSGDm) Lin et al. 2021 with the Decentralized FedAvg algorithm. We show the training loss versus R for everparameterizedd CNN model using MNIST dataset in both the cases. We see DSGDm outperforms the Decentralized FedAvg algorithm as expected.
5. Comparison with different topologies in the decentralized case. Fig. shows the training loss versus the communication rounds R for overparameterized CNN model using MNIST dataset with T = 10 for four different topologics. Since server topology has X2 = 0, it outperforms the network with ring lopology and a random (doubly) stochastic matrix. However, the torus topology does not satisly the conditions required, ie., symmetric and doubly stochastic matrix, and hence cannot be used for corroborating our thcoretical findings. Nevertheless, we have conducted experiments with torus topology, and Fig. 5 shows that the torus has the worst convergence performance. One reason for this could be that the ring topology has more structure, i.e., has symmetric and doubly stochastic mixing matrix P
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
opposed to the torus topology. The theoretical analysis of networks with general topology is relegated to our future work.
6 Conclusion
In this work, we performed a theoretical analysis of the well known FedAvg algorithm for the class of smooth non-convex overparameterized systems in the interpolation regime. We considered the decentralized sctting where nodes communicate over an undirected graph. In this regime, it is well know that neural networks with non-convex loss functions typically satisfy an inequality called LojakkLooasiewiczz (PL) condition. Assuming PL condition, we showed that the FedAvg algorithm achieves linear convergence rate O(T log(L/e)), where is the desired solution accuracy, and T is the number of local SGD updates at each node. As opposed to the standard analysis of the FedAvg algorithm, we showed that our approach does not require bounded heterogeneity, variance, and gradient assumptions. We captured the heterogeneity in FL training through sample-wise and local smoothness of loss functions. Finally, we carried out experiments on multiple real- world datasets to confirm our theoretical observations.
Acknowledgements:: This work in part supported by SERB-CRG (grant number: and SERB-MATRICS (grant number: MTR (TTR/2021000057)) We thank the anonymous reviewers for their helpful comments which have improved the paper.
References
Sai Aparna Aketi, Abolfazl Hashemi, and Kaushik Roy. Global update tracking: A decentralized learning algorithm for heterogeneous data. Advances in Newral Information Processing Systems, 36, 2024.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Raef Bassily, Mikhail Belkin, and Siyuan Ma. On exponential convergence of sgd in non-convex over- parametrized learning. arXiv preprint arXix:1811.02564 2018.
Aleksandr Bexnosikov, Pavel Dvurechenskii, Anastasia Koloskova, Valentin Samokhin, Sebastian U Stich, and Alexander Gasnikov. Decentralized local stochastic extra-gre bataa-ggaaienttt for variational inequalities. Ad- vances in Neural Information Processing Systems, 15:38116-381311 2022.
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. Xiv preprint arXiv:1910.144 14425, 2019.
Farzin Haddadpour, Mohammad Mahdi Kamani, Mchrdad Mahdavi, and Viveck Cadambe. Local sgd with periodic averaging; Tighter analysis and adaptive synchronization. Adv. Neural Inf. Process., 32, 2019.
Roger A Horn and Charles R Johnson. Matriz analysis. Cambridge university press, 2012.
Yifan Hua, Kevin Miller, Andrea Bertozzi, Chen Qian, and Bao Wang. Efficient and reliable overlay networks for decentralized federated learning. SIAM Journal on Applied Mathematics, 82(4):1558 1586, 2022.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradien methods under the balsak-lojasiewiez condition. In ECML PKDD, pp. 795-811. Springer, 2016.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Scbastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inl. Conf. on ML, pp.
PMLR, 2020.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd on heterogeneous data. arXiv preprint arXiv:1909.00 04700, 2019.
Anastasia Koloskova, Tao Lin, Sebastian U Stich, and Martin Jaggi. Decentralized deep learning with arbitrary communication compression. arXiv preprint Xrv:v:1907 09356, 2019.
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In Inl. Conf. on MI, pp. 5381-5393. PMT.R, 2020.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05440 2016.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint vrXio:1907.02160 2019a.
Xiang Li, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Commuicctiion-fffcent local decentralized sgd methods. Xr preprint ar 1rc::119100000 09126, 2019b.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Newral Information Processing Systems, 30, 2017.
Tao Lin, Sai Praneeth Karimireddy; Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accel- crating decentralized deep learning on heterogencous data. arXiv preprint (xXf:22022.4761,, 2021.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Toward theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning. arXiv preprinl arXi::2003.0000 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization pver-parameterized non- linear systems and neural networks. Applied and Compulational Harmonic Analysis, 59:85-115 116,, 2022a.
Wei Liu, Li Chen, and Wenyi Zhang. Decentralized federated learning: Balancing communication and computing costs. IEEE Transactions on Signal and Information Processing over Networks, 8:131 143, 2022b.
Siyuan Ma, Racf Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern pver-parametrized learning. In Int. Conf. on ML, pp. 3325 3334. PMI.R, 2018
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effeieett learning of deep networks from decentralized data. In Artificial intelligence and stats., pp. 1273-1282 PMLR, 2017.
Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies, Shigang Li, and Dan Alistarh. Asyuchronous decen- tralized sgd with quantized and local updates. Advances in Newral Information Processing Systems, 34: 6829-688 0299- 2021.
Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In Int. Conf. on ML, pp. 8119 8129. PMLR, 2021.
Quynh N Nguyen and Marco Mondelli. Global convergence of deep networks with wide layer followed by pyramidal topology. Adv. Neural Inf. Process., 33:11961-11999 2020.
Zhaonan Qu, Kaixiang Lin, Jayant Kalagnanam, Zhaojian Li, Jiayu Zhou, and Zhengyuan uan Zhou. Federated learning's blessing: Fedavg has linear speedup. arXiv preprint 0rXiv:2007.05620 2020.
Tomer Sery, Nir Shlezinger, Kobi Cohen, and Yonina C. Eldar. Over-the-air federated learning from heteroge-
ncous data. IEEE Transactions Signal Processing, 69:3796-383 3811, 2021. doi: TT..0//TSP..021..33.
Bingqing Song, loamnis Tsaknakis, Chung- Yiu Yau, Hoi To Wai, and Mingyi Hong. Distributed optimization for overparameterizee problems: Achieving optimal dimension independent communication complexity. Advances in Newral Information Processing Systems.
Sebastian U Stich. Local sgd converges fast and communicates little. or Xiv preprint or
2018.
14

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Tao Sun, Dongsheng Li, and Bao Wang. Decentralized federated averaging. arXiv preprint arXiv:2104.11375, 2021.
Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking: Momentum acceleration for decentralized deep learning heterogeneous data. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL http:///ppppppoeeee..eeetttttaaa
Jianyu Wang and Gauri Joshi. Cooperative sgd: unified framework for the design and analysis of communictiion-effccent sgd algorithms. arXiv preprint arXiv:1808. 808.0160, 2018.
Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local- update sgd algorithms. Journal of Machine Learning Research, 22((133)1 -0, 2021. URL http: //t////mrr..
org/papers/v22/20- 147. html
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor A novel framework for the analysis and design of heterogeneous federated learning. IEEE Transactions on Signal Processing, 69:5234-5244 2021. doi: 10.1109/TSP:20 2021.311111....1
Shiqiang Wang, Tiffany Tuor, Thcodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal Selected Areas in Communications, 37(6):1205-122) 2019.
Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Memahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In Int. Conf. on ML, pp. 10334-10343. PMLR, 2020a.
Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281-6292, 2020b.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint 1111:22101.11203, 2021.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In Tnl. Conf. on ML, pp- 7184-7193. PMLR, 2019a.
Hao Yu, Sen Yang, and Shenghuo Zhu, Parallel restarted sgd with faster convergence and less communication Demystifying why model averaging works for deep learning. In Proceedings of the AAAT Conference Artificial Intelligence, volume 33, pp. 5693-5700, 2019b.
Anastasia Koloskova Yue Liu, Tao Lin and Sebastian U. Stich. Decentralized gradient tracking with local steps. Optimization Methods and Software, 0(0):1-28, 2024. doi: 01018//00156788.2222222333 URL https: https:///oi..oo htt////i..ogg/..../ 1ttp:/10000000110551111
2024. 2322095
Miaoxi Zhu, Li Shen, Bo Du, and Dacheng Tao. Stability and generalization of the decentralized stochas- tic gradient descent ascent algorithm. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL http::/ppppaeeeeeeeeeeeeeeroeaae
Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized sgd and average- direction sam are asymptotically equivalent. In International Conference on Machine Learning, pp. 43005 43036. PMLR, 2023b.
15


Reviews:
[{"review_id": "HzARRmPEJD", "reviewer": "Reviewer_pU3c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper provides a new analysis of Decentralized FedAvg, focusing on the interpolation regime and non-convex functions satisfying the PL inequality. The federated optimization under PL seems to be a relatively unexplored area, and the results have significant implications for decentralized learning.\n\nWeakness\n\n- The convergence analysis under the PL setting seems to be quite standard, can you highlight any technical novelty in case i missed something?\n\n- Is there any stronger baseline algorithm that you can compare with?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "FedAvg is a well-known algorithm for federated learning. While it has been studied in the centralized setting, the convergence of FedAvg in the decentralized setting is not clear. This paper makes an important contribution to exploiting the convergence of decentralized FedAvg under the PL condition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The comparison between underparameterized and overparameterized settings seems a bit distracting, as it primarily shows the advantages of overparameterized models, which is known before. I am okay if you keep it, but since you are working with the PL setting and linear convergence is expected, could you plot the convergence of the optimality gap f(w^t) - f(w^*), where w^* is an approximate optimal solution? For instance, you could run any centralized algorithm for a sufficient amount of time to obtain this. So far, all the losses seem to be sublinear.", "broader_impact_concerns": "Not available.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ljvBglxHsv", "reviewer": "Reviewer_7fXb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength. Compared to existing literature, the paper extends the analysis in Koloskova et al., (2020) from strongly convex settings to non-convex functions satisfying Polyak--Łojasiewicz condition; compared to the existing work by Song et al., (2022) which also dealt with the PL conditions, the paper allows mini-batched SGD on each machine, as opposed to full gradients. The results are technically new.\n\nWeakness. Given a lot of existing works in this literature, the novelty of this paper is relatively limited. The relaxation from strongly convex functions to PL conditions is well-troden path in optimization literature. Furthermore, there are already existing works under the PL conditions for deterministic setups. In the interpolation regime, extension from deterministic to stochastic settings while keeping the linear convergence requires some technicalities, but is not very surprising.\n\nMoreover, the results in this paper are weak compared to many existing works. The number of communication rounds needs to scale as $O(T^2 \\log (1 / \\varepsilon)/ \\mu)$, which is worse than prior papers in the strongly convex settings. This slowdown is reflected in the $O (1/T^2)$ stepsize requirement in Theorem 1. As the length of inner loops grow larger, the actual move it can make becomes smaller.\n\nIt is not clear to me whether the stepsize choice in Corollary 2 is always attainable. In particular, Theorem 1 requires that $\\eta \\leq \\frac{\\mu}{\\zeta_8 T^2}$, while Corollary 2 sets $\\eta = \\frac{\\mu}{(\\mu^2 + 8) T}$. Comparing the two facts yields $T \\leq \\frac{\\mu^2 + 8}{\\zeta_8}$. Suppose that $\\mu$ is $O(1)$ and the smoothness constants etc are large, we may have $\\frac{\\mu^2 + 8}{\\zeta_8} < 1$, and there is no possitive integer $T$ satisfying this relation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a decentralized federated learning problem where the population objective satisfies the Polyak--Łojasiewicz condition and the individual losses are zero at the population-level optimum (so that no noise at optimum). This corresponds to the ``interpolation regime'' in modern ML models. The paper analyzed decentralied FedAvg algorithm where each device runs a mini-batched SGD algorithm and the information is exchanged every $T$ rounds through a mixing matrix. The paper proved linear convergence with an outer-loop iteration complexity of order $O(T^2 \\log (1 / \\varepsilon))$, where $T$ is the number of steps in an inner loop and $\\varepsilon$ is the desired accuracy level.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The writing needs significant improvements. For example, Theorem 1 has 12 stepsize conditions involving a lot of messy notations. It is not clear which condition is relevant. It would be much better to simplify it significantly by considering the usual scaling of these parameters and removing the irrelevant conditions. Also, it would be better to state the main theorem in the form of final complexity guarantees instead of per-iteration progress.\n\nHere are some minor comments:\n- There are lots of notation conflicts. For example, lemmas indexed as Lemma 1 and Lemma 2 appeared on both page 5 and page 8, and they are completely different lemmas.\n- Table 1: I don't understand ``SC, C and NC represent Server, Decentralized, Strongly\nconvex, Convex and Non-convex settings, respectively.''. It seems that ``Server, Decentralized'' should not be here.", "broader_impact_concerns": "I don't see any potential ethical concerns or broader impact issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "la5TiQJxAU", "reviewer": "Reviewer_ztBN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper discusses a topic of interest for the modern machine/deep learning community. Decentralised optimization is crucial in modern learning systems while interpolation is a practical assumption which can explain many phenomena of fast convergence of SGD.\n\nWeaknesses: The paper does not do a good job in explaining the exact contribution compared to related work. I suspect that this contribution is quite limited. This is because interpolation is well-known to be able to provide linear rates for SGD, while decentralised SGD is also very-well studied. \n\nMy current understanding is the following and I would like the authors to follow up and clarify the situation:\n\n[The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning] gives linear convergence of SGD under the interpolation assumption. \n\n[ A unified theory of\ndecentralized sgd with changing topology and local updates] gives (as a side result) linear convergence of decentralised SGD for overparametrized problems in the *strongly convex* case.\n\n[Distributed optimization\nfor overparameterized problems: Achieving optimal dimension independent communication complexity.] gives linear convergence of deterministic  decentralised GD under the interpolation and PL assumptions.\n\nThe current paper gives linear convergence of decentralised SGD under the interpolation and PL assumptions. In addition, the paper includes the scenario where the nodes communicate after T local iterations (but this is usually quite trivial).\n\nThe paper also claims to have removed some standard assumptions on bounded variance, heterogeneity etc., but I don't see any of them in [The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning]. Could the authors comment on that?\n\nIf my understanding is correct, I think that the paper contains a good to know scenario but the novelty is limited. I leave it to the editor to decide whether this is enough for publication in TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper shows a linear rate for decentralised SGD where nodes communicate after a number of local iterations. In order to show linear rate, an interpolation assumption is used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to clarify my questions and modify accordingly the introduction and related work sections in order to make the contribution clear.", "broader_impact_concerns": "Nothing in particular.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The responses from the authors helped to clarify a few points and also led to improved presentation of the work. The main criticism is the incremental nature of the work, as it puts together several aspects that have been well studied separately (decentralized learning in the interpolation regime, for PL objectives, with local updates). Yet, combining these aspects is relevant and overall, the wok satisfies TMLR's criteria.
Decision: Accept (poster)