On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Authors: Jianyu Wang
Rudrajit Das
Gauri Joshi
Satyen Kale
Zheng Xu
Tong Zhang
Venue: Accepted by TMLR ()

Abstract: Existing theoretical results (such as (Woodworth et al., 2020a)) predict that the performance of federated averaging (FedAvg) is exacerbated by high data heterogeneity. However, in practice, FedAvg converges pretty well on several naturally heterogeneous datasets. In order to explain this seemingly unreasonable effectiveness of FedAvg that contradicts previous theoretical predictions, this paper introduces the client consensus hypothesis: on certain federated datasets, the average of local models updates on clients starting from the optimum is close to zero. We prove that under this hypothesis, data heterogeneity does not exacerbate the convergence of FedAvg. Moreover, we show that this hypothesis holds for a linear regression problem and some naturally heterogeneous datasets such as FEMNIST and StackOverflow. Therefore, we believe that this hypothesis can better explain the performance of FedAvg in practice.

Published in Transactions on Machine Learning Research (06/2024)
On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Jianyu Wang"
Banya, www.. www.gghuu@gmallcomm com
Cornegie Mellon University
Rudrajit Das University of Tezos at Austin
vdas@utteaasco edu
Gauri Joshi Cornegie Mellon University
geariffaendrew.cm.ccom
Satyen Kale Google Research
sutyen@@satennkale.com
Zheng Xu Google Research
zusheng@google.com com
Tong Zhang University of Blinois Artuna-Champaign
tongzhong@tonghanggntt
Reviewed on OpenReview: https: o/penreee...... net/ forun? 2F76Ga4EPs
Abstract
Existing theoretical results (such as /Woowwothh et al. 2020a predict that the performance of federated averaging (FedAvg) is exacerbated by high data heterogeneity. However, in practice, FedAvg converges pretty well on several naturally heterogeneous datasets. In order to explain this seemingly unreasonable effectiveness of FedAvg that contradicts previous theoretical predictions, this paper introduces the client consensus hypothesis: the average of local models updates on clients starting from the optimum is close zero. We prove that under this hypothesis, data heterogeneity does not exacerbate the convergence of FedAvg. Morcover, we show that this hypothesis holds for lincar regression problem and some naturally heterogeneous datasets such FEMNIST and StackOverflow. Therelore, believe that this hypothesis can better explain the performance of FedAvg in practice.
1 Introduction
Federated learning (FL) is an emerging distributed training paradigm (Kairouz et al., 2019 Wang al. 2021), which enables large number of clients to collaboratively train powerful machine learning model without the need of uploading any raw training data. One of the most popular FL. algorithms is Federated Averaging (FEDAvC), proposed by McMahan et al. (2017). In each round of FEDAVG, small subset of clients are randomly selected to perform local model training. Then, the local model changes from clients are aggregated at the server to update the global model. This general local-update framework only requires infrequent communication between server and clients, and thus, is especially suitable for FL settings where the communication cost a major bottleneck.
Due its simplicity and empirical effectiveness, FEDAVG has become the basis of almost all subsequent FL optimization algorithms. Nonetheless, its convergence behavior especially when the clients have
CCurrently Apple.

---

Published in Transactions on Machine Learning Research (06/2024)
heterogeneous data has not been fully understood yet. Existing theoretical results such Woodworth et al. (2020a); Glasgow et al. (2021) predict that FEDAVG's convergence is greatly affected by the data heterogeneity. In particular, when the local gradients on clients become more different from each other (i.c., more data heterogeneity), FVDAVG may require much more communication rounds to converge. These theoretical predictions match with the observations on some datasets with artificially partitioned or synthetic non-IID data (Hsu et al., 2019 Li et al. 2020a; Zhao et al. 2018). However, on many real-world FL training tasks, FEDAVG actually performs extremely well (MMMahan al. 2017 Charles et al. 2021), which appears to be unreasonable based the existing theory. In fact, many advanced methods aimed mitigating the negative effects of data heterogeneity performs similar to FEDAVG in real-world FI, training tasks. For example, SCAFFOLD (Karimireddy et al. (020b) needs much fewer communication rounds to converge than FEDAVG in theory and when run on a synthetic dataset. However, SCAFFOLD and FEDAVG are reported lo have roughly identical empirical performance on many realistic federated datasets, see the results in (Reddi
ct al. 2021
Wu al. 2023 Li ct al. 2021).
The above gap between threotical results and practical observations motivates us to think about whether the current theoretical analyses are too pessimistic about FEDAVG, since most of them only focus the worst cases. It remains as an open problem whether the data heterogencity modeled by theory and simulated via artificially constructed non-IID datasets matches the heterogeneity in real-world applications. Is it possible that the data heterogeneity in practice has some special properties that allow FEDAVG to enjoy good convergence? The answers to the above questions will serve as important guidelines for the design and evaluation of future federated algorithms.
Main Contributions. In this paper, we take the first step towards explaining the practical effectiveness of FEDAVG under a special but realistic hypothesis. In particular, contributions are follows.
By performing experiments on naturally heterogeneous federated datasets, we show that previous theoretical predictions do not align well with practice. FEDAVG can have nearly identical performance on both IID and non-lID versions of these datasets. Thus, previous worst-case analyses may be too pessimistic for such datasets. This is likely because the heterogencity metric used by existing analyses may be too loose for naturally heterogeneous datasets.
In order to explain the practical effectiveness of FEDAVG, we propose a client consensus hypothesis: the average of local model updates starting from the optimum is close zero. For smooth and strongly-convex functions, we prove that under this hypothesis, there is no negative impact on the convergence of FEDAVG even with unbounded gradient dissimilarity.
We further validate that the client consensus hypothesis can hold in many scenarios. We firstly consider a linear regression problem where all the clients have the same conditional probability of label given data, and show that the client consensus hypothesis holds. Besides, we empirically show that natural federated datatsets such FEMNIST and StackOverflow satisfy this hypothesis. Indeed, data heterogeneity has very limited impact these datasets.
We would like to clarify that we are not trying to provide any practical approach/criterion /criterion to predict which datasets FedAvg will converge on. We are just diving deep into the notion of heterogeneity and trying to provide some insights on which notion seems to be more aligned with the behavior of FedArg in practice.
2 Preliminaries and Related Work
Problem Formulation. We consider a FL setting with total M clients, where each client e has local objective function Fe(w) defined its local dataset D.- The goal of FL training is to minimize global objective function, defined weighted average over all clients:
M
F(w) > (w)) (F))),
(1)
2

---

Published in Transactions on Machine Learning Research (06/2024)
where Pe is the relative weight for client c. For the case of writing, in the rest of this paper, we will use [e([ =c P.a.. to represent the weighted average over clients.
Update Rule of FodAvg. FEDAVG (McMahan et al. 2017 is a popular algorithm to minimize I without the need of uploading raw training data. In round of PED,VG client c performs H steps of SGD from the current global model w(O) to a local model w (t.H)) with a local learning rate 17. Then, at the server, the local model changes are aggregated to update the global model as follows:
((++)) = w aBe/w!!) w ((
Here a denotes the server learning rate. Our results in this paper are for the full-device participation case, iee, when all the clients participate in each round. We discuss how our results can be extended to the partial-device participation case at the end of Section 4.2
Theoretical Analysis of Avd When clients have homogeneous data, many works have provided error upper bounds to guarantee the convergence of FEDAVG (also called Local SGD) (Stich) 2019 Yu et al. 2019b;
Wang & Joshi 2018 Zhou & Cong 2018 Khaled et al. 2020 Li et al.] 2019). In these papers, FEDAVG was treated as method to reduce the communication cost in distributed training. It has been shown that in the stochastic setting, using proper H > 1 in FEDAVG will not negatively inffuence the dominant convergence rate. Hence FEDAVG can save communication rounds compared to the algorithm with H 1. Later in Woodworth et al. (2020b), the authors compared FEDAVG with the mini-batch SGD bascline, and showed that in certain regimes, FEDAVG provably improves over mini-batch SGD. These upper bounds on FEDAVG was later proved by Glasgow et al. (2021) to be tight and not improvable for general convex functions.
When clients have heterogeneous data, in order to analyze the convergence of FEDAVG, it common to make the following assumption to bound the difference among local gradients.
Assumption 1 (Bounded Gradient Dissimilarity). There erists positive constant & such that V w Rd, the difference between local and global gradients are uniformly bounded:
Eg VVPwm)) <² <
This assumption first appeared in decentralized optimization literature LLin et al. 2017 Yuan et al. 2016
Assran et al.
2018 Koloskova et al. 2020 Wang et al. 2022), and has been subsequently used in the analysis
of FEDAVG and related algorithms Yu et al. 2019a Khaled et al. 2020; Karimireddy et al. 2020b Reddi
et al.] 2020 Wang et al. 2020a bj Haddadpour & Maludavi, 2019: Karimireddy et al. |020aa] Das et al. 2022
Zindari et al. 2023 Crawshaw et al. 2024).. Under the bounded gradient dissimilarity assumption, FVDAVG cannot outperform the simple mini-batch SGD baseline unless ( is extremely small (C < 1/T where T is the total communication rounds) (Woodworth et al.) (020a)): the deterministic version of FEDAVG (i.c., Local GD) has even slower convergence rale than vanilla GD (Khaled et. al. (2020). Again, these upper bounds match a lower bound constructed in Glasgow et al.) (2021) for general convex functions, suggesting that they are tight in the worst casc. In this paper, we do not aim to improve these bounds, which are alrcady tight. Instead, we argue that since the existing analyses only consider the worst case, they may be too pessimistic for practical applications. The data. heterogeneity induced by the bounded gradient dissimilarity assumption may be different from the real-world heterogeneity.
Finally, we note that there is another line of works, namely, Malinovskiy et al. 2020 Charles & Konečný (2021) Charles & Rush ) ; using a different analysis technique from the above literature. They showed that FEDAVG (with full client participation) is equivalent
performing gradient descent on a surrogate loss function. However, far this technique still has some limitations. It can only be applied to deterministic settings with quadratic (or very special class of) loss functions. Additionally, Wang et al. (2023a) propose a heterogencity-driven Lipschitz assumption to botter capture the effect of local steps, and derive a convergence result. for AVG with this assumption. Gu et al. (023) discuss the reasons why FEDAVG can have better generalization than mini-batch SGD.
Comparisons of Data Heterogeneity Assumptions. In Table 1 we summarize some commonly used data heterogeneity assumptions in literature. Tt is worth highlighting that previous literature, no matter
3

---

Published in Transactions on Machine Learning Research (06/2024)
which heterogeneity measures they used, suggested that only when all local functions are the same share the same optimum, there are additional error terms (i.e., <= 0) caused by data heterogeneity. However, in our paper, we show that even if local functions are heterogeneous and have different local optima, data heterogencity can have no negative impact in some regimes (which likely happen in practice). This new result helps us gain a deeper onderstanding of the great performance of FedAvg observed in practice and cannot be obtained from previous works.
Among all these heterogencity measures, gradient dissimilarity (at optimum) is considered as the most general one and widely used in literature [Wang et al., 2021). But will discuss in the paper, it can be pessimistic in practice. Besides, the gradient diversity assumption used in (Li et al. 2020a Haddadpour & Mahdavi, 2019) implicitly forces all local functions share the same optimum. So it is much more restrictive than the gradient dissimilarity.
Table 1:
Summary of data heterogeneity measures paausurre/ssuumtionsss used literature. The above table is adapted from Table in the survey (Kairouz et al. 2019). In the above works, their error upper bounds have the same dependency on C, though ('s definition different.
Name
Definition
Example usage
grad. dissimilarity
E. V.. FF(w)] S²
Woodworth al.
2020a
grad. dissimilarity at opt. grad. diversity
E.l VF F[l"" <
Khaled et al. [2020
Koloskova et al. 2020
E. VF((w)
al. 2020a
Haddad & Malidavi 2019
general grad. diversity grad. norm opt. diff
Ee (e((w)
12
V w c
Wang el al. (2020
Karimireddy et al. 2020b
(W))ll
< <²
eb al. [019bb
2020b
[B w" <²
Wang et a.. 20236
3
Mismatch Between Theory and Practice on FedAvg
In this section, we will introduce the existing convergence analysis in detail and compare the theory with experimental results. We shall show that there is gap between the theory and practice of FEDAVG.
3.1
Existing Theoretical Analysis of FedAvg
Besides the bounded gradient dissimilarity assumption 3 the analysis of FEDAVG relies on the following common assumptions.
Assumption 2 (Lipschitz Smoothness). There erists consland L such that, Vw, E Rdd Ve,
[[VF(() - (F(ll)]? s L|(w ull?
(4)
Assumption 3 (Local Unbiased Gradient). Local stochastic gradient on each client c unbiased with expectation (ww) and bounded variance d²
Using the above assumptions, previous works derived an upper bound for the optimization error with non-convex objective functions. We take the following theorem from (Jhunjhuwwala al. 2022 an example.
Theorem 1. Under Assumplions the global gradient norm min, E
3 if FEDAVG's 's learning rates satisfy n S 1/8LH, 1/24THH, then ((w)) ()))) 12 can be upper bounded by
(
FM )( F
(
+ + +0(P)2HH()))
o
+ +
(5)
anHT
M
Theorem g shows that when the learning rates a, 17 are fized and the communication round T is limited, data heterogeneity always introduces an addition term O({2) to the optimization error bound. is worth noting that, with an optimized learning rate, the above upper bound matches a lower bound constructed in Glasgow et al. (2021) for general convex functions, suggesting that it is tight. In the worst case, the convergence of
4

---

Published in Transactions on Machine Learning Research (06/2024)
FEDAVG will become worse compared to the homogeneous setting. Although some papers argued that the data heterogeneity only influences the higher order terms when using optimized learning rates (Yu et al.) 2019a; Khaled et al. 2020), this conclusion only holds asymptotically w.r.t. T. Also, in some special cases, FEDAVG's convergence rate with data heterogencity can be substantially slower. For instance, when there is no stochastic noise = 0, F AVG homogeneous selting can achieve a. rate of T-1 but with heterogeneous data, the rate degrades to T-2/3 (Woodworth et al. 2020a Wang et al. 2021 Jhunjhunwala et al. 2022)
3.2 Empirical Observations on FedAvg
Results on Artificial Non-IID Datasets. In order to corroborate the above theoretical results, most of previous works constructed datasets with artificially high data heterogeneity. Given common benchmark classification dataset (such as MNIST, CIFAR-10/100), researchers simulated the data, heterogeneity by assigning different class distributions to clients. For example, each client may only hold one or very few classes of data (Zhao et al.] [2018), or has data for all classes but the amount of each class is randomly drawn from a Dirichlet distribution (Hsu et al. 2019). On these datasets, the empirical convergence of FEDAvG is greatly impacted and its final accuracy is much lower than the one in the centralized training sctting. As shown in Zhao et al. (2018), AvD'' final lest accuracy on non-lID versions of CIFAR-10 can be 10 40% lower than its on the standard IID version.
Results on Natural Non-IID Datasets. While the negative results artificial non-lII) datasets are widely cited to claim FEDAVG suffers due to data heterogencity, we doubt whether the results are representative and general enough to cover all practical applications. Is it possible that there are some scenarios where the data heterogeneity has benign effects and has different characteristics from the heterogeneity simulated through these originally IID datasets?
We conduct some experiments on StackOverflow, naturally non-IID split dataset for next-word prediction. Each client in the dataset corresponds to unique user the Stack Overflow site. The data of each client consists of questions and answers written by the corresponding user. More details about the experimental setup, such as optimizer and learning rate choices, can be found in the Appendix. From the naturally heterogeneous SlackOverflow dataset, we create its ITD version by aggregat ing and shuflling the data from all clients, and then re-assigning the IID data back to clients. Surprisingly, the results in Figure [ show that the convergence of FEDAVG nearly identical (with limited communication rounds) on the new IID dataset and its original non-IID version. This observation contradicts the conventional wisdom that data heterogeneity always adds additional error to the optimization procedure, well as the empirical results artificial lID datasets. There is indeed gap between the theory and practice of FEDAVG.
StackOverflow
0.25
0.20
Accuracy
0.15
0.10
0.05
No IID
0.00
200
400
600
1000 1200 1400 1600 Round
800
Figure 1: Results the naturally non-IID dataset StackOverflow. The IID version of StackOverflow is created via aggregating and shuffling all the data from clients. We observe that FEDAVG achieves roughly the same convergence in both IID and non-IID settings.
Experimental results from some previous papers serve as additional evidence support our claim. For instance, AVG converges significantly faster than FEDSGD Shakespeare and StackOverflow datasets
5

---

Published in Transactions on Machine Learning Research (06/2024)
Local minimum
Local updates
Optimum
Local update s
Optimum
Local minimum
Client drift
Client consensus
Figure Comparison between client drift and client consensus. (Left) Previous works focus on the worst case where the average of clients' local updates (or drifts) drives the global model away from the optimum: (Right) Our client. consensus hypothesis states that the average local drifts at the optinum very small close to zero. This structured heterogeneity does not impact the convergence of FEDAVG.
even with strong heterogencity [McMahan et al. 2017 Charles et al. 2021 J. Even though a more sophisticated method like SCAFFOLD does belter on artificial datasels like EMNIST, it does not outperform FED AvG on many naturally heterogeneous datasets like FEMNIST, StackOverflow, etc., shown in (Reddi al. 2021
Wu ct al. 2023
Li ct al. 2021 Explanations to these observations still remain mysteries.
4 Client Consensus Hypothesis
In previous sections, saw that there is gap between the theory and practice of FEDAVG. While theory predicts that F FVDAVG performs worse in the presence of data heterogeneity, did not observe this on naturally non-IID datasets. Also, many advanced methods designed to tackle data heterogeneity problem do not outperform FEDAVG. In order to explain the above phenomenon, in this section, we propose the client consensus hypothesis. Many realistic federated datasets may satisfy this special property such that data heterogeneity has very limited impacts on the convergence of FEDAVG.
4.1 Key Insights
FEDAVG is commonly viewed as a perturbed version of vanilla SGD. To see this, we can consider the accumulated local updates on each client as a pseudo-gradient, which is an approximation of the batch client gradient. When clients have heterogencous data, the average the very different from the original global gradient. Especially, when the global model approaches the optimum (or stationary point) with constant learning rate, since the average of pseudo-gradients is not zero, the global model camnot slay and may drift away towards different point. This is referred to as client drift problem in literature (Karimireddy et al.) 2020b). An illustration is provided in Figure 2
across all clients can be
The above insight is true in the worst case. But it is possible that real-world FL datasets are far away from this worst case. In this paper, we make a hypothesis about a special class data heterogeneity. clients can still have drastically different data distributions and local minima. But, the global optimum (or stationary point), clients' local updates (i.c., pscudo-gradients) cancel out with cach other and hence, the average pseudo-gradient at the optimum is or close lo zero. As a consequence, when the global model reach the global optimum, it can stay there and will not drift away. That is, clients reach some kind of consensus at the optimum. For this reason, we name the hypothesis as client consensus hypothesis. We will show it in Section 4.2 that this special class of non-IIDness has no negative impacts on the convergence of FEDAVG. Therefore, the hypothesis can serve as a possible explanation of the effectiveness FEDAVG.
particular,
More formally, we define the deterministic pseudo-gradient for client c as follows:
(c(w)) A
(ff) nH C
1
(w
(6)

---

Published in Transactions on Machine Learning Research (06/2024)
where W2GGD denotes the locally trained model at client e after performing H steps of GD from w using a local learning rate 77. Given this definition, the client consensus hypothesis is formally stated below. Assumption 4 (Client Consensus Hypot On real-world federrated dalasels, for the values of v). H used in FEDAVG, the average
(H)
at the optimum (i.e., average client model drift at the optimeum)
p - [
is very small close to zcro, where w is the global optimaum stationary point of the global objective
Note that the Client Consensus Hypothesis is satisfied whenever FEDAVG converges with a constant learning rate. Thus, in a sense, it is the minimal assumption under which one can expect to prove improved convergence rales for AVD P can be interpreted as the average drift al opliman:; this quanlity akin helerogeneity metric and our theoretical results will be in terms of it.
Connections to Existing Analysis with Bounded Gradient Dissimilarity. Here, we discuss the connections between client consensus hypothesis and previous convergence analysis with bounded gradient dissimilarity assumption. As we mentioned before, FEDAVG can be viewed as a perturbed version of vanilla SGD. This alternative view is also critical in the convergence analysis. A key step in previous FEDAVG analysis is to bound the perturbations, i.e., the difference ween the average pseudo-gradient Ee[ge] and the batch global gradient VF(w) E. [V.()]]. Previous works always upper bound this difference using Jensen's 's incquality follows: (G)()) V (w))]] (ell)G(()) (F((1)))
(7)
Then, the right-hand-side (RHS) can be further bounded using the gradient dissimilarity assumption. However, we note that the above upper bound omits the correlations among different clients. While the RHS (i.e., average of squared lg differences) can be large or unbounded, the LHS (i.e., squared Eg norm of the average difference) can be small or just zero, espocially at the optimum w". Therefore, using 7 in the analysis may result in pessimistic estimate.
Instead, in this paper, the client consensus hypothesis states that the LHS of (7) at the optimum is close to, if not, zero. Using the hypothesis and our new analysis, we no longer need to provide an upper bound of the RHS of ( for all points. As consequence, the new analysis does not require the gradient dissimilarity to be bounded.
4.2 Convergence of FedAvg under Client Consensus Hypothesis
In this subsection, we will provide a convergence analysis for FEDAVG under the client consensus hypothesis (Assumption 4 for strongly-convex loss functions. In particular, we show that when p p= in Assumption data heterogeneity has no negative impact on the convergence of FEDAVG.
Besides the pseudo-gradient defined in 6), we need to define its stochastic version:
Go(w)= A nH (w (H)
where w (H) denotes the locally trained model at client using H steps of SGD with learning rate n, starting from w. Let us define 9 = [c.] and G == [G.]. Then, we have the following result.
Theorem 2. Under Assumptions to 4 when each local objective function is p-strongly conver and the learning rates satisfy a < 1/4, min 1/T, 1/T, 1/HH, after T rounds of FEDAVG, we have
E
w (T)
w0
||
< (1 - 2 u)²
(0) - w"
+ 2amH
max Varl
20
20p²
+ 2 max E. (w) +
(9)
42
where E[|,,, Varl-] are taken with respect to random noise in stochastic local updates, and dc(w) ()n(()))((()))))) denotes the ilerate bias between local GD and local SGD on client
(w,
(H)
7

---

11 into 9 and optimizing the Published in Transactions on Machine Learning Research (06/2024)
2 observe that the stochastic noise during local updates inffuences the second and the third terms on the right-hand-side of (9). The upper bounds of these two terms only depend on the dynamics of SGD, which has been well understood in literature. Specifically, in Khaled et al. (2020), the authors show that El)w(()) El < 2n/ Ho?. As a consequence, we directly obtain that
From Theorem
W (H) E[ #1)
20²
Varls(w)]
(10)
122
MH
As for the iterate bias, one can obtain
E.e (1)((())1 S 1,
(11)
the proof of which is provided in the Appendix. After substituting 10 and
learning rates, we can obtain the following convergence rate for FVDAVG
Corollary 1 (Convergence Rate for Strongly Convex Functions). In the same setting as Theorem when a = 1/4, = OlI/lIT), the convergence rate of FEDAVG $
w δ MHT HT2 + =
(12)
If clients perform local GD instead of local SGD, then when 7 min(1/1////// L, l/ult)), we have
w w = o (cxp (- [_imiiff m H} + p²)
T
(13)
16k
where L/p denotes the condition reumber.
Effects of Data Heterogeneity. In the special regime of R 0, i.e., when the client consensus hypothesis holds, Theorem 2 and Corollary [ state that data heterogeneity docs not have any negative impact on the convergence of FEDAVG. However, previous works based on gradient dissimilarity, even if p 0, there is an additional error caused by the positive gradient dissimilarity. Compared to the centralized setting where M 1, 12 suggests that FEDAVG can provide linear speedup due to the usage of multiple workers.
Extensions.
The above analysis can be extended to various settings. Below provide several examples.
(1) Client Sampling: If we consider client sampling in FEDAVG, then only the variance term in (9) will change while the other terms will be unaffected. One can obtain new convergence guarantees by analyzing the variance of different sampling schemes and then simply substituting them into (9) Standard techniques to analyze client sampling (Yang et al. 2020) can be dircctly applicd.
(2) Third-order Smoothness: When the local objective functions satisfy third-order smoothness ([[V=F((w) V F²()( < Q [|w [[wwll),,, ul],, the bound of the iterate bias d(w) can be further improved while all other terms remain unchanged. According to Glasgow et al. 2021)) ) one can obtain [|[(w)]l I 1 HQQ2. In the special case when local objective functions are quadratic, we have Q = 0. That is, there is no iterate bias. As a consequence, the convergence rate of FEDAVG can be significantly improved.
Morcover, a result for the general convex case is provided in the Appendix. Analysis for non-convex functions requires a different framework and non-trivial. So we leave it for future work.
Comparison with Previous Works. In Table 2 summarize the convergence rates of FEDAVG in different papers. All previous results depend on the gradient dissimilarity upper bound Ç, which is generally large for heterogeneous data settings. However, in our result, under client consensus hypothesis, we show that the effects of data heterogeneity can be mcasured by average drift at optimum p, which can be close to zero even in presence of strong data heterogeneily, as we showed in the quadratic example and experiments on FEMNIST and StackOverflow. When the average drift optimum is zero, we can get an improved convergence rate compared existing results.
10(.) hides log Factors.
8

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Comparison with existing results for strongly convex objectives functions with deterministic local updates. In the table, the error measured by the distance the optimum |/ =='l2 and L/j the condition number. Also, we omit logarithmic factors. Compared previous results, we show that in the considered setting: (i) FEDAVG enjoys linear convergence to the global optimum, and (ii) the multiple local steps of FEDAVG mitigate the impact of of ill-conditioning (high condition number).
Algorithm
Worst-case error
Comm. rounds to attain error when p
GD
exp( T/k)
((glog(1)))
FEDAVG
Koloskova et al. 2020
<2/72 1/ +
0(1/²)
FEDAVG Woodworth et al. 2020a
0(1/2)
FEDAVG
(urs)
(1, p
(1, k/H) log(1/6))
5
Validating the Client Consensus Hypothesis
Here we will present some evidence to show that the client consensus hypothesis is realistic and practical; we do so (a) theoretically, in a linear regression setting, and (b) empirically, on some naturally split datasets such as FEMNIST and StackOverflow.
5.1 Linear Regression Example
Intuition on When Client. Consensus Hypothesis Holds. One of the key insights of the client consensus hypothesis is that clients do reach some consensus and a single global model can work reasonably well for all clients though they have heterogencous data. Inspired by this, we assume that all clients have the same conditional probability of the label given data, i.e., pe(y)=r) = p(yl æ)) Ve, where ær denote the input data and its label, respectively. In this case, clients still have heterogeneous data distributions, as they may have drastically different pe(x)) and po( v). However, the client's updates should not conflict with each other, the learning algorithms tend to learn the same x)y]=) on all clients.
Let us study concrete linear regression setting satisfying the above property. Specifically, we assume that the label of the ith data sample on client e is generaled as follows:
Ve. (w" Re,1) + Cc.i:
(14)
where w" E & denotes the optimal model, and 6z.s Pe is a zero-mean random noise and independent from De.i (this is common assumption in statistical learning). We also assume that all ||æ,,حi| have bounded variance. Note that both w" and Pe are the same across all the clients, i.e., all clients have the same label gencration process and hence, the same conditional probability p(y[z). Our goal is to find the optimal model w" given a large amount of clients with finite data samples (which is a common cross-device FL setting (Kairouz et al., 2019)). ). We use the squared loss function which makes our objective function quadratic; specifically, it is:
Fa(w)
(ye, w xr..))
i=1
2 (w w" A.(w - w²) b_ 6w(w= (w- w²) ww) const.,
(15)
where A. = En Te T T/n, b. = E Le,inn The minimizer of local objective Fe(w) is w = w" + A b. which is different from the global minimizer w" as b. + 0.
Client Consensus Hypothesis. In this problem, we can show that client consonsus hypothesis holds. particular, one can show that the average pseudo-gradient at the optimum p is
E.lG.(w")) )) E.
1
-
#-1
(16)
F h=0
9

---

Published in Transactions on Machine Learning Research (06/2024)
Due to the independence of ce,s and Fe,ás for any choice of H, p p= |Ee[Ge("") + 0 almost surely when the number of clients M + co.
Unbounded Gradient Dissimilarity. In contrast, if we check the gradient dissimilarity, we have:
Ee [VF((") VF(w")] Ee l6b.1.
(17)
Observe that e can have extremely large variance such that the gradient dissimilarity bound Ç is arbitrarily large. As a result, existing analyses, which rely on the bounded gradient dissimilarity, may predict that FEDAVG much worse than its non-local counterparts. However, by simple manipulations on the update rule of FEDAVG, one can prove the following theorem.
Theorem 3. Suppose that the weighting of the clients is uniform, and each client has a small finite amount of dala. For the above linear regression selling, the ilerales of Local GD (i.e., deterministic version of FVDAVG) satisfy the following equation almost surely as the number of clients goes to infinity:
w (T) - = E2 (I--)))])"
nAa)"]]
(w - w").
(18)
The proof is relegated to the Appendix. From Theorem it is clear that if the learning rate is properly set such that (I n))) is positive definite, then performing more local updates (larger H) will lead to faster linear convergence rate O(exp(-T)) to the global optimum w*. That is, local GD is strictly better than vanilla GD. However, previous papers based on gradient dissimilarity will get a substantially slower rate of O(1/T9). In this example, while gradient dissimilarity can be arbitrarily large, data heterogeneity does not. have any negative impacts FEDAVG as the client consensus hypothesis holds.
5.2 Experiments on Naturally Non-IID Datasets
In this subsection, we empirically show that the proposed client consensus hypothesis approximately holds across multiple practical training tasks.
In Figure 3 we first run mini-batch SGD on Federated EMNIST (FEMNIST) (MMMahan et al. 2017 and StackOverflow Next Word Prediction datasets (Roddi et al. 2019 to obtain an approximation for the optimal model w² Then we evaluate the average drift at optimum p= E. B. (ww")) and its upper bound as given in on these datasets. We summarize the important observations below.
As shown in Figures 3a and 3b the average drift (or very close to zero on naturally non-III datasets FEMNIST and StackOverflo.. The proposed client consensus hypothesis is true on these two realistic datasets.
at the optimum, i.c. p, is
From Figures 3a and 3b we also observe that there a large gap betweeen p and its upper bound as given in This suggests that previous analyses using this upper bound can be loose.
We run the same set of experiments on a non-III CIF dataset. Figure 3c shows that on this artificial dataset, the client consensus hypothesis no longer holds. The average drfit at optimum p is pretty close to its upper bound and far from zero. This is the scenario where FEDAVG fails with heterogeneous data and faces the client drift problem.
The drastically different observations on FEMNIST, StackOverflow and artificial CIFAR-100 highlight. that there are various kinds of data hetcrogeneity. For the hetcrogeneity satisfying client consensus hypothesis, it may have very limited impacts on the convergence of AVG
Furthermore, run FEDAVG FEMNIST dataset following the same setup (Reddi et al. 2021 and check the difference between the average of secudo-rradeent and the true batch gradient at several intermediate points on the optimization trajectory. For each point, we let clients perform local GD with the same local learning rate for multiple steps. As shown in Figure 4 we observe a significant gap between the norm of average differences (red lines) and its upper bound 7 (i.c., the average of la difference, blue lines). Especially, at the 50th and th rounds, the upper bound is about 10 times larger. These observations suggest that the upper bounds based on the gradient dissimilarity are loose in practice.
10

---

Published in Transactions on Machine Learning Research (06/2024)
FEMNIST
StackOverflow
U1FAR-100, Round 5000
1.6
120
1.4
100
1.2
1.0
0.125
* avg. of L2 diff.
0.100
avg. L2 diff. L2 of avg. diffs.
Blas
L2 of avg. diffs.
0.075
40-
0.4
0.2
20-
avg. L2 diff.
L2 of avg. diffs.
0.0
0
25
50
75
125 150 175 200 Local Steps
100
25
75
100 125 150 175 200 Local Steps
25
50 100 125 150 175 200 # Local Steps
(a) FEMNIST.
(b) StackOverflow.
(c) Artificial Non-IID CIFAR-ION
Figure 3: Difference between the average pseudl-gradient and the global gradient at the optimal point w² three different datasets. We observe that the norm of the average gradient differences at w² (red line) nearly remain zero on all natrual non-llD datasets but its upper bound used in previous analyses 7 (blue lines) slowly become larger when H increases. On a artificial non-IID CIFAR-100 datasct, these two values are pretly close to each other.
FEMNIST, Round 0
FEMNIST, Round 50
FEMNIST, Round 100
25
1.0
17.5
0.8
15.0
12.5
19
* avg. of L2 diff.
10.0
* avg. of L2 dift.
L2 of avg. diffs.
12 of avg. diffs.
10-
7.5
avg. of
12 of avg. diffs.
25 50 75 100 125 150 175 200 Local Steps
50 75 100 125 150 175 200 # Local Steps
25
50
75
150 175 200 Local Steps
100 125
(a) Round 0.
(b) Round 50.
(c) Round 100.
Figure 4: Difference between the average cenudo-gradient and the global gradient FEDAVG's optimization trajectory. There is significant gap between the average gradient differences (red line) and its upper bound (blue lines). Both of them increase and then saturate when incrcasing H.
6 Conclusions
In this paper, aim to bridge the gap between theory and practice of the popular FEDAVG algorithm. We found that previous analyses based on the bounded gradient dissimilarity assumption can be too pessimistic for practical applications. On some natural federated datasets, FEDAVG may have identical performance on both IID and non-IID settings. In order to explain this phenomenon, we introduced the client consensus hepot and formally proved that under this hypothesis, data heterogeneity does not exacerbate the convergence of FVD More importantly, we show that this hypothesis holds for a linear regression problem and many practical federated datasets, including FEMNIST and StackOverflow. So the proposed hypothesis is realistic and can better explain the empirical success of FEDAVG.
Given the above contributions, several future directions are ripe for exploration. Our client consensus hypothesis is expressed in terms of a quantity (p) that is akin to a heterogeneily metric; it would be interesting to come up with a formal metric that applies to all settings and can replace the existing loose gradient dissimilarity metric. Perhaps such metrics could also be helpful in guiding the design principles for FL algorithms. Besides, it may be also worthwhile design a more practical criterion check whether dataset. satisfies the client consensus hypothesis.
11

---

Published in Transactions on Machine Learning Research (06/2024)
References
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Michael Rabbat. Stochastic gradient push for distributed deep learning, arXiv preprint orXiv:1811.1079,, 2018.
Zachary Charles and Jakub Konečný. Convergence and accuracy trade-offs in federated learning and meta- learning. In International Conference on Artificial Intelligence and Statistics, pp. 2575-2583. PMLR, 2021.
Zachary Charles and Keith Rush. Iterated vector fields and conservatism, with applications to federated learning. In International Conference on Algorithmic Learning Theory, pp. 130-147. PMLR, 2022.
Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. Advances in Newral Information Processing Systems, 34, 2021.
Michael Crawshaw, Yajie Bao, and Mingrui Liu. Federated learning with client subsampling, data heterogeneity, and unbounded smoothness: A new algorithm and lower bounds. Advances in Neurol Information Processing Systems, 36, 2024.
Rudrajit Das, Anish Acharya, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S Dhillon, and Ufuk Topcu. Faster non-convex federated learning via global and local momentum. In Uncertainty in Artificial Intelligence, pp. 496-506. PMLR, 2022.
Margalit Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. urXiv preprinl ar arXiv:2111.037/4 2021.
Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local SGD generalize better than SGD? In The Eleventh International Conference on Learning Representations, 2023. URL
httpss///oenrrevie net/forummiddssvCuu..
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXit:1910.14495 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown, Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXic:1909.000 060900 2019.
Divyansh Jhunjhunwala, PRANAY SHARMA, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tackling the variance due to partial client participation in federated learning, In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint 8rXiv:1912.04977 2019.
Sai Pranecth Karimireddy, Martin Jaggi, Satyen Kale, Mchryar Mohri, Sashank J Roddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint ar trxxi:20008.0060 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for on-device federated learning. In Proceedings of the International Conference Machine Learning, 2020b.
A Khaled, K Mishchenko, and P Richtárik. Tighter theory for local SGD on identical and heterogeneous data. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Scbastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning, 2020.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the /CFF/CVF conference on computer vision and pattern recognition, pp- 2021.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of the Conference on Machine Learning and Systems, 2020a.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiu preprint orXio:1907.021800 2019.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang, On the convergence of FedAvg on non-lID data. In International Conference on Learning Representations, 2020b. URL https
//openreview. Cuopenrerie..net//ooummmttaaaaee
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems, pp. 5336 5346, 2017.
Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtarik. From local sge to local fixed-point methods for federated learning. In International Conference Machine Learning, pp. 6692-62 6701. PMLR, 2020.
H Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, et al. Communication-elfiient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Stalislics, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.0022 2020.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. orXiv preprinl 1rXii:1904.00 09237, 2019.
Sashank J. Reddi, Zachary Charles, Manzil Zahoer, Zachary Garrett, Keith Rush, Jakub Koncčný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL
Sebastian Stich. Local SGD converges fast and communicates little. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Jianyu Wang and Gauri Joshi. Cooperative SGD: A unified framework for the design and analysis of communictiion-effiieent SGD algorithms. arXie preprint or arxv::1006 005.07776, 2018.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33, 2020a.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. SlowMo: Improving communication- efficient distributed SGD with slow momentum. In Proceedings of the International Conference on Learning Representations (ICLR), 2020b.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. XrXiv preprint arXiv:2107.0623 2021.
Jianyu Wang, Anit Kumar Sahu, Gauri Joshi, and Soummya Kar. Matcha: A matching-based link scheduling strategy to speed up distributed optimization. IEEE Transactions on Signal Processing, 70:5208 5221, 2022.
Jiayi Wang, Shiqiang Wang, Rong-Rong Chen, and Mingyue Ji. new theoretical perspective on data heterogeneity in federated optimization. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunilies, 2023a.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Jiayi Wang, Shiqiang Wang, Rong-Rong Chen, and Mingyue Ji. Rethinking the data heterogencity in federated learning. In 2023 57th Asilomar Conference on Signals, Systems, and Compulers, pp. 628. LEEE, 2023b.
Blake Woodworth, Kumar Kshitij Patel, and Nathan Srebro. Minibatch vs local sgd for heterogeneous distributed learning. arXiv preprint arXiv:2006.0 735, 2020a.
Blake Woodworth, Kumar Kshitij Patel, Sebastian U Stich, Zhen Dai, Brian Bullins, H Brendan McMahan, Ohad Shamir, and Nathan Srebro. local SGD better than minibatch SGD? In International Conference Machine Learning, 2020b.
Feijie Wu, Song Guo, Zhihao Qu, Shiqi He, Ziming Liu, and Jing Gao. Anchor sampling for federated learning with partial client participation. International Conference on Machine Learning, pp. 374996 PMLR, 2023.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. In International Conference on Learning Represenlations, 2020.
Hao Yu, Rong Jin, and Sen Yang. On the lincar spocdup analysis of communication efficient momentum SGD for distributed non-convex optimization. In International Conference on Machine Learning, 2019a.
Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. (1993-7700 2019b.
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 26(3):1835 1854, 2016.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-IID data. arXiv preprint XrXi 00582, 2018.
Fan Zhou and Guojing Cong. On the convergence properties of k-step averaging stochastic gradient descent algorithm for nonconvex optimization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAT), pp. 2219-3222 2018.
Ali Zindari, Ruichen Luo, and Sebastian Stich. On the convergence of local sgd under third-order smoothness and hessian similarity. In OPT 2023: Optimization for Machine Learning, 2023.
14

---

Published in Transactions on Machine Learning Research (06/2024)
A
Experimental Details
On FEMNIST, StackOverflow, and CIFAR-100 datasets, we strictly follow the training setup given in Reddi ct al. 2020). Both models are neural networks and honce the objective functions are non-convex. In Figure 8
we used a trained model (obtained after training with mini-batch SGD) as an approximation of the global optimum. Then, a large set of clients are selected to perform local model training from the optimum caleulate the gradient bias. Details on the local training can be found in the following table.
Table 3: Details local training in Figure 3
Dataset
Model
Loss function
of clients
Local optimizer
Local learning rate
FEMNIST
ConvNet
Cross Cross-Entrope Entropy
500
GD
0.1
StackOverflow
LSTM
Cross-Entropy
1000
GD
0.5
CIFAR-100
ConvNet
Cross-Entron Eotsoy
200
GD
0.5
B Proof of Theorem 2
B.1 Preliminaries
In this subsection, will first introduce several useful lemmas, which relate to the propertics of the deterministic pseudo-gradients. Before diving into the proof details, would like to first introduce lemma, which will be frequently used in the subsequent sections.
Lemma 1 (Mean Value Theorem). Suppose function F is twice differentiable, then
VF,(w) VF(uu) = Ac(w, u) (w n)
(19)
where Ac(w, u) = So F2F s(w u)) If F 3 L, then it follows that μ = Ac( = L for any
Lemma 2 (Convexity, Smoothness & Co-coercivity). When each local objective function F L- Lipschitz smooth and j--ttrongly convex, for any w, u E Rd, we have
T 1 u||2 < (G(w) - G(u). u) S I | -ull [[((w) - (())ll (g(w) - G(u), w - u)
(20)
(21)
where === )n) and L. = [1 (1 h|/ (1)H)
Proof. Let us first focus on the pseudo-gradient on a specific client c. According to the definition of pseudo-gradient, we have
(H((((w) - Go(u)]
=w u -w (1) - u (1)) =
(22)
ww (II 11--11 1) - ui (1- - 17( V(V (M-1) VF (M-1))
(23)
= (I (H (I u (11-1))
(24)
where 24 follows from Lemma 1 and De is a symmetric matrix satisfying μ - De 1 L. Repeating the above procedure, we can obtain that
H-1
nH (Q((w)) - Gc(u)] = u
II (I-nD(())) ()D( ww)
(25)
km0
I
[I (1))
(w (w-w) w)
(26)
k=0
15

---

Published in Transactions on Machine Learning Research (06/2024)
As consequence, we have
HH-
H (Ge(w) - (c(u), u) =1/w0 u/1²
II (T n ( u),, w
(27)
k=0
Note that, due to Cauchy- Schharr incquality,
H-1
II (I (D ((w -w); 1
II k=0
I --
D/
11 uul
(28)
k=
s - m//e) 11 u112
(29)
That is,
H-1
- (1 mm1) 1|20 =ul²= s
II ( - u) u S -pp)
||0 -ull²
uel²
(30)
k=0
It follows that,
(1 - m/)"
1 1 (1 m1)""
| ulI² s (G.(w) Ge(u), u) -
|1 - -ull
(31)
nH
nH
Taking the average over all clients, we complete the proof of 20
Next, we are going to prove
21 Note that
#-1
w
(H)
(
(I
1jD
(k)
(w
(32)
A=0
H
D (k)
w0
(33)
<(1 - m/p) 112 - 21l
(34)
As a result, we have
( -1)--((0 (1) = | (f) E. Euu!")) mm
-IE ( u ||?
(35)
<[(1 7111 "12 || w/12
(36)
Then, according to the definition of pscudo-gradients, one can obtain
+ (H) 2
x7² 2 (9(ww) 9(u)ll2
w u (H)
(37)
||w u|1 + w (H) r (1)
w u, (H) u
(38)
<(1 (1 (hlii 11
u||² |/
2 w u, w 2a (1))
- (1 - -odi []
(1 - m/) #1 || - ull2
(39)
=[1 + (1 (nlii #1 ||l - -ull² - (w- u, w (H) re (11
(1 - (1 (h1 #) w u w (H) u (H)
(1 -ati- 1 [] (1 (nlll | w/ 2u
|1
(40)
=[1 + (1 - (nlii "1 || - u||² w u, w (1f) u m1)
+ (1 - (1 (nli H) w u|| (w u. (H) - u (1)t)))
- [1 + (1 mu) H
- (1 muu H1 w ull
(41)
16

---

Published in Transactions on Machine Learning Research (06/2024)
Note that nH (w- u, ((w) ((u)) w/w -ul2 ((w u. w -u and nHH 1+ np)" nHH
- (1 m//)* H we have
( s(u)l
S(I + 11) (w - u ( ((u)) Li 11 ull
(42)
= (w u, u.g(w) G(u))
jú [ 11 ull² - (w u, ((w) ((w)))
(43)
<I (w ((w) - ((u))
(44)
where the last inequality is due to the smoothness of the pseudo-gradd gradient 20
B.2
Main Proof
In the analysis below, we first analyze the training progress within one round. Suppose the current global model is un and the next round's global model is w+. Without otherwise stated, the expectation and variance Var are conditioned on the current global model w. For the ease of writing, we define effective learning rate ā an..
First, according to the update rule of FEDAVG, we have
E /
B
w0|]
=E a)(w) w"
(45)
[B()(w)) - w ||
+ + [arl(()w)
(46)
w w01l + ã² |(( 2ā (w - w ([(,[(())) + Varl((w))
(47)
Also, note that [[ÿ(w))] = ((w) - ((w") + g(ww)) + 6(w). So one can obtain
E w w w¹1 < 1100 w0 1| 20 [[( - G(w") )12 - 2 (w w", g((w) - G(w"))
2 [9(w") 5( 2 (w w/, (0==("")) + d(wr) + a² Varl (w)
(48)
5(1 aلj) I w/ 11 + 28 [[( - G(w"))())))))) )1( - ā (w w ((w) ((w"))
2 [9(w") 5(w))1 2 (w w", G(w") + 8(w)) + a²
(49)
where the first inequality uses the fact ||a + b11 < 2 1|a|12 + |bb| and the second inequality comes from the strongly-convexitty of the pseudo-gradiess Now let us check the value of the following terms:
Ti =2 [[(() - G(w"))(" -w-w - w", w", "0w)((w)= ((w")) 2 (w w", "(("") + o(w)).
(50)
According to the co-coercivity of the pseudo-gradient, we have
Ti < 2āL 1 (w - ww"w", (o")((w) - ((w") )) (w - w", (", + d(w))
(51)
< 2aī 1] (w (w-wi w' g(w) ((w")) || w² |[((w)) 5(w)]l
1
(52)
where the last inequality comes from Young's inequality. When aj S aï < 1/4, we have
Ti s ww w0 112 + cl/w w" w0112 1 l1g(w=) + 6(w))12 2 2119(w") 6(w))11
(53)
17

---

Published in Transactions on Machine Learning Research (06/2024)
where the last incquality is obtained by setting fi/2. Then, substituting 53 into 49 ap S al I 1/4 (that is, ā S 1/(47)),
and noting that
2ā
B /u0 w011² <(1 [(j) | w" w011² + 20 +
(9(w") ((w))1 + ā Varl((())
(54)
j
<1 - ā
|| w" w01l3 | +
50
[g(w") ) + 8(w))1 a² VarlS(w)]
(55)
2
à² Var w) ) + 5
5
<1 (i
|| w011 ||
[(6(w) 1
|g(w")||
(56)
( ap)
w w01l 1|
+ ã² max Varlg
5
5a
max 1 w
12
[((w")) |||
(57)
Jt
After lotal T communication rounds and laking the total expectation, we end up with
IF
w (T)
w0
5(1 am)²
w w'
max [arl](w)]
w
5p²
max E. [lōc(w))l
(58)
w0
n12
When nH S 1, one can easily validate that
1 (1 mp) 15
(59)
11
So it follows that
2
1
220 - w 1|
2amH
E
w (T)
<(1
2 p))
max Varlÿ(w)]
20
20p² 12
2 max E. ((
(60)
At last, in order to satisfy aT. < 1/4, one can set < 1/8, such that
1+ (1 - (1 a(1 (1 m)) s 2a 1
aा. onH
(61)
n
C Bound on Iterate Bias
In this section, we will provide an upper bound for the iterate bias we have
According to the local update rules,
w (H) G. Blw! (m)
c,G
(H-1)
(fww!--11) 11 V.
(
(H
n V w 1I
(62)
w C,
(1-1)
Elun!
(111 7) P. ((-1))
V
R
(2 (H
+ 17
[ F:ww!!
11
V F. [1 --1)
(63)
<(1 mpn
w (1-1)
Blw!! H1 11 1>|
+7
(w
V F. (E (H- 111)
(64)
For the second term, we have
EVFF(W! (F.(E(
< VF FF (H-
V (E
(65)
H El (1 ||²
<IºE
W
(66)
<2n 22 111-11 -1)
(67)
18

---

Published in Transactions on Machine Learning Research (06/2024)
where the last incquality comes from previous works Khaled et al. (2020): one can obtain
Glasgow al. 2021
As result,
w (H) G,CD EE|
5(1 m/4))
w
(H-1)
E (H - 11
+ 22n Lo - 1)
(68)
H-1
<v2n/ Lo
mm11 n1 n
(69)
h=0
W-1
<v2n2 Lo
(1 (1
Σ h
(70)
h=0
バ=0
H Σ
<v2nT² Lo
(1 mur -
Σ h
(71)
(1- mu) H7
ry -H
(72)
MuH
According to the definition of 8(w), we obtain
T 22 2 (H -1)
E. [()]l s B. w W.GD
(8)
(73)
4
where the last inequality comes from the fact that S μ.
Proof of Corollary 1
D.1 Deterministic Setting
When clients perform local GD in each round, there is no stochastic noise. So the error upper bound ( simplified as follows
can
w (
w0
S (1 111-20111 pz) (0) - w01 -w0l- + 112
20p²
(74)
2
If Hμ 2 L, then the maximal learning rate is 1/ HH. When = 1/8, the upper bound becomes
w (T) -w""
w0
< 1
1
T
[0 (0)
w ||
20p² +
(75)
16
20p² w"
S exp
(0)00)
+ 122
(76)
16
If Hp & L, then the maximal learning rate is 17 1/L. When 1/8, the upper bound becomes
20p²
w"
(0) w
(77)
16T.
n2
nHT
(0) w² 1|2
20p²
S exp
(78)
16L
112
We can summarize the above two bounds follows:
w (T)
w0
|| Sexp
T
minfk, H1)
|1000) 110000-00
-woll
20p²
(79)
166
222
T
=0 exp
min (ri H} + p2)
(80)
16x
19

---

Published in Transactions on Machine Learning Research (06/2024)
D.2 Stochastic Setting
Substituting the upper bounds for Var[G(w)] and d(w) inlo (77
and setting 1/8.
w (t++)) w0
1| 5(1 - (1))
w (1)
w"
+
a
20²
50 0²
1)
+
5ā
[9((=))]
(81)
M
H
a²
(t))) 2s* ||2
20
320g
L
5p²
S( app
02
(82)
MH
H.H
p
After minor rearrangement, we can get
5p
| [100 2n ||2
5
20
320
E w (1+1)
w
S (1 opp
+ 2
+ 3
(83)
MH
ju
A ffter total T communication rounds,
5p²
5p"
200 2
320a
E w ((+1))
112 I (1 aa))
(0)
w"
(84)
1T2
jMH
72-
12 If we set ā = aF: where In Ainmmx(frop ( rop (o118)))))rrr1(((822)0)10))))
then it follows that
E (t+2)
w²
20p²
80
128002
+ exp (-%)
(85)
122
HH
H+HT2
80 11)
1280o 12
(86)
P2MHT
# H²22
o²
a²
(87)
MHT
HT
where ① hides logarithmic factors.
E Extensions to General Convex Functions
In this section, are going to extend Theorem 2 to general convex settings for deterministic FEDAVG. This extension can help to show that our conclusion "FedAvg can have identical convergence rate in homogeneous and heterogeneous settings" is not only constrained to the strongly convex settings.
When the average client drift at optimum is zero and there is no stochastic noise, we have
|| w 20 = w w² al(w))l
(88)
w' 2a (w
w" ([() ((w"))())))))))))))))())) a² [[[(w) ((w")11
(89)
<(1 am] ||w
w" w011
(1 aï) [g(w) - [[
(90)
L
<(1 - am) || - w w01
a
[[((w) ((
(91)
2L
where the last inequality is due to aT. < 1/2. In the general convex setting, we have = 0. According to the definitions of J. in Lemma it follows that z 0 and = 2/nH. Substituting these values into 91
we
obtain
w w/ w|w w01 on ([[(w) g(w")|1
(92)
4
A After minor rearrangement, we have
((( 9(w")|1 anH
[|
w0112 /w= 20 |13)
(93)
20

---

Published in Transactions on Machine Learning Research (06/2024)
Taking the average from t to t 1, 1,
T
w w $12
(((w(0) 9(w0
(94)
T
anHi
If we set a = 1/4, 1/L, then
16L w (0(0) ==1l
((w - 0(wi))|²
(95)
HT
The above rate the same as GD for general convex functions and data heterogencity does not have negative impacts, as [[(w)] 0.
F Proof of Theorem 3
Proof. According the local update rule, we have
w (th+t) w (t.b) 1/ V/V (V((tt())))))
(96)
(t.h) - Ac(w (f.1)) - w²) be
(97)
(I ( + +1
(A bo).
(98)
Subtracting w w + A, on both sides, it follows that
w (1,++1) w. = (I 11A0) (w!lit) w² = (I --,)))) h+1 (w0 /+t
(99)
w²
(100)
Setting H, we have we we get
(t.H)
(I A). (w)(t) - + w Recall the definition of pseudo-gradient
(6)
(e(
(w) - wl((HH)
(101)
n.
[ (1 -Ao) (l (w - -w w.). According to the global update rule FEDAVG, one can obtain that
1
(102)
nH
w (++1) (t) En
(103)
(t) aB.
- (I- 1)
(I-nnA)))) (wo()) -w:))--w))))
(104)
(1) aB (I (1-(1- ((I-1AA0) (IIn)A.) (wi()) w0)
alE, [(I (I - =1 (w- - -w:)]
(105)
Subtracting w" on both sides and setting a 1, we have
(1+1) w² -Ee (I- nA.)) |1 (w)(0) w0") -Be E. (I- (on-nAA)) )) (w= w:)
(106)
wHG
=Eo (I- n7.)) #1 (w(()) w0) -nHG(w")
(107)
where G(w=") EwGc(
21

---

Published in Transactions on Machine Learning Research (06/2024)
Now, prove that ((w") 0 almost surely as M oo this synthetic problem. According to the definition of A be, we obtain that
((w))=Ba [1I (1 -Ac) '
(108)
=B, (I- (I A A
D Ee
(109)
Since the noise are independent of &o, &a is a zero-mean random variable that depends on client c. Since we have assumed that all ||æم and Eo have bounded variance, we know that B[s21 < co. Since we have a uniform weighting on the M clients, it follows EclEc] = O(1/vM) with probability 1 1-0 (1), and as M + oo, we have [Cc] + 0 almost surely.
Hence, from 107 we conclude that
(1++1) ~² = Eg (I- nAc)
"]]
1+1 (w) w
(110)
almost surely as M oc, which proves the desired result.
22


Reviews:
[{"review_id": "O37CdOJQ66", "reviewer": "Reviewer_Vef9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The theory developed in this paper is elegant and looks fundamental. I think it is likely to be the correct way to explain the performance of FedAvg.\n\n- The argument/logic is convincing, as it not only gives theoretical justification, but also verifies the hypothesis for typical real datasets.\n\nWeakness:\n\n- Some of the technical discussions/interpretations are not very accurate.\n\n- The technical depth seems limited although the focus is on finding the right hypothesis. As a theory result, this might be a minor weakness.\n\n- I’m not sure how fundamental FedAvg algorithm is, and thus this questions the motivation/significance. Alternatively, it would be great if you can justify if you can apply your technique/hypothesis to other related algorithms/problems.\n\n- The non-convex case is not discussed, and even the hypothesis seems to only focus on the convex case (since it concerns the global optimum).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper observes that the FedAvg algorithm performs well on heterogeneous data in practice, but existing theory results (i.e., upper bounds) is very loose and fails to justify this good performance on heterogeneous data. This paper analyzes why the existing analysis is not tight, and proposes a new hypothesis called “client consensus hypothesis”, which basically says that the average change/updates from clients tends to zero if starting from the (global) optimum. This hypothesis is verified on the task of linear regression and on real datasets. The main result is an improved convergence bounds for FedAvg which is independent of whether or not the data is heterogeneous, assuming the client consensus hypothesis holds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In the abstract, when you mention “existing theoretical results”, maybe give some representative reference already here?\n\n- In the abstract, the “client consensus hypothesis” may need to be rephrased, as I don’t understand this sentence from first reading it. a)“average of client updates” — maybe clarify that this is the amount of update to the gradient etc.? b) “on certain datasets” sounds weird, since any arbitrary property can be observed on “certain” dataset. So it might be good to make a stronger claim, such as “on many typical federated datasets”. Or, as a hypothesis which aims to talk about the property itself, you can remove the half sentence “on certain federated datasets”\n\n- “Unreasonable practical effectiveness” — this “unreasonable” may not be a proper word in my opinion. It is true that it performs differently than previous theory results suggest, but this does not mean the practical effectiveness is “unreasonable”, since otherwise it would mean the experiments are not correct (which is not the case). Instead, it is the existing theory that is “unreasonable” (since it does not capture the practical cases well enough). Actually, for the title, it looks better to me if you simply remove the word “unreasonable”, so it reads “on the effectiveness of FedAvg with heterogeneous data”\n\n- Reading your “Main Contributions”, it is mentioned that the hypothesis is satisfied as long as FedAvG converges. Does this mean as long as FedAvG converges, then it converges fast (since your theorem shows that as long as the hypothesis holds then it converges fast)? Is this message/interpretation correct? Please clarify (and possibly reflect in the paper).\n\n- After reading Theorem 1, I’m not sure about many of your descriptions/interpretations: you seem to say previous works, including Theorem 1, says that FedAvg does *not* perform well on heterogeneous data. However, Theorem 1 is only an upper bound, so it only means FedAvg is *not known to be good* for heterogeneous data. This is much different from saying they show FedAvg is bad. Maybe you should rephrase many of your descriptions, to emphasize that a) FedAvg works well in practice and b) existing theorem fails to justify this fact (and their upper bound is very loose for heterogeneous setting).\n\n- Did you formally define or model the data heterogeneity?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cg0DUkceTq", "reviewer": "Reviewer_ZSqV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Many of existing FL papers are based on similar assumptions: Lipschitz smoothness and bounded variance. This paper points out that the assumptions are too pessimistic without considering the cancelling effect of the local gradients. This approach seems novel.\n2. The authors do not only derive better convergence results based on the new assumption, but also verify it in linear regression and FEMNIST and Shakespeare.\n\nWeaknesses:\n1. This paper doesn't propose a new algorithm of FL, but rather analyze the existing effectiveness of FedAvg.\n2. It says that label shift split of CIFAR-100 is pathological, but it is a standard FL dataset. Whether it is pathological is still a question. For example, we could have many different users, and some users have more classes of cats than others, which is totally reasonable. Therefore, I think this paper is a bit overclaim. There are still a lot of cases when FedAvg suffer and existing FL algorithms can improve, and one should just call them pathological just because they don't satisfy the client consensus hypothesis.\n3. The datasets (FEMNIST, StackOverflow) tested are still small scale. \n4. The main contribution revolves around a new assumption of FL which may or may not hold in practice. The authors should add more results on how to verify/falsify the assumption in FL problems.\n5. Since this is an analysis paper, I would expect more comprehensive analysis on more datasets, like classic CIFAR-10/100. Also, why does the main assumption not hold in these label shift settings?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges common theoretical assumptions of federated learning like Lipschitz smoothness and local unbiased gradient. Instead of reusing these assumptions, it proposes client consensus hypothesis, which essentially says that data local heterogeneity will cancel out. Based on this assumption, the paper derives a new convergence bound and verifies this assumption in synthetic (linear regression) and real datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most of the changes have been mentioned above. In addition, I would request the authors to:\n\n1. finish the theoretical proof of eq. (16) and show exactly why it converges to 0.\n2. Remove the usage of \"pathological\" for CIFAR-100 since many people may not agree it's pathological. I think they are just different kinds of data heterogeneity.\n3. Add more analysis on Assumption 4 and show in which cases (covariate shift, label shift, etc.) Assumption 4 is more likely to hold.\n4. Add a table to compare existing assumptions/your assumptions and existing convergence bounds/your bounds (from Appendix A/B).\n5. Compare FedAvg with the centralized setting since the latter is an upper bound. This will further demonstrate that FedAvg is not affected by data heterogeneity.", "broader_impact_concerns": "This is more like an analysis paper and I'm not sure if broader impact is required here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ARCtgvZsgl", "reviewer": "Reviewer_TjzG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The ideas are simple and clear, and should be simple to confirm and reproduce.\n* Research questions are well defined and clearly guide the paper.\n* The paper provides important intuition about when FedAvg can be expected to be effective and when not.\n* The writing is accessible.\n\nWeaknesses\n* Experiments could have been carried out at a larger scale to give a better impression of how often the hypothesis holds in practice over some large set of real problems.\n* The linear regression example is a clear demonstration of the case when the 'client consensus' hypothesis holds. It would be very clear if the paper also gave an equally simple example where the hypothesis does not hold. In general, the paper could benefit from providing the reader a more clear intuition to which cases the hypothesis is likely to hold.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks into an apparent mismatch between theory and practice in federated learning with FedAvg. Current theory suggests that client heterogeneity incurs a strong penalty on the convergence, whereas empirical results on at least some heterogeneous data sets are not improved by shuffling, indicating that heterogeneity is not always a practical problem. The paper proposes a simple condition (that the client gradients average to zero at the optimum, i.e. that FedAvg converges) and show that this leads to better convergence bounds. This is most clearly shown in a simple linear regression example.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I noticed that there is only one paper listed from 2023 in the references. Perhaps there are more recent studies that could be relevant to include?\n\nThe statement that SCAFFOLD and FedAvg have roughly identical performance is supported only by one non-peer-reviewed reference (Reddi et al. 2020). I do not consider this claim to be well enough established. Most other work I have seen, seem to indicate that SCAFFOLD works quite well in many settings, including some real world applications. There are also extensions of SCAFFOLD that improve further upon this, indicating that variance reduction can yield strong improvements. \n\n\"mysterious inconsistency\" I am not sure there is any mystery or inconsistency. But a gap between theory and practice, perhaps.\n\nIs \"simulated through pathological datasets\" an accurate characterization of previous work? In which exact sense can you underpin that most simulation studies that are pathological?\n\nThe second bullet point in the main contribution says \"...whenever FedAvg converges...we prove..there is no negative impact on the convergence...even with unbounded gradient dissimilarity.\" I believe I understand the gist of what you're trying to convey, but I find it a bit unclear. \n\n\"We further validate that the client consensus hypothesis is realistic.\" I'm uncertain about what exactly needs to be validated, given that the hypothesis revolves around the convergence of FedAvg. As far as I know, the criteria for the convergence of FedAvg are well established. I'm open to clarification if there are nuances I may have missed.\n\nQuestion: Is discussing only the full device participation case is a significant limitation of the paper. Presumable the influence of data heterogeneity is significantly stronger in the partial participation case? For cases when your 'consensus hypothesis' does not hold strongly, does partial participation not influence performance?\n\nI respectfully disagree with the characterization of classification dataset with unequal class balance across clients as 'pathological'. In many instances, such imbalances hold practical significance. \n\n\"In order to support the above conjecture...\" I believe it would be beneficial to provide clearer motivation regarding the presence of a gap between theory and practice. While discovering one or more examples demonstrating natural heterogeneity where shuffling doesn't improve convergence isn't evidence that data heterogeneity isn't a practical concern in other scenarios.\n\nI think some effort should be put into examining how often the 'consensus hypothesis' holds in relevant scenarios, or to define more clear criteria for when it holds, or to outline methods to to determine if it holds. The paper seems to suggest that this hypothesis often holds, but there is not much clear evidence to support this.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a compelling analysis of the FedAvg algorithm's performance in federated learning with heterogeneous data. It introduces the "client consensus hypothesis" and demonstrates through both theoretical analysis and empirical validation that this hypothesis can explain the observed effectiveness of FedAvg.

Summary of Reviewer Comments:

- Reviewer TjzG appreciates the clarity and simplicity of the ideas but suggests more extensive experiments and clearer examples where the hypothesis does not hold.
- Reviewer ZSqV finds the approach novel and the verification on real datasets convincing but suggests additional results to verify or falsify the hypothesis in various scenarios.
- Reviewer Vef9 considers the theory elegant and fundamental but notes that some technical interpretations are not very accurate and that the paper lacks discussion on non-convex cases.


Recommended Changes and Authors' Responses:

- The authors have added more recent references and corrected claims about SCAFFOLD's performance.
- The term "pathological datasets" was changed to "datasets with artificially high heterogeneity" to avoid controversy.
- The authors provided additional discussions to clarify the client consensus hypothesis and its applicability.
- Tables comparing existing assumptions and convergence bounds were moved to the main text for better visibility.

Conclusion:
The paper addresses a significant issue in federated learning and provides valuable insights through the proposed hypothesis and its validation. The authors have adequately responded to the reviewers' concerns, making necessary revisions to improve the clarity and comprehensiveness of the paper. Therefore, I recommend acceptance with minor revisions to further enhance the empirical validation and address the remaining technical inaccuracies.
Decision: Accept (poster)