Title: Revisiting Consensus Error: A Fine-grained Analysis of Local SGD under Second-order Data Heterogeneity
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Averaging

Local SGD

Convergence Analysis

Communication Complexity

Data Heterogeneity
Abstract: Local SGD, or Federated Averaging, is one of the most widely used algorithms for distributed optimization. Although it often outperforms alternatives such as mini-batch SGD, existing theory has not fully explained this advantage under realistic assumptions about data heterogeneity. Recent work has suggested that a second-order heterogeneity assumption may suffice to justify the empirical gains of local SGD. We confirm this conjecture by establishing new upper and lower bounds on the convergence of local SGD. These bounds demonstrate how a low second-order heterogeneity, combined with third-order smoothness, enables local SGD to interpolate between heterogeneous and homogeneous regimes while maintaining communication efficiency. Our main technical contribution is a refined analysis of the consensus error, a central quantity in such results. We validate our theory with experiments on a distributed linear regression task.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new analysis of the convergence properties of Local SGD, a.k.a. Federated Averaging, is provided.\n\nSTRENGTHS AND WEAKNESSES:\nLocal SGD-type methods are very important and I appreciate this kind of papers with an in-depth study of the convergence properties.\n\nI would like to see a discussion of the following papers with important insights.\n\n1) In a series of papers [A,B,C], Mangold et al. have studied the properties of Local SGD, showing that linear speedup can be shown with a careful analysis.\n\n2) Variance-reduction is not discussed. In my opinion the Scaffnew algorithm [D,E] is the state of the art of local training type methods. So, heterogeneity / client drift can be tackled using control variates, in the convex setting (the relevance of variance reduction in local SGD-type methods for nonconvex problems remains an open question). So, the real question is how the error of the stochastic gradients behave.\n\n3) In the deterministic setting, the error of Local GD has been characterized in [F]. It depends on the gradients of the functions $F_m$ at the global solution $x^\\star$, which is a natural measure of heterogeneity, also standard in the analysis of SGD. So, I don't think the Euclidean geometry on the primal solutions, as considered in Assumptions 3-5, is the right criterion. It should be more the first-order information on the gradients that matters.\n\n\n\n\n[A] Magold et al. \"Scafflsa: Taming heterogeneity in federated linear stochastic approximation and td learning\" 2024\n\n[B] Mangold et al. \"Refined Analysis of Constant Step Size Federated Averaging and Federated Richardson-Romberg Extrapolation\" 2025\n\n[C] Mangold et al. \"Scaffold with Stochastic Gradients: New Analysis with Linear Speed-Up\" 2025\n\n[D] Mishchenko et al. \"ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!\" 2022\n\n[E] Malinovsky et al. \"Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning\" 2022\n\n[F] Malinovsky et al. \"From local SGD to local fixed point methods for federated learning,\" 2020\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a refined convergence analysis of Local SGD under the second-order heterogeneity and the third-order smoothness settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. This paper provides a more refined analysis for Local SGD compared to Patel et al. [25].\n2. The topic of explaining the unreasonable effectiveness of Local SGD is important.\n3. The amount of work in this paper is sufficient, and the theory is solid and rigorous.\n\n**Weaknesses**:\n\nI have not found any serious weaknesses. Some minor problems (and questions) are as follows:\n\n1. 1-st line of the equation between Lines 1209 and 1210: Should \"$+$\" be \"$-$\" before $2\\langle\\xi_t^m,\\xi_t^n \\rangle$?\n\n2. 1-st line of the equation between Lines 1213 and 1214: Same as Point 1.\n\n3. 1-st inequality of the equation between Lines 1213 and 1214: It is well known that $(x-y)^4 = x^4 - 4x^3y + 6x^2y^2 - 4xy^3 +y^4$. Thus, is it necessary to use Cauchy Schwartz inequality for $\\mathbb{E}[{(\\langle\\xi_t^m,\\xi_t^n \\rangle)^2}]$ here?\n\n4. 3-rd line of the equation between Lines 1229 and 1230: Should \"$=$\" be \"$\\leq$\"?\n\n5. 2-rd inequality of the equation between Lines 1253 and 1254: It uses $||x_t - \\eta \\nabla F(x_t) - x^\\star + \\eta \\nabla F(x^\\star) || \\leq (1-\\eta\\mu)^2 ||x_t - x^\\star||^2$ here. In fact, it is well known that $||x_t - \\eta \\nabla F(x_t) - x^\\star + \\eta \\nabla F(x^\\star) || \\leq (1-\\eta\\mu) ||x_t - x^\\star||^2$ (see [+1]'s Lemma 6). So I want to know whether this trick helps improve the existing bound. If not, why we use it here?\n\n6. 6-th inequality of the equation between Lines 1278 and 1279: Should \"$(\\nabla^2 F_m(\\hat x_t^m) - \\nabla^2 F_m(x_t) + \\nabla^2 F_m(x_t) - \\nabla^2 F(x_t)  )$\" be  $(\\nabla^2 F_m(\\hat x_t^m) - \\nabla^2 F_m(x_t) + \\nabla^2 F_m(x_t) - \\nabla^2 F(x_t) + \\nabla^2 F(x_t) )$? Is one term $\\nabla^2 F(x_t)$ missing?\n\n7. Line 1279: What does \"$\\in m \\mathcal{H}_t$\" mean?\n\n8. The last equality of the equation between Lines 1293 and 1294: The \"$=+$\" is a typo.\n\n9. Line 1295: The $\\eta$ before $\\nabla F_m(x_t^m)$ is missing.\n\n10. Line 1302: Should \"Using Lemma 16\" and \"taking a whole square\" be exchanged?\n\n11. Line 1362: Where the fact $(1-\\eta \\mu)^K\\leq \\frac{1}{2}$ is used?\n\n12. Line 1371: The phrase \"holds the hides\" seems a typo.\n\n13. Line 1394: The definition of $\\gamma_j = j- \\delta(j)$ is inconsistent in the context. Should it be $\\gamma_j = j-\\delta(t)$? In addition, the sentence \"note that the term with $1/\\gamma_{t-1}$ becomes 1 when $t-1=\\delta(t)$\" is also unclear.\n\n14. 3-rd inequality of the equation between Lines 1395 and 1396: Is $\\sum_{j=\\delta(t)}^{t-1}\\frac{1}{j+1-\\delta(t)}$ be upper bounded by $\\ln (t-\\delta(t))$? In fact, I think it is upper bounded by $1+\\ln(t-\\delta(t))$ if using the integrals. In addition, where the fact $..\\leq \\frac{1}{\\eta \\mu}$ before Line 1396 is used?\n\n15. Line 1401: $\\delta(j) = K(r-1)$ is unclear. Should it be \"$\\delta(j) = K(r-1)$ when $K(r-1) \\leq j \\leq Kr-1$\"?\n\n[+1] SCAFFOLD Stochastic Controlled Averaging for Federated Learning, ICML, 2020.\n\nIf the problems I raised above are wrong, please free free to point them out.\n\nQUESTIONS:\n**Questions**:\n\nThis paper gives a more fine-grained analysis of Local SGD. This may lead to different conclusions compared to the existing works in FL. Recently, [+2] showed that sequential FL is better than parallel FL (Local SGD) when the heterogeneity is relatively high, by comparing their bounds of sequential FL with the bounds of Local SGD [+1]. I am curious to know whether the new bounds (established in this paper) will cause a different conclusion compared to [+2]. Could you kindly compare the bounds between sequential FL and Local SGD? If possible (the time of rebuttal is enough), please provide some preliminary experiments (on quadratics or linear regression). I think the comparison can help improve the impact of this paper.\n\n[+2] Sharp Bounds for Sequential Federated Learning on Heterogeneous Data, JMLR, 2025.\n\n**Suggestions**:\n\n1. Figures: It would be better to use PDF format for figures.\n\n2. Comparison with the related works: It would be better to add the tables to present the differences (of the upper and lower bounds) compared to the existing works like Patel et al. [25]'s Table 1. They can be added in an additional page (if accepted) or in the appendices.\n\n3. Assumptions: Currently, the assumption $\\mathbb{E} ||\\nabla f(x;z) - \\nabla F_m(x) ||_2^2 \\leq \\sigma^2$ is seen as one part of Assumption 2. Since this assumption has appeared multiple times in the proofs, I recommend to write it as an independent assumption.\n\n4. Lines 222 and 228: It would be better to use `\\text{MB-SGD}` and `\\text{L-SGD}` in LaTex to improve the readability.\n\n5. Appendices: The current organization of the appendices are not clear, especially the sections of the upper bounds (Appendices E-J). The titles like \"Double Recursion for Consensus Error\" are not friendly for the readers who are unfamiliar with the settings. I recommend to use \"Proof of Theorem 1\" (see [+3]'s TOC of appendices) and put the current titles at the beginning of the corresponding sections to explain the purpose of the sections.\n\n6. Proofs: Adding some sketches of the proofs can improve the readability (see [+1]).\n\n7. Lemma 21: It would be better to add the proof of Lemma 21 for completeness.\n\n[+3] Minibatch vs Local SGD with Shuffling Tight Convergence Bounds and Beyond, ICLR, 2022.\n\n**Score**:\n\n1. Currently I give the score of 4. This is only because I have not yet checked all the proofs (I mainly checked the proofs of the upper bounds), due to the very heavy workload of NeurIPS. I know this is unfair for this good paper, so I will also consider the reviews of the other reviewers.\n2. However, I cannot guarantee a higher score (i.e., 6) for now, as this topic (the unreasonable effectiveness of Local SGD) has been studied for several years and this theory is still limited to strong convexity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the convergence behavior of Local SGD in the presence of heterogeneous data across clients. The authors focus on understanding how second-order heterogeneity affects the performance and communication complexity of Local SGD. The main contributions are:\n\n1. A new lower bound that quantifies how convergence degrades with increasing second-order heterogeneity.\n\n2. A new convergence rate for local SGD.\n\n3. A synthetic regression experiment validating the theoretical predictions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nI like section 6 which presents simple synthetic experiments to show the motivation.\n\n**Major concerns**:\nMy major concerns lie in the theoretical results.\n\n1. Theorem 1 presents a lower bound for **convex quadratic** functions.\n\n(a) Does the same lower bound hold for **any** functions satisfying the blanket assumptions? If not, the authors may consider to state their first contribution more precisely in Section 1 to avoid overclaiming.\n\n(b) In line 151, the authors state that \"when $\\phi_\\star$ is small and $K$ is large, the lower bound is dominated by the term $\\tau B^2 / R$\". I cannot totally agree with it. In the extreme case where $\\phi_\\star = 0$, all the local quadratic functions have the same optimum, so communication is not needed at all. Then, without stochasticity, the convergence rate is lower bounded by the worst conditional number of all the local functions, not $\\tau$, and the convergence rate should be linear, not $\\mathcal{O}(1/R)$. So the statement is somewhat pointless or inaccurate.\n\nAnother case is $\\phi_\\star$ is small but nonzero. Then when $K \\to \\infty$, agents never communicate and the algorithm never converges. This phenomenon, again, is not captured by the term $\\mathcal{O} (\\tau B^2 / R)$.\n\n(c) In line 158, the authors state that \"we extend the construction of Patel et al. by introducing an additional dimension\". I was only able to understand this sentence after reading the appendix, which indicates a need for clearer exposition in the main text. In addition, this sentence does not justify why we need $\\tau$ instead of other quantities, e.g., the worst condition number of all local quadratic functions; it seems the choice of $\\tau$ is arbitrary...\n\n2. The authors claim that Lemma 2 is novel and distinct from existing literature, but I have reservations about this assertion. In a recent line of research on decentralized optimization [1,2], a similar proof technique has been introduced. In particular, those papers consider using graph sequences with finite-time consensus in decentralized optimization. So consensus error will somehow \"disappear\" periodically due to the suitable choice of communication networks. The same proof technique has also been utilized in local updates (see, e.g., [3] and references therein). The authors may consider to compare the current work with this line of research.\n\n(a) Lemma 2 requires $\\eta < 1/H$, but in line 213, the authors state that \"our bound does require setting $\\eta = \\mathcal{O}(1/K)$\".\n\n(b) In line 214, the authors state that \"an alternative bound in Appendix I.3 that avoids this ($\\eta = \\mathcal{O}(1/K)$)\". However, I find the bound in Lemma 32 not informative at all: the consensus error will never vanish under the stated stepsize condition. Lemma 32 simply says that the difference between $x_m^t$ and $x_n^t$ is bounded by that between $x_m^\\star$ and $x_n^\\star$, which is trivially true because $x_m^t$ converges to $x_m^\\star$ as $K \\to \\infty$. If the algorithm does not converge with this stepsize choice, it is unclear what value these results provide.\n\n3. Theorem 2 states the convergence rate of local SGD.\n\n(a) How should I compare Theorem 1 and Theorem 2? Does the lower bound in Theorem 1 match the upper bound Theorem 2? If they are incomparable, then what is the connection between Sections 3 and 4?\n\n(b) Assumption 1 says $\\mu \\geq 0$, but Theorem 1 certainly fails when $\\mu=0$, as pointed out in lines 244-249.\n\n(c) Assumption 1 imposes $Q$-Lipschitz continuity of the Hessian, which, I believe, is referred to as “third-order smoothness” in the paper. However, Theorem 2 does not involve $Q$ at all. Does this imply that $Q$ has no impact on the convergence rate? If so, then what is the purpose of Section 5 then?\n\n(d) In lines 226-227, the authors claim that \"with a small $\\tau$, Local SGD can converge much faster than mini-batch SGD\". I am not convinced by this claim and believe it requires further justification. First, the dominating term in both rates is $\\sigma^2 / (\\mu^2 M K R)$ rather than the exponential term.  As such, the two rates appear essentially the same. the parameter $K$ has different interpretations in Local SGD and mini-batch SGD, so I do not believe the two are directly comparable. Therefore, the additional factor of $K$ in the exponential term of Theorem 2 does not carry much significance in my view.\n\n(e) In lines 227-228, the authors claim that \"when $K \\to \\infty$, the communication complexity of Local SGD for target accuracy $\\epsilon$ and large $K$ satisfies ...\" However, this is misleading. As $K \\to \\infty$, communication effectively ceases, and Local SGD will not converge to the global optimum. Therefore, the limit $K \\to \\infty$ does not reflect a meaningful or practical convergence behavior.\n\n4. In Section 5, the authors restrict the attention to quadratic objectives and provide an upper bound involving $Q$ (defined in Assumption 1).\n\n(a) How should we compare Theorem 2 and Theorem 4. Logically speaking, the bound in Theorem 4 should be no larger than that in Theorem 2. Is that correct?\n\n(b) How should we compare Theorem 1 and Theorem 4? Do the lower and upper bounds match? At least on the surface, they do not appear to align. How, then, should these two results be interpreted? Is one of them not tight, or are both potentially loose?\n\n(c) I do not see the motivation to include $Q$ in the analysis. In other words, why should $Q$ be the correct measure, rather than, e.g., the condition numbers of all the quadratic functions?\n\nIn addition to the technical issues discussed above, I strongly encourage the authors to improve the overall clarity and presentation of the manuscript.\n\n1. The introduction is difficult to follow, as it attempts to present too much information at once, which makes the narrative feel overloaded. I also recommend that the authors present the motivation of the work more clearly without too many technical details, so that readers can easily understand the problem setting and the significance of the results. A clearer and more focused introduction would greatly improve the readability and impact of the paper.\n\n2. On first reading, I expected Section 3 to start presenting the main results of the paper. However, the section opens with an extensive discussion of related literature, which detracts from the focus and makes me really confused about the purpose of this section.\n\n3. The authors frequently reference forward to materials that appear later in the paper. While occasional forward referencing is acceptable, excessive use can hinder readability and is generally discouraged in academic writing. \n\n(a) For example, in the very first paragraph of Section 1, the authors refer to Figure 2, which appears only in the appendix. This raises confusion: if the figure is essential for understanding the setup, it should be included in the main text; otherwise, if it is non-essential, the reference should be removed to avoid disrupting the flow of reading.\n\n(b) A similar issue arises in the second paragraph, where the authors refer to Equation (2) on page 2 before establishing sufficient context. Forward references like this can be disorienting for readers and should be used sparingly. I recommend reordering the content to introduce necessary concepts before referencing them.\n\n[1] Ying et al. Exponential graph is provably efficient for decentralized deep training. NeurIPS, 2021.\n\n[2] Nguyen et al. On graphs with finite-time consensus and their use in gradient tracking. SIAM J. on Opt., 2025.\n\n[3] Alghunaim. Local exact-diffusion for decentralized optimization and learning. IEEE TAC, 2024.\n\nQUESTIONS:\nSome minor concerns are listed here.\n\n1. In line 15, the authors state that \"$F_m$ is the stochastic objective\". This is not accurate: there is no stochasticity in $F_m$.\n\n2. In line 110, the authors state that \"without this condition, some clients may not benefit from collaboration\". This argument is vague to me. In terms of convergence rate, distributed/decentralized methods are almost always slower than the vanilla counterpart. \"Collaboration\" is necessary for other (realistic) reasons.\n\n3. In lines 133-136, the authors mention \"distributed proximal-point methods\" and cite [50]. But [50] proposes a Newton-type distributed method.\n\n4. In line 329, what does it mean by \"monotonic relationship\"? The plot is not monotonically increasing.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript studies distributed optimization using Local SGD with full node participation. The authors present a refined analysis for convex objective functions with high-order smoothness by introducing the second-order data heterogeneity / similarity, providing both upper and lower bounds on convergence complexity. The results emphasize that Local SGD performs well in homogeneous data scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n- The paper presents a new theoretical lower bound for Local SGD under second-order similarity conditions and includes sufficient technical details.\n\n__Weaknesses__\n\n- The significance of the contribution is unclear. The main conclusion—that Local SGD performs well in homogeneous data settings—is intuitive, as model divergence across nodes is minimal in such scenarios, reducing the need for frequent communication. While the analysis is refined, it does not yield fundamentally new insights compared to prior work.\n\n- There are several overstatements. For instance, in line 25, the claim that Local SGD “consistently outperforms other first-order methods” is inaccurate. In heterogeneous settings, methods like SCAFFOLD [19] have been shown to outperform Local SGD. Similarly, in line 33, SCAFFOLD is proven to have advantages over SGD.\n\n- The comparison between Assumption 6 (second-order similarity) and Assumption 7 (first-order similarity) is not entirely fair, as Assumption 6 additionally requires twice differentiability, making it more restrictive in a different sense.\n\n- This work only analyzes the case when the second-order similarity is small, but does not discuss what happens when it is large. For example, in the comparison between Equation (3) and Lemma 2, when \\tau is large, the consistency error bound given by Lemma 2 may not be tighter than that of Equation (3), due to its dependence on critical parameters such as 1/\\mu and K. Similar issues also appear in the discussion of the main results. This one-sided analysis further limits the significance of the work—especially given that data heterogeneity is common in federated learning scenarios.\n\n- Theorem 2 (and other theorems) does not clearly display the dependency on \\tau. Upon closer inspection, the term R (related to \\tau) appears in the denominators of multiple terms, implying that when data heterogeneity is small, the steady-state error actually increases. However, the authors only highlight the positive effect on convergence speed.\n\n- The experiments only show the communication efficiency of Local SGD, without evaluating its impact on sampling or iteration complexity—particularly relevant since it may increase the steady-state error.\n\n- The experiments are conducted only on synthetic settings, lacking real-world datasets to adequately support the theoretical claims.\n\nQUESTIONS:\n- Comparing Figures 1a and 5, it seems that as R increases, the average best-final error gap narrows. If R grows further, will the convergence errors of Local SGD continue to converge?\n\n- The algorithm assumes full node participation, which limits scalability. What challenges would arise if node sampling were incorporated?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper explores the advantages of Local SGD over mini-batch SGD in heterogeneous settings. The authors derive new upper and lower bounds on the convergence of Local SGD and demonstrate its superior communication complexity, particularly in settings with low second-order heterogeneity.

STRENGTHS AND WEAKNESSES:
Strengths:
This paper makes a valuable theoretical contribution by clarifying why Local SGD can outperform mini-batch SGD—a phenomenon widely observed in practice but not well-established in theory. The work is also clearly written.

Weaknesses:
some assumptions are quite restrictive, such as requiring the objective function to be strongly convex (μ>0).

QUESTIONS:
One of the paper's key contributions is presented as removing the assumption of uniformly bounded first-order heterogeneity. However, prior work in similar convex settings, such as Koloskova et al. (2020), also avoids this assumption, instead using bounded first-order heterogeneity at the optimum.
Given that this paper also appears to rely on a similar condition (Assumptions 4 and 5), could you clarify how your contribution differs from what could be derived from these existing results?

I would be happy to reconsider my score after clarification in this regard.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03