Title: Unveiling the Power of Multiple Gossip Steps: A Stability-Based Generalization Analysis in Decentralized Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: stability and generalization; DSGD; Multiple Gossip Steps; generalization error; excess error
Abstract: Decentralized training removes the centralized server, making it a communication-efficient approach that can significantly improve training efficiency, but it often suffers from degraded performance compared to centralized training.
Multi-Gossip Steps (MGS) serve as a simple yet effective bridge between decentralized and centralized training, significantly reducing experiment performance gaps. 
However, the theoretical reasons for its effectiveness and whether this gap can be fully eliminated by MGS remain open questions.
In this paper, we derive upper bounds on the generalization error and excess error of MGS using stability analysis, systematically answering these two key questions.
1). Optimization Error Reduction: MGS reduces the optimization error bound at an exponential rate, thereby exponentially tightening the generalization error bound and enabling convergence to better solutions.
2). Gap to Centralization: Even as MGS approaches infinity, a non-negligible gap in generalization error remains compared to centralized mini-batch SGD ($\mathcal{O}(T^{\frac{c\beta}{c\beta +1}}/{n m})$ in centralized and  $\mathcal{O}(T^{\frac{2c\beta}{2c\beta +2}}/{n m^{\frac{1}{2c\beta +2}}})$ in decentralized).
Furthermore, we provide the first unified analysis of how factors like learning rate, data heterogeneity, node count, per-node sample size, and communication topology impact the generalization of MGS under non-convex settings without the bounded gradients assumption, filling a critical theoretical gap in decentralized training. Finally, promising experiments on CIFAR datasets support our theoretical findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the efects of using Multi-Gossip Steps (MGS) in decentralized learning. From experimental records, MGS seems to be an effective method in improving performance and efficiency in decentralized learning, and a central question is to study whether MGS can make decentralized learning as good as the centralized version. \n\nIn this paper, they want to answer from the theoretical perspective why MGS works better by studying the generalization error and excess error of the MGS.They contribute an upper bound of the generalization error as sum of a term $I_1$ related to model stability and a term $I_2$ related to optimization error. Moreover, they attempt to compare the MGS with the centralized learning scheme when the number of gossip steps is large. They discover that the error bound obtained by taking the number of gossip steps to be large does not match the centralized SGD. If the error bound is tight, then this result indicates a negative answer to the centrall question aforementioned.  \n\nFrom these bounds they also offer suggestive strategies to practitioners on how to reduce generalization error, including increasing the MGS steps, decreasing the learning rate etc. These findings are also validated through experiments on the CIFAR datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper tackles an important and fundamental problem in decentralized federated learning from a theoretical perspective. While much of the literature focuses on optimization error, this work takes a significant step by analyzing the generalization error of Decentralized SGD (DSGD) under MGS, which is of broad interest to the community and believed to be challenging.\n\nThe authors present their main claims in a reasonably clear and accessible manner. Notably, their analysis goes beyond the claimed focus on the number of gossip steps but also explores the influence of other key hyperparameters, providing a more comprehensive understanding of DSGD behavior.\n\nAlthough I have not verified the proofs in the appendix line by line, the results appear to be well-supported under the stated assumptions, and the bounds themselves seem plausible. The paper also offers thoughtful discussions on the implications of their findings, including practical takeaways and insights that could inform future work or empirical practices.\n\nSome of the claims initially seem counterintuitive, but the experimental results are convincing and effectively reinforce the theoretical conclusions. To the best of my knowledge, this is the first work that directly studies the generalization behavior of DSGD with message gossiping, and it presents a well-rounded treatment of this question.\n\nOverall, this is a solid and insightful contribution, and I believe it opens up several promising directions for follow-up work. I recommend acceptance to NeurIPS.\n\nQUESTIONS:\nWhile I understand the rationale behind the claims in Remark 5 as derived from your bounds—and I appreciate that they are supported empirically—I wonder if there are more intuitive explanations for choosing strategies 2 and 6. Specifically:\n1. It would be helpful to provide more insight into why a smaller learning rate improves generalization error. The explanation around Lines 280–281, while mathematically sound, appears closely tied to the specific form of the bound and may not provide a clear, generalizable intuition for practitioners.\n2. I am also curious about the recommendation to increase $m$. Intuitively, increasing $m$ makes the algorithm more decentralized and arguably further from the centralized baseline, which one might expect to hurt performance due to increased communication noise or model divergence. Some discussion reconciling this intuition with the observed benefit in your bounds would strengthen the impact of the result.\n\nMinor:\n1. In Line 142, do you mean $S^{(ij)}=(S_1,...,S_{j-1},S_j^{(i)}, S_{j+1},...,S_m)$. \n2. In Line 211, I guess the second $\\theta_k^{(T)}$ is supposed to be $\\theta_k^{(T)}(i,j)$?\n3. In Line 234, from Lemma 3, it seems to miss a square root.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on DSGD-MGS which is a Decentralized Stochastic Gradient Descent (DSGD) algorithm with Multiple Gossip Steps (MGS) after each model update step. It establishes the generalization error and excess error bounds for the DSGD-MGS algorithm in non-convex settings without the bounded gradients assumption. The paper established theoretical results showing that MGS can exponentially reduce the generalization error bound but however can not recover the performance of the centralized SGD algorithm. The paper also outlines the influence of various factors such as dataset size, number of nodes, spectral gap, MGS count, learning rate etc on the generalization error. The paper validates the theoretical findings with empirical experiments on CIFAR-10 dataset  (non-IID, $\\alpha=0.1$) trained on LeNet with 50 clients.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper claims to be the first to establish the generalization error and excess error bounds for the DSGD-MGS algorithm in non-convex settings without the bounded gradients assumption.\n2. The paper clearly mentions all the assumptions and theoretically shows the mechanism by which MGS enhances the generalization performance of decentralized training models through an exponential reduction $O(e^{\\frac{-\\delta \\gamma Q}{4}})$ in optimization error.\n3. The paper relies on on-average model stability to remove the bounded gradient assumption making the theoretical results more general.\n4. The paper clearly presents the remarks for the theorems explaining the impact of various terms on generalization error. The paper also makes recommendations regarding the hyper-parameters such as number of nodes, spectral gap, learning rate etc that will result in reduction in the generalization error based on the theoretical results.  \n5. The paper present experimental results, on a small scale dataset and model architecture i.e., CIFAR-10 with Lenet, validating the theoretical claims. Ablation studies are presented in Appendix (Figure 4 and Figure 5). \n\nWeaknesses:\nNo major weakness.\n\nQUESTIONS:\n1. What are the dataset, model architecture and number of nodes used for Figure 1 plot? It is important to highlight the number of nodes in Figure 1 as it indicates a practical upper bound for number of MGS steps. \n2. Spectral gap is defined as $\\delta = 1- |\\lambda_2|$ which indicates that larger spectral gap results in better connectivity with $\\delta =1$ for fully connected graphs. However, remark 5 indicates to use graphs with smaller spectral gap to reduce the generalization error bound. Also, for fixed graph structure (say ring), increasing number of nodes $m$ (reducing $\\delta$) will result in poor performance empirically but the remark 5 suggests to increase the number of nodes. This is a bit counterintuitive, can authors shed some light on point 2 and 5 of remark 5?\n3. Consider an decentralized setup with n nodes and doubly stochastic weight matrix (say undirected ring topology) and full communication in each MGS. In this scenario, when we set number of multi-gossip steps to n at each iteration, DSGD-MGS should recover the same performance as centralized SGD. In the extreme case, one could have a fully connected network with DSGD-MGS and this should be able to recover centralized SGD generalization performance at least for linear neural networks. However, the paper claims that decentralized training cannot fully achieve the generalization performance of centralized training solely by increasing the number of MGS steps. Why is this the case? In particular, what is the bottleneck here when functionally DSGD with fully connected graph ($\\delta=1$) is same as centralized SGD? \n4. Can the authors add centralized SGD baseline to Figure 2?\n5. $\\gamma$ is introduced in Lemma 2 but not described till Theorem 2. Please include $\\gamma$'s equation in Lemma 2 itself.\n\nLIMITATIONS:\nMulti-step gossip comes at the additional cost of communication which is a major overhead for decentralized settings that will limit the practical application of the DSGD-MGS method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents analysis of multiple gossip steps (MGS) approach to decentralized SGD, i.e., a framework in which several rounds of model averaging are performed between stochastic gradient updates. The results include on‑average model stability bounds that explicitly depend on the number of gossip steps and the spectral gap, demonstrating that the error decays exponentially -- in non-convex settings, without making assumption of bounded gradients, and allowing for heterogeneous data distributions. Empirical results on CIFAR‑10 align with the theory, showing that increasing the number of gossip steps significantly improves test accuracy. While MGS tightens the decentralization-induced gap, the analysis shows that a residual asymptotic generalization gap relative to centralized SGD remains even as the number of gossip steps grows to infinity.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and technically solid -- as far as I can tell, the presented derivations are correct. However, the contributions are somewhat incremental and the framework raises some concerns, as outlined below:\n\n(1) Previously, Le Bars et al. [ICML '24] used a technique for deriving optimization-dependent generalization bounds for DSGD. The paper under review applies this technique to the analysis of gossip-style local averaging of SGD updates.\n\n(2) The MGS framework assumes sampling one point from a local dataset, computing SGD, and then running multiple gossip steps. However, in addition to inducing latency, the cost of repeatedly communicating the model can be rather expensive -- both from the bandwidth as well as energy perspective. For instance, Savazzi et al. [An Energy and Carbon Footprint Analysis of Distributed and Federated Learning, '22] argue that in many cases communication exceeds computation cost. It may therefore be more natural if the nodes perform mini-batch processing in the same vain done by the centralized scheme which is anyways used as a benchmark.\n\n(3) Performing MGS is predicated upon the assumption of networks with stable links yet in practice the nodes may be available only intermittently. Moreover, in any given round, the communication may be asymmetric -- if A communicates with B, B need not necessarily be able to compute an update and communicate with A. This would violate the doubly-stochastic gossip matrix assumption. It is not clear if/how the current analysis may be extended to such more realistic scenarios.\n\n(4) Theorem 2, a main result, relies on the PL condition yet abstract/introduction do not mention this -- the way they are written now, a reader gets an impression that the results are more general. If indeed necessary, the assumption that PL condition holds should probably be listed along with Assumptions 1-3 in the main paper, rather than given in the appendix.\n\n(5) Experimental results are somewhat limited, considering only CIFAR-10 and LeNet.\n\nQUESTIONS:\n1) Can the theoretical analysis be extended to the case where local nodes perform mini-batch updates, followed by MGS? If so, the local batch size would become a variable of interest.\n\n2) Even if theoretical analysis appear challenging, an experimental study of the interplay between communication and computation (i.e., (M)GS + local mini-batch) would provide beneficial insight.\n\n3) Can PL condition be relaxed or is it necessary?\n\n4) Could the framework be extended beyond synchronous and symmetric gossip?\n\n5) Broader empirical evaluation would be appreciated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}