Title: Sharp Gaussian approximations for Decentralized Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Distributed Systems

SGD

Gaussian approximation
Abstract: Federated Learning has gained traction in privacy-sensitive collaborative environments, with local SGD emerging as a key optimization method in decentralized settings. While its convergence properties are well-studied, asymptotic statistical guarantees beyond convergence remain limited. In this paper, we present two generalized Gaussian approximation results for local SGD and explore their implications. First, we prove a Berry-Esseen theorem for the final local SGD iterates, enabling valid multiplier bootstrap procedures. Second, motivated by robustness considerations, we introduce two distinct time-uniform Gaussian approximations for the entire trajectory of local SGD. The time-uniform approximations support Gaussian bootstrap-based tests for detecting adversarial attacks. Extensive simulations are provided to support our theoretical results.


Peer reviews:
comments: SUMMARY:
This paper investigates the statistical properties of local SGD in decentralized federated learning (DFL) settings, focusing not only on convergence but also on refined distributional approximations. The authors propose Berry-Esseen bounds for final iterates and develop time-uniform Gaussian approximations to enable inference and adversarial attack detection. Two bootstrap frameworks, Aggr-GA and Client-GA, are introduced, and extensive simulations validate the theoretical results.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1.	This paper addresses an important gap in the statistical understanding of local SGD beyond convergence. 

2.	It provides novel Berry-Esseen bounds and time-uniform Gaussian approximations. The proposed approximations support practical applications such as bootstrap inference and attack detection. Solid empirical validations that match theoretical insights.

3.	Both centralized (Aggr-GA) and decentralized (Client-GA) solutions, can balances accuracy and privacy trade-offs. 

**Weaknesses:**

1.	While simulations are extensive, real-world experiments (e.g., NLP or vision FL benchmarks) are lacking.

2.	Presentation could be improved, the notation and organization are occasionally dense and hard to follow.

QUESTIONS:
1.	Are the results still valid under significant client heterogeneity?
2.	All analysis assumes a fixed step size, can the results extend to adaptive or momentum-based variants, e.g. Adam?

LIMITATIONS:
The paper lacks real-world experiments on standard federated learning tasks (e.g., image or text classification).
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies stochastic gradient descent in the framework of decentralized federated learning, and provides a non-asymptotic versions of the Central limit theorem for the convergence of the model parameter to global optimum. In Section 2, the authors generalize Berry-Essen-type bounds for the last iterate of SGD as well as for the Polyak-Ruppert average to the case of federated learning. The proof of this section are based on the framework of randomized concentration inequalities by [Shao and Zhang, 2022].\n\nIn Section 3, the authors present two ‘time uniform’ approximations of SGD iterates averaged across clients. That is, they show that strong approximations in the form of Gaussian processes with explicit covariance structures depending on the parameters of the optimization problem can be constructed. The authors derive stochastic bounds on the deviation of the original SGD process from these approximations. The authors motivate their studies of ‘time uniform’ approximations by detecting anomalies in the SGD trajectory\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths***\n\nThe authors obtain novel theoretical results in the federated learning setting. To the best of my knowledge, this is the first time Berry-Eseen type bounds have been proven for the federated learning setup. The considered strong approximation result for SGD training trajectories by a Gaussian process is also novel, to the best of my knowledge.\n\n***Weaknesses***\nTheorem 2.2 in its current form is unclear to me. First, a well-known asymptotic normality results on the last iterate of SGD suggest that one should consider $(\\theta_n - \\theta^\\star) / \\sqrt{\\eta_n}$ in order to have a non-trivial limiting variance. Hence, I guess, there is a normalizing factor $n^{\\beta/2}$ missing in the left-hand side. Second, the claim in lines 208-209:  \"In particular, this rate is slower than the corresponding rate for the Polyak-Ruppert averaged version\" is incorrect - when setting $\\beta = 1$, you have exactly the same rate $1/\\sqrt{n}$. The problem here is not in the rate, rather in the realizability of the corresponding step sizes (in such a case $\\eta_k = c/k$ with some lower bound on the constant $c$, which is instance-dependent). Another issue with the last iterate is that it yields a suboptimal limiting covariance matrix, unlike the Polyak-Ruppert procedure.\n\n***Minor points***\n1. Statements of \"informal\" theorems are too vague, especially the one of theorem 1.1. I would indicate arguments in $d_{Berry-Esseen}$, or at least mention in the statement that the Polyak-Ruppert averaged version of SGD is considered;\n2. There is a clash of notations related to $A$. It is used both for generic matrix in notations, generic convex set in the definition of the convex distance (see eq. (2.3)), at the same time, due to Theorem 2.1, it has a separate meaning (Hessian at the optimum). Please change. \n3. Vectors $g_k$ are intensively used in the statement of theorem 2.1 and below, but are not defined in the main text. From the context I can infer that $g_k (\\theta, \\xi) = \\nabla f_k (\\theta, \\xi)$. Please fix this discrepancy;\n4. Line 188, still statement of Theorem 2.1: \n***assume that $\\max_{k \\in [K]} \\mathbb{E}[|g_k * (\\theta_K^\\star,\\xi)|^2] = \\mathcal{O}(1)$***\nFirst, there is a typo here, you do not need $*$. Second, writing here $\\mathcal{O}(1)$ is very confusing. Please write the assumption you impose formally.\n5. Appendix A, line 989: \"Assumption A.3 is much weaker compared to assumption A2(p) in [Sheshukova, 2025]\" It is not true, Assumption A.3 is exactly assumption A2(p)- (iii) in the mentioned paper. At the same time, assumption A.2 in the current paper is indeed weaker than the a.s. Lipshitzness imposed in A2(p)-(ii) in [Sheshukova, 2025]. \n5. What is $E$ in the line 1054 in appendix? \n6. There is a $\\nabla$ symbol missing in assumption A.2, see line 979\n7. Some constructions are clearly too verbose. As a particular example: line 257, “Hence posing a hindrance to performing” --> why not just \"hindering\"? As I have already mentioned in the weaknesses section, I recommend the authors to re-read the main text carefully and update the ambiguous or verbose phrasing.\n\nQUESTIONS:\n1. What is the effect of data heterogeneity across different clients in Theorem 2.1? In the current version, I guess, it is completely hidden in the constants, at the same time, it will be very interesting to quantify, how this factor affects the non-asymptotic CLT.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents sharp Gaussian approximation results for local SGD in decentralized federated learning (DFL), going beyond standard central limit theory to enable statistical inference with finite sample guarantees. The main contributions are a Berry-Esseen theorem for both the final and Polyak-Ruppert averaged iterates, and two distinct time-uniform Gaussian approximations (Aggr-GA and Client-GA) for the entire trajectory of local SGD updates. The theoretical results are validated through extensive simulations demonstrating improved approximation rates compared to traditional methods and revealing key computation-communication trade-offs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Strong technical novelty in deriving first Berry-Esseen bounds and time-uniform Gaussian approximations for local SGD\n- Rigorous theoretical analysis with comprehensive proofs\n- Extensive empirical validation of theoretical claims through well-designed experiments\n- Clear distinction from prior work by going beyond central limit theory\n- Direct implications for statistical inference and anomaly detection in DFL\n\nWeaknesses:\n- Limited discussion of computational complexity of the proposed methods\n- Lacks ablation studies comparing different variants of the Gaussian approximations\n- Could benefit from more real-world applications beyond synthetic experiments\n- Synchronization effects could be explored more thoroughly\n- Some assumptions like strong convexity may be restrictive\n\nQUESTIONS:\n1. How do the approximation rates scale with non-IID data distributions across clients?\n2. What are the computational trade-offs between Aggr-GA and Client-GA in practice?\n3. How robust are the results to violations of the strong convexity assumption?\n4. Could the methods be extended to non-convex settings?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present two generalized Gaussian approximation results for decentralized federated learning with local updates and explore their theoretical and practical implications. The theoretical development appears solid and technically nontrivial.\n\nSTRENGTHS AND WEAKNESSES:\nIn this paper, the authors present two generalized Gaussian approximation results for decentralized federated learning with local updates and explore their theoretical and practical implications. The theoretical development appears solid and technically nontrivial. However, the paper is somewhat difficult to follow. Many important concepts and theorem statements are introduced without sufficient explanation or discussion, making it challenging for readers who are not already familiar with this specific area to fully grasp the contributions and their implications.\n\nQUESTIONS:\nMy major comments are as follows:\n\n1. Motivation \n\nThe authors should elaborate on why traditional convergence analysis (i.e., focusing on optimization error) is not sufficient in modern decentralized learning settings. It is important to explain why the statistical properties investigated in this paper are important. A more thorough discussion connecting statistical properties with optimization and generalization error would greatly benefit readers unfamiliar with this line of work.\n\n2. Choice of Online Setting\n\nThe paper considers an online federated setting, rather than the more common federated learning scenario where each client holds a local dataset. It would be valuable to consider, or at least discuss, how the proposed theoretical results would extend to or differ in the standard offline setting. Additionally, analyzing how sample size, local update number, and total iteration number affect the statistical properties in such settings would be more general and practically meaningful.\n\n3. Clarification on Decentralization\n\nThe paper claims to investigate decentralized federated learning, but repeatedly refers to a ``moderator'' that aggregates client information and redistributes it according to some policy in Sections 2.1 and 2.1.1. This resembles centralized federated learning more than a truly decentralized settings. In typical decentralized Local SGD, messages are shared peer-to-peer over a communication graph.  It should be clearly justified and discussed.\n\n4. Dependence on $\\tau$, Network Topology, data heterogeneity\n\nThe authors mention that the data can be heterogeneous and describe the network topology in Section 2. However, none of the main theoretical results appear to explicitly capture the impact of $\\tau$ (the number of local updates), the network topology, or data heterogeneity. The absence of such dependencies should be discussed.\n\n5. Privacy and Robustness Claims Lack Support\n\nThe authors claim in Section 3 that analyzing the full trajectory of Local SGD can help maintain privacy and detect adversarial attacks. However, I was unable to find any formal results, theoretical justifications, or empirical validations to support these claims. More detailed discussion and supporting experiments are necessary.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two generalized Gaussian approximation methods for local SGD in Decentralized Federated Learning (DFL). First, the authors derive Berry-Esseen bounds for the final local SGD iterates, enabling valid multiplier bootstrap procedures. Second, they provide two sharp time-uniform Gaussian approximations over the local SGD trajectory. Specifically, the theoretical results proposed in this paper are related to the choice of the number of iterations $n$, and the number of clients $K$, the step size $\\beta$. These results enable the development of valid and powerful statistical inference methods, thereby enhancing the performance of federated learning in terms of privacy, robustness, and computational efficiency. Empirical results not only validate the theoretical analysis in the paper but also demonstrate the effectiveness of the proposed methods compared to off-the-shelf Brownian-motion-based approximation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The idea proposed in the paper is easy to follow, and the proposed asymptotic statistical guarantees for local SGD under DFL are potentially impactful. \n2. The elaboration of the proposed theoretical results is sufficient and intuitive.\n3. Both theoretical analysis and empirical results are provided to justify the effectiveness of the proposed Aggr-GA and Client-GA.\n\n### Weaknesses\n1. The pseudocode for the two sharp time-uniform Gaussian approximations could be provided.\n2. Some of the interpretations of empirical results are inconsistent when comparing the main body and the appendix. For example, $\\tilde{d}_c$ does not linearly increases with $K$ for fixed $n$ in Figure 1 right, and is caused by the different dominate term in Eq. (2.5) when increasing $K$. However, the analysis in Section 4.1 says that $\\tilde{d}_c$ increases with $K$ for fixed $n$.\n3. Additional comparative analysis with other approaches in the literature could strengthen the findings.\n4. Typos in line 320, \"we discuss the Berry-Esseen error Berry-Esseen error $d_C$...\".\n\nQUESTIONS:\nAre there any empirical results to indicate the effectiveness of the proposed GA methods for detecting adversarial attacks?\n\nLIMITATIONS:\nThis paper focuses on proposing the asymptotic statistical guarantees beyond convergence properties of local SGD under DFL. However, more experimental evidence could be provided to demonstrate the effectiveness and potential impact of the proposed sharp time-uniform Gaussian approximations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-20"}