Title: Gaussian Approximation and Concentration of Constant Learning-Rate Stochastic Gradient Descent
Venue: NeurIPS 2025
Accepted: yes
Keywords: stochastic gradient descent

constant step size

Gaussian approximation

concentration
Abstract: We establish a comprehensive finite-sample and asymptotic theory for stochastic gradient descent (SGD) with constant learning rates. First, we propose a novel linear approximation technique to provide a quenched central limit theorem (CLT) for SGD iterates with refined tail properties, showing that regardless of the chosen initialization, the fluctuations of the algorithm around its target point converge to a multivariate normal distribution. Our conditions are substantially milder than those required in the classical CLTs for SGD, yet offering a stronger convergence result. Furthermore, we derive the first Berry-Esseen bound -- the Gaussian approximation error -- for the constant learning-rate SGD, which is sharp compared to the decaying learning-rate schemes in the literature. Beyond the moment convergence, we also provide the Nagaev-type inequality for the SGD tail probabilities by adopting the autoregressive approximation techniques, which entails non-asymptotic large-deviation guarantees. These results are verified via numerical simulations, paving the way for theoretically grounded uncertainty quantification, especially with non-asymptotic validity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a  theoretical analysis of stochastic gradient descent (SGD) with a constant learning rate. Unlike traditional analyses that require decaying step sizes or stationarity assumptions, the authors provide both finite-sample and asymptotic results under milder conditions. The key contributions include:\n\n1. The authors develop a new linear approximation technique that enables a CLT for SGD iterates starting from arbitrary initialization, without requiring convergence to stationarity. The fluctuations of the iterates around the target parameter converge to a multivariate Gaussian distribution as the step size \\\\(\\gamma \\to 0\\\\).\n  \n 2.  The paper derives the  Berry–Esseen-type bound for constant-rate SGD, quantifying the distance between the SGD distribution and its Gaussian limit. The bound shows the approximation error is of order $\\mathcal{O}(\\sqrt{\\gamma})$, which is optimal given the non-vanishing bias of the SGD iterates.\n  \n  3.  The authors provide explicit upper bounds for the $p$-th moments of the error $\\mathbb{E}[|\\theta_n - \\theta^*|^p]$, establishing how higher-order moments decay with the number of iterations.\n  \n  4.  By approximating the SGD dynamics with a nonlinear autoregressive process, the paper derives sharp large-deviation bounds that combine sub-Gaussian and polynomial decay terms, allowing high-probability guarantees even in heavy-tailed noise settings.\n  \n  5.  Numerical experiments on logistic regression with heavy-tailed data validate the theoretical bounds. The proposed Nagaev-type inequality closely matches the empirical tail probabilities and significantly outperforms Markov-type bounds in tightness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.  The paper establishes the first Berry--Esseen bound and Nagaev-type inequality for constant learning-rate SGD.\n 2. The central limit theorem applies to non-stationary SGD sequences with arbitrary initialization, which is more aligned with practical scenarios.\n 3. The work provides moment bounds and concentration inequalities that are valid for finite samples.\n 4. The use of linear and autoregressive approximations, functional dependence measures, and Lyapunov equations is mathematically rigorous and innovative.\n 5.  Numerical studies effectively support the theoretical claims, demonstrating the sharpness of the proposed tail bounds.\n\n  \n\nWeaknesses\n\n1. The simulations are confined to a single logistic regression setting, limiting insight into performance across diverse models or datasets.\n  \n 2.  Although milder than prior work, the assumptions (e.g., strong convexity, Lipschitz continuity) may still be restrictive for many non-convex or real-world scenarios.\n  \n 3. The focus is on theoretical analysis; no new algorithm or implementation technique is proposed, which may limit the paper’s immediate practical impact.\n  \n 4.  There is no discussion on computational efficiency or how the proposed bounds scale with dimensionality in practical settings.\n\n5. It is not clear how the authors estimate the covariance matrix $\\Gamma$ and how this would influence the final bounds.\n\nQUESTIONS:\n1. How do the assumptions in this paper (e.g., strong convexity, Lipschitz continuity) limit the applicability of the results to non-convex optimization problems commonly encountered in deep learning?\n\n2.  The paper presents a quenched central limit theorem that applies to non-stationary SGD. How does this differ from classical CLTs for SGD that require asymptotic stationarity? What practical advantage does this offer? \n\n3.  Why is the Berry--Esseen bound particularly important for understanding the behavior of SGD in finite-sample regimes? How does this bound compare to previous results for decaying learning rates?  The matrix $\\Gamma$ is typically unknown and should be estimated.  How are the authors going to estimate it. \n\n4.  The simulations focus on logistic regression with heavy-tailed inputs. Would the theoretical bounds hold or degrade for more complex, non-linear models (e.g., neural networks)?\n\n 5.  The authors claim their bounds are sharp. What empirical or theoretical evidence supports this, and under what conditions might the bounds be loose or overly conservative?\n\n 6.  The paper discusses optimal scaling of $\\gamma$ in terms of $n$ for convergence. How should practitioners choose $\\gamma$ in practice based on this theory, especially when $n$ is unknown?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

comments: SUMMARY:
The paper provides a Berry-Esseen bound for the last-iterate error of SGD, with an $\tilde{\mathcal O}(n^{-1/2})$ (dimension dependence suppressed) normal approximation error over all convex sets in $\mathbb R^d$.

STRENGTHS AND WEAKNESSES:
The paper is very crisply written, well structured, and bears a very elegant and important result. While it is long known that the last iterate of SGD is asymptotically normal, this paper's quantitative version of such asymptotically normality opens up numerous downstream inference/uncertain quantification tasks.

I only have good things to say about this paper. A minor weakness might be that some potentially interesting dependences, in particular the one on $d$, are suppressed while directly porting the BE result by Shao & Zhang.

QUESTIONS:
- Can the authors spell out the dependence of $C$ on other problem parameters, in particular the dimension $d$, after Theorem 3.4?

- Related, it is known in the i.i.d.\ random vector case that by considering all closed balls instead of all convex sets in $\mathbb R^d$, the Berry-Esseen bound's $O(d^{1/4})$ dependence on $d$ can be removed. Can the authors achieve the same via taking $\mathcal V$ to be all closed balls?

- Would the authors be able to say a few things about the BE analysis of the Polyak-Ruppert averaged iterate? The PR average has a smaller variance in its limiting Gaussian distribution, right? So its BE approx might be of more practical interest.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the distribution of SGD iterates under a constant learning rate. They prove that under certain assumptions such as strong convexity and Lipschitz, fluctuations around its mean converge to a multivariate normal distribution. Then they find the error of convergence through Barry-Essen bound in Gaussian approximation, characterize the large tail deviation using Nagaev type inequalities and find condition for convergence of higher order moments. Deviating from past works, that require a stationary distribution, this works provides the results for a finite step-size reflecting practical behaviour.\n\nSTRENGTHS AND WEAKNESSES:\nThis is an interesting paper that characterizes the converges of SGD iterates under certain conditions. Based on the research interest on practical deep learning such as finite-step size, I think this is a timely topic. \n\nStrengths:\n\n1) The paper is generally well written and the notations are explained carefully for ease of the reader. \n2) The remarks after every theroem explains the role of each hyperparameter. \n\nWeakness:\n\n1) *Lack of convincing experiments*: It would have made the paper stronger if the distribution was analyzed for more loss functions. It is well known that logistic regression is not strongly convex, although the theoritical assumption has strong convexity in it. I think the basic loss function is a quadratic where both the strong convexity parameter and the lipschitz parameter depend on the eigenvalues of the Hessian, which is easily calculated. If one can characterize the distribution is on a simple quadratic, then that would validate all the theorems. Currently this is missing. \n\n2) *What if stochastic noise is Gaussian*: If the noise in the gradient assumes a normal distribution and loss is quadratic, then the distribution of the SGD iterates I think should also be an exact normal distribution with 0 error in Barry Essen type inequality. Does the current theorem cover this case? For example in 3.4, will these two assumptions make the Barry-Essen bound go to zero?\n\n3) *Question on loss surface assumptions*: Although strong convexity and lipschitz assupmtions are slightly strong, yet necessary for a nice result, I assume what happens for general ovreparameterized deep neural networks where these conditions do not hold (consider a saddle near a minima). What will the distribution of SGD iterates look like? Will they still converge to a Gaussian distribution? This is not necessarily a weakness but I want to hear the author's comments. \n\nRegarding the originality, I am not aware of all the papers on characterizing the distribution convergence of SGD iterates, but I agree that a finite step-size is indeed an interesting result.\n\nQUESTIONS:\nPlease see points 2 and 3.\n\nLIMITATIONS:\nAs I said, the assumptions on the loss function are too restrictive. The most basic loss to analyze the convergence is a quadratic, which the authors skipped to for a logistic regression loss. I would suggest l2 regularized logistic regression since it is strongly convex, as a better alternate experiment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a detailed finite-sample and asymptotic theory for constant learning-rate SGD. By leveraging linear approximations and functional dependence techniques, it derives a quenched central limit theorem, a Berry-Esseen bound, and Nagaev type tail inequalities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** This work introduces novel approximation techniques and establishes sharp finite sample guarantees, providing the first Berry–Esseen bound and non-asymptotic tail bounds for constant step size SGD. It advances our understanding of the non-stationary behavior of SGD beyond the classical results.\n\n\n**Weaknesses:** The paper primarily lists various techniques and results, but it leaves much of the intuition and practical usefulness of these tools insufficiently explained. Moreover, the experimental section is quite limited, focusing only on verifying the Nagaev type tail inequality through synthetic logistic regression experiments in low dimensions, and testing just three learning rates. It does not explore higher-dimensional scenarios or examine other theoretical results, such as the Berry-Esseen bound or the central limit theorem.\n\nQUESTIONS:\n1. What is the implication of the bound in Theorem 3.1? Furthermore, how does Theorem 3.1 contribute to the main results?\n\n2. What is the definition of $\\beta^*$  on page 7?\n\n3. Expanding the current experiments to include more diverse scenarios, additional step sizes, and higher dimensional cases would strengthen the practical validation of the tail bound.\n\n4. The Berry-Esseen bound in Theorem 3.4 is valuable and novel. However, its practical usefulness remains somewhat unclear. The paper does not discuss how the bound scales with dimension, nor does it provide numerical experiments to illustrate how tight or meaningful the bound is in practice. Including explicit dimension dependence, practical guidance on estimating the constants involved, and empirical validation would significantly strengthen the impact and applicability of this result.\n\n5. The theoretical contributions of this work are valuable and novel, but the lack of explanatory context and practical examples significantly reduces readability and impact. With improvements in exposition and additional experiments, the paper could be a strong contribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives a variety of quantitative statistical properties of SGD iterates when the averaged loss is strongly convex and the step size is constant. In particular, it gives variations on the $p$-th moment bounds, the central limit theorem, Berry-Esseen bounds, and large-deviation inequalities. The Berry-Esseen bounds and large deviation inequalities give quantitative bounds on the deviation of the iterates from Gaussian random variables.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The results give a more more detailed statistical analysis of SGD than currently available. While basic convergence properties are well-understood, the results in this paper give precise quantitative bounds on the deviation from Gaussian random variables. \n- The writing is well-structured and precise. \n\n# Weaknesses\nThe only minor weakness is that the writing is highly terse, and some parts end up appearing unmotivated or not quite justified. Some examples:\n- In the proof of Theorem 3.4. It isn't obvious why the quantity in (27) is being bounded until much later, when a result from Shao and Zhang is invoked.\n- The use of the Lindeberg-Feller CLT to prove Theorem 3.3 could be explained more thoroughly. It seems that it would follow from Assumption 2.2 and the construction of L, but more  details would be nice. (Namely, justifying the Lindeberg condition.)\n\nA bit of restructuring to help the reader (and this reviewer!) understand the ideas behind of the proofs would be helpful.\n\nQUESTIONS:\n- Can the constants on the theorems be quantified? \n- Would it be possible to modify the methods to handle non-strongly convex problems?\n- In the simulation, values of constants associated with the asymptotic distribution are used, and this is also remarked earlier to suggest that part of the Nagaev-type bounds are tight. Could the use of the constants from the asymptotic distributions be justified rigorously?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}