Title: Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent
Venue: NeurIPS 2025
Accepted: no
Keywords: Stochastic Gradient Descent

Polyak-Ruppert averaging

Berry-Esseen type bounds

Bootstrap methods
Abstract: In this paper, we establish the non-asymptotic validity of the multiplier bootstrap procedure for constructing the confidence sets using the Stochastic Gradient Descent (SGD) algorithm. Under appropriate regularity conditions, our approach avoids the need to approximate the limiting covariance of Polyak-Ruppert SGD iterates, which allows us to derive approximation rates in convex distance of order up to $1/\sqrt{n}$. Notably, this rate can be faster than the one that can be proven in the Polyak-Juditsky central limit theorem. To our knowledge, this provides the first fully non-asymptotic bound on the accuracy of bootstrap approximations in SGD algorithms. Our analysis builds on the Gaussian approximation results for nonlinear statistics of independent random variables.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript, the authors investigate the limit theorems for Noisy SGD\n$$\n\\theta_{k} = \\theta_{k} -\\alpha_{k}(\\nabla f(\\theta_{k-1}) + \\xi_{k}).\n$$\nThe sequence of averaged iterates is defined as $\\sqrt{n}(\\bar{\\theta}_{n} - \\theta^{\\ast})$.\n\nUnder some assumptions, the authors obtain the convergence rate of the averaged iterates to a zero-mean Gaussian distribution with covariance matrix $\\Sigma_{\\infty}$ in terms of convex distance $\\mathrm{d}_{\\mathrm{C}}(\\cdot,\\cdot)$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The author provide the lower bounds on the convex distance $\\mathrm{d}_{\\mathrm{C}}(\\sqrt{n}(\\bar{\\theta}_n - \\theta^{\\ast}),\\mathcal{N}(0,\\Sigma))$.\n2. It is interesting to note that the multiplier bootstrap procedure is employed to study the convergence result of SGD.\n\n\nWeaknesses:\n\n1. Since the authors assume the decomposition $F(\\theta_{k-1}, \\xi_k) = \\nabla f(\\theta_{k-1}) + \\xi_{k}$ holds, they actually study the noisy gradient descent (7) rather than the general SGD (2). I suggest the author should clarify this aspect in advance.\n2. As we know $\\nabla f(\\theta)=\\mathbb{E}_{\\xi\\sim\\mathbb{P}}[F(\\theta, \\xi)]$, I wonder whether these assumptions (A1)-(A4) are mutually compatible, as their compatibility is crucial for the validity of the results. I would appreciate it if you could give some explanation.\n3. Could you please give an example of the objective function that satisfies A1-A4?\n4. In the following references, the authors have considered the limit theorems for SGD, the results in which are closed to this manuscript. Could you please provide a detailed comparison between these literatures and your works? I believe this would help us better understand the contributions of your research.\n\n    [1] Jose Blanchet, Aleksandar Mijatović, and Wenhao Yang. Limit Theorems for Stochastic Gradient Descent with Infinite Variance. arXiv:2410.16340v3, 2024.\n\n    [2] Jerome Sacks. Asymptotic distribution of stochastic approximation procedures. The Annals of Mathematical Statistics, 29(2):373–405, 1958.\n5. I would be glad to see more validation of the theoretical result. So could you please provide some numerical experiments to illustrate their theoretical results?\n\nQUESTIONS:\nThe questions that I am concerned about have been listed in the 'Weaknesses'.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the properties of a bootstrap approximation for Polyak-Ruppert SGD iterates.  For strongly convex optimization problems, they establish a rate of convergence in the convex distance of $1/\\sqrt{n}$, matching the Berry-Esseen rate.  To prove this result, the authors show that the bootstrap is able to approximate an intermediate covariance matrix that differs from the Polyak-Juditsky covariance matrix.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors establish a sharp rate of convergence for the bootstrap approximation, improving on other plug-in approaches in the literature.\n- The observation that the bootstrap offers a more direct approximation of the sampling distribution, leading to faster rates of convergence than methods that target the asymptotic distribution, may be of interest in other related problems.   \n- The paper is well-written overall.  \n\nWeaknesses:\n- It appears that some additional regularity conditions that are not needed for the CLT are required to study the bootstrap.  However, the authors do a good job of explaining the conditions and citing examples where similar assumptions are made.\n- While the results are sharp, it appears that the result holds only for $d$ fixed and for very well-behaved optimization problems.\n\nQUESTIONS:\n- While the authors establish a sharper rate of convergence for the bootstrap, the technical conditions are stronger.  It would be appreciated if the authors could discuss if there are any substantial differences in the conditions compared to plug-in approaches to inference.  \n- In their discussion of the proof of Theorem 1, the authors provide an expression for $\\Sigma_n$.  It would be helpful if the authors briefly discuss why plug-in estimators for this quantity cannot readily be constructed. \n-  The paper of Shao and Zhang (2022) also considers SGD as an application of their Berry-Esseen bound for non-linear statistics.  It would be helpful if the authors could explain the differences of the proof of Theorem 2 from what is considered in this paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of constructing statistically valid confidence sets for the optimum $\\theta^$ of a strongly convex objective when using stochastic gradient descent (SGD). The authors analyze a multiplier bootstrap procedure for Polyak–Ruppert averaged SGD iterates, which involves generating perturbed SGD trajectories to approximate the distribution of $\\sqrt{n}(\\bar{\\theta}_n - \\theta^)$. They prove, under suitable smoothness and noise conditions, that the bootstrap can accurately approximate the quantiles of the exact error distribution at finite sample sizes. Notably, their non-asymptotic bounds show an approximation error of order $n^{-\\xi/2}$ (up to log factors) for step-size schedules $\\alpha_k\\propto k^{-\\xi}$ with $\\xi\\in(0.5,1)$. This provides the first rigorous finite-sample guarantee for bootstrap-based confidence intervals in SGD. Furthermore, the analysis reveals that the bootstrap’s validity hinges on approximating a finite-$n$ covariance ($\\Sigma_n$) rather than the usual asymptotic covariance $\\Sigma_1$, avoiding the need to directly estimate $\\Sigma_1$. In addition, the paper establishes new rates for the convergence of the averaged SGD iterate’s distribution to normality (the Polyak–Juditsky CLT): the distributional distance to $N(0,\\Sigma_1)$ is shown to be $O(n^{-1/4})$ for a specific step-size schedule, and a matching lower bound is provided to demonstrate the tightness of this rate. Overall, the contributions advance the theory of uncertainty quantification for SGD by providing non-asymptotic bootstrap confidence guarantees and deeper insight into the distributional behavior of SGD iterates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a thorough theoretical analysis with clear assumptions (A1–A9) and complete proofs (deferred to appendices). The main theorems are stated with high-probability guarantees and rates, and the authors even derive a lower bound to confirm that their key upper bound is tight. \n2. The contributions advance the theoretical state-of-the-art in several ways. Establishing finite-sample convergence rates up to $O(1/\\sqrt{n})$ for distributional approximations is non-trivial and improves upon earlier results that were either asymptotic or slower. This is the first work to show that a bootstrap method can achieve a faster distributional convergence rate than the classic CLT rate for SGD (which, in terms of Berry–Esseen bounds for $\\sqrt{n}(\\bar{\\theta}_n-\\theta^*)$, was effectively slower). \n3. The presentation of the paper is clear with a nice flow. \n\n**Weaknesses**\n\n1. The main limitations in quality stem from the restrictive conditions required. Notably, the analysis assumes the stochastic gradient noise is essentially bounded (almost sure boundedness of $\\zeta_k$ in Assumption A2(iii)) and that the stochastic gradient is co-coercive (Assumption A4). These conditions are quite strong and may not hold in many practical scenarios. \n\n2. The paper’s novelty is incremental in the sense that it builds directly on an existing methodology (the bootstrap algorithm from [16]) and extends recent theoretical frameworks. The work is a natural next step after [38] and [46], and some techniques are adapted from those sources (as the authors acknowledge, e.g. using techniques from [42] and [46] in proving their Theorem 2). Thus, the contribution is not a brand-new algorithm or completely unconventional theory, but rather a meaningful extension and generalization of prior art.\n\nQUESTIONS:\n1. Relaxing the Noise Assumption: One major assumption is the almost sure boundedness of the stochastic gradient noise (Assumption A2(iii) and related conditions). How sensitive are the results to this assumption? Could the analysis be extended to allow sub-Gaussian noise or heavier-tailed noise (perhaps with slower rates or additional logarithmic factors)? \n\n2. Empirical Illustration: To increase the practical impact, consider adding a small-scale experiment or simulation. For example, run SGD on a simple logistic regression or quadratic problem and construct confidence intervals using the multiplier bootstrap vs. using a plug-in covariance estimate.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the rate of convergence of the average of the iterates from time zero to n called the Polyak--Ruppert averaging. It is well known that the fluctuations of this estimatoir are Gaussian, and the paper provides some finite $n$ bounds on this rate of convergence. The paper also studies the bootstrap proceduce, which does not depend on the limiting covariance $\\Sigma_\\infty$.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem of the rate of convergence studied in this paper is very natural and the paper provides a nice precise derivation of this rate of convergence by approximating the iterates with the iterates of a perturbed SGD. It is not surprising that such rates of convergence can be proved, but the analysis seems to be carefully done. \n\nHowever, I have a few concerns that the scope of this paper might be quite narrow. Namely the convexity and coercivity assumptions seem to be quite restrictive. Rigorous bounds even under some strong assumptions on the objective are welcome, but perhaps its scope is too narrow for a venue like NEURIPs. The usage of the bootstrap procedure does not seem well motivated in this paper, and for the purpose of proving a rate a convergence does not seem like a necessary intermediate step.\n\nQUESTIONS:\n1. The convex distance is a bit strange to me. Why is this distance used instead of a more standard ones like the Kolmogorov distance or the total variation distance. Does this metric have some nice properties that differ from these more standard ones?\n\n2. Why do we study the bootstrap method? What are the main barriers from proving a rate of convergence directly on the usual SGD iterates.\n\n3. How necessary are the assumption in this paper? In particular, if these assumptions where to be weakened, will the fluctuations no longer be Gaussian?\n\n4. The importance of the role of $\\Sigma_n$ and $\\Sigma_\\infty$ raised in point 2 on line 63 is a bit unclear. There should be concentration of the empirical average, so the error introduced from the approximation should be controlled. It is not clear that going through the bootstrap gives a more precise rate on convergence.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}