Title: Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime
Venue: NeurIPS 2025
Accepted: yes
Keywords: last-iterate

low-noise

interpolation

realizable

sgd

stochastic

convex

smooth
Abstract: We study population convergence guarantees of stochastic gradient descent (SGD) for smooth convex objectives in the interpolation regime, where the noise at optimum is zero or near zero. The behavior of the last iterate of SGD in this setting---particularly with large (constant) stepsizes---has received growing attention in recent years due to implications for the training of over-parameterized models, as well as to analyzing forgetting in continual learning and to understanding the convergence of the randomized Kaczmarz method for solving linear systems. We establish that after $T$ steps of SGD on $\beta$-smooth convex loss functions with stepsize $0 < \eta < 2/\beta$, the last iterate exhibits expected excess risk $\widetilde{O}(\tfrac{1}{\eta (2-\beta \eta) T^{1-\beta\eta/2}} + \tfrac{\eta}{(2-\beta\eta)^2} T^{\beta\eta/2} \sigma_\star^2)$, where $\sigma_\star^2$ denotes the variance of the stochastic gradients at the optimum. In particular, for a well-tuned stepsize we obtain a near optimal $\widetilde{O}(1/T + \sigma_\star/\sqrt T)$ rate for the last iterate, extending the results of Varre et al. (2021) beyond least squares regression; and when $\sigma_\star=0$ we obtain a rate of $O(1/\sqrt T)$ with $\eta=1/\beta$, improving upon the best-known $O(T^{-1/4})$ rate recently established by Evron et al. (2025) in the special case of realizable linear regression.


Peer reviews:
comments: SUMMARY:
This paper studies the last-iterate convergence of SGD. For the general smooth convex setting with optimally tuned learning rates, it obtains an improved instance dependent bound which replaces the uniform noise bound in the previous work by the variance at optimum. This directly implies an improved bound for the realizable setting, that goes beyond the OLS regime. For the realizable setting with the greedy stepsize $1/\beta$, it obtains a strictly improved convergence guarantee which has a faster rate and goes beyond OLS.

STRENGTHS AND WEAKNESSES:
**Strengths:** studying the last-iterate convergence of SGD is an important problem both theoretically and practically. This paper provides solid advancement on this problem. The technical framework and novelty are well explained: the proof outline clearly describes the roadmap, making it easier for the readers to tell which parts (step 0,1,3) mostly follow from the literature and which (step 2,4,5) are novel technical contributions. In my opinion, the overall approach still lies in the classic framework, but the technical contributions are solid and necessary for obtaining the improved rates. Hence I recommend accept.

**Weaknesses:** the improvement is mainly in the realizable setting with the greedy stepsize. For the general setting, the improvement is somewhat minor, but it's still a strict improvement since the algorithm requires no knowledge of the variance at optimum.

QUESTIONS:
I wonder if last-iterate convergence guarantees can be potentially obtained for the agnostic setting with the natural $\eta=1/\beta$ choice. If it's not theoretically impossible, what's the main technical difficulty?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies population convergence guarantees of stochastic gradient descent (SGD) for smooth convex objectives in the interpolation (low noise) regime.\n\n- The paper establishes a convergence rate of $\\widetilde{O}(1/(\\eta T^{1-\\beta\\eta/2}) + \\eta T^{\\beta\\eta/2}\\sigma_{\\star}^2)$ for the last iterate of SGD with step-size $\\eta \\leq 1 /\\beta$ on $\\beta$-smooth convex loss functions, where $\\sigma_{\\star}^2$ is the variance of the stochastic gradients at the optimum. With a well-tuned stepsize, this translates to a $\\widetilde{O}(1/T + \\sigma_{\\star}/\\sqrt{T})$ rate.\n\n- The paper further extends the analysis to without-replacement SGD, block Kaczmarz method, and continual linear regression, leading to some better convergence rates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper derives a $\\widetilde{O}(1/T + \\sigma_{\\star}/\\sqrt{T})$ rate for the last iterate of SGD with a well tuned step-size on smooth convex loss functions. It improves previous result by replacing the global noise variance bound with the variance at the optimum, or under the same set-up, it extends convergence guarantees for averaged iterates  to the last iterate of SGD, matching (up to logarithmic factors) the optimal rates achieved by averaging.\n\n- The paper further extends the analysis to other settings.\n\n- The paper is in general clearly written.\n\n**Weakness**\n\n- I agree with that  deriving convergence results for the last iterate, rather than the averages, of SGD is interesting. Given that the connection between the last iterate and averages has been extensively explored in the literature, the contribution feels somewhat incremental to me.\n- Could the theoretical results of this paper offer practical insights into the numerical experiments? For instance, Theorem 1 suggests that the step-size in case (ii) should outperform the choice in (i). Is this actually observed in practical applications or numerical simulations?\n- While the paper focuses on the convex setting—a reasonable starting point for theoretical analysis—this is somewhat limiting given the prevalence and importance of nonconvex optimization in modern machine learning applications.\n-  As the authors acknowledge, when $\\eta = {1\\over \\beta}$, the convergence rate is $1/\\sqrt{T}$ in the interpolating regime, which falls short of the optimal $1/T$ rate achievable with averaged SGD.  Moreover, what is the theoretical motivation for considering this particular step-size setting?\n\nQUESTIONS:\n- See the weakness part.\n- Is there any connections between the Zamani and Glineur technique with [1, Theorem 2]?\n- How can the result from Srebro et al. (2010), which uses an approximation error rather than $\\sigma^*$, be refined to obtain the error bound presented in Table 1.\n\n\n[1] Francesco Orabona. Last iterate of sgd converges (even in unbounded domains), 2020. URL\nhttps://parameterfree.com/2020/08/07/last-iterate-of-sgd-converges-even-in-unbounded-domains/.\n\nLIMITATIONS:\nprobably yes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a significantly improved last-iterate convergence analysis of SGD for smooth, convex functions in the interpolation and low noise regimes. This is the first result showing a $1/T$ last-iterate rate in the interpolation regime (previous results had achieved this rate only for ordinary least squares), and this is the first result showing a $1/\\sqrt{T}$ last-iterate rate with a greedy stepsize $\\eta = 1/\\beta$ (where $\\beta$ is the smoothness constant), where previous work had shown a $1/T^{1/4}$ rate and only for ordinary least squares.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This is an important problem: the last-iterate convergence of SGD (especially under interpolation) is fundamental for optimization with clear motivation for machine learning.\n2. The improvement over previous work is clear. Better rates are established and in more general settings (see Table 1).\n3. The paper is very well-written and clearly communicates the novelty in the analysis compared to previous work. I think the proof outline and proof itself (sections 3.1 and 3.2) are informative and transparent about the ideas.\n\nWeaknesses:\n1. The $1/T^{1/4}$ rate for greedy stepsize with interpolation does not match lower bounds ($1/T$) or upper bounds for the average iterate ($1/T$), though I think this is fine. The authors acknowledge this gap as an open problem (line 77) and I think this is gap is totally reasonable: you can't solve everything in one paper.\n\nQUESTIONS:\n1. Can you explain what exactly is the low-noise regime? The introduction (line 62) says that $\\sigma_* \\ll 1$, but the statement of Theorem 2 only requires that $\\sigma_* < \\infty$. Assuming that low-noise refers to $\\sigma_* < \\infty$, why call this low-noise? Bounded variance (or bounded variance at optimum) is a common assumption in optimization, but is this condition often referred to as low-noise?\n2. Liu and Zhou [22] have results for both in-expectation and high-probability. Do you think your techniques can be easily extended to allow for high-probability guarantees, assuming say sub-gaussian noise as they do?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}