Title: Asymptotic theory of SGD with a general learning-rate
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning Rate Schedules

Convergence Analysis

Stochastic Gradient Descent

Online Learning
Abstract: Stochastic gradient descent (SGD) with polynomially decaying step‐sizes has long underpinned theoretical analyses, yielding a broad spectrum of statistically attractive guarantees. Yet in practice, such schedules find rare use due to their prohibitively slow convergence, revealing a persistent gap between theory and empirical performance. In this paper, we introduce a unified framework that quantifies the uncertainty of online SGD under arbitrary learning‐rate choices. In particular, we provide the first comprehensive convergence characterizations for two widely used but theoretically under-examined schemes—cyclical learning rates and linear decay to zero. Our results not only explain the observed behavior of these schedules but also facilitate principled tools for statistical inference and algorithm design. All theoretical findings are corroborated by extensive simulations across diverse settings.


Peer reviews:
comments: SUMMARY:
This paper studied the convergence analysis of SGD with general learning rate that is more aligned with practice. They introduce a unified framework that quantifies the uncertainty of online SGD under arbitrary learning-rate choices. Specifically, they explored cyclical learning rates and linear decaying learning rate. Experiments validate the theoretical findings.

STRENGTHS AND WEAKNESSES:
Strength: 1) the problem they studied is important to analyze the theoretical performance of practical working step sizes. 2) the analyzing tools are innovative. The theoretical finding is novel and inspiring. 

Weaknesses: (1) For the cosine learning rate, which is widely used in practice, only experimental results are provided, with no accompanying theoretical guarantees. 

(2) Theorems include too many assumptions, and their roles—as well as the classes of problems that satisfy them—are not clearly explained. For example, how can condition (2.7) be satisfied in practice? 

(3) The experimental evaluation is limited. Although the authors validate various theoretical results through experiments, all experiments are conducted on linear regression problems. No large-scale experiments are included to demonstrate the scalability of the theoretical conclusions to more complex models.

(4) It would be helpful to include a direct comparison of convergence rates with those from existing literature to highlight the effectiveness and novelty of the new proof technique.

QUESTIONS:
1. Reference error: Line 271 ?? 
2. See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a unified finite-sample analysis of vanilla SGD under arbitrary learning-rate schedules. 1) General moment bound (Thm 2.1). For any sequence $\\{\\eta_t\\}$, even when $\\eta_t$ depends on the optimisation horizon $n$, they bound the $p$-th moment error by an explicit two-term expression that separates the effect of the initialisation from a weighted sum of squared step-sizes. 2) Specialising Thm 2.1 yields what appears to be the first non-asymptotic MSE guarantee for the popular “linear-D2Z” policy $\\eta_t=\\eta(1-t/n)$ . 3) The authors prove convergence in distribution of the iterates to a cyclostationary process when $\\eta_t$ is $T$-periodic, filling a gap in existing theory . 4) Simulations on a handful of convex models illustrate the contrasting transient and terminal errors of five schedules (constant, cosine, linear-D2Z, polynomial, mixed-polynomial).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. Treating horizon-dependent and cyclical rates within the same inequality is elegant and may simplify future analyses.\n2. The general bound recovers known optimal rates for polynomially decaying steps and constant steps (up to logarithmic factors).\n3. Providing the first rigorous account of linear-decay and cyclical learning rates bridges a real gap between theory and what deep-learning practitioners actually use.\n5. Despite heavy notation, the paper is well structured (Section 1 to  2 to 3) and key ideas are summarised before each proof .\n\nWeaknesses: \n1) In my view, it's good to bring the technical novelty in the proofs to the audience's attention.\n2) The global strong convexity rules out the non-convex settings (deep nets, GANs) where exotic schedules are most used. So it could be nice to have these results under milder assumptions like PL inequality.\n\nQUESTIONS:\n1) In the first line of Assumption 2.2, an identity matrix in the right-hand side of inequality is missing. Why is this assumption called local strong convexity? What is the point of this assumption brought in the main body when it's not used?\n2) What is $|\\cdot|_{p}$ in Equation (2.2)? Where is it defined?\n3) Typo in line 200, \"Section 2.3\".\n4) Please provide a citation for line 207.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes theoretically some learning rates schedules commonly used in practice in ML, the cyclical learning rate and the linear decay to zero learning rate. \nMore precisely, it generalizes previous analyses in terms of mean-squared error to not only tackle polynomially decreasing learning rates, but also more learning rate schedules that previously lacked theoretical support, such as the linearly decaying to zero learning rate (D2Z) and cyclical learning rates.\nAdditionally, the results are verified experimentally.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n-  1. I think this paper makes good contributions to the literature of SGD by focusing on learning rate schedules that are recent and more used in practice.\n- 2. The analysis for the linear decaying learning rate looks interesting, as it it expressed in a general $\\ell_p$ norm form which looks novel compared to state of the art results for this setting such as Defazio et. al. (2023)\n- 3. Perhaps more importantly the analysis for cyclical schedules looks interesting and novel since for its proof it uses tools from non-linear time series theory (from Li et. al. 2024), which is not very standard in convergence proofs in optimization (which are usually rather based on judicious refactoring of norm inequalities and using the assumptions ((local) strong convexity, etc) of the problem). The way that the results of Li et. al. are used seem novel up to my knowledge (I checked the paper from Li et. al. and did not see any proof for a cyclic schedule there).\n- 4. Finally, the experiments show well the validity of the results in practice, in particular, they are able to showcase the periodicity in the standard deviations, when using a cyclic schedule.\n\n# Weaknesses\n\n- 1. I think it would be good for that paper to compare in slightly more details to the result from Defazio et. al. (2023), in particular, in 2.3, what do the authors mean by “its non-asymptotic convergence rate\nremains unknown” ? From my understanding, the work from Defazio is non-asymptotic already ? (as it gives a bound for a given iteration number ?)\n- 2. Further on this point, it seems to me at first sight that the proof in the case of linear decay, in terms of the p-norm, is somehow a standard modification of a usual proof of convergence in such setting, but using generalizations of $\\ell_2$ inequalities to p-norms, but I may be wrong. If so, maybe the authors could describe in more details the fundamental difference with other usual proofs, except the use of the p-norm. (I guess in any case it can always be a mildly interesting feature to show such proof in the p-norm).\n\n- 3. **Minor typo:** in 3.2, the reference for the Figure 1 is missing (it is written Figure ??)\n\nQUESTIONS:\n- 1. Could the authors provide maybe more details on how this work, for the convergence rate in the case of the linear schedule, compares with the work of Defazio, and whether there is any fundamental difficulty to obtain their result compared to the one in Defazio ? (even if, I agree, this work is general as it uses $\\ell_p$ norms for the moments, and also, the result for linear decaying learning rate is just one result amongst many results proven in the paper so that does not put in question the rest of the paper). Especially when p=2: the result of Defazio is a convergence result in objective function, but wouldn’t one be able to obtain a result in terms of $\\ell_2$ distance to optimality directly from the objective function bound, by using Assumption 2.1 at the optimum ?\n- 2. Why using the p-norm in Theorem 2.1 (instead of simply the $\\ell_2$ norm) ? I may have missed the motivations behind it but if it is not present in the paper I guess the authors could elaborate on why it is important to prove a result in p-norm.\n\nLIMITATIONS:\nno specific limitation", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the gap between the well-established theoretical analyses of stochastic gradient descent (SGD) with polynomially decaying step-sizes and the practical use of alternative learning rate schedules such as cyclical learning rates and linearly decaying to zero (Linear-D2Z), which have been widely adopted in deep learning but lack comprehensive theoretical understanding. The authors develop a unified framework that provides non-asymptotic mean-squared error bounds for SGD iterates under arbitrary learning rate schedules, generalizing classical results and explicitly capturing the influence of initial conditions and step-size sequences. They offer the first rigorous convergence characterization for the popular Linear-D2Z schedule and introduce novel asymptotic theory for cyclical learning rates, showing that SGD with cyclical rates converges to a cyclo-stationary distribution exhibiting periodic behavior, in contrast to the stationary distribution convergence of constant step-size SGD. Numerical experiments validate these theoretical insights, demonstrating fast early convergence and low final error for linearly decaying schedules and cyclical variance patterns for cosine schedules.\n\nSTRENGTHS AND WEAKNESSES:\nI think in general the paper is well written, the motivation is clear and the contribution too. \n\nOne of my main issues with the paper is that I find the title and some parts of the abstract and introduction a bit misleading. I do not want to diminish the contribution of the paper, but they way it is presented, especially in the title for a \"general learning-rate\", the reader might be misled thinking that the proposed framework generalizes all the previous works so far, or is insightful for them. However this is not the case, as for constant learning rate, as the authors point out, later on, their Theorem 2.1 only yields a O(1) bound, which does not seem useful. I do not have an issue with this per se, but with the way the framework is presented and encourage the authors to change this.\n\nThe authors make several assumptions which I do not seem to be too restricting and are standard assumptions in works like this, and they further give comments on these assumptions and relaxations in case where they are possible which I appreciated.\n\nIn Remark 2.4 the authors say that some of the ηi’s maybe taken to be large which helps in faster convergence. This can also be seen empirically in Figure 4. Could the authors be a bit more specific to how they define \"large\" and provide a bit more clarification on this? A histogram for the norms might be interesting. Another thing that I observe in the is: in first example in Figure 4 with 0.1 as the multiplicative constant, the norms of these fluctuations seem to start to be larger and larger as the process progresses. Why did the authors consider 10^4 iterations and not more? For the smaller constant multipliers it seems that the behavior does not change but for the case 0.1 I think it would be useful to further run the simulation. \n\nAlthough the paper is of theoretical nature, it would be beneficial and I would have been keen to give a higher score if the authors have performed more extensive experimental analysis, for example using simple networks as LeNet or AlexNet would suffice on simple datasets like MNIST or CIFAR-10 that do not require intensive computational resources. I think this would make the paper stronger.\n\nAlso, can the authors repeat their experiments with initializations following a zero-centered Gaussian distribution? I think this initialization is, at least from my experience in similar works, a lot more standard. I am not sure why did the authors decide to fix it throughout all experiments and it raised slight confusion, although I believe that the results and findings should not change.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper studies how stochastic gradient descent (SGD) behaves when using different kinds of learning rate schedules—especially ones commonly used in practice, like linearly decaying learning rates and cyclical ones. These types of schedules are widely used in training deep learning models but haven’t been fully studied in theory. The authors develop a general framework to analyze these learning rates, providing both theoretical results and experiments. The main contributions include error bounds for general step sizes (Theorem 2.1), analysis of linearly decaying learning rates (Theorem 2.2), and new results about cyclical learning rates (Theorem 2.4), showing that they converge to a repeating (cyclostationary) pattern.

STRENGTHS AND WEAKNESSES:
Strengths
The new theroretical analysis of linear decay to zero SGD.

Weakness
In the introduction, the paper does not explain what $\eta_t$ and $\eta_{n,t}$ represent, which may confuse readers unfamiliar with the notation. The schedule $\eta_t = \eta(1 - t/n)$ is introduced without much context or justification. This type of schedule assumes that the total number of iterations $n$ is known in advance, which is often unrealistic in practice—especially for stochastic algorithms where the convergence time is not fixed and may vary depending on the data or noise. Additionally, the paper does not cite some relevant prior work.
Assumption 2.3 seems to be more restrictive than usual assumption on boundedness of conditional expectation. In this form this assumption requires that noisy gradient could not depend on past trajectory. 

I will reconsider my opinion based on author rebuttal.

QUESTIONS:
Please comment about indicate weaknesses. Does convexity assumptions could be relaxed?

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-21