Title: Stability and Sharper Risk Bounds with Convergence Rate $\tilde{O}(1/n^2)$
Venue: NeurIPS 2025
Accepted: yes
Keywords: algorithmic stability

generalization bounds

excess risk bounds

stochastic gradiet descent
Abstract: Prior work (Klochkov \& Zhivotovskiy, 2021) establishes at most $O\left(\log (n)/n\right)$ excess risk bounds via algorithmic stability for strongly-convex learners with high probability. We show that under the similar common assumptions — Polyak-Lojasiewicz condition, smoothness, and Lipschitz continous for losses — rates of $O\left(\log^2(n)/n^2\right)$ are at most achievable. To our knowledge, our analysis also provides the tightest high-probability bounds for gradient-based generalization gaps in nonconvex settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a method to show generalization bounds under uniform stability of gradients for methods optimizing Lipschitz, smooth and strongly convex functions. Starting with a generic generalization bound that replaces that dependence on the Lipschitz constant in prior work to expectation of squared length of the gradient, the paper then shows sharper bounds for ERM, PGD and SGD. Notably, in the low noise regime ($F(w^∗) <O(1/n)$), the bounds for ERM, PGD  is $\\tilde{O}(1/n^2)$. For SGD in a single pass ($T=n$), the bound is $\\tilde{O}(1/n)$ in high probability, and when $T=n^2$, the bound is $\\tilde{O}(1/n^2)$, sharper than previously known results. Another result is a constant-level improvement to concentration of sums of vector valued functions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- I find the paper technically strong and all results are presented clearly. In particular, all results are presented in detailed comparison with prior works, which highlights where both the improvements and limitations are.\n- The improvements in the bounds are meaningful and interesting. I think improving the generalization bound from $\\tilde{O}(1/n)$ (limit of prior works) to $\\tilde{O}(1/n^2)$ even under more restrictive assumptions is non-trivial. \n- I don't find any big issues in the proofs except some minor typos.\n\n**Weaknesses**\n- As mentioned above, the main weakness of the paper comes from the fact that the analysis requires stronger assumptions than do prior works (eg, PL vs SC, requiring lower bounds for the sample size). Here, I think the authors should revise the language in the abstract (\"under the same common assumptions\"), which is not correct.\n- Since the paper mainly focuses on the dependence on the sample size $n$, it is unclear how the bounds in terms of problem parameters are improved or degraded. Here a comparison with prior work will be helpful.\n\nMinor:\n- Sentence fragments in Line 337-339. \n- Missing $C_p$ in Eq (7) Line 530.\n\nQUESTIONS:\n- Can the authors provide a comparison with prior works in terms of the dependence on the problem parameters (Lipschitzness, smoothness, strong convexity parameters)?\n\nLIMITATIONS:\nAdequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce new high-probability excess risk upper bounds based on uniform stability in gradients. They first develop a general theoretical framework for deriving such bounds, under the assumptions of a Polyak–Łojasiewicz (PL) condition, smoothness, and Lipschitz continuity of the loss function. They then apply this framework to several learning algorithms, including Empirical Risk Minimization (ERM), Projected Gradient Descent (PGD), and Stochastic Gradient Descent (SGD). In particular, they show that in low-noise regimes, the excess risk bounds can achieve the fast rate of $\\tilde{O}(1/n^2)$, notably without any dependence on the dimension $d$.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nOverall, the paper presents a very interesting, original, and tighter approach to deriving high-probability upper bounds on the excess risk of learning and optimization algorithms. I particularly appreciate that the authors focus on high-probability bounds, in contrast to the vast majority of stability-based generalization results that primarily address the expected excess risk. Moreover, the theoretical results are thoroughly analyzed, the optimality of the bounds is discussed, and the comparisons to related literature are well-executed and informative.\n\n### Weaknesses\n\nOne weakness I see concerns the notion of stability in gradients that the authors adopt. From my understanding, the stability upper bound holds almost surely with respect to the randomness of the algorithm $A$, which is not standard and appears to be a significant limitation that deserves further discussion. Besides that, the paper is relatively dense, and the writing could be improved in places to enhance clarity and accessibility. For example, it is not entirely clear whether all results need to be stated in the main text (e.g., Theorem 1 does not seem essential and could be moved to the appendix). On the other hand, some of this saved space could be better used to provide more intuition or discussion about the proof techniques behind Theorems 2 and 3, which are the core contributions of the paper. Beyond these points, please see my detailed suggestions and questions below.\n\nQUESTIONS:\n1) Could you confirm that the upper bound in Definition 2 holds almost surely with respect to the randomness of the algorithm $A$? If so, this is not standard and actually corresponds to a stronger notion of stability than what is typically considered in the literature, where an expectation is usually taken with respect to the randomness of $A$ (see, e.g., [23]). I would appreciate it if the authors could clarify this point and clearly discuss this aspect in the main text.\n\n2) The results in Section 3 are derived under the Polyak–Łojasiewicz (PL) condition, while in the applications, the objectives considered are strongly convex. Could the application part be extended to handle objectives that satisfy only the PL condition, without requiring strong convexity? Some discussion on this point would help clarify the generality of the framework.\n\n3) I believe there may be a missing $M^2$ factor in the third term of the upper bound in Theorem 3. While $M^2$ can be regarded as a constant, it is not purely numerical and should likely appear explicitly in the bound. Could the authors clarify this point? Also, is it possible that the dimension $d$ is implicitly hidden in $M$? An explicit statement on this would be useful.\n\n4) In Definition 3, is $\\rho$ meant to be a constant? If so, I suggest using a different notation, as $\\rho$ is already used to denote the distribution of $z$. Additionally, did you absorb the dependence on $\\rho$ into $C$ in the proof of Proposition 1? As above, it would be clearer if these constants appeared explicitly in the final bounds.\n\nLIMITATIONS:\n### Limitations \n\n- Please see my earlier comment regarding the specific notion of stability in gradients adopted in this paper. I am concerned that this almost-sure notion of stability might be more restrictive to establish for randomized optimization algorithms (beyond SGD), and could potentially lead to suboptimal bounds, especially in terms of the number of iterations $T$ required to achieve the stated rates. I would strongly encourage the authors to discuss this limitation explicitly. I am open to raising my overall score if this point is properly addressed.\n\n- I believe Theorem 1 and Remark 2 could be moved to the appendix. This would free up space in the main text and allow the authors to expand on the proof sketch or main ideas behind Theorem 2, which currently deserve more emphasis.\n\n- I recommend that the authors include precise cross-references to the proofs provided in the appendix. At present, the appendix sections are not explicitly cited in the main text, which makes it harder for readers to follow the technical arguments.\n\n- It should be clearly stated in the notation section that the symbol $\\lesssim$ is used to hide (numerical?) constant factors in the upper bounds, while $\\mathcal{O}$ and $\\tilde{\\mathcal{O}}$ follow the Landau notation conventions for asymptotic comparisons (the latter hiding polylogarithmic factors). In machine learning papers, $\\lesssim$ and $\\mathcal{O}$ are sometimes used interchangeably, but here they have distinct meanings. Clarifying this would avoid confusion.\n\n### Typos\n\n- l.12 \"algorithmic\" $\\rightarrow$ \"algorithmic stability\"?\n- l.50 \"and $\\mathcal{O}(1/n)$\"$\\rightarrow$ \"and terms in $\\mathcal{O}(1/n)$\"?\n- l.120 \"under\" $\\rightarrow$ \", the\"\n- l.164 \"stable\" $\\rightarrow$ \"stability\"\n- l.171 \"$\\gamma$-smoothness\" $\\rightarrow$ \"$\\gamma$-smooth\"\n- l.198  \"even IF the algorithm\"\n- l.235-236 \"can BE up to\"? \n- l.254 \"analysis\" $\\rightarrow$ \"analyze\"\n- Proof of Theorem 2 (l.630), first equality: there is an extra bracket \"]\". This is repeated several times in the proof.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the question whether tighter high-probability excess risk bound tighter than $O(1/n)$ can be obtained using algorithmic stability-based methods and under common assumptions (e.g., strong convexity, smoothness, Lipschitz, PL ...).\nTo answer this question, an dimension-free $\\tilde{O}(1/n^2)$ bounds are derived under Lipschitzness, smoothness and PL condition. \nA dimension-free improved gradient generalization is first proved to potentially allow $O(1/n)$ rates under suitable stability and regularity. Then combined with PL condition, the generalization of gradient is translated into the generalization of model weight, where the square in the PL condition promotes the rate to potentially $O(1/n^2)$ under suitable stability. Finally, the stability bounds are applied to ERM, PGD and SGD by proving plugging-in their stability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. This paper derives the first dimension-free $\\tilde{O}(1/n^2)$ bound on excess risk under commonly assumed strong convexity/PL, smoothness, Lipschitzness. The results are carefully compared (e.g., Table 1 and the remarks after each Theorems) and are optimal under some conditions, demonstrating the superiority in convergence rate or dimension dependence. \n2. The idea of using PL condition to bring a square in stability bounds and accelerate its rate is quite novel.\n3. The contribution is not limited to generalization of risks, but also the generalization w.r.t. gradients. The new generalization bound w.r.t. gradient has the same convergence rage but has dimension-free requirements on $n$ and also applies to non-convex and non-smooth cases under Lipschitzness and SGC.\n4. The paper is clearly structured and well written.\n\n## Weaknesses\n\n\n1. For SGD results to have $O(1/n^2)$ rate, the step number has to be $T = n^2$. Moreover, Theorems 6 and 7 have $\\log^{\\{3, 5\\}} \\frac{1}{\\delta}$ terms of very high order.\n2. Aside from the use of PL condition and the consequential order improvement in convergence rate, the technique to achieve them is based on Klochkov and 147 Zhivotovskiy [19]. Nevertheless, I believe this is minor compared to the $\\tilde{O}(1/n^2)$ rate.\n3. Minor/Typos: \n   1. The sentence in L267-L269 is quite confusing: \"However, the generalization analysis method proposed by Klochkov and Zhivotovskiy [19] does not use smoothness assumption, which only derive ... under strongly convex and smooth assumptions.\"\n\nQUESTIONS:\n1. Is it possible to decrease the order of $\\log \\frac{1}{\\delta}$ terms in Theorems 6 and 7?\n2. The following questions may be not necessary. It would be greatly appreciated if the authors could satisfy our curiosity. In [19], an $\\tilde{O}(1/n)$ bound is established under strong convexity and Lipschitzness, and in this paper an $\\tilde{O}(1/n^2)$ bound is established under strong convexity, Lipschitzness and Smoothness, which can be seen as the Lipschitzness of gradients or \"first-order Lipschitzness\". Therefore, is it possible that given strong convexity and $0-, 1-, \\dots p-$th order Lipschitzness (perhaps with more minor assumptions), an $\\tilde{O}(1/n^{p+1})$ convergence can be established. It is particularly interesting after realizing that given high-order Lipschitzness (e.g., 2-order Lipschitzness on Hessian) and extra assumptions of boundedness of hypothesis space, one can prove lower-order Lipschitzness by integral and obtain lower-order (at least 2-order) convergence. Then the only remained question is whether the high-order Lipschitzness can be \"fully\" exploited to achieve $(p+1)$-order convergence.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents high-probability excess risk bounds for empirical risk minimization (ERM) problems and projected gradient descent (PGD)/ stochastic gradient descent (SGD) algorithms. The authors achieve dimension-free, O(1/n) generalization bounds by generalizing the standard
concept of algorithmic stability of loss function to its gradient. This approach allows for sharper generalization bound. 

This work also combines the derived generalization bound on gradient and analysis on optimization error for strongly convexity or Polyak-Lojasiewicz (PL) condition and implies a sharp bound on the excess risk.

STRENGTHS AND WEAKNESSES:
Strengths 
- The generation bounds were developed before, but the one in this paper is dimension free. 
- The results are well-presented. 

Weaknesses
- The technical novelty is somewhat limited, with proofs heavily relying on appropriate combinations or refinements of existing techniques (e.g., Zhang & Zhou 2019, Klochkov & Zhivotovskiy 2021, Fan & Lei 2024).
- In the strongly convex/ PL setting, if $\mu$ is small, the bound is not good as it has $1/\mu^3$ 
- The assumption $n \geq \frac{16\gamma^2 \log(6/\delta)}{\mu^2}$ and $F(w*) < O (1/n)$ might not be practical in the strongly convex applications.

QUESTIONS:
Comment: The authors should specify what you mean by PGD. I thought it was GD but I have to find the original reference to know it was projected GD.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03