Title: Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-convex optimization

stochastic optimization

second order stationary points

generalized smoothness
Abstract: We study the optimization of non-convex functions that are not necessarily smooth (gradient and/or Hessian are Lipschitz) using first order methods. Smoothness is a restrictive assumption in machine learning in both theory and practice, motivating significant recent work on finding first order stationary points of functions satisfying generalizations of smoothness with first order methods. We develop a novel framework that lets us systematically study the convergence of a large class of first-order optimization algorithms (which we call decrease procedures) under generalizations of smoothness. We instantiate our framework to analyze the convergence of first order optimization algorithms to first and *second* order stationary points under generalizations of smoothness. As a consequence, we establish the first convergence guarantees for first order methods to second order stationary points under generalizations of smoothness. We demonstrate that several canonical examples fall under our framework, and highlight practical implications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general framework for establishing high-probability convergence of an algorithm to a target solution set $\\mathcal{S}$ and for analyzing its oracle complexity.\nUsing this framework, the authors prove convergence of several algorithms to first-order or second-order stationary points, assuming the objective function $F$ satisfies self-bounding regularity properties, a generalization of $(L\\_0, L\\_1)$-smoothness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe proposed perspective of unifying convergence proofs offers a systematic and fundamental view of how such arguments are structured, making it a valuable contribution to the literature.\nThe central question of finding first- and second-order stationary points without assuming Lipschitz continuity of the gradient or Hessian is well-motivated and clearly explained.\nGiven the fact that self-bounding regularity conditions generalize the recently popular $(L_0, L_1)$-smoothness, the main results showing convergence to SOSPs under second- and third-order self-bounding conditions seem compelling.\n\n\n**Weaknesses**\n\nThe unification comes at a cost. Section 2.2 is difficult to follow and contains formalisms that appear unnecessarily complex.\nThe precise roles and meaning of $\\mathcal{A}\\_2$ and $\\mathcal{R}$ are not clearly explained and lead to confusion (lines 156–158).\nFor instance, why not simply define $\\mathcal{A}$ as a procedure that outputs $\\mathcal{A}\\_1$ (the output with high probability decrease of $F$) and $\\mathcal{R}(\\mathcal{A}\\_2)$ (the candidate solution), given that $\\mathcal{R}$ and $\\mathcal{A}\\_2$ always appear together?\nThis would eliminate the need to repeatedly use cumbersome notation involving the set of sequences $\\cup\\_{n=0}^\\infty (\\mathbb{R}^d)^n$ in defining $\\mathcal{A}$.\nOverall, the writing and notation in this section can, and should, be significantly improved for clarity and simplicity.\n\nThere are also numerous issues with notation, typos, and minor errors in the proofs. A few examples include:\n- The notation $\\mathcal{A}(u\\_0) = (u\\_0 - \\eta \\nabla F(u\\_0)) \\times u\\_0$ is not appropriate for denoting an element of a product set and should instead be written as a tuple; the current form is confusing.\n- The definition of a decrease procedure does not clearly distinguish between randomness due to the algorithm itself and noise from sampling (e.g., for SGD and restarted SGD).\n- In line 229, is it necessary to define $\\mathcal{R}$ for sequences of any length?\n- In the proof of Theorem 2.1, using $\\mathcal{A}$ to denote an event overlaps confusingly with the notation for the algorithm.\n- Also in that proof, the authors write $\\mathcal{E}_n \\cap \\mathcal{B} \\cap \\mathcal{C}$ even though $\\mathcal{C} \\subseteq \\mathcal{B} \\subseteq \\mathcal{E}_n$, which is needlessly confusing.\n- In lines 1181 and 1182, $p\\_{n-1}$ should likely be $p\\_{n+1}$.\n- In line 1205, the first inequality appears to have an incorrect sign.\n- In line 1206, the authors presumably meant $k'=1$ is not possible so $k' \\ge 2$, as with $k'=0$ the summation up to $k'-1$ is vacuous?\n- In line 1209, $p_i$ is in place of $w_i$.\n- In line 1211, the left side of the implication should be $N \\le \\sum_{i=0}^{k'} t_{oracle}(w_i)$?\n\nEven though the paper is lengthy and involves many intricate parts, I believe it is the authors' responsibility to ensure that the paper is presented in a clear and polished manner to merit publication.\n\nQUESTIONS:\n- How meaningful is the third-order self-bounding regularity? While I understand that the second-order version is related to $(L\\_0, L\\_1)$-smoothness (which is already well understood and whose relevance to applications is widely accepted in the literature) it is unclear whether the same holds for the third-order condition. Although the authors demonstrate that it holds for phase retrieval and matrix PCA, it would be helpful to provide a more general discussion regarding its practical significance.\n\n- How general is Assumption 3.1? It is clearly more general than the assumption of uniformly bounded variance, but it may be beneficial if the authors could further clarify its scope by drawing more explicit connections to prior work such as [1], devoted to relaxing assumptions on the noise.\n\n- Are there no experiments on phase retrieval or matrix PCA? If so, the last paragraph of Subsection 3.6 (lines 370–374) may be better separated from the rest of the subsection, as it gave the misleading impression that it was pointing to experimental results on those problems.\n\n[1] Gower et al., SGD: General Analysis and Improved Rates. ICML, 2019.\n\nLIMITATIONS:\nThe paper does not explicitly discuss its limitations, but this seems acceptable given the theoretical nature of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
An open problem in ML is the development of optimisers that can provably avoid strong saddle points. The authors define a condition SOSP of the estimated optimal point, saying that the norm of the gradient at the should be less than epsilon, and the minimal eigenvalue of the Hessian should be more than -sqrt(epsilon). The paper proposes some new variants of gradient descent, adaptive gradient descent, and stochastic gradient descent, and shows that these variants, under suitable assumptions on the target function, are able to find solutions of the SOSP problem, and give complexity bounds for these solutions.

STRENGTHS AND WEAKNESSES:
Strengths: The paper has some novel theoretical results for finding FOSP and SOSP about several deterministic and stochastic optimisation algorithms. Many of these bounds are sharp.

Weaknesses: Assumption 1.1 of the paper on the Hessian being upper bounded by a function of function value (||Hess f(x)||<= rho_1(f(x))$ are not unreasonable, but it is unfortunate that the algorithms explicitly depends on this function rho_1 (see equation (4)).
Since in practical ML problems, Assumption 1.1 may be difficult to verify.
There are no convincing numerical experiments showing the usefulness of the algorithms developed in this paper.

QUESTIONS:
Would it be possible to develop a truly adaptive version of the algorithm where the algorithm's steps do not depend on the parameters in Assumptions 1.1-1.2, but these only affect the convergence rates?

LIMITATIONS:
Limitations have been adequately addressed.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This purely theoretical paper addresses the problem of finding first- and second-order stationary points (FOSP and SOSP) using first-order optimization methods under weak regularity assumptions: second and third order self-bounding regularity. To that end, a decrease procedure is introduced showing when first order methods decrease the value function, leading to convergence guarantees. It is shown that many common first order algorithms (GD, Adaptive GD, SGD, Perturbed GD and Restarted SGD) are covered by the framework and be analyzed accordingly.

STRENGTHS AND WEAKNESSES:
Strengths

- The problem seems important for the ML community, especially for non-convex optimziation
- Authors provide a thorough and unified convergence analysis for a wide variety of first-order algorithms
- The high level introduction of the decreasing procedure is clear and very helpful, improving the paper's readability.

Weaknesses

- Experimental results could have been introduced in the main paper, even briefly.
- Generality of the self-bounding regularity is not extensively discussed, it is a generalization of (L_0, l_1)-smoothness but are there many known functions satisfying self-bounding regularity and not (L_0, L_1)-smoothness? How frequently do such functions arise in deep learning?

QUESTIONS:
- How common are the self-bounding regularity conditions in modern ML, eg in deep learning? 
- You notice that the choice of the step size \eta should depend on suboptimality at initialization, how sensitive is the method in practice ?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-10