Title: Balancing Gradient and Hessian Queries in Non-Convex Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: non-convex optimization

critical point

gradient query

Hessian query

dimension-dependent method
Abstract: We develop optimization methods which offer new trade-offs between the number of gradient and Hessian computations needed to compute the critical point of a non-convex function. We provide a method that for a twice-differentiable $f\colon \mathbb{R}^d \rightarrow \mathbb{R}$ with $L_2$-Lipschitz Hessian, and input initial point with  $\Delta$-bounded sub-optimality and sufficiently small $\epsilon > 0$ outputs an $\epsilon$-critical point, i.e., a point $x$ such that $\|\nabla f(x)\| \leq \epsilon$, using $\tilde{O}(\Delta L_2^{1/4} n_H^{-1/2}\epsilon^{-9/4})$ queries to a gradient oracle and $n_H$ queries to a Hessian oracle. As a consequence, we obtain an improved gradient query complexity of  $\tilde{O}(d^{1/3}L_2^{1/2}\Delta\epsilon^{-3/2})$ in the case of bounded dimension and of $\tilde{O}(\Delta^{3/2} L_2^{3/4}\epsilon^{-9/4})$ in the case where we are allowed only a single Hessian query. We obtain these results through a more general algorithm which can handle approximate Hessian computations and recovers known prior state-of-the-art bounds of computing an $\epsilon$-critical point, under the additional assumption that $f$ has an $L_1$-Lipschitz gradient, with $O(\Delta L_2^{1/4}\epsilon^{-7/4})$-gradient queries.


Peer reviews:
comments: SUMMARY:
This paper studies non-convex minimization and offers new trade-offs between the number of gradient and Hessian computations.

The previous methods use $\mathcal{O}(\epsilon^{-1/4})$ gradient oracles in the subroutines to decrease the function value by a factor of $\Omega(\epsilon^{3/2})$, which leads to the $\mathcal{O}(\epsilon^{-7/4})$ queries to the gradient oracle.
This paper enhances the subroutine to $\mathcal{O}(\delta^{1/2}\epsilon^{-1/4})$ by incorporating in $\delta$-inexact Hessian oracle, which allows the reuse of the Hessian.
By properly tuning $\delta$, the proposed method can achieves better computational complexity than the lazy Hessian (Doikov et. al, 2023).

I find the theoretical results presented in this paper are interesting and novel. However, this paper lacks of experiments to check the efficiency of the proposed method, while it requires additional computational cost of the Hessian and its decomposition.

STRENGTHS AND WEAKNESSES:
**Strengths:** the theoretical results obtained by this paper is novel and significant in non-convex optimization. The new trade-off between the gradient and Hessian oracles largely improves the prior-art.

**Weakness:** the paper lacks of experimental part to verify the efficiency of the proposed methods and to validate the proper trade-off between the gradient and Hessians.

QUESTIONS:
1. Theorem 2 requires the condition such that $H \preceq L_1 I$ (line 156), I am wondering how can this be satisfied in Algorithm 2 when calling the Critical-or-Progress sub-routine?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-17

comments: SUMMARY:
The paper studies trading off gradient and Hessian computations for a faster convergence rate for computing the critical point in non-convex optimization. In particular, the paper develops a method that achieves the best known dimension-independent rate and improves upon prior works in finite dimensions.

STRENGTHS AND WEAKNESSES:
Strength: 

1. The paper has technical contributions and significance. The proofs are based on fine-grained analysis, coupled with delicate algorithm design, e.g., the choice of parameters in Eq. (4). Moreover, the theories possess sufficient generality to cover known results for gradient methods. 

2. The result is strong in improving the dimensional dependence compared to prior works under the same condition (Corollary 1, Table 1). 

3. Balancing the use of gradient and Hessian information for better overall cost (measured in gradient queries) is interesting. 


Weaknesses: 

1. The paper only considers gradient queries and does not provide computational complexity analysis. In particular, some steps might take a significantly larger amount of time to compute, e.g., transformed Hessian $\hat{H}$ in Eq. (2). It would be clearer to provide overall computational complexity in comparison with pure gradient methods and second-order methods. 

2. The writing is very technical. While this is not necessarily a weakness, it would be better to provide more intuition illustrating why a faster rate can be obtained.

QUESTIONS:
1. How easy is it to compute $\hat{H}$ and $\hat{H}^{-1}$ in Algorithm 1 for high-dimensional problems? Similarly, the Newton step in Algorithm 3?

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of finding a first-order stationary point of a smooth non-convex function using access to a gradient oracle and an approximate Hessian oracle. The core idea is to apply the restarted nonconvex accelerated gradient method of Li & Lin (2023), preconditioned by an approximate Hessian. Moreover, the Hessian estimate is updated in a lazy manner only when the distance between the current iterate and the last reference point exceeds a predefined threshold. This threshold controls the trade-off between gradient and (approximate) Hessian queries: a smaller threshold leads to more frequent Hessian updates and also fewer gradient steps due to more accurate second-order information. As a corollary, the authors derive a new dimension-dependent complexity bound using only gradient access, as well as a bound that requires only a single Hessian query.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- This paper combines several techniques from nonconvex optimization—including restarted nonconvex accelerated gradient methods, preconditioning, and lazy Hessian updates—in a non-trivial and interesting way. The result is an algorithm that explicitly balances the trade-off between gradient and Hessian computations.\n- Notably, the proposed method achieves a new complexity bound of $\\tilde{O}(d^{1/3}\\epsilon^{-3/2}+d)$ when the function has $L_2$-Lipschitz Hessian and only gradient access. This improves the dimension dependence over Doikov et al. (2023) and yields a better dependence on $\\epsilon$ than Li & Lin (2023) when the dimension is sufficiently small.\n\n## Weaknesses\n- The paper currently lacks a detailed discussion on the computational cost. Specifically, constructing the modified approximate Hessian matrix $\\hat{H}$ requires full eigendecomposition, which will incur a cost of $O(d^3)$.\n\nQUESTIONS:\n## Clarifying Questions\n- The construction of the modified approximate Hessian estimate $\\hat{H}$ in (3) appears somewhat ad hoc. In particular, the origin of the logarithmic terms is unclear. While the authors mention that these terms introduce spectral gaps and facilitate the use of matrix perturbation theory, a more detailed explanation or motivation for this specific form would be helpful.\n- Regarding the results in Corollary 1, since each approximate Hessian computation requires $2d$ gradient queries, I believe the complexity bound should be $\\tilde{O}(d^{1/3}L_2^{1/2}\\Delta \\epsilon^{-3/2} +d)$. Alternatively, if this cost is not counted explicitly, then the bound applies only when $\\epsilon \\leq L\\_2^{1/3} \\Delta^{2/3} d^{-4/9}$.\n\nLIMITATIONS:\nPlease refer to the \"Weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops novel optimization algorithms that achieve improved trade-offs between gradient and Hessian oracle queries for computing ε-critical points of nonconvex functions. The key contributions are:\n\n1. ​**Main Trade-off Result (Theorem 1):​**​ For twice-differentiable functions with L₂-Lipschitz Hessians and initial Δ-bounded suboptimality, the algorithm computes an ε-critical point using nₖ Hessian queries and $\\tilde{O}(\\Delta L_2^{1/4} n_H^{-1/2} \\epsilon^{-9/4})$ gradient queries with δ-approximate Hessians. This generalizes prior gradient-only methods.\n\n2. ​**Dimension-Dependent Improvement (Corollary 1):​**​ In d-dimensional space, the algorithm achieves $\\tilde{O}(d^{1/3} L_2^{1/2} \\Delta \\epsilon^{-3/2})$ gradient-only complexity via finite differencing, improving over prior $\\tilde{O}(\\sqrt{d})$ results when $\\epsilon$ is sufficiently small.\n\n3. ​**Single-Hessian-Query Complexity:​**​ When limited to just one exact Hessian query, the method attains $\\tilde{O}(\\Delta^{3/2} L_2^{3/4} \\epsilon^{-9/4})$ gradient queries, improving previous bounds by a polynomial factor.\n\n4. ​**Technical Innovations:​**​ The approach introduces (a) a restart mechanism with adaptive Hessian updates (Algorithm 2), (b) a reduction technique to handle functions without gradient Lipschitz continuity (Algorithm 3), and (c) careful analysis of accelerated gradient descent under approximate Hessian-induced norms.\n\nThese results advance nonconvex optimization by unifying and extending prior state-of-the-art complexities while enabling new computation trade-offs between oracle types.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- ​**Originality:​**​ The paper introduces a fundamentally new perspective on balancing gradient and Hessian queries in non-convex optimization, an underexplored dimension in complexity analysis. The approach of using δ-approximate Hessians to define custom norms (Ĥ in Eq. 2) and the reduction technique for unbounded gradient Lipschitz (Algorithm 3) represent novel algorithmic innovations not seen in prior work.\n\n- ​**Technical Quality:​**​ The theoretical analysis is rigorous and comprehensive. The authors provide full proofs for all claims (Appendix A–F), leveraging advanced tools like Davis-Kahan theorem for spectral perturbation analysis (Lemma 7) and carefully structured induction arguments. The main results (Theorems 1–4) strictly generalize prior bounds (e.g., recovering [28] when $\\theta = L_1$).\n\n- ​**Significance:​**​ The results meaningfully advance non-convex optimization theory by:  \n  (i) Establishing the first trade-off rates between gradient/Hessian queries (Theorem 1),  \n  (ii) Improving dimension-dependent complexity to $\\tilde{O}(d^{1/3})$ (Corollary 1), beating the prior $\\tilde{O}(\\sqrt{d})$ barrier for $d ≥ 7$,  \n  (iii) Enabling non-trivial critical point computation with only one Hessian query – a practical scenario where the paper shows $\\tilde{O}(\\epsilon^{-9/4}) $ gradient complexity, polynomially better than prior arts.\n\n- ​**Clarity:​**​ The paper is exceptionally well-structured. The abstract and introduction clearly contextualize contributions against 15+ cited works (Table 1 is particularly effective). Algorithm descriptions are precise, and §2 provides excellent high-level intuition before technical details. Notation is consistent and well-defined throughout.\n\n## Weaknesses\n\n- ​**Practical Limitations:​**​ While theoretically novel, the algorithms rely on expensive operations like spectral decomposition of $\\tilde{H}$ (Eq. 2) and may incur large constant factors hidden in $\\tilde{O}(\\cdot)$. The logarithmic dependencies on problem parameters (e.g., $\\log(L_1/\\theta)$) could be prohibitive in real-world implementations, though this is common in theoretical optimization.\n\n- ​**Optimality Questions:​**​ While the paper improves upper bounds, it doesn't establish matching lower bounds for the new trade-off regimes. The relationship to the known $\\Omega(\\epsilon^{-12/7}$ gradient-only lower bound [11] is discussed, but Hessian-query lower bounds remain open.\n\n- ​**Scope Constraints:​**​ The analysis assumes exact access to a $\\delta$-approximate Hessian oracle. In practice, such oracles might require auxiliary computations (e.g., via finite differencing in Corollary 1), but the complexity of implementing these oracles isn't incorporated into the main trade-off results.\n\nQUESTIONS:\n**1. On the tightness of dimension-dependent complexity (Corollary 1):​**​  \nThe $\\tilde{O}(d^{1/3})$ gradient complexity in Corollary 1 improves prior $\\tilde{O}(\\sqrt{d})$ bounds but still has a polynomial dimension dependence. Could you comment on whether this exponent is theoretically tight? Specifically:  \n- Is there a matching lower bound for $d^{1/3}$ in this Hessian-approximation-via-finite-differencing setting?  \n- Does the exponent $1/3$ stem from fundamental limitations or current proof techniques?  \n*Actionable request*: Provide intuition for the $d^{1/3}$ term (e.g., whether it arises from the $\\min(c_\\delta)$ trade-off in Theorem 1 or finite differencing error propagation). A clear explanation would strengthen significance; conversely, failure to clarify may weaken perceived novelty.  \n\n​**2. Handling stochastic or inexact Hessian approximations:​**​  \nTheorem 1 assumes access to a deterministic $\\delta$-approximate Hessian oracle. Many practical implementations (e.g., via sketching or subsampling) yield *stochastic* approximations with variance $\\sigma^2$.  \n- Can your analysis accommodate stochastic Hessians with $\\mathbb{E}[\\|H_x - \\nabla^2 f(x)\\|^2] \\leq \\sigma^2$?  \n- If so, how would this affect the complexity (e.g., would terms like $\\Delta \\sigma / \\epsilon$ appear)?  \n\n\n**3. Practicality of spectral computations for $\\widehat{H}$:​**​  \nAlgorithm 1 requires spectral decomposition of $H_{x^{(0)}}$ to construct $\\widehat{H}$ (Eq. 2), which costs $O(d^3)$ and negates low-query benefits in high dimensions.  \n- Can $\\widehat{H}$ be applied implicitly (e.g., via Krylov methods) without full decomposition?  \n- Does your analysis support approximate eigenvectors (e.g., from power iteration)?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}