Title: A Regularized Newton Method for Nonconvex Optimization with Global and Local Complexity Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: smooth nonconvex optimization; regularized Newton method; worst-case complexity; adaptive algorithm
Abstract: Finding an $\epsilon$-stationary point of a nonconvex function with a Lipschitz continuous Hessian is a central problem in optimization. Regularized Newton methods are a classical tool and have been studied extensively, yet they still face a trade‑off between global and local convergence. Whether a parameter-free algorithm of this type can simultaneously achieve optimal global complexity and quadratic local convergence remains an open question. To bridge this long-standing gap, we propose a new class of regularizers constructed from the current and previous gradients, and leverage the conjugate gradient approach with a negative curvature monitor to solve the regularized Newton equation. The proposed algorithm is adaptive, requiring no prior knowledge of the Hessian Lipschitz constant, and achieves a global complexity of $O(\epsilon^{-\frac{3}{2}})$ in terms of the second-order oracle calls, and $\tilde O(\epsilon^{-\frac{7}{4}})$ for Hessian-vector products, respectively. When the iterates converge to a point where the Hessian is positive definite, the method exhibits quadratic local convergence. Preliminary numerical results, including training the physics-informed neural networks, illustrate the competitiveness of our algorithm.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on regularized Newton methods, which are widely studied yet still face a trade-off between global and local convergence. A key open question is whether a parameter-free algorithm can achieve both optimal global complexity and quadratic local convergence. To tackle this issue, the authors propose a regularizers using current and previous gradients, and apply a conjugate gradient approach with a negative curvature monitor to solve the regularized Newton equation. The algorithm is adaptive and does not require prior knowledge of the Hessian Lipschitz constant. The proposed method achieves a global complexity of $O(\\epsilon^{-\\frac{3}{2}})$ in terms of second-order oracle calls and $\\tilde{O}(\\epsilon^{-\\frac{7}{4}})$ for Hessian-vector products. When the iterates converge to a point where the Hessian is positive definite, the method demonstrates quadratic local convergence. Preliminary numerical results highlight the competitiveness of the proposed algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n- The overall structure of the paper is clear and well-organized, with a thorough literature review.\n- The explanation of the overall algorithmic process is comprehensive, allowing readers to clearly follow the algorithm's workflow. Additionally, although I have not thoroughly checked every step of the proof, it appears that the theoretical justifications are detailed and rigorous, providing solid theoretical guarantees.\n- The paper introduces an adaptive Newton-CG algorithm, which cleverly detects the local region in practice. This enables the algorithm to adaptively balance the trade-off between global convergence and local convergence.\n- The design for solving the Newton equation for subproblems is innovative, allowing for a cost-effective way to determine the presence of negative curvature directions.\n- The experimental setup is detailed, with numerous experiments conducted to validate the effectiveness of the proposed algorithm.\n\n### Weaknesses:\n- Some of the symbols in the paper need clearer explanations. For example, the meaning of the symbol $\\lesssim$ is not explained. Although line 87 mentions \"The notations... are used in their standard sense to represent asymptotic behavior,\" a more detailed explanation would help readers who are not familiar with complexity theory.\n- On line 260, \"lemma\" should be \"proposition\". Additionally, on line 272, the definition of $\\alpha$ should be provided. On line 303, there is a missing space before the final \"and\" in the sentence. Lastly, on lines 951 and 956, the formulae should end with a comma instead of a period.\n- The paper is somewhat lengthy. Although this is done for theoretical completeness, it makes the review process for a conference submission more difficult.\n\nQUESTIONS:\n- In Table 1, why is the $\\tilde{O}(1)$ term missing from the iteration complexity corresponding to He et al. [29, Theorem 1]? From the regularization coefficient, it seems they also assume a lack of prior knowledge about $L_H$ and make an estimate accordingly. Additionally, the phrase \"$\\epsilon$ is used in the regularization coefficient or minimal eigenvalue computation\" is unclear. Could the authors briefly explain why $\\epsilon$ is involved in these two places? A simple clarification here might enhance the readability.\n- In line 13 of Algorithm 1, is there any theoretical guarantee for the value of $\\alpha$? Could the authors provide more insight into this?\n- Another point of interest is whether the value of $\\theta$ in the algorithm could be adaptively selected rather than using a constant value throughout the entire algorithm? This could potentially improve performance.\n- It appears that ARC$_q$K shows excellent time efficiency in most experiments. However, while the authors mention using slightly fewer Hessian evaluations in line 306, it does not seem to result in a better time efficiency. Could the authors further emphasize the practical advantages of the proposed algorithm compared to other methods, particularly in real-world applications? Highlighting these advantages will help readers understand the practical utility of the algorithm better.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an adaptive Newton’s method that achieves both global and local convergence. In general, optimization algorithms require different hyperparameter settings to ensure global versus local optimal convergence, and these settings are often heuristic and not known in advance. The proposed method addresses this challenge by introducing a regularization term delta_k, which adaptively monitors the algorithm’s behavior to balance global and local convergence. This adaptive scheme enables the algorithm to achieve optimal performance in both regimes, in contrast to previous methods that rely on fixed hyperparameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n(1) The algorithm is an adaptive Newton-CG, which is both locally and globally convergent. The technique is intuitively convincing.\n\n(2) Theoretical analyses are thorough and rigorous, including the results in the main content and appendices.\n\n(3) Experiments are comprehensive, including CUTEst and PINNs.\n\n**Weaknesses:**\n\nThe method requires the computation of exact Hessian matrix.\n\nQUESTIONS:\n**Questions:**\n\n(1) When training deep neural networks, how to handle the Hessian matrix? In general, the exact computation of Hessian on the full batch is difficult. Do you use Hessian-free (Hessian-vector products) methods or quasi-Newton methods that does not use the exact Hessian? Or do you adopt lazy Hessian?\n\n(2) Can the algorithm be extended to be stochastic? Most of machine learning problems may require mini-batch sampling. What is the theoretical convergence guarantees?\n\n(3) In Figure 2, ARC and ANRCG performs comparably. Can you please explain? I think ARC is an adaptive version of CR, which also enjoys the (optimal) global and local convergence. The idea of ANRCG should be similar to ARC, am I correct? I also know another parameter-free Newton’s method (https://arxiv.org/abs/2311.11489) that enjoys global and local convergence, can you discuss? I guess the core idea should be very similar.\n\n(4) What are the theoretical convergence results if without Lipschitz smoothness? Is it possible?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to create a parameter-free algorithm that achieves optimal convergence rates, both locally and globally.  They propose a new class of regularizers based on gradient information and leverage a CG approach with potential early exit if negative curvature information is detected.  Their algorithm leverages the “Capped CG” approach that is essentially CG that additionally exits with negative curvature conditions either if detected explicitly or convergence rates are detected to be too slow, in which case the CG iterates are regenerated to find negative curvature (B1 page 17).  It may also terminate if the convergence of CG exceeds the theoretical threshold denoted by $J(M, \\bar\\rho, \\xi) + 1$.  An addition LipEstimation function is used to improve the estimate of the Lipschitz factor M that is either increase by a factor of $\\gamma$ or decreased by a factor $1/\\gamma$.\nTheir algorithm is described in “Algorithm 1” where the function NewtonStep is called potentially twice, first with a trial step and if the step returns FAIL, a fallback step is used.  The heart of NewtonStep is CappedCG based on Royer [49].  Additionally, it provides the choice regularized based on the estimate of $ L_H$,$ M$, as well as $\\omega_k$ based on past gradient norms.  They seek to balance the need for the regularization to be proportional to $\\sqrt(g_k)$ far from the solution but smaller than $g_k$ when close.  That is, unmodified Newton’s method without a regularizer would ideal if near a positive-definite solution point.  So in a sense the regularize must converge at a much slower rate until sufficiently close to a solution.  So the magic is to detect that the algorithm is behaving like it is sufficiently close or far while balancing that fact that the estimate of $L_H$ itself maybe part of the problem.\nAfter CappedCG, post processing handles the CG converged cases, versus max iteration and/or negative curvature found. They provide the option for a back-tracking like line search to find $\\beta^m$ where $m$ is an integer as close to $0$ as possible to get the desired decrease.  In practice, it seems that $m_{max} = 1$ works, indicating the machinery of the algorithm is correctly balancing the step.  (If larger values of $m$ were needed, this would hint that the direction was too large and the system was not sufficiently regularized for the current location.)\nThey provide comprehensive theoretical convergence proofs as well as numerical results with some CUTEr problems and a physics-informed neural network.\n\nSTRENGTHS AND WEAKNESSES:\nWeakness and Strength\nThe paper has an impressive amount of theory backing and seems to achieve its goals of proving desire convergence theoretical bounds for the proposed algorithm.  The numerical results hint that the method is competitive against similar algorithms in this genres.  It provides sound/comprehensive algorithms for generating steps, determining the regularization sequence sizing, accounting for negative curvature, and improving the estimate for $L_H$.  It is sufficiently balanced in that, according to page 40, 1123, the choice of $m_{max}=1$ seems to work well, implying that the algorithm often can take full steps along the generated direction.  The algorithm has a good number of hyperparameters; it appears that very little tuning was needed, and good default values are known.\n\nUltimately, I think this is a great paper for an optimization journal where significantly longer lengths are permitted.  The algorithm and ingredients is sufficiently complex and interdependent that paper is not comprehensible without going deep into the appendix. Algorithm 1 is itself based on CappedCG and LipEstimation, defined in the appendix, which are critical players to understand the algorithm. The Appendix itself ends on page 48 and the material is not supplemental but critical to the entire paper.  Thus this is not a 9 page paper but a much larger optimization paper that has essentially moved its body to the index.\nLastly, the algorithm seems overly complex for what it is attempting.  It is essentially using CG algorithm with early exits if negative curvature is detected.  It seems a cubic regularized variant of shifted Steihaug-Toint algorithms that behave very similarly: https://optimization-online.org/wp-content/uploads/2004/09/960.pdf and work well for the CUTEr problems, as does Steihaug-Toint itself: https://epubs.siam.org/doi/10.1137/0720042.  The method seems similar in nature to papers like this: https://arxiv.org/pdf/2311.11489 .\n\nQUESTIONS:\nIs it possible to clarify the algorithm and make it easier to follow, as done by some of the earlier links?  I needed to jump around a lot to follow.\n\nFor the CUTEr problems, I think it is important to compare to a simple shifted Steihaug-Toint variant and explicitly report the number of iterations, evaluations, and matrix-vector products in a table, or reference such tables from similar papers.  \n\nI think the paper has a lot of strong results, but it is just too long for an article of this length.  Is it possible to lay out the information for better clarity, making it less necessary to flip back and forth from the appendix to the starting page?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors propose a novel regularized Newton’s method and demonstrate that two classes of regularizers can achieve both optimal global convergence rates and quadratic local convergence. These new regularizers are constructed using current and historical gradient information, and the authors employ the conjugate gradient method with negative curvature to solve the regularized Newton equation. Notably, the proposed algorithm requires no prior knowledge of the Hessian Lipschitz constant while simultaneously attaining optimal global complexity and quadratic local convergence. Empirical results from numerical experiments align well with the theoretical analysis.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a high-quality theoretical optimization submission with a clear structure and well-written language. It offers two notable theoretical contributions to second-order Newton’s methods:\n\n1. The proposed algorithm is adaptive and parameter-free, meaning it does not require any prior knowledge of the Hessian Lipschitz constant or other parameters. All assumptions are clearly stated in Assumption 2.1, which is relatively simple. This provides a practical advantage over other algorithms that require parameters of the objective function, which can be difficult to estimate in real-world applications.\n\n2. The proposed regularized Newton’s method simultaneously achieves optimal global complexity and quadratic local convergence rates, making it a standout contribution. Combined with its parameter-free nature, this algorithm is both theoretically significant and practically applicable to general non-convex optimization problems.\n\nFurthermore, the authors validate their theoretical claims through numerical experiments on problems such as physics-informed neural networks, with empirical results consistent with the theoretical analysis.\n\nWeaknesses:\n\n1. Although the iteration complexity is clearly stated, the paper should provide more detailed analysis of the computational complexity per iteration, particularly regarding the computational cost of the capped conjugate gradient method used in the algorithm. While the oracle complexity is presented in Theorem 2.3, some of the oracle complexity discussion currently in the appendix should be moved to the main text to improve accessibility.\n\n2. The authors should clarify and be more specific in Theorem 2.2 regarding the local quadratic convergence rates. Instead of the Big-O notation $g_{k + 1} = \\mathcal{O}(g_k^2)$ it would be helpful to present the exact form of the local quadratic convergence rate. Additionally, the precise size of the neighborhood where quadratic convergence occurs and the number of iterations required to reach this neighborhood should be clearly stated.\n\n3. In the numerical experiments, it is recommended that the authors compare the proposed Adaptive Regularized Newton-CG method against established quasi-Newton methods such as BFGS, DFP, SR1, and L-BFGS. Moreover, conducting experiments on both convex and non-convex problems would provide a more comprehensive assessment of the algorithm’s performance across different settings.\n\nIn summary, this submission is a high-quality optimization paper and deserves a publication.\n\nQUESTIONS:\nCould the authors conduct numerical experiments on simpler or toy objective functions, in addition to the more complex non-convex problems? Currently, the results focus on challenging benchmarks such as the CUTEst suite and physics-informed neural networks. While these are valuable examples of non-convex problems, the reviewer is interested in seeing how the proposed algorithm performs on much simpler non-convex functions—for example, a quadratic function with an indefinite Hessian matrix. Such experiments would provide clearer intuition about the behavior and effectiveness of the proposed regularized Newton’s method, complementing the results on more complicated deep neural network models.\n\nLIMITATIONS:\nThe authors has presented some limitations in last discussion section of this submission. Please check the other weakness section for all limitations of this submission.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-11"}