Title: Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: nonconvex optimization

generalized smoothness

first-order methods
Abstract: We study nonlinearly preconditioned gradient methods for smooth nonconvex optimization problems, focusing on sigmoid preconditioners that inherently perform a form of gradient clipping akin to the widely used gradient clipping technique. Building upon this idea, we introduce a novel heavy ball-type algorithm and provide convergence guarantees under a generalized smoothness condition that is less restrictive than traditional Lipschitz smoothness, thus covering a broader class of functions. Additionally, we develop a stochastic variant of the base method and study its convergence properties under different noise assumptions. We compare the proposed algorithms with baseline methods on diverse tasks from machine learning including neural network training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers preconditioned gradient methods for nonconvex optimization problems under the generalized smoothness condition. It first introduces preconditioned gradient descent with momentum when the full gradient is available. In addition, the authors propose a stochastic variant of the preconditioned gradient descent method and study its convergence guarantee.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Extension of the preconditioned gradient descent to the stochastic setting is an interesting topic to explore.\n* The authors provide a rigorous theoretical analysis of the proposed methods. \n* This paper presents empirical evidence to demonstrate the effectiveness of the proposed methods.\n\nWeakness:\n\n* The authors do not provide a comparison of the convergence rates of the proposed methods with existing approaches under the generalized smoothness setting. The theoretical advantages of the proposed methods over existing approaches are not clearly articulated.\n\n* In Theorem 3.1, the R.H.S. of (15) has a non-vanishing term of $\\sigma^2$, which implies that the proposed stochastic preconditioned gradient descent cannot have exact convergence to the stationary point.\n\n* In Theorem 3.4, the choice of batch size is too large. To obtain $\\epsilon$-stationary point, one needs to choose a batch size $K = 1/ \\epsilon$, which is impractical for large-scale machine learning applications.\n\nQUESTIONS:\n1. With an appropriate choice of the reference function $\\phi$, anisotropic smoothness is equivalent to $(L_0, L_1)$-smoothness. The $(L_0, L_1)$-smoothness framework already encompasses a wide range of machine learning applications. Therefore, a key concern is the significance of further generalizing from $(L_0, L_1)$-smoothness to anisotropic smoothness in the context of machine learning. Specifically, I would like to know which machine learning applications fall within the class of anisotropically smooth functions but not within the class of $(L_0, L_1)$-smooth functions.\n\nLIMITATIONS:\nThe theoretical guarantees for the stochastic variants of preconditioned gradient descent appear to be limited, as highlighted in Weaknesses (2) and (3).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates smooth noncovnex optimization problem using anisotropic gradient descent (GD) method & its extensions. They combine the notion of momentum with nonlinear preconditioned GD & propose a Heavy-Ball (HB)-type method with a convergence rate of $\\mathcal{O}(L/K)$ for a strongly convex reference function $\\phi$ & $f$ being anisotropically smooth relative to $\\phi$ (a notion that was proposed by previous works). Later, they extend their convergence result to the scenario when $f$ satisfies a generalized PL inequality and $\\phi$ is 2-subhomogenous. Additionally, they study their method for functions that satisfy a preconditioned-smoothness assumption which contains $(L_0-L_1)$-smoothness as a special case. A convergence analysis of the stochastic preconditioned GD method under a more general noise assumptions revealed that the convergence is not guaranteed without variance reduction techniques (like increasing batch sizes). Numerical experiments revealed competetive behaviour of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1- The problem is well-motivated. Although, the gradient clipping method is more explored and studied in the literature, its general form (1) is less attended. This is the general motivation in this work. (Lines 42-54)\n\n2- The contribution is good. To further study the general form of nonlinear preconditioned GD, they propose an HB-type method with its convergence analysis under the assumptions of anisotropic smooth relative to $\\phi$ (lines 155-156 theorem 2.1), a generalized PL inequality where $\\phi$ is 2-subhomogenous (lines 197-201 theorem 2.3), and a preconditioned Lipschitz continuity (lines 225-228 theorem 2.6) . A stochastic extension is also studied (for the first time as the authors claim) under a slightly more general noise assumption (lines 241-243 theorem 3.1). Additionally, they show that if the mini-batch size increase (classical approach to reduce the variance estimation error), the true convergence is possible in theorem 3.4.\n\n\n**Weaknesses**\n\n1- The writing could improve, sometimes the reader might feel lost or diconnected. Adding a bit more coherency might help with this (this point exists throughout the whole paper. Examples are Paragraph on line 22 is incoherent with the previous paragraph, paragraph on line 203 hard to read sentence on line 207).\n\n2- ADAM is a momentum-based algorithm, however it was not compared in the Numerical simulations (Figure 1, right).\n\nQUESTIONS:\nFirst, thank you for your efforts.\n\n1- In line 117 it is mentioned that this work mainly focuses on strongly convex reference functions. Then they propose some examples. What are other examples of strongly convex reference functions?\n\n2- Did you ensure that all the methods in your simulations are compaired fairly in terms of the choice of the step-sizes? I'm mainly referring to the neural network simulations where the step-size was chosen through sweeping on $\\{1,0.1,0.01\\}$.\n\n3- Why ADAM was not compared in the training on CIFAR10 ResNet18 experiment with momentum?\n\n4- Do you think it would be possible to extend these results to the constrained optimization?\n\nThanks!\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretically rigorous and practically motivated study of nonlinearly preconditioned gradient methods (NPGMs), specifically extending them with momentum (m-NPGM) and developing a stochastic variant. The paper analyzes convergence under anisotropic smoothness, a generalization of Lipschitz smoothness, and establishes novel results under generalized PL conditions and (L0, L1)-smoothness. It also provides empirical comparisons on deep learning and matrix factorization tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: 1) Theory: This work introduces a novel heavy-ball extension of nonlinear preconditioning methods with provable convergence guarantees in the general nonconvex setting under a generalized PL condition. This is the first result regarding the convergence of the method with momentum under anisotropic smoothness conditions.\n2) Assumption and setting: the analysis is established under anisotropic smoothness and preconditioned Lipschitz conditions, which cover many applications. In particular, the analysis is a non-trivial extension of the previous work due to the particular structure of the anisotropic descent inequality and the lack of global upper bounds for the gradient difference term. Therefore, the techniques developed here have the potential to be applied to develop and analyze other algorithms in this setting.  \n\nWeakness: The presentation of the content lacks details. All the main results are presented right after the assumptions/conditions, without any elaboration on the proof sketch and technical novelty. The paper discusses a variety of settings, but could focus on one or two of them to further elaborate the technical details. The experimental section could be strengthened with ablation on choice of reference function and effect of momentum coefficient on convergence, etc.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a nonlinearly preconditioned heavy-ball-type algorithm and provides its convergence guarantees in the general nonconvex setting and under a generalized PL condition. It also presents a stochastic variant and analyzes its convergence behavior under both new and standard noise assumptions. Finally, it tests the proposed methods and shows their good performance on various tasks, including neural network training and matrix factorization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This paper extends the nonlinearly preconditioned gradient descent method by incorporating a momentum term and studies the convergence behavior under different conditions.\n\n2. This paper also proposes a stochastic variant of the base method and analyzes it under different noise assumptions.\n\n3. The numerical experiments consider two different applications and show the good performance of the proposed methods.\n\n**Weaknesses**\n1. Some concepts are used without any explanations, such as $\\Phi$-convexity. It may be better to give a brief introduction in the appendix.\n\n2. It is better to compare the standard convergence analysis of gradient clipping under the $(L_0, L_1)$-smooth condition and the one under this anisotropic smoothness. For example, does the paper obtain a stronger result in the stochastic setting? Does the analysis become simpler? Are the assumptions weaker? I think the current manuscript lacks a discussion on this aspect.\n\nQUESTIONS:\n1. It is better to explain sigmoid preconditioners in the main text.\n2. Is there any relationship between the so-called Hyperbolic Gradient Descent with the gradient descent in hyperbolic space [1]?\n3. Is it better to use $\\cosh(\\cdot)-1$ instead of $\\cosh-1$?\n4. To obtain the next iterate, Equation (2) is minimized over what?\n5. Define $\\delta_{\\{0\\}}(x)$ in Line 103\n6. Define $\\text{rge}$ in Equation (3)\n7. It is better to give the formulation of the stochastic optimization problem at the beginning of Section 3.\n8. What about the numerical performances of algorithms induced by other reference functions?\n9. Is it possible to include numerical experiments on PL-type functions?\n---\n[1] Wilson, B., & Leimeister, M. (2018). Gradient descent in hyperbolic space. arXiv preprint arXiv:1805.08207.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes nonlinearly pre-conditioned gradient methods (NPGMs) that combine the dual-space/anisotropic-descent framework with heavy-ball momentum and with a stochastic variant. It establishes global convergence bounds for the momentum version under anisotropic smoothness and shows a linear rate when a generalized PL condition holds. For the stochastic setting, the authors derive non-asymptotic rates under a novel noise assumption that subsumes bounded variance and prove linear convergence under generalized PL. Finally, they include experiments on MNIST/CIFAR-10 classification and matrix-factorization tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. The paper is well structured and has a nice flow.
2. They provide rigorous proofs for all stated theorems, and the assumptions are clearly presented. In particular, they both deterministic and stochastic analyses, including mini-batch dependence.
3. The authors give the first convergence proof for heavy-ball momentum inside anisotropic preconditioning setting.

**Weaknesses**
1.  In the statement of Theorem 2.1 the authors make the restriction $\beta<0.5$, limiting practical momentum choices (usually $\beta=0.9$), however, they note this as an open question. 
2. The stochastic results (Section 3) cover only the base (no-momentum) method. Following the work from Section 2, one would expect results for the momentum-based method. After all, this is what practitioners usually deploy. 
3. Empirical accuracy gains over tuned SGD/Adam are modest on CIFAR-10; error bars overlap after 200 epochs.

QUESTIONS:
See weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02