Title: Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
Venue: NeurIPS 2025
Accepted: yes
Keywords: gradient descent

optimization

momentum

natural gradient

Adam

learning rules

theory

optimal control

normative framework

variational principles

continual learning
Abstract: Learning rules—prescriptions for updating model parameters to improve performance—are typically assumed rather than derived. Why do some learning rules work better than others, and under what assumptions can a given rule be considered optimal? We propose a theoretical framework that casts learning rules as policies for navigating (partially observable) loss landscapes, and identifies optimal rules as solutions to an associated optimal control problem. A range of well-known rules emerge naturally within this framework under different assumptions: gradient descent from short-horizon optimization, momentum from longer-horizon planning, natural gradients from accounting for parameter space geometry, non-gradient rules from partial controllability, and adaptive optimizers like Adam from online Bayesian inference of loss landscape shape. We further show that continual learning strategies like weight resetting can be understood as optimal responses to task uncertainty. By unifying these phenomena under a single objective, our framework clarifies the computational structure of learning and offers a principled foundation for designing adaptive algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDifferent learning rules, such as variants of gradient descent, are often compared empirically. Theoretical analysis generally looks at convergence rates or guarantees. This paper takes a slightly different perspective and tries to understand what problem each rule is implicitly solving. By framing parameter updates as a multi-step, continuous-time optimal control problem, the authors find they can derive many popular learning rules from a common framework. The objective for deriving the parameter updates is composed of two terms: a parameter change cost and the loss function. Different choices of the parameter change cost and approximations of the loss function give us different learning rules. Common tricks, like momentum, naturally fall out of the framework due to the multi-step optimization. Using a second-order approximation of the loss can give Netwon's method, gradient descent, or an in-between \"ballistic\" learning rule depending on the choice of time step and temporal discount factor. Incorporating different metrics in the parameter change cost can respect parameter geometry. For instance, using the Fisher information matrix gives us natural gradient descent. An important take-away from this analysis is that natural gradient descent is not actually second-order optimization. Rather, the metric governs the parameter velocity penalty and the Hessian measures loss landscape curvature. Making assumptions about partial controllability of parameters, such as existing external \"forces\" on parameters that may act in a state-dependent way, can produce non-gradient rules, such as rotational dynamics, and account for weight decay. Next, assuming the true loss is only partially observable and must be estimated, they show that adaptive optimizers, like Adam and RMSprop, can be derived. Finally, they show that some common continual learning strategies can fall out of modeling weight uncertainty and making the learning dynamics sensitive to this uncertainty.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Building a better theoretical understanding of what implicit assumptions different learning rules make and when to use them is an important problem. This paper provides a new perspective on this problem, deriving many different learning rules under a common framework.\n- New learning rules can possibly be derived from this framework, making different assumptions on parameter geometry, partial controllability, and partial observability. In fact, the ballistic learning rules appear to be novel, although it is not evaluated empirically.\n- The insight on the relationship between natural gradient descent and second order methods appears to be novel and makes a valuable distinction between the two.\n- The view of adaptive learning rules as modeling partial observability also appears novel and provides a nice explanation for what these algorithms are doing under the hood.\n- The discussion of continual learning algorithms, which may inject noise or reset unused weights, as being tied to a weight-uncertainty-aware learning rule is a nice justification of choices that are often empirically motivated.\n\nWeaknesses:\n- While the paper is primarily focused on theory, and is deriving known algorithms from a new perspective, there are some new variants of common learning rules put forth. It would be interesting and valuable to compare these different choices on some simple examples to better understand what trade-offs they make.\n- The framing of momentum as falling out of a continuous-time formulation appears in Wibisono et al. [49], which is briefly mentioned in the related work section. While this work does not consider partial observability, partial controllability, weight uncertainty, or second-order methods, it does account for momentum falling out of a multi-step perspective on optimization. Moreover, it uses the Bregman divergence as a distance metric. Natural gradient descent can be viewed as gradient descent with a specific choice of Bregman divergence [1, 2]. Therefore, while I do think the contributions of this paper are substantial and novel, the paper could do a better job relating its innovations to this prior work and making the distinctions clear.\n\nReferences:\n[1] G. Raskutti and S. Mukherjee, \"The information geometry of mirror descent,\" IEEE Transactions on Information Theory, 2015.\n[2] N. Wagener, C. Cheng, J. Sacks, and B. Boots, \"An online learning approach to model predictive control,\" RSS, 2019.\n\nQUESTIONS:\n- Are there more subtle differences between Wibisono et al. [49] and the formulation considered in this paper, when not considering partial observability or controllability?\n- What are some situations in which one would want to use the ballistic version of the learning rule? What are some trade-offs of these different choices besides Hessian computation?\n\nLIMITATIONS:\n- The authors are clear that their analysis does not consider comptuational requirements and efficiency in its definition of optimal.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a general framework to model learning dynamics. The focus is not a parameter that minimize a loss, but rather a trajectory (on parameter space) that minimize a novel objective: an integral over the trajectory of a combination of loss and parameter change. The optimal solution is given by the Euler Lagrange equations providing two boundary conditions: an initial parameter, and the integral to be minimized.\n\nDifferent hyperparameters characterize the weight of these two components, as well as their specific form. Importantly, for some specific choices of hyperparameters, the optimal solution can be expressed analitically, and the discrete time difference along such optimal trajectory corresponds to learning rules prescribed by established optimizers. \nThis motivates the relevance of this novel framework: it can recover and generalize multiple optimizers.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written an easy to read. The various elements are properly discussed and the Appendix provides several useful examples and a clear derivation of the results.\n\nThe proposed framework is, to my knowledge, novel. The derivation are correct and multiple learning rules are derived from the same framework. For this reason I consider this work very relevant and useful for the community.\n\nWhat I personally find extremely interesting is the analysis in Section 4, where the loss is assumed to be quadratic and the parameter geometry is given by the Fisher. This lead to a proper distinction of Hessian and Generalized Gauss Newton which, as the authors also notice, are too often simply viewed as surrogates of each other. The authors framework provides a rigorous derivation of their role.\n\nI found the assumption in Section 5 on gradient noise to be a bit arbitrary. It would be very helpful if the authors can show some references motivating (at least partially) this assumption. Aren't there other works that motivates Adam through this kind of assumption? \n\nI like the presence of the $f$ term in the general form, but I personally don't see how the rotational term can be of any use in practice. Is there a practical use case that can be shown?\n\nOverall, I really like the proposed framework and the approach of finding solutions through Euler Lagrange. My only concern (which is actually my only main concern about the paper) is on the boundary conditions. First of all, this part is a bit swiped under the carpet in the Appendix, and I don't really like that. The authors should be more clear about this. \nThen, more specifically, no problem about the initial parameter condition, that's very reasonable. My concern is about \"The remaining degree of freedom\" that \"is chosen so that J is minimized\". This feels circular. Isn't the whole point of Euler Lagrange solution to minimize J? My understanding is that Euler Lagrange provides LOCALLY optimal solutions, i.e. optimal among small perturbations. How is this compatible with your choice of boundary condition? Please elaborate on this.\n\nSome minor details:\n\n-In Line 42 I'd recommend changing \"sensitive to structure in parameter space\" to \"sensitive to non-local structure in parameter space\" to better clarify what is actually being generalized of the natural gradient descent methods\n\n-In Line 855 there is a typo: \"J to minimized\" should be changed in \"J to be minimized\"\n\n-In Equations 123 and 124, isn't that $J$ supposed to be a $\\mathbf{J}$? Is it a typo or am I missing something\n\nQUESTIONS:\nCan the authors elaborate on the second boundary condition of the Euler Lagrange solutions?\n\nIn Line 1099 you state that Equation (133) \"matches the main text form up to the prefactor $1/(2v_i^2$\". Referring to the equation before line 231 in the main text. Can the authors expand on this mismatch? Why is there such difference? And how is the equation in the main text derived?\n\nCan the authors provide an example where rotational forces exists?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework to derive generally learning rules retrieving known ones in the process. Namely, the authors propose to define learning rules by finding continuous-time parameter curves minimizing ${}^{(1)}$ some average future discounted costs comprising the loss but also some parameters-specific costs enforcing e.g. smoothness of parameter changes or beliefs in the process changes. \nThe authors illustrate their framework to retrieve (i) momentum SGD, (ii) natural gradient descent, (iii) adaptive algorithms à la Adam. They also apply their framework to get insights on continual learning approaches. A long appendix provides detailed computations and explanations for each setting\n\n\n${}^{(1)}$ the authors solve the Euler-Lagrange equations associated to their objective, but the Euler-Lagrange equations only give stationary points for the objective in terms of continuous time curves.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The idea of considering \"long term\" optimization horizons could be interesting. The authors should integrate algorithms like Lookahead [1] in their framework since it bears similar motivations. \n- The derivations in the appendix are clear. \n- The paper is well written and pleasant to read (see small remark about tone below though).\n- Some new algorithms are derived, though not tested. \n- While the momentum and natural gradient viewpoints are very well known from a usual variational viewpoint, the viewpoint for Adam appears new.\n\n**Weaknesses**\n\n*Main*\n- The field of optimization may be older than the authors themselves. There already exists frameworks in optimization to design algorithms in a principled way. An essential example is the one developed by e..g A. Nemirovisky and D. Rudin [2] that influenced optimization for years. Their framework defines what could the best possible algorithm for a given class of problems, read for example [3] for a gentler introduction. Such a framework had the benefits to understand how different algorithms compare given some preliminary information on it. It also bore fruits: the accelerated algorithm of Nesterov was motivated by finding the best algorithm for smooth and convex problems. In comparison, the framework proposed by the authors does not help us understand how these algorithms compare for a given class of problems. The framework provides ad-hoc explanations to well-known algorithms.\n- \"it it makes it easier to generate new learning rules from a principled starting point, and to justify them without empirical guesswork\" Actually an algorithm is made to be applied. Experiments are the gold standard, the truth to understand the relevance of the theory. The ultimate test for the proposed framework is to get a new algorithm that perform better in practice (like sharpness aware minimization was designed after studying sharpening effects). The authors actually have some new algorithms like the \"ballistic one\", why aren't they tested empirically? At least each intuition that the framework provides could be tested empirically to see how relevant the framework is. Why aren't each intuition tested empirically to analyze their relevance? Is it left for other people to do this work?\n- Numerous approximations are sometimes needed to find the desired algorithm in the given framework (see Adam for example).\n- All these algorithms had previous justifications: momentum is well known to be motivated from a physics viewpoint (it's called Polyak's heavy ball), Adam is justified as a form of Adagrad, Adagrad itself is justified a s a form of a normalized gradient, there are many studies from those viewpoints that gave new intuitions like understanding how normalized gradient can adapt to the edge of stability phenomenon. The authors need to justify what brings their approach. \"One rule to rule them all\" is not a good motivation (especially given that their framework won't encompass all possible algorithms).\n\nThe paper presents many algorithms through the given framework. It could be much better to focus on one algorithm each time and provide numerical experiments illustrating either new algorithms that the framework gave rise to (like the ballistic one) or to illustrate empirically some intuitions that the framework gave. Even the \"Newton's method\" that the framework gives in section 3 is not the actual Newton method. Maybe it gives a much better algorithm, maybe not at all. Probably this framework completely misses the need to adjust for the learning rate once a learning rate rule is found. A Newton method needs a priori to be damped for example, is it the case for what the authors propose? There could be interesting fruits from the framework but currently trying to frame each algorithm through one lens does not add much relevance to the approach. Experiments will prove the relevance.\n\n*Minor*\n- \"Natural gradient descent is not second-order optimization in disguise\" The authors are presenting generic preconditioning methods. Indeed they are not presenting natural gradient descent which is one particular instance. Updates using pre-conditioners are well motivated from a variational viewpoint. Actually one can go beyond using simple Mahalanobis distances and consider Bregman divergences, or any gradient mapping coming from a partial linearization of the objective. \n\n*Additional notes*:\n- A humble tone may be best to invite the reader to understand the relevance of the idea rather than a forceful tone affirming for example that \"[this] framework is significant\". A paper is written to present an idea not to argue or convince about its significance, let the reader judge by themselves. \n- There is a missing identity matrix in the \"effective learning rate\" of the Newton step and it's not an \"effective learning rate\", learning rates are scalars, a Newton method does more than multiplying with a scalar.\n\n[1] Zhang, Michael, et al. \"Lookahead optimizer: k steps forward, 1 step back.\" Advances in neural information processing systems 32 (2019).\n[2] Nemirovskij, Arkadij Semenovič, and David Borisovich Yudin. \"Problem complexity and method efficiency in optimization.\" (1983).\n[3] Nesterov, Yurii. Introductory lectures on convex optimization: A basic course. Vol. 87. Springer Science & Business Media, 2013.\n\nQUESTIONS:\n- How can the framework explain why Adam could work better than SGD with momentum? More generally the choice of objective to solve for with Euler-Lagrange equations appear as arbitrary as choosing an algorithm, and does not provide much insights on the choice of the algorithm.\n- Could the authors use their framework to\n  - derive Nesterov's accelerated method?\n  - derive learning-rate tuners such as line-searches?\n  - derive algorithms using exponential moving average?\n  - derive sign-sgd (a variant of sgd that was linked to adam)\n- A current difficulty of usual frameworks is to be able to take into account phenomena like edge of stability [1]. Does the proposed framework help to integrate such effects?\n- Could the authors test some new algorithms they propose?\n\n\n[1] Cohen, Jeremy M., et al. \"Gradient descent on neural networks typically occurs at the edge of stability.\" arXiv preprint arXiv:2103.00065 (2021).\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
In this work, the authors develop a framework for describing existing learning algorithms. Across a variety of algorithms, they work towards identifying rules for which each particular algorithm is the optimal algorithm.

STRENGTHS AND WEAKNESSES:
Strengths:

* the authors study the important problem of characterizing learning algorithms
* normative rules for learning algorithms is an interesting way to characterize learning algorithms 

Weaknesses:

* no experiments for identifying whether or not the identified rules are actually realistic in practice (as a basic example: do continuous time approximations actually match what happens in practice? one would think a priori that infinitely many extraordinarily small learning rates would actually significantly change navigation of the loss landscape for DL optimization tasks). 
* the paper is rather confusing, treating prior work as well known and obvious (e.g., the amari insight on line 141) and considered theory as obvious (e.g., line 118 or line 80). ideally, the authors would work to guide the reader better through their work.

QUESTIONS:
how do the results relate to practice? are there experiments that would made this paper more relevant to practice?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-12