Title: Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: continuous approximation

momentum

HB
Abstract: This paper establishes a continuous time approximation, a piece-wise continuous differential equation, for the discrete Heavy-Ball (HB) momentum method with explicit discretization error. Investigating continuous differential equations has been a promising approach for studying the discrete optimization methods.  Despite the crucial role of momentum in gradient-based optimization methods, the gap between the original dynamics and the continuous time approximations due to the discretization error has not been comprehensively bridged yet. In this work, we study the HB momentum method in continuous time while putting more focus on the discretization error to provide additional theoretical tools to this area. In particular, we design a first-order piece-wise continuous differential equation, where we add a number of counter terms to account for the discretization error explicitly. As a result, we provide a continuous time model for the HB momentum method that allows the control of discretization error to arbitrary order of the learning rate.  As an application, we leverage it to find a new implicit regularization of the directional smoothness and investigate the implicit bias of HB for diagonal linear networks, indicating how our results can be used in deep learning. Our theoretical findings are further supported by numerical experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper highlights the differences between the discrete and continuous time heavy-ball (HB) momentum methods.\nThe authors explore the continuous-time HB method and report notable findings that distinguish their work from previous studies.\n\n- Research Question\n    \n    - The discretization error between gradient flow (GF) and gradient descent (GD) has been studied in [Barrett and Dherin, 2022; Miyagawa, 2023; Rosca et al., 2023]. Rather than GD, the authors focus on the HB method, which has yet to be explored.\n        \n    - More specifically, the continuous time HB as a surrogate model of the discrete HB has been formalized as a rescaled gradient flow (RGF: Eq. (1)) [Kovachki and Stuart (2020)], but the authors point out that it cannot differentiate gradient descent (GD) and HB (Sections 3 and 4). Another continuous time HB has been proposed by [Ghosh et al. (2023)], which reduces the discretization error compared to RGF, but it is limited to the second order of the step size. [Cattaneo et al. (2023)] has analyzed Momentum and Adam but is also limited to the second order.\n        \n- Solution\n    \n    - To address this issue, the authors propose HB Flow (HBF), a first-order piece-wise continuous differential equation built upon RGF, that approximates the discrete HB momentum method and can control discretization error to *arbitrary order* of the step size (learning rate).\n        \n    - The idea is to add a number of counter terms to RGF order-by-order (Theorem 2.1), akin to backward error analysis [Hairer et a., 2006; Miyagawa, 2023; Rosca et al., 2023].\n        \n- Applications\n    \n    - Examining the directional smoothness and the diagonal linear network, the authors demonstrate that HBF captures the differences between HB and GD, which has been impossible by RGF.\n        \n        - The authors show that the learning dynamics of HB exhibits smaller directional smoothness compared to GD (Fig. 1), through the lens of HBF.\n            \n        - In addition, they find the implicit bias of HB for the diagonal linear network (Theorem 3.1) that cannot be obtained by RGF.\n            \n    \n- Overall, the paper argues that, despite its popularity, continuous-time analysis fails to fully capture the learning dynamics of discrete-time GD methods when neglecting discretization error. Although this limitation has been noted in prior work, such as [Miyagawa, 2023], it is worth emphasizing that the issue is general and extends to various optimization methods such as the HB momentum method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nPlease see also the Summary field:\n> - (Novelty) The authors explore the continuous-time heavy-ball (HB) method and report notable findings that distinguish their work from previous studies.\n> - (Novelty & Technical Contribution) The authors show that the learning dynamics of HB exhibits smaller directional smoothness compared to GD (Fig. 1), through the lens of HBF.\n> - (Novelty & Technical Contribution) They find the implicit bias of HB for the diagonal linear network (Theorem 3.1) that cannot be obtained by RGF.\n> - (Broader Impact) The paper argues that, despite its popularity, continuous-time analysis fails to fully capture the learning dynamics of discrete-time GD methods when neglecting discretization error. Although this limitation has been noted in prior work, such as [Miyagawa, 2023], it is worth emphasizing that the issue is general and extends to various optimization methods such as the HB momentum method.\n\n- The proposed HBF might be seen as a natural progression of [Kovachki and Stuart (2020); Miyagawa, 2023; Rosca et al., 2023]. In fact, the the authors’ approach and contents in Section 2 in the paper are similar to  Sections 3 & 4 in [Miyagawa, 2023]. However, I do not think that this is a weakness because:\n    \n    - (Clarity, Novelty, & Technical Contribution) The contribution is clear$\\textemdash$as summarized in the Summary field above$\\textemdash$and significant for the field of optimization, as it helps bridge the gap between conventional continuous-time theory and practical discrete-time methods.\n        \n    - (Technical Contribution) In addition, the derivation of HBF is nontrivial$\\textemdash$performing all-order backward error analysis involves careful calculations, including the use of Lie derivatives$\\textemdash$demonstrating the paper's technical soundness.\n        \n    - (Novelty & Technical Contribution) All analyses in Section 3 and 4 are built upon the proposed HBF and thus are novel, indicating HBF can distinguish GD and HB unlike RGF.\n        \n- (Soundness) The newly-observed implicit regularization of HBF discussed at the end of Section 2 matches our intuition: HB prefers smaller directional smoothness to GD because of the momentum (inertia).\n    \n- (Empirical Validation) (If my concern below is addressed) The experiment in Figure 2 is convincing: larger $\\alpha$ leads to smaller discretization error. Although the model is small, this choice is acceptable given that computing higher-order derivatives becomes prohibitively expensive for large models.\n\n\n### Weaknesses\n\n- (See Questions below) It would be helpful to clarify under what conditions the proposed HBF fails to serve as a valid and accurate approximation of the discrete time HB method. \n\n- (See Questions below) I am concerned about $\\eta_\\mathrm{Euler}$.\n\n- Additionally, I recommend that the authors provide their code for reproduction.\n\n\n## Minor Comments\n\n- [Comment (minor)] I recommend that the authors explicitly refer to $\\gamma_k$ as the *counter term* in line 134. Currently, the term “counter term” appears only in the Abstract, Introduction, and Appendix A.1, which may lead to confusion for readers.\n    \n- [Comment (minor)] The colors in Figure 2(a) are hard to distinguish (three blues).\n\n- [Typo] The numerator of $\\frac{ }{q!}$ in Eq. (50) is missing.\n    \n- [Typo] (Lines 188, 193, 219, and 223) Theorem 3.1 should be Theorem 2.1?\n\n- [Typo] (Line 831) Please Appendix C. -> Please see Appendix C.\n    \n- [Grammatical error] (Line 184) … it is worth to mention… -> … it is worth mentioning…\n    \n- [Comment (minor)] (Lines 192--194)\n    \n    > … the recursive manner in Theorem 3.1 provides a possibility to calculate the involved terms automatically by using software for symbolic mathematics such as SymPy, ….\n    \n    Totally agree $\\textemdash$ combining SymPy with AI coding tools makes tedious analytical math way easier.\n\n\n## Review Summary\nThe paper presents strong technical contributions, offers novel insights, and is clearly written. While it raises a few concerns noted above, I believe the strengths outweigh them. I recommend acceptance and look forward to the discussion with the authors.\n\nQUESTIONS:\n- When does the approximation accuracy of HBF for the discrete HB method deteriorate?\n    \n    - I suspect it occurs in regions where the derivatives of $L$ are large and where $1 - \\mu \\ll \\eta$, causing the Taylor series to diverge.\n        \n    - Does this happen in practical settings?\n        \n    - Could you provide rough bounds on the derivatives required to keep the discretization error small?\n        \n    - It would also be helpful to provide a rough estimate of the $\\alpha$\\-th order discretization error of a $d$\\-dimensional DNN at the $k$\\-th step from the results in Section 2.2, for example.\n        \n- (Eq. (17) & (18)) This question relates to the one above. When $\\mu \\sim 1$, it appears that the dynamics are dominated by higher-order derivatives or even a constant term, rather than the first-order gradient. I am concerned that this does not reflect the correct learning dynamics. Why is that the case?\n    \n- How can we extend the authors’ approach to adaptive learning rate, e.g., Adam? What is the challenge specifically? The authors mentioned the extension to Adam in the Limitation section, and I agree this is an interesting and important step toward practically relevant scenarios in view of the prevalence of Adam.\n    \n    - To my knowledge, existing continuous-time formulations of Adam are quite complex. How might one begin to approach this problem?\n        \n- (Theorem 2.1) Can the result be extended from $\\alpha \\geq 1$ to $\\alpha > 0$?\n    \n- To my understanding, a difference between the paper and [Miyagawa, 2023] is, aside from HB and GF, that the former proposes a piecewise continuous differential equation, whereas the latter provides a globally continuous one. Is it correct?\n    \n- (Appendix C.2) Was $\\eta_\\mathrm{Euler} = \\eta/10$ sufficiently small? The authors can check this, for example, by changing $\\eta/10, \\eta/100,... $ and confirming the learning trajectory does not change dramatically or the change is much smaller than the discretization error, the quantity is of interest.\n    \n- How long did it take to run the experiments?\n\nLIMITATIONS:\nYes, but additional discussions can be added: under what conditions the proposed HBF fails to serve as a valid and accurate approximation of the discrete-time HB method?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new continuous ODE model that can reduce the discretization error from the discrete HB method.\nBy ultilizing the Taylor expansion and discrete matching on the learning rate, the discretization error can be arbittrary order of the learning rate. This result provides a reliable foundation for analyzing the momentum methods in the continuous time limit.\n\nSTRENGTHS AND WEAKNESSES:\nThe theoretical results are very attractive, and can be considered as a good extension of previous works. \nWhile (Kovachki and Stuart, 2020) and (Ghosh et al., 2023) explicitly anaylzed the discretization error to $\\mathcal{O}(\\eta)$ and $\\mathcal{O}(\\eta^2)$, this paper extended the result to $\\mathcal{O}(\\eta^{\\alpha})$ with arbitrary positive integer $\\alpha>0$.\nThis continuous model is applied to analyze the regularization term for the directional smoothness and the implicit bias of diagonal linear networks.\nThis bridge the gap between the discrete HB and the its precise characterization in continuous time.\nAlthough I did not check all the proofs in detail, I believe the overview of the proof is correct.\nThe following parts could be improved a bit more:\n\n1. The numerical experiments are far from enough to reflect the correctness of the main Theorem 2.1. This paper only conduct results on 2d-simple model and 2-layer diagonal linear networks for simple regression problems. More experiments on deep learning taskes should be explored to examing the discretization errors. What I expect to see is the discretization errors vs iterations on different $\\alpha$, like the Figure 2(b) in your paper (but in different tasks), or Figure3-4 in Miyagawa (2023).\n\n2. The mathematical results Theorem 2.1 are got by ultilizing the Taylor expansion and discrete matching on the learning rate used in (Miyagawa ,2023), (Rosca et al., 2023) for GD, and extended the techniques to HB, which covers the theoretical results of previous works  (Kovachki and Stuart, 2020) and (Ghosh et al., 2023). From this point of view, the theoretical contributions are limited.\n\n3. I have some difficulties when I first read the paper and before searching the cited works. I think the reason is that the \"overview of our approch\" in page 3 full fill of mathematical symbols without explicit explaning, and the following proofs are based  on the overview. The motivation and main technniques in this overview should be well-organized.\n\n4. Some typos: lacking $\\eta$ for the discrete GD in Table 1, lacking $\\frac{\\nabla L \\nabla^2 L}{2}$ in equation (20).\n\nQUESTIONS:\n1.see the weakness.\n\n2.How can the theoretical insights from Theorem 2.1 be used to improve practical optimizer design?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a continuous time framework--HB Flow (HBF)--for the heavy-ball momentum method (HB). The authors show HBF approximate HB up to arbitrary order. They also study the implicit bias of HBF on diagonal linear networks which brings insights into the implicit bias problem in momentum methods. Experiments on simple neural networks are done to verify their results.\n\nSTRENGTHS AND WEAKNESSES:\n*I have reviewed this paper last year. I don't see many significant changes. So I am borrowing some of my previous review here, which the authors had not yet addressed.*\n\nStrengths.\n\n1. The authors propose HB flow--a continuous-time approximation of the heavy ball momentum method. By adding back the discretization error in each time interval, the HB flow achieves an arbitrarily close distance to the discrete-time dynamic. This is justified rigorously in theorem 2.1\n\n2. More precisely, the distance between two dynamics is measured using $L^\\infty[0, T]$ on the trajectory space. The $L^\\infty$ is qualified by an upper bound in the learning rate $\\eta^\\alpha$. The authors give the specific form for the case $\\alpha = 1, 2, 3$ as examples, which help the readers to understand the construction of the continuous dynamic.\n\n3. The authors computed the implicit bias of HBF for diagonal linear networks in theorem 3.1 which makes this framework useful in studying the generalization properties of HB. This type of behavior cannot be captured by lower order models like RGF.\n\n4. In numerical experiments, Figure 2 illustrate the trajectories in a clear way and verifies main results from the paper.\n\n5. I see the authors added one experiments about the implicit regularization. \n\nWeaknesses.\n\n1. I found the main theorem hard to follow. Specifically, Theorem 2.1 is stated with many notations without enough intuition/explanations. I am trying to understand this at a high level. So basically find the difference between continuous and discrete dynamics and use Taylor expansion for it. The order $\\alpha$ in $\\eta^\\alpha$ just means how many times we do Taylor expansion in $t$ because the time interval is of length $\\eta$. Although the computation needed to find the exact expressions is not trivial, one could argue this is not mathematically sophisticated.\n\n2. Some notations are not consistently defined. For example, on page 4, the authors stated $G_k \\in R^d$, i.e. a d dimensional vector, but $G_k$ is a function. Similar situation for $\\gamma_k$.\n\nQUESTIONS:\n1. In Figure 2(a), I noticed that the $\\alpha=2$ case gives a better approximation away from the convergence point while $\\alpha=3$ is the opposite. I wonder if the authors have some intuition for this behavior.\n\n2. This analysis can be generalized to the stochastic case by for example replacing $\\nabla L$\n by $\\tilde{\\nabla} L$\n where $\\tilde{\\nabla}$\n denotes the approximate gradient. The approximation can be done by either introducing a diffusion term [1] or an index switching process [2]. I think this might be an interesting future direction.\n\n[1] Stochastic modified equations and dynamics of stochastic gradient algorithms I: Mathematical foundations, Li et al.\n\n[2] Analysis of Stochastic Gradient Descent in Continuous Time, Latz.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a continuous ode approximation of the popular Heavy-ball momentum optimization algorithm. Compared to the previous studies, the framework proposed in this paper can achieve an error as small as arbitrary order of the small interpolation, i.e. the learning rate $\\eta$. Based on this conclusion, authors propose a new implicit bias of diagonal linear networks. While simple and idealized, this model usually serves as the first step for multiple theoretical studies.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n1. Compared to the previous studies, the framework proposed in this work can better capture the optimization trajectories of the Heavy-ball momentum, as it can achieve an arbitrarily small error, which, also claimed as the major technical contribution of this work.\n\n2. The study of the implicit bias of momentum implies the potential benefits of studying the the higher-oder approximations.\n\n**Weaknesses:**\n\n1. While I found the approximation w.r.t. an arbitrary order of the interpolation size $\\eta$ is relatively interesting, the intuition behind such an approach is hard to follow, especially after reading the proof details in the Appendix. First of all, from my understanding, the presentation of equation (6) is quite different from the standard form of second-order Taylor expansions. Indeed, after checking the definition of $\\eta^2 I_k^+$ in line 140, it is exactly $\\beta(t_{k+1}) - \\beta(t_{k}) - \\eta\\dot{\\beta}(t_{k}^+)$. Therefore, from my perspective, term $I_k^+$ is entirely determined by $\\beta(\\cdot)$ and is fixed through the entire stage. However, from the proof and the discussion of the proof, it seems that $I_k^+$ is **specifically designed** to match the term of $\\gamma$. Besides, what is the advantage to utilize the calculation of equation (47)? It seems that it still introduces the $n$-th order differential calculations here. I wonder more intuitive explanation of such a calculation. In addition, I hope that authors can also clarify some of the notations. For example, is the domain of $G_k(\\beta)$ only from $\\beta_k$ to $\\beta_{k+1}$, or the entire space? If it is the former, then $G_k(\\beta)$ would always be a multiple of $\\nabla L(\\beta)$, in which case it is unclear what is gained by defining distinct $G_k(\\beta)$ functions. If it is the latter, then how should we interpret the expression $G_{k-1}(\\beta_{t_k})$ in equation (44)? Moreover, when using the shorthand $G_k$, what is the intended independent variable?\n\n2. The use of the notation $O(\\cdot)$ can lead to significant incorrectness in induction and should therefore be avoided in such contexts. To illustrate what I claim, let's consider a classical paradox when using the $O(\\cdot)$ in induction. For any integer $K$, it is obvious that we can write $K=\\sum_{k=1}^K a_k$, with $a_k=1$. Clearly, $a_1 = O(1)$ holds. Next, assume that at the $k'$-th iteration $\\sum_{k=1}^{k'} a_k = O(1)$, which leads to the conclusion that $\\sum_{k=1}^{k'+1} a_k = O(1) + 1 = O(1)$. Consequently, for any integer, no matter how large it is, we can always claim that it is $O(1)$, which, is obviously incorrect. To make induction mathematically rigorous, authors should explicitly specify the universal absolute constant hided in $O(\\cdot)$, and keep such a constant fixed during the proof. \n\n\nI would like to increase my score if the authors could provide further illustration and modify their proof (especially the $O(\\cdot)$ notation used in their induction proof of Lemma A.2.)\n\nQUESTIONS:\nWhile not a major issue, as a study including the implicit bias of momentum, I suggest the author could consider discussing [1, 2, 3] in the related works. \n\n[1]. Wang et al. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. ICML.\n\n[2]. Wang et al. Does Momentum Change the Implicit Regularization on Separable Data? Neurips.\n\n[3]. Zhang et al. The implicit bias of Adam on separable data. Neurips.\n\nLIMITATIONS:\nThe potential incorrectness in the proof by using the notation of $O(\\cdot)$ in induction.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}