Title: EAGLE: Early Approximated Gradient-based Learning-rate Estimator
Venue: NeurIPS 2025
Accepted: no
Keywords: optimization algorithms

adaptive learning rate

gradient-based optimization

curvature estimation

neural network training

convergence acceleration
Abstract: Improving the learning efficiency of deep learning models remains a significant research focus. In this paper, we propose EAGLE (Early Approximated Gradient-based Learning-rate Estimator), a novel optimization method that accelerates parameter optimization. Firstly, to achieve faster loss convergence, EAGLE possesses the unique parameter update rule that leverages the local curvature of the loss landscape, derived from gradient variations between consecutive training steps. Secondly, to enhance training stability, it introduces a branching mechanism that adaptively switches to the existing Adam update rule under specific conditions where the EAGLE update rule might become unstable (e.g., extremely small gradient differences or locally upward convex shapes). In experiments on the GLUE SST-2 text classification task using a pre-trained GPT-2 model, EAGLE reached respectively the SGD with momentum’s final loss value 6.83× faster and the Adam’s final loss value 6.77× faster. Similarly, on the CIFAR-10 image classification task using a pre-trained ViT-B/16 model, EAGLE reached respectively the SGD with momentum’s final loss value 3.41× faster and the Adam’s final loss value 6.60× faster. To ensure reproducibility and promote further improvements, our code is publicly available on GitHub: https://github.com/keiotakmin/EAGLE


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEAGLE is studied as a preconditioner for single-node optimization. In the context of deep neural network (DNN) optimization, computing the Hessian matrix and its inverse introduces significant computational overhead. As a result, lightweight methods like Adam, which adaptively adjust the learning rate, are commonly preferred. EAGLE, as formulated in Eq. (1), achieves adaptive learning rate adjustment by utilizing update differences in parameters and gradients. Additionally, it demonstrates stable behavior by switching to Adam in response to the scenarios of the loss landscape. Through two DNN benchmark tasks, EAGLE consistently outperforms Adam in both convergence speed and final performance.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n- (S1) The authors provide an intuitive explanation of EAGLE’s behavior using illustrations based on the loss landscape. In addition, they propose a mechanism that switches to Adam to mitigate instability in convergence.\n\n- (S2) The experimental results consistently demonstrate that EAGLE outperforms Adam across two benchmark tests.\n\n- (S3) The source code is publicly available, allowing readers to easily try out the method.\n\n__Weaknesses__\n- (W1) Insufficient Motivation for Introducing the Update Rule\n\nLine 68 states: “EAGLE update rule can be theoretically justified as an approximation of Newton’s method.” However, does the preconditioning matrix used in Eq. (1) actually approximate the inverse Hessian $H^{-1}$?\n\nAlthough the example presented from line 75 attempts to explain the motivation for the update rule, it feels more like an empirical discussion. From a reviewer’s perspective, this is insufficient as a principled motivation for the preconditioner formulation.\n\n- (W2) Lack of Convergence Analysis\n\nEven if limited to convex functions, a theoretical convergence analysis would be valuable.\n\nThe switching mechanism between Adam and EAGLE is currently based on empirical heuristics. How does this switching behavior affect the convergence rate?\n\n- (W3) Small-Scale Experiments in Section 4\n\nWhile the comparisons between SGDm, Adam, and EAGLE are informative, many other optimizers have been proposed in this field. The scope of optimizer comparison is too limited. \n\nThe choice of benchmark tasks (CV and NLP) seems reasonable, but if the authors wish to emphasize empirical evidence, larger-scale benchmark evaluations would strengthen the argument.\n\n- (W4) Lack of Runtime Comparison on PyTorch\n\n- Since Eq. (1) appears to use update differences of parameters and gradients, implementing it on PyTorch and comparing the actual runtime should be feasible. (It is anticipated that EAGLE is unlikely to be critically slow.)\n\nQUESTIONS:\nNone in particular.\n\nLIMITATIONS:\nThe limitations of the empirically conducted experiments are discussed in Section 5; however, the social impact of the proposed method is not addressed, either in the main text or in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the *Early Approximated Gradient based Learning-rate Estimator* (EAGLE) optimisation algorithm, which adaptively switches between ADAM and a secant-method-type update rule. The authors evaluate EAGLE on language and vision fine-tuning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper addresses the important question of adaptive step-size selection in machine learning tasks.\n- The algorithm description is clear, and the overall paper is well-written and easy to follow.\n\n## Weaknesses\nThe primary weakness of this work is its limited novelty. The proposed EAGLE update rule is the component-wise secant method\n$$\n\\theta_{n+1} \\gets \\theta_n - \\eta \\frac{\\theta_n - \\theta_{n-1}}{\\nabla L (\\theta_n) - \\nabla L(\\theta_{n-1})} \\odot \\nabla L (\\theta_n),\n$$ \nwhich in its core has been known since 1800 BC [1], and has been around in its modern form since the 16th century [2]. Hence the core method itself lacks novelty, and key historical and modern references are missing.\nFurthermore, adaptive switching between different optimization rules have also been extensively studied in the last century [3-5], and more recently in [6] and references therein.  This literature is not discussed in the current submission. Although the specific adaptive conditions employed in this work appear novel, the sign-based condition closely resembles known methods such as Brent’s algorithm [7].\n\nAdditionally, the experimental setup has methodological weaknesses. The hyper-parameter tuning process is not transparently described, making it unclear whether the baseline methods were tuned. Moreover, the provided implementation uses AdamW for EAGLE within its adaptive switching mechanism, whereas the baseline utilizes Adam with L2 regularization. Given that AdamW is known to outperform Adam with L2 regularization [8], this discrepancy favours the proposed method.\n\nFinally, the work does not provide any theoretical guarantees for the proposed algorithm.\n\nBelow a list of possible typos. \n- Line 18: Intrduction -> Introduction\n- Line 235: Tab -> Table\n\n---\n\n### References\n\n[1] Papakonstantinou and Tapia, Origin and Evolution of the Secant Method in One Dimension, Amer. Math. Monthly, 2013.\n\n[2] Cardano, Artis magnae, sive de regulis algebraicis (Ars magna), Nuremberg, 1545.\n\n[3] Wu et al., Google’s neural machine translation system: Bridging the gap between human and machine translation, arXiv, 2016.\n\n[4] Akiba et al., Extremely large minibatch SGD: Training resnet-50 on ImageNet in 15 minutes, arXiv, 2017.\n\n[5] Keskar and Socher, Improving Generalization Performance by Switching from Adam to SGD, arXiv, 2017.\n\n[6] Schröder et al., Switching between Numerical Black-box Optimization Algorithms with Warm-starting Policies, arXiv, 2022.\n\n[7] Brent, Algorithms for minimization without derivatives, Courier Corporation, 2013.\n\n[8] Loshchilov and Hutter, Decoupled Weight Decay Regularization, ICLR 2019\n\nQUESTIONS:\n1. How sensitive are EAGLEs hyper-parameters? Given EAGLE adds additional knobs to turn, and can reduce to ADAM in case they are chosen large, the cost of hyper-parameter tuning should be discussed.\n2. How large is the memory overhead of EAGLE compared to Adam?\n3. How does the optimiser perform compared to other curvature-aware optimisers such as LION [9] or MUON [10, 11]? Even small-scale comparisons would help to position the contribution more clearly.\n \n---\n\n### References\n\n[9] Chen et al., Symbolic Discovery of Optimization Algorithms, NeurIPS 2023\n\n[10] Jordan et al., Muon: An optimizer for hidden layers in neural networks, Blog post, 2024 \n\n[11] Shen et al, On the Convergence Analysis of Muon, arXiv 2025\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EAGLE (Early Approximated Gradient-based Learning-rate Estimator), a novel optimization algorithm designed to accelerate training in deep learning models by incorporating lightweight curvature estimates from gradient differences. EAGLE proposes a new update rule that approximates second-order information without explicit Hessian computation and integrates an adaptive switching mechanism to fall back on Adam updates in unstable regimes. The method is evaluated on text classification (SST-2 with GPT-2) and image classification (CIFAR-10 with ViT), demonstrating significantly faster convergence and improved final accuracy compared to SGD with momentum and Adam.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n- The paper considers an important problem in optimization—accelerating convergence via curvature-aware updates without explicit Hessian computation. The proposed method is conceptually simple and easy to implement, with a clear mechanism for fallback stability via integration with Adam.\n\n**Weakness**:\n- The motivation behind this work is not clearly proposed. Although the authors claim to focus on estimating second-order curvature information without explicitly computing the Hessian, the rationale for the proposed method is neither intuitively explained nor theoretically grounded. I can understand the intuition that motivates the EAGLE update rule: since the Hessian $H$ is defined as $H = \\frac{\\partial g}{\\partial \\theta}$ , then in the limit $\\Delta \\theta \\to 0$, the ratio $\\frac{\\Delta g}{\\Delta \\theta}$ can approximate the Hessian. Consequently, the authors adopt $\\frac{\\Delta \\theta}{\\Delta g}$ as a proxy for $H^{-1}$. However, the paper lacks any discussion that would clarify or justify this approximation from either a mathematical or statistical perspective. The idea is merely presented through a toy quadratic example without elaborating on its limitations or applicability beyond this setting.\nSpecifically, I consider this form of curvature approximation to be relatively trivial. Crucially, the authors do not provide any analysis showing that this estimate has desirable statistical properties—e.g., whether it correlates with the true Hessian (or its inverse) in expectation or higher moments. In contrast, recent works such as Sophia [1] have proposed more principled approaches to curvature estimation using only first-order information. These methods introduce well-defined estimators for diagonal Hessian elements or Gauss-Newton approximations, and rigorously demonstrate that such estimators are unbiased or consistent under certain conditions. Further, they provide convergence guarantees—at least in the convex case—supporting the soundness of their methods. In comparison, EAGLE lacks all of these elements. The paper provides neither theoretical insight nor statistical justification for its update rule beyond a single-step illustration on a quadratic function. There is no analysis even within this simple quadratic setting (e.g., convergence upper bounds, stability regions), let alone any discussion of how the method behaves under the far more complex and non-convex landscapes characteristic of neural network training. As a result, I find it hard to be convinced that the proposed update rule offers a meaningful improvement over existing preconditioning or adaptive optimization methods. Without a clearer motivation, stronger justification, and a connection to existing literature on curvature estimation, the value of this contribution remains doubtable.\n\n- The authors propose EAGLE as a curvature-aware optimizer that conditionally falls back to Adam when the gradient difference $\\Delta g$ is below a threshold, in order to avoid instability due to potentially unbounded update magnitudes. While this adaptive switching mechanism is a reasonable design choice for ensuring training stability, the paper lacks any substantial analysis on how often and under what conditions this fallback is actually triggered during training. In particular, it is well known that during the pretraining or early fine-tuning phases of large models, the gradient norm tends to decrease rapidly—especially in the initial loss-drop phase—and then oscillates around a small value with low variance as the model gradually converges. In such stages, the difference between consecutive gradients $\\Delta g$ is typically very small. As a result, I suspect that the proposed method may align with Adam for the majority of training steps, with the EAGLE update rule being applied only in the very early stages where gradients are still large and varying.\nThis concern is partially supported by the loss curves reported in the paper: the major divergence between EAGLE and baseline optimizers (SGD, Adam) occurs in the first few thousand steps, after which the curves follow similar trajectories. This suggests that the benefit of EAGLE might stem primarily from selecting larger effective step sizes during the early phase, rather than maintaining a consistently different or curvature-aware update trajectory throughout training. Given this, I believe the paper would strongly benefit from a detailed analysis of the dynamics of the update rule selection. Without such analysis, it is reasonable to interpret EAGLE as an ad-hoc adjustment of the learning rate schedule in the early stage, which then defaults to Adam-like behavior for the majority of training. This significantly weakens the argument that EAGLE provides a fundamentally new optimization behavior.\n\n- The experimental results presented in the paper are not sufficiently convincing to establish the practical effectiveness of the proposed method. All evaluations are conducted solely in the context of fine-tuning pre-trained models. However, fine-tuning tasks are often relatively easy optimization problems, where convergence is less challenging and many simple methods—even some zeroth-order techniques—can yield competitive performance. As such, the ability to accelerate convergence in this context does not necessarily imply that the proposed optimizer is generally effective or robust. To truly demonstrate the validity, even training a smaller architecture like GPT-2 from scratch on a mid-sized corpus would be informative. This kind of experiment would reveal whether the curvature approximation strategy employed by EAGLE can consistently support convergence over a long training horizon and under diverse gradient regimes. If the method fails to converge in this setting—or proves to be unstable—its utility would be severely limited.\nFurthermore, given that EAGLE introduces non-negligible additional computation per step (as acknowledged in the paper),in fine-tuning, where a wide variety of optimizers already perform well with minimal tuning, the computational overhead of EAGLE is hard to justify unless its advantages are clearly established in more general or challenging settings.\n[1] Hong Liu, Zhiyuan Li, David Hall, Percy Liang, Tengyu Ma, Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training\n\n\n\nGiven the concerns outlined above regarding the lack of theoretical justification, insufficient analysis of the update mechanism’s behavior, and limited experimental validation, I believe this paper requires substantial revisions before it can meet the standards expected for publication at NeurIPS.\n\nQUESTIONS:\nsee Weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a novel optimization method that accelerates loss convergence during early stages of training.

STRENGTHS AND WEAKNESSES:
Strengths:
1. A novel update rule utilizing the local curvature of the loss function between consecutive steps.
2. The authors provides an adaptive switching mechanism, switching update rules according to different constraints.

Weakness:
1. The EAGLE update rules was only tested in small benchmark scale.
2. The EAGLE method didn't compare with existing methods eg. AdamW, Lion, Sophia, etc. 
3. Limited Novelty. Curvature from gradient differences and hybrid optimizers have prior works eg. AdaQN, Shampoo+momentum, Sophia, AdaHessian methods, the paper does not differentiate clearly from these.

QUESTIONS:
1. Would it be possible if authors could provide a comparison with state-of-art works mentioned in the Weakness? I would be raise my score if EAGLE works better than those.
2. Could the authors provide more training experiments on larger models eg. Llama350M, Llama1b? To test whether this method have a boarder utility.
3. Could the authors clarify the difference between EAGLE work and existing optimization works eg. AdaQN, AdaHessian. It would lack of novelty if EAGLE method is mainly about specific threshold formula, which constitutes only an incremental improvement.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-03