A second-order-like optimizer with adaptive gradient scaling for deep learning

Authors: Jerome Bolte, Ryan Boustany, Edouard Pauwels, Andrei Purica
Primary area: optimization
Keywords: deep learning, second-order methods, stochastic optimization, dynamical systems

Abstract:
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.

Decision: reject
Reviewer scores: 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729097590309

thread_type:
official_review

<thread_object>
creator: reviewer_bdcw

<content>
summary:
The paper combines RMSprop adaptive gradient scaling to the existing INNA optimizer, which is based on the Dynamical Inertial Newton. The obtained optimizer leverages second-order information while having similar computational requirements to first-order methods. With appropriate hyper-parameters the optimizer matches and in some cases outperforms AdamW.

strengths:
The paper is well written, and easy to follow.
-    The derivation of the algorithm is well motivated.
-    They leverages second order information without requiring the hessian.
-    The algorithm seems to perform on par or even outperform AdamW in certain settings like language modeling.
-    The experiments cover a good amount of tasks between image classification and language modeling.
-    The authors open-source the code.

weaknesses:
-   The introduction of the paper makes a big deal about the energy consumption of training large language models, but the authors don't measure or show how their method compares to the competitors in terms of wall-clock time in their experiments. Particularly interesting would be seeing how wall-clock time on ImageNet or language modeling tasks compare.
-   The method lacks comparison with more recent optimizers other than AdamW. Sophia for example is another fast and efficient second-order optimizer paper, which the author cite, but don't compare with. Lion is another recent and high performing optimizer, which was cited but not compared with either.
-   Numeric results are reported without standard deviation/standard error or confidence intervals

questions:
-   How does INNAprop compare in terms of wall-clock time to AdamW or Sophia?
-   How does INNAprop compare to Sophia/Lion and other more recent high performing optimizers?
-   Is the increase in test accuracy of INNAprop compared to AdamW statistically significant? What are the error ranges?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646827911

thread_type:
official_review

<thread_object>
creator: reviewer_xyed

<content>
summary:
The authors introduce a second-order optimization method, INNAprop, which combines the INNA method with RMSprop, while not increasing the memory-footprint. The method is evaluated on a few image classification benchmarks, and GPT-2 with the OpenWebText dataset, and directly compared to AdamW with somewhat favorable results.

strengths:
It is a comprehensive and well-written paper that does a good job of explaining the motivation for, and the math behind, the proposed method. I do appreciate the main thesis of the work, and the effort that went into it. I like the discussion of, and approach to, hyperparameter tuning, as well as the comprehensive Appendix.

weaknesses:
My main objection is the empirical evaluation. Firstly, the reported improvements (on test sets) are generally very marginal. Secondly, the results are not pushed to SOTA-level --- even for the chosen architectures (that are pretty old and pretty small). Thirdly, there are too few experiments to convince me that the proposed method really works better than, or even as well as, AdamW. Since the method is not faster, or less computationally demanding, I find myself unconvinced of its overall usefulness.

questions:
I noticed that you mentioned the lack of computational resources two times (lines 259 and 359). You should know, that I am very sympathetic to that challenge. Still, my suggestion is that you bite the apple and invest that extra time and computational cost --- to really make a convincing case for your method :-)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725012906

thread_type:
official_review

<thread_object>
creator: reviewer_9gye

<content>
summary:
This paper proposes a new first-order optimization algorithm: INNAprop. The algorithm combines the INNA method with RMSProp to derive a novel optimization scheme. The authors evaluate INNAprop empirically against AdamW, the de facto standard deep learning optimizer. They explore a set of diverse tasks spanning image classification and natural language processing using several different classes of deep learning architectures.

strengths:
I thought the introduction was strong and successfully explained the need for improved optimization algorithm algorithms in terms of real-world impact. The authors back this up by deriving a memory-efficient version of their algorithm matching the memory overhead of existing approaches.

A thorough discussion of the algorithm and how it can be derived is provided. The authors also discuss comparisons between INNAprop and several other methods in the literature, in terms of its derivation. This is a valuable contribution that helps to place the proposed method among existing work. On the surface, I felt that the combination of the two methods could be of limited novelty. However, after reviewing the derivation I feel this constitutes a reasonably strong contribution.

The paper is clear and well-written. The authors provide a thorough set of diverse experiments to investigate the proposed method and include multiple runs to account for variability. From my review of the manuscript, all results look sensible and technically sound.

weaknesses:
Claiming second-order without computing second-order statistics doesn't seem accurate to me. The finite difference approximation may not be sufficiently accurate to achieve these benefits. However, I am not overly familiar with this approximation that the authors adopt from prior work. One way to alleviate this concern would be to provide additional justification for this terminology. Or, to include empirical evidence that the finite-difference approach is providing a good approximation of the actual second order statistics.

One of the claimed contributions is that INNAprop "matches or outperforms AdamW in both training speed and final accuracy on benchmarks". However, Fig. 2 shows the faster training speed of AdamW (the final error bars overlap for test accuracy so I'd agree that they match performance). Fig. 3 shows INNAprop as faster to converge for ResNet, but slower once again for ViT. The test accuracy is once again matched in both experiments. Fig. 4 shows a similar convergence speed with a modest (~1%) increase in accuracy for INNAprop. Finally, Fig. 5 shows a modest win for INNAprop on the smaller settings and matched performance for the larger model. For the LoRA finetuning (Fig. 6), there is a small win for the small setting and INNAprop matches AdamW on the medium and large setting.

The improvement over Adam variants is small across the board. The notable wins are on vision finetuning (Fig. 4) and small-scale GPT-2 training (Fig. 5). There are many methods that propose similarly slim margins and none are adopted by the community. Furthermore, under scrutiny from third-party implementations, the gains are often difficult to replicate [1]. Sadly, my honest opinion is that this method will not gain traction with the community --- it is unclear when it should be used over AdamW.

While often spurious, it is common for optimization algorithms to include theoretical analysis of their convergence. I do not consider this a critical weakness of this paper.

[1] "Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers", Schmidt et al.

questions:
Can you explain how you "systematically favored AdamW through the choice of recommended hyperparameters (scheduler, learning rates, weight decay)"? In more established tasks, I feel this is a reasonable claim. However, where INNAprop showed wins (finetuning on an uncommon dataset and training tiny GPT-2 models), I expect that standard AdamW hyperparameters are suboptimal. How much compute budget was spent tuning AdamW compared to INNAprop? I am also concerned that two different settings of INNAprop are shown in all experiments against a single choice of hyperparameters for AdamW.

In the appendix, you derived the DINAdam variant. Do you also explore this algorithm empirically?

With my initial review, I am recommending that the paper be rejected. I believe the paper is technically correct and the proposed algorithm is sufficiently novel. However, I do not believe that the algorithm provides a meaningful practical contribution for the wider community at this time. My concern is that a practitioner would always choose AdamW over INNAprop due to the marginal expected gains, increased risk of failure, and the additional overhead of parameter tuning required. My score would change if the authors could provide clear guidance on when a practitioner would choose INNAprop over AdamW and expect a win. 

Minor points:

- "we provide quite extensive experiments" vague statement in abstract. I would prefer that this be made more precise.
- "extra second-order geometrical intelligence" vague technical term. What do you mean by this? I think this should be clarified in the main text.

-------
**Post-rebuttal:**

Following the response from the authors, I have increased my score (and reduced my confidence). This is due primarily to the author's point that several results that I considered to be minimal improvements are in fact significant relative to the current state-of-the-art.

I retain concerns that the proposed algorithm is unlikely to see widespread community adoption. However, the paper is sound and proposes a novel technique. I would prefer that we present novel and correct methods to the community for their consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730841507574

thread_type:
official_review

<thread_object>
creator: reviewer_mr5g

<content>
summary:
This paper introduces INNAprop, a combination of the INNA optimizer with RMSprop gradient scaling. The resulting method has memory requirements similar to methods such as AdamW. Experiments show that this optimizer is competitive with AdamW.

strengths:
The derivation of the algorithm is clear and the performance often seems competitive with AdamW. Large-scale models such as ViT-B/32 and GPT-2 were trained.

weaknesses:
The INNAprop algorithm seems like a relatively straightforward extension from INNA. From what I can tell Algorithm 2 can be derived directly from the INNA equations in Table 2 and adding the factor $\\frac{1}{\\sqrt{v_{k+1}} + \\varepsilon}$ to the gradient. The motivation from a continuous dynamical systems perspective and the derivation done in appendix B seem equivalent to those from the INNA paper. This means that the theoretical insights from this paper seem limited.

This in itself wouldn't be a bad thing if the paper could make strong experimental claims about this small tweak to INNA leading to consistently stronger performance. However, this seems difficult to conclude for a few reasons, such as:

* No comparisons were made with INNA (without the RMSprop scaling) or other algorithms (Adam, ADAGRAD, SGD)
* In figures 3 and 6 INNAprop is sometimes indistinguishable from AdamW

Another surprise is that the optimal values for $\\alpha$ and $\\beta$ seem quite different to those found in the INNA paper (which ends up using $(0.1, 0.1)$ and $(0.5, 0.1)$) and it would be great to understand why. (In general, I am not convinced that the comparison with AdamW was the way to go, given that the proposed algorithm is far more similar to INNA. It would be insightful to know how the RMSprop extension changes INNA's behaviour and the choice of optimal hyperparameters for INNA.)

I understand that with limited compute it can be difficult to run large amounts of experiments, but I don't think that in it's current form this paper really provides new insights to the community: The RMSprop addition to INNA is pretty straightforward, and whether it really makes a difference is still hard to say.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732490718847

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment to all reviewers (1/2)

comment:
We thank all the reviewers for their constructive feedback, which will helped us a lot in improving our paper.

We agree that we did not properly emphasize several important points, starting with the hypertuning protocol and several performance reports which are essential to the understanding of this paper.

We, of course, hope the referees will take into account the new insights we bring to our work through their evaluations.

**AdamW is favored**.
AdamW has been systematically advantaged by using state-of-the-art parameters. These parameters are reused as is for INNAprop, and *only afterwards*, $\alpha$ and $\beta$ are optimized on CIFAR-10 (ResNet18, VGG11).

*We did not optimize these parameters for any of the other benchmark we adress* (we only observed that lower weigh-decay for ViT-B/32 seems better for INNAprop).

This is summarized below:

--Hyperparameter tuning strategy--

| **Parameters**              | **AdamW tuning**                | **INNAprop tuning**        | **Comparative advantage** |
|----------------------------|----------------------------------|----------------------------|----------------------------|
| Learning rate              | Literature or grid search tuning | Reused from AdamW          | AdamW favored             |
| Step size scheduler        | Literature                      | Reused from AdamW          | N/A                        |
| Weight decay               | Literature or grid search tuning | Reused from AdamW          | AdamW favored             |
| RMSprop parameter          | Default or literature            | Reused from AdamW          | AdamW favored             |
| Inertial parameters ($\alpha, \beta$) | N/A                            | Tuned on CIFAR-10          | N/A                        |

Despite this obvious handicap, INNAprop performs much better, especially considering the extensive tuning previously invested in AdamW.


In our coming revision, we will optimize INNAprop with tuning learning rates, weight decay and RMSProp parameters for CIFAR-10 experiments. Due to the long runtime of these experiments, we will add the results as soon as they are completed.


**AdamW & INNAprop**.
INNAprop is closely tied to Adam, as it incorporates an additional friction term that implicitly leverages the Hessian of RMSProp, as shown in Equation 5. Furthermore, by setting $\alpha=\beta = 1$, we empirically recover the behavior of AdamW. Experiments demonstrate that this tuning consistently aligns with AdamW, see Figure 7, 8, 9, suggesting that AdamW can be seen as a special case within the broader INNAprop family. This connection should have been emphasized in our pedagogical introduction. We have revised Remark 3 and Appendix B.1 to provide a clearer explanation. We have added illustrations in Figures 7, 8, and 9 in Appendix B.1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732647686377

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment to all reviewers (3)

comment:
Dear Reviewers,

We have uploaded a new version of the paper, incorporating your feedback and addressing the points raised in your reviews.

We have added figures to highlight the gains of INNAprop, as shown in Tables 2, 3, and 4. Experiments requiring more time, as mentioned in our responses, will be added once completed.

As the update deadline is tomorrow, we are available to make any further adjustments or address additional concerns. Please feel free to reach out with any suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies a new optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. The resulting method has memory and computational requirements similar to AdamW.  The paper demonstrates that this algorithm performs somewhat better than AdamW algorithm.

The reviews for the paper are slightly negative. The main concern of the reviewers is limited novelty and weak empirical analysis. I agree with the reviewers that the novelty is somewhat limited. The authors need to at least provide more solid empirical analysis to justify the claim of outperforming AdamW.  The presentation of the paper is poor and can also be improved. I also think it is worth comparing the performance of algorithm with approaches like Shampoo (even though these approaches are expensive, it is worth comparing the quality-computation tradeoff). For these reasons, I recommend rejection.