A second-order-like optimizer with adaptive gradient scaling for deep learning

## Abstract

Abstract In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum.
After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at https://github.com/innaprop/innaprop .

## 1 Introduction

As deep learning models grow in size, massive computational resources are needed for training, representing significant challenges in terms of financial costs, energy consumption, and processing time (Susnjak et al., ; Varoquaux et al., ). According to the UN’s Environment Programme Training, the Big Tech sector produced between two and three percent of the world’s carbon emissions in 2021; some estimations for the year 2023 go beyond 4%, see the latest Stand.earth reports, and also (Schwartz et al., ; Strubell et al., ; Patterson et al., ) for related issues. For instance, training GPT-3 is estimated to require 1,287 megawatt-hours (MWh) of electricity, equivalent to the annual usage of over 100 U.S. households (Anthony et al., ; Patterson et al., ). Similarly, the financial cost of specialized hardware and cloud computing is extremely high. OpenAI claimed that the training cost for GPT-4 (Achiam et al., ) exceeded 100 million dollars. The PaLM model developed by Google AI was trained for two months using 6144 TPUs (^1^11Tensor Processing Unit.) for 10 million dollars (Chowdhery et al., ). All this implies a need for faster and more cost-efficient optimization algorithms. It also suggests that early stopping (Prechelt, ; Bai et al., ) in the training phase is a desirable feature whenever possible.

We focus in this work on computational efficiency during the training phase and consider the problem of unconstrained minimization of a loss function $\mathcal{J}\colon\mathbb{R}^{p}\to\mathbb{R}$, as follows

$$ $\displaystyle\min_{\theta\in\mathbb{R}^{p}}\mathcal{J}(\theta).$ (1) $$

##### Continuous dynamical systems as optimization models.

To achieve higher efficiency, it is necessary to deeply understand how algorithms work and how they relate to each other. A useful way to do this is by interpreting optimization algorithms as discrete versions of continuous dynamical systems (Ljung, ), further developed in (Harold et al., ; Benaïm, ; Borkar & Borkar, ; Attouch et al., ; Aujol et al., ; Castera et al., ).
In deep learning, this approach is also quite fruitful; it has, in particular, been used to provide convergence proofs or further geometric insights (Davis et al., ; Bolte & Pauwels, ; Barakat & Bianchi, ; Chen et al., ).

In the spirit of Castera et al. (; ), we consider the following continuous-time dynamical system introduced
in Alvarez et al. () and referred to as DIN (standing for “dynamical inertial Newton”):

$$ $\displaystyle\underbrace{\ddot{\theta}(t)}_{\text{\rm Inertial term}}+\underbrace{\alpha\,\dot{\theta}(t)}_{\text{\rm Friction term}}+\underbrace{\beta\,\nabla^{2}\mathcal{J}(\theta(t))\dot{\theta}(t)}_{\text{\rm Newtonian effects}}+\underbrace{\nabla\mathcal{J}(\theta(t))}_{\text{\rm Gravity effect}}=\ 0,\qquad t\geq 0,$ (2) $$

where $t$ is the time, $\mathcal{J}\colon\mathbb{R}^{p}\to\mathbb{R}$ is a loss function to be minimized (e.g., empirical loss in DL applications) as in [Equation 1](#S1.E1), assumed $C^{2}$ with gradient $\nabla\mathcal{J}$ and Hessian $\nabla^{2}\mathcal{J}$. A key aspect of Equation ([2](#S1.E2)) that places it between first- and second-order optimization is that a change of variables allows to describe it using only the gradient $\nabla\mathcal{J}$, since $\nabla^{2}\mathcal{J}(\theta(t))\dot{\theta}(t)=\frac{d}{dt}\nabla\mathcal{J}(\theta(t))$ (see Section [2.2](#S2.SS2) for details). This greatly reduces computational costs, as it can be discretized as a difference of gradients which does not require Hessian vector product, making it possible to design more practical algorithms, as shown in Chen & Luo (); Castera et al. (); Attouch et al. ().

We recover the continuous-time heavy ball system by assuming $\alpha>0$, and removing the geometrical “damping” term in Equation ([2](#S1.E2)) through the choice $\beta=0$. A discrete version of this system corresponds to the Heavy Ball method (Polyak, ), which is at the basis of SGD solvers with momentum in deep learning (Qian, ; Sutskever et al., ). By allowing both $\alpha$ and $\beta$ to vary, we recover Nesterov acceleration (Nesterov, ; Su et al., ; Attouch et al., ).

##### Adaptive methods.

Adaptive optimization methods, such as RMSprop (Tieleman et al., ) and AdaGrad (Duchi et al., ), modify the update dynamics by introducing coordinate-wise scaling of the gradient based on past information. These methods can be modeled by continuous-time ODEs of the following form, expressed here for the simple gradient system:

$$ $\displaystyle\dot{\theta}(t)+\frac{1}{\sqrt{G(t,\theta(t))+\epsilon}}\odot\nabla\mathcal{J}(\theta(t))=0,\quad t\geq 0,$ (3) $$

where $\epsilon>0$, $G(t,\theta(t))\in\mathbb{R}^{p}$ represents accumulated information. The scalar addition, square root, and division are understood coordinatewise and $\odot$ denotes the coordinate-wise product for vectors in $\mathbb{R}^{p}$. In AdaGrad or RMSprop, $G(t,\theta(t))$ is defined as an accumulation of squared gradient coordinates of the form:

$$ $\displaystyle G(t,\theta(t)):=\int_{0}^{t}\nabla\mathcal{J}(\theta(\tau))^{2}\,d\mu_{t}(\tau),$ (4) $$

for different choices of $\mu_{t}$ (uniform for AdaGrad and moving average for RMSprop). Both approaches scale the gradient based on accumulated information on past gradient magnitudes, improving performance, particularly in settings with sparse or noisy gradients (Duchi et al., ; Tieleman et al., ).

##### Our approach.

We combine the “dynamical inertial Newton” method (DIN) from Equation ([2](#S1.E2)) with an RMSprop adaptive gradient scaling. This allows us to take into account second-order information for the RMSProp scaling. Computationally, this second-order information is expressed using a time derivative. In discrete time, this will provide a second-order intelligence with the same computational cost as gradient evaluation. The resulting continuous time ODE is given as follows:

$$ $\displaystyle\ddot{\theta}(t)+\alpha\,\dot{\theta}(t)+\beta\,\frac{d}{dt}\text{RMSprop}(\mathcal{J}(\theta(t)))+\text{RMSprop}(\mathcal{J}(\theta(t)))=0,\qquad\,t\geq 0$ (5) $\displaystyle\text{where }\text{RMSprop}(\mathcal{J}(\theta(t)))=\frac{1}{\sqrt{G(t,\theta(t))+\epsilon}}\odot\nabla\mathcal{J}(\theta(t))$ $$

with $G$ of the form ([4](#S1.E4)) with an adequate time-weight distribution $\mu_{t}$ corresponding to the RMSProp scaling. A discretization of this continuous time system, combined with careful memory management, results in our new optimizer INNAprop, see Section [2.1](#S2.SS1).

##### Relation with existing work.

To improve the efficiency of stochastic gradient descent (SGD), two primary strategies are used: leverage local geometry for having clever directions and incorporate momentum to accelerate convergence. These approaches include accelerated methods (e.g., Nesterov’s acceleration (Nesterov, ; Dozat, ), momentum SGD (Polyak, ; Qian, ; Sutskever et al., ), and adaptive methods (e.g., Adagrad (Duchi et al., ), RMSProp (Tieleman et al., )), which adjust learning rates per parameter.

Adam remains the dominant optimizer in deep learning. It comes under numerous variants proposed to improve its performance or to adapt it to specific cases (Dozat, ; Shazeer & Stern, ; Reddi et al., ; Loshchilov & Hutter, ; Zhuang et al., ). Adafactor (Shazeer & Stern, ) improves memory efficiency, Lamb (You et al., ) adds layerwise normalization, and Lion (Chen et al., ) uses sign-based momentum updates. AdEMAMix (Pagliardini et al., ) combines two EMAs, while Defazio et al. (Defazio et al., ) introduced a schedule-free method incorporating Polyak-Ruppert averaging with momentum.

One of the motivations of our work is the introduction of second-order properties in the dynamics akin to Newton’s method. Second-order optimizers are computationally expensive due to frequent Hessian computations (Gupta et al., ; Martens & Grosse, ) and their adaptation to large scale learning settings require specific developments (Jahani et al., ; Qian et al., ). For example, the Sophia optimizer (Liu et al., ), designed for large language models, uses a Hessian-based pre-conditioner to penalize high-curvature directions. In this work, we draw inspiration from the INNA optimizer (Castera et al., ), based on the continuous time dynamics introduced by (Alvarez et al., ), which combines gradient descent with a Newtonian mechanism for first-order stochastic approximations.

Our proposed method, INNAProp, integrates the algorithm INNA, which features a Newtonian effect with cheap computational cost, with the gradient scaling mechanism of RMSprop. This framework preserves the efficiency of second-order methods and the adaptive features of RMSprop while significantly reducing the computational overhead caused by Hessian evaluation. Specific hyperparameter choices for our method allow us to recover several existing optimizers as special cases.

##### Contributions.

They can be summarized as follows:

- $\bullet$
We introduce INNAprop, a new optimization algorithm that combines the Dynamical Inertial Newton (DIN) method with RMSprop’s adaptive gradient scaling, efficiently using second-order information for large-scale machine learning tasks.
We obtain a second-order optimizer with computational requirements similar to first-order methods like AdamW, making it suitable for deep learning (see Section [2.2](#S2.SS2) and Appendix [B](#A2)).
- $\bullet$
We provide a continuous-time explanation of INNAprop, connecting it to second-order ordinary differential equations (see [Section 2](#S2) and [Equation 5](#S1.E5)). We discuss many natural possible discretizations and show that INNAprop is empirically the most efficient. Let us highlight a key feature of our method: it incorporates second-order terms in space without relying on Hessian computations or inversions of linear systems which are both prohibitive in deep learning.
- $\bullet$
We show through extensive experiments that INNAprop matches or outperforms AdamW in both training speed and final accuracy on benchmarks such as image classification (CIFAR-10, ImageNet) and language modeling (GPT-2) (see Section [3](#S3)).

We describe our algorithm and its derivation in Section [2](#S2). Hyperparameter tuning recommendations and our experimental results are provided in Section [3](#S3).

## 2 INNAprop: a second-order method in space and time based on RMSProp

### 2.1 The algorithm

Our method is built on the following [Algorithm 1](#alg1), itself derived from a combination of INNA
(Castera et al., ) and RMSprop (Tieleman et al., ) (refer to Section [2.2](#S2.SS2) for more details). The following version of the method is the one we used in all experiments. It includes the usual ingredients of deep-learning training: mini-batching, decoupled weight-decay, and scheduler procedure. For a simpler, “non-deep learning” version, refer to Algorithm [2](#alg2) in Appendix [B](#A2).

Figure: Algorithm 1 Deep learning implementation of INNAprop

In [Algorithm 1](#alg1), SetLrSchedule is the “scheduler” for step-sizes; it is defined as a custom procedure for handling learning rate sequences for different networks and databases. To provide a full description of our algorithm, we provide detailed explanations of the scheduler procedures used in our experiments (Section [3](#S3)) in Appendix [D](#A4), along with the corresponding benchmarks.

###### Remark 1 (Well posedness)

Observe that, for all schedulers $\gamma_{k}<\beta$ for $k\in\mathbb{N}$, so that INNAprop is well-posed (line 13 in Algorithm [1](#alg1), the division is well defined).

###### Remark 1 (Well posedness)

### 2.2 Derivation of the algorithm

There are several ways to combine RMSprop and INNA, or DIN its second-order form, as there exist several ways to do so with the heavy ball method and RMSprop. We opted for the approach below because of its mechanical and geometrical appeal and its numerical success (see Appendix [B](#A2) for further details).
Consider the following dynamical inertial Newton method (Alvarez et al., ):

$$ $\displaystyle\ddot{\theta}(t)+\alpha\,\dot{\theta}(t)+\beta\,\frac{d}{dt}\nabla\mathcal{J}(\theta(t))+\nabla\mathcal{J}(\theta(t))=\ 0,\quad t\geq 0,$ (6) $$

as in [Equation 2](#S1.E2) and replacing $\nabla^{2}\mathcal{J}(\theta(t))\dot{\theta}(t)$ by $\frac{d}{dt}\nabla\mathcal{J}(\theta(t))$. We use finite differences with a fixed time step $\gamma$ to discretize this system, replacing in particular the gradient derivatives by gradient differences:

$$ $\frac{d}{dt}\nabla\mathcal{J}(\theta(t))\simeq\frac{\nabla\mathcal{J}(\theta_{k+1})-\nabla\mathcal{J}(\theta_{k})}{\gamma},$ $$

where
$\theta_{k},\theta_{k+1}$ correspond to two successive states around the time $t$.

Setting $\nabla\mathcal{J}(\theta_{k})=g_{k}$, we obtain

$$ $\displaystyle\frac{\theta_{k+1}-2\theta_{k}+\theta_{k-1}}{\gamma}+\alpha\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{{g_{k}}-g_{k-1}}{\gamma}+g_{k-1}=0.$ (7) $$

To provide our algorithm with an extra second-order geometrical intelligence, we use the proxy of RMSprop direction in place of the gradient.

Choose $\sigma>0$ and $\epsilon>0$, and consider:

$$ $\displaystyle v_{k+1}=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ (8) $\displaystyle\frac{\theta_{k+1}-2\theta_{k}+\theta_{k-1}}{\gamma}+\alpha\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}}{\gamma}+\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}=0.$ (9) $$

Although this system has a natural mechanical interpretation, its memory footprint is abnormally important for this type of algorithm: for one iteration of the system ([8](#S2.E8))-([9](#S2.E9)), it culminates at 6 full dimension memory slots, namely $g_{k-1}$, $g_{k}$, $\theta_{k-1}$, $\theta_{k}$, $v_{k}$, and $v_{k+1}$ before the evaluation of ([9](#S2.E9)).

Therefore, we proceed to rewrite the algorithm in another system of coordinates. The computations and the variable changes are provided in Appendix [B](#A2). We eventually obtain:

$$ $\displaystyle v_{k+1}=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k},$ $\displaystyle\theta_{k+1}$ $\displaystyle=\left(1+\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $$

which only freezes 3 full dimension memory slots corresponding to $v_{k}$, $\psi_{k}$, $\theta_{k}$. As a result, the memory footprint is equivalent to that of the Adam optimizer (see [Table 2](#A1.T2)).

###### Remark 2 (On other possible discretizations)

(a) If we use the proxy of RMSprop directly with INNA (Castera et al., ), we recover indeed INNAprop through a rather direct derivation (see Appendix [C.1](#A3.SS1) for more details). Our motivation to start from the “mechanical” version of the algorithm is to enhance our understanding of the geometrical features of the algorithm.
(b) RMSprop with momentum (Graves, ) is obtained by a discretization of the heavy ball continuous time system, using a momentum term and an RMSprop proxy. It would be natural to proceed that way in our case, and it indeed leads to a different method (see Appendix [C.2](#A3.SS2)). However, the resulting algorithm appears to be numerically unstable (see Figure [7](#A3.F7) for an illustration).
(c) Incorporating RMSprop as it is done in Adam by using momentum leads to a third method (see Appendix [C.3](#A3.SS3)). This algorithm appears to be extremely similar to NAdam Dozat (); it was thus discarded.

###### Remark 3 (A family of algorithms indexed by $\alpha,\beta$ )

INNAprop can be seen as a family of methods indexed by the hyperparameters $\alpha$ and $\beta$. When $\beta=0$, we recover a modified version of RMSprop with momentum (Graves, ) (see Appendix [B.1](#A2.SS1)). For $\alpha=\beta=1$, INNAprop with its default initialization, boils down to AdamW without momentum ($\beta_{1}=0$), see Appendix [B.1](#A2.SS1) and [Table 2](#A1.T2). In the next sections, we explain how these hyperparameters ($\alpha,\beta$) have been tuned on “small size” problems.

###### Remark 2 (On other possible discretizations)

###### Remark 3 (A family of algorithms indexed by α , β 𝛼 𝛽 \alpha,\beta )

## 3 Empirical evaluation of INNAprop

We conduct extensive comparisons of the proposed algorithm and the AdamW optimizer, which is dominantly used in image classification (Chen et al., ; Zhuang et al., ; Touvron et al., ; Mishchenko & Defazio, ) and language modeling tasks (Brown et al., ; Hu et al., ; Liu et al., ). Hyperparameter tuning (Sivaprasad et al., ) is a crucial issue for this comparison, and we start with this. As a general rule, we strive to choose the hyperparameters that give a strong baseline for AdamW (based on literature or using grid search). Unless stated differently, our experiments use the AdamW optimizer (^2^22[https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html)) with its default settings as defined in widely-used libraries like PyTorch (Paszke et al., ), Jax (Bradbury et al., ), and TensorFlow (Abadi et al., ): $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\lambda=0.01$ and $\epsilon=1e-8$. For INNAprop, unless otherwise specified, the default settings for the RMSprop component align with those of AdamW: $\sigma=0.999$ and $\epsilon=1e-8$.

The INNAprop method and the AdamW optimizer involve different classes of hyperparameters; some of them are common to both algorithms, and some are specific.

Our hyperparameter tuning strategy for both algorithms is summarized in Table [1](#S3.T1).

We begin this section with the tuning of parameters $\alpha,\beta$ for INNAprop on CIFAR10 with VGG and ResNet architectures and then use these parameters on larger datasets and models. We use as much as possible the step size scheduler and weight decay settings reported in the literature for the AdamW optimizer, which we believe to be well-adjusted and provide adequate references for each experiment. These are used both for AdamW and INNAprop. With this protocol, we only perform minimal hyperparameter tuning for INNAprop for larger-scale experiments. This is due to constrained computational resources. We aim to demonstrate the typical performance of the Algorithm [1](#alg1), rather than its peak performance with extensive tuning.

**Table 1: Summary of our hyperparameter tuning strategy for INNAprop and AdamW.**
| Algorithm | Parameter | Tuning |
| --- | --- | --- |
| AdamW & INNAprop | Scheduler and weight decay | AdamW literature, minimal tuning |
| AdamW & INNAprop | RMSprop parameters | AdamW default values, or AdamW |
|  |  | literature (for e.g GPT-2 training) |
| INNAprop | Dynamical parameters $\alpha,\beta$ ([6](#S2.E6)) | Tuned on CIFAR10 |
| AdamW | AdamW parameters $\beta_{1}$ | AdamW default values, or AdamW |
|  |  | literature |

### 3.1 Tuning INNAprop on CIFAR-10 with VGG11 and ResNet18

##### Hyperparameter tuning:

We tune ($\alpha,\beta$) using VGG11 (Simonyan & Zisserman, ) and ResNet18 (He et al., ) models trained on CIFAR10 (Krizhevsky & Hinton, ), together with the initial learning rate $\gamma_{0}$ to ensure proper training. We fix a cosine scheduler where $T_{\text{max}}=200$ and $\gamma_{\min}=0$ (see [Appendix D](#A4) for more details) and we consider two weight decay parameters $\lambda=0$ or $\lambda=0.01$. Our experiment suggests using an initial learning rate $\gamma_{0}=10^{-3}$, which is the baseline value reported for AdamW in this experiment (see [Appendix E](#A5)). For INNAprop, we optimize only the hyperparameters $\alpha$ and $\beta$, using test accuracy and training loss as the optimization criteria. A grid search is performed over $(\alpha,\beta)\in\{0.1,0.5,0.9,\dots,3.5,4.0\}$ using optuna (Akiba et al., ). In Figure [1](#S3.F1), we detail the obtained training loss and test accuracy for various $(\alpha,\beta)$ configurations over short training durations (20 epochs) and long training durations (200 epochs) for VGG11 with weight decay $\lambda=0.01$. Our criteria (short and long training duration) are chosen to find parameters $(\alpha,\beta)$ that provide a rapid decrease in training loss in the early stages and the best test accuracy for long training duration.

These results highlight a tendency for efficient couples; we choose for further experiments the values $(\alpha,\beta)=(0.1,0.9)$ which correspond to aggressive optimization of the training loss for short training durations, and $(\alpha,\beta)=(2.0,2.0)$ which provides very good results for longer training durations. Additional results for VGG11 and ResNet18 with and without weight decay are in Appendix [F.4](#A6.SS4), which are qualitatively similar.

Figure: (a) 20 epochs
Refer to caption: /html/2410.05871/assets/x1.png

##### Validation and comparison with AdamW:

We confirm our hyperparameter choices ($\gamma_{0}=10^{-3}$, $\lambda=0.01$) by reproducing the experiment with 8 random seeds and comparing with AdamW using the same settings. Based on hyperparameter tuning, we select two pairs of $(\alpha,\beta)$ with different training speeds. As shown in Figure [2](#S3.F2) (and [Appendix F](#A6) for ResNet18), with $(\alpha,\beta)=(0.1,0.9)$, INNAprop improves training loss and test accuracy rapidly by the 100th epoch, maintaining the highest training accuracy. With $(\alpha,\beta)=(2.0,2.0)$, INNAprop trains more slowly but achieves higher final test accuracy. This is aligned with the experiments described in [Figure 1](#S3.F1).

Figure: Figure 2: Training VGG11 on CIFAR10. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 8 random seeds.
Refer to caption: /html/2410.05871/assets/x3.png

###### Remark 4 (Trade-off between fast learning and good generalization)

For CIFAR-10 experiments, INNAprop offers flexibility in adjusting convergence speed through $(\alpha,\beta)$. Faster training configurations generally lead to weaker generalization compared to slower ones, highlighting the trade-off between quick convergence and strong performance on unseen data (Wilson et al., ; Zhang et al., ).

###### Remark 4 (Trade-off between fast learning and good generalization)

### 3.2 Extensive experiments on large-scale vision models

In this section, we present experimental results on large-scale vision benchmarks, using the hyperparameters selected as described in Section [3.1](#S3.SS1).

##### Resnets on ImageNet:

We consider the larger scale ImageNet-1k benchmark (Krizhevsky et al., ). We train a ResNet-18 and a ResNet-50 (He et al., ) for 90 epochs with a mini-batch of size of 256 as in Chen et al. (); Zhuang et al. (). We used the same cosine scheduler for both AdamW and INNAprop with initial learning rate $\gamma_{0}=10^{-3}$ as reported in Chen et al. (); Zhuang et al. (); Chen et al. (). The weight decay of AdamW is set to $\lambda=0.01$ for the ResNet18, instead of $\lambda=0.05$ reported in Zhuang et al. (); Chen et al. () because it improved the test accuracy from $67.93$ to $69.43$. The results of the ResNet18 experiment are presented in [Figure 11](#A6.F11) in [Appendix F](#A6). The figure shows that our algorithm with $(\alpha,\beta)=(0.1,0.9)$ outperforms AdamW in test accuracy (70.12 vs 69.43), though the training loss decreases faster initially but slows down towards the end of training.

For the ResNet50, we kept the value $\lambda=0.1$ as reported in Zhuang et al. (); Chen et al. (). For INNAprop, we tried two weight decay values $\{0.1,0.01\}$ and selected $\lambda=0.01$ as it resulted in a faster decrease in training loss. We report the results in [Figure 3](#S3.F3), illustrating the advantage of INNAprop. As noted in Section [3.1](#S3.SS1), INNAprop with $(\alpha,\beta)=(0.1,0.9)$ reduces training loss quickly but has lower test accuracy compared to AdamW or INNAprop with $(\alpha,\beta)=(2.0,2.0)$. For $(\alpha,\beta)=(2.0,2.0)$, the loss decrease is similar to AdamW, with no clear advantage for either method. This obviously suggests developing scheduling strategies for damping parameters $(\alpha,\beta)$. This would require a much more computation-intensive tuning, far beyond the numerical resources used in the current work.

##### Vision transformer (ViT) on ImageNet:

We performed the same experiment with a ViT-B/32 architecture over 300 epochs with a mini-batch size of 1024, following Defazio & Mishchenko (); Mishchenko & Defazio (). For AdamW, we used a cosine scheduler with a linear warmup (30 epochs) and the initial learning rate and weight decay from Defazio & Mishchenko (). For INNAprop, we tested weight decay values of $\{0.1,0.01\}$, selecting $\lambda=0.1$ for better test accuracy. Results in [Figure 3](#S3.F3) show the advantage of INNAprop. For faster convergence using INNAprop $(0.1,0.9)$, we recommend a weight decay of $\lambda=0.01$ (see Figure [12](#A6.F12) in the Appendix).

Figure: Figure 3: Training a ResNet50 (top) and ViT-B/32 (bottom) on ImageNet. Left: train loss, middle: Top-1 test accuracy (%), right: Top-1 train accuracy (%). 3 random seeds.
Refer to caption: /html/2410.05871/assets/x4.png

In the ImageNet experiments, we evaluated INNAprop for rapid early training and optimal final test accuracy without tuning $(\gamma_{0},\alpha,\beta)$. For ViT-B/32 with $\lambda=0.1$, INNAprop achieved lower training loss and higher final test accuracy than AdamW (75.23 vs. 75.02). For ResNet-18, INNAprop also outperformed AdamW in final test accuracy with $\lambda=0.01$ (70.12 vs. 69.34).

##### Fintetuning VGG11 and ResNet18 models on Food101:

We fine-tuned ResNet-18 and VGG-11 models on the Food101 dataset (Bossard et al., ) for 20 epochs, using pre-trained models on ImageNet-1k. Since weight decay and learning rate values for AdamW were not found in the literature, we chose the default AdamW weight decay value, $\lambda=0.01$. We used a cosine scheduler and tried one run for each initial learning rate value in $\{10^{-5},5\times 10^{-5},10^{-4},5\times 10^{-4},10^{-3}\}$. The best result for AdamW was obtained for $\gamma_{0}=10^{-4}$, and we kept the same setting for INNAprop. The results are depicted in [Figure 4](#S3.F4), where INNAprop performs no worse than AdamW on three random seeds.

Figure: Figure 4: Finetuning a VGG11 on Food101. Left: train loss, middle: test accuracy (%), right: train accuracy (%). Qualitatively similar results for ResNet18 are in [Figure 10](#A6.F10) in [Appendix F](#A6). 3 random seeds.
Refer to caption: /html/2410.05871/assets/x5.png

##### Conclusion and recommendation for image classification:

Tuning $(\alpha,\beta)$ significantly impacts training. Based on heatmaps in Section [3.1](#S3.SS1) and figures in Section [3.2](#S3.SS2), we recommend using $\alpha=0.1$ and $\beta\in[0.5,1.5]$ for shorter training (e.g., fine-tuning). For longer training, $\alpha,\beta\geq 1$ is preferable. In both cases, our algorithm either matches or outperforms AdamW.

### 3.3 Pre-training and fine-tuning GPT2

In this section, we present experimental results on large-scale language benchmarks, using the hyperparameters selected as outlined in Section [3.1](#S3.SS1).

##### Training GPT-2 from scratch:

We train various GPT-2 transformer models from scratch (Radford et al., ) using the nanoGPT repository(^3^33[https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)) on the OpenWebText dataset. For all models, gradients are clipped to a norm of 1, following Mishchenko & Defazio (); Liu et al. (); Brown et al. (). We use AdamW with hyperparameters from the literature (Liu et al., ; Brown et al., ), the standard configuration for LLM pre-training. The reported RMSprop parameter $\beta_{2}=0.95$ is different from Adam’s default (0.999), the weight decay is $\lambda=0.1$ and $\gamma_{0}$ depending on the network size (see Brown et al. (); Liu et al. ()). For example, GPT-2 small works with an initial learning rate $\gamma_{0}=6\times 10^{-4}$. For INNAprop, we keep the same values for $\lambda$ and $\gamma_{0}$ as AdamW, and use the RMSprop parameter $\sigma=0.99$ (corresponding to $\beta_{2}$ for Adam), which provides the best results among values $\{0.9,0.95,0.99\}$ on GPT-2 mini. We use this setting for all our GPT-2 experiments with $(\alpha,\beta)=(0.1,0.9)$ and $(\alpha,\beta)=(2.0,2.0)$. The results are in [Figure 5](#S3.F5). INNAprop leads to a faster decrease in validation loss during the early stages compared to the baseline for GPT-2 models of Mini (30M), Small (125M), and Medium (355M) sizes. Its performance could be further improved with more thorough tuning of hyperparameters ($\alpha,\beta,\sigma,\lambda$).

Figure: Figure 5: GPT-2 training from scratch on OpenWebText.
Refer to caption: /html/2410.05871/assets/x6.png

##### Fine-tune GPT-2 with LoRA:

Using LoRA (Hu et al., ), we fine-tune the same GPT-2 models on the E2E dataset, consisting of roughly 42000 training 4600 validation, and 4600 test examples from the restauration domain. We compare AdamW and INNAprop for 5 epochs, as recommended in Hu et al. (). We use for both algorithms the same linear learning rate schedule, the recommended mini-batch size, and the RMSprop parameter ($\beta_{2}=\sigma=0.999$); these are listed in Table 11 in Hu et al. (). The results are displayed in Figure [6](#S3.F6), where we see the perplexity mean result over 3 random seeds. INNAprop with $(\alpha,\beta)=(0.1,0.9)$ consistently achieves lower perplexity loss compared to AdamW across all GPT-2 fine-tuning experiments.

Figure: Figure 6: Perplexity test with GPT-2 E2E Dataset with LoRA finetuning on five epochs. Three random seeds.
Refer to caption: /html/2410.05871/assets/x7.png

## 4 Conclusion

We introduce INNAprop, an optimizer that leverages second-order geometric information while maintaining memory and computational footprints similar to AdamW. Experiments on text modeling and image classification show that INNAprop consistently matches or exceeds AdamW’s performance.

In our approach, we systematically favored AdamW through the choice of recommended hyperparameters (scheduler, learning rates, weight decay). Hyperparameter tuning for friction parameters $(\alpha,\beta)$ was conducted using a grid search on CIFAR-10 (see [Figure 15](#A6.F15)). We suspect that further experiments in that direction could greatly improve the efficiency of INNAprop. In particular, a dynamic scheduler for $(\alpha,\beta)$ could be very beneficial, but we were unable to explore this due to resource limitations.

For language models, INNAprop with $(\alpha,\beta)=(0.1,0.9)$ performs consistently well across all training durations, both for pre-training from scratch and for fine-tuning. In image classification, the same hyperparameter choice accelerates short-term learning, while higher values like $(\alpha,\beta)=(2.0,2.0)$ improve test accuracy during longer training runs. Moreover, $(\alpha,\beta)=(2.0,2.0)$ proves effective for fine-tuning, offering a good balance between convergence speed and final accuracy. These experiments illustrate consistent performances of the proposed method over a diversity of benchmarks, architecture, and model scales, making INNAprop a promising competitor for the training of large neural networks. Future research will be focused on the design of schedulers for the hyperparameters $\alpha$ and $\beta$.

## Acknowledgements

The authors acknowledge the support from the AI Interdisciplinary Institute ANITI (ANR-19-PI3A-0004), ANRT and Thales LAS France for Ryan B’s grant. Jérome B. and Edouard P. are supported by the Air Force Office of Scientific Research FA8655-22-1-7012, ANR Chess (ANR-17-EURE-0010), ANR ESRE (ANR-21-ESRE-0051) and ANR REGULIA. Access to MesoNET resources was granted under allocation m23038. We thank Nicolas Renon, Christophe Marteau, Laurent Cabanas for their advice on the Turpan HPC supercomputer, and Céline Parzani and Mélodie Angeletti for IT support.

## Appendix A Reminders of known optimization algorithms

Considering the problem in [Equation 1](#S1.E1) and setting $\nabla\mathcal{J}(\theta_{k})=g_{k}$, we outline several well-known update rule optimizers.

**Table 2: Update rules considered for known optimizers. SGD is due to (Robbins & Monro, ), Momentum to (Polyak, ), Nesterov to (Nesterov, ), RMSprop + Momentum to (Graves, ), Adam to (Kingma & Ba, ), NAdam to (Dozat, ) and INNA to (Castera et al., ).**
| $\text{SGD}(\gamma_{k})$ |
| --- |
| $\begin{aligned} \theta_{k+1}=\theta_{k}-\gamma_{k}g_{k}\end{aligned}$ |
| $\text{Adam}(\gamma_{k},\beta_{1},\beta_{2},\epsilon)$ |
| $\begin{aligned} &m_{0}=0\text{,}\,v_{0}=0\\<br>&m_{k+1}=\beta_{1}m_{k}+(1-\beta_{1})g_{k}&\\<br>&v_{k+1}=\beta_{2}v_{k}+(1-\beta_{2})g_{k}^{2}\\<br>&\theta_{k+1}=\theta_{k}-\gamma_{k}\frac{m_{k+1}}{\sqrt{v_{k+1}}+\epsilon}\end{aligned}$ |
| $\text{NAdam}(\gamma_{k},\psi,\beta_{1},\beta_{2},\epsilon)$ |
| $\begin{aligned} &m_{0}=0\text{,}\,v_{0}=0\\<br>&\mu_{k}=\beta_{1}(1-\frac{1}{2}0.96^{k\psi})\\<br>&m_{k+1}=\beta_{1}m_{k}+(1-\beta_{1})g_{k}\\<br>&v_{k+1}=\beta_{2}v_{k}+(1-\beta_{2})g_{k}^{2}\\<br>&\theta_{k+1}=\theta_{k}-\gamma_{k}\frac{\mu_{k+1}m_{k+1}+(1-\mu_{k})g_{k}}{\sqrt{v_{k+1}}+\epsilon}\end{aligned}$ |

## Appendix B Derivation of INNAprop from DIN

We consider ([9](#S2.E9)) which was a discretization of ([6](#S2.E6)), namely:

$$ $\displaystyle v_{k+1}=\sigma_{2}v_{k}+(1-\sigma_{2})g_{k}^{2}$ (10) $\displaystyle\frac{\theta_{k+1}-2\theta_{k}+\theta_{k-1}}{\gamma^{2}}+\alpha\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}}{\gamma}+\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}=0.$ (11) $$

This gives

$$ $\displaystyle\frac{1}{\gamma}\left(\left(\frac{\theta_{k+1}-\theta_{k}}{\gamma}+\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}\right)-\left(\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right)\right)=\;$ $\displaystyle-\alpha\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}$ $$

and thus

$$ $\displaystyle\frac{1}{\gamma}\left(\left(\frac{\theta_{k+1}-\theta_{k}}{\gamma}+\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}\right)-\left(\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right)\right)$ $\displaystyle=\;$ $\displaystyle\left(\frac{1}{\beta}-\alpha\right)\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\frac{1}{\beta}\left(\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right).$ $$

Multiplying by $\beta$, we obtain

$$ $\displaystyle\frac{1}{\gamma}\left(\left(\beta\frac{\theta_{k+1}-\theta_{k}}{\gamma}+\beta^{2}\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}\right)-\left(\beta\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta^{2}\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right)\right)$ $\displaystyle=\;$ $\displaystyle\left(1-\alpha\beta\right)\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\beta\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}$ $$

after rearranging all terms

$$ $\displaystyle\frac{1}{\gamma}\left(\left(\beta\frac{\theta_{k+1}-\theta_{k}}{\gamma}+\beta^{2}\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}+(\alpha\beta-1)\theta_{k}\right)-\left(\beta\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta^{2}\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}+(\alpha\beta-1)\theta_{k-1}\right)\right)$ $\displaystyle=\;$ $\displaystyle-\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\beta\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\>.$ $$

Setting $\psi_{k-1}=-\beta\frac{\theta_{k}-\theta_{k-1}}{\gamma}-\beta^{2}\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}-(\alpha\beta-1)\theta_{k-1}$, we obtain the recursion

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma_{2}v_{k}+(1-\sigma_{2})g_{k}^{2}$ (12) $\displaystyle\frac{\psi_{k}-\psi_{k-1}}{\gamma}$ $\displaystyle=-\frac{\psi_{k-1}}{\beta}-\left(\alpha-\frac{1}{\beta}\right)\theta_{k-1}$ (13) $\displaystyle\frac{\theta_{k+1}-\theta_{k}}{\gamma}$ $\displaystyle=\frac{-1}{\beta}\psi_{k}-\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\left(\alpha-\frac{1}{\beta}\right)\theta_{k}$ (14) $$

We can also rewrite the above as follows:

$$ $\displaystyle v_{k+1}=\sigma_{2}v_{k}+(1-\sigma_{2})g_{k}^{2}$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k},$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}\left(1+\gamma\left(\frac{1}{\beta}-\alpha\right)\right)-\frac{\gamma}{\beta}\psi_{k}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}.$ $$

We can save a memory slot by avoiding the storage of $\psi_{k}$:

$$ $\displaystyle\quad\psi_{k+1}=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k},$ (15) $\displaystyle\Leftrightarrow$ $\displaystyle\quad\psi_{k}=\frac{\beta}{\beta-\gamma}\left(\psi_{k+1}-\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}\right)=\frac{\beta}{\beta-\gamma}\psi_{k+1}-\frac{\beta}{\beta-\gamma}\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}$ $\displaystyle\quad\theta_{k+1}=\theta_{k}\left(1+\gamma\left(\frac{1}{\beta}-\alpha\right)\right)-\frac{\gamma}{\beta}\psi_{k}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $\displaystyle\qquad\quad=\theta_{k}+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}+\frac{\gamma}{\beta-\gamma}\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $\displaystyle\qquad\quad=\theta_{k}+\left(1+\frac{\gamma}{\beta-\gamma}\right)\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $\displaystyle\qquad\quad=\theta_{k}+\left(\frac{\beta}{\beta-\gamma}\right)\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $\displaystyle\qquad\quad=\theta_{k}+\left(\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $\displaystyle\qquad\quad=\left(1+\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ (16) $$

Finally, we merely need to use 3 memory slots having the underlying dimension size $p$:

$$ $\displaystyle v_{k+1}=\sigma_{2}v_{k}+(1-\sigma_{2})g_{k}^{2}$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k},$ $\displaystyle\theta_{k+1}$ $\displaystyle=\left(1+\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $$

Figure: Algorithm 2 INNAprop

### B.1 Equivalence between a special case of INNAprop and Adam without momentum

In this section, we demonstrate that INNAprop with $\alpha=1$ and $\beta=1$ is equivalent to Adam (Kingma & Ba, ) without momentum ($\beta_{1}=0$). To illustrate this, we analyze the update rules of both algorithms. We assume that the RMSprop parameter $\beta_{2}$ (for Adam) and $\sigma$ (for INNAprop) are equal. Starting with INNAprop, we initialize $\psi_{0}=(1-\alpha\beta)\theta_{0}$. For $\alpha=1$ and $\beta=1$, this simplifies to $\psi_{0}=0$. The update for $\psi$ becomes:

$$ $\displaystyle\psi_{k+1}$ $\displaystyle=\left(1-\frac{\gamma}{\beta}\right)\psi_{k}+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}=(1-\gamma)\psi_{k}$ $$

Given that $\psi_{0}=0$, it follows that $\psi_{k}=0$ for all $k$. The parameter update rule for INNAprop is:

$$ $\displaystyle\theta_{k+1}$ $\displaystyle=\left(1+\frac{\gamma(1-\alpha\beta)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $$

Replacing $\alpha=1$, $\beta=1$, and $\psi_{k}=0$, we get:

$$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\gamma\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $$

Here, $g_{k}$ is the gradient, and $v_{k+1}$ is the exponential moving average of the squared gradients:

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ $$

The Adam optimizer uses two moving averages, $m_{k}$ (momentum term) and $v_{k}$ (squared gradients):

$$ $\displaystyle m_{k}$ $\displaystyle=\beta_{1}m_{k-1}+(1-\beta_{1})g_{k}$ $\displaystyle v_{k}$ $\displaystyle=\sigma v_{k-1}+(1-\sigma)g_{k}^{2}$ $$

Setting $\beta_{1}=0$, the momentum term $m_{k}$ simplifies to $m_{k}=g_{k}$. The update rule becomes:

$$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\gamma\frac{g_{k}}{\sqrt{v_{k}}+\epsilon}$ $$

This matches the form of Adam’s update rule without the momentum term, confirming that INNAprop with $\alpha=1$ and $\beta=1$ is equivalent to Adam with $\beta_{1}=0$.

Figure: Algorithm 3 INNAprop with $(\alpha,\beta)=(1,1)$

## Appendix C Alternative discretizations

### C.1 An alternative derivation of INNAprop

As mentioned in [Remark 2](#Thmremark2), we can obtain INNAprop easily from INNA (Castera et al., ). The algorithm INNA writes (see [Table 2](#A1.T2)):

$$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}+\gamma_{k}\left((\frac{1}{\beta}-\alpha)\theta_{k}-\frac{1}{\beta}\psi_{k}\right)$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}+\gamma_{k}\left((\frac{1}{\beta}-\alpha)\theta_{k}-\frac{1}{\beta}\psi_{k}-\beta g_{k}\right)$ $$

Rearranging the terms and saving a memory slot — use $\psi_{k+1}$ in the second equation instead of $\psi_{k}$, (see [Equation 16](#A2.E16) for details)— yields

$$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}$ $\displaystyle\theta_{k+1}$ $\displaystyle=\left(1+\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta g_{k}$ $$

Now, use the RMSprop proxy directly within INNA. Using the usual RMSprop constants $\sigma\in[0,1]$ and $\epsilon>0$, we obtain:

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ $\displaystyle\psi_{k+1}$ $\displaystyle=\psi_{k}\left(1-\frac{\gamma}{\beta}\right)+\gamma\left(\frac{1}{\beta}-\alpha\right)\theta_{k}$ $\displaystyle\theta_{k+1}$ $\displaystyle=\left(1+\frac{\gamma(1-\beta\alpha)}{\beta-\gamma}\right)\theta_{k}-\frac{\gamma}{\beta-\gamma}\psi_{k+1}-\gamma\beta\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ $$

This is INNAprop and the derivation is much more direct, although less illustrative of the geometric features.

### C.2 A variant of INNAprop with momentum

##### The algorithm.

We follow the rationale behind the algorithm RMSprop  with momentum (Graves, ). We therefore start with [Equation 9](#S2.E9) using the RMSprop  proxy for the gradient:

$$ $\displaystyle v_{k+1}=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ $\displaystyle\frac{\theta_{k+1}-2\theta_{k}+\theta_{k-1}}{\gamma}+\alpha\frac{\theta_{k}-\theta_{k-1}}{\gamma}+\beta\frac{\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}}{\gamma}+\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}=0.$ $$

Rearranging terms, we have

$$ $\displaystyle v_{k+1}=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ $\displaystyle\theta_{k+1}=\theta_{k}+(1-\alpha\gamma)(\theta_{k}-\theta_{k-1})-\beta\gamma\left(\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right)-\gamma^{2}\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}$ $$

Let us introduce a momentum variable $m_{k}=\theta_{k-1}-\theta_{k}$ to obtain:

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ (17) $\displaystyle m_{k+1}$ $\displaystyle=(1-\alpha\gamma)m_{k}+\gamma^{2}\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}+\beta\gamma\left(\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}-\frac{g_{k-1}}{\sqrt{v_{k}}+\epsilon}\right)$ (18) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-m_{k+1}$ (19) $$

As previously need now to optimize the dynamics in terms of storage. For this we rewrite [Equation 18](#A3.E18) as

$$ $\displaystyle m_{k+1}=am_{k}+bg_{k}-c{g_{k-1}}.$ (20) $$

where $a=(1-\alpha\gamma)$, $b=\beta\gamma$ and $c=\gamma(\beta-\gamma)$.
Writing $\tilde{m}_{k}=m_{k}-\frac{c}{a}g_{k-1}$, we have

$$ $\displaystyle\tilde{m}_{k+1}$ $\displaystyle=m_{k+1}-\frac{c}{a}g_{k}$ $\displaystyle=am_{k}+bg_{k}-c{g_{k-1}}-\frac{c}{a}g_{k}$ $\displaystyle=a\left(m_{k}-\frac{c}{a}g_{k-1}\right)+\left(b-\frac{c}{a}\right)g_{k}$ $\displaystyle=a\tilde{m}_{k}+\left(b-\frac{c}{a}\right)g_{k}.$ $$

Therefore, using this identity, we may rewrite the following

$$ $\displaystyle m_{k+1}$ $\displaystyle=am_{k}+bg_{k}-c{g_{k-1}},$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-m_{k+1}$ $$

as

$$ $\displaystyle\tilde{m}_{k+1}$ $\displaystyle=a\tilde{m}_{k}+\left(b-\frac{c}{a}\right)g_{k},$ $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\tilde{m}_{k+1}-\frac{c}{a}g_{k}.$ $$

Recalling that $a=(1-\alpha\gamma)$, $b=\beta\gamma$ and $c=\gamma(\beta-\gamma)$. Finally, we get the following recursion which is an alternative way to integrate RMSprop to INNA:

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma v_{k}+(1-\sigma)g_{k}^{2}$ (21) $\displaystyle\tilde{m}_{k+1}$ $\displaystyle=(1-\alpha\gamma)\tilde{m}_{k}+\gamma^{2}\left(\frac{1-\alpha\beta}{1-\alpha\gamma}\right)\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ (22) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\tilde{m}_{k+1}-\frac{\gamma(\beta-\gamma)}{1-\alpha\gamma}\frac{g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ (23) $$

but as shown below through numerical experiments, the factor $\gamma^{2}$ is poorly scaled for 32 bits or lower machine precision.

##### Numerical experiments.

Using CIFAR-10 dataset, we train a VGG11 network with the momentum version of INNAprop with the hyperparameters $(\alpha,\beta)=(0.1,0.9$) above. We used a cosine annealing scheduler with $\gamma_{0}=10^{-3}$ and no weight decay. As seen in Figure [7](#A3.F7), the training loss stops decreasing between the 125th and 150th epochs. Upon closely examining the algorithm in this regime, we observe that at the end of training, $\gamma_{k}^{2}$ falls below the numerical precision, resulting in unstable behavior in [Equation 22](#A3.E22).

Figure: Figure 7: The version of INNA with momentum of Section [C.2](#A3.SS2) is an unstable method.
Refer to caption: /html/2410.05871/assets/x8.png

### C.3 An approach à la Adam

In this section, we mimic the process for deriving Adam from the heavy ball with a RMSprop  proxy, see, e.g., Kingma & Ba (); Ruder (), by simply replacing the heavy ball by DIN(^4^44Note that DIN with $\beta=0$ boils down to the heavy ball method.). We call this optimizer DINAdam.

From ([6](#S2.E6)), we infer the discretization:

$$ $\frac{\theta_{k+1}-2\theta_{k}+\theta_{k-1}}{\gamma^{2}}+\alpha\frac{\theta_{k+1}-\theta_{k}}{\gamma}+\beta\frac{g_{k}-g_{k-1}}{\gamma}+g_{k}=0.$ (24) $$

Rearranging terms, we have

$$ $\theta_{k+1}=\theta_{k}-\frac{\gamma^{2}}{1+\alpha\gamma}g_{k}+\frac{1}{1+\alpha\gamma}(\theta_{k}-\theta_{k-1})-\frac{\beta\gamma}{(1+\alpha\gamma)}(g_{k}-g_{k-1})$ (25) $$

By introducing the new variable $m_{k}=(\theta_{k-1}-\theta_{k})/\eta$ and setting $\eta>0$, we can rewrite equation ([25](#A3.E25)) as:

$$ $\displaystyle m_{k+1}$ $\displaystyle=\frac{1}{(1+\alpha\gamma)}m_{k}+\frac{\gamma^{2}}{(1+\alpha\gamma)\eta}g_{k}+\frac{\beta\gamma}{(1+\alpha\gamma)\eta}(g_{k}-g_{k-1})$ (26) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\eta m_{k+1}$ (27) $$

To follow the Adam spirit, we set $\sigma_{1}=\frac{1}{(1+\alpha\gamma)}$ and $(1-\sigma_{1})=\frac{\gamma^{2}}{(1+\alpha\gamma)\eta}$. Solving for $\gamma$, we get

$$ $\displaystyle\frac{\alpha\gamma}{1+\alpha\gamma}=\frac{\gamma^{2}}{(1+\alpha\gamma)\eta}\Rightarrow\gamma=\frac{\eta}{\alpha}$ $$

Then, we find the following recursion:

$$ $\displaystyle m_{k+1}$ $\displaystyle=\sigma_{1}m_{k}+(1-\sigma_{1})g_{k}+\beta\alpha\sigma_{1}(g_{k}-g_{k-1})$ (28) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\eta m_{k+1}$ (29) $$

From Equation ([28](#A3.E28)), we make a change of variable $\tilde{m_{k}}=m_{k}-\alpha\beta g_{k-1}$ to save a memory cell.

$$ $\displaystyle\tilde{m}_{k+1}$ $\displaystyle=\sigma_{1}\tilde{m}_{k}+(1-\sigma_{1}+\beta\alpha\sigma_{1}-\beta\alpha)g_{k}$ (30) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\eta(\tilde{m}_{k+1}-\alpha\beta g_{k})$ (31) $$

Using the usual RMSprop constants $\sigma_{2}\in[0,1]$ and $\epsilon>0$, we obtain:

$$ $\displaystyle v_{k+1}$ $\displaystyle=\sigma_{2}v_{k}+(1-\sigma_{2})g_{k}^{2}$ (32) $\displaystyle\tilde{m}_{k+1}$ $\displaystyle=\sigma_{1}\tilde{m}_{k}+(1-\sigma_{1}+\beta\alpha\sigma_{1}-\beta\alpha)g_{k}$ (33) $\displaystyle\theta_{k+1}$ $\displaystyle=\theta_{k}-\eta\frac{\tilde{m}_{k+1}-\alpha\beta g_{k}}{\sqrt{v_{k+1}}+\epsilon}$ (34) $$

Figure: Algorithm 4 DINAdam

###### Remark 5

The way RMSprop is added in INNAprop and DINAdam is different. In INNAprop, RMSprop is incorporated directly during the discretization process of Equation ([9](#S2.E9)) for all gradients. However, in DINAdam, RMSprop is added only at the last step, as shown in Equation ([32](#A3.E32)), and only on the gradient in the $\theta_{k+1}$ update. This is how RMSprop was combined with heavy ball to obtain Adam.

###### Remark 6

After setting $\alpha=1$ and $\beta=0$, we obtain Adam update rules. If $\beta\neq 0$, DINAdam is very close to NAdam algorithm. Hence, we did not investigate this algorithm numerically.

###### Remark 5

###### Remark 6

## Appendix D Scheduler procedures

Cosine annealing (Loshchilov & Hutter, ). Let $\gamma_{k}$ represent the learning rate at iteration $k$, $T_{\text{max}}$ be the maximum number of iterations (or epochs), and $\gamma_{\text{min}}$ be the minimum learning rate (default value is 0). The learning rate $\gamma_{k}$ at iteration $k$ is given by:

$$ $\gamma_{k}=\gamma_{\text{min}}+\frac{1}{2}(\gamma_{0}-\gamma_{\text{min}})\left(1+\cos\left(\frac{k}{T_{\text{max}}}\pi\right)\right)$ $$

This scheduler was employed in all image classification experiments except for ViT.

Cosine annealing with linear warmup (Radford et al., ). Let $\gamma_{k}$ represent the learning rate at iteration $k$, $\gamma_{\text{min}}$ the minimum learning rate, $\gamma_{0}$ the initial learning rate, $T_{\text{warmup}}$ the number of iterations for the warmup phase, and $T_{\text{decay}}$ the iteration number after which the learning rate decays to $\gamma_{\text{min}}$. The learning rate is defined as follows:

$$ $\gamma_{k}=\begin{cases}\gamma_{0}\cdot\frac{k}{T_{\text{warmup}}},&\text{if }k<T_{\text{warmup}}\\ \gamma_{\text{min}}+\frac{1}{2}\left(\gamma_{0}-\gamma_{\text{min}}\right)\left(1+\cos\left(\pi\cdot\frac{k-T_{\text{warmup}}}{T_{\text{decay}}-T_{\text{warmup}}}\right)\right),&\text{if }T_{\text{warmup}}\leq k\leq T_{\text{decay}}\\ \gamma_{\text{min}},&\text{if }k>T_{\text{decay}}\end{cases}$ $$

This scheduler was applied in experiments involving training GPT-2 from scratch and for ViT.

Linear schedule with linear warmup (Hu et al., ). Let $\gamma_{k}$ represent the learning rate at iteration $k$ and $T_{\text{max}}$ be the maximum number of iterations, $T_{\text{warmup}}$ be the number of warmup steps, and $\gamma_{\text{min}}$ be the minimum learning rate after warmup (default value is typically set to the initial learning rate, $\gamma_{0}$). The learning rate $\gamma_{k}$ at iteration $k$ is given by:

$$ $\gamma_{k}=\begin{cases}\gamma_{0}\cdot\frac{k}{T_{\text{warmup}}}&\text{if }k<T_{\text{warmup}},\\ \gamma_{0}\cdot\left(1-\frac{k-T_{\text{warmup}}}{T_{\text{max}}-T_{\text{warmup}}}\right)&\text{otherwise}.\end{cases}$ $$

This scheduler was used for fine-tuning GPT-2 with LoRA.

## Appendix E Choosing hyperparameters α 𝛼 \alpha and β 𝛽 \beta for INNAprop

### E.1 Comparison with AdamW

For VGG and ResNet training on CIFAR10, the literature suggest using initial learning rate $\gamma_{0}=10^{-3}$ with a learning rate schedule (Mishchenko & Defazio, ; Defazio & Mishchenko, ; Yao et al., ; Zhuang et al., ). Our experiment fix a cosine scheduler where $T_{\text{max}}=200$ and $\gamma_{\min}=0$ as it achieves a strong baseline for AdamW (Loshchilov & Hutter, ; Mishchenko & Defazio, ). We set weight decay $\lambda=0.1$. Then, we tune the initial learning rate $\gamma_{0}$ among $\{10^{-4},5\times 10^{-4},10^{-3},5\times 10^{-3},10^{-2}\}$. In Figure [8](#A5.F8), we report the performance in
terms of training loss and test accuracy for AdamW. These results confirm the usage of $\gamma_{0}=10^{-3}$.

Figure: (a) Performance rankings with VGG11.

## Appendix F Additional experiments

### F.1 CIFAR10 experiments

Figure: Figure 9: Training ResNet18 on CIFAR10. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 8 random seeds.
Refer to caption: /html/2410.05871/assets/x9.png

### F.2 Food101 experiments

Figure: Figure 10: Finetuning a ResNet18 on Food101, same as [Figure 4](#S3.F4) for ResNet18. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds.
Refer to caption: /html/2410.05871/assets/x10.png

### F.3 ImageNet

Figure: Figure 11: Training ResNet18 on ImageNet. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds.
Refer to caption: /html/2410.05871/assets/x11.png

Figure: Figure 12: Fast training ViT/B-32 on ImageNet with weight decay $\lambda=0.01$ for INNAprop ($\alpha,\beta)=(0.1,0.9)$. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds.
Refer to caption: /html/2410.05871/assets/x12.png

### F.4 Heatmap for preliminary tuning of α 𝛼 \alpha and β 𝛽 \beta

Figure: (a) 20 epochs
Refer to caption: /html/2410.05871/assets/x13.png

Figure: (a) 20 epochs
Refer to caption: /html/2410.05871/assets/x15.png

Figure: (a) 20 epochs
Refer to caption: /html/2410.05871/assets/x17.png

## Appendix G Experimental Setup

### G.1 CIFAR-10

We used custom training code based on the PyTorch tutorial code for this problem. Following standard data-augmentation practices, we applied random horizontal flips and random offset cropping down to 32x32, using reflection padding of 4 pixels. Input pixel data was normalized by centering around 0.5.

| Hyper-parameter | Value |
| --- | --- |
| Architecture | VGG11 and ResNet18 |
| Epochs | 200 |
| GPUs | 1$\times$V100 |
| Batch size per GPU | 256 |
| Baseline LR | 0.001 |
| Seeds | 8 runs |

| Hyper-parameter | Value |
| --- | --- |
| Baseline schedule | cosine |
| Weight decay $\lambda$ | 0.01 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.999 |
| $\sigma$ (for INNAprop) | 0.999 |

### G.2 Food101

We used the pre-trained models available on PyTorch for VGG11 and ResNet18.(^5^55[https://pytorch.org/vision/stable/models.html](https://pytorch.org/vision/stable/models.html)).

| Hyper-parameter | Value |
| --- | --- |
| Architecture | VGG11 and ResNet18 |
| Epochs | 200 |
| GPUs | 1$\times$V100 |
| Batch size per GPU | 256 |
| Baseline LR | 0.001 |
| Seeds | 3 runs |

| Hyper-parameter | Value |
| --- | --- |
| Baseline schedule | cosine |
| Weight decay $\lambda$ | 0.01 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.999 |
| $\sigma$ (for INNAprop) | 0.999 |

### G.3 ImageNet

We used the same code-base as for our CIFAR-10 experiments, and applied the same preprocessing procedure. The data-augmentations consisted of PyTorch’s RandomResizedCrop, cropping to 224x224 followed by random horizontal flips.
Test images used a fixed resize to 256x256 followed by a center crop to 224x224.

#### G.3.1 ResNet18

| Hyper-parameter | Value |
| --- | --- |
| Architecture | ResNet18 |
| Epochs | 90 |
| GPUs | 4$\times$V100 |
| Batch size per GPU | 64 |
| Baseline LR | 0.001 |
| Seeds | 3 runs |

| Hyper-parameter | Value |
| --- | --- |
| Baseline schedule | cosine |
| Weight decay $\lambda$ | 0.01 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.999 |
| $\sigma$ (for INNAprop) | 0.999 |

#### G.3.2 ResNet50

| Hyper-parameter | Value |
| --- | --- |
| Architecture | ResNet18 |
| Epochs | 90 |
| GPUs | 4$\times$V100 |
| Batch size per GPU | 64 |
| Baseline LR | 0.001 |
| Mixed precision | True |
| Seeds | 3 runs |

| Hyper-parameter | Value |
| --- | --- |
| Baseline schedule | cosine |
| Weight decay $\lambda$ | 0.1 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.999 |
| $\sigma$ (for INNAprop) | 0.999 |

#### G.3.3 ViT/B-32

| Hyper-parameter | Value |
| --- | --- |
| Architecture | ViT/B-32 |
| Epochs | 300 |
| GPUs | 8$\times$A100 |
| Batch size per GPU | 128 |
| Baseline LR | 0.001 |
| Seeds | 5000 |

| Hyper-parameter | Value |
| --- | --- |
| Baseline schedule | cosine |
| Warmup | linear for 30 epochs |
| Weight decay $\lambda$ | 0.1 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.999 |
| $\sigma$ (for INNAprop) | 0.999 |

### G.4 GPT2 from scratch

We followed the NanoGPT codebase (^6^66[https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)) and we refer to (Brown et al., ) as closely as possible, matching the default batch-size and schedule.

| Hyper-parameter | Value |
| --- | --- |
| Architecture | GPT-2 |
| Batch size per gpu | 12 |
| Max Iters | 100000 |
| GPUs | 8$\times$A100 |
| Dropout | 0.0 |
| Baseline LR | refer to (Brown et al., ) |
| Warmup Steps | 500 |

| Hyper-parameter | Value |
| --- | --- |
| Seeds | 5000 |
| Weight decay $\lambda$ | 0.1 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.95 |
| $\sigma$ (for INNAprop) | 0.99 |
| Gradient Clipping | 1.0 |
| Float16 | True |

### G.5 GPT-2 with LoRA

We followed the LoRA codebase (^7^77[https://github.com/microsoft/LoRA](https://github.com/microsoft/LoRA)) and we refer to (Hu et al., ) as closely as possible, matching the default batch-size, training length, and schedule. We train all of our GPT-2 models using AdamW (Loshchilov & Hutter, ) and INNAprop on E2E dataset with a linear learning rate schedule for 5 epochs. We report the mean result over 3 random seeds; the result for each run is taken from the best epoch.

| Hyper-parameter | Value |
| --- | --- |
| Architecture | GPT-2 |
| Batch size per gpu | 8 |
| Epochs | 5 |
| GPUs | 1$\times$A100 |
| Dropout | 0.1 |
| Baseline LR | 0.0002 |
| Warmup steps | 500 |

| Hyper-parameter | Value |
| --- | --- |
| Seeds | 3 runs |
| Weight decay $\lambda$ | 0.01 |
| $\beta_{1},\beta_{2}$ (for AdamW) | 0.9, 0.98 |
| $\sigma$ (for INNAprop) | 0.98 |
| Learning Rate Schedule | Linear |
| LoRA $\alpha$ | 32 |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: The derivation of the algorithm is clear and the performance often seems competitive with AdamW. Large-scale models such as ViT-B/32 and GPT-2 were trained.
summary: This paper introduces INNAprop, a combination of the INNA optimizer with RMSprop gradient scaling. The resulting method has memory requirements similar to methods such as AdamW. Experiments show that this optimizer is competitive with AdamW.
weaknesses: The INNAprop algorithm seems like a relatively straightforward extension from INNA. From what I can tell Algorithm 2 can be derived directly from the INNA equations in Table 2 and adding the factor $\\frac{1}{\\sqrt{v_{k+1}} + \\varepsilon}$ to the gradient. The motivation from a continuous dynamical systems perspective and the derivation done in appendix B seem equivalent to those from the INNA paper. This means that the theoretical insights from this paper seem limited.

This in itself wouldn't be a bad thing if the paper could make strong experimental claims about this small tweak to INNA leading to consistently stronger performance. However, this seems difficult to conclude for a few reasons, such as:

* No comparisons were made with INNA (without the RMSprop scaling) or other algorithms (Adam, ADAGRAD, SGD)
* In figures 3 and 6 INNAprop is sometimes indistinguishable from AdamW

Another surprise is that the optimal values for $\\alpha$ and $\\beta$ seem quite different to those found in the INNA paper (which ends up using $(0.1, 0.1)$ and $(0.5, 0.1)$) and it would be great to understand why. (In general, I am not convinced that the comparison with AdamW was the way to go, given that the proposed algorithm is far more similar to INNA. It would be insightful to know how the RMSprop extension changes INNA's behaviour and the choice of optimal hyperparameters for INNA.)

I understand that with limited compute it can be difficult to run large amounts of experiments, but I don't think that in it's current form this paper really provides new insights to the community: The RMSprop addition to INNA is pretty straightforward, and whether it really makes a difference is still hard to say.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Can you explain how you "systematically favored AdamW through the choice of recommended hyperparameters (scheduler, learning rates, weight decay)"? In more established tasks, I feel this is a reasonable claim. However, where INNAprop showed wins (finetuning on an uncommon dataset and training tiny GPT-2 models), I expect that standard AdamW hyperparameters are suboptimal. How much compute budget was spent tuning AdamW compared to INNAprop? I am also concerned that two different settings of INNAprop are shown in all experiments against a single choice of hyperparameters for AdamW.

In the appendix, you derived the DINAdam variant. Do you also explore this algorithm empirically?

With my initial review, I am recommending that the paper be rejected. I believe the paper is technically correct and the proposed algorithm is sufficiently novel. However, I do not believe that the algorithm provides a meaningful practical contribution for the wider community at this time. My concern is that a practitioner would always choose AdamW over INNAprop due to the marginal expected gains, increased risk of failure, and the additional overhead of parameter tuning required. My score would change if the authors could provide clear guidance on when a practitioner would choose INNAprop over AdamW and expect a win. 

Minor points:

- "we provide quite extensive experiments" vague statement in abstract. I would prefer that this be made more precise.
- "extra second-order geometrical intelligence" vague technical term. What do you mean by this? I think this should be clarified in the main text.

-------
**Post-rebuttal:**

Following the response from the authors, I have increased my score (and reduced my confidence). This is due primarily to the author's point that several results that I considered to be minimal improvements are in fact significant relative to the current state-of-the-art.

I retain concerns that the proposed algorithm is unlikely to see widespread community adoption. However, the paper is sound and proposes a novel technique. I would prefer that we present novel and correct methods to the community for their consideration.
soundness: 4
strengths: I thought the introduction was strong and successfully explained the need for improved optimization algorithm algorithms in terms of real-world impact. The authors back this up by deriving a memory-efficient version of their algorithm matching the memory overhead of existing approaches.

A thorough discussion of the algorithm and how it can be derived is provided. The authors also discuss comparisons between INNAprop and several other methods in the literature, in terms of its derivation. This is a valuable contribution that helps to place the proposed method among existing work. On the surface, I felt that the combination of the two methods could be of limited novelty. However, after reviewing the derivation I feel this constitutes a reasonably strong contribution.

The paper is clear and well-written. The authors provide a thorough set of diverse experiments to investigate the proposed method and include multiple runs to account for variability. From my review of the manuscript, all results look sensible and technically sound.
summary: This paper proposes a new first-order optimization algorithm: INNAprop. The algorithm combines the INNA method with RMSProp to derive a novel optimization scheme. The authors evaluate INNAprop empirically against AdamW, the de facto standard deep learning optimizer. They explore a set of diverse tasks spanning image classification and natural language processing using several different classes of deep learning architectures.
weaknesses: Claiming second-order without computing second-order statistics doesn't seem accurate to me. The finite difference approximation may not be sufficiently accurate to achieve these benefits. However, I am not overly familiar with this approximation that the authors adopt from prior work. One way to alleviate this concern would be to provide additional justification for this terminology. Or, to include empirical evidence that the finite-difference approach is providing a good approximation of the actual second order statistics.

One of the claimed contributions is that INNAprop "matches or outperforms AdamW in both training speed and final accuracy on benchmarks". However, Fig. 2 shows the faster training speed of AdamW (the final error bars overlap for test accuracy so I'd agree that they match performance). Fig. 3 shows INNAprop as faster to converge for ResNet, but slower once again for ViT. The test accuracy is once again matched in both experiments. Fig. 4 shows a similar convergence speed with a modest (~1%) increase in accuracy for INNAprop. Finally, Fig. 5 shows a modest win for INNAprop on the smaller settings and matched performance for the larger model. For the LoRA finetuning (Fig. 6), there is a small win for the small setting and INNAprop matches AdamW on the medium and large setting.

The improvement over Adam variants is small across the board. The notable wins are on vision finetuning (Fig. 4) and small-scale GPT-2 training (Fig. 5). There are many methods that propose similarly slim margins and none are adopted by the community. Furthermore, under scrutiny from third-party implementations, the gains are often difficult to replicate [1]. Sadly, my honest opinion is that this method will not gain traction with the community --- it is unclear when it should be used over AdamW.

While often spurious, it is common for optimization algorithms to include theoretical analysis of their convergence. I do not consider this a critical weakness of this paper.

[1] "Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers", Schmidt et al.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: I noticed that you mentioned the lack of computational resources two times (lines 259 and 359). You should know, that I am very sympathetic to that challenge. Still, my suggestion is that you bite the apple and invest that extra time and computational cost --- to really make a convincing case for your method :-)
soundness: 2
strengths: It is a comprehensive and well-written paper that does a good job of explaining the motivation for, and the math behind, the proposed method. I do appreciate the main thesis of the work, and the effort that went into it. I like the discussion of, and approach to, hyperparameter tuning, as well as the comprehensive Appendix.
summary: The authors introduce a second-order optimization method, INNAprop, which combines the INNA method with RMSprop, while not increasing the memory-footprint. The method is evaluated on a few image classification benchmarks, and GPT-2 with the OpenWebText dataset, and directly compared to AdamW with somewhat favorable results.
weaknesses: My main objection is the empirical evaluation. Firstly, the reported improvements (on test sets) are generally very marginal. Secondly, the results are not pushed to SOTA-level --- even for the chosen architectures (that are pretty old and pretty small). Thirdly, there are too few experiments to convince me that the proposed method really works better than, or even as well as, AdamW. Since the method is not faster, or less computationally demanding, I find myself unconvinced of its overall usefulness.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: -   How does INNAprop compare in terms of wall-clock time to AdamW or Sophia?
-   How does INNAprop compare to Sophia/Lion and other more recent high performing optimizers?
-   Is the increase in test accuracy of INNAprop compared to AdamW statistically significant? What are the error ranges?
soundness: 2
strengths: The paper is well written, and easy to follow.
-    The derivation of the algorithm is well motivated.
-    They leverages second order information without requiring the hessian.
-    The algorithm seems to perform on par or even outperform AdamW in certain settings like language modeling.
-    The experiments cover a good amount of tasks between image classification and language modeling.
-    The authors open-source the code.
summary: The paper combines RMSprop adaptive gradient scaling to the existing INNA optimizer, which is based on the Dynamical Inertial Newton. The obtained optimizer leverages second-order information while having similar computational requirements to first-order methods. With appropriate hyper-parameters the optimizer matches and in some cases outperforms AdamW.
weaknesses: -   The introduction of the paper makes a big deal about the energy consumption of training large language models, but the authors don't measure or show how their method compares to the competitors in terms of wall-clock time in their experiments. Particularly interesting would be seeing how wall-clock time on ImageNet or language modeling tasks compare.
-   The method lacks comparison with more recent optimizers other than AdamW. Sophia for example is another fast and efficient second-order optimizer paper, which the author cite, but don't compare with. Lion is another recent and high performing optimizer, which was cited but not compared with either.
-   Numeric results are reported without standard deviation/standard error or confidence intervals

Meta-review: This paper studies a new optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. The resulting method has memory and computational requirements similar to AdamW.  The paper demonstrates that this algorithm performs somewhat better than AdamW algorithm.

The reviews for the paper are slightly negative. The main concern of the reviewers is limited novelty and weak empirical analysis. I agree with the reviewers that the novelty is somewhat limited. The authors need to at least provide more solid empirical analysis to justify the claim of outperforming AdamW.  The presentation of the paper is poor and can also be improved. I also think it is worth comparing the performance of algorithm with approaches like Shampoo (even though these approaches are expensive, it is worth comparing the quality-computation tradeoff). For these reasons, I recommend rejection.

Decision: Reject