A second-order-like optimizer with adaptive gradient scaling for deep learning
Authors: Jerome Bolte
Ryan Boustany
Edouard Pauwels
Andrei Purica
Venue: Accepted by TMLR ()

Abstract: In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second order information and rescaling while keeping the memory and compute requirements of standard DL methods as AdamW or SGD. INNAprop is evaluated on CIFAR-10, Food101, and ImageNet with ResNets, VGG, DenseNet, and ViT. We also train GPT-2 (OpenWebText) from scratch and with LoRA fine-tuning (E2E). INNAprop consistently offers close performance to AdamW, while performing significantly better in our LLM training experiments, achieving faster convergence and higher accuracy with minimal hyperparameter tuning, even at large scale. Our code is public.

Published in Transactions on Machine Learning Research (11/2025)
A second-order-like optimizer with adaptive gradient scaling for deep learning
Jérôme Bolte
Bewmme.bolt@@@ut-ooptooo....
Toulouse School of Economics, University of Toulouse Capitale, Toulouse, Pronce
Ryan Boustany
Toulouse School of Economics, University of Toulouse Copitole, Thales LAS France
Edouard Pauwels
Toulouse School of Economics, University of Toulouse Copitole, Toulouse, Pronce
Andrei Purica
andrei.puricc@@@uleesrroo..m
Thales LAS France
Reviewed on OpenReview: https: // openreview. net/ forum? id= 3khttitDDDD
Abstract
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. Tt leverages second order information and rescaling while keeping the memory and compute requirements of standard DL methods as AdamW or SGD. INNAprop is evaluated on CIFAR-10, Food101, and Ima- geNet with ResNets, VGG, DenseNet, and ViT. We also train GPT-2 (OpenWebText) from scratch and with LoRA fine-tuning (E2E). INNAprop consistently offers close performance to AdainW while performing significantly better our LLM training experiments, achieving faster convergence and higher accuracy with minimal hyperparameter tuning, even at large scale. Our code is public
1 Introduction
As deep learning models grow in size, massive computational resources needed for training, representing significant challenges in terms of financial costs, energy consumption, and processing time (Susnjak et al. Varoquaux et al. 2024). According to the UN's Environment Programme Training, the Big Tech sector produced between two and three percent of the world's carbon emissions in 2021; some estimations for the year 2023 go beyond 4%, see the latest Stand.earth reports, and also (Schwartz et al., 2020 Strubell et al.) Patterson al. 2021 for related issues. For instance, training GPT-3 estimated to require 1,287 megawatt-hours (MWh) of electricity, which is equivalent to the annual usage of over 100 U.S. households
2024
2020
Anthony al. 2020 Patterson et al.) 2021). OpenAI claimed that the training cost for GPT-4 (Achiam)
et al. 2023) exceeded 100 million dollars. The PaLM model developed by Google AI was trained for two months using 6144 TPUs for 10 million dollars (Chowdhery et al.) 2023). This calls for faster and more cost-efficient optimization algorithms. also suggests that early stopping (Prechelt) 2002 Bai et al. 2021 in the training phase desirable feature whenever possible.
https://github. com/ http:///wwccninnncc/eaaaa inaprop/tnnappop

---

Published in Transactions on Machine Learning Research (11/2025)
We focus in this work the computational efficiency during the training phase and consider the problem of unconstrained minimization of loss function J: R R, as follows
min J(0). DERP
(1)
Continuous dynamical systems as optimization models. To achieve higher efficiency, a useful way is
to view algorithms as discrete versions of contimuous dynamical systems (Ljung 1977p, further developed in
(Harold al. 1997 Benaïm 2006 Borkar Borkar 2008 Attouch et al. 2016 Aujol et al. 2019). In deep
learning. this approach is also quite fruitfull; it has, in particular, been used to provide convergence proofs or
further geometrie insights (Davis et al. 2020: Bolte & Pauwels) 2020 Barakat & Bianchi 2021 Chen el. al.
2023a).
In the spirit of Castera et al. [2021) we consider the following contimuous-time dynamical system introduced in Alvarez et al. (2002) and referred to as DIN (standing for "dynamical inertial Newton"):
o(t)
a(tt)
+ + + VJ((()) =
(DIN)
Inertial term
Friction term
Newtonian effects
Gravity effect
where is the time, J: R - R is loss function to be minimized (e.g., empirical loss in DL applications) in Equation 1 assumed C with gradient V and Hessian V A key aspect of Equation (DIN) that
places it between first- and second order optimization is that a change of variables allows to describe it using only the gradient VJ, since (O(((())) computational costs, as it can be discretized as a difference of gradients which does not require Hessian vector product, making possible design more practical algorithms, as shown in Chen & Luo (2019): Castera et al. 2021 Attouch al. (2022).
कi (O(t)) (see Section 2.2 for details). This greatly reduces
We recover the contimuous-time heavy ball system by assuming a > and removing the geometrical "damping" term in Equation (DIN) through the choice / = 0. A discrete version of this system corresponds to the Heavy
Ball method (Polyak 1964), which is at the basis of SGD solvers with momentum in deep learning (Qian
1999
Sutskever et. al.
2013). By allowing both a and / to vary, we recover Nesterov acceleration (Nesterov
1983
Su et al. 2016
Attouch et al. 2019).
Adaptive methods. Adaptive optimization methods, such as RMSprop Tieleman et al. 2012) and AdaGrad (Duchi et al. 2011), use coordinate- wise scaling to stabilize the system and improve training. These methods can be modeled by contimuous-time ODEs of the following form:
0(1) VG(1) +
O VJ(O(() = 0, 12 0,
(2)
where e > O, G(I) E RP represents accumulated information. The scalar addition, square root, and division are understood coordinatewise and @ denotes the coordinate-wis product for vectors in Rº. In AdaGrad or RMSprop, G(t) is a gradient amplitude average of the form:
G(I== = J ((((()))24u(7))
G(I)
(3)
for different choices of 14 uniform for AdaGrad and moving average for RMSprop, where the square operation applies coordinatewise. This generally improves performance, see (Duchi et al.
2011 Tieleman
et al. 2012).
Our approach. We combine the *dynamical ial Newton" method (DIN) from Equation DIN with an RMSprop adaptive gradient scaling. We thus take into account second order information for the RMSprop scaling. Computationally, this sccond order information is expressed using a time derivative. In discrete time, this will result in second order like behavior with the same computational cost as gradient evaluation. The resulting continuous time ODE is:
2

---

Published in Transactions on Machine Learning Research (11/2025)
d (RMSprop(7(00)) + RMSprop((((())) = 0
0(1) + aa(() + 3
(4)
dt
where (MS))) (G(L (.(((t)).
Here, G is of the form with an adequate time-weight distribution 14 corresponding to the RMSprop scaling. A discretization of this contimuous time system, results in our new optimizer INNAprop, see Section 2.1 Let us emphasize that the system
(4) is the superposition of three dynamical systems: the gradient flow, the heavy ball flow, and the continuous Newton flow Alvarez D & Pérez C (1998). at Vf(((t)++
v f(0(t))( -RM (M =
f(((()) = 0. When adapted to the RMSprop setting it becomes di RMMSpopp( (B(t))
0. It is easily checked that this this system drives RMSpropp[][tt (O(t)) to zero exponentially fast as in Alvarez D) & Pérez C (1998). We use the term "second order information" " to reflect the fact that the underlying continuous time system features second order derivatives of the cost inducing fast convergence. Note that the resulting algorithm remains first order method, based only on gradient oracles, this is a key aspect of our approach motivated by deep learning applications.
Relation with existing work. To improve the efficiency of stochastic gradient descent (SGD), two primary strategies are used: leverage local geometry for having clever directions and incorporate momentum to accelerate convergence. These approaches include accelerated methods (e.g., Nesterov's acceleration
[Nesterov] 1983 Dozat, 2016). momentum SGD (Polyak) 1964 Qian 1999; Sutskever ct al.) 2013). and
adaptive methods (e.g., Adagrad (Duchi al. 2011 RMSprop (Tieleman al. 2012 which adjust.
learning rates per parameter.
Adam remains the dominant optimizer with numerous variants proposed to improve its performance or better
suit specific cases (Dozal, 2016 Shazeer & Stern
2018: Reddi et al., 2019 Loshchilov & Hutter 2017 Zhuang
et al. 2020). Adafactor (Shazeer & Stern 2018 improves memory efficiency, Lamb (You et al., 2019 adds
layerwise normalization, and Lion (Chen et al., 2023b) uses sign-based momentum updates. AdEMAMix
PPallardiii et al.] 2024) combines two EMAs, while Defazio al. (Defazio et al., 2024] introduced a schedule-free method incorporating Polyak-Ruppert averaging with momentum.
One of the motivations of our work is the introduction of second order properties in the dynamics akin to Newton's method. second order optimizers are computationally expensive due to frequent Hessian computations (Gupta et al.] 20188] Martens & Grosse 2015). Their adaptation to large scale learning settings
require specific developments (Jahani et al. 2021 Qian et al. 2021 Liu et al. 2023
Contributions.
Our method, LNNAprop, integrates the algorithm INNA (Castera al.) 2021), which features com- putationally cheap Newtonian effect with the gradient scaling mechanism of RMSprop. We improve the memory footprint of INNA by saving memory slot. Computational costs are similar to first order methods like AdamW, making INNAprop suitable for deep learning (see Section 2.2 and Appendix B Specific hyperparameter choices for our method allow us to recover several existing optimizers as special cases. explanation of INNAprop, connecting it to second order ordinary differential
We provide a contimuous-timg equations (see Section and Equation (4)). We discuss many natural possible discretizations and show that INNAprop empirically the most efficient. We show that second order terms in space may be approximatod without resorting to Hessian computations inversions of linear systems which are both prohibitive in deep learning.
We show through extensive experiments that INNAprop offers close performances to AdamW and sometimes significantly better in both training speed and final accuracy on benchmarks such as image classification (C1FAR-10, ImageNet) and language modeling (GPT-2) (see Section 3 for which we also consider the Muon optimizer.
We describe our algorithm and its derivation in Section Hyperparameter tuning recommendations and our experimental results provided in Section
3

---

Published in Transactions on Machine Learning Research (11/2025)
2 INNoprop: second order method in space and time based on RMSprop
2.1 The algorithm
Algorithm D which blends INNA and RMSprop, is the method we used in all experiments. It includes the usual ingredients of deep-learning training: mini-batching, decoupled weight-decay, and scheduler procedure. For a simpler, "non-deep learning" version, refer to Algorithm in Appendix
Algorithm Deep learning implementation of INNAprop
Loss function: [() J Σm (0) for 0 E RP.
2: Learning step-sizes: 7/k == (SetLarchhcuule()))err
To is called the initial learning rate.
3: Hyper-parameters: a E [0, 1], 2 0, 3 > suPREN TR, 2 0, 6 = 10-8
4: Mini-batches: (B*) C. {I,...,NY N chosen with usual protocols, see e.g. Krizhevsky et al. (2012)
Initialization: time step < 0, parameter vector 8o. %b = 0, 4/o (1 a,3)0
for k to K do B& EneB, (Ox)
It: _
8: T t SetLrSchedule(())
see above
f₂ +- (1 \7k) 0²
decoupled weight decay
10: 1 oun (1 o)
11: 0 Uk+1/ / ot)
12: 1141 f 1 - 2k
Tik () -aa) a) On
13: 0k+1 1 2=(1-08)
02: 8-70 shat1 7&B (gkl(V/kt1 +c)
B-nk
14: return 0k+1
SetLrSchedule is the "scheduler' for step-sizes. It is custom procedure for handling learning rate sequences: detailed explanations of several procedures along with the corresponding benchmarks in Section may be found in Appendix D
Observe that, for schedulers 7a < B for E N, so that INNAprop is well-posed (see line 13 in Algorithm I
2.2 Derivation of the algorithm
We start with the INNA optimizer of Castera et al. (2021) without minibatch, which is an Euler discretization of the system [IN]. Introducing, for each is given in the phase space form as as follows
step-size 7t and an estimate Ok for (J(a), the algorithm
c 1/x (1-
+ 7h
1 -a)ap
3--)0e Ok.
Bx+1 Ax 1 [1+7 ())
2/4 Bg
(INNA)
A direct modification of the gradient estimate gu allows to include the RMSprop mechanism.
Uk+11 = o4k + (1 - o'go
1 7/
1h+1 = 4h
+ 7k
Ok:
(INNAprop-O)
8
3
1
7k
Oc+1 0k
1 + 7t
th
8
3
/uk+1 +
The final version of the recursion in Algorithm 1 the following features:
is variation of the basc recursion in [NNAprop-O] with
bias correction step for the RMSprop term, decoupled weight decay and learning rate schedulers. These typical in modern deep learning optimizers.
4

---

Published in Transactions on Machine Learning Research (11/2025)
An update form equivalent to [NNApop--)] which allows for naive implementation with in place updates for the variables 0 and v, contrary to (INNAprop-O), see Appendix B.1 for more details.
A few remarks are in order.
Remark 1 (On memory savings) The formulation of ANNop used in Algorithm [ directly allows for in place updates, potentially saving memory. Alternatively careful implementation of the original form of INNAprop-O), the use of additional computational devices, such compilers, could possibly lead to similar memory savings.
Remark 2 (A family of algorithms indexed by 8) INNAprop is family of methods indexed by the hyperparameters a and B, with an interesting special case. For a = 3 = 1, INNAprop with its default initialization, boils down to AdamW without momentum (31 = 0), see Appendix B.3 and Table 5 This correspondence is confirmed empirically in Figure 9 and Figure S in Appendix [B.3 for a VGG and a ResNet model. Regarding the parameter a in Algorithm | line 12 and 13, it may be interpreted
having an effect line While this intuition makes sense, varying À only affects
that is similar to uncoupled weight decay the 0 variable while a affects both 0 and v. We confirm in the experimental section that the parameter a plays a role which goes beyond that of a hidden uncoupled weight decay parameter.
Remark 3 (On other possible discretizations) It can be checked that the basic recursion [LNNAprop-D can be interpreted as an Euler discretization of the continuous dynamics 4 see Appendix B.2 for more details. We mention for completeness different attempts at discretizing the continuous time system in resulting two algorithmic schemes which were not considered for further evaluation.
(a) RMSprop with momentum (Graves] 2013) is obtained by discretization of the heavy ball continuous time system, using an RMSprop proxy instead of the gradient. It would be natural to proceed similarly in our case with the system (4) instead of the heavy ball system. This leads to a different method (see Appendix
C.1 which appears to be numerically unstable on early experiments (see Figure 10 for an illustration).
(c) For Adam, the RMSprop mechanism is applied not in the momentum term but only in the update, see
4 leads to a third method, detailed in Appendix C.2 We did not push the invostigations for this method as essentially amounts to run the NAdam algorithm Dozat 2016
Table 5 Doing similarly with the system
for different syper-parameles choices (see Appendix
C.2 for details).
3 Empirical evaluation of INNAprop
We conduct exlensive comparisons of the proposed algorithm and the AdamW optimizer, which dominantly used in image classification (Chen et al.) 20188] Zhuang et al.) 2020 Touvron et al. 2021 Mishchenko & Defazio 2023) and language modeling tasks (Brown al.) 2020 Hu et al. 2021 Liu et al. 2023). Hyperparameter tuning (Sivaprasad et al., 2020 is a crucial issue for this comparison, and we start with this. As a general rule, we strive to choose the lyperparameters that give a strong baseline for AdamW (based on literature or using grid search). In particular, most of our experiments use the AdamW optimizer 2 with its default settings as defined in widcly-used librarics (Paszke et al.) 2019 Bradbury et al. 2018 Abadi et al.] 2016): 31 = 0.9, 82 = 0.999, x = 0.01 and e == 10-8, For INNApr, unless otheewise specified, the default settings for the RMSprop component align with those of AdamW: a = 0.999 and c <= 10-8
The INNAprop method and the AdamW optimizer involve different classes of hyperparameters; some of them are common to both algorithms, and some are specific. Our hyperparameter tuning strategy for both algorithms is summarized in Table D AdamW is favored whenever possible. We emphasize that AdamW, which is regarded as one of the if not the state-of-the-art algorithm, has benefited from a long history of extensive testing by a large research community. Favoring AdamW through robust, setups provides this method with a considerable advantage. It should also be noted, that the notion of 'state of the art' docs not necessarily correspond to an absolute optimum, as performance may depend on specific implementations and hardware configurations which not an aspect we take into account presently.
caammuity-vvllddttee
htteee/r//ypoocccccocrrrrrcltlse Adamp.........a........am.m..... Saamm.hte html
5

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Hyperparameter tuning strategy for INNAprop and AdamW.
Parameters
AdamW tuning
INNAprop tuning
Advantage
Learning rate
Literature or grid search tuning
Reused from AdamW
AdamW
Step size scheduler
Literature
Reused from AdamW
N/A
Weight decay
Literature or grid search luning
Reused from Adam
AdamW
RMSprop parameter
Default or literature
Reused from Adam
AdamW
Inertial parameters (a, 8)
N/A
Tuned on CIFAR-10
N/A
We begin this section with the tuning of parameters a, for INNAprop on CIFAR10 with VGG and ResNet architectures and then use these parameters on larger datasets and models. As explained previously, we use as much as possible the step size scheduler and weight decay settings reported in the literature for the AdamW optimizer, which we believe to be well-adjusted and provide adequate baselines for each experiment. These are then reused for INNAprop, whence, with this protocol, merely tune and B. This approach is guided by constrained computational resources. It is, however, very interesting to look for INNAprop's peak performance with extensive tuning.
Remark 4 (Choice of baselines) Although SGD with momentum is a classical baseline for CNNs, recent works such as Lion (Chen et al. 2023bb and Prodigy (Mishchenko & Defazio 2023 show that AdamW can outperform it when properly tuned. In light of this, we adopt Adamw do not include very recent optinizers such Muon (Jordan et al., 2024), Scion (Pethick et al.] 2025), SchedulerFree (Defazio et al., 2024 they currently lack established benchmark protocols and reliable hyperparameter defaults. A comprehensive comparison would require extensive tuning, which is beyond the scope of this work.
our primary baseline. We also
3.1 Tuning INNAprop on CIFAR-10 with VGG11 and ResNet18
Hyperparameter tuning: We consider VGG11 (Simonyan & Zisserman 2014h and ResNet18 (He et al., 2016) models trained on CIFAR10 (Krizhevsky & Hinton 2010). We fix a cosine scheduler where Tmax = 200, as recommended for AdamW, and Ymin = (see Appendix D for more details) and consider two weight decay parameters À = 0 or 0.01 (defaut value for AdamW). We tune the initial learning rate "o only for AdamW. We find 70- 10-3, which is also the baseline value reported for AdamW in this experiment (see Appendix
For INNAprop, we tune only and 3 using 70- 10-3 from AdamW. Using optuna (Akiba et al. 2019 perform a grid scarch over (0 E {0.1, ................................ ..3.,,......4444 4.0}. In Figure | we detail the obtained training loss and lest accuracy for various (ax, 3) configurations over short. training durations (20 epochs) and long training durations (200 epochs) for VGGIL. Our criteria (short and long training duration) are chosen to find parameters (a. 9) that provide a rapid decrease in training loss in the early stages and the best test accuracy for long training duration.
We observe that (a (0.1,0.9) gives one of the best training losses for short durations, while (a, () (2.0, 2.0) works well for longer durations. Similar results for VGG11 and ResNet18 with and without weight decay are in Appendix F.4
Validation and comparison with AdamW: We validate this observation by repeating the experiment with & random seeds and comparing it to AdamW under the same settings with three common architectures: VGG11 (Sionyyan & Zisserman (2014), R8 He al. (2016), both used for tuning, and an additional DenseNet121 architecture Huang et al. (2017) verify that our choices transfer to different models. We show in Figure 13 that the batch size has a rather limited influence.
6

---

Published in Transactions on Machine Learning Research (11/2025)
Log-scaled Training Loss
Test Accuracy
4.0
-02
4,0
3.5
3.5
0.4
3.0
2.5
2.5
2.0
(a) 20 epochs
8
1.5
-0.6
1.5
-10
0.1
0.5
0.9
1.0
1.5
2.5
3.0
4.0
0.1
0.5
0.9
1.0
2.0 2.5
3.0
3.5
4.0
a
Log-scaled Training Loss
Test Accuracy
-15
4.0
91.0
3.5
-.0
90.5
90.0
(b) 200 epochs
1.5
89.5
89.0
-35
88.5
-.00
0.1 0.5 0.9 1.0
2.0 2.0
3.5
4.0
0.1 0.5 0.9 101221025500000 4.0
3.
Figure 1: Log-scale training loss and test accuracies for hyperparameters (a, with VGGI1 on CIFAR10 at 20 and 200 epochs. Optimal learning rate To = 10-3 and weight decay A = 0.01, with one random seed.
92
10a
91
20-4
96
89
94
Train
10-2
Test
88
92
86
10
85
86
160 Epochs
150
200
50
100
150
200
50
100
150
200
Epochs
Epochs
AdamW
INNAprop, (a, f) (0.1,0.9)
INNAprop, (a, [)= (2.0,2.0)
Figure 2: Training VGG11 on CIFAR10 with random seeds.
In Figure 2 (refer to Appendix F for additional experiments with the ResNet-18), 18), we observe that with (a, B) (0.1, 0.9), INNAprop improves training loss and test accuracy quickly by the 100th epoch, achieving the highest training accuracy. With (a (2.0, 2.0), INNAprop trains more slowly, but achieves higher final test accuracy. This is aligned with the experiments described in Figure [] In Table 2 we compare the performance of different networks on CIFAR-10 using INNAprop and AdamW optimizers. INNAprop outperforms AdamW for this benchmark.
Let us now use these tunings for large-scale models.
3.2 Extensive experiments on large-scale vision models
Let present experiments large-scale vision benchmarks with the hyperparameters of Section 3.1

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Test accuracy (%) of ResNet-18, VGGI1, and DenseNet121 on CIFAR-10. AdamW weight decay & learning rate are optimized. Results are averaged over eight runs.
Model
Optimizer
Test accuracy
Training on CIFAR-10 over 200 epochs
AdamW INNAprop (a 2.0, 3 2.0)
91.14
R8sNet18
91.58
AdamW
90.79
VGG11
INNAprop (a = 2.0, 3 = 2.0)
90.99
AdamW INNAprop (a 0.1, 3 0.9)
86.19
DenseNet121
86.91
We consider the ImageNet-1k 1k benchmark (Krizhevsky et al. 2012), training for epochs on ResNet-18 and ResNet-50 (He al.] 2016) with a mini-batch size of 256, following Chen ct al. (2023b); Zhuang
Resnets on ImageNNe::
cosine scheduler for AdamW, initial lized with (2023b]; Zhuang et al. (020); Chen et al. (2018) and applied it to INNAprop. The weight decay of AdamW was set to À 0.01 for the ResNet18, instead of (2018) because it improved the test accuracy from 67.93 to 69.43. The results of the ResNet18 experiment are presented in Figure 16 in Appendix F The figure shows, despite the lack of hypertuning, that our algorithm with (a, 3) = (0.1, 0.9) outperforms Adamm in test accuracy (70.12 vs 69.34). However, observe that the training loss decreases faster initially but slows down towards the end of training.
al. (2020). We used reported in Chen et al.
learning rate of %0 10
A = 0.05 reported in Zhuang et al. (2020); Chen et al.
For the ResNet50, we kept the value À = 0.1 reported in Zhuang al. 2020 Chen al. (0018) For INNAprop, we tried two weight decay values (0.1, 0.013 and selected À = 0.01 as il resulted in a faster decrease in training loss. We report the results in Figure g illustrating the advantage of INNAprop. As noted in Section 3.1 INNAprop with (a, 3) = (0.1, trains faster but has a lower test accuracy compared to AdamW/ / MINNAANAAOOppp with (a, = (2.0, 2.0). For (a, 3) (2.0, 2.0), the loss decrease is similar to Adamw, with no clear advantage for either method. This obviously suggests developing scheduling strategies for the damping parameters (a, B). This requires a much more computation-intense
tuning, far beyond the numerical resources used in the current work. In Table 3 present the performance of INNAprop achieved using minimal hyperparameter tuning, explained in Table [
Vision transformer (ViT) on ImageNet: We performed the same protocol (state-of-the-art AdamW
parameters) with a ViT-B/32 architecture over 300 epochs with a mini-batch size of 1024, following Defazio & Mishchenko (2023): ; Mishchenko & Defazio (2023). . We used a cosine scheduler with a linear warmup (30 epochs) and the initial learning rate and weight decay from Defazio & Mishchenko (2023) We confirm
Figure 18 that this is a favorable weight decay choice for AdamW on this problem. For INNAprop, we testod
weight decay values of (0.1, selecting 0.1 for better test accuracy. Results in Figure 3 show the
advantage of INNAprop. For faster convergence using INNAprop (0.1, 0.9), we recommend weight decay of
X = 0.01 (sec Figure 17 in Appendix
In the ImageNet experiments, we evaluated INNAprop for rapid early training and optimal final test accuracy without tuning (70, a, .). For VIT-B/32 with À = 0.1, INNAprop achieved higher training loss and a similar final test accuracy compared AdamW (75.23 vs. 75.02). Note that the choice of weight decay is tuned for confirming the comment in Remark regarding the
AdamW on this problem (see Figure 18 Appendix F
role of X and
8

---

Published in Transactions on Machine Learning Research (11/2025)
90
80
60
ResNet50
Tran
138 1est
50
00
104
6x010
00
Tran los
80
VIT-8/32
Tog
10
20
100
150
250
300
100
150
Epocha
INNAprop. (0.1.0.9)
INNAprop, (2.0,2.0)
Figure Training a ResNet50 (top) and ViT-B/32 (bottom) on ImageNet with 3 random seeds.
Table 3: Top-1 and Top-5 accuracy (%) of ResNet-18, ResNet-50, and ViT-B/32 ImageNet. Results are averaged from three runs for ResNets and one run for ViT-B/32.
Model
Optimizer
Top-1
Top-5
Train from scratch ImageNet Net
AdamW INNAprop (a = 0.1, 8 = 0.9)
69.34
88.71
ResNet18
70.12
89.21
AdamW
76.33
93.04
ResNet50
INNAprop (a 1.0, 8 1.0)
76.43
93.15
AdamW
75.02
91.52
ViT-B/32
INNAprop (a = 0.1, 3 = 0.9)
75.23
91.77

---

Published in Transactions on Machine Learning Research (11/2025)
Fintetuning VGG11 and ResNet18 models on Food101: We fine-tuned ResNet-18 and VGG-11 models on the Food101 dataset (Bossard et al., 2014) for 20 epochs, using pre-trained models ImageNet-1k. Since weight decay and learning rate values for AdamW were not found in the literature, we chose the default AdamW weight decay value, À 0.01. We used a cosine scheduler and tried one run for cach initial learning rade value (10-5; 10-5, 10-1,35 5.x x 10-*, 10-3). The best result for AdamnW was obtained for "o = 10-1, and we kept the same setting for INNoprop.
78
77
16:4
accuraty
75
96
20
10-1
72
92
Epochs 15
Epochs
Epochs
(0.1,0.9)
Figure 4: Finetuming VGG11 Food101. Left: train loss, middle: test accuracy (%), right: train accuracy
(%). Qualitatively similar results for ResNet18 are in Figure 14 in Appendix random seeds.
See for this Figure 4 where INNAprop performs no worse than AdamW on three random seeds.
Conclusion and recommendations for image classification: Tuning (a. significantly impacts training. Based on heatmaps in Section 3.1 and figures in Section 3.2 we recommend using = 0.1 and 3 E [0.5, 1.5] for shorter training (e.g., fine-tuning). For longer training, a, B 2 1 is preferable. In both cases, our algorithm either matches or outperforms AdamW.
3.3 Pre-training and fine-tuning GPT2
We present experimental results on LLMs using the hyperparameters selected in Section 3.1
GPT-2 Mini
GPT-2 Small
GPT-2 Medium
4.4
3.4
3.15
3.30
Valisaion
Jalccsionnn loso
3.3
balldtion loss
3.05
3.2
3.00
2.50
3.6
2,85
3.4
20K
40K
60K
B3K
100K
20K
40K
60K
8BK
100K
OK
40K
BOK
100K
Number steps
Number steps
Number of steps
SapNa-G
Figure 5: GPT-2 training from scratch on OpenWebTextt (Sophia-G unstable mini and medium). The implementation of the Muon optimizer is adapted from the Modded-NanoGPPP see main text.
Training GPT-2 from scratch: We trained various GPT-2 transformer models from scratch (Radford et al. 2019) using the nanoGPT repository on the OpenWebText dataset Gokaslan & Cohen (2019). For all models, gradients are clipped to a norm of 1, following Brown et al. (2020). We use AdamW with hyperparameters from the literature (Liu et al., 2023 Brown et al.) 2020), the standard configuration for
LLM pre-training. The reported RMSprop parameter 82 0.95 different from AdamW's default (0.999),
https://gitthb. com..///aacaaaaattt/ooood"""
10

---

Published in Transactions on Machine Learning Research (11/2025)
GPT-2 Small
GPT-2 Medium
GPT-2 Large
3.8
3.30
Puplecky
0.7
Perpleviy te5t
3.5
Perpextty
3.20
3.5
3.10
staps
Number staps
steps
AdamIW
NNAprop, (a. B)- (0.1,0.9)
Figure 6: Perplexity test with GPT-2 E2E Dataset with LoRA finetuning on five epochs. Three random seeds.
the weight docay 0.1 and %o depending on the network size. For example, GPT-2 small works with an initial learning rate To 6 10-4, For INNAprop, we keep the same values for and % AdamW, and the RMSprop parameter
0.99 (corresponding to for AdamW), which provides the best results among values {0.9, 0.95, 0.99} GPT-2 mini. We use this setting for all our GPT-2 experiments with (a, 3) = (0.1, 0.9). The results are in Figure INNAprop leads to faster decrease in validation loss during the early stages compared to the baseline for GPT-2 models of Mini (30M), Small (125M), and Medium (355M) sizes. Its performance could be further improved with a more thorough tuning of all hyperparameters (a, 3, o, A). For GPT-2 small, we also include a comparison with two recent algorithus. Sophia-G, using the hyperparameters provided in the literature [Liu al. 2023). The Muon optimizer [Joddnn al. (2024), based the GPT-2 training on 8xH100 cluster. For Muon, took the minibatch size, schedule and weight decay from this repository. We adapted the number of GPUs to fit our cluster, also impacting the total number of training tokens. We also used the same nanoGPT architecture as we used for AdamW and INNAprop and the Text dataset.
repository c which part of collaborative competition for fast
Comparision with INNA As a sanity check, we evaluate INNA on GPT-2 Mini and compare to INNAprop and AdamW. Following Castera et al. (2021), we used the recommended hyperparameters (a, () (0.5, 0.1) and tested learning rates (le 4, le 3, le 2, le - 1}, selecting To 0.1 the best. Figure shows that INNAprop and AdamW outperform INNA in both convergence speed and final validation loss.
GPT-2 Mini
4.0
3.9
Validation loss
3.8
3.6
3.5
3.4
Number of steps
AdamW
INNAprop, (a. B) = (0.1, 0.9)
INNA, (a, 8) (0.5, 0.1)
Figure 7: Validation loss comparison during GPT-2 mini training from scratch on the OpenWebText datasct.
http://gtthib.ca//nbbmo///////
http://ggthhb.cc///llleldaaaaann nanogpt
11

---

Published in Transactions on Machine Learning Research (11/2025)
Fine-tune GPT-2 with LoRA: Using LoRA (Hu et al. 2021), 1, fine-tune the same GPT-2 models the E2F dataset Novikova et al. (2017), consisting of roughly 42000 training examples, 4600 test examples from the restauration domain. We compare AdamW and ANNoprop for 5 epochs, as recommended in Hu
ct al. (2021). ). We use statc-of-tho-art paramoters for AdamW: linear learning rate schedule, the recommended bati- size, and the RMSprop parameter (32 = = 0.999), these are listed Table 11 in Hu al. (2021). The same parameters are reused without further funing for Aprop,,,, yet the latter outperforms AdamW. The results are displayed in Figure [ and Table where we see the perplexity mean result 3 random seeds. INNoprop with (a, B) (0.1, 0.9) achieves lower perplexity loss compared to AdamW across all GPT-2 fine-tuning experiments.
we emphasize the capabilities of ANNoprop compared to AdamW in the context of early training where gains are considerable.
We summarize the performance of our algorithm on LLMs (see Table 4)
Table 4: Comparison of GPT-2 training on OpenWebText (validation loss) and LoRA fine-tuning on E2E (perplexity).
Model
AdamW
INNAprop
Steps to match AdamW
GPT-2 from scratch (best validation loss)
GPT-2 mini
3.57
3.47
51,000 (1.96X faster)
GPT-2 small
3.03
2.98
79,000
(1.26x faster)
GPT-2 medium
2.85
2.82
83,000 (1.2x faster)
GPT-2 with LoRA (best perplexity test)
GPT-2 small
3.48
3.44
19,000 (1.31x X faster)
GPT-2 medium
3.20
3.17
20,000 (1.25x faster)
GPT-2 large
3.09
3.06
20,000 (1.25x faster)
4 Conclusion
INNAprop leverages second order geometric information a cheap manner while maintaining memory and computational footprints similar to AdamW. Experiments on text modeling and image classification show that INNAprop consistently matches or exceeds AdamW's performance.
We systematically favored AdamW by adopting the schedulers, learning rates, and weight decay, which result from extensive empirical efforts to optimize the method's performance across diverse tasks. For INNAprop, hyperparameter tuning for friction parameters (a.3) was conducted using a grid search on CIFAR-10, see Figure 21 We found that the resulting method performs very well across variety of settings (both in models and datasets), and outperforms AdamW for certain tasks such as GPT-2 training, without any additional hyperparameter tuning. We believe that further experimental studies with INNAprop could greatly improve its efficiency.
hyperparameters, including
For language models, (a, B) (0.1, works well for both pro-training and fine-tuning. In image classification, (a, 8) = (0.1, 0.9) speeds up early training, while (a, B) = (2.0, 2.0) improves test accuracy in long runs and during fine-tuning.
Overall, INNAprop shows strong performance across different benchmarks and model scales, making it a promising option for training large models. Future rescarch will focus on the design of schedulers for hyperparameters and B.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgements
Access to McsoNET resources in Toulouse was granted under allocation m23038. The authors thank Nicolas Renon and Calmip support team for their help in setting up the experiments. The authors thank AI Interdisciplinary Institute ANITI funding, through the French "Investments for the Future - PIA3" program under the grant agreement ANR-22-IACI-0000,, Air Force Office of Scientific Research, Air Force Material Command, USAF, under grant numbers FA8655-22-1-7012, ANR MAD ANR-21-CE23-155 1529, ANR Regulia. JB and EP thank TSE-P and acknowledge support from ANR Chess, grant ANR-17-FURF--0000 EP acknowledges support from IUF and ANR Bonsai, grant ANR-23- -A-R-23-CE23-0112-001
References
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pele Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265 283, 2016. URL https://www usenix .rgg https//eee/illleeeoofffreeeeeeee hwwpreecce/ood1///ddddiiie owwp://1////o1166--11 pdf
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl arXiv:2303.08721 2023.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework, 2019.
Felipe Alvarez, Hedy Attouch, Jérôme Bolte, and Patrick Redont. A second-order gradient-like dissipative dynamical system with hessian-driven damping.: Application to optimization and mechanics. Journal mathématiques pures et uppliquées, 81(8):747 779, 2002.
F Alvarez D and JM Pérez C. A dynamical system associated with newton's method for parametric approximations of convex minimization problems. Applied Mathematics and Optimization, 38(2):193-217, 1998.
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. orXiv preprint arXit:2007.03051, 2020.
Hedy Attouch, Juan Peypouquet, and Patrick Redont. Fast convex optimization via inertial dynamics with hessian driven damping. Journal of Differential Equations, 261 (11)))357734 5783, 2016.
Hedy Attouch, Zaki Chbani, and Hassan Riahi. Rate of convergence of the nesterov accelerated gradient method in the subcritical case < 3. ESAIM: Control, Optimisation and Calculus of Variations, 25:2, 2019.
Hedy Attouch, Zaki Chbani, Jalal Fadili, and Hassan Riahi. First-order optimization algorithms via inertial systems with hessian driven damping. Mathematical Progremming, pp. 1 43, 2022.
Jean-Francois Aujol, Charles Dossal, and Aude Rondepierre. Optimal convergence rates for nesterov accoleration. SIAM Journal on Optimization, 29(4):3131-3153, 2019.
Yingbin Bai, Erkum Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Nearal Information Processing Systems, 34:24392 -24403, 2021.
Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM Journal on Optimization, 31(1):244-273 2021.
Michel Benaïm. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII, pp. 1-68. Springer, 2006.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Jérôme Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programmingg pp. 1 1-33, 2020.
Vivek S Borkar and Vivek S Borkar. Stochastic opprocimation: a dynamical systems viewpoint, volume 9.
Springer, 2008.
Lukas Bossard, Matthieu Guillaunin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Seplember 6-12, 2014, Proceedings, Parl VT 13, pp. 446-461. Springer, 2014.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang, JAX: composable transformations of Python+NumPy programs, 2018. URL http:
com googl///gglleaaao
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Camille Castera, Jérôme Bolte, Cédric Févotte, and Edouard Pauwels. An inertial newton algorithm for deep learning. The Journal of Machine Learning Research, 22(1):5977 6007, 2021.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.0m 06766, 2018.
Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As Iyapunov prodicts. Xiviv preprint XXi::3310005800 2023a.
Long Chen and Hao Luo. First order optimization mothods based on hessian-driven nesterov accolerated gradient flow. arXiv preprint arXiv:1922.000 09276, 2019.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.0666 2023b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-111 113, 2023.
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lee. Stochastic subgradient method converges on tame functions. Foundations of computational mathematics, 20(1):119- 2020.
Aaron Defazio and Konstantin Mishchenko. Learning-rate-ires learning by d-adaptation. In International Conference on Machine Learning, pp. 7449-7479. PMLR, 2023.
Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, et al. The road less scheduled. arXiv preprinl ar Xix:244005 15682, 2024.
Timothy Dozat. Incorporating nesterov momentum into adam, 2016.
John Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Aaron Gokaslan and Vanya Cohen.
Openwebtext
corpus.
http://Skyylio0077gibbbb io/
OpenVebTextCorpus] 2019.
Alex Graves. Generating sequences with recurrent neural networks. arXie preprint arXiv:1388.08 2013.
Vincet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. International Conference on Machine Learning, pp. 1842 1850. PMLR, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2025)
J Harold, G Kushner, and George Yin. Stochastic approximation and recursive algorithm and applications. Application of Mathemalics, 35(10), 1997.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
Gao Huang, Zhnang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the TEEE conference on compuler vision and paltern recognition, pp. 7700-4708, 2017.
Majid Jahani, Sergey Rusakov, Zheng Shi, Peter Richtárik, Michael W Mahoney, and Martin Takáč. Doubly adaptive scaled algorithm for machine learning using second-order information. arXiv preprint arXiv:2109.055 050959 2021.
Keller Jordan, Yuchon Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers neural networks, 2024. URL httpp:://kelleeroordann github
io/posts/muon/
ar preprint Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.698 69800 2014.
Alex Krizhevsky and Geoffre Hinton. The cifar-10 dataset, 2010.
Alex Krizhevsky, Llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. pp. 1097-1105, 2012.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. ai preprint NNw:2235.14444 2023.
Lennart Ljung. Analysis of recursive stochastic algorithms. IEEE transactions on automatic control, 22(4): 551-575, 1977.
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1rXii:1608..030 2016.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051: 2017.
James Martens and Roger Grosse. Optimizing neural networks with cromecker-fcctored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint Xrv:222006.00 06101, 2023.
Yurii Evgen'evich Nesterov. A method of solving a convex programming problem with convergence rate o(1/k2). In Doklady Akademii Nauk, volume 269, pp. 543-547. Russian Academy of Sciences, 1983.
Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-lo-end generation. In Proceedings of the 18th Annuol Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrücken, Germany, 2017. URL https: http://wwxxvvv.oaab/b///// www./bgg/////eeoooooooo/oooeeeee 1100 09254 arXiv:1706.09254.
Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. orXiv preprinl ar 0et::4409.03339 2024.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca. Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygekzimer, d' Alcch-Buu,, E. Fox, and Garnett. (eds.), Advonces Newral Information Processing Systems 32, pp. 8024-80000 8034. Curran Associates, Inc., 2019.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Mique Munguia, Daniel Rothchild, David So, Mand Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.1083 10350, 2021.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained Imos. arXiv preprint arXic:2502.0755 2025.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964.
Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pp. 55 69. Springer, 2002.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145-151, 1999.
Xun Qian, Rustem Islamov, Mher Safaryan, and Peter Richtárik. Basis malters: better communicationn-lliiend second order methods for federated learning. arXie preprint arXiv:2111.01847, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.0992 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arxiv: (xXi::669.04747, 2016.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63 (12):54-63, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596 4604. PMLR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 2014.
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. Optimizer bench- marking needs account for hyperparameter tuning. In International conference on machine learning, pp. 9036 9045. PMLR, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13693-13696, 2020.
Weijie Su, Stephen Boyd, and Emmanuel J Candes, A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1 43, 2016.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Teo Susnjak, Timothy R McIntosh, Andre LC Barczak, Napoleon H Reyes, Tong Liu, Paul Watters, and Malka N Halgamuge. Over the edge of chaos? excess complexity as roadblock to artificial general intelligence. arXiv preprint arX7::2407.09662, 2024.
Nya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference machine learning, pp. 1139-1141 PMLR, 2013.
Tijmen Ticleman, Geoffrey Hinton, al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural nelworks for machine learning, 4(2):26 31, 2012.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Gaël Varoquaux, Alexandra Sasha Luccioni, and Meredith Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in ai. arXic preprint 0rXiu:2409.144 14160, 2024.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoncy. Adahossian: An adaplive second order optimizer for machine learning. In proceedings of the AAAI conference artificial intelligence, volume 35, pp. 10665-10666 2021.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. Xiv preprint arXic: 190600220 2019.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advonces in neural information processing systems, 15118795-18800 2020.
17

---

Published in Transactions on Machine Learning Research (11/2025)
This is the appendix for 'A second-order-lik like optimizer with adaptive gradient scaling for deep learning'.
Contents
A Reminder optimization algorithms
19
B Additional details on the derivation of INNAprop
20
C Alternative discretizations
23
D Scheduler procedures
26
E Choosing hyperparameters and I for INNAprop
26
F Additional experiments
27
Experimental Setup
31
18

---

Published in Transactions on Machine Learning Research (11/2025)
A Reminder on optimization algorithms
Considering the unconstrained minimization of J (see Equation I and using the shorthand VT((k)) gk+ we outline in the table below the structure of several well-known optimizers.
Table 5: SGD is due (Robbins Monro 1951). Momentum (Polyak 1964). Nesterov to Nesterov
1983), RMSprop + Momentum to Grrvvss 2013), Adam to (Kingma & Ba 2014h, NAdam to Dozat 2016 and INNA to Castera ct al. 2021
SGD(1k)
SGD Momentum(144 B1)
Ok+1 0s To.9k
Vo 0
uR+1 8i00 + (1 -1-33 B1)g
Adam Adam(1): (7k: $1, c2
0.+1 02 Uk+1
my = 0, Up =
m B1mx (1 B1)gN
RMSprop Momentum(ny, 31, 32, c)
Uk+1 = 328k + (1 32)
Uo = 1,mp =
m
0k+1 = 0% - 7k uk+1 +
uk+1 820% - (1 - B2)92
Ok
m++1 = 81mp + Vu+++ +
0.41 0₂ 7mm++1
NAdam(T4 4; B1, ()
0, 0
INNA(7K, B) 2/b (1 ar3)6u
μ4 = (1 - 0.96
2
m Bimm + (1 - -Bi)ggk
th++ th + 7h ( -a)0
-3 -//%
Uk+1 824k + (1 -32)
1 (1 )0k)gk
0x41 02 + 7k
a
2)22
1/2
-3g
ak+1 O2 7kk UK++
INNAprop( a, (,)
DINAdam(n.og T1, 01, (,
v (1 a,3)0u
uk+1 = O + (1 o)
mo vo
m 01 mm (1 01) (1 BBa)gk
Mh+1 = w/k (1 7k
+ 7k
3
Uk+1 02 + (1 o2)
7k (1 33a
0k+1 0g- 7
14+1 a.9r
0x41
1 +
3 7kk
3 7
Vk-1
7h
/uk+1 +
/ ut+1 +
19

---

Published in Transactions on Machine Learning Research (11/2025)
B Additional details on the derivation of INNAprop
B.1 An equivalent basic update
Starting again from the basic recursion INNAprop-O]
Uk+1 otk + (1
7kk
1
Wh+1
7
Ok.
8
3
1
7k
9k
Ok+1 = On
1 (1+ 7k
Uu
3
3
Vu++T
+
we obtain an equivalent rewrite of the update starting with the following equivalent relation between the and 1
xk41 1/2
(1 (1-
7k
7h
1
Ok
(5)
B
3
B
1
th
uk+1
7kk
B 7h
3
This allows to obtain an equivalent update for the variable as follows
Jk
Oz+1 0s
1 7k
7h
Uk++
+
1
7kk
9k
02 + 7k
Wh+1
+
7kk
a
0z
-k.B
B
7kk
Ykk
3
Uk+1 +e
7
7
0x + 7
1
a
0%
1b+1 - 3
B 7kk
7k
/ek+1 tE
1
7o - (aa
Wk41
gk
(6)
8 7
8 7
Uk+1 +e
This results in the following equivalent basic algorithm. This is the form we use in Algorithm | allows for a direct naive implementation using in place updates for both variables &i and
because it
Algorithm INNAprop
1: Objective function: [0) for E IRP.
2: Constant step-size: 7 0 1], ,,0,3,,,
Hypar-parametors:
Initialization: Op to 0, th a-3))0
for k to K do
gk (O(O))
7: u + JUR + (1 - σ 9g
8: 1/tt1 + 1 - 4b + ? [ - a 0
0 +- +1 2(-a88)
0x 3-7 4b441 BB I+e
10: return 0k+1
20

---

Published in Transactions on Machine Learning Research (11/2025)
B.2 Direct derivation from the autonomous continuous time system
Let us recall the continuous time system in
d +()++ + (MSprop((((0())) RMMPp))(((SSpp dt
(7)
RMSprop( = VG(t) VJ)((())
G(t) = / V
The purpose of this section is to verify that the recursion in (INNApro--Op can be understood as a direct Euler discretization of the system (7 for a certain probability measure μ4 over R4 as a time space. We will consider here a constant step size y for simplicity.
Let first consider the dynamics G. Set dru(r) bexp(b(T ()) for some b 0 to be precised latter. We note that 2 dT) bddu(()). The dynamics in G can therefore be equivalently expressed
dt G(t) (VJ(((t)) S (I(((T))² d dt duu(T)
-V - (]((T)))
=VVJ)(())) G(t))
In [INAApop--). in principle, the step size y is small and is close 1 (for example in ADAM's default parameters). We will consider the same asymptotic regime as in (Barakat & Bianchi, 2021 Assumption 2.4) and set b 1-8 Now assuming that 0x is given direct Euler discretization with step size y gives
uk+1
Uk 1 (J(Bk)² - vK)
which is the first line of the recursion INNAprop-O
The rest of the argument similar the derivation the INNA algorithm in Castera et al. (2021 based on phase space representation of the (DIN) system Alvarez al. (2002). Introducing
= - - 3²RMSprop((((())) - - (a3 1)0(t),
the system 7 can be equivalently written as
(t) (t)
0(t)
(5)
((t)
0(t) - (RMSprop(((((t))
and the last two lines of the recursion in [NNAprop-D can be seen a direct Euler discretization of this system.
B.3 Equivalence between a special case of INNAprop and Adam without momentum
In this section, we demonstrate that INNAprop with a and 8 1 is equivalent Adam (Kingma & Ba 2014) without momentum (31 0). To illustrate this, analyze the update rules of both algorithms. We assume that the RMSprop parameter &r (for Adam) and (for INNAprop) are equal. Starting with
INNAprop, we initialize vo = (1 - a.3) For and B = 1, this simplifies to v/p 0. The update for tb becomes:
Wk-1 (1-7) 7 the + (1
0x (1 r)
21

---

Published in Transactions on Machine Learning Research (11/2025)
Given that do 0, it follows that 4/x 0 for all k. The parameter update rule for INNAprop is:
Ox+1 = 1 7(1 - ag5)
7
9h
On
3 --7 Wk+1 7B
B -
Vuk+i +
Replacing 1, 3 = 1, and 4/1 0, we get:
gk
0 0%
vu++1 +
Here, Ib is the gradient, and v is the exponential moving average of the squared gradients:
uk+1 = 0%k + - (1-o)gg
The Adam optimizer uses moving averages, mi (momentum term) and up (squared gradients):
= f1m--1 + +1 31)9k
2 00k-1 (1 1 -0)9
Setting 31 = 0, the momentum term my simplifies to my = 9k-
The update rule becomes:
Sk
Ok+1 O2
Vuk +
This matches the form of Adam's update rule without the momentum term, confirming that INNAprop with a 1 and B is oquivalent to Adam with $1 O.
A Aloorihhm 3 INNAprop with (a, = (1,1)
Objective function: [(0) for E IR".
Constant step-size: 7
Hyper-parameters: a E [0, 1], 2 0, 8> 10
4: Initialization: time step k + 0, parameter vector 8o = 0.
5: for A: = to K do
6: gk = ((t)))
t + otk (1 g)g
Ōk+1 vk+1/(1 ot)
Ok+1 O2 7 gk/(( (vu+z +c)
10: return 0k+1
16
1.0
30
10
Text acC REC
Train a0au/ay
96
Tra
10
10.
AdemW
INNAprop, f)= (0.1,0.9)
INNAppop, (a. 8)= (1.0,1.0)
Figure 8: Training VGG11 on CIFAR10. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with random seeds. AdamW is considered without momentum illustrate the connection with INNAprop, the two curves are indistinguishabl.
22

---

Published in Transactions on Machine Learning Research (11/2025)
10
100
1071
Taatt Recuriny
Train accurasyy
Train
10
10
200
200
100
Epachs
AdamW
INNAprop, (a. B) (0.1,0.9)
INNAprop, (d. B}r (1.0,1.0)
Figure 9: Training NesNet18 on CIFAR10. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with random seeds. AdamW is considered without momentum to illustrate the connection with INNAprop, the two curves are indistinguishable.
C Alternative discretizations
C.1 A variant of INNAprop with momentum
The algorithm. We follow the rationale behind the algorithm RMSprop with momentum GGrves 2013
We therefore start with Equation DIN using the RMSprop proxy for the gradient as in direct discretization gives:
Uk+1 OUk (1 o)ge
0k+1 20x + 02-1
Ox 0x-1
+ vet+ite
9h
/ tc
9k-1
aa- 7
7
Uk te
Rearranging terms, we have
U ouk + (1 - -
0++1 0% + (1-1aa a0)((x - 0k-1)) -B7
(y
gle
9h-1
2
9k
/u++1 foeti+i +
uuh +
v Uk +
Let us introduce a momentum variable my 0x-1 - 0g obtain:
Uk+1 oUh + (l-
(8)
Mk+1 (1 (1-00 a-)77) mz + 22 xk-1 + BY
Uok +e ( te
9k
9k-1
(9)
Vuk +
0x+1 = 0g m
(10)
As previously we might have an equivalent rewrite of this recursion which allows for in place updates. For this we rewrite Equation
m am, + bgk c
(11)
where a = (1 - -a7), b= By and c 7(3 - 7)) Writing mh mi
- - we have
c
mi++1 = m*+1
99k
a
= am, + bgk - cgk 1
gk
c
+ (b
c
= a m+
- [k-1
gh
ami + (b.
c
a
gk.
23

---

Published in Transactions on Machine Learning Research (11/2025)
Therefore, using this identity, may rewrite the following
M = anTT + byk c0k-1,
0&+1 = 0k - -
M&+1 amiu + (b -) gk.
Bx+1 0% m gk-
Recalling that a = -1 -1- b= b= By and c 7(3 -7). Finally, we get the following recursion which is an
alternative way to integrate RMSprop to INNA:
uk+1 0tk + 5 0)
(12)
M = - a1 + 92
ag
gk
(13)
an
Uu++ +
7/ -7) 7)
gk
0K+1 = 02 - mi 1 0z v
(14)
Uk++1 +
but as shown below through numerical experiments the resulting algorithm performs poorly. Indeed, the factor 72 poorly scaled for 32 bits or lower machine precision.
Numerical experiments. Using CIFAR-10 dataset, train VGG11 network with the momentum version of INNAprop with the hyperparamcters (a, () (0.1.0.9) above. We used cosine annealing scheduler with 7 10-3 and no weight decay. As seen in Figure 10 the training loss stops decreasing between the 125th and 150th epochs. Upon closely examining the algorithm in this regime, we observe that al the end of training, 2f falls below the numerical precision, resulting in unstable behavior in Equation
196
meeining Loss
20-1
10-2
125
Figure 10: The version of INNA with momentum of Section
C.1 is an unstable method.
C.2 An approach à la Adam
In this section, we mimic the process for deriving Adam from the heavy ball with a RMSprop proxy, see, o.g., Kingma & Bal (2014h: Ruder (2016), by simply replacing the heavy ball by DIN We call this optimizer DINAdam.
From DIN)
infer the discretization:
0k+1 20p + 0x-1 + a
0k+1
9k
-k-1
+ go 0.
(15)
722
Note that DIN with boils down the heavy hall method.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Rearranging terms, we have
72
By
x++1 0%
9k
(0x 0k-1)
(9) -
(16)
+ an
1+ay"
(1 + ar)
By introducing the now variable my = (0x-1
0x)/n and sctting 7 > 0, we can rewrite equation 16 as:
1
72
By
m4+1 (1+
m +
(1 ay))"
9h (1+a7)n (gx: 9k-1)
(17)
0k+1 04 mp1++1
(18)
To follow the Adam spirit, we sot 01 (1407] and (1-01)
Solving for y, we get
[1+a7)n
72
1+
(1+a7)n
Then, we find the following recursion:
m m (1 )g + +,800 (gx. gk-
(19)
B 0x 1p+++1
(20)
From Equation
19 we make a change of variable my = ml a.3gk-1.
m = - + (1 - 01 + Bao - Ba)gg
(21)
0h-1 02 - n + az3ga))
(22)
Using the usual RMSprop constants 02 E [0 and > 0, we obtain:
Uk+1 = 02 + (1 o2)
(23)
M = ky + (1 (1 - or3)gh
(24)
0x+1 = 0₂ 17 ml++1 + 0r
(25)
uu++1 +e
Algorithm DINAdam
Objective function: T(0) for E RD
Constant step-size: >>>
Hyper-parameters.. (01, 02) [0, 1J2 0,800, 1
4: Initialization: fo. 0, mo = 0.
5: for &: = to K do
6: gk = (0()))
7: Uk+1 - 02th + (1-00 (1-02)94
m + o + (1 o1) az)gk
9: 0 +- 0: me+1+a091
/oz+i+e
10: return 0k+1
Remark 5 The way RMSprop is integrated within INNAprop and DINAdam different. In RMSprop is incorporated directly during the discretization process of Equation 4 and in particular the RMSprop scaling apply to all gradient terms the recursion. However, DINAdam, RMSprop is added only at the last step, as shown in Equation 23), and only affects the momentum term. This is how RMSprop combined with heavy ball to obtain Adam.
25

---

Published in Transactions on Machine Learning Research (11/2025)
Remark 6 After setting = and B 0, we obtain Adam update rules. If * 0, DINAdam is qualitatively very close to the NAdam algorithm, with a different. choice of hyperparameters. This is seen for from Algorithm 4 to be compared with Table 5 (for aB < 1). In both cases, the update is a weighted average between the current gradient and discounted average of past gradients, making it possible to recover each recursion equivalently with different hyperparameter choices (including the step-size).
Scheduler procedures
Cosine annealing (Loshchilov & Hutter 2016). Let 7' represent the learning rate at iteration k, Tmax be the maximum number of iterations (or epochs), and Ymin be the minimum learning rate (default value is 0). The learning rate 7 iteration & is given by:
k
(70 7mnnn) (1+ cos
7h
Tmin
T
This scheduler was employed in all image classification experiments except for VIT.
Cosine annealing with linear warmup (Radford et al., 2018). Let 7 represent the learning rate at iteration k Tmin the minimum learning rate, to the initial learning rate, warmmp the number of iterations
for the warmup phase, and Tdecay the iteration number after which the learning rate decays to Ymin- The learning rate is defined as follows:
700 Twarmpp'
if k T Twarmup
7h
Timn + (70 -mmin)
(1 cos
(T
k- T Twarmup Taweay Twarmop
T Tarrmpp S S Taecay
Tmins
Tdecay
This scheduler was applied in experiments involving training GPT-2 from scratch and for VIT.
Linear schedule with linear warmup (Hu et al., 2021). Let " represent the learning rate itcration i and Tmax be the maximum number of iterations, T warmuu be the number of warmup steps, and Tmin be the minimum learning rate after warmup (default value is typically set to the initial learning rate, 70). The learning rate "h iteration & is given by:
if l T warrmup:
Twannup
7k
Twarmup max Twarmup
otherwise.
This schoduler was used for fino-tuning GPT-2 with LoRA.
E Choosing hyperparameters a and 3 for INNAprop
E.1
Comparison with AdamW
For VGG and ResNet training CIFAR10, the literature suggest using initial learning rate %
10 with
learning rate schodule (Mishchenko & Defazio 2023 Defazio & Mishchenko, 2023 Yao et al. 2021 Zhuang
et al. 2020). Our experiment fix a cosine scheduler where T Tmax = 200 and 7min O as it achieves a strong
baseline for AdamW (Loshchilov & Hutter 2016 Mishchenko & Defazio 2023). We set weight decay 0.1.
Then, we tune the initial learning rate To among {10-2.5 5 10-a, 10
10-3 10-23. In Figure we
report the performance terms of training loss and test accuracy for AdamW. These results confirm the
usage of "t 10-3
26

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Performance rankings with VGGII.
70
Train loss
Test accuracy (%)
10-3
0.00041
91.02
5 X 10 10-3
0.00047
90.86
x 10
0.00048
90.79
10-2 2
0.00057
90.41
10-4
0.00081
88.49
(b)
Performance rankings with ResNet18.
Train loss
Test accuracy (%)
10
0.00040
92.1
X 10-3
0.00049
91.84
10-4
0.00094
92.32
10-2
0.00057
90.41
10-4
0.0018
87.85
Figure 11: Comparative performance of the training loss and test accuracy according %o- We trained VGG11 and ResNet18 models CIFAR10 for 200 epochs.
F
Additional experiments
F.1
CIFAR10 experiments
98
accuracy
88
Test
86
94
90
100 Epochs
200
100
200
100
150
200
Epachs
Epochs
AdamW
INNAprop. (a, =)= (0.1,0.9)
INNAprop, (a, f)= (2.0,2.0)
Figure 12: Training Net18 on CIFAR10. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 8 random seeds.
95.0
accuracy
92.5
algo
AdamW
Innaprop (0.1, 0.9)
Innaprop (2.0. 2.0)
100
200
500
Batch size
Figure 13: Influence of the minibatch size for training ResNet18 CIFAR10. The learning rate is tuned for each batch size. The influence of the batch size is relatively moderate for all three algorithms.
27

---

Published in Transactions on Machine Learning Research (11/2025)
F.2 Food101 experiments
10a
70
98
10-4
Test
75
94
10:4
72
90
10
15
20
15
20
15
20
Epochs
Epochs
Epochs
AdamW
INNAprop. (a, 8)- (0.1,0.9)
INNAprop, (a, B)- (2.0.2.0)
Figure 14: Finetuning ResNet18 on Food101, same as Figure for ResNet18. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds.
F.3 ImageNet
90
70
80
8x
Top trann 30curao)
70
80
89
10'
00
80
tran SCC//S))
30
67
b2c
ResNet18
lest
80
2.300
80
50
88
Epocct
AdamW
INNAprop. (a. (0.1.0.9)
INNAprop, (a. (1.0,1.0)
Figure 15: Training ResNet50 (top) and ResNet18 (bottom) ImageNet. Left: train loss middle: Top-1 test accuracy (%), right: Top-1 train accuracy (%). 3 random seeds.
28

---

Published in Transactions on Machine Learning Research (11/2025)
70
102
Top- Urain WOTLENCO
3x10
00
50
2x10
20
Epochs
AdamW
INNAprop. (a, BF (0.1.0.9)
INNAprop, (a. B) (2.0.2.0)
Figure 16: Training ResNet18 on ImageNet. Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds. The superiority of INNAprop consistent from epochs 70 to 90. Note that we chose 90 epochs be consistent with the hyper-parameter settings reported in the literature Chen al. (202bb).
6'100
70
60
Top-1 Dai 4422
Top-1 bast
1x10
80
3x10
10
250
190 Epochs
Epocha
Fpochs
AdamW
INNAprop. (a, By (0.1,0.9)
INNAprop, (a, f)* (2.0,2.0)
Figure 17: Fast training VIT/B-32 on ImageNet with weight decay À = 0.01 for INNAprop (a. (0.1, Left: train loss, middle: test accuracy (%), right: train accuracy (%), with 3 random seeds.
10
00
Top-1 104n MCCOURRI ENCLLENT
4x10
Top-1 best
60
40
3x10
20
10
50
130
100 Epochs
200
210
300
50
100
100 Epochs
200
200
300
50
100
150 Escohs
200
250
300
AdamW, A 0.1
AdamW. 0.01
AdamW. 0.3
Figure 18: Tuning the weight decay for the AdamW optimizer the VrT/B-32 ImageNet training, with 3 random seeds.
29

---

Published in Transactions on Machine Learning Research (11/2025)
F.4
Heatmap for preliminary tuning of and B
Log-scaled Training Loss
Test Accuracy
3.0
3.0
84
2.0
2.0
1.5
80
1.0
0.9
-0.8
0.9
-78
0.1
1.0
0.1
0.1
0.1
0.5
0.9
1.0
2.0.2.22
(a) 20 epochs
Log-scaled Training Loss
Test Accuracy
91.0
-250
3.0
3.0
90.5
2.0
33.00
2.0
90.0
-325
09.5
1.8
10-
0.9
-3.50
0.9
89.0
-3.75
88.5
0.1
0.1 0.5 09
1.1.5.292
4,00
0.1
0.155
(b) 200 epochs
Figure 19: Log-scale training loss and test accuracies for (a, () hyperparameters with VGGII on CIFAR1O at different epochs. Optimal learning rale 7/o = 10 weight decay À
Lag -saled Training
Accuracy
0.2
88
3.5
3.5
86
2.5
2.5
84
-0.5
82
1.0
0.6
80
-0.77
78
0.5
0.5
0.1
76
2.0
(a) 20 epochs
Log-scaled Training Loss
Accuracy
4,0
92.0
3.5
355
25
23
90.5
1.5
1.5
90.0
1.0
100
83.5
89.0
03
0.5
88.5
0.0.500.
(b) 200 epochs
Figure 20: Log-scale training loss and test accuracies for (a, (8) hyperparameters with ResNet18 CIFAR10 at differrnt epochs. Optimal learning rate "o 10 weight decay À = 0.01.
30

---

Published in Transactions on Machine Learning Research (11/2025)
Log-scaled Training Loss
Test Accuracy
-0.2
88
3.5
-0.3
3.5
2.5
2.5
84
2.0
2.0
1.0
1.0
-0.7
78
0.1
0.8
0.1
76
1.0
0.1
0.5
0.9
1.0
3.0
3.5
4.0
(a) 20 epochs
Log-scaled Training Loss
Test Accuracy
92.0
3.5
-20
33
91.5
2.5
2.5
2.0
2.0
90.5
1.5
90.0
-30
1.0
1.-
89.0
0.5
-35
0.5
88.5
(b) 200 epochs
Figure 21: Log-scale training loss and test accuracies for (a, B) hyperparameters with ResNet18 on CIFAR10 at different epochs. Optimal learning rate "No 10-3 weight decay X = 0.
G
Experimental Setup
G.1 CIFAR-10
We used custom training code based on the PyTorrc tutorial code this problem. Following standard tata-angmentttion practices, we applied random horizontal flips and random offset cropping down to 32x32, using reflection padding of 4 pixels. Input pixel data normalized by centering around 0.5.
Hyper-parameter
Value
Hyper-parameter
Value
Archilecture
VGGII and ResNet18
Baseline schedule
cosine
Epochs
200
Weight decay X Br, B2 (for AdamW)
0.01
GPUs
IxV100
0.9, 0.999
Batch size per GPU
256
a (for INNAprop)
0.999
Baseline LR
0.001
Seods
8 runs
G.2 Food101
We used the pre-trained models available on PyTorch for VGG1I and ResNet18
Hyper-parameter
Value
Value
Architecture
VGG11 and ResNet18
Baseline schedule
cosine
Epochs
200
Weight decay À
0.01
GPUs
IxV100
B1, B2 (for AdamW)
0.9, 0.999
Batch size per GPU
256
(for INNAprop)
0.999
Baseline LR
0.001
Seeds
3 runs
htal
31

---

Published in Transactions on Machine Learning Research (11/2025)
G.3
ImageNet
We used the same code-base as for CIFAR-10 experiments, and applied the same preprocessing procedure. The data-augmentations consisted of PyTorch's RandomResizzdCrop cropping to 224x224 followed by random horizontal flips. Test images used a fixed resize to 256x256 followed by a center crop to 224x224.
G.3.1 18
Hyper-parameter
Value
Value
Architecture
18
Baseline schedule
cosine
Epochs
90
Weight decay X
0.01
GPUs
Vx00
31, B2 (for AdamW)
0.9, 0.999
Batch size per GPU
64
a (for INN Aprop)
0.999
Baseline LR
0.001
Seeds
3 runs
G.3.2 ResNet50
Hyper-parameter
Value
Value
Architecture
18
Baseline schedule
cosine
Epochs
90
Weight decay X
0.1
GPUs
V100
31, B2 (for AdamW)
0.9, 0.999
Batch size per GPU
64
a (for INNAprop)
0.999
Baseline LR
0.001
Mixed precision
True
Seeds
3 runs
G.3.3 ViT /B-3B
Hapam-parameter
Value
Hyper-parameter
Value
Architecture
VIT/B-3)
Bascline schedule
cosine
Epochs
300
Warmup
linear for 30 epochs
GPUs
8xA100
Weight decay
0.1
Batch size per GPU
128
31, B2 (for AdamW)
0.9, 0.999
Baseline LR
0.001
(for INNAprop)
0.999
Seeds
5000
G.4 GPT2 from scratch
We followed the NanoGPT codebase and we refer to the default batch-size and schedule.
Brown et al.
2020 as closely possible, matching
Syype-parameter Architecture
Value GPT-2
Hlpper-paameete
Value
Seeds
5000
Batch size per gpu
12
Weight decay À
0.1
Max Iters
100000
B1, B2 (for AdamW)
0.9, 0.95
GPUs
4xA0000
(for INNAprop)
0.99
Dropout
0.0
Gradient Clipping
1.0
Baseline LR
refer to
(Brown al. 2020)
Float16
True
Warmup Steps
500
htttps://githbb.com http//ghhhh.cmm//ppppttttaaaaaa
32

---

Published in Transactions on Machine Learning Research (11/2025)
G.5 GPT-2 with LoRA
We followed the LoRA codebase and we refer to (Hu el al. 2021 closely possible, matching the default
batch-size, training length, and schedule. We train all of our GPT-2 models using AdamW (Loshchilov & Hutter 2017) and Aprop on E2E datasct Novikova ct al. (2017) with a linear learning rate schedule for 5 epochs. We report the mean result over random seeds; the result for each run is taken from the best epoch.
Hyper-parameter
Value
Hlpee-parameter Seeds Weight decay À
Value
Architecture
GPT-2
runs
Batch size per gpu
8
0.01
Epochs
31.82 B2 (for AdamW)
0.9, 0.98
GPUs
1x A100
(for INNAprop)
0.98
Dropout
0.1
Learning Rale Schedule
Linear
Baseline LR
0.0002
LoRA a
32
Warmup steps
500
33


Reviews:
[{"review_id": "dfb4DvGyoo", "reviewer": "Reviewer_3aip", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- **Rigorous algorithmic foundation**: The algorithmic derivation is well-grounded in continuous-time ODE theory with detailed step-by-step discretization. The explicit variable usage ensures memory equivalence with AdamW (see Algorithm 1 and derivation in Section 2). The connection to previous inertia-based and adaptive methods is appropriately covered.\n- **Fair experimental protocol**: The hyperparameter tuning protocol is transparently described and consistently favors AdamW. The authors make considerable effort to ensure AdamW is used in its optimized or literature-preferred settings, reducing confounding effects from suboptimal baselines (as detailed in Table 1 and corresponding sections).\n- **The empirical evaluation is relatively comprehensive in scope**: Experiments are reported across vision (CIFAR-10, Food101 and ImageNet with different model architectures ResNet, VGG, DenseNet, ViT) and language modeling (GPT-2, including both pretraining and LoRA fine-tuning). The evaluation spans both small- and large-scale scenarios, including training speed and accuracy comparisons.\n- **Strong language modeling performance**: The empirical results for GPT-2 pretraining and fine-tuning demonstrate that INNAprop outperforms AdamW, representing a notable achievement for the proposed method.\n\n## Weaknesses\n- **Overreliance on “favoring AdamW” protocol**: While the decision to always optimize/tune AdamW hyperparameters is commendable for fairness, it sometimes leads to the impression that INNAprop’s absolute best performance is unexplored, potentially underrepresenting its peak capability. I'm not asking the authors to do massive hyperparameters tuning, but some interesting questions might be ignored. For instance, given its purported second-order-like information benefits, how does INNAprop respond to batch size variations? Such experiments could illuminate its robustness characteristics.\n- **Insufficient intuitive explanation**: Although the derivation is mathematically clear, the paper lacks intuitive discussion about why INNAprop should outperform alternatives like DINAdam or INNA. While INNAprop incorporates second-order information for RMSprop scaling, which is an advantage compared to RMSprop, the specific advantages over INNA remain unclear. This gap makes the motivation for developing INNAprop appear incomplete, merely another deep learning optimizer with integrated second-order-like information rather than a principled advancement.\n- **Unclear method relationships**: The connections and comparisons between INNAprop, DINAdam, INNA, NAdam, and AdamW lack clarity (see critical comments 1-4 below).\n- **Questionable vision results**: The empirical results in computer vision are not entirely convincing (see critical comments 6-8 below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper introduces INNAprop, a deep learning optimizer derived by combining the INNA (dynamical inertial Newton) method with RMSprop’s adaptive gradient scaling approach. The resulting optimizer is designed to enjoy second-order-information-like benefits, while maintaining the same memory requirements as AdamW, making it suitable for deep learning.\n- The authors show through extensive experiments that INNAprop is competitive to AdamW across a variety of benchmarks, including image classification datasets with modern CNN and ViT architectures, as well as language modeling and fine-tuning tasks with GPT-2.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical comments to address\n\n1. It seems that the starting of the derivations of INNAprop with momentum in Appendix C.2 is exactly the same as the starting of the derivations of INNAprop in Appendix B. How do they result in different methods as stated in Remark 1 (b)?\n2. Perhaps I missed something. The authors claim to \"improve the memory footprint of INNA by saving a memory slot.\" However, examining INNA's update rules in Table 5, while INNA maintains vector $\\psi_k$, it doesn't compute $v_k$ (unlike INNAprop). Shouldn't their memory footprints stay the same ?\n3. **Unclear claims in Remark 2**: Several statements require clarification, such as \"When $\\beta = 0$, we recover a modified version of RMSprop with momentum\" (discussed in Appendix B.1 for Adam, not RMSprop) and \"If β ≠ 0, DINAdam is very close to NAdam algorithm\" (lacking further discussion despite Table 5 providing update rules for known optimizers).\n4. The authors claim that INNAprop benefits from second-order information. I have concern about this claim. First, I'm not familiar with the literature of dynamical inertial Newton and finite difference approximation applied on it. In my opinion, there is a distinction between using actual second-order information and its approximations. How effective is the finite difference approximation applied to dynamical inertial Newton methods? Given the extensive literature cited, can the authors provide additional evidence for this research direction's general effectiveness?\n5. There is this statement \"by setting $\\alpha = \\beta = 1$, we empirically recover the behavior of AdamW\" in Remark 2. Isn't this claim addressed in Table 3 for INNAprop ($\\alpha = \\beta = 1$) ? I find the result of INNAprop ($\\alpha = \\beta = 1$) in Table 3 strange, as it is not discussed elsewhere in the paper.\n6. Regarding Figure 3, the authors claim that \"in the ImageNet experiments, we evaluated INNAprop for rapid early training and optimal final test accuracy without tuning $(\\gamma_0, \\alpha, \\beta)$. For ViT-B/32 with $\\lambda = 0.1$, INNAprop achieved lower training loss and higher final test accuracy than AdamW (75.23 vs. 75.02).\" However, for ViT-B/32, Figure 3 shows AdamW achieving lower training loss with similar final test accuracy to INNAprop. Thus, the claimed advantage is not clearly evident.\n7. Regarding Figure 14, the authors claim that \"our algorithm with $(\\alpha, \\beta) = (0.1, 0.9)$ outperforms AdamW in test accuracy (70.12 vs 69.34).\" While INNAprop slightly outperforms AdamW in final test accuracy (70.12 vs. 69.34), it performs worse during most of training. This raises concerns about whether training was deliberately stopped at an advantageous moment for INNAprop. For instance, what will happen if we leave the training longer ?\n8. Based on comments 6 and 7, the claim that the algorithm \"either matches or outperforms AdamW\" on large-scale vision models appears unsupported.\n\n## Suggestions for strengthening the work\n\n- Add missing citations for OpenWebText and E2E datasets.\n- Include proper discussion of DenseNet, which currently appears only in the abstract and Table 2.\n- Add DINAdam and INNAprop to Table 5 for clearer comparison.\n- **Include INNA comparisons in the main paper**: While I acknowledge Appendix F.5 presents INNA experiments in language modeling, these results should be appeared in the main text. Since INNAprop already demonstrates strong language modeling performance, it would be more valuable to evaluate whether INNAprop outperforms INNA in computer vision tasks with a clear margin. Such comparisons would better justify the proposed method's advantages over its direct predecessor.\n- As mentioned previously, explore INNAprop's robustness through batch size variation experiments to showcase additional advantages of second-order information beyond faster convergence.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vL3yPg08Zw", "reviewer": "Reviewer_MYuc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\nI think the proposal of INNAProp is quite interesting. There is no issue with the direction of this study; the optimizer is well-motivated. And I think the experimental results are positive, in the sense that INNAProp passes the \"sanity check\" -- I believe it can work, and indeed it seems to work -- which is not trivial.\n\nI don't believe INNAProp has any theoretical advantage over AdamW; and I don't believe that, with thorough hyper-parameter tuning, one can consistently outperform another in practice. But what makes INNAProp unique, is that the slot variable $\\psi_t$ only keeps track of the model parameter, but does not depend on the gradient at all; yet it can achieve an effect of momentum. This might be useful in some scenarios, for example when the gradient calculation is unreliable.\n\n## Weakness:\n\nIn the technical details, this paper needs to improve a lot.\n\n1. The theoretical part is difficult to follow; especially Section 2.2, _Derivation of the algorithm_: I couldn't understand how the algorithm was derived from the 2nd order ODE, at all; the narration refers to Equation (7) and (8), but Equation (8) is not presented in the main paper -- it jumps to the Appendix. And I still couldn't quite get the essence after reading the Appendix, until I read Castera et al.'s JMLR paper. You are not making a big theoretical advancement in this paper, so you have a very good existing framework in Castera et al.'s paper to follow -- yet why is the writing of this paper so messy? This is almost irritating. Furthermore, the \"rewriting the algorithm in another coordinate system to save memory footprint\" part -- as I discussed above, this doesn't make much sense. Don't mention it. Save some space.\n\n2. One big flaw in the experiments, is that the \"$\\alpha$\" hyper-parameter in INNAProp is almost the same as the $\\lambda$ hyper-parameter for weight-decay. So the settings in this paper, which align $\\lambda$ for INNAProp and AdamW, but tune $\\alpha$ separately, is not actually a \"head-to-head\" comparison between INNAProp and AdamW. This discrepancy seems to manifest in the ImageNet experiments in Section 3.2: the one performing better on training set performs worse on test set, and vice versa -- we know that ImageNet data is noisy and it requires very strong weight decay to achieve the best test accuracy -- so these results probably just reflect the difference of weight-decay strength; they don't indicate either INNAProp or AdamW is better.\n\n3. More generally, on ImageNet with ResNet and ViT, there are certain numbers which we refer to as SOTA -- one has to achieve those numbers in order to make the results convincing. It's unlikely that INNAProp can achieve a better number than the best tuned AdamW on these tasks, but there is a chance that INNAProp can have a traning loss curve that shows faster convergence. The [LION paper](https://arxiv.org/abs/2302.06675) and its repo can be a good place to find the SOTA settings. For Transformer LMs, one needs to carefully determine the model size and training steps, and do thorough hyper-parameter tuning to make the results convincing -- the default settings in most open-sourced repos are far from optimal. The [Chinchilla paper](https://arxiv.org/abs/2203.15556) is the scheme to follow for such evaluation. I understand this is not easy; it requires a lot of experiments to write a convincing paper on a new optimizer. But this is the way it is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically investigates an optimizer which combines the Inertial Newton Algorithm (INNA, [Castera et al., JMLR 2021](https://www.jmlr.org/papers/volume22/19-1024/19-1024.pdf))) with RMSProp. The Inertial Newton Algorithm can be summarized as below:\n\n$$\\theta_{t+1}=\\theta_t + \\gamma_t((\\frac{1}{\\beta}-\\alpha)\\theta_t-\\frac{1}{\\beta}\\psi_t - \\beta g_t)$$\n\n$$\\psi_{t+1}=\\psi_t + \\gamma_t((\\frac{1}{\\beta}-\\alpha)\\theta_t-\\frac{1}{\\beta}\\psi_t)$$\n\nwhere $t$ is the step, $\\theta_t$ the model parameter, $g_t$ the gradient, and $\\gamma_t$ the learning-rate; $\\psi_t$ is a slot variable which tracks a decayed running average of the model parameter, and $\\alpha$, $\\beta$ are hyper-parameters.\n\nIntuitively, INNA is considered as a variant of SGD with heavy-ball momentum and weight-decay. More precisely, in the above, $-\\alpha\\theta_t$ which comes from the \"friction\" term in INNA's framework, is corresponding to weight-decay; and the $\\frac{1}{\\beta}(\\theta_t-\\psi_t)$ term can be considered as some kind of momentum, because it calculates the difference between the current model parameter and a running average of previous ones, and uses it to achieve an \"inertia\" effect in the update. INNA is heuristically derived from a 2nd order ODE as an analogue to Newtonian Mechanics.\n\nBased on INNA, this paper proposes INNAProp, by replacing the $g_t$ term with a RMSProp-style normalized gradient:\n\n$$\\theta_{t+1}=\\theta_t + \\gamma_t((\\frac{1}{\\beta}-\\alpha)\\theta_t-\\frac{1}{\\beta}\\psi_t - \\beta \\frac{g_t}{\\sqrt{\\hat{\\nu}_{t+1}}+\\epsilon})$$\n\n$$\\psi_{t+1}=\\psi_t + \\gamma_t((\\frac{1}{\\beta}-\\alpha)\\theta_t-\\frac{1}{\\beta}\\psi_t)$$\n\nwhere $\\hat{\\nu}_{t}$ is the bias-corrected running average of gradient squares, as usual. So INNAProp can be viewed as a variant of AdamW, of which the (first) momentum and (decoupled) weight-decay are implemented in INNA's style.\n\nIn addition, this paper proposes to rearrange the formula above, by updating $\\psi_{t+1}$ before $\\theta_{t+1}$, and using $\\psi_{t+1}$ instead of $\\psi_{t}$ when updating $\\theta_{t+1}$. As below:\n\n$$\\psi_{t+1}=\\psi_t + \\gamma_t((\\frac{1}{\\beta}-\\alpha)\\theta_t-\\frac{1}{\\beta}\\psi_t)$$\n\n$$\\theta_{t+1}=\\theta_t + \\gamma_t(\\frac{1-\\alpha\\beta}{\\beta-\\gamma_t}\\theta_t-\\frac{1}{\\beta-\\gamma_t}\\psi_{t+1} - \\beta \\frac{g_t}{\\sqrt{\\hat{\\nu}_{t+1}}+\\epsilon})$$\n\nAlthough this rearrangement is mathematically equivalent, the authors claim that \"it can save the memory footprint of the algorithm by avoiding the storage of $\\psi_t$\". However, I don't believe this rearrangement can achieve any observable effect in saving the memory footprint in practice. (Because, what this rearrangement saves is the temporary scratch space of one tensor; during update, the computer is going to calculate the updates for a lot of tensors, while the temporary scratch space is reused. So the effect of this rearrangement is negligible. -- If the authors have different opinions, please show me an ablation experiment comparing the memory usage and running speed.)", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides revision of the theoretical writing and more convincing experimental results, as described above, I'm also curious about how INNAProp performs with the [original Transformer on WMT](https://arxiv.org/abs/1706.03762). The original Transformer, especially with PostNorm, usually requires momentum and learning-rate warmup to be trained properly; it should be good to know if the INNA-style momentum can achieve the same on the Transformer-WMT task. A good repo where one can find the SOTA Transformer-WMT settings is [init2winit](https://github.com/google/init2winit).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wxRLnJjPCk", "reviewer": "Reviewer_Coz7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The algorithm design is very clearly motivated and easy to follow: it combines mechanisms from two directions (INNA and RMSProp) with the hope of getting the best of the two worlds.\n* The experimental evaluation is very fair and thorough work. It covers several data domains and model architectures, runs multiple seeds, and properly accounts for hyperparameter tuning.\n\nWeaknesses:\n\n* The main question I had after reading the paper is: what exactly is the motivation to propose yet another optimization method? As mentioned above, the steps of the derivation are clear, and the experiments also are insightful. From the current version, it seems that the method lacks either a theoretical access or a clear empirical advantage over AdamW. \n\n* The method seems to be quite sensitive to the choice of $\\alpha$ and $\\beta$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new optimizer which combines the core concepts of INNA and RMSProp/Adagrad style preconditioning of the gradient.\nAn extensive experimental evaluation the performance of the optimizer in comparison to AdamW on machine learning tasks is provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This section is more \"Questions\":\n\n* You mention other recent optimizers (ScheduleFree, Muon etc.). It would be interesting to see one of these methods for the GPT-2 tasks in comparison, where INNAprop performs very well. For me it would be sufficient to reuse the reported hyperparameter values of these methods, as the setup should be very similar and one can assume that the reported values are tuned. (If INNAprop performs worse than those methods, this would not alter my rating of the submission.)\n\n* Rather unrelated to the proposed method: in Fig 5 AdamW seems to be better than Sophia, however the original Sophia paper claims a big advantage over AdamW. Can you comment on this phenomenon? Is it because you were not able to reproduce the (supposedly) superior performance of Sophia, or because you tuned AdamW better than the baseline from the Sophia paper?\n\n* From the Resnet50/Imagenet plot, it seems that green performs very well in the beginning, but only blue is able to match the final acccuracy of AdamW. Have you tried to schedule $\\alpha$ and $\\beta$ over training in order to get fast initial convergence, and good final performance? \n\nMinor:\n\n* is the term \"geometrical intelligence\" standardly used? I am not sure what it means.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)