In Search of Adam’s Secret Sauce
Authors: Antonio Orvieto
Robert M. Gower
Venue: NeurIPS 2025 oral ()
Keywords: Adam
Language Models
Transformers
Adaptive Methods
Optimization
Primary area: deep_learning
TL;DR: Adam with equal betas works well, and its form can be simplified for further insights

Abstract: Understanding the remarkable efficacy of Adam when training transformer-based language models has become a central research topic within the optimization community. To gain deeper insights, several simplifications of Adam have been proposed, such as the signed gradient and signed momentum methods. In this work, we conduct an extensive empirical study — training over 1,500 language models across different data configurations and scales — comparing Adam to several known simplified variants. We find that signed momentum methods are faster than SGD, but consistently underperform relative to Adam, even after careful tuning of momentum, clipping setting and learning rates. However, our analysis reveals a compelling option that preserves near-optimal performance while allowing for new insightful reformulations: constraining the Adam momentum parameters to be equal, $\beta_1=\beta_2$. Beyond robust performance, this choice affords new theoretical insights, highlights the "secret sauce" on top of signed momentum, and grants a precise statistical interpretation: we show that Adam in this setting implements a natural online algorithm for estimating the mean and variance of gradients—one that arises from a mean-field Gaussian variational inference perspective.

In Search of Adam's Secret Sauce
Antonio Orvieto ELLIS Institute Tübingen, MPI-IS Tübingen AI Center, Germany
Robert M. Gower
CCM, Flatiron Institute, Simons Foundation New York, US
Abstract
Understanding the remarkable efficacy of Adam when training transformer-based language models has become central research topic within the optimization community. To gain deeper insights, several simplifications of Adam have been proposed, such as the signed gradient and signed momentum methods. In this work, we conduct an extensive empirical study training over 1,500 language models across different data configurations and scales - comparing Adam to several known simplified variants. We find that signed momentum methods are faster than SGD, but consistently underperform relative to Adam, even after careful tuning of momentum, clipping setting and learning rates. However, our analysis reveals compelling option that preserves near-optimal performance while allowing for new insightful reformulations: constraining the Adam momentum parameters to be equal, 31 82. Beyond robust performance, this choice affords new theoretical insights, highlights the "secret sauce" on top of signed momentum, and grants a precise statistical interpretation:
show that Adam in this setting implements natural online algorithm for estimating the mean and variance of gradients one that arises from mean-fieed Gaussian variational inference perspective.
Introduction
Despite decade of research into efficient and performant adap- tive optimizers for deep learning, the de facto choice for large- scale training today remains Adam [Kingma and Ba, 2014], especially for training language models (LMs) [Grattafiori et al., 2024, Liu et al., 2024]. At the root of this choice is the peculiar geometry of optimization landscapes induced by the transformer architecture [Noci et al., 2022, Zhang et al.,
160M params.. All methods tuned best.
29.
Signum+wwd Ir decay to 3.28 tok
28
AdamW, decay to
AdammW, decay to 2.4B tok
pag Perpleetty
26
25
2024a], well the noisy/unbalanced nature of tokenized text data [Zhang et al., 2020a, Kunstner et al., 2024].
24
25% slower
22
In recent years, the surge of extremely large-scale and expensive-to-preet -er-ttrr--rerinn-- language models has further pushed the community to better understand Adam's 's performance and to pro- pose faster, efficient, and robust alternatives. Towards achieving this goal, contemporary studies [Kunstner et al., 2023, Bernstein and Newhouse, 2024] have brought up a close similarity be- tween the performance of Adam and SignSCD [Bernstein et al., 2018] with momentum. While such results are extremely valu- understanding. they are precise enough
21
2.0
2.5
3.0
seen tokens (B)
Pretraining on SlimPa- jama with Chinchilla-optimal [Hoff- mann et al., 2022] scaling. Both mo- mentum and learning rates for Signun extensively tuned ($3). While Sägnum closes 96% of the perplexity gap between Adam and SCD with mo- mentum (Table 1), still results in 25% slowdown Adam achieves the performance with 3/4 of the budget.
Figure
able forward already at a scale of 160M parameters we found that extensive tuning of Signum (SignSGD with momentum), while closing 96% of the perplexity gap between SGD and Adam, results in 25% effective slowdown (Figure l).
antonioQtue. ellis.eu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: (Signum closes 96% of the perplexity gap between Adam and SGD) Validation perplexity comparison of widely used optimizers that interpolate between SGD and A Aaam evaluated on language modeling task (160M parameters, 3.2B SlimPujama tokens, sequence length 2048, batch size 256 Chinchilla optimal). We report the mean and 2-sigma interval of validation perplexity (on 100M held-out tokens) across 3 initialization seeds. Weight decay is always decoupled [Loshchilov and Hutter, 2019] and set to 0.1 [Biderman et al., 2023, Liu et al., 2024] except for SCD where we further tune (SB). RHSprop does not use momentum, and Gclip is global norm clipping 1 (before applying momentuan) Cclip is coordinate-wise clipping (after applying momentum). Other hyperparameters, for all other methods, are carefully tuned, see e.g. Figure 2 and $3.
To optimally tune hyperparameters (e.g. Figure 2), we performed a total of 582 full training runs.
Adam
Si Signum
RMSprop
SGD+Cclip
SignSGD
SCD+Gelip
SGD
Val ppl. 21.86 0.21 23.23+ 0.16 27.04 0.34
33,40# 0.39
0.57 37.76= 0.61 53.62+ 5.14
While for large-scale training, the slowdown in Figure 1 is not acceptable, it may seem unnecessary or anachronistic to further explain it, in light of recent algorithms claiming to have further improved the performance of Adam, e.g. Muon [Jordan et al., 2024, Liu et al., 2025, Shah et al., 2025], Scion [Pethick al., 2025], and Shampoo-based [Gupta et al., 2018] methods such as SOAP [Vyas et al., 2025]. However, a close inspection of such optimizers reveals that, while gains over vanilla Adam are solid, most of these methods still use A dam on a specific subset of parameters: For instance, in recent scaled-up versions of Muon [Liu et al., 2025, Shah et al., 2025], Adam is used to update embedding, LM heads and normalization parameters 2, and on the other parameters the Muon update is normalized to have a similar RMS value similar to the Adam update. Further, SOAP's improvements stem from the application of Adam the preconditioner's eigenbasis.
The discussion above and the results in Figure inspires us to further dissect- - once again [Balles and Hennig, 2018] - the mechanisms of Adam compared to those of simpler methods in language modeling with transformers.
Towards improving our understanding of Adam, we make the following contributions:
We perform a large-scale evaluation (~ 10 thousand NVIDIA A A100-SXM4-80GB GPU hours) of the performance of established algorithms which claim a theoretical empirical similar- ity/dissimilarity with Adam on 160M parameters LMs with usual configurations [Biderman et al., 2023, Black al., 2022], at a compute-optimal budget on different datasets, at different batch-sizes and sequence lengths (up to 2048 tokens). Crucially, we sweep over all momentum parameters for each method, for each learning rate in our grid for each of our settings. We find that, while clipping and sign descent methods can close most of the gap with SGD, their performance is not satisfactory in comparison to Adam (Figure 2). We make all of our data, e.g. loss dynamics for all our settings, publicly available at https: //gtthub com cww/aooviiiieeeeeeeeeoc.oooooeee corvieto//a wwtaaaett/eerreeeteeeeeeeeeeeeee SecretSauce.
Through our extensive tuning of Adam (e.g., Figure 2, comprising 200 distinct hyperparameter settings), we identify one simplification that does perform well: that of setting 31 = B2 (emerging practical choice in contemporary literature [Zhao et al., 2025, Shah et al., 2025, Cattaneo and Shigida, 2025, Zhang et al., 2025]). We validate this finding ($3.2) at different batchsizes, data source, token budget, sequence length and larger scale (410M): 31 = 33 performs at near cctr-optimality across the majority of our experiments, see Figure 3. Given the breadth of our evaluation and the robustness of this finding, we recommend adopting Bn f3 as the default setting for Adam for training language models at similar data and parameter scales. More broadly, this perspective one one-parameter optimizer (as Signum).
suggests that Adam can be effectively treated
We show in $4, that reducing 31 = B2_ 3 to a single parameter, leads to a surprising new interpretation of Adam: it is built on top of nontrivial yet principled online method for estimating mean and variance of the gradients. Indeed, we can view the two momentum buffers as the result of an online Gaussian Variational inference method for tracking the mean and variance of the gradients as they change across iterations. This viewpoint directly adds to the discussion by Balles and Hennig [2018], yet affords more precision induced by our empirically-infrmee simplification.
We offer toy quadratic example illustrative of our findings, building on top of recent works on the peculiar landscape of transformer-base
language modeling problems [Noci et al., 2022, Zhang et al., 2024a]. This example replicates the gaps between tuned SGD, Signum, and Adam with equal betas in a 9-dimensional setting, helpful for future research and to gain intuition.
*Coincidentally, the ones that were shown to be most sensitive during training [Zhao et al., 2025, Kunstner et al., 2024]. Scion claims a greater independence from Adam, yet adopts an architecture where normalization layers have trainable gain parameters. While results are promising, experiments in the usual setup are needed.
2

---

Signum+wd, Signumeed, Signumewd. Signumeed Signum-wd, 0.9875
28
dlp - True
dlip Falso
23
learning
learning
leaming
leamning
learning rate
AdamW. 9: = 0.0
AdamiW, B: 0.9
AdamiW, P: 0.95
AdamW. 0 0.975
AdamiW, 0.9875
23.5
Signum
82 0.6
A:
23.0
test
P: 0.95
22.5
P2 0.375
equal betas
9: 0.9875
best with equal
92 0.99375
22.0
0.996875
best
equal
21.5
10
10
10
10
10
10 10
10
10
10
learning
learning
leatnnng
leaming
Figure 2: Training total of 265 language models with 160M parameters with 3.2B SlimPajama-Be tokens, sequence length of 2048, butch size of 256. Shown is the final test perplexity 100M held-out tokens. Some underperforming runs
shown keep focus the most interesting range. For careful dexcription of our tuning grid, see $A. Takeaway I: Validation perplexity of highly tuned (65 hyperparameter configurations) gigguum with weight decay 0.1 top row is around 23.23 (see Table for multiple seeds at optimal tuming). We ablate on the momentum parameter, learning rate, and presence of global clipping before averaging. The green horizontal line the second row (200 Adam runs, with
best performance of Signum is reported weight decay of 0.1).. Most Adam runs perform better than optimally tuned Sägnum Takeaway 2: For each B1. the optimal corresponding f3 (after learning rate tuning) similar. The higher B1, the higher B3 for optimal performance (optimal Bs are correlated).
Preliminaries and Related Works
For signal (sk)REM and | [0, 1), define the 3-normalized exponential moving average:
g [sk] = S + B)sk.
EMAg [so] == So (or zero).
(1)
The Adam optimizer [Kingma and Ba, 2014] without bias correction takes the following form:
?) VEMA,,, [o +
Wk++1
wy
EMA [gh]]
(Adam)
where all division and multiplications element- wise, Wk. 9k Idd are model parameters and gradi- ents at iteration k me the scheduled learning rate, and 6 > 0 is a small constant. RMSprop [Tiellman and Hinton, 2012] is earlier method that sets B1 0.
One special case, and simplification, Adam is to consider 31 B3 = = which gives SignSGD:
Wk++
maiinggg))
(SignSGD)
A practical variant of SignSGD, which has shown strong performance in language modeling [Kunstner al., 2023]. first computes an exponential moving average (EMA) or momentum of the gradients before applying the sign operator [Bernstein et al., 2018]:
Wi++1
W/ s) EMA [o]]]-
(Signum)
In practice, every language modeling pipeline (see e.g. (Karpathy, 2022]) incorporates some gradient clipping strategy [Pascanu et al., 2013]. a component known to stabilize training in the autoregressive
'e show in Table 3 that the presence of bias correction does not affect our results at the best hyperparameter configuration. However, for all runs, the Pytorch implementation including this factor, for simplicity.

---

setting and to make gradients more robust to the stochasticity of language IZhang et al., 2020b].
Global norm clipping (that we abbreviate Gclip), processes gradients fresh out of the backward pass:
Gclip]ax] min
1 9k.
In our experiments, we start from vanilla SGD with momentum: Wk+1 = wy - EMA [gk]] and ablate on the positive effect of Gelip before applying momentum. Regarding coordinate clipping (Cclip), softer version of sign, we consider applying it to EMA, in connection with Signum.
Research on Adam, a short summary. Despite initial concerns on generalization [Wilson et al., 2017] and convergence [Reddi et al., 2018], after the introduction of decoupled weight decay (i.e.. AdamW [Loshchilov and Hutter, 2019] 9J) Adam rapidly became the de-facto standard optimizer in deep learning, with works highlighting its landscape adaptation properties [Orvieto al., 2022] and its debated connections to empirical Fisher preconditioning [Kunstner et al., 2019].
With the advent of Transformers [ [Vaswani et al., 2017], early works noticed an intriguing gap with SGD performance language modeling [Xiong et al., 2020] (much larger than what can be observed, e.g., in CNNs on image data), that was initially attributed to heavy-tail noise in text data [Simsekli et al., 2019, Zhang et al., 2020a] suggesting Adam performance to be correlated with its adaptive coordinate clipping mechanism [Zhang et al., 2020a].
As models became larger and more hardwaare-demanding, interest spiked in the community to reduce the memory footprint of Adam [Li et al., 2023, Zhang et al., 2024b] and to search for more efficient options [Chen al., 2023, Liu et al., 2023]. Current trends, draw an intriguing connection between Adam and SignSCD [Bernstein and Newhouse, 2024], and in particular with its momentum variant: Signum [Berstein al., 2018]. This connection was first suggested in early attempts to understand Adam's 's empirical performance [Balles and Hennig, 2018], and has recently gained renewed attention in light of transformer architectures and their heterogeneous optimization landscapes [Noci et al., 2022, Zhang et al., 2024a, Tomihari and Sato, 2025, Kunstner et al., 2024, Zhao et al., 2025]. These landscape- based arguments are now more compelling, as recent evidence shows that Adam and signed momentum methods outperform SGD even in deterministic settings [Kunstner et al., 2023].
Our approach. Although recent literature highlights many connections between Adam and simpler methods such as Signum-wii which involve fewer hyperparameters, the computational demands of thoroughly studying Adam on small- to medium-scale language models remain prohibitive for most academic optimization researchers. This challenge is amplified by the combinatorial explosion of hyperparameter configurations required for rigorous comparisons. In $3, we aim to provide a comprehensive empirical reference for optimizer performance across a range of language modeling settings. Our key findings are distilled into two main takeaways (Figure 2), which are further supported by theoretical insights in $4.
Experiments
In our experiments, we systematically explore Transformer-basel language models using nanoGPT [Karpathy, 2022] implementation" enhanced by recent advancements such as Rotational Positional Embeddings [Su et al., 2024], RMSNorm normalization [Zhang and Sennrich, 2019], and SwiGLU activation functions [Shazeer, 2020]. We adopt a robust training protocol inspired by successful practices established in large language models like LLaMa [Touvron et al., 2023], GPT-neox [Black et al., 2022], GPT-J [Wang and Komatsuzaki, 2022] and Pythia [Biderman et al., 2023], leveraging techniques including bfloat 16 precision, linear warm-up followed by a cosine an- nealing schedule [Loshchilov and Hutter, 2016], and global gradient norm clipping (unless specified). Our model configurations follow [Biderman et al., 2023] and are presented, alongside detailed description of all tuning settings and resources, in SAA
3.1 Extensive benchmarking at 160M parameters
We conduct 475 compute-optimal pretraining runs on the SlimPajama-627 dataset [Soboleva et al., 2023], using a sequence length of 2048, batch size of 256, and decoupled weight decay of 0.1 [Loshchilov and Hutter, 2019] (except for SGD). We always report validation perplexity on

---

held-out subset of 100M tokens. Results from these tuning sweeps are summarized in Table 1, Figure 2, and Appendix B.1. The runs span the following configurations:
SGD (131 runs): Tuned parameters include weight decay (too large causes instability), global norm clipping (Gelip). We also consider clipping coordinates after applying momentum (Cclip). For all these options, momentum and learning rates are independently tuned.
RMSprop (48 runs): Tuned parameters include momentum the preconditioner and learning rate.
Signum (70 runs): Tuned parameters include global norm clipping, momentum, and learning rate.
Momentum on SignSGD (35 runs): This variant inverts the order of the sign and EMA operations (and performs worse than Signum). Clipping has no effect here due the sign operation.
AdamW (200 runs): Tuned parameters include both momentum terms and the learning rate.
Two additional seeds are provided for the best performing hyperparameter settings, see Table. 1.
Choice for betas grid. While we vary the learning rate by powers of two, our choice of moving average parameters is guided by recent insights into Adam scaling behavior [Malladi et al., 2022, Com- pagnoni al., 2025]: we choose 3 = 1 1-x(1- S where Bbase 0.9 and E {2 15- 2- ...., 2²). This makes it such that the accumulation factor 1/(1 - 3) (/( Bmase))
Takeaway 1. As shown in Figure and Table 1, optimally tuning Signum with weight decay leads to significant improvements over standard SGD, in line with recent findings [Kunstner et al., 2023, Zhao et al., 2025]. Nonetheless, Adam consistently outperforms the alternatives across most settings, suggesting that it retains a key advantage-a "secret that continues to set it apart from better-understood methods in large-scale optimization tasks.
This gap is not limited to this specific setup. In $3.2 we discuss results on another dataset (Fineweb), with disabled weight decay, and shorter sequence lengths. Further, we ablate other potential confounders (initialization of moving averages, bias corrections, Adam value) in $3.3.
Takeaway 2 (a). In Figure 2, we clearly see that 31 Ag yields near-optimal performance in Adam, for the five 31 values we con- sidered. In $ 3.2 we show similar results different batch sizes, different sequence lengths, and with disabled weight decay on a dif- ferent dataset. We also extend this observation to 410M parameters models (Figure 5). This empirical finding serves as a basis for our theory in $4.
batch size 512
-0
0.975
batch size 256
0
0.95
Takeaway 2 (b). As corollary to Takeaway 2, Figure 3 shows
batch 128
-
that the best performance is not only achieved when 31 B2, but
0.975
-.08
also improves as the two values become closer. Among 500 runs on
0.95
0.9
models, we observe a clear correlation: lower loss is associated with smaller differences between 3, and B3. This suggests that gradient smoothing (31) and preconditioner smooth- ing (32) should not be treated as independent operations-optinal performance often arises when they act in concert.
09
0.755
Figure 3: Summary of the results in $B.4. At different batch sizes, for each $1 E [0.9, 0.95, 0.975), show the best-performing B9 (highest score, yellow) and the gap between its performance and that of other options in the grid. We notice high correla- tion between beta values (e.g.. B2 0.9875 is a terrible option at B1 = 0.9, but a good one at 81 = 0.975). While results are noisy, notice that 8 = Ba never degrades performance 0.3 points. contrast (Table 1. the gap with Signum can be as high as 1.37 points.
To put to the test our second takeaway in different training settings, we consider shorter sequence lengths (512, Figure 14), higher/lower batch sizes (Figure 16 & Figure 17), different data (Fineweb) and absence of weight decay (Fig, 18). See discussion in $3.2.
Standard choice for betas. While in standard deep learning (also Pytorch default) 3g > 31 (0.999, 0.9), in language modeling the choice 8, 0.9, B2 0.95 is much more common. A lower value for 32 was shown to help mitigate loss spikes [Cattaneo and Shigida, 2025, Compagnoni et al., 2025], and several recent studies have started to adopt 31 = B2 0.95 as default [Zhao et al., 2025, Shah et al.. 2025, Zhang et al., 2025]. All our findings confirm this choice for tuning (see e.g. Figure 2), of which we evaluate validity extensively for several values of 31.
than
5

---

AdamW. A, 0.9
AdomW, By 0.95
AdamW. By 0.975
16.4
16.3
9: 0.8
final test ppl
16.2
P2 0.9
80
16.1
P: 0.975
16.0
best with
equal betas
9b 0.9875
Pb 0.99375
15.9
equal betas
best with equal betas
10
10
10-2
10 10
10
learning rale
learning
learning rale
Figure 5: The final validation performance (100M held-out tokens) for 44 trained LMS with 410M parameters trained on B SlimPajana tokens (Chinchilla--e oppiimalll--.ama....... Equal betas yields near-optimal performance. gradient clipping and a batch size of 512 (scaled by 2 compared to Figure 2, as suggexted by Zhang et al. [2025]). Sequence length is 2048, weight decay is 0.1. Note that the standard setting (0.9, 0.95) is quite suboptimal here.
use
also noted first by Reddi et al. [2018], Alacaoglu et al. [2020] for AMSgrad, and later by Zhang et al. [2022] for Adam, where is shown that if 82 is large enough and [3 < vB2 it converges to the neighborhood of critical points. Further, Xie and Li [2024] showed that weight decay in Adamw leads convergence constrained minimizer only if 32 $1.
Theoretical relations between betas. We note that correlation between 3 parameters
3.2 Ablations
More Tokens. We find our Takeaway also hold at a higher token budget. In $B.2, we show trend very similar to Fig. 2 for models trained for x the Chinchilla-optimal budget.
Different batch size. We find our Takeaway 2 to be robust to batch size. In the same setting as Figure yet at a slightly lower compute budget due to hardware limitations (2.5B parameters), we find that, even batch size 128 and 512 the choice 31 82 yields near-optimal performance. This step involves training 500 models, see $B.4 for visualizations similar to Figure 2 and a discussion.
Different sequence length. In $B.3, we find Takeaway 2 to also hold at shorter sequence length of 512 (Figure 14). We note that here performance of Signum is closer to that of Adam compared to Figure 2 yet, Adam is still superior by substantial margin ( 00.7 validation perplexity), Takeaway This pattern agrees well with the results in [Zhao et al., 2025], who found other methods to be competitive with Adam at short context lengths. Our experiments in Figure 14 and Figure 2 suggest that Adam performance particularly shines at higher sequence lengths.
Different data and weight decay. In Figure 18 we test both Takeaway 1 and Takeaway 2 on Fineweb [Penedo et al., 2024]. We take this opportunity to also deactivate weight decay () = 0), as the optimal Signum learning rates in Figure 2 suggest decoupled weight decay = Aryw acts differently for the two methods, likely needing different tuning. When deactivated, we still see substantial gap between Signum and Adam, as well as strong performance with equal betas.
Larger Models. We restrict our attention the SlimPajama dataset and to validation of Take- away Results presented in Figure 5, com- prising 44 full compute-opy -pptmmll training runs of 410M parameter models, which confirm yet again strong and near-optimal performance at 31 = .2.
24.0
11c-00000
23.8
vanilla (G)
=1e-09 (Gi)
== (GI)
vanila (21)
23.4
=1e (2))
23.2
3.3 Checking for confounders
2x110 3x100"4H 6
10-3
rate
When comparing Signum with Adam, here are few confounders that might affect results:
Figure 4: Adding an mollifier to Stgnum i.e., using ma/( mm e) offered little no improvement. We also both initialization (ZI) and gradient ini- tialization (GI) for m, and find similar results with no significant improvement. e = le is significantly worse, hence is not shown. Similar finding Figure 7,
The value of in Adam was shown to be im- portant numerical stability, and might affect performance (uuan and Gao, 2020]. We show in Table 2 that one can choose an extremely small value in our setting. We cross-check the impact of including an factor in Signum: we found that

---

little can be gained from this strategy (Figure 4). In short, we found that is not crucial parameter our setup. This is also liked our findings on adaptive mollifiers, cf. $4.
Initialization of moving average parameters. In Figure 4 we also ablate on initialization of the moving average in Signum and found substantial differences. We perform this same ablation for Adam and report comprehensive results with seeds in $B.6.
Bias correction. While bias correction in Adam is helpful in early training, final validation perfor- mance is almost unchanged, see the full training curve and results with seeds BB.6.
Table Effect of in AdamV- other purumeters optimally tuned for = 10 (setting: Figure 2). All values between E [10 10 18] result in similar validation perplexity.
= le -3 3
6= le
6
6=
Le
8
6=
Le
10
c le
12 c le 15
Val ppl
23.341 0.31 21.56. 21.86.1 0.21
21.87.1 0.04
21.89 0.2
21.911 0.18
New Viewpoints of Adam
We now show that restricting to the case 31 32
yields useful interpretation of Adam. Since the Adam update is coordinate-wise it suffices to analyze a single scalar gradient 9k E R. Moreover, ablations (Table 2, Table 3) indicate that neither the e-term nor the bias correction significantly affect performance. Thus, for clarity, we set c = and study the simplified Adam update:
[94
d2
(2)
We next rewrite (proof in the Appendix) the update to explicitly highlight the role of variance. Proposition 1. Let m. EMAg(gu]. Then the update (2) admits the equivalent representation:
mi
dx
(3)
m
EMA (m gk )2]
This shows that the denominator depends on the exponential moving average the squared deviation between the momentum m and the incoming gradients Sk, with an interesting multiplier B. As we demonstrate in the next section, this quantity is in fact an online estimator of the gradient variance.
4.1 Adam Estimates Mean and Variance using Variational Inference
We show that Adam admits a natural interpretation as an online variational inference method, where
m& == [MA and o = [M - )
correspond to online estimates of the mean and variance of the stochastic gradients. We reintroduce Adam through this lens.
Suppose we are given a sequence of stochastic gradients {91...., J. where each is sampled from an unknown Gaussian distribution whose mean and variance may vary with Rather than taking steps directly along these noisy gradients, we aim to estimate their mean and variance online and use these estimates define a more informed search direction.
At iteration k; let (my, o denote our current estimates of the gradient mean and variance, respec- tively. Upon receiving new gradient sample gk+1 ~ (M,m o²) with unknown (m, o²), we wish to update our estimates to (m++1, x that it becomes more likely that g6+1 ((m++1,
drawn from
Since we also expect the underlying distribution to vary slowly over time, prefer that (mm++1, 1) remain close the previous estimate (m,,, o.).. These two goals-fitting the new observation and ensuring smooth apdates-cal be traded off via a regularized maximum likelihood problem:
m, min log p mmm o²)+ + KL (N((s, of) (m),))))) m,
7

---

where P(gk+1 ) is the Gaussian likelihood, A 2 0 is regularization parameter, and KL
denotes the Kullback -eiblec Leibler divergence:
m. m, -logplok+1
log 0² + 2g (1++1 - -m)²
(5)
KL (N(m,, [I [oo
(m+ m)² log of o² ]
(6)
0²
o²
The following result, proved in the appendix, characterizes the solution of (4), showing that the moving averages used in Adam correspond exactly to instance of online variational inference:
Theorem 4.1. Let 3 = 1+A Then the solution to the optimization problem (4) is given by
m 3m, + (1 8) EMA [gk+]]
(7)
o 3o B(1 33) 9k+1) 2 EMAs [(m. gk+1)
(8)
As consequence, the Adam update direction in (3) can be rewritten
my
sign (m²) Vmg + V1+ m
mi
d2 m SEMA (ma-1 9k
(9)
This shows that Adam may be interpreted as an adaptive mollified variant of Signum, where the mollification depends on the local noise-to-signal ratio. This mollified viewpoint aligns well with one of the first papers on understanding Adam [Balles and Hennig, 2018]. as discussed after Proposition 1.
Using these insights, we can better formalize the noise-toe signal interpretation of Adam [Balles and Hennig, 2018] (see also $4.2). Let ma/ax denote the signal-to-noise ratio (SNR). We show that Adam can be viewed steepest descent method whose trust region is modulated by the SNR.
To build this connection, consider first the Signum update. It corresponds to the steepest descent direction under lx -norm constraint [Balles and Hennig, 2018], solving
-m))n(m)) = argmin m- subject to [0] < 1.
(10)
DER
That is, Signum selects the direction most aligned with -m within unit trust region.
In contrast, Adam can be interpreted as steepest descent method with variable trust region, defined by the (inverse) signal-to-noise ratio:
sign(me)
argmin my subject to 10| S VT+ PER
(11)
VT+0//
Here, the effective step size shrinks when the noise dominates the signal, and expands toward 1 as uncertainty decreases, In this sense, Adam adapts its update magnitude according to a confidence- weighted trust region.
4.2 Comparison with Balles and Hennig [2018)
Balles and Hennig [2018] first drew a connection between Adam, Signum and Signal-to-noise Ratio regularization. Their observation as follows. Let m. EMA,, [gk]] and EMAA,, lo2]. We can trivially re-write the Adam direction as
mk
m&
de
/Uk
/mg
m
If now assume that of :- m is measure of variance, then dividing the Adam direction through by (mz]) as done in (9), we arrive at a Signal-to-noise Ratio regularized variant the Signum method. In particular, the noise goes to zero (o + 0), we arrive at the Si Signum method.
The missing piece in their insight was to show when and if the term v - -
? is measure variance.
We show that 31 = B2, a choice that was not common the time of Balles and Hennig [2018], allows for more precise claims: Proposition shows that when 81 ;3 = 3 the term - -mī is
Default parameters have for long been B1 = 0.9, 92 0.999, see :/docs://doc... http://cocsppp.cc.h.oocoo org /ocs/ stable/ htwpl///geeeratee/ttoooo optim Adam. html.
8

---

precisely equal to DEMAA[[mx-1111 ) which in turn is a online estimate of variance (Theorem 4.1).
We further show that Ui m only has a precise variance interpretation for the case 11 :2 indeed, we prove in SC.2 that Adam can be represented
dx
(12)
V m + EMA, [amm bgk) )1
for some a, v R and 7 E (0, 1) if and onty if 81 32. In other words, connecting 2 m to variance estimation, and turn Adam to an SNR-controlled trust region method (11), can only be done precisely for the case of equal betas.
Ablating hyperparameters in our reformulation. While (12) reduces to Adam with equal betas if and only if a, b 1 and 3 = y T, we found it interesting to consider (12), with b 1, as a new method with no precise connection to simultaneous variance and mean estimation, with hyperparameters 8, T. In $C.4, we train 150 additional language models ablating such parameters, and found no advantage in setting 3 * r or T # 7. We believe such evidence further strengthens our claims: best performance is aligned to the theoretical choice 7 = B.
5
Why an adaptive trust region? Insights from heterogeneous quadraties
Homegenecus Hessian
Heterogeneous Heszian
Adam (equal Bs), Ir=0.003
10a
10a
Adam (equal 6s), 1r=0.01
Adam (equal (s), Ir-0.03
107
10a
5GD (+m), In-3e-55
loss
10:
10a
SGD (+m), Ir=000001
SGD (+m), In-0,0003
10-1_
10-1_
Signum, Ir=0,003
Signum, In=0.01
10-1
10-3
Signum, Ir-0.03
100
200
300
400
100
200
400
500
iteration
iteration
10a
10a
dim. 0 dim dim. dim.
Eigervalues (both Hessians)
EMA(Imc-3 -0,7)
10
105
block
3
block
102
block
dim.
10: 1.
1-
dim.
dim.
dim.
10
10
100
200
300
400
500
0
100
200
300
400
500
Iterationn
Iteration
Figure 6: Top row: Training performance (median and 25%/75% quantiles over 10 seeds) of SGD, Signum and Adam on two 9-dimensional convex quadratic problems ($D) inspired by Zhang et |2024a]. All optimizers use moving average parameters set to 0.95, with 10% warmup followed by cosine decay to zero. Both problems share the same Hessian eigenspectrum and have a 3 x 3 block structure. The landscape on the left is homogeneous, with each block containing both large and small eigenvalues. The landscape the right is heterogeneous, with each block having eigenvalues of different magnitudes. In this setting, Adam clearly outperforms SGD, with Signum closing part of the gap. Bottom row: Dynamics of the variance term in Proposition 1. The value of this term varies both across iterations and across blocks, adapting to the local curvature structure. This adaptive behavior improves performance over Signum in the heterogeneous setting.
While our theoretical analysis in $4 offers a new perspective on Adam, it is not tied to any specific architecture. To enhance intuition and provide a controlled setting for future work, we validate our findings on a simplified model of transformer loss landscapes introduced by Zhang et al. [2024a], building signal propagation theory [Noci et al., 2022].
As noted in Zhang et al. [2024a], Kunstner et al. [2024], Zhao et al. [2025], the landscape of autoregressive language models is highly heterogencous: Hessian blocks associated with semantically distinct parameter groups (e.g.. normalization layers, embeddings, or softmax-relatee parameters) exhibit markedly different eigenspectra and thus demand different learning rates. In contrast to homogeneous models (e.g., CNNs), this heterogeneity is where Adam significantly outperforms SGD [ef. Zucchet and Orvieto, 2024].
Figure illustrates this point. On a toy heterogeneous quadratic landscape, tuned Adam with equal 3 values substantially outperforms tuned SGD with momentum, echoing results from Zhang et al.

---

[2024a]. We also observe that Signum closes much of the gap but still falls short of Adam. This is consistent with our findings in Table I for language models.
In Proposition 1, we showed that the key difference be- tween Signum and Adam lies in the variance correction term 9EMA,[[
Fixed mollifier Signum
10a
10.0
9k " in the denominator. Un- derstanding how this term evolves is essential: it can- not be approximated by a constant. In the second row of Figure 6, we observe that the variance estimate not only varies over time, but also differs in scale across the three blocks blocks-mimicking mimicking the parameter groupings in trans- former models. This block-wise variation reinforces the idea that the variance term dynamically adapts to the local curvature and cannot be substituted by fixed value. In Figure and we show similar effect heterogeneous quadratic and language models, respectively: replacing [EOAA[[m==- [m==1 - gk ]"] with a fixed constant cannot pro- vide the same adaptive effect.
104
0.1
10a
10a
10-2
10-4
100
200
300
400
500
lteration
Figure Counterpart of Figure for the heterogeneous quadratic example. We do not observe gains with a fixed mollifier my// / / + e. Placing inside or outside V: has qualitative effect after tuning.
3EMA,
6 Conclusion
We have presented an extensive numerical study of Adam, comparing it against several proposed simplifications. Our main finding is that, on generative language modeling tasks, Adam significantly outperforms these simplified variants. Notably, we observe that setting 31 = _32 is often optimal or near-optimal. Based on this observation, recommend Adam with 31 = f2 simplified model, and we provide new variational inference interpretation for this setting.
Our findings come with some limitations. First, numerical experiments fix grid over the hyperparameters; the results are therefore sensitive to the choice of grid, and different grids may lead to different conclusions. However, for all our hyperparameters, curves demonstrating that we are always at optimality inside the grid (and not at the edge). Second, while 31 32 often performs well, note that at small batch sizes, Figure 3 suggests slight shift. Finally, although Theorem 4.1 shows that Adam's 's two momentum buffers can be interpreted as online estimates of the gradient's mean and variance, it does not explain why these estimates should be arranged into the quotient used in Adam (9). Lemma in [Balles and Hennig, 2018] can provide starting point further dissect this interesting choice and explore alternatives.
show explicitly all tuning
Acknowledgements
We would like to thank Niccolo Ajroldi, Sam Liang, Weronika Ormaniec, and Enea Monzio Com- pagnoni for their comments. We additionally thank the NeurIPS 2025 and ICMI 2025 HiLD workshop reviewers for their valuable feedback and references. Antonio Orvieto acknowledges the financial support of the Hector Foundation, and is thankful for the compute made available by MPI-IS and the Tübingen AI ecosystem.
10

---

References
Ahmet Alacaoglu, Yura Malitsky, Panayotis Mertikopoulos, and Volkan Cevher. A new regret analysis for adam-type algorithms. In International conference on machine learning, pages 202-210. PMLR, 2020.
Lukas Balles and Philipp Hennig. Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients. In ICML, 2018.
Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint 20Xi::2409.20325, 2024.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In ICML, 2018.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. 2022.
Matias D. Cattaneo and Boris Shigida. Tuning adam(w): Default B2 may be too large, 2025. URL bects://mdcatte
gi github..... www.chhw....p...ee.eeeee wwtpipppppe//aattaaaaeeee Shigida 2025_____________11 2aaiila_222____mnnaeee TuningAdam pdf.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et Symbolic discovery optimization algorithms. Advances in neural information processing systems, 36:49205-4999 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of SDEs: Theoretical insights the role of noise. In ICLR, 2025.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with oo-awareness. Advances in neural information processing systems, 35, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion, 2018.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendrieks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 155556, 2022.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan github. potpts/mmmmmo...../////
Andrej Karpathy. Nanogpt, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv.1412.6988, 2014.

---

Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. In Advances in Neural Information Processing Systems, 2019.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise not the main factor behind the gap between sgd and adam on transformers, but sign descent might be, In ICLR, 2023.
Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems, 2024.
Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36:15136-15171 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:24/2.19437, 2024.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training, arXiv preprint arXiv:2305. 14342, 2023,
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 2022.
Toan Q Nguyen and Julian Salazar, Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05899, 2019.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvicto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 2022.
Antonio Orvieto, Jonas Kohler, Dario Pavllo, Thomas Hofmann, and Aurélien Lucchi. Vanishing curvature in randomly initialized deep relu networks. AISTATS, pages 7942-7975, 2022.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML, 2013.
Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: http:////ppe/eiieeee net/forumiid n6SCkn2QaG.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained Imos. arXiv preprint arXiv:2502.07559, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations, 2018
12

---

Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of muon for pretraining, arXiv preprint arXiv: arxiiv2505.00222,, 2025.
Noam Shazeer. Glu variants improve transformer. arXiv preprini arXiv:2002.05202, 2020.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient
noise in deep neural networks. In ICML, 2019.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023. URL https://huggingffan co/ cestaastssceeebbeee
j am2-66BB..
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical Report, 6, 2012.
Akiyoshi Tomihari and Issei Sato. Understanding why adam outperforms sgd: Gradient heterogeneity in transformers. arXiv preprint arXiv:2502.00221, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In ICLR, 2025.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model. 2021. URL https://github. com/kinggoffol//mesh--roe http://gfofottoeettrrrrrrmrrrraa page 8, 2022.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stem, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances neural information processing systems, 30, 2017.
Shuo Xie and Zhiyuan Li. Implicit bias of adamw: lo -norm constrained optimization. In ICML, 2024.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference machine learning, pages 10524-10533, PMLR, 2020.
Wei Yuan and Kai-Xin Gao. Eadam optimizer: How impact adam. arXiv preprint arXiv:2011.0221 1.200/1.01110 140, 2020.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How does critical batch size scale in pre-training? In ICLR, 2025.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In ICLR, 2020a.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances Neural Information Processing Systems, 33:15383-15393, 2020b.
13

---

Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. Advances in Neural Information Processing Systems, 2022.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why trans- formers need adam: A hessian perspective. Neural Information Processing Systems, 2024a.
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more, 2024b.
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham M Kakade. Decon- structing what makes good optimizer for autoregressive language models. In ICLR, 2025.
Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. Advances Neural Information Processing Systems, 2024.
14

---

Contents
Introduction
2
Preliminaries and Related Works
3 Experiments
3.1
Extensive benchmarking at 160M parameters
3.2 Ablations
3.3
Checking for confounders..
New Viewpoints of Adam
4.1
Adam Estimates Mean and Variance using Variational Inference
4.2
Comparison with Balles and Hennig [2018)
8
5
Why an adaptive trust region? Insights from heterogeneous quadratics
Conclusion
10
Experimental details
16
A.1 Experiments on SlimPajama 160M parameters model
17
A.1.1
Sequence Length 2048, Batch size 256, 3.2 Tokens (6200 gradient steps)
17
A.1.2
Sequence Length 2048, Batch size 256, 6.4 Tokens (12400 gradient steps)
18
A.1.3 Sequence Length 512, Batch size 256, 3.2 B Tokens (24800 gradient steps)
18
A.1.4 Sequence Length 2048, Variable batch size, 2.5 B Tokens
19
A.2 Experiments on SlimPajama - 440M parameters model, 8.2 tokens
19
A.3
Experiments on Fineweb 160M parameters model, 3.2B tokens no weight decay
20
Complementary Experimental Results
20
B.1
Tuning for Table 1
20
B.2 Effect of More Training Tokens Figure
................................
22
B.3 Effect of Shorter Sequence Length in Figure 2
23
B.4
Batch size ablation for Figure
24
B.5 Figure on Fineweb (no weight decay)
25
B.6 Effect of Bias Correction and Zero Initialization on Adam
25
Missing proofs and derivations
26
C.I Proof of Proposition
26
C.2 Generalization of Proposition Necessity of equal betas for variance interpretation
26
C33 Proof of Theorem 4.1
28
C.4 Performance of generalized Adam reformulation
29
D Toy Quadratic Example
30
15

---

E Signal Processing Perspective
31
A Experimental details
For pre-training Transformers Causal Language Modeling, we build upon the nanoGPT [Karpathy, 2022] implementation, augmenting it with Rotational Positional Embedding [Su et al., 2024], RM- SNorm [Zhang and Sennrich, 2019]. and SwiGLU [Shazeer, 2020J. All our models have vocabulary size of 50280 and make use of GPT-Neox tokenizer [Black et al., 2022]. We adopt an enhanced training recipe, made popular by large language models such LLaMa [Touvron et al., 2023]. These modifications include: training in bfloat16; employing a linear learning rate warm-up for 10% of the training steps, followed by cosine annealing to le 5. Global norm clipping is used (unless specified or ablated upon) for gradients with norms above (on the raw gradient, as a first step). We have no weight tying between the embedding and the last linear layer. We always report validation perplexity on a separate subset consisting of 100M tokens. Seeds, when provided, are relative to distinct network initialization.
Computational Resources. All our experiments at 160M parameter scale are performed single NVIDIA A100-SXM4-80GB. At compute optimality (most of our experiments) each run takes approximately 5.83 hours. Our runs at 410M parameter scale are performed on 8 NVIDIA A 1000-SXM4-00GB GPUs, and each run here takes approximately 4.83 hours. For all our runs, we fill up memory and optimize to minimize the gradient accumulation steps (usually, around 8).
Code.
All our runs use the repository
http:://sthaa..co//iccoo-----
Model settings (160M). We use the same configuration [Biderman et al., 2023]: https:// 1ariiia-16mmmmm
gi github.co http:///tttteee////////////////a
Layers: 12 Transformer (Vaswani al., 2017] layers
Attention heads: 12
Hidden size: 768
Attention implementation: Flashattention [Dao et al., 2022].
MLP type: SwiGLU [Shazeer, 2020] with expansion factor 8/3.
Backbone: PreL.N transformer [Xiong et al., 2020] with skip connections.
Normalization: RMSnorm [Zhang and Sennrich, 2019] for both Attention and MI.P
Position embeddings: Rotary embeddings (RoPE) to 25% of dimensions ([Su et al., 2024])
Initialization: the MLP and Attention output weights are initialized with variance 0.02/ /2 laylay (scaling also similar to [Radford et al., 2019]). All other weights (com- prising embeddings) [2019], Wang and Komatsuzaki [2022], Sec. 2.2). Biases are always initialized at zero.
initialized with standard deviation of 0.02 (Nguyen and Salazar
Precision: Mixed precision FP16 enabled.
Dropout: Disabled for both hidden and attention layers (see also Chowdhery et al. [2023]).
Model settings (410 M). We use the same setting as [Biderman et al., 2023], configuration
be found here: https:/ htpps://githbb wttp:///ittteeel/ttt//////aaaaaa pythia-410mm deduped. yml
Layers: 24 Transformer layers
Attention heads: 16
Hidden size: 1024
Other settings as 160M parameters.
16

---

A.1
Experiments on SlimPajama 160M parameters model
On the Cerebras SlimPajama-622 627B [Soboleva et al., 2023] dataset: attps://hugg" co/
datasets/ wwwebras/Siiem amaa- a at a 160M scale we present three experimental sections:
Section A.1.1 core setting, ablating on all optimizers.
Section A.1.3 ablating smaller sequence length.
Section A.1.4 ablating at different batch sizes.
A.1.1 Sequence Length 2048, Batch size 256, 3.2 B Tokens (6200 gradient steps)
This setup comprises a total of 747 full training runs. We always use warm-up (10%) and cosine anneal until a learning rate of le - 5. This setting is Chinchilla-optimal (20 tokens/parameter).
here denotes the weight decay, always decoupled [Loshchilov and Hutter, 2019].
Core experiments: These are the core experimental results for this setting.
SGD with momentum and global norm clipping to 1 (Gelip, dampening to 1 (3)
84 full runs (Figure 8, top).
(n, 1) E [2.0, 1.0, 0.5, 0.25, 0.125, 0.0625, 0.03125] x [0.9875, 0.975, 0.95, 0.9)
[0, le 3, 1e 4].
SGD with momentum 3 with (1) global norm clipping of raw gradient to 1 (Gelip) and (2) coordinate clipping (Cclip) to after momentum is applied. Dampening is set to 1 - B,
X (weight decay) set to 0, as the previous point revealed decreasing performance on SGD
24 full runs (Figure 8, bottom).
(7. 3, A) E [2.0, 1.0, 0.5, 0.25, 0.125, 0.0625]
x [0.9875, 0.975, 0.95, 0.9]
SGD with momentum B (vanilla, dampening to 1 3, no clipping). À 0 (weight decay). 28 full runs (Figure 9)
(n, B) E [0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625]
x [0.9875, 0.975, 0.95, 0.9].
Adam with global norm clipping to 1 and with 0.1 (weight decay) and 1c (usual Pytorch setup, see also Biderman et al. [2023]).
-20 full runs (Figure 2)
(n, B1 B2) E [0.016, 0.008, 0.004, 0.002, 0.001]
[0.9875, 0.975, 0.95, 0.9, 0.8]
x [0. (0.996875, 0.99375, 0.9875, 0.975, 0.95, 0.9, 0.8, 0.6]
Adam without global norm clipping and with À 0.1 (weight decay) and 1e (usual Pytorch setup, see also Biderman et al. [2023]). 165 full runs (Figure 12)
(n. B1, 32) E [0.032, 0.016, 0.008, 0.004, 0.002, 0.001]
[0.975, 0.95, 0.9, 0.8]
x [0.9875, 0.975, 0.95, 0.9, 0.8, 0.6]
RMSprop implemented using the AdamW W Pytorch class using B1 0. We again use
X = 0.1 (weight decay) and 1e
488 full runs (Figure 10).
(n, 32) 0.004, 0.002, 0.001, 0.0005, 0.00025, 0.000125] [0.9875, 0.975, 0.95, 0.9, 0.8, 0.6, 0.4, 0.00
17

---

Signum with weight decay À 0.1 as also suggested by [Zhao et al., 2025] (their Figure 5, top left panel). We ablate on presence global norm gradient clipping (to norm 1). 70 full runs (Figure 2).
(n, clip) [0.004, 0.002, 0.001, 0.0005, 0.00025, 0.000125, 0.0000625)
[0.9875, 0.975, 0.9,
Falee
Note that Signum with and without gradient clipping are different methods: here, clipped gradients are first averaged and only then the sign is taken. Instead, clipping on the EMA of signed gradients (next method) should have effect (apart from non-determinism).
EMASign with weight decay À = 0.1. We ablate on the presence of global norm gradient clipping (to norm 1) out of mistake: the two methods are equal! -70 70 runs (35 duplicate runs) (Figure 11)
(n, B, clip)
[0.001, 0.0005, 0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625] x [0.9875, 0.975, 0.95, 0.9, 0.8]
[True,False)
Ablations:
These ablations were performed to test side-claims paper.
Adam with global norm clipping to and 0.1, 31 .2 0.95, 0.008 (best setup from Figure 2). We report performance for seeds using different values. 18 full runs (Table 2).
E [le 3, le 6, le 8, le 10, le 12, le 15).
and influence of initializing exponential moving averages to zero (default, ZD) or to the stochastic quantity of interest (gradient initialization, GI). the same time, we to remove bias correction. These experiments are presented with 3 random initialization seeds:
9 full runs (Table 3).
Signum with global norm clipping to and À = 0.1, 3 = 0.95 (best setting from Figure 2): ablate fixed mollifiers for zero-initialized (ZI) gradient-initialized (GI) momentum. The mollified we study mz/ m/x(y/m (y/m + c):
(n, c) E [0.001, 0.0005, 0.00025, 0.000125] x [1e 3, le - 6, lc - 99]
12 full runs (Table 2). We additionally test the influence of ZI vs. with three random seeds at e = 0.
5 5 full runs (Table 3).
for the hesf-perg variants of core experiments, we initialize the model with two other random seeds. This accounts for 14 additional full runs (Table 1).
Other:
A.1.2 Sequence Length 2048, Batch size 256, 6.4 B Tokens (12400 gradient steps)
The setup here is exactly as in $A.1.1, but we train for 2x the token budget. We test our core claim (3) = 32 works well), and hence run:
* Adam with global norm clipping to and with À = 0.1 (weight decay) and = le 8.
168 full runs (Figure 13)
(7, B1 82) E [0.032, 0.016, 0.008, 0.004, 0.002, 0.0001,0.0000 00.9855, 0.975, 0.95, 0.9]
x [0.99375, 0.9875, 0.975, 0.95, 0.9, 0.8]
A.1.3 Sequence Length 512, Batch size 256, 3.2 B Tokens (24800 gradient steps)
This setup comprises total of 55 full training runs. We test our core claims (Signum underperforms Adam, 31 18 works well) at smaller sequence length. Setting is exactly the same as $A.1. for all methods, unless stated otherwise.
18

---

Adam, we limit this ablation to 31 = 0.95,
(n. 52) E [0.001, 0.002, 0.004, 0.008, 0.016] x [0.99375, 0.9875, 0.975, 0.95, 0.8)
full runs (Figure 14).
Signum, we do full ablation using global norm gradient clipping 1.
(n, B) E [0.0000625, 0.000125, 0.00025, 0.0005, 0.001, 0.002) x (0.9875, 0.975, 0.95, 0.9, 0.8]
full runs (Figure 14).
A.1.4 Sequence Length 2048, Variable batch size, 2.5 B Tokens
We use here a slightly reduced token budget (2.5B, 20 tokens for every non-embedding parameter) and run the same Adam tuning experiment presented in Figure 2 for batch size 256. We actually run this experiment again at a batch size of 256, and test batch sizes of 128 and 512 reducing or doubling the number of steps accordingly (same token budget). The sequence length is still 2048, and the dataset SlimPajama, Due to the reduced number of tokens, each run takes approximately 4.7 hours on our hardware. We implement variation of batch size using gradient accumulation (4, 8, 16) at micro-batte size of 32 sequences. This setup comprises total of 500 full training runs.
Adam with À 0.1 (weight decay) and le 8 (usual setup, Biderman al. [2023]), we clip gradients to global norm 1.
For batch size 256:
(n, 92) [0.016, 0.008, 0.004, 0.002, 0.001]
x [0.9875, 0.975, 0.95, 0.9, 0.8]
[0.996875, 0.99375, 0.9875, 0.9.5.99.55. 0.995, 0.9,
For batch size 128 and 512:
(17, B1, 82) E [0.0005, 0.001, 0.0014 0.002, 0.0028, 0.004, 0.0056, 0.008, 0.0112, 0.016]
00.975, 0.9
1 - [4, 1, 0.5, 0.25] (1 (1-3,) 31)
(i.c. 3 higher and 2 lower values in grid)
Note that here we overturned the learning rate, the reason for this is the square root scaling law in Malladi et al. [2022], Compagnoni et al. [2025]: if batch size scales by 2, learning rate should scale as V2. We see in $B.4 that this indeed seems to hold true, yet noise prevents us from making precise verification claims.
500 full rans ($B.4).
A.2 Experiments on SlimPajama 410M parameters model, 8.2 B tokens
All experiments here use the Cerebras SlimPajama-66778 [Soboleva al., 2023] dataset: https:
/ //hugg ingf ace co/ dataastsss////////// cereberrrrrsseeeee hatpaaasssmmaaaaaaaaeaaaaaaaaeee aaPPaaamam6777 We focus on evaluating whether
31 = 32 yields good performance in this settings. We scale up the batch size by factor 2 compared to Section A. as suggested by [Zhang et al., 2025]. We perform our experiments at compute optimality (8.2B tokens, 20 tokens per parameter).
Adam with À = 0.1 (weight decay) and = le 8 (usual setup, see Biderman et al. [2023]), we clip gradients to global norm 1:
31 0.9
(n, B2) E [0.0166 0.008, 0.004, 0.002] x [0.95, 0.8]
31 0.95
82) E [0.016, 0.008, 0.004, 0.002] [0.9875, 0.975, 0.95, 0.9]
31 0.975
(n, 82) E [0.016, 0.008, 0.004, 0.002] x [0.99375, 0.9875, 0.975, 0.95]
44 full runs (Figure 5).
19

---

A.3 Experiments on Fineweb 160M parameters model, 3.2B tokens no weight decay
While testing our claims on different dataset, we also crucially remove weight decay here. Our setting is otherwise identical to that of $A.11.1; Sequence length is 2048, batch size is 256, model has 160 parameters and we train on 3.2B tokens from Fineweb [Penedo et al., 2024] https:// huggingface ace. co/ Battassts//uuggingaaee////
Adam with À = 0 (no weight decay!) and = le - 8 (usual setup, see Biderman et al. [2023]). We clip gradients to global norm 1.
(n, 92) [0.032, 0.016, 0.008, 0.00040..0020..000
x [0.975, 0.95, 0.9] [0.9875, 0.975, 0.95, 0.9.0.8)
full runs (Figure 18)
Signum with À = 0 (no weight decay) also suggested by [Zhao al., 2025] (Figure top left panel). We clip gradients to global norm 1.
(n, B) E [0.004, 0.002, 0.001, 0.0005, 0.0000625, 0.00025, 0.000125] x [0.975, 0.95
24 full runs (Figure 18).
B
Complementary Experimental Results
The results in this section complement the discussion $3. We organize them in 5 subsections, and report all technical details in $A.
$B.1 outlines all hyperparameter tuning curves for the setting in Table for SGD (with/without clipping and with/without weight decay) - Figure and 9, RMSprop without momentum Figure 10, and momentum on top of SignSCD Figure 11.
$B.3 validates that 81 = 32 is a prroog-perfomming option for Adam at a shorter sequence length. Here, we also show that Signum performance still suboptimal (cf. Figure 2).
$B.4 validates that 31 B2 is strong- peroog-peroorming option for Adam across different batch- sizes. This data, comprising training 500 models, is summarized in Figure
$B.5 reproduces the Signum- Adam gap on Fineweb [Penedo et al., 2024]. Compared to Figure 2 and the settings above, here we compare at zero weight decay to eliminate this additional confounder.
$B.6 confirms the validity of our findings when ablating nuances of Signum and Adam such as initialization and bias correction. These findings complement $3.3.
B.1 Tuning for Table 1
Setup Summary. 160 M parameters LM on SlimPajama, trained for 3.2 B tokens at batchsize of 256 2048 sequence length.
Comment. Our objective here is to tune to best, despite the combinatorially exploding number of options, our methods in Table 1. Details regarding our hyperparameters grid and model configurations reported in $A. We remind that tuning for Signum and Adam is presented directly in the main paper as Figure 2. All figures below show optimal tuning jointly learning rate and momentum space. While tuning for RMSprop and momentum on SignSGD is straightforward, SGD requires more attention: we found that removing weight decay was always beneficial when global norm clipping the raw gradient, hence we adopt this option also for the non-clipped variant, and for the variant that includes an additional coordinate clipping step after applying momentum. We believe this is due to the decoupled nature of weight decay, combined with the high learning rates required for good performance in SGD.
Finalizing Table 1. After careful tuning, we select for each method the best configuration (given by figures below) and run two additional seeds to report final results with 2-sigma confidence bars.
20

---

SGD + Gcllp, 0.9875
SGD + Gclip. 0.975
SGD + Gclip, 0.95
SGD Gclip. 0.9
final test ppl
100
80
wd 0.0001
wd 0.001
60
40
100
10-2
10a
10
100
Tearning rate
learning rate
learning rate
learning rate
SGD CCllo p - 0.9675
SG0 P 0.975
SGD CC # 0.85
SCD + CCli. p 0.9
50
Srra 1est 5p)
35
10a1
10
10
10
107
learnin rato
leaming rate
leaming ratc
Icaming rate
Figure 8: (top) SGD with global norm clipping. found beneficial remove weight decay: the best setting achieves 37.53 ppl, while a slightly larger wd leads 38.11. weights decay of 0.001 is too large and yields 93.7 best validation perplexity. (bottom) SGD with global norm clipping on raw gradients, followed by coordinate clipping momentum. We remove weight decay suggested by the plot. We observe an improvement 5 perplexity points.
Plain SGD. 0 0.9875
Plain SGD. P 0.975
Plain SGD, 8 0.95
Plain SGD, 0.9
learning rate
learning rate
learning rate
learning rane
Figure 9: SGD without coordinate-wise clipping at zero weight decay (as suggested by Figure 8
0.9875
0.975
0.4
40
38
final best ppl
36
final test ppl
final pp!
final best ppl
final test po!
final test ppl
34
32
final
final
30
28
10
10
learning rate
learning rate
leaming rete
leaminng rate
learning rate
learning rate
learning rate
learning rate
Figure 10: RMSprop with decoupled weight decay 0.I. Implemented with Pytorch AdamW setting B1 = 0.
0.9875
Bi 0.975
Bi 0.95
B1 =
Bi
ppl
34
clip
Faise
32
10
10
10
10
10
10
10-2
10
10
learning rate
learning rate
leamiing rate
leaming rate
Figure 11: Momentum SignSGD with decoupled weight decay. We implement this just for completeness to show that it performing worse than S Signum Clipping has mathematically no effect (we did not notice at first, so we show the result anyways).
21

---

Ada ciipiingt
AdamiN ino cipping). 0.9
Adamm (re clisping, 0.55
Adami dipping). 2.05
85
best with
8: 0.9
equal batas
27.5
0.975 3.
22.0
with
betas
equai
21.5
10
10
10
Teaming
learning
Figure 12: Adam without global norm clipping on gradients with decoupled weight decay. Compared to
Figure 2, here we do not clip gradients as a first preprocessing step. Performance is slightly worse, and results noisier. The best setting, among the
tried, is 31 32 0.9. Note, however, that for large/small 3, s.
we observe that some specific configuration with high 32 can beneficial (while still suboptimal if 91 8
tuned). In practice, best performance can also be achieved this setting by merely tuning 31 92 = resulting in drastic hyperparameter grid size reduction.
B.2 Effect of More Training Tokens Figure
We part of the experiments in Figure twice the token budget. Results conceptually very similar, and show that, on top of 31 Bg being a performance choice for Adam W, that there exists strong correlation between values (see Fig. 3).
AdamW, 0.9875
22.0
21.5
A: = 0.8
21.0
9:
0.95
20.5
A: = 0.975
0.9875
20.0
R. 0.98375
best with equal betas
19.5
equai betas
equal
best with
best with
equal betas
10
10
10
10
10
10
10
10
leamning
learning
learning
learning
Figure 13: Adam W. same setting as Figure 2, but trained for twice the number of tokens.
22

---

B.3 Effect of Shorter Sequence Length in Figure
We run part of the experiments in Figure at a lower sequence length (512), for a batch size 256 sequences (as Figure 2). The model here still sees 3.2B tokens (compute optimal), but number of effective optimizer steps is 4 times bigger compared to the 2048 sequence length setting. While we still observe a sizeable gap between Signum and Adam, we note that this is smaller compared to Figure 2, as noted also by Zhao et [2025] similar setting.
wid 5 0.9875
Signum-wd, # 0.975
Signumewd, # 0.95
Signum+wd, F 0.9
Signum+wd, F 0.8
28
27
26
25
learning rate
learning rate
learning rate
learning rate
leaming rate
AdamW,
25.5
25.0
Signum
performance
P. 0.8
final best pol
A: 0.9
24.5
A:
24.0
8: 0.9875
best
8: 0.99375
B2
23.5
10-3
10
learning rate
Figure 14: Adam W vs Signum, same setting as Figure 2, but at a smaller sequence length (512).
23

---

B.4 Batch size ablation for Figure
We rum part of the experiments Figure 2 lower and higher batch size. All other details remain the same and are summarized in $A except for the number of steps performed: due to limitations in our resources, we chose here train models for 2.5B tokens i.c. slightly undertrained setting (optimal would be 3.2B). In line with [Malladi et al., 2022, Compagnoni et al., 2025] we consider half-steps when tuning. All experiments use weight decay of 0.1.
Despite some imperfections and noise in performance, we notice that [1 = .3 is strong choice even different batch sizes, our Takeaway 2.
Adami,,
Adam/W
AdamW, 0.9875
26.0
25.5
8; 0.8
Ema Eaad
24.5
0.95
with
0.9675
23.5
0.996875
23.0
22.5
leamiin
learning
learning rate
leaming
learning
Figure 15: Adam, batch size 256 trained for 2.5B tokens. Other settings setting Figure 2.
AdamiW,
Adamnty,
26.50
28.25
28.00
2 0.6
25.75
25.50
8: 0.9
0.95
25.25
25.00
B, 0.9875
24.75
0.59375
24.50
24.25
10
10
10
10
leaming
Figure 16: Adam, batch size 512 trained for 2.5B tokens. Other settings are same setting as Figure 2.
AdaniK, 0.9
AdamIN, 9, 0.95
AdamtW, 0.975
23.6
8; 0.8
234
8.95
232
0.975
0.5875
WITH 2u
B: 0.99375
22.6
10
10
10
10
30
10
leaming rate
leaming rate
leaming rate
Figure 17: Adam, batch size 128 trained for 2.5B tokens. Other settings setting Figure 2.
24

---

B.5 Figure 2 on Fineweb (no weight decay)
Finally, we evaluate our findings both strong performance of equal [s in Adam and substantial gap with Signum on a different dataset (Fineweb [Penedo et al., 2024D. All other experiments in this paper are performed on SlimPajama. To add an additional axis of variation compared to previously presented settings, we here remove weight decay from all methods.
Acim and Signum,
Adam and Signum, B m 0.95
Adam and Signumm f = 0.975
20.5
28.0
Signum
Adom, 8: 0.8
27.0
0.95
26.5
Adam, P. 0.975
bust with
25.5
best with
equal
25.0
learning rate
learring
learring rate
Figure 18: Adam and Signum (no weight decay) Fineweb. Other settings are same Figure 2. For visualization purposes, here we rescaled the visualized learning rate of Signum by factor
B.6 Effect of Bias Correction and Zero Initialization on Adam
The findings below complement our discussion in $3.3.
Table 3: ZI denotes Zero init of EMA parameters, GI denotes init of EMA parameters to the measurement at first iteration, BC denotes Bias Correction. Not doing 7J means we initialize m and v at go and gō respectively. Defautt for Adam is 7J and BC. Default for Signum+ WD is less clear. We found that initialization does not affect much performance in Signum, yet it does in Adam. Performing bias correction is not as important as initialization in Adam All other parameters for BC and Z1.
ablation are fixed to the optimal ones found default settings
Adam (+ZI+BC)
Adam (+7I-BC)
Adam (+GI-BC)
Signum (+GI)
Signum (+7I)
ppl
21.86.1 0.21
21.891 1 0.16
22.58 0.35
23.23 0.16
23.30.1 0.25
Optimal AdwmWW tuning
Difference between Nethods
Abolune Oiferencc (tainppl)
10
Neplodty
10
30-3
207
20
Medami Acamm no RC
Figure 19: Effect of eliminating bias correction in Adam The difference between variants vanishes iterations progress. Plotted is average dynamics over random seeds.
25

---

C
Missing proofs and derivations
C.1 Proof of Proposition
Proposition 1. Let my = 3 [gk). Then the update (2) admits the equivalent representation:
me
dR
(3)
Vmg +
EMA (m 9k
Proof of Proposition 1. For this proof will use the abbreviation
2k 3MAg 19
With this abbreviation the Adam update (2) can be written
mk
dk
Vuk
V m
+
m
Next will show that U3 m /EMA, and mg+11 we have that
(m--- D
g Indeed by expanding the update of uk+1
2
BUx + (1 3) (Bmx + (1 3)
Uk- M
Buk + (1 3) (B9 & + -3) + 2.311 3)
BUk 3 (1 8) 23(11 13
BUK - 3m + 3m 32m + (1 - 3) 2, - 33
S(u: - m + 3(1 B) + B(1 B) 2,(1 3)
A(0 - mm) + 3( (m 9c+1)²
By setting 6% 2k m we have that
8 = 36x + 31 - (m - g - -EMA, [ - -9x)²
where we used the definition the EMA recurrence in (1).
C.2 Generalization of Proposition Necessity of equal betas for variance interpretation Proposition 2. Adam with hyperparameters B1, 32 (0, 1) has update of form
mk
V + YEMA, (am b
for some a. b, R and E (0, 1) if an only if 31 B2-
Proof of Proposition 2.
Let us expand the expression.
Uk+1 - m = B2uk + (1 - 32) - (Bim* + (1 (1)gx+1)²
B2uk + (1 - 32) - [89m3 +(1 11--8p 31)22 + 2,3 - -3:) 31)m49k+11
=
B22k - B9 + [11 - B2) -11-31) - 2,3 (1 B1)ma9t+1
The case of equal betas. Notice that if 31 = :2== 3, then
(1 3) (1 9)² = B - (1 + 82 - 2,3) = 1- 132 + 23 = B(1 - 13,
and so the expression gets simplified:
Uk+1 m = 3i - m + 3(1 - -B)122 2m
Now add and subtract Bmi get
Uk+1 m B(vx m + 3(1 - B)(m + -2_1 -2mggk+1]
where the last term is the perfect square (m4 2
26

---

The general setting, One might hope for the "stars aligning" into a perfect square also in the general setting. For this to happen, we need to require that the term
(1 32) (1 3
23r (1 Bi 31)mgg++1
allows for such simplification to happen. That is, assume to start from
(am, b1)=1)² amms= 2abmag9++1 + b² 9
we need
b² = (1- - (1- (1 3z) ab = 31 31)-
so
B1 - B1)
a
v B2) - (1 31)2
Therefore:
m v(1 (1) - (1 - 91) 2
(1
v(1 -32) (1- - 81)²
87(1 3,)
(1- 82) - (1 - 31)2 m + [[1 - B2) - (1 - 31) - -2 - 31)mmak+1
Therefore, in the general setting, we can write
89( - 31)2
2k+1 m = B2
B9 +
(1 52) - (1- 31))) m
2
B1 B1)
mh v(1 - B2) (1 (1-11)) gh++1
v(1 52) - (1 B1)2
Massaging a bit, get
BT(1 - B2)
Uk+1 -m2+1
$2
(1 - B2) (1 B1)2 m +
2
Bi 1 -B1)
+ ((1 B2) (1 B1)²
mg v 32) - (1 - 3,) gx+1
which implies
B7(1 (2))
Uk+1 m2 B2 (2x
B3 B2) - 32(1 - 31)2 m +
2
81 - 31)
((1 (2) - 31)2 mk v(1 - (2) - (1 - (, 92
Therefore, the formula holds true if and only if
87 - 32)
32(1 - 32) - 32 - 31)2 1.
That is, if and only if The condition simplifies, as it reads:
83 B2) = 32 B2) -B2(1- - B1)
87 - 8, = B2 - 32 - 82 - 320 + 23132
which simplified is
B7 + 22 28132 = 0.
i.e.
- -81- =0 + 3 = 32.
27

---

C3 Proof of Theorem 4.1
Theorem 4.1. Let 1+A Then the solution the optimization problem (4) is given by
m = 8mm + (1 - 3)gk=1 = EMA
(7)
o o _o B(1 3)(m) 9k+1)² = EMMAs [Im 9k+1)
(8)
Proof. Recall that
1ogp(0++1 m. o,
log
(gt+1 - 0+++-
20
KL (N(me, [I = 2
o
+
(m, m)²
1 log
0
g²
0²
Therefore
-log p(gk++ m, o 1
F(m, o²)
KL (W o
log 0² + (9h+1- - mm)² +
20 (9h+1
1
σ
mm - m)²
log
2
21
0²
g²
Since we are not optimizing for oº. we can replace log the following objective function
(::²)
llog(o") and drop constants, gives
1 11 X log(o?) +
1 (g m)²
1 (o (m+ m))) const.
min Fmm, o²) m,o-20
202
Stationarity in m: Differentiating in m and setting to zero gives
OF
1 σ² (g m)
(ms m) 0.
dm
do
Multiplying by λo we get:
Ag mg
N m) (m² m) =0
m
(13)
Stationarity in o²: Differentiating in o² and setting to zero gives
OF 11+1 -02
2 (g m)² + 1 - (o + (mz m)²)
X
Multiplying both sides by 204, and rr-arnanging gives:
x 02 = (g- - m)² x (o (m= - m)2)
Multiplying through by T+X gives
0² A(g - 30) + [o] + (mx - m)?]
(14)
1
Now using m
Ag+-m from (13) we have that 14X
9 m7
A(m² - -9)
-
1 1+X
my m
1 X
Therefore:
(g- m2m²
X2 m )
(g - m)²
(m - -m)²
(1 + A)2
(1 + X)2
28

---

Using the above in the expression for σ² in (14), we get:
0² A(g mx)²
ox
1+X
This, together with (13) gives the final solution
my + Ag
of 1+X +
A(g - m
m
and
o
(1+
If use the standard momentum parameterization, which corresponds to B T+X arrive the stated results (7) and (8) of the theorem.
C.4 Performance of generalized Adam reformulation
As described in $4.2, we here consider performance the update direction:
mk
dx m + y EMA, m 9k
(Ada Var)
This reduces to Adam with equal betas soon as B 7 T but cannot be written an Adam update as soon as B of 7 or 7 + T (see proof in $C.2). Further, our theory $4 shows that 3 = 7 = T is the only theoretically grounded choice for precise online variational inference interpretation, also in this setting, i.e. when considering of = EMA, [(mx-1 - with optimal performance.
- 33. We wonder this insight correlates
As one can see Figure 20, we found that setting / y leads near optimal performance in all settings.
AdamVar, 0.9
AdaniVar, 8 0.35
AdamVar, 8 0.975
Fast pn)
23.0
T-0.9
22.5
0.9875
best with bata
bate
21.5
10:3
10-2
10
10-2
10-3
10-2
learning rate
learning rate
learning rate
AdemVar. f = 0.9
AcamVar, 8 0.95
AdamVar, 8 0.975
24.0
23.5
tealtes ppa
23.0
0.95 (y 1)
22.5
0.9875 (y 13
22.0
bata tau
bata tau
21.5
10
leaming
learning
leaming rate
Figure 20: Performance of AdaVar aligns with our theoretical insights. Setup for these experiments is exactly the same as for Figure 2.
29

---

D Toy Quadratic Example
Hession magnitudel
Egenvv
Hession magnitude)
10a
102
0
101
Figure 21: (left) Heterogemeous and (right) Homogeneous Hessian considered in $5.
Our setup here is inspired directly from the results and discussions in Zhang et al. [2024a]. Specifically, we consider the loss
1 L(w) Hw
2
where construct the Homogeneous and Heterogeneous Hessians using the following procedure:
We fix the eigenvalues, equal in both cases, to
sig(Hhom) eig [ha) = {1, 2, 3, 99, 100, 101, 4998, 4999, 5000).
We choose both Hessians to be block-diagonal, with blocks of size 3 x 3. The homogeneous Hessian has eigenvalues of different magnitude in each block, while the Heterogeneous keeps similar magnitudes in each block.
H_ dedailllll..mmmmm _eet het [[1,2,3], [99, 100, ,01]] [4998, (4998,4999,,,,, 5000]] H_details_hom = [[1,98 49988, 100, 49999] 1,,10,,5000]]
For each block, we apply a random rotation to the diagonal matrix of eigenvalues, specific to each block. Each rotation is sampled from the Haar measure by decomposition of random positive semidefinite matrix AAT where A E R3x3 has i.i.d. Gaussian entries.
The result is shown in Figure 21.
Next, to introduce stochasticity in this setting, we simply take the square root of the Hessian define a 9 x 9 design matrix X
= x= H
and subsample a number (the batchsize) of of X each iteration.
30

---

E Signal Processing Perspective
In this last section, we examine Adam through a signal processing lens, to get qualitative insights into its distinction with Signum and other Si SignSGD with momentum variants. Setting 31 = A = 3, we can write the Adam update, without bias correction (see $B.6) as simply
du - -(VENAssOR) +e)
EMAg [gx]
where (gx)x k is the gradient signal. One might wonder this special case allows for a simpler graphical interpretatoin of Adam. To do this, consider here fixing the gradient signal, and see how different methods process this signal.
Graphical intuition. We denote by dy the update of Adam once sees gradient signal (gi)gga and plot its dynamics as a function of synthetic one-dimensional gradient in Figure 22.
9e
d, if =00
d Wiif:: =B
-
8f 0,9987
d, 9: = 0.99375
-1
de 9. - =8 0.996875
100
200
300
400
500
600
700
800
timestamp
Figure 22: Filtering effect for same 31 $2-
In the example of Figure 22, we chose the synthetic gradient signal
9k 1.8 sin(0.03k)) (xp(k)xxppp-))))
this is a damped periodic signal plotted in red. Note that this is pure filtering, there is no loss or learning process. We note the following:
1. 31 = =B2 = 0 is obviously just sjgn( This is plotted for comparison.
2. For any B1 = Bg + O, dg bounded by in magnitude. It's dynamics however, for e.g. 31 B > 0 is smooth and follows more closely the gradient, while being bounded. It is somehow a rescaled version. More on this later.
3. Very interestingly, dę blind the decay term exp(-0.0025k). the output is perfectly periodic for every 31 = B2.
Towards proceeding, note that d, cannot be reduced momentum the sign sign on the moomentumSSinnum): both variants actually destroy the signal shape, while dy maintains the shape of the original signal and has clear invariance properties. The behavior of signSGD with momentum (2 variants) is shown in Figure 23: as one can see, the behavior is drastically different from dę in Figure/ 22, an enlargement is shown Figure 24.
We now try to formalize some of the properties we observe.
very special operator T on gradient sequences (gx) E G C loo (with normed vector space structure and notation). We can identify four distinctive properties. T: (gx) 220 (dx)
Properties. Adam be
1. It is causal.
2. It is invariant positive scaling: T(a g) T(g), for any > 0.
3. It is odd: T(-g) = -T(g).
4. It has bounded infinity norm: |[ < 1 for all g to..
31

---

5. Density: For any b E I-1, 1] and any arbitrary 0, there exists (g+) such that de b.
are amazed by these rich set of properties, thickening our interest in better understanding the properties of Adam mollification, which we study in $4.
gt:
(Miggiig 8=0.00
(MAgiigg 8r,8=0.9
(MAntiign( 9=0.975
EMAnlsignigu) f500
EMAgtsigniaa) -=0.993772 99375
EMAglsign(gs)) if $-0.996875
100
200
300
400
500
600
700
800
timestamp
Sign(EMAgGG..1 6=0.0
sign(EMAn(a.1 600099
sign(EMAn(a.J) p=0095
signtEMAntg.J] 8-0.975
signtEMAglg.J) if 8-0.9875
sign(EMAp(g.J) if p-0.99375
sign(EMAp(g.11 if 8-0.996875
300
400
800
timestamp
Figure 23: Filtering induced by signSGD with momentum (2 variants, the one below is Signum). Compare with Figure 22.
d, -
sign(EMAy(a.)) 8=0.9 =f
if d, 0.99775
sign(EMAg(g.)) if
-1
50
100
150
200
250
300
timestamp
Figure 24: idam-like filtering compared sign of EMA (Signum), detail.
We hope this investigation ispires future effors in understanding these intriguing phenomena and properties. We conclude the paper with quote, stolen from the Bernt Øksendal masterpiece book SDEs:
have not succeeded in answering all our problems. The answers have found only serve to raise whole set of new questions. In some ways we feel we are as confused ever, but we believe and about more important things.
confused higher level
Posted outside the mathematics reading room -Tromsø University
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction claims that 31 = .2 is a nearly the optimal choice when it comes searching grid of choices. We support this claim with ample numerical evidence. The abstract and introduction also claim that we present a variational inference perspective of Adam, which do give in full detail in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We highlight the limitations of our work in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: We provide the complete proof and assumptions for our theoretical result in Theorem 4.1 in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiments are based on the repository https: //i gigg........... com/ Ni Viccolo-Ajroldi/ wwwrro/iipppannn/teee//ee/////ee
with no modifications on the architecture and training pipelines (except for ablations e.g. removing bias correction). We provide both code for reproducing our plots from data, and all the hyperparameters and exact details of each experimental setup are reported in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
34

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the code for reproducing our plots. We provide the data and main plots at httts:://gtthii com/ wtt.aaatooooooo/eee SecretSauce..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
coiiiisssscccco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips...c.. cttpp/ccppbbbc/guuie guwdes/ / /CodeSubmis ss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, see Section A for all experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In all Tables we report both the mean and 2-sigma confidence of validation perplexity over 3 network initialization seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the introduction we detail that our experiments required over 10 thousands
NVIDIA A100-SXM4-80GB hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://nnuuripie httpp//cccccc..eeeccccceeeeeeeee
Answer: [Yes]
Justification: We have read the code of ethics. Our work poses no clear negative societal impact and potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
There is societal impact of our work. Our work is regarding a general purpose method (Adam) for training generative language models. Its societal impact will then depend on who and for what end, will someone use these methods.
36

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our code is built top of publicly available nanoGPT [Karpathy, 2022] implementation' S, which includes a MIT licence, in the variant provided by https:// gi thub. wwwc/oom//iiccccecccc www.//iicooooAjooooo..
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
37

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release training curves for all our runs in the form of CSV to reproduce our plots upon acceptance. We are submitting as supplementary a zip file containing such files and example code plot results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
38

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines: No components of our work used generated responses from LLM.
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. wtt.cc////oooc/cceccooc
Please refer to our LLM policy (https: : htt.::pp.......eeeeeee////////// for what should or should not be described.
39


Reviews:
[{"review_id": "yJM7anDWh6", "reviewer": "Reviewer_89qA", "summary": "This paper presents an extensive empirical study on why Adam works so well for training Transformer models and reports two key observations:\n* Signum closes most of the perplexity gap between SGD and Adam but still suffers from a slowdown, hinting that Adam contains additional “secret sauce.”\n* The larger the chosen $\\beta_1$, the larger the optimal $\\beta_2$. $\\beta_1 = \\beta_2$ is near-optimal.\n\nThe authors then reinterpret Adam with $\\beta_1 = \\beta_2$ as a mollified version of Signum whose mollification factor is the local gradient noise-to-signal ratio. They argue this adaptive mollification is Adam’s hidden advantage and confirm the claim in a controlled setting.", "questions": "1. Zhao et al. report that the performance of Signum matches Adam well. What do you think might be the reason of the discrepancy?\n2. In a recent work, Srećković et al. argue that batch size plays an important role in the gap between Adam and SGD. Do you think Signum can match Adam if the batch size is better tuned? How do you think batch size will affect the \"secret sauce\" of Adam?\n\nReference\n\n[1] Zhao, Rosie, et al. \"Deconstructing What Makes a Good Optimizer for Language Models.\" OPT 2024: Optimization for Machine Learning.\n\n[2] Srećković, Teodora, Jonas Geiping, and Antonio Orvieto. \"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling.\" arXiv:2506.12543.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The authors conducted extensive grid search on hyperparameters and claim to open the experiment data to the public. This not only substantiates the findings of the paper, but also facilitates future research on this topic.\n* The results challenge the common $\\beta_1=0.9$, $\\beta_2=0.95$ Adam training recipe for LLMs and suggest that $\\beta_2=0.9$ might be the better default, potentially saving compute and simplifying future hyper-parameter tuning for Adam-like optimizers.\n\nWeaknesses\n* As shown in Fig. 2-4, $\\beta_1 = \\beta_2$ generally performs well but is not always the optimal choice, implying additional factors affecting the training dynamics of Adam.\n* This paper focus on Chinchilla-optimal settings, whereas many modern LLMs are trained on far more data. It is unclear whether the reported trends still hold in such regime.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their reading and appreciation of our work. It truly means a lot to us to hear that you also believe our empirical evidence to be extensive and of practical interest — especially for future research.\n\nWe also like to answer in detail your questions, as we believe those are very interesting: we will highlight these points in our revised manuscript and also include the new results we present next.\n\n---\n\n**1) Optimality**\n\nWhile $\\beta_1=\\beta_2$ performs in our experiments better than the usual choice $0.9, 0.95$ we do not claim that $\\beta_1=\\beta_2$ is always the best choice -- but instead that \n\na. it is quite convenient for tuning, while leading to near-optimal results.\n\nb. brings about theoretical insights.\n\n---\n\n**2) Training on more tokens.**\n\nThis is a setting that definitely helps deliver our point, and we thank the reviewer for bringing this up. We were able to run a quite comprehensive ablation. The results below are in the setting of Figure 2, but we train for 6.4B tokens instead of 3.2B. We performed a total of 168 full training runs, each taking approximately 10 hours on a A100 with 80GB. For each choice of $\\beta_1,\\beta_2\\in [0.9, 0.95, 0.975, 0.9875]\\times[0.8 ,0.9, 0.95, 0.975, 0.9875, 0.99375]$ we report the best learning rate in the grid $[0.005, 0.001, 0.002, 0.004, 0.008, 0.016, 0.032]$. We highlight the choice $\\beta_1=\\beta_2$ below. We report validation perplexity.\n\n| β₁  / β₂  | 0.8    | 0.9    | 0.95   | 0.975  | 0.9875 | 0.99375 |\n|---------------|--------|--------|--------|--------|--------|---------|\n| **0.9**           | 20.15  | ***20.00***  | 20.00  | 19.94  | 20.14  | 20.8    |\n| **0.95**          | 20.14  | 19.93  | ***19.87***  | 19.93  | 19.88  | 20.24   |\n| **0.975**         | 262.12 | 20.01  | 19.80  | ***19.73***  | 19.72  | 19.74   |\n| **0.9875**        | 5797      | 189    | 20.40     | 19.88  | ***19.81***  |  19.87 |\n\nAs the reviewer can notice, $\\beta_1=\\beta_2$ is always near-optimal (gap of 0.01 ppl), with a slight imperfection at $\\beta_1=0.9$. Note that this imperfection is also quite apparent in Figure 2. This does not quite matter because $\\beta_1=0.9$ leads anyways to suboptimal results. Notice how, as reported in Figure 3 in the paper, we notice a strong correlation between $\\beta_1$ and $\\beta_2$: the higher $\\beta_1$, the higher $\\beta_2$ has to be for optimality. \n\nNote that crucially, if one only tries values on the diagonal $\\beta_1=\\beta_2$, one gets to a perplexity which is only suboptimal by 0.01 ppl. This type of optimality also holds in Fig. 2 and 4.\n\n---\n\n**3) Question: Signum vs Adam + Batch Size effect**\n\nThis is a crucial point, which we again thank the reviewer for bringing up, this is worth a paragraph in our potential camera-ready. We believe our results are in *perfect agreement* with the 2 papers you mention. We would, however, like to note that we have already partially addressed this point in our paper (lines 196 in the main text and 491 in the appendix). \n\nLet us expand: we first summarize what the two papers you mention say and then compare them with our claims.\n\n***Summary***: Zhao et al. report that Signum and Adam perform quite similarly. Srećković et al. show that the gap between Adam and SGD shrinks to almost zero at very low batch-sizes. Srećković et al. was published after the Neurips deadline, so that is why we could not compare with this work.\n\n***Comparison***: The finding of  Srećković et al. was additionally confirmed by [A], which goes one step further and test a few optimizers other than Adam and SGD. In particular, they compare Adam with Adafactor — which (similarly to Signum) can be seen as a simplification of Adam. Note how in Figure 1(a) in [A] they show how the gap between Adam and Adafactor shrinks as the batch size decreases. We also notice a similar fact and already commented in line 196 of our submission, pointing to our Figure 12 in the appendix: at a shorter sequence length of 512 — i.e. a smaller number of tokens/iteration — the gap between Signum and Adam shrinks. We did this experiment to compare indeed with Zhao et al., that uses a fixed sequence length of 512. Hence, we believe our results are in perfect agreement with the 2 papers you mention.\n\nThere is a final important point to stress regarding shrinkage of the gap at low batch sizes: this setting is not practical for parallel computing, where batch sizes are often a standard. Our result in Figure 1 illustrates the gap between Signum and Adam in a practical scenario, with commonly used values of batchsize $\\times$ sequence length.\n\n\n[A] Marek, Martin, et al. \"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful.\" arXiv preprint arXiv:2507.07101 (2025)."}, {"review_id": "2NdlvhqbKp", "reviewer": "Reviewer_9fwq", "summary": "This is a very well-written submission offering a thorough analysis of the effectiveness of the Adam optimizer both experimentally and theoretically, arriving at a novel interpretation of the essential features of Adam, in turn leading to a set of simple, specific recommendations for best practices.", "questions": "Fig. 2, increase some of the fonts in the figure legends/labels? (Same for Fig. 4). Also, indicate more clearly that the top row is for Signum, and the bottom row is for Adam?\n\nEquation right before L90, defining `clip`. If abbreviating to \"Gclip\" in the text, maybe define `gclip` or `Gclip` in the equation?\n\nL111, what is a \"landscape-based argument\"?\n\nL114, \"less hyperparameters\" --> \"fewer hyperparameters\" \n\nL119, remove superfluous \"4\"?\n\nL229, \"Proposition 1\": state that the proof is in the Appendix?\n\nre: Theorem 4.1: state that the proof is in the Appendix?\n\nThe Conclusion seems to underplay the key theoretical findings, e.g. Proposition 1 and Theorem 4.1 (and the main other derivations in the Appendix). It seems the Conclusion could be expanded?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strengths of the work are: very high quality writing and overall excellent presentation, thorough and large-scale experimental grounding of the conceptual analysis, and very clear (at least in the foundation) theoretical analysis; elegant and simple concluding recommendations. The experimental evaluation should serve as a good reference for the behavior of Adam, Signum (and Sign) for the broader community. Also, AFAICT the work provides a good summary of relevant work in the community over the years.\n\nThe main weakness is what is IMO a slightly unclear, disconnected relationship between main body and Appendix.  The main body clearly depends on the Appendix material in several places, but this is not really mentioned in the main body. Conversely, there seems to be a fair bit of material in the Appendix that, though interesting and related, does not directly support anything in the main body. Ideally, there would be more of an organize, obvious relationship between main body and Appendix.\n\nOther weaknesses are relatively minor: the figures are a bit cramped/unclear in places, and some parts of the text could be clarified too.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their work and their pleasant comments on our paper! We also thank them for their questions and suggestions, which will help further improve the quality of our work.\n\nWe provide separate answers for your main points.\n\n---\n\n**1) Disconnected Appendix**\n\nThe reviewer is absolutely right, the main reason for this is that we did not want to mention precise places in the appendix due to the separate main paper and appendix deadlines (e.g., adding a new figure would compromise the ordering stated in the main paper). We will make sure everything in the appendix is linked semantically to the main paper and properly connected. Another reason for this disconnect was the tight space in the main paper, insufficient to properly comment all our side results. This can be though of course done in the potential camera-ready.\n\n---\n\n**2) Improved Figures and Conclusion**\n\nWe are committed to delivering the best possible graphical results, and therefore will devote efforts to addressing all reviewers' comments, such as those regarding Figure 4. We will also expand our conclusion section to address the theoretical aspects of the paper better.\n\n---\n\n**3) Other points**\n\n1. We will define Gclip in the equation. This helps clarity, thanks!\n2. With \"landscape-based arguments\" we mean arguments that attribute the Adam-SGD gap not to noise in the gradient but on the Hessian structure of Transformers. An example is the paper \"Why Transformers Need Adam: A Hessian Perspective\" by Zhang et al. We will specify in our updated version.\n3. Typos and other suggestions: thanks so much, we already updated our LaTeX file with your comments!"}, {"review_id": "0tTb6Knfgw", "reviewer": "Reviewer_rfMp", "summary": "This paper shows, through a large-scale experiment, that the Adam optimiser generally performs better when $\\beta_1 = \\beta_2$. The authors then propose a simplified version of Adam without bias correction, where  $\\beta_1 = \\beta_2$ and $\\epsilon = 0$. They show that when $\\beta_1 = \\beta_2$, Adam provides an online estimate of the mean and (centred) variance of the gradient. Further analysis also highlights a close relationship with signum, where the derived version of Adam is shown to be a signum modulated by the squared coefficient of variation (SCV). In other words, the higher the spread/uncertainty in a specific region of the loss landscape, the smaller the update magnitude.", "questions": "[1] In Proof C3, l. 530, I was unable to reproduce the given objective function. Instead, when substituting Eqs. 5 and 6 into Eq. 4, I have:\n$$\\\\frac{\\\\lambda + 1}{2} \\\\log \\\\sigma^2 + \\\\frac{1}{2\\\\sigma^2}\\[ (g_{k+1} - m) + \\\\lambda(\\\\sigma^2_k + (m_k - m^2)) \\] - \\frac{\\\\lambda}{2} \\\\log \\\\sigma_k^2 - \\\\frac{\\\\lambda}{2}.$$ \nI assume the last term is what goes into $const$ but the coefficient for $\\\\log \\\\sigma^2$ and $\\\\sigma^2_k + (m_k - m^2)$ are different and I am not sure what happened to $ \\frac{\\\\lambda}{2} \\\\log \\\\sigma_k^2$. Could the authors explain how they arrived at their results? I will be happy to raise my score if I missed a simplification or if there was a typo somewhere that is corrected and does not impact the results of Sec. 4.1. \n\n[2] l. 514, I do not understand how the authors went from $\\\\beta \\\\delta_n + \\\\beta(1- \\\\beta)(m_k - g_{k+1})^2$ to $\\\\beta EMA_{\\\\beta}[(m_{k-1} - g_k)^2]$. If the authors could provide the complete derivation, it would be much appreciated and may increase my score. It would also be great to have an interpretation of the $m_k^2$ factor in the denominator of Eq. 3.\n\n[3] There are several small typos in the maths, which did not impact the results, but should be corrected: \n\n- l. 241 $\\\\sigma_k$ should be $\\\\sigma_k^2$ to be consistent with Eq. 8. \n\n- l. 521 shouldn't we have $\\\\beta(v_k - m_k^2)$ instead of $\\\\beta(v_k - m_k)$ as we add and subtract $\\\\beta m_k^2$?\n\n- Before l. 523, when developing the square, we should have $a^2m^2_k$ instead of $a^2m_k$.\n\n- l. 529 not a math typo but an equation which is not referenced: it should be `substituting (5) and (6) into (4)'.\n\n[4] Could we see a comparison of Adam and the update proposed in Eq.9 to assess how much the various assumptions made during the theoretical analysis impact the practical application?\n \n[5] Could the authors explain in what regard their claim that Adam is an online estimate of the mean and variance of the gradient is novel compared to what is stated in the initial paper? I would be happy to raise my score if a small paragraph discussing the differences between the two were added somewhere in the paper, as this may help other readers better understand the novelty of the paper.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "___Quality:___\n\n[+] Most of the derivations and proofs are correct.\n\n[+] The proposed empirical experiment is done on several models and datasets with multiple seeds and an extensive grid search.\n\n[-] I could not reproduce some of the derivations. This is my primary concern as one of the results directly impacts the soundness of Thm 4.1 and the results of Section 4.1. (see questions 1-2)\n\n[-] There are some minor typos in several equations, which did not impact the final results (see question 3) \n\n[-] The impact of the different assumptions made for the theoretical analysis is not assessed experimentally (see question 4).\n\n___Clarity:___\n\n[+] The paper is well-written and easy to follow.\n \n[-] Most of the figures are too small and not black and white printer-friendly. (see comments on formatting concerns)\n\n___Significance:___\n\n[+] The authors provide a large-scale experiment and a theoretical analysis, which are, to my knowledge, new.\n\n[+] This may be used by others to propose a new Adam-based optimiser\n\n[+] The results highlight the importance of having optimisers robust to a heterogeneous landscape, especially in NLP. This may help others to design more efficient optimisers.\n\n___Originality:___\n\n[+] The paper provides an interesting analysis of Adam, which is, to my knowledge, novel, but I am not an expert in the field, and I may not be aware of closely related work.\n\n[-] In the original Adam paper, section 2, the authors said that `the moving averages themselves are estimates of the first moment (the mean) and the second moment (the uncentred variance) of the gradient'. As one of the central claims of the paper is to show that Adam is an online estimate of the mean and variance of the gradient, it may not be very novel. (see question 5)", "was_revised": true, "final_justification": "The authors have satisfactorily answered my questions, and I am now confident that all the derivations are correct and sufficiently detailed. I have thus raised my score to 5.", "rebuttal": "Thank you for the time and effort you have put into reading our paper. \n\nBelow we answer all of your questions, including your main question on verifying the proof of Theorem 4.1 (*which resulted from a small type-O*). See below. We hope, given our response, that the reviewer will reconsider their position. We thank the reviewer for this process, which helps us improve clarity in our work.\n\n---\n\n**Q1:**  “Reproduce the given objective function when substituting Eqs. 5 and 6 into Eq. 4”\n  \n**A1:** We very much appreciate the reviewer going through our proofs. The reason you could not re-deduce the objective is because of a type-O in our appendix: The appendix uses $1/\\lambda$ instead of $\\lambda$ multiplying the KL divergence. That is, in the appendix we have considered the objective\n$$ - \\log p(g_{k+1} \\mid m, \\sigma^2) + \\tfrac{1}{\\lambda }\\mathrm{KL}\\left(\\mathcal{N}(m_k, \\sigma_k^2)\\,\\|\\,\\mathcal{N}(m, \\sigma^2)\\right) $$\n\n\n This is the only reason your deduced objective does not match the one in the objective in the proof. We apologize for this type-O. In our revision, we will fix this, and take your advice and give all the explicit steps of the proof, starting by stating\n “Substituting (5) and (6) into (4) gives the objective …”' This will result in a clearer proof. \n \n---\n\n**Q2:** \"I do not understand how the authors went from  $\\beta \\delta\\_k + \\beta(1-\\beta)(m\\_k-g\\_{k+1})^2$ to $\\beta EMA\\{\\beta}(m\\_k - g\\_{k+1})^2$?.. provide derivation, …  may increase my score.\"\n   \n**A2:** Certainly, this derivation relies on applying the recursive definition of EMA given in Eq (1).   The short answer, is that the equivalence holds because the EMA as an operator is linear, and consequently $\\beta EMA_{\\beta}[(m_{k-1}-g_k)^2] =EMA_{\\beta}[\\beta(m_{k-1}-g_k)^2].$ In case it helps, we also give a more detailed proof below using induction.\n\nOur objective is to prove that, given $\\delta_k$ defined recursively via:\n\n$\\delta\\_{k+1} =  \\beta \\delta\\_k + \\beta(1-\\beta)(m\\_k-g\\_{k+1})^2 $\n\nit follows that  $\\delta_k = \\beta EMA_{\\beta}[(m_{k-1}-g_k)^2]$, where by definition the EMA is also defined recursively via\n\n$ EMA_{\\beta}[(m_{k}-g_{k+1})^2] = \\beta EMA_{\\beta}[(m_{k-1}-g_k)^2] +(1-\\beta) (m\\_k-g\\_{k+1})^2.$\n\n\n\n**Base case**. First consider the base case $k=0,$ where we initialize $m_{-1}=m_0 = g_0,$ and thus\n\n $\\delta\\_0 = 0 = \\beta (m_{-1}-g_0)^2 = \\beta EMA\\_{\\beta}[(m_{-1}-g_0)^2]$, \n\nsince we also initialize EMA at the first element of the sequence. Thus the equivalence holds for $k=0.$\n  \n\n**Induction hypothesis**. For $k>0$, assume we have that $\\delta_k = \\beta EMA_{\\beta}[(m_{k-1}-g_k)^2],$ and let us now prove the equivalence for $k+1$ using this induction hypothesis. By the recursive definition of $\\delta_{k+1}$ we have that\n   \n$\\delta_{k+1} = \\beta \\delta_k + \\beta (1-\\beta) (m_k - g_{k+1})^2  $\n   \n$= \\beta^2 EMA_{\\beta}[(m_{k-1}-g_k)^2] + \\beta (1-\\beta) (m_k - g_{k+1})^2$\n   \n$= \\beta( \\beta EMA_{\\beta}[(m_{k-1}-g_k)^2] +  (1-\\beta) (m_k - g_{k+1})^2)$\n  \n$=  \\beta EMA_{\\beta}[(m_{k}-g_{k+1})^2]$\n   \nWhere in the second equality we used the induction hypothesis, and in the last equality we used the recursive definition of EMA. \n\n\n---\n\n**Q3:**  interpretation of the $m_k^2$  factor in the denominator of Eq. 3?\n  \n**A3:** The interpretation we give is that, because of the denominator which includes the $m_k^2$ term, we can re-write Adam (for $\\beta_1=\\beta_2$) as Eq (9), where it is clear that Adam is a mollified variant of Signum.\n  \n---\n\n**Q4:** There are several small typos in the maths\n  \n**A4:** Thank you very much for these, and reading our paper and appendix so carefully. We corrected all of them in our latex file.\n\n---\n  \n**Q5:** Could we see a comparison of Adam and the update proposed in Eq.9?\n  \n**A5:** The method in Eq. 9 is equivalent to Adam with $\\beta_1 =\\beta_2$ and $\\epsilon=0$, which is the method we most experimented with. \n\n---\n   \n**Q6:** What regard their claim that Adam is an online estimate of the mean and variance of the gradient is novel compared to what is stated in the initial paper?\n  \n**A5:** We apologize that we may not have understood your question here. When you ask “novel compared to what is stated in the initial paper”, here we assume you mean the origin Adam paper [1]. Please let us know if this is not the case, and we will gladly interact and answer. \n\nThis viewpoint we give of Adam in Theorem 4.1 as a formal online estimator of mean and variance is entirely new. Despite years of research into Adam, to the best of our knowledge, we have never seen this viewpoint of Adam before. As compared to the original Adam paper [1], the authors introduce Adam as a “ ... the method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients”. That is, they refer to the $v_k$ buffer as a type of second moment estimator of the gradient. But there is no formal proof, or variational viewpoint that shows this. Compare this to our Theorem 4.1 that formally shows how Adam arise from finding the mean and variance that maximizes the likelihood of observing a gradient drawn from a Gaussian, subject to KL regularization term that encodes how the distribution over gradients changes gradually over iterations.\n\nThe only formal proof in the original Adam paper [1] is Theorem 4.1, which attempts to establish the online regret of the iterates of Adam, and which is a notoriously incorrect proof (a counter example emerged in [2])\n\nTo conclude, Let us also emphasize a crucial aspect of our theory: it is derived from the $\\beta_1=\\beta_2$ insight. Proposition 2 in the appendix shows that under no other choice we can get an explicit variance-like term in the reformulation. This is important, since the connection to SignSGD with momentum is only possible only if one can properly \"split\" the second order term $v$ into a squared mean and variance term. To the best of our knowledge, we are also the first to analyze this setting.\n\n\n\n[1] Adam: A Method for Stochastic Optimization, Diederik P. Kingma, Jimmy Ba, ICLR 2014\n\n[2] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. \"On the Convergence of Adam and Beyond.\"\nInternational Conference on Learning Representations, ICLR 2018."}, {"review_id": "FBGoLDqX9T", "reviewer": "Reviewer_Gwdg", "summary": "This paper firstly provides a comprehensive empirical study of the Adam optimizer on pertaining language models with extensive model types and hyperparameter settings. Through these experiments, the authors found out that the optimal beta parameter setting in their experiments is $\\beta_1=\\beta_2$, in which case they theoretically derived a new interpretation of the Adam updates as estimating the mean and variance of the stochastic gradients through variational inference. Under this framework, they interpret the Adam update as direction search within an adaptive trust region based on the ratio between mean and variance of the gradients (signal-to-noise ratio), contrasting with Signum (SignSGD with momentum), whose trust region is fixed in the lens of this framework. In addition, a landscape heterogeneity interpretation of such advantage is provided and validated with a constructed quadratic example.", "questions": "1. The empirical and theoretical analysis is done by contrasting with SignSGD and those interpolating between SGD and Adam, in particular for the theoretical analysis. The reviewer is wondering do the authors have additional comments on Adam if in contrast with other recent optimizers like Shanpoo [Gupta et al., 2018, Vyas et al. 2025], Lion [Chen et al. 2023], which are mentioned in the related work but not quite discussed. For Lion in particular, as a generalized version of SignSGD, does its improvement on SignSGD, to some extent, close this gap to Adam, or seem to be orthogonal efforts?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1. The empirical study of Adam is very comprehensive on training language models.\n2. The finding of $\\beta_1=\\beta_2$ as the optimal Adam hyperparameter setting is of practical guidance to the broader ML community.\n3. The theoretical interpretation of Adam from the variational inference perspective is novel to the best of the reviewer's knowledge, and the reviewer finds the adaptive trust-region perspective of Adam when contrasting with SignSGD very insightful.\n\n**Weaknesses:**\n1. Even though many hyperparameter combinations are tried out, the space grows exponentially, and the grid included in the paper could still be limited, looking at one parameter, e.g., $\\eta$, alone, as the authors acknowledge this limitation in the conclusion\n2. The experiments are only conducted for training language models, excluding other tasks or data like images, which could limit the generality of the findings.\n3. The variational inference interpretation only holds when $\\beta_1=\\beta_2$, while other values in different settings may still show superior, sometimes optimal performance, which could limit the generality of this theory, or leave a gap between theory and practice.\n4. Bias correction seems not included in the theoretical analysis. Though the emperical results do not seem to be affected, the bias correction of Adam is usually regarded as an important theoretical component.", "was_revised": true, "final_justification": "The authors addressed my questions in the rebuttal, and the paper is overall insightful. Therefore, I'm keeping my score and supporting the acceptance of this paper.", "rebuttal": "We thank the reviewer for their time, their insightful comments, and for finding our work comprehensive, practical, and novel!\n\nWe answer your questions below:\n\n---\n\n**1) Other domains**\n\nThe reviewer is absolutely correct: this question is of extreme interest. We limited ourselves to the LM setting, as this discussion already encompasses multiple scenarios (batch sizes, regularization, data, and scale), is compute-intensive and often used to showcase the Adam-SGD gap. We, however, note that a concurrent paper, \"Matias D. Cattaneo and Boris Shigida. Tuning Adam(w): Default beta2 may be too large, 2025\", which we cite in line 332, provides ablations on small-scale vision domains confirming the default $\\beta_2=0.999$ in Adam is too high.\n\n---\n\n**2) Variational Interpretation for $\\beta_1\\ne\\beta_2$**\n\nOur variational interpretation stems directly from our empirical finding that $\\beta_1=\\beta_2$ often leads to optimal results (e.g., Figure 2 and Figure 4). \n1. Let us elaborate better what we mean: while for a fixed arbitrary $\\beta_1$ it might indeed be that $\\beta_2>\\beta_1$ leads to *slightly* better results (maybe due to noise, e.g. Figure 2, $\\beta_1=0.9875$), at the optimal $\\beta_1$ we have both in Figure 2 and Figure 4 that $\\beta_2=\\beta_1$ leads to best perplexity. This implies that if one tunes only $\\beta_1$ and sets $\\beta_2=\\beta_1$, then the optimal result can be achieved.\n2. Our point is that Adam with $\\beta_1=\\beta_2$ is not only a good method but also a statistically principled method. Our novelty in this regard is to demonstrate that **only** in this setting does Adam admit a strong statistical viewpoint, as we show in Appendix C.2. We will reference this appendix and provide further commentary on it in our manuscript. Thanks for the question!\n3. To conclude, $\\beta_1=\\beta_2$ leads to a simplified and statistically principled method, which performs well (= much better than Signum) and often optimally in practice. This is not only interesting but also gives clear guidelines for tuning.\n\n---\n\n**3) Bias Correction**\n\nWe thank the reviewer for this careful question. Our analysis doesn't include bias correction because of the way we initialize the momentum buffers in Eq (1). Bias correction was introduced because the momentum (or EMA) estimate is biased for stationary signals when you initialize momentum at zero. We take care of this issue by instead initializing our momentum estimates at the first sample. This already guarantees an unbiased estimate of the momentum estimate, see Section 2.3 here for a discussion.\n\nIn any case, we also show in Section B.5. (referenced at line 221) that bias correction does not affect qualitatively our results. We agree that some works devote much focus on this, but also point out that bias correction can be seen as simply modifying learning rate scheduling (e.g. see equation 5 in https://arxiv.org/pdf/2003.02395), and our statistical interpretation is independent of scheduling.\n\n---\n\n**4) Other methods, e.g., Shampoo, Lion**\n\nIn this paper, we address the question of understanding Adam with *simpler* methods that claim *theoretical* similarity with this optimizer. Both Shampoo and Lion are designed to surpass Adam, not to simplify it or to explain its update rule. For this reason, we believe comparing our methods with these would distract from the message in our paper, which is not devoted to finding the best method, but rather to understanding the Adam optimizer and how to simplify its tuning process.\n\n[1] MoMo: Momentum Models for Adaptive Learning Rates, Fabian Schaipp, Ruben Ohana, Michael Eickenberg, Aaron Defazio, Robert M. Gower, ICLR 2024"}]

Author rebuttal:

Meta-review: This paper presents a thorough and compelling investigation into the Adam optimizer, a cornerstone of modern deep learning. Through an extensive empirical study encompassing over 1,300 language model trainings, the authors arrive at a significant and practical finding: constraining Adam's momentum parameters to be equal ($\beta_1 = \beta_2$) yields robust, near-optimal performance. This discovery is substantiated by a novel theoretical interpretation, framing Adam under this configuration as a natural online algorithm for estimating gradient statistics, derived through a mean-field variational inference perspective.

The manuscript is well-written, clearly organized, and makes several key contributions to the community:

- It provides a definitive empirical analysis, demonstrating that while Signum narrows the performance gap with SGD, Adam's adaptive trust region—governed by the signal-to-noise ratio of the gradients—consistently provides a measurable advantage.

- It offers a valuable practical guideline, showing that the simple constraint $\beta_1 = \beta_2$ is highly effective, thereby significantly reducing the hyperparameter search space for practitioners.

- It delivers a profound theoretical insight, reformulating Adam's update rule through a clean variational inference framework that elucidates its underlying mechanics.

The authors have satisfactorily addressed all concerns raised by the reviewers during the discussion period. It is noted that the related work section is comprehensive, though it miss a relevant work (https://openreview.net/forum?id=TBJCtWTvXJ), which provides a complementary theoretical justification for $\beta_1 = \beta_2$ from the perspective of mitigating loss spikes.

In summary, this work is grounded in extensive experimentation and deep theoretical analysis. Its insightful conclusions regarding both the performance and interpretation of Adam are timely and will be of substantial value to the  community. For these reasons, I strongly recommend acceptance.
Decision: Accept (oral)