Fantastic Pretraining Optimizers and Where to Find Them
Authors: Kaiyue Wen
David Leo Wright Hall
Tengyu Ma
Percy Liang
Venue: ICLR 2026 Poster (2026)
Keywords: optimizer
benchmarking
pretrain
Primary area: optimization
TL;DR: Fair comparisons reveal that new optimizers only offer modest speedups over AdamW; matrix level optimizers' advantage shrinks with scale—from 1.4× at 0.1B to just 1.1× at 1.2B parameters—once proper hyperparameter tuning and evaluation are applied.

Abstract: AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2$\times$ speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups.
To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B–1.2B parameters) and data-to-model ratios (1--8$\times$ the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training.
First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. 
Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1$\times$ for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay.
Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners --- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4$\times$ over AdamW for 0.1B parameter models to merely 1.1$\times$ for 1.2B parameter models.

Published as a conference paper at ICLR 2026
FANTASTIC PRETRAINING OPTIMIZERS AND WHERE TO FIND THEM
Kaiyne Wen
David Hall
Tengyn Ma Stanford University
Stanford University Kaiyuew@stanfordd.om
Stanford University dlwhăcs stanford.edu
tengyumalstanforr
Percy Liang
Stanford University
pliangêcs stanford. edu
ABSTRACT
AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct systematic study of ten deep learning optimizers across four model scales (0. (0.1B-1.25 parameters) and data-to-model ratios (1 (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperaaraeeter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well selletuned baselines is lower than claimed and decreases with model size to only 1.1x x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices preconditioners multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for O.IB parameter models merely 1.1 x for 1.2B parameter models.
flip during training due to learning rate decay. Through
INTRODUCTION
Pretraining has been the most computationally expensive component in the training pipeline for large language models, accounting for over 95% of the cost in DeepSeek V3 DeepSeck-Al et al. (2025b), and the additional RL training cost in DeepSeek R1 DeepSeek-Al et al. (2025a) is also comparatively much smaller. Until recently, Adam W has been the standard optimizer. Recent studies have introduced novel optimizers that claim to accelerate pretraining by 1.4x to 2x compared to AdamW Liu et al. (2024a); Vyas et al. (2025); Liu et al. (2025a); Yuan et al. (2025); Liang et al. (2025); Wang et al. (2025); Liu et al. (2025c); Pethick et al. (2025); Ma et al. (2025), yet these optimizers have yet been widely adopted in real-world pretraining DeepSeek-Al et al. (2024); Yang et al. (2025); Grattafiori et al. (2024), with notable exceptions of Kimi K2 Team et al. (2025b), which uses the Muon-clip optimizer (a variant of Muon Jordan et al. (2024)). and GLM 4.5 Team et al. (2025a), which uses Muon.
We pinpoint two problems in optimizer evaluation the evaluation process of these optimizers that both undermine confidence in new methods and limit their practical adoption. First, some baselines suffer from improperly tuned hyperparameters. Second, many experiments are confined to smaller-scale settings, leaving unanswered questions about how these optimizers perform in broader, more realistic scenarios. To address these issues, we benchmark eleven optimizers, including AdamW, in rigorously controlled setup, focusing on two key questions:

---

Published as a conference paper at ICLR 2026
C4/EN Loss vs Weight Decay
C4/EN on 130M models
4.0
AdamW w/ Ir 6
3.37
Lion
3.9
Mars
AdamW
Nesterov AdamW
3.36
3.8
AdamW w/ Ir 8
3.35
wd 0.6 is optimal for Lion
3.7
3.34
3.6
3.33
Tuning LR of AdamW 2x speedup
3.5
2000
4000
6000
8000
10000
3.32
0.0
0.2
0.4
0.6
0.8
1.0
Step
Weight Decay
C4/EN Loss Scaling For 520M Models
Speedup Model Size (8x Chinchilla)
1.5
AdamW
Muon
3.10
AdamW
Muon
Speedup wrr.t AdamW
1.4
NAdamW
Soap
Mars
Soap
Optimizers' speedup w.r.t. Adam decreases with model size
3.05
NAdamW
Kron
1.3
Matrix- based optimizers (solid) consistently outperform
3.00
scalar-based optimizers (dashed)
1.2
2.95
1.0
2.90
130M
300M
520M
1.2B
Model Size
2
8
Chinchilla Ratio
Figure Top Left: The commonly used AdamW baseline for optimizer design under-tuned. Up to a 2x speedup is achievable by tuning a single hyperparameter (learning rate) in the GPT-3 recipe Brown et al. (2020) for 100M model (adopted in Liu al. (2024a); Wen al. (2024); Yuan et al. (2025); Liang et al. (2025); Wang et al. (2025)). highlighting the importance of proper hyperparameter optimization. Top Right: Fixing hyperparameters across optimizers does not guarantee fair comparison. Shared hyperparameters such as learning rate and weight decay are commonly set to constant in previous studies. However, even conceptually similar optimizers may correspond very different optimal hyperparameters. Bottom Left: Speedup decays with model size. While some optimizers show high (1.3-1.4x) speedup over AdamW models under 1B parameters, the speedup decays with model size to only 1.1x x for 1. 1.2B parameters. Bottom Right: bassix optimizers consistently outperform scalar-based optimizers (AdamW, Nesterov Adam Mars) and three matrix-basee optimizers (Kron, Soap. Muon) trained with different Chinchilla ratios of data shown. Matrix-based optimizers achieve consistent speedup over scalar- scalar-based optimizers. Furthermore, the three matrix-based optimizers converge to a similar loss in an overtrained setting.
optimizers. The loss curves for three
1. How to ensure hyperparameter optimality? Previous works typically rely on manual hyperparameter selection and keep common hyperparameters such as learning rate and weight decay fixed across optimizers (e.g. Liu et al. (2025a)). This tuning process may result in weak baseline, the hyperparameters chosen may favor the proposed optimizers rather than for Adam W. We validate this concern by showing that tuning just hyperparameter in the widely adopted GPT-3 recipe (introduced in Brown al. (2020) and used in Liu et al. (2024a); Wen et al. (2024); Yuan et al. (2025); Liu et al. (2025d:c); Liang et al. (2025); Wang et al. (2025)) can yield 2x speedup for pretraining (Figure top left). To address, this we perform coordinate descent in the hyperparameter space on N hyperparameters for all eleven optimizers, iterating until convergence across for each of six different settings for models with up 0.5B parameters to ensure that the hyperparameters are ncar-optimal for each setting.
2. How do speedups differ across different scaling regimes? Large language models are trained in different regimes: depending on the model size, the data-to-model ratio (the number of tokens over the number of parameters) can range from 1 to more than 50 times the Chinchilla optimal (about 20 based on Hoffmann

---

Published as a conference paper at ICLR 2026
et al. (2022)). Typical optimizer experiments in 1x Chinchilla optimal regimes raise concerns about new optimizer's effectiveness in high data-to-model ratio. To address this, we benchmark the optimizers in four distinct data-to-model ratios (1x, 2x, 4x and 8x the Chinchilla optimal regime) and scale up to 1.2B parameter models (following Everett et al. (2024); Zhang et al. (2025a)).
Concretely, we employ the Llama 2 architecture Touvron et al. (2023b); Grattafiori al. (2024) (ranging from O.IB to 1.2B parameters) and a data mixture similar to OLMo 2's OLMo et al. (2025). We focus on the final validation loss on the C4-EN mixture as a known proxy for downstream performance Bhagia et al. (2024), while also tracking exact downstream performance on various benchmarks. As previous works Vyas et al. (2025); Liu et al. (2025a) show that the step-wise computation overhead of matrix-based optimizers can be reduced to under 10% through proper implementation, we primarily compare algorithms by the number of tokens needed to reach given loss.
Our empirical results show the necessity of careful hyperparameter tuning and end-of-training evaluations across a range of model scales and data-to-model ratios:
1. Hyperparameter transfer between optimizers is non-trivial. Even similar optimizers may need very different hyperparameters (e.g., Lion's optimal weight decay =0.6 vs. AdamW's =0.1, Figure 1, top right), so fixing hyperparameters across optimizers can lead to unfair comparisons.
2. The speedup of new optimizers is lower than claimed and diminishes with model size. Many reported speedups of 2x simply reflect a weak baseline. Against our well-tuned AdamW baseline, the speedup of alternative optimizers does not exceed (Figure 4). Furthermore, while new optimizers such as Muon and Soap show 1.3x speedups for small models (0.1B), the speedups diminish to around 1.1x for 1.2B parameter models at 8x Chinchilla Ratio (Figure 1, bottom left), a regime that is not tested in previous works studying the scaling law of these optimizens
3. Early-stage loss curves can mislead significantly. During learning rate decay. loss curves of different optimizers may cross multiple times (Figure 6), so judging optimizers using intermediate checkpoints may result in different ranking than comparing models at the target training budget.
Our benchmarking also reveals new insights about optimizer design:
1. Matrix-based optimizers consistently outperform scalar-based optimizers for small models. Scalar- based optimizers (e.g., AdamW, Lion, Mars, etc.) update each parameter individually using scalar operations. After proper tuning, all scalar- based optimizers achieve similar optimization speeds to Adam with an average speedup ratio of less than 1.2x. Matrix-based optimizers (e.g.. Kron, Muon, Soap, etc.) leverage the inherent matrix structure of neural network parameters and precondition gradients using matrix multiplication. Despite their diverse update rules, matrix based optimizers all deliver approximately a 1.3x speedup over AdamW (Figure 1, bottom right) for models under 520M parameters.
2. Optimal choice of optimizer shifts depends on data-to-model ratios. A winner in the 1x Chinchilla regime may be suboptimal when tata-to-modell ratio increases. For example, while Muon is consistently the best optimizer in smaller Chinchilla ratio regimes, it is outperformed by Kron and Soap when the mata-to-model ratio increases to 8x or larger (Figures 4 and 5).
2 RELATED WORKS
Due to page limit, a more detailed discussion of the related work is deferred to Appendix A.
Optimizers for Pretraining. Research on optimization for deep learning builds on classical stochastic methods Robbins and Monro (1951); Nesterov (1983); Duchi et al. (2011) and neural network-spechs -ppecffc insights Sutskever al. (2013), leading to adaptive methods such as Adam Kingma and Ba (2017) and its variants (c.g.. Adam W Loshchilov and Hutter (2019), Nesterov Adam Dozat (2016)). Subsequent improvements have addressed convergence guarantees, variance reduction Yuan et al. (2025); Pagliardini et al. (2024), memory efficiency Shazeer and Stern (2018); Zhang et al. (2025b), and large-batch training You et al. (2020). In parallel, matrix-prece preiiidprrcooiioneedd methods such as Shampoo Gupta et al. (2018), Muon Jordan et al. (2024), and Scion Pethick et al. (2025) have emerged, alongside approaches leveraging Hessian information Yao et al. (2021) or symbolic discovery Chen et al. (2023). Another popular line of research is designing better
'The original Soap paper Vyas et al. (2025) investigate model sizes up to 0.6B parameters. and Kimi's paper on Muon Liu et al. (2025a) only considers 1x Chinchilla regime.

---

Published as a conference paper at ICLR 2026
optimizers specifically for pretraining. These optimizers are our main objects of study. An (incomplete) list of optimizers and their claimed speedups over AdamW includes Sophia Liu et al. (2024a) (2x), ), Soap Vyas et al. (2025) (1.4x), Muon Jordan et al. (2024); Liu et al. (2025a) (2x), MARS Yuan et al. (2025) (2x), Cautious AdamW Liang et al. (2025) (2x). ). Block-wiss wiick-wise// Learning Rate Adam Wang et al. (2025) (2x), FOCUS Liu et al. (2025c) (2x), SWAN Ma et al. (2025) (2x), DION Ahn et al. (2025) (3 x3), and SPlus Frans et al. (2025) (2x). We present a comparison of setups with these prior works in Appendix H.
Re-evaluation Methodology. Our work is rigorous evaluation of optimizers for pretraining. Rigorous evaluation has been an important part of deep learning research to clarify the current status of research and move the community forward. Jiang et al. (2019) critically examines metrics for predicting LLMs' generalization capability and has facilitated research on understanding loss landscape sharpness and generalization and new optimizers such SAM Foret al. (2021). Reevaluations for optimizers in different settings than have also shown insight on optimizer design Schmidt et al. (2021); Zhao et al. (2025); Kasimbeg et al. (2025).
Comparison with concurrent work Semenov al. (2025) Semenov al. (2025) study the same problem of benchmarking optimizers for pretraining and reach similar high-level conclusions on the importance of weight decay, learning rate decay, and the benefits of variance-reduced Adam variants. Our results diverge in the relative ranking of matrix-leve optimizers: while we find Muon competitive with or surpassing Mars, they report the opposite. We attribute this differences in experimental regimes- larger-batch TPU-v5lite setting favors matrix-level methods, whereas their smaller-batch GPU setting amplifies the advantages of variance reduction. Additional differences in learning rate sweeps and Muon uningg strategies further contribute to the discrepancy, underscoring the sensitivity of optimizer benchmarking to batch size, hardware, and hyperparameter tuning.
METHODOLOGY
This section outlines our experimental design and evaluation protocol. Section 3.1 introduces the general setup, followed by a three-phase hyperparameter-tume tunnnn framework: Phase (Section 3.2) uses coordinate-descoe across model sizes and data ratios to identify scaling-sensitive parameters; Phase II (Sec- tion 3.3) refines these on mid-scale settings and selects optimizers: Phase III (Section 3.4) extrapolates hyperparameter scaling laws to 1.2B parameter and 16x Chinchilla regime. Op- timal configurations and loss trends appear in Appendix C.
Optimizer
References
Alg
Baseline
sweeps
AdamW
Loshchilov and Hutter (2019)
Alg.
Variance-reduced AdamW Variants
NadamW
Dozat (2016) Yuan et al. (2025)
Alg. 2
Mars
Alg. 5
Cautious
Liang al. (2025)
Alg. 7
Memory-efficient Optimizers
Lion
Chen et al. (2023)
Alg. 3 Alg. 6
Adam-mini
Zhang et al. (2025b)
3.1 GENERAL EXPERIMENTAL SETUP
Matrix-based Optimizers
Following OLMo et al. (2025), we conduct all our experiments on large-scale pretrain- ing corpus composed of three publicly avail- able datasets, tokenized with the Llama3 tok- enizer: DCLM-baseline (3.8 trillion tokens, Li et al. (2025b)), StarCoder Data (0.25 trillion tokens, Lozhkov al. (2024)), and ProofPile (55 billion tokens, Azerbayev et al. (2024)).
Muon
Jordan et al. (2024)
Alg. 8
Scion
Pethick al. (2025)
Alg. 9
Kron (PSGD)
Li (201 8a; 2022)
Alg. 10
Soap
Vyas al. (2025)
Alg. 11
Heesian-Aprroximation Optimizers
Sophia
Liu et al. (2024a)
Alg. 4
Our benchmarks cover four model sizes derived from the architecture Touvron et al. (2023a;b); Grattafiori et al. (2024), with approximately 130M, 300M, 520M, and 1.2B parameters. Each variant uses a fixed sequence length of 4,096 and 32 transformer layers (following Liu et al. (2024b)), differing only in hidden dimension, intermediate dimension, and number of attention heads. Detailed hyperparameters are summarized in Table 4. Training is implemented in JAX and executed TPU v5 hardware. We employ a mixed precisore scheme (parameters in
Table 1: Optimizers under study

---

Published as a conference paper at ICLR 2026
Stage
LR
WD
min Ir ratio
Warmup
Max Grad Norm
Batch
Val. Loss
Init 0.008
0.1
0
1000
256
3.298
Round 1
0.008
0.1
0
2000
256
3.282
Round 2
0.008
0.1
0
2000
128
3.263
Best
0.008
0.1
0
2000
128
3.263
Table 2: Illustrative coordinate-desse steps for AdamW on the 130M 1x Chinchilla regime. Changed hyperparameter values are highlighted in red; Unchanged hyperparameters: 31 = 0.9, 32 0.98, e 10
fp32 and activations in bfl6). For each model, use 20 times its non-embedding parameter count to compute the Chinchilla optimal tota-t ratio based on Hoffmann et al. (2022). We will use nx Chinchilla to represent training the models for times the Chinchilla optimal number of tokens.
Our primary evaluation metric for the model is the language modeling loss on the English split of the C4 dataset Raffel al. (2023), which has been shown to be strong proxy for downstream performance Bhagia et al. (2024). We also track downstream accuracy and bits-per-byte on the following suite of benchmarks: ARC (Easy and Challenge) Clark et al. (2018), BoolQ Clark et al. (2019), COPA Gordon al. (2012), CommonsenseQA Talmor et al. (2018), HellaSwag Zellers et al. (2019), LAMBADA Paperno et al. (2016), OpenBookQA Mihaylov et al. (2018), PIQA Bisk al. (2020), WSC273 Levesque al. (2012), and Winogrande Sakaguchi et al. (2020).
Our study includes a wide range of eleven optimizers listed in Table 1. Due to the page limit, we defer the exact algorithm descriptions of these optimizers to Appendix B. We selected these eleven optimizers according to three guiding principles: (i) include widely adopted baselines such as Adam W and Lion; (ii) cover recently proposed optimizers; and (iii) when multiple methods share similar update rule, choose few representative algorithms. These choices ensure both breadth and depth in our comparison. We group the optimizers under study into five general classes:
1. Our baseline algorithm is AdamW, with m, and o, being first- and second-order momentum of gradient. Adam W's update rule is wc+1 W/
2. Reducing the variance of updates is a shared motivation behind many optimizers. For example, Nesterov AdamW incorporates the Nesterov lookahead technique to estimate gradients more accurately, with the following update rule: Wi+1 wt 17
31 (1-mm11--11111
17 Wt- W-.-
vor+e
3. Optimizers such as Lion aim reduce the memory required by AdamW by only keeping the first- order momentum. Lion is observed to perform better than AdamW at a small scale Liang et al. (2025); W++1 w. 7) signn(222) + (1 32)9:), where m, the first-order momentum.
4. Other optimizers like Muon leverage the matrix structure of neural networks and perform preconditioning of gradients through matrix multiplication instead of scalar multiplication. For Muon, the key operation is called Newton Newton-Schulz: Schulz: NS(M) = M (aM + bMT M + c(M M)?). With appropriate a, b, c, one can prove that (5) (M) R arg max max||Ollppp= 01 Tr(O M) when ||M]]op 1. Muon has the following update rule: Wi+1 wi, - 7 (5 (82m, + (1 - 32)9(). This done for all the matrices except the token classification head and the embedding in the network, which are optimized by AdamW.
5. Optimizers including Sophia are motivated by the famous Newton's 's method and use Hessian vector product to approximate the diagonalized version of the Hessian matrix empirically.
3.2 PHASE I: FINE-GRAINED HYPERPARAMETER COORDINATE DESCENT
Fixed or lightly tuned baselines can severely understate an optimizer's capability and lead to overstated speedup claims. For instance, single tweak to the learning-rate schedule with peak learning rate 6c-4 in the GPT-3 recipe Brown et al. (2020) can produce nearly 2x speedup (Figure 1, left). To avoid such artifacts, we perform an exhaustive, one-at-a at-amtttteeeeaaeee sweep over each hyperparameter, identify the set of near-best configurations in each regime, and then determine which knobs truly require re-tuning scale changes.
We defer the result of Sophia Appendix C.4.

---

Published as a conference paper at ICLR 2026
Optimizer
Scaling-Sensitive Hyperparameters
AdamW
learning rate, warmup, weight decay, batch size learning rate, warmup
Nesterov AdamW
Muon
learning rate learning rate, warmup, block size
Soap
Table 3: Scaling-sensitiv hyperparameters identified in Phase I (Section 3.2) for selected subset of optimizers, see full list at Table 5. To reduce Soap's memory, we apply parameter blocking as in Anil et al. (2021).
C4/EN Loss for 130M Model
C4JEN for 300M Madel
CA/EN Loss for 520M Model
3.55
Adam-Mii
Castious
3.25
Adams-ini
Castious
3.10
AdaamMMni
Lion
Soap
Lion
Soap
Lien
Soap
AdamW
Muon
Adammi
Muon
AdaamW
Muan
3.45
NAdmmW
3.20
NAdamW
Scion
NAdamW
Scionn
3.40
Mars
Kron
3:15
Mars
Kron
3.00
Mars
kron
3.10
3.30
2.95
3.25
3.05
2.90
Chinchilla Ratio
Chinchilla
HellaSwag Pert for 130M Model
HellaSwag Perf for 300M Model
HellaSwag Perf for 520M Model
0.40
0.50
0.58
Accuracy
0.38
AdamW
0.46
0.48
AdamW
0.54
0.56
AdamW
0.36
NAdamiW
NAdami
0.52
NAdamI
Mars
0.34
Muon
0.42
Muon
0.50
Muon
Soap
Soap
0.32
Kionn
0.40
Kron
0.46
Kron
0.38
2
4
8
1
2
Chinchilla Ratio
Chinchilla Ratio
Chinchilla Ratio
Figure 2: Main Results For Phase & II. Top: We plot the validation loss on C4/EN for the experiments in Phase I and Phase II. Every point corresponds to the optimal loss achieved at the corresponding Chinchilla ratio for each optimizer. Bottom: we plot the HellaSwag performance corresponding to the selected run for a subset of optimizers: the A baseline, the top 2 most performant sealar-based optimizers, and the top 3 most performant matrix-based optimizers. Analysis is deferred to Section 4.1.
For each optimizer, define discrete grid for every hyperparameter (e.g., for AdamW, our swept hyperparameters include learning rate, weight decay, warmup steps, B,, 82, c, gradient- norm clipping, and batch size). Starting from initial hyperparameter configuration similar the one in the original paper or is commonly used in pretraining in each iteration, we hold all but one hyperparameter fixed at the best values, then search the whole grid for that parameter, and accept the new value if the validation loss improves by more than A = 3 x 10-3 We repeat passes until no parameter update yields further significant gain. We perform this sweeping for different settings, namely 130M, 300M, 500M at xx Chinchilla and 130M 2x, 4x, 8x Chinchilla. One exemplary hyperparameter optimization procedure for A on a model with 300M parameters and xx Chinchilla shown in Table 2.
Result of Phase I. By the end of Phase I, we have, for each optimizer and each of the six regimes (130M, 300M, 500M at 1x; 130M at 2x, 4x, 8x Chinchilla), identified a coordinate-wiss local optimum of hyperparameters- when all hyperparameters except one are held fixed, minimizes validation loss.
3.3 PHASE II: COORDINATE DESCENT ON SCALING-SENSITIV HYPERPARAMETERS
While the extensive coordinate descent guarantees coordinate-wise optimality, it is too costly to perform on larger-scale experiments. We empirically observe two crucial properties of the sweeping that allow us to simplify the descent procedure: (1) Losses are sensitive to only a subset of hyperparameters: many have little effect on performance when perturbed from their optimal values. (2) Among the sensitive hyperparameters, the optimal settings for most remain stable across scales, tuning is needed only at smaller scales.
6

---

Published as a conference paper at ICLR 2026
Dar vs Ogg (Model Size: 130M)
Day vs Dege (Model Size: 300M]
Day vs Dag (Model Size: 520M)
Acam Mini
Lion
Adam-Mini
Lion
Adam- Niri ModeeW
Lion
20
NAdamm
Muoe
10-
MAdemm))
Muan
10-
Muon
Tokens Needed by Adamm// Chinchii
Eautinus
Stap
Tokens Necied AdammW/ Chinchala
Soap
Tokkons Neeted by
Ademm Chinchile
Mars
Kron
Mars
Speedup
Speedup
Speaduu
1.0-12x
2x
12 1.3x
1.2 1.3x
12 1.3x
1.3-1.4x
1.3-1.4x
1.3-1.4x
8
0
Tokens / Chinchilla
Takens Chinchilla
Tokens / Chinchilla
Figure 3: Speedup of different optimizers across scale. We estimate the speedup of different optimizers by fitting a scaling law for Adam W and then map the loss of different optimizers to the corresponding equivalent data budget. We observe that (i) The highest speedup is capped at 1.4x; (ii) matrix-based optimizers consistently outperform scalar-based optimizers and show an increasing speedup with data budget.
Building on these observations, we can simplify our hyperparameter search by identifying scaling-sensitive parameters, which (i) crucially influence the final performance, and (ii) change with respect to the model scale. We define the approximate-optmmal configuration for a given regime as the set of all hyperparameter tuples whose final loss lies within A₂ -6.4e-3 of the regime's best-observed loss L;, where r denotes regime/setting, that is, r is a pair of choice of model size and data budget. Concretely, let ey denotes hyperparameter (where h is the index) and e be the tuple of all hyperparameters.
1. For each regime r, let C, {c: L(c) < L; + A2) be all the hyperparameter configurations in our coordinate descent procedure that yield approximately optimal loss.
2. A hyperparameter Cд is called mealing-insensiivee if there exists a single value un such that there exists E C, such that C₁ = 2% for every regime in Phase I. Otherwise, op is scaling-sensitive, meaning its optimal value shifts depending on model size or data budget.
We carry forward only the scaling-sensitiig hyperparameters (shown in Table 5) into Phase II, thereby focusing our next round of coordinate-descen sweeps on the hyperparameters that truly require re-tuning across scaling regimes. We then perform sweeping for another different settings, namely 300M, 500M at 2x, 4x, 8x Chinchilla.
Result of Phase II. Combined with the results in Phase we obtain a set of near-optimal hyperparameters and their corresponding losses for 12 different settings (130M, 300M, 500M and xx, 2x, 4x, 8x Chinchilla).
To quantify the speedup of different optimizers over the baseline AdamW, we fit how AdamW's s loss scales with data budget for each model size with the following functional form: (D(D) = a D Suppose an optimizer achieves loss Lopimizer at data budget Doptimizer- We calculate the corresponding data budget needed for A to achieve this loss, denoted by Adamw by finding the solution to the equation We then use D Dopiiizer as the estimated speedup ratio.
D-B- Bx + 3N.
LN( (Adamw) = Luptimizer
Through this set of experiments, observed that matrix- optimizers consistently outperform scalar- based optimizers, but all optimizers' speedup ratios over Adam W do not exceed 1.4x.
3.4 PHASE III: HYPERPARAMETER SCALING LAW FOR FURTHER EXTRAPOLATION
Having obtained optimized byperparameter settings from Phase (Section 3.3), we now a smooth scaling law that predicts the optimal value of each scaling-sensitive hyperparameter function of model size N and data budget D. Concretely, we model the optimal value for each scaling-sensitive hyperparameter h as: D) a A + 8, where A, B, a, and / are learned coefficients.
We estimate these parameters via non-linear least-squares on the 12 observed (N, D,h) triples for each optimizer, minimizing the squared error between predicted and actual optimal hyperparameter values. To test the quality of our prediction, we ran a full Phase I sweep at N = 1.2B and Chinchilla = 1 for AdamW. Comparing the identified optimum against our fitted hyperparameters, we observe that our predicted hyperparameters yield final loss within 3e-3 of the optimal configuration, showing that our hyperparameter scaling law can effectively predict the optimal hyperparameters. We then performed two case studies to further extrapolate our benchmarking:

---

Published as a conference paper at ICLR 2026
Dar vs Oug (Model Size: 130M0
Day vs Dege (Model Size: 300M]
Day vs Dag iModel Size: 520M)
Acimm Mini
Lion
Adam-Mii
Lion
Adam-Niri MAdamW
Lion
Tokens Needed by Adammt// Chinchii
20
NAdamm
Muoe
Stap
Tokens Needed Adammi/ Chinchlli
10-
MAdummi
Soap
Muen
Tokenss Noeded by
Adammi/ Chinchill
10-
Muon
Scep
Mars
Kron
Mars
Krun
8-
Mara
Speedup
Speedup
Speeddup
1.0-122
1.0-3 Zx
1.0-1.2x
12. 1.xx
1.2.1.30
12.11.3
1.3-1.4x
3.3-1.4x
1.3-1.4x
2
4 Takens / Chinchilla
0
Tolens/ / Chinchilla
Tokens /
Chinchilla
Figure 4: Speedup of different optimizers across scale. We estimate the speedup of different optimizers by fitting a scaling law for Adam W and then map the loss of different optimizers to the corresponding equivalent data budget. We observe that (i) The highest speedup is capped at 1.4x; (ii) matrix-based optimizers consistently outperform scalar-based optimizers and show an increasing speedup with data budget.
1. To test the effect of scaling up model sizes, we train 1.2B models using AdamW, Nesterov AdamW, and Muon on I to 8x Chinchilla ratio.
2. To further test the effect of different optimizers when data-to-model ratios are high, we train 130M and 300M models using AdamW, Nesterov Ad Muon, and Soap on 16x Chinchilla ratio.
Results of Phase III. We demonstrate two potential shortcomings of Muon optimizers, which is the best optimizers in Phase and Phase II, through experiments in this phase: (i) while Muon's 's speedup persists for models up to 1.2B parameters, the speedup decreases to under 1.2x; (ii) With a 16x Chinchilla ratio, Soap outperform Muon on the 130M and 300M models.
4 EMPIRICAL FINDINGS
4.1
MAIN RESULTS
Results on LLLB-O0.-pparamtte models. Figure 2 shows the validation loss curves for the 130M, 300M, and 520M models with varying Chinchilla ratios (1 to 8) in our benchmark. We further show that HellaS accuracy improvements closely mirror validation-loss gains. This is consistent with prior works that show lower losses translates to better downstream accuracy Bhagia et al. (2024); Liu et al. (2025b). Across all model scales and compute budgets, both the variance-reduced Adam variants (NAdamW, Mars, Cautious) and the matrix-based optimizers deliver speedups over the Adam W baseline. However, no method achieves 2 step-wiss acceleration claimed in previous literature. We note that Soap Vyas al. (2025) is one of the few works that conduct independent hyperparameter sweeping for the baseline, and it indeed reports a speedup closest to the actual observed improvement. Following the methodology defined in Section 3.3, we calculate the estimated speedup ratio of different optimizers in Figure 4 and the highest speedup ratio is 1.4x. From the measured speedups, three patterns stand out in this computation regime:
1. Matrix-based methods outperform scalar-based methods. The speedup ratio increases with data budget yet decreases with model size. For every model size, matrix-based optimizers (Soap, Muon, Kron, Scion-solid curves) consistently drive validation loss below that of their scalar-based counterparts (dashed curves). In the base (1x x Chinchilla) compute regime, Muon performs best, but at 8x Chinchilla compute, the advantage shifts to Soap and Kron. As shown by the super-linear trend in Figure 4, the speedup ratios of these three optimizers grow with increasing data budget. To the best of authors' knowledge, this dependency on data budget is not noted in prior works, which typically only experiment on one mata-to-model ratio.
However, the greatest gains occur on the 130M model, after which the speedup decreases to roughly 1.3x for larger model sizes. We will affirm this observation further in the experiments of 1.2B models, showing that the speedup ratios of matrix- based optimizers decrease to 1.1 x for the 1.2B model.
2. rediatio-reducctinn techniques provide a small but clear lift. Within the scalar-based family, all variance- reduced Adam variants (NAdamW, Mars, Cautious) surpass vanilla Adam exxept for a small lag at the smallest experiment. Muon combines mutrix-based updates with Nesterov momentum, illustrating how variance reduction compounds with matrix adaptation can yield greater efficiency. This result is very different from the 2x speedup reported in prior works and we attribute this disparity to the better-tuned baseline.
3. Memory-efficient variants of AdamW closely track the performance of AdamW. The two memory- AdamW variants (Lion, Adam-mini) despite their reduced auxiliary caute-llosely track the performance of Adam with a slowdown of at most 5% and sometimes even perform better than AdamW. Interestingly, Lion and Adam-mini show different scaling trends regarding model size: the disadvantage of Lion relative to Adam W widens while the disadvantage of Adam-mini over AdamW narrows.
8

---

Published as a conference paper at ICLR 2026
CA/EN Loss for 1.26 Model
(Model 1.28)
CAJEN Loss for 300M Model
2
AdamW NidamW
NAdamiy Muon
3.25
2
Seap
Tokans Nesded by
Adtemtl Chinchile
10
3.20
Adamil
Seap
2.86
2.84
3.15
2.82
2.80
Speedup
1.0-12x
2.78
1.2 1.3x
2.76
1.3-1.4x
3.00
16
Chinchilla Ratio
Tokens/ Chinchilla
Chinchilla Ratio
Figure 5: Case Studies. Left: Validation loss scaling on 1.2B model for AdamW, NAdamW, Muon and Soap. Muon and Soap still offer significant speedup over Adam W but no longer significantly speed up over NAdamW. Mid: Estimated speedup ratio with the same methodology Figure 4, we observe that Muon and Soap's speedup decays with model size to only 1.1x. Last: Experiment with 300M 16x Chinchilla setting, Soap outperform: Muon when data-to-model ratio further increases.
Results on -2B-parameters Models. Using the hyperparameter scaling law fit (Section 3.3), we scale up the model size to 1.2B to examine how the speedup of optimizers scales with model size, We observe that NAdamW, Muon, and Soap still deliver speedup over AdamW, but the speedup deminishes to 11.1 x for all these optimizers (Figure 5, Left and Mid) and no longer leads to downstream improvements (Table 6). While the exact reason for the speedup decay is not clear, conjecture that the landscape-ddaptation advantage of matrix-based optimizers plays a smaller role at the largest model scale we tested, where the noise factor may be more important for training dynamics, leading to similar performance between Muon and Nesterov AdamW.
High data-to-model Ratio. In our previous experiments, Muon is outperformed by Soap in the 8x Chinchilla regime for the 130M and 520M models. We further train three 300M models 16x Chinchilla and verify that Muon is no longer the optimal optimizer when the data-to-model ratio increases (Figure 5, right). We conjecture that the second-order momentum maintained by Soap and Kron becomes more effective, and adaptivity to heterogeneity in parameter directions may lead to larger speedup when the data-to-moder ratio increases. We also perform similar experiments on 130M models and reach the same results (deferred to Appendix C.5).
4.2 NECESSITY OF RIGOROUS BENCHMARKING
Our systematic sweeps uncover both universal optimization principles and surprising optimization-specifee nuances, which call for rigorous study when designing future optimizers.
First, find that even a superior optimizer can underperform less advanced method when its hyperparameters are not precisely tuned. In our exhaustive grid searches, slight deviations from each optimizer's ideal learning rate or other critical hyperparameter often lead to degradation in validation loss that is large enough to flip the ordering (Figure 6, Left). This hyperparameter sensitivity means manual selection without systematic sweeps will likely produce arbitrary ranking of the optimizers. To further demonstrate this, we plot how validation losses vary when only one of the hyperparameters deviates from optimal value for Muon, Soap. Mars on 520M model in 1x Chinchilla regime (Figure 6, Mid). The ordering of the optimizers can easily flip if one chooses sub-optimal hyperparameter.
Second, using the same hyperparameters for different optimizers do not guarantee fair comparison between optimizers. For example, while weight decay is essential across all optimizers for optimal performance, the optimal decay strength varies markedly between optimizers. As shown in (Figure I, top right), optimal weight decay coefficients differ between the optimizers studied. We also note the optimal weight decay for Kron is larger than the conventional 0.1 and is approximately 0.5, which is crucial for Kron to outperform Adam
Third, early training behavior can be highly misleading. Validation-loss curves during this initial phase tend to exaggerate performance gaps (Figure I, bottom right) and, in some cases, even reverse the eventual ranking, (Figure 6, right). Many optimizers exhibit rapid early descent followed by plateauing, meaning that assessments based solely on losses before the end of the training trajectories fail to predict final outcomes.
4.3 COMMON PHENOMENA ACROSS OPTIMIZERS
Through logging the evolution of weight and gradient norms, we discovered some shared optimization phenomena across optimizers.
Parameter norms typically track learning rate decay when there is weight decay We observe that the parameter norms of all optimizers show a similar pattern of increase and decrease, closely aligning with the increase and decrease of the learning rate if there is decay phase. However, the absolute values of parameter norms very different across optimizers (Figure 7, Middle Left).

---

Published as a conference paper at ICLR 2026
1.20
5230 Chili loss on cRty
3.6-
520e Chinchéla less on Careh
52de Ja Chischil vos Carth
3.15
3.10
3/4
3.4
g
1.2
Mart 0e-3
311
10-
Winn WD-3.7
2.4
Soap [e-)
3.-
may maleed
3.-
Misn UD: 3
2.2
Soap fe-3
Muon WD-333
20000
40020
50000
50020
5.07
3.03
20003
68003
60000
8C550
5020
19000
15000 20203
Figure 6: Necessity of Careful Tuning. Left: 2x the optimal learning rate diminishes Soap's ' loss improvement over Mars on 520M 8x experiment; Mid: Variation of loss when only one hyperparameter differs from optimal learning rate and runs converge to within 0.02 of optimal. The order of optimizers may flip arbitrarily if rigorous tuning is missing. Right: Changing single hyperparameter like weight decay may lead to misleading faster loss improvement but plateaus later.
Leaming Rate For
Model Chinchil
(g Noem For 28 Model Ex Chincills
Generatization Aczowa Dptimizers
de-3
3-3
Maon
70030 Stop
80020
Must
Mum
Ködant!
3
4.-
60030
2.5
Adamw
2--3
50020
2.0
5.4
40000
1.5
1-33
10030
1.0
Man
500
20020
10030
Stop
Stap
300%
150k
2.6 28 3.0
3.2 Trining Loss
34
38 38
Figure 7: Common Phenomena Across Optimizers. Left: Learning rate used for different optimizers. Middle Left: Parameter norm of all optimizers shows a similar trend of increment and decrease, closely aligning the increasing and decaying of learning rate schedule. Middle Right: Gradient norm increases during learning rate decay. However, this increase does not lead to a loss increase. Right: The training loss and evaluation loss follows the same trend for all optimizers.
Gradient norm increases during learning rafe decay. Across all the optimizers, the gradient norms increase during the training run. However, this increase does not lead to a loss increase. Similar to the parameter norms, the absolute values of gradient norms are not consistent across optimizers. These two phenomena are also reported in the previous work Defazio (2025), where the author also provides a theoretical explanation for both phenomena. We provide additional evidence that these two phenomena not artifacts of the chosen optimizer Adam W but rather common phenomena across optimizers (Figure 7, Middle Right).
Different optimizers have similar generalization behavior, For architecture design, it has been observed that different architectures can have vastly different generalization behaviors Lu et al. (2025). However, this is not the case for optimizers, and the evaluation losses and training losses follow roughly the same trend for all optimizers at the regimes we experimented on (Figure 7, Right).
5 CONCLUSION AND LIMITATIONS
We benchmarked 11 optimizers in pretraining and found that their true gains over AdamW are much smaller than previously reported. Our results highlight three key lessons: (i) many claimed speedups stem from insufficient hyperparameter tuning, as fair sweeps climinate most of the apparent ad advantages; (ii) comparisons based on early or inconsistent evaluation can be misleading, since optimizer rankings often change over the full training trajectory; and (iii) even the berf-performing alternatives provide only modest speedups, which further diminish with model scale, dropping to 1.1x x at 1.2B parameters. This benchmarking study has the limitation that it does we haven't scaled to models larger than 1.2B parameters. However, we believe that evaluating optimizers on models of comparable size to prior studies is still valuable, as it reveals that insufficient tuning is major cause of subpar speedups. LLM is used to fix grammar mistakes for this paper.
LEDNOWLEDGEMENT
This work supported by the Google TPU Research Cloud (TRC), the Stanford HAI HAl-Google Cloud Credits Program, the DARPA AIQ program, Schmidt Sciences and NSF RI 2045685, and is a part of the Marin Project. The authors thank Evan Walters, Omead Pooladzandi, Jiacheng You, and Zhiyuan Li for discussions and help during our research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates, 2020. URL
org/abs/20022 11803.
K. Ahn, B. Xu, N. Abreu, and J. Langford. Dion: Distributed orthonormalized updates, 2025. URL https: /ttp///axvvvvv..//////eee/e..av. orggabs//504/.4 05295.
E. AI, :, I. Shah, A. M. Polloreno, K. Stratos, P. Monk, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Tanwer, J. Shah, K. Nguyen, K. Smith, M. Callahan, M. Pust, M. Parmar, P. Rushton, P. Mazarakis, R. Kapila, S. Srivastava, S. Singla, T. Romanski, Y. Vanjani, and A. Vaswani. Practical efficiency of muon for pretraining, 2025. URL www.thttp//... http//axvvv.//// hwtp://////aaaa/ooooooooo 2at/////s/22..... 02222.
R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer. Scalable second order optimization for deep learning, 2021. URL https: www.vvixvvvvv/////vv///..//aaavv wtw.aatww......oooooooooo.oooorl abs / 2002. 09018.
Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. McAleer, A. Q. Jiang, J. Deng, S. Biderman, and S. Welleck. Llemma: An open language model for mathematics, 2024. URL
0bs/2330.1080
S. Becker and Y. L. Cun. Improving the convergence of back- propaarooagttion learning with second order methods. D. S. Touretzky, G. E. Hinton, and J. Sejnowski, editors, Proceedings of the 1988 Connectionist Models Summer School, pages 29-37. San Francisco, CA: Morgan Kaufmann, 1989.
S. Bergsma, N. Dey, G. Gosal, G. Gray, D. Soboleva, and J. Hestness. Power lines: Scaling laws for weight decay and batch size in Ilm pre-training, 2025. URL https: /ttttp:://rxxi.... /ttp://aa......//////.........
J. Bernstein and L. Newhouse. Modular duality in deep learning, 2024a. URL BtLtttts/////xi..oom abs/2411 0. 21265.
Bernstein and L. Newhouse. Old optimizer, new norm: An anthology, 2024b. URL httptt:p///ae org/abs/ /2409.24999 24..22032........
J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar. signSGD; Compressed optimisation for non-convex problems. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 560-569. PMLR, 10-15 Jul 2018. URL ht.ts::
pro.eediiggg.................... ml press / v80 http/bbeeeeeinnneeeeeeeeeeeeeeee htmml
A. Bhagia, Liu, A. Wettig, D. Heineman, O. Tafford, A. H. Jha, L. Soldaini, N. A. Smith, D. Groeneveld, P. W. Koh, J. Dodge, and H. Hajishirzi. Establishing task scaling laws via compute-efficient model ladders, 2024. URL https: / arxiv. org/ abs 2412. 04403.
Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 7432-7439, AAAI Press, 2020. doi: 10.1609/aaai. 04440566239) URL htwps:tt////tooc /tti:://doooo org/ 10. / Saai.v34i0n 6239,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models few-shot learners, 2020. URL https:t///// http:s://axxiii http///xxivvvo// org/ aaaaabba/2000 2200 14165.
X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, H. Pham, X. Dong, T. Luong, C.-J. Hsich, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms, 2023. URL pttttps://axxiv.oo/ /bbs//// 2302.06675.
C. Clark, K. Lee, M.WW. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of NAACL-HLT 2019, pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (0.18653/v//119. 11NN199-100.. 1300. /t..httss//aallttthooooo
org /N1 9-1000/.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. CoRR, http://aa /atpiv/aaxiv..o wtggabs/////0 05457.
2018. URL.

---

Published as a conference paper at ICLR 2026
DeepSeek-AI, ;, X. Bi, D. Chen, G. Chen. S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y. Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y. Piao, J. Qiu, H. Qu, T. Ren, Z. Ren, C. Ruan, Z. Sha, Z. Shao, J. Song, X. Su, J. Sun, Y. Sun, M. Tang, B. Wang, P. Wang. S. Wang. Y. Wang. Y. Wang. T. Wu, Y. Wu, X. Xie,
Z. Xie, Z. Xie, Y. Xiong, H. Xu, R. X. Xu, Y. Xu, D. Yang. Y. You, S. Yu, X. Yu, B. Zhang, H. Zhang, L. Zhang, L. Zhang. M. Zhang, M. Zhang, W. Zhang, Y. Zhang, C. Zhao, Y. Zhao, S. Zhou, S. Zhou, Q. Zhu, and Y. Zou. Deepseek Ilm: Scaling open-source language models with longtermism, 2024. URL https: arwiv. org/ abs/ 2a.b//400 2401. 02954.
DeepSeek-AL. D. Guo, D. Yang. H. Zhang, J. Song. R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wanng X. Bi, X. Zhang, X. Yu, Y. Wu, Z. Wu, Z. Gou. Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang. B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang. C. Ruan, D. Dai, D. Chen, D. Ji, F. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang. H. Bao, H. Xu, H. Wang. H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. J. I.. Cai, Ni. J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang. L. Zhang, L. Zhao, L. Wang. L. Zhang, L. Xu, L. Xia, M. Zhang. M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q Wang. Q Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang. R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang. S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng. W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie,
X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao Y. Sun, Y. Wang. Y. Yu, Y. Zhang, Y. Shi. Y. Xiong. Y. He. Y. Piao, Y. Wang. Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou,
Y. Wang. Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang,
Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Xu, Z. Xie,
Z. Zhang, Z. Hao, Z. Ma. Z. Yan. Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 2025a. URL https://///... / /tt///aavvvvv//////e////e//...o/ wwt.///g//////a/oooooooooo
12948.
DeepSeek-AL, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang. D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang. H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu, J. Li, J. Song, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Wang. L. Zhang. M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li. N. Tian, P. Huang, Wang, Zhang, Q. Wang. Q. Zhu, Q. Chen, Q. Du. R. J. Chen, R. L. Jin, R. Ge, R. Zhang, R. Pan, R. Wang, R. Xu, R. Zhang, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Pan, Wang, T. Yun, T. Pei, T. Sun, W. L. Xiao, W. Zeng, W. Zhao. W. An. W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang. X. Q. Li, X. Jin, X. Wang, X. Bi.
X. Liu, X. Wang. X. Shen, X. Chen, X. Zhang, X. Chen, X. Nie X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yu, X. Song, X. Shan, X. Zhou, X. Yang, X. Li, X. Su, X. Lin, Y. K. Li, Y. Q Wang, Y. X. Wei, Y. X. Zhu, Y. Zhang, Y. Xu, Y. Xu, Y. Huang, Y. Li, Y. Zhao, Y. Sun, Y. Li, Y. Wang, Y. Yu, Y. Zheng, Y. Zhang. Y. Shi, Y. Xiong, Y. He, Y. Tang. Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Wu, Y. Ou,
Y. Zhu, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Zha, Y. Xiong, Y. Ma, Y. Yan, Y. Luo, Y. You, Y. Liu, Y. Zhou,
Z. F. Wu, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Huang, Z. Zhang, Z. Xie, Z. Zhang, Z. Hao, Z. Gou,
Z. Ma. Z. Yan, Z. Shao, Z. Xu, Z. Wu, Z. Zhang, Z. Li, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Gao, and Z. Pan. Deepseek-v3 technical report, 2025b. URL https: / http/aaxvvv./////////////a org/ htt///bs//22111 24122412.
A. Defazio. Why gradients rapidly increase near the end of training, 2025. URL abs/2506. 02285.
A. Defazio and K. Mishchenko. Learning-rate- learning by d-adaptation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1449-7479, PMLR, 23-29 Jul 2023. URL. https: / / ptwc/eooooeoo//oooooooooo ngs.mn.......................n ml r. press / v22 / de fazi o233a .t..m...l..................a
A. Defazio. A. Cutkosky, H. Mchta, and K. Mishchenko, Optimal linear decay learning rate schedules and further refinements, 2024a. URL https:://axxii...tt org. abs/22100 07831.
A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled, and A. Cutkosky. The road less scheduled, 2024b. URL https: / /ttp/vxivvv.////////////////aav/ www.wiiwwioiiooooooooooooooooooe /tt/////55555 152...
12

---

Published as a conference paper at ICLR 2026
T. Dozat. Incorporating nesterov momentum into adam. 2016.
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
R. Eschenhagen, A. Immer, R. Turner, F. Schneider, and P. Hennig, Kronecker-factored approximate curvature for modern neural network architectures. Advances in Neural Information Processing Systems, 36: 2023.
R. Eschenhagen, A. Defazio, T.-H. Lee, R. E. Turner, and H.-J. M. Shi. Purifying shampoo: Investigating shampoo's s heuristics by decomposing its preconditioner, 2025. URL https::/arxiv..o abs/ 2506. 03595.
K. Everett, L. Xiao, M. Wortsman, A. A. Alemi, R. Novak, J. Liu, I. Gur, J. Sohl-Dickstein, L. P. Kaelbling, J. Lee, and J. Pennington. Scaling exponents across parameterizations and optimizers, 2024. URL https: / httpi//.rggabb//////// 05872.
P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2021. URL https: /tte://///xxii wwtpaaawwwww..aa//a//aoaaaa 2101..0112. 01412.
K. Frans, S. Levine, and P. Abbeel. A stable whitening optimizer for efficient neural network training, 2025. URL httph:tt:/ //arxi:///rxxv.... org/ abaa 07254.
A. Gordon, Z. Kozareva, and M. Roemmele. Semeval-2011 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In E. Agirre, J. Bos, M. Diab, S. Manandhar, Y. Marton, and D. Yuret, editors, *SEM 20/2: The First Joint Conference on Lexical and Computational Semantics (SemEval 2012), pages 394-398, Montréal, Canada, June 7-8 2012. Association for Computational Linguistics. URL https:/
ogy- org/ //1 112-111555 1052/.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang. A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, H. AlBadawy, E. Lobanova, F. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, Zhang, G. Synnaeve, G. Lee, G. L.. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang. G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadcokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins. L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev. N. Chatterji. N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong. R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre. R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang. S. Hosseini, S. Chennabasappa, S. Singh. S. Bell, S. S. Kim, S. Edunov, S. Nie. S. Narang, S. Raparthy, S. Shen, S. Wan. S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan. S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov. T. Xiao, U, Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. F. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang. C. Kim, C. Zhou, C. Hu, H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin,
13

---

Published as a conference paper at ICLR 2026
D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich. D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, F. Arcaute, F. Dunbar, F. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, R. Caggioni, Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi. Zhang, G. Lakshminarayanan H. Inan, H. Shojanazeri, H. Zou, H. Wang. H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj. I. Molybog, I. Tufanov, I. Leontiadis, L.-E. Veliche, I. Gat, J. Weissman, J. Geboski. J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong. J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang. K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg. L. A. L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson. M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark,
M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li. R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. I. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang. S. Z.Zang S. Wang. S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfiel,, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang. Z. Zhao, and Z. Ma. The llama herd of models, 2024. URL https: : / //arxvivv... www.iiccgg.aab.ee.. 2407 22 21793.
V. Gupta, T. Koren, and Y. Singer. Sahampoo: Preconditioned stochastic tensor optimization, 2018. URL https:// http://axxvvvvee/ wwwpa/wwwwp...///////..2 09568.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks,
J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models, 2022. URL https: /ttpa:://aaxaaa arxiv. wwg.///i..../oo.o.ooooooooooooot abs 2203.223.22 15556.
S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long. Z. Zheng. Y. Fang, Y. Huang, W. Zhao, X. Zhang, Z. L. Thai, K. Zhang, C. Wang, Y. Yao, C. Zhao, J. Zhou, J. Cai, Z. Zhai, N. Ding, C. Jia, G. Zeng, D. Li, Z. Liu, and M. Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. 06395.
URL hwtpt:tts///ettt
M. Ivgi, O. Hinder, and Y. Carmon. Dog is sgd's best friend: A parameter-free dynamic step size schedule, 2023. URL https: / arxii.iiviv..... orgg abs 22022.3.2.222 12022.
Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and Bengio. Fantastic generalization measures and where to find them, 2019. URL https: http//axxvvv..o//// org/ abs 1912. 02.78.
K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL
P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang. Z. Nado, S. Medapati, P. Hennig, M. Rabbat, and G. E. Dahl. Accelerating neural network training: An analysis of the algoperf competition, 2025. URL https: //tp://///iiiii/ org/
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL abs/1412 6980.
T. Large, Y. Liu, M. Huh, H. Bahng, P. Isola, and J. Bernstein. Scalable optimization in the modular norm, 2024. URL https: //tpxaaa.vvvv///////////ee .rrg absg//bb/// abs/ 2bbs/2005.. 14813.
H. J. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. KR, 2012: 13th, 2012.
14

---

Published as a conference paper at ICLR 2026
H. Li, W. Zheng, J. Hu, Q. Wang, H. Zhang, Z. Wang, S. Xuyang, Y. Fan, S. Zhou, X. Zhang, and D. Jiang. Predictable scale: Part i - optimal hyperparameter scaling law in large language model pretraining, 2025a. URL wwtphhtpp////....m http///xxvvvvoo// org/ aww.aaaaa50 2503. 04715.
J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Gadre, H. Bansal, E. Guha, S. Keh, K. Arora, S. Garg, R. Xin, N. Muennighoff, R. Heckel, J. Mercat, M. Chen, S. Gururangan, M. Wortsman, A. Albalak, Y. Bitton, M. Nezhurina, A. Abbas, C.-Y. Hsieh, D. Ghosh, J. Gardner, M. Kilian, H. Zhang. R. Shao, S. Pratt, S. Sanyal,
G. Ilharco. G. Daras, K. Marathe, A. Gokaslan, J. Zhang, K. Chandu, T. Nguyen, I. Vasiljevic, S. Kakade, S. Song. S. Sanghavi, F. Faghri, S. Oh, L. Zettlemoyer, K. Lo. A. El-Nouby, H. Pouransari, A. Toshev, S. Wang. D. Groeneveld, L. Soldaini, P. W. Koh, J. Jitsev, T. Kollar, A. G. Dimakis, Y. Carmon, A. Dave, L. Schmidt, and V. Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2025b. URL https: /ttppp.s//aaxi..oo// org/ abs/ 00t.//44...19119 11794.
X. Li. Black box lie group preconditioners for sgd, 2022. URL wtw.tttpps//aaii.oo///ooooo/ wwwpx/iwwwrgg////op// 04422.
X.-L. Li. Preconditioned stochastic gradient descent. IEEE Transactions on Neural Nenworks and Learning Systems, 29(5):145-1146,, May 2018a. ISSN 2162-2388. doi: 10.1109/mbs:201122677777 URL
http://dx... doi .rg. /1/g///..00 1109 /TNNLS 2011..677977.8.
X.-L. Li. Preconditioner on matrix lie group for sgd, 2018b. URL http:::/aaravv..or///bbb///// 10232.
K. Liang, L. Chen, B. Liu, and Q. Liu. Cautious optimizers: Improving training with one line of code, 2025. URL https::///// /ecpss//aazeee attpaaaxvvv..o....aaaaaaa org// abs / 411. 16085.
H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training, 2024a. URL https::////... /atpss///arxviie atwpaaiivvvv/io ort. 2305. 14342.
J. Liu, J. J.Su,X, X. Yao, Z. Jiang, G. Lai, Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Yiu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang. Muon is scalable for Ilm training, 2025a. URL https: / aawxiv. org/ abs abs/2502 2502. 1 169669 82.
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond, 2021. URL https: / /atxiv..vivvv////////.aa org/ 1aabbs/1990 1908.998.000 03265.
Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin. Regmix: Data mixture as regression for language model pre-training, 2025b. URL https:t/////// :/apxiiv////xv....//a///am . org/ abs /2407.01492
Y. Liu, Z. Liu, and Gore. Focus: First order concentrated updating scheme, 2025c. //arxiv. / htabs//550.122/.
URL https:
Z. Liu, C. Zhao, F. landola, C. Lai, Y. Tian, Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi, L. Lai, and V. Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases, 2024b. URL https: /atp:://aaxvvveee wwwp//iiiorrroooooooooe /ass////////// /a////bs/////// /2402. 14905.
Z. Liu, Y. Liu, E. J. Michaud, Gore, and M. Tegmark. Physics of skill learning, 2025d. URL https: //arxiv. orggorrgaooooo aa///////// /2501. 12391.
Loshchilov and F, Hutter. Decoupled weight decay regularization, 2019. URL URRhstps://axxi..oom abs abs/1711.00 05101.
A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Loirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y. Belkada, Z. Wang, Q. Liu, D. Abulkhanov, Paul, Z. Li, W.DD Li, M. Risdal, J. Li, J. Zhu, T. Y. Zhuo, E. Zheltonozhskii, N. O. O. Dade, W. Yu, L. KrauB, N. Jain, Y. Su, X. He, M. Dey, E. Abati, Y. Chai, N. Muennighoff, X. Tang, M. Oblokulov, C. Akiki, M. Marone, C. Mou, M. Mishra, A. Gu, B. Hui, T. Dao, A. Zebaze, O. Dehaene, N. Patry, C. Xu, J. McAuley, H. Hu, T. Scholak, S. Paquet, J. Robinson, C. J. Anderson, N. Chapados, M. Patwary, N. Tajbakhsh, Y. Jernite, C. M. Ferrandis, I. Zhang. S. Hughes, T. Wolf, A. Guha, L. von Werra, and H. de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL /tt.htts::////xtii http://axvvvvv///
http://////aiiiooo///a////////// wwt..ogggbb///....... 2402. 19173.
X. Lu, Y. Zhao, S. Wei, S. Wang, B. Qin, and T. Liu. How does sequence modeling architecture influence base capabilities of pre-trained language models? exploring key architecture design principles to avoid base capabilities degradation, 2025. Jttpttts: //axxiivv.../////////a/////aa org/ abs/2505. 18522.
15

---

Published as a conference paper at ICLR 2026
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate, 2019. URL htwps: /atps://arra wawiiv.vivv..///vi///aa//a/iaav. wwwpeiiw.w......ee..e....oooeeee
/ 902. 09843.
Y. Luo, X. Ren, Z. Zheng, Z. Jiang. X. Jiang, and Y. You. Came: Confidence guiieedguddddddd adaptive memory efficient optimization, 2023. URL https: http::////xvvooooeeeooeee obg/ /tab///3.... 2307 .0077.0247
C. Ma, W. Gong, M. Scetbon, and E. Meeds. Swan: Sgd with normalization and whitening enables stateless Ilm training, 2025. URL htw.t:tp////. htttttps:/aaxiiv.rrrg/. atapaaxxivv.ooo http//bs//22// /tt.a////222211..1......
M. Marek, S. Lotfi, A. Somasundaram, A. G. Wilson, and M. Goldblum. Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful, 2025. URL
http://axxvv.rrgg.o 2507 07101.
Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curvature, 2020. abs/ /atbbs//553.0000 5555..5.67.
URL https::p////..
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381 1-911, Brussels, Belgium, Oct-Nov 2018. Association for Computational Linguistics. doi: 10.18653/w//D1--4 1260. URL https: http:/://claattthlloooo org org/ 1260/.
K. Mishchenko and A. Defazio, Prodigy: An expeditiously adaptive parameter-free learner, 2024. URL https://arxiv. org/ h2tp://bb///....../// 06101.
I.-V. Modoranu, M. Safaryan, G. Malinovsky, E. Kurtić, T. Robert, P. Richtárik, and D. Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence. Advances in Neural Information Processing Systems, 37:1 43, 2024.
D. Morwani, I. Shapira, N. Vyas, E. Malach, S. Kakade, and L. Janson. A new perspective on shampoo's preconditioner, 2024. URL httpt: /arxivv. wwwoorrg//s.... 2406 17748.
Y. Nesterov. A method for solving the convex programming problem with convergence rate o (1/k2). In Dokl akad nauk Sssr, volume 269, page 543, 1983.
T. OLMo, P. Walsh, I.. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, M. Guerquin, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg. D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. olmo furious, 2025. URL
https: www.aaiivvvvo....a .rg.
M. Pagliardini, P. Ablin, and Grangier. The ademamix optimizer: Better, faster, older, 2024. URL attps://arxxt httpvv.ivvvvvvv//. orriii.orrraaae abtp//b////400 03137.
D. Paperno, G. Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In K. Erk and N. A. Smith, editors, Proceedings the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 11111165/11111111111 144.. URL https:
T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher. Training deep learning models with norm-constrained Imos, 2025. URL https: /ttp///axvvvvv/. org /abs///////// 2502.. 07529.
C. Raffel, N. Shazeer, Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li. and J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL htttt:t/////ai// abs/ 1910.1111111 10683.
S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018.
: revneeview.... net/ forum? d=ryQu7E-RR.
H. Robbins and S. Monro. stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
16

---

Published as a conference paper at ICLR 2026
K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 8732-8740. AAAI Press, 2020. doi: 10. 1609/aai..344055663..... URL https: httpp::///aai.... aaai. v34 05.0005.00 6399.
//w.///oii/ooo/ http:////oooooo//////// 1609/
T. Schaul, S. Zhang, and Y. LeCun. No more pesky learning rates, 2013. URL htRLttts///axii.oorr abs/ / 110//2206... 1106.
R. M. Schmidt, F. Schneider, and P. Hennig. Descending through crowded valley benchmarking deep learning optimizers, 2021. URL /ttp:://aaxiiiioiooee /tlaa/iii...gg/////////7./ 01547.
A. Semenov, M. Pagliardini, and M. Jaggi. Benchmarking optimizers for large language model pretraining, 2025. URL https: / arxiv. org/ abaaabss/////
N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL https: / /awpivxivvi/.ii//aaaa/aaaarlaa . owg. aataaaas//100 1803.04235......
I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1139-1147, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https: h http://pooceeeng..e/eeoo/o ml wawpinns..l.......ss... w.....peessvv8 sutskeveri 3. html.
A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. CoRR, abs/1811.00937, 2018. URL http://arxiv.oort /abs///a///// 1881. 00937.
S. Taniguchi, K. Harada, G. Minegishi, Y. Oshima, S. C. Jeong, G. Nagahara, T. liyama, M. Suzuki, Y. Iwasawa, and Y. Matsuo. Adopt: Modified adam can converge with any Bo with the optimal rate, 2024. URL https: / //axxvivvvvv////////e///aaaavv owg//org//.....ooooo........o..e http//bs//211// 2411. 02853.
Team, A. Zeng. X. Lv, Q. Zheng. Z. Hou, B. Chen, C. Xie, C. Wang. D. Yin, H. Zeng. J. Zhang, K. Wang, L. Zhong, M. Liu, R. Lu, S. Cao, X. Zhang, X. Huang, Y. Wei, Y. Cheng, Y. An, Y. Niu, Y. Wen, Y. Bai, Z. Du, Z. Wang, Z. Zhu, B. Zhang, B. Wen, B. Wu, B. Xu, C. Huang, C. Zhao, C. Cai, C. Yu, C. Li, C. Ge, C. Huang, C. Zhang, C. Xu, C. Zhu, C. Li, C. Yin, D. Lin, D. Yang, D. Jiang, D. Ai, E. Zhu, F. Wang. G. Pan, G. Wang. H. Sun, H. Li, H. Li, H. Hu, H. Zhang, H. Peng. H. Tai, H. Zhang, H. Wang. H. Yang.
H. Liu, H. Zhao, H. Liu. H. Yan, H. Liu, H. Chen, J. Li, J. Zhao, J. Ren, J. Jiao, J. Zhao, J. Yan, J. Wang. J. Gui, J. Zhao, J. Liu, J. Li. J. Li, J. Lu, J. Wang, J. Yuan, J. Li, J. Du, J. Du, J. Liu, J. Zhi, J. Gao, K. Wang, L. Yang. L. Xu, I.. Fan, I.. Wu, L.. Ding, L.. Wang, M. Zhang, M. Li, M. Xu, M. Zhao M. Zhai P. Du, Q. Dong, S. Lei, S. Tu, S. Yang, S. Lu, S. Li, S. Li, Shuang-Li, S. Yang, S. Yi, T. Yu, W. Tian, W. Wang, W. Yu, W. L. Tam, W. Liang, W. Liu, X. Wang, X. Jia, X. Gu. X. Ling, X. Wang, X. Fan, X. Pan, X. Zhang, X. Zhang, X. Fu, X. Zhang, Y. Xu, Y. Wu, Y. Lu, Y. Wang, Y. Zhou, Y. Pan, Y. Zhang. Y. Wang. Y. Li, Y. Su,
Y. Geng, Y. Zhu, Y. Yang. Li, Y. Wu, Y. Li, Y. Liu, Y. Wang. Y. Li, Y. Zhang, Z. Liu, Z. Yang, Z. Zhou,
Z. Qiao, Z. Feng, Z. Liu, Z. Zhang, Z. Wang. Z. Yao, Z. Wang. Z. Liu, Z. Chai, Z. Li, Z. Zhao, W. Chen, J. Zhai, B. Xu, M. Huang, H. Wang, J. Li, Y. Dong, and J. Tang. Glm-4.5: Agentic, reasoning, and coding 06471
(arc) foundation models, 2025a. URL https: petctts://aree //aaxi//..ii//// www.vvvoooo orgg
K. Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, Z. Chen, J. Cui, H. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, H. Gao, P. Gao, T. Gao,
x. Gu, L. Guan, H. Guo, J. Guo, H. Hu, X. Hao, T. He, W. He, W. He, C. Hong, Y. Hu, Z. Hu, W. Huang, Z. Huang, Z. Huang, T. Jiang, Z. Jiang, X. Jin, Y. Kang, G. Lai, C. Li, F. Li, H. Li. M. Li, W. Li, Y. Li, Y. Li, Z. Li, Z. Li, H. Lin, X. Lin, Z. Lin, C. Liu, C. Liu, H. Liu, J. Liu, J. Liu, L. Liu, S. Liu, T. Y. Liu, T. Liu, W. Liu, Y. Liu, Y. Liu, Y. Liu, Y. Liu, Z. Liu, F. Lu, I.. Lu, S. Ma, X. Ma, Y. Ma, S. Mao, J. Mei, X. Men, Y. Miao, S. Pan, Y. Peng, R. Qin, B. Qu, Z. Shang, I. Shi, S. Shi, F. Song, J. Su, Z. Su, X. Sun, F. Sung, H. Tang, J. Tao, Q. Teng, C. Wang, D. Wang. F. Wang, H. Wang. J. Wang, J. Wang. J. Wang. S. Wang, S. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang. Z. Wang, Z. Wang. Z. Wang, C. Wei, Q. Wei, W. Wu, X. Wu, Y. Wu, C. Xiao, X. Xie, W. Xiong, B. Xu, Xu, J. Xu, L. H. Xu, L. Xu, S. Xu, W. Xu, X. Xu,
Y. Xu, Z. Xu, J. Yan, Y. Yan, X. Yang. Y. Yang. Z. Yang. Z. Yang, Z. Yang. H. Yao, X. Yao, W. Ye, Z. Ye, B. Yin, L. Yu, E. Yuan, H. Yuan, M. Yuan, H. Zhan, D. Zhang. H. Zhang, W. Zhang, X. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Z. Zhang, H. Zhao, Y. Zhao, H. Zheng, S. Zheng, J. Zhou, X. Zhou, Z. Zhou, Z. 7hu, W. Zhuang, and X. Zu. Kimi k2: Open agentic intelligence, 2025b. URL https: : /ttp//aaiiv/////////////.//..a.a org/ abs/ abs/2507 205344.
17

---

Published as a conference paper at ICLR 2026
T. Tieleman. Lecture 6.5 6.p-rmpp Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26, 2012.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro,
F. Azhar, A. Rodriguez. A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023a. URL https: // www.i.
abs 2302. 13971.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, Lavril, Lee, D. Liskovich, Y. Lu, Y. Mao. X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu. Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL ptccps://ariiie org. abs/ / 2337 09288.
N. Vyas, D. Morwani, R. Zhao, M. Kwun, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade. Soap: Improving and stabilizing shampoo using adam, 2025. URL https: org org/ /bsp///bb/ 11321.
Wang, M. Wang, Z. Zhou, J. Yan, W. E, and L. Wu. The sharpness disparity principle in transformers for accelerating language model pre-training, 2025. URL httttsss/aazx...er//a/////////
K. Wen, Z. Li, J. Wang, D. Hall, P. Liang, and T. Ma. Understanding warmup-stable-deun wndeeesaandiingggwauuuuuuuabbbaa river valley loss landscape perspective, 2024. URL hRR..tttss///aar...orr////////// 05192.
learning rates: A
X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2024. URL https: : aaaiv.iiiivv..
Z. Xie, X. Wang, H. Zhang, I. Sato, and M. Sugiyama. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum, 2022. URL httttpp://axxxi..orggg/// 2ttbbb///066.55.....0 15815.
A. Yang, A. Li, Yang, B. Zhang, B. Hui, Zheng, Yu, C. Gao, C. Huang, Lv, Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang. J. Tu, J. Zhang, J. Yang. J. Yang, J. Zhou, Zhou, Lin, K. Dang, K. Bao, K. Yang. L. Yu, L.. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025. URL https::///// /tttpp:://axxtt //tr///aaavvv//////////eeeeaawe/ org. abs/ 25002 09388.
G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation, 2020a. URL https://aaee wrtp/bs/199// 04760.
G. Yang. Tensor programs ii:: Neural matrix laws. arXiv preprint arXiv:2009.10683 2020b.
G. Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:200.115548, 2020c.
G. Yang. Tensor programs Wide feedforward or recurrent neural networks of any architecture are gaussian processes, 2021. URL https: //axxiv, org/ abs/ 1940.
G. Yang and E. Littwin. Tensor programs ivb: Adaptive optimization the infinite-width limit, 2023. URL https: / /arriviv....////////aa/////aaava wwwpid/..rr/bb////////a 01814.
G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL https: http:://axxvvvvo absb/bb3.3.. 03466.
G. Yang, J. Simon, and Bernstein. spectral condition for feature learning, 2024. URL https:
/ //arxiv.. org/ http///s.//3..... 1718.1
18

---

Published as a conference paper at ICLR 2026
Z. Yao, A. Gholami, S. Shen, M. Mustafa, K. Keutzer, and M. Mahoney. Adahessian: An adaptive second order optimizer for machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 10665-10673, May 2021. doi: 10. 10.1609/aaai, htt.aaaiiii.ii12.277.... URL https://///s http///ssaaaaaaaaaaaaaaaaaaaaaaa . org/ nddx. php/AAA htt.A//aarrileeeeeee
Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks, 2017. URL https:
//arxiv. abs 1708. 03888.
Y. You, J. Li, Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, Song, J. Demmel, K. Keutzer, and C.-J. Hsieh, Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint a9Xiv:1904.00 1904,0004220 2019.
Y. You, Li, S. Reddi, Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.J. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2020. URL https: http://awxvv..rraarr./aa/a////ae
904. 00962.
H. Yuan, Y. Liu, S. Wu, X. Zhou, and Q. Gu. Mars: Unleashing the power of variance reduction for training large models, 2025. URL https: // //arxiv...//////////// org/ arggabs/////eooo...o/..g/o/ 2411 10438.
M. Zaheer, Reddi, D. Sachan, Kale, and Kumar. Adaptive methods for nonconvex optimization. Advances in neural information processing systems, 31, 2018.
M. D. Zeiler. Adadelta: An adaptive learning rate method, 2012. URL UtL/ 5701.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4791-4800, 2019. doi: 10. 1655//11/P19-1472 WtLhttps:// /ttp:://///oi...o11...1.1111 V1/P19-144
H. Zhang, D. Morwani, N. Vyas, J. Wu, D. Zou, U. Ghai, D. Foster, and Kakade, How does critical batch size scale in pre-training?, 2025a. URL https: arxiv. org/ hat//ab///a abs/2410, 226666.
M. R. Zhang, J. Lucas, G. Hinton, and J. Ba. Lookahead optimizer: k steps forward, step back, 2019. URL https:/ / //aaaaavvvooo/// wwrrxiv.org//ee. worg/bbs//i /1977. 08610.
Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. P. Kingma, Y. Ye, Z.-Q. Luo, and R. Sun. Adam-mini: Use fewer learning rates gain more, 2025b. URL httpt::tt///rttt
org/ abs 2406. 16793.
Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-effieient Ilm training by gradient low-rank projection, 2024.
204.bbb577 03507.
R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. Kakade. Deconstructing what makes good optimizer for language models, 2025. URL
H. Zhu, Z. Zhang, W. Cong. X. Liu, S. Park, V. Chandra, B. Long. D. Z. Pan, Z. Wang, and J. Lee. Apollo: Sgd-like memory, adamw-level performance, 2025. URL https: aarii..iiiv org/ /abs////////// 1abbs/2444 22112 05270.
Zhuang, Tang, Y. Ding, Tatikonda, N. Dvornek, X. Papademetris, and J. Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients, 2020. URL https: wwt.:vv.rrgga////aaaaaoooeee 07468.
19

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Optimizers for Deep Learning. A long line of work has studied optimization for deep learning, incorporating insights from classical optimization literature Robbins and Monro (1951); Nesterov (1983); Duchi et al. (2011) and domain knowledge about deep neural networks Sutskever et al. (2013). (i) Early insights motivated the rise of optimizers that use adaptive learning rates based on second-order momentum Tieleman (2012); Zeiler (2012); Kingma and Ba (2017). Adam Kingma and Ba (2017) later became the default baseline for optimizers with adaptive learning rates. Since then, improvements over Adam and SGD have been proposed, with notable examples including Nesterov Adam Dozat (2016) and Adam W with decoupled weight decay Loshchilov and Hutter (2019). Other improvements include addressing the convergence of Adam on convex loss Reddi et al. (201 8); Zaheer et al. (201 8); Taniguchi et al. (2024), considering interpolation between Adam and SGD to improve generalization Luo et al. 2019); Xie et al. (2022), performing further variance reductions on optimizer updates Liu al. (2021): Zhang et al. (2019); Yuan et al. (2025); Xie et al. (2024); Pagliardini et al. (2024); Zhuang et al. (2020), incorporating momentum on weights Ivgi et al. (2023); Defazio et al. (2024b), allowing easier hyperparameter tuning Defazio and Mishchenko (2023); Mishchenko and Defazio (2024); Defazio et al. (2024a), reducing memory usage by incorporating the structure of neural networks Shazeer and Stern (2018); Zhang et al. (2025b); Zhu et al. (2025); Luo et al. (2023); Modoranu et al. (2024); Zhao et al. (2024), and modifying the algorithm to allow larger batch sizes You et al. (2017: 2020). (ii) Starting from Preconditioned SGD Li (2018a) and Shampoo Gupta et al. (2018), another line of optimizer design Morwani et al. (2024); Eschenhagen et al. (2023): Martens and Grosse (2020); Li (2018b); Eschenhagen et al. (2025) began incorporate matrix preconditioners instead of simple scalar preconditioners. Techniques including learning rate grafting Agarwal et al. (2020), blocking, and distributed methodology Anil et al. (2021) have since been proposed. These matrix-based approaches later led to the theory of modular duality in deep learning optimization Bernstein and Newhouse (2024a:b); Large et al. (2024) and new optimizers including Muon Jordan et al. (2024) and Scion Pethick et al. (2025). (iii) Motivated by Newton's s algorithm, there has been line of work that tries to incorporate Hessian information Becker and Cun (1989); Yao et al. (2021); Schaul et al. (2013); Yao et al. (2021). (iv) Symbolic discovery of optimizers Chen et al. (2023) has discovered eeemory-efficieee SignGD Bernstein et al. (2018) variant called Lion, which claims outperform Adam wide range of tasks.
Optimization for Pretraining. Since Brown et al. (2020), the cost of pretraining has increased dramatically, One of the challenges is how to choose hyperparameters with minimal cost. A pivotal line of work in this direction is the tensor program series that allows for extrapolating some hyperparameters across scales Yang (2020b:c;a; 2021); Yang and Littwin (2023); Yang et al. (2022; 2024). Empirical results suggest that fitting power law to scale hyperparameters is now common practice for large models Li al. (2025a); Everett et al. (2024); Liu et al. (2025a); DeepSeek-AI et al. (2024); Zhang et al. (2025a). We have also incorporated this approach in our paper. Another popular line of research is designing better optimizers specifically for pretraining. These optimizers are our main objects of study. An (incomplete) list of optimizers and their claimed speedups over AdamW includes Sophia Liu et al. (2024a) (2x), Soap Vyas et al. (2025) (1.4x), Muon Jordan et al. (2024); Liu et al. (2025a) (2x), MARS Yuan et al. (2025) (2x), Cautious Adam Liang et al. (2025) (2x), ), Block-wise Learning Rate Adam Wang et al. (2025) (2x), FOCUS Liu et al. (2025c) (2x), SWAN Ma et al. (2025) (2x), DION Ahn et al. (2025) (3x), and SPlus Frans et al. (2025) (2x). We present comparison of setups with these prior works in Appendix H.
Benchmarking Optimizers Schmidt et al. (2021) re-evaluated optimizers at that time and showed that (i) which optimizer is optimal is problem-specifie and (ii) rigorous hyperparameter tuning is required and uncqual tuning can account for most of the claimed speedup. Unlike Schmidt et al. (2021), we evaluate the LLM pretraining task and include modern optimizers they did not test. However, we arrive at a similar conclusion: rigorous and fair hyperparameter tuning is still not the norm, but rather the exception in optimizer design research. Zhao et al. (2025) also examines how different optimizers perform on the pretraining tasks. However, the focus of Zhao al. (2025) is on understanding loss structure, and the optimizers tested in the paper are shown to have slower convergence compared to AdamW, whereas the optimizers tested in this paper show small but significant improvements in convergence speed. The Algoperf competition Kasimbeg et al. (2025) evaluates different optimizers across different settings and arrives at similar conclusion that (i) matrix- optimizers and (ii) denoising methods such as Nesterov momentum lead to speedup over AdamW. Our works focus on the pretraining setting and investigates the effect of scaling dataset and model sizes.
Comparison with concurrent work Semenov et al. (2025) Semenov et al. (2025) investigates the same topic of benchmarking optimizers for pre-training. These two works agree on many high-level points, c.g., (i)
20

---

Published as a conference paper at ICLR 2026
non-zero weight decay and decaying to small learning rate are essential for pretraining, (ii) variance-reduced A variants such as Mars show non-trivial speedups over vanilla Adam
However, our results differ in the relative performance of matrix-level optimizers. Our paper shows that matrix-level optimizers such as Muon can achieve significant speedups over variance-reduced versions of AdamW, such as Mars, their study finds that AdEMaMix and Mars outperform Muon. Our initial investigation suggests this discrepancy largely stems from the differences in the batch sizes used in experiments. Their most extensively tuned experiments on 130M models use batch size of only O.IM and 0.02M tokens, whereas our experiments operate with tuned batch sizes that are not smaller than 0.4M tokens. This difference is likely result of different hardware regimes. We leverage 128 TPU-v5lite chips (which approximately equivalent to 12 H100 GPUs), where only larger batches (larger than 0.4M) can fully utilize the parallelism in the compute. On the other side, their experiments appear to be primarily conducted on 1-8 H100 GPUs, where smaller batches might be preferable. Since Mars and AdEMAMix both perform gradient averaging and variance reduction, these methods are advantangeous in their noise-dominated small-batch regime, whereas in our larger- batch setting these benefits diminish and matrix-level optimizers become more competitive.
In their larger scale experiments, Semenov et al. (2025) increase both model size 720M parameters and batch size to 1M tokens, respectively, which is closer to our setting. However, we differ in hyperparameter tuning methodology: (i) we conduct more extensive sweeps over learning rates on our 520M scales experiments and generally find higher values (4e-3 to 8e-3) compared to their choices (1e-3 to 2e-3), which are transferred from smaller scale and for some optimizers are not tuned; and (ii) for Muon, we separately tune the learning rates for the embedding layers and the matrix weights, which also improves its performance. These differences highlight the sensitivity of optimizer benchmarking to hardware and tuning strategies, underscoring the importance of carefully controlled experimental design when comparing optimizer performance.
B OPTIMIZER DEFINITIONS
In this section, we present the algorithm for each optimizer we evaluated.
the following notation: w. for model parameters, 9t. for gradients step for learning rate, for weight decay, 31, 82 for moment decay rates, for numerical stability, fnorm for gradient norm, and m, v for first and second moments, All operations are element-wise unless specified.
Throughout this section,
Algorithm AdamW
Hyperparameters: B1, 32, X, Inorm
State: m. V Update Rule:
0: = St min Jnorm
1
m, = 31 m, + (1 (1) 9r:
1t2 = 32 1; + (1 - B2) 82
m,
Ut
m.
0r =
/1 1
- $1
my
W++1
W/ 17
1 \ we-
V0g +
21

---

Published as a conference paper at ICLR 2026
Algorithm Nesterov AdamW
Hyperparameters: 31, 82, e, n, Inorm
State: m,, U
Update Rule:
92 gr min (1, Inorm
|1ge l12
31 m + (1 31) 9r.
U1 32 Ut (1 (2) 92.
m = 31 m, + (1 31) 9
m
V1
m
83
m
W/+1
Wt 7
V0r + 6
Algorithm Lion
Hyperparameters: 31, B2, n, X, Juorm
State: m
Update Rule:
9. min(1, (1, Inorm
||
mr B1 m + (1 B1) gr.
m4+1 B2 m4-1 (1 (32) gc:
We+1 w. ssiin sm..) Aw
22

---

Published as a conference paper at ICLR 2026
Algorithm Sophia-H
Hyperparameters: {me M=1. x, k, B1, 82, 7, State: me 0, h1-k 0, 01
For t 1.....,
9t VoL.(0.)
my = 31 m4-1 + (1 - f1) gt
If mod K = 1:
V gt T, Vou,
h - TOw
he 12 + (1 - B2) h
Else: ht = h4-1
0, = 0, 0c
Oc+1 Og m clip max(y hesey 1)
Algorithm MARS
Hyperparameters: B1, 32, 7, e, T,, X, Fnorm
State: m, 9t-1
Update Rule:
B1
G. 9t + 7
(gt 9:
1 31
à = min(1,
Inorm
||ce|12 }.
m, 31 mp-1 + (1 31) ct:
Dt = 32 2/-1 + (1 - (32 E2
m, m
0 Ut
g
my
W++1
Wi !| Vig + c
71 Wt.
23

---

Published as a conference paper at ICLR 2026
Algorithm Adam-mini
Hyperparameters: 31, 32, e, 7, Inorm
State: m (with the same shape as w), v (one scalar for each block)
Setup:
1. Partition all parameters into param blocks: please refer to Zhang et al. (2025b) for the exact partition scheme.
2. We will use wt, and Me. to denote the parameters block at step
Update
9. = min (1, Inorm
11 ll
m4 B1 m4-1 + (1 - 31) 9t.
m
m = - 8
for each block E param_block
vr B2 1t- (1 me)m g2.0)
0,, 1 35
W/ - 11 m/ (v,6 + e) - A 1.-
Algorithm Cautious
Hyperparameters: BI, 32, X, Inorm
State: m, V Update Rule:
Inorm
9: = min
||9:|12
me = B1 m + (1 - B1) gc,
Ut B2 14-1 + (1 - (22 0r-
m+
0t
m =
1 31
rit
4
S; 1l [u: gt > 0),
v Ug +
0 U
Wh+1 wi 0+ 17 A wt-
(s)
24

---

Published as a conference paper at ICLR 2026
Algorithm Muon
Hyperparameters: B, v, e, 31, $2, Addams AAdmm: X, Inorm
State: m
Update Rule For Weights in LM Head, Embedding, and LayerNorm: Same Adam W Update Rule For Matrices in Transformer Layer:
gt min (1, Inorm
99 2
m4 = 3 m4-11 + gr:
3 m + Dr,
u NewtonSchula(u, steps 5);
V max(1, 1, rows(w))
cols())
u 8 u,
W++1 We 17 17 Wy.
Newton-Schulz Orthogonalization (Operating on Matrices):
X
X
transpose <- (rows(X) cols( (X)),
x
transpose: X f X for = 1
A X
B -4.750 A + 2.0315 A
X < 3.4445 X + B BX,
transpose: X XT, return X.
Algorithm Scion
Hyperparameters: 3, n, e, 3, 32, DSignGD Inorm
State: m
Update Rule For Matrices in LM Head and Embedding:
G: gr min(1, Inorm
||9a||2
m 31 m + (1 - B1) 9:,
Wi-1 = 7/ggnGD sign m.))
Update Rule For Matrices Transformer Layer Same as Muon
25

---

Published as a conference paper at ICLR 2026
Algorithm 10 PSGD Kron
Hyperparameters: 31, 7 X. 6, Inorm normalize grads, partition_ _nttoo__t____ anno_blockss merge_small simes_dmmlld,,,,,,,__- block_siee size, target mergede Lesiidiim_iiiie sime_szze, Pupd(t), Setup:
If mnerge_small dims is True, then try merging small dimensions into single dimension with size target_ lerngedddim_stt dim_size greedily.
If partition_grads_inte geradd_into_blocks is True, then partition all parameters into block_sizz x block_siz blocks.
Define unfold, the function that unfolds all dimensions except the i-th dimension into single dimension and fold, as the inverse function.
State (per block (): Denote w()) as the parameters in block l and assume it has shape d1 x xd2 x dn.
1 with the same shape as (
0 (i 1, n), lower-triangular matrix with shape d, x d,
1.
if Inorm 0: g: 9: min Inorm else 9t 119e|12
9.
if carmalize_gradee Ôe 1lăell2 + else 9: = 9e
#4
31) 9c, Me 11-1 + (1 31)
Pu +- 1 Br
2. Balance check: bal_ctr + bal_etr + 1.If bal_ctr 2 100, then Q! + balanco (Q.
bal ctr O. Balance function: balance(Q,)) performs the following steps:
(a) Compute the maximum absolute value for each row/column of Q!" norms;
maxj,& (
(b) Calculate the geometric mean: gmean, (TT norms,
(c) Scale each element: Q (2) gmean
norms
(d) Return the balanced
Q!")
3. Preconditioner update: with probability Pupa(1) (t),
(a) Random probe: ((O) tree_random_lkker like( (o))
(b) Dampen: K + -p( mean((a! ((1)
(c) Conjugate sketch (B):
( = v(0), x fold; ((Q!0)² -
unfold((X
B x(m).
(d) Pre-sketch (A):
A y(0) ((0) yO) fold, (Φ. un(old, A= y(n).
(e) For each i 1,...,n. n
M, (nfold;(A),
C, unfold
Ti = M,M, To = =C,C'"
8 max
(Ti + T2) u,
u,v
Q ! a
T T2 Q!
4. (Precondition gradient)
( ((0),
(( fold, (Q unfold, (G(il))),
7/00 ((d)
5. (Weigh-decay & update)
26
(6)
(
+ w (
(8)
< 1 g)
(8)
9b
gi
W
wi

---

Published as a conference paper at ICLR 2026
Algorithm 11 SOAP
Hyperparameters: 31, 32, μ., k, e, block guorm
Iuorm
State: m, v, GA. GB. QA. QB
Partition all parameters into block_ size x block_size block
Update Rule For Each Block:
Inorm
0: min (1,
11 2
9: - Q De QB.
my = 31 m 1 + (1 - 31) Dc.
1 = B2 1-1 + (1- 32) 02.
f = m
i 2t
B
32
11++1 W2 1/ Q7(
mh
QB:
Vūt +
= MA= + (1 - 11) dt D2 +el, =n= + (1- M) de + el,
ift mod k = 0: Q^ QR(GA Qa),
Qp QR(Gn QB).
Batch Size
524288
1048576
2097152
adamw 3.046
3.055
muon
3.029
3.056
B.
EQUIVALENCE BETWEEN OUR PARAMETERIZATION AND CONSTRAINED SCION
We note that our parameterization for Scion is equivalent a constrained -Siinn--Snnnn formulation, with the only difference being that we did not separately rescale the weight decay for the embedding and language-modd model head parameters. Concretely, one can establish mapping between the two schemes as follows.
Let us consider a linear layer with weight W, input dimension din, and output dimension dout Suppose the constrained-Sett update is written in the form
AW -/ r 0 n/ W
where O is the orthogonalized momentum (after appropriate scaling), r/ is the "base" learning rate, and r is spectral radius parameter for constrained Scion.
In contrast, our version applies update
AW = 0n - 17 W W where v is the learning rate and À the weight decay coefficient. We can match the two updates by selecting
x=1
= r/ r,
in the constrained Scion setting (or in the unconstrained case). This ensures that
0 7 X W - 1 W.
For layers involving the "sign" component (e.g. the signed momentum S), the constrained-Sen Scinn variant further Thus the update takes the form
scales by a factor a
S dim
AW
n
1/ W.
which corresponds to
n'r 11
di
dn
r
in the original formulation with weight decay.
27

---

Published as a conference paper at ICLR 2026
Optimizer
Scaling-Sensitive
Hyperparameters
A
learning rate, warmup, weight decay, batch size learning rate, warmup
Nesterov A
Lion
learning rate, beta2
Adam-mini
learning rate, weight decay, warmup
Cautious
learning rate, batch size
Mars
learning rate, warmup, betal learning rate, learning rate learning rate, warmup, block size learning rate
Scion
betal, # decay steps in WSD Hu et al. (2024)
Muon
Soap
Kron
Table sealing-sensitvee hyperparameters identified in Phase (Section 3.2). To reduce the memory required by Soap, apply the parameter blocking as in Anil et al. (2021)
C OMITTED EXPERIMENTS
C.1 ARCHITECTURE HYPERPARAMETER DETAILS
Model
Params
Seq Len
Hidden Dim
Inter Dim
# Layers
# Heads
Llama-130M
130M
4096
512
2048
32
8
Alama-3001
300M
4096
768
3072
32
12
Alama-520M,
520M
4096
1024
4096
32
16
Llama-1.22 1.2B
1.2B
4096
1536
6144
32
24
Table Detailed architecture hyperparameters for each model size we studied.
C.2
SCALING SENSITIVE HYPERPARAMETERS
C.3
SCALING LAW
Based on Appendices D and E, we fitted our scaling law for the 1.2B run of Muon, NAdam and Adam we rondd the fitted value our grid of hyperparameter and used hyperparametens shown in Appendix F. After training the models, we fitted scaling law for Muon and AdamW of the following form:
L(N, D) aN +DD
The fitted values are
For Adam W. a 21,4289, 0.1555, 276.4235, B 0.2804, 7 1.7324, with RMS error of x 10
For Muon a 32, 7458, A = 0.1864, 3 59.0221, B = 0.2074, = 1.8063, with RMS error of 5 10-3
This scaling law predicts that when the parameter scale reaches 7B, Muon will actually result in higher loss compared to AdamW in 1x Chinchilla regime.
C.4 SOPHIA EXPERIMENTS
We performed Phase experiment with Sophia, with detailed hyperparameter settings shown in Appendix D. We found that Sophia tends to underperform AdamW in smaller compute regimes and eventually slightly outperforms AdamW when either the model size or the data size increases.
We also note that the evaluaton in Sophia Liu et al. (2024a) largely follows the correct procedure of the comparisons in our paper the peak learning rate was tuned to be optimal for the baseline. However, as the data loader in the codebase used did not fully randomize the order of the data, the optimal peak learning rate in
28

---

Published as a conference paper at ICLR 2026
that code base for AdamW is significantly smaller than the optimal peak learning rate for fully randomized data loader setting. turns out that Sophia doesn't offer significant speedup over Adam
C4/EN Loss Model Size at 1x Chinchilla
C4/EN Loss for 130M Model
3.55
AdamW
-
Sophia
AdamW
Sophia
Loss at 1x Chinchilaa
3.5
3.50
3.4
3.45
3.40
3.3
3.35
3.2
3.30
3.25
130M
300M
520M
1
2
8
Model Size
Chinchilla Ratio
Figure 8: Sophia Experiments. Left: Loss curve of Sophia and AdamW in 1x Chinchilla setting, Right: Loss curve of Sophia and AdamW for 130M model size.
C,5 HIGH DATA-TO-MODEL RATIO
C4/EN Loss for 130M Model
Muon
NAdamW
3.50
AdamW
Soap
3.45
3.40
3.35
3.30
3.25
3.20
8
16
Chinchilla Ratio
Figure 9: More Case Studies. Experiment with 130M 16x Chinchilla setting, SOAP outperforms Muon in the overtraining setting.
C.6 EVALUATION PERFORMANCE
Table 6: Benchmark performance of 1.2B models with different optimizers and Chinchilla scaling.
Optimizer LAMBADA OpenBook Wino PIQA BoolQ WSC273
Hella ARC-C. ARC-E COPA
Avg
8x Chinchilla (193B)
AdamW 67.16
41.40
64.96
76.12
68.59
82.78
67.56
43.43
74.49
85.00
67.
NAdamW 67.84
40.20
64.80
77.15
68.10
83.52
67.40
43.34
73.61
81.00
66. 66.70
Muon 67.53
39.80
67.09
77.09
68.81
80.95
67.67
43.34
73.53
84.00
66.98
29

---

Published as a conference paper at ICLR 2026
Table 7: Evaluation Performance for Mars, Model Size
130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.327
0.377 0.402 0.451
OPENBOOKQA
0,274
0.304 0.298 0.302
WINOGRANDE
0.521 0.512 0.526 0.522
PIQA
0.625 0.652 0.662 0.678
BOOLQ
0.616 0.504 0.569 0.557
WSC273
0.557 0.560 0.579 0.590
HELLASWAG OSHOT
0.308 0.343 0.365 0.399
ARC CHALLENGE
0.225 0.258 0.272 0.247
ARC EASY
0,462 0.517 0.532 0.544
COPA
0.650 0.690 0.660 0.710
FINAL C4 LOSS
3.537 3.396 3.323 3.247
Table 8: Evaluation Performance for Mars, Model Size = 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.450 0.493
0.514 0.538
OPENBOOKQA
0.300 0.312 0.328 0.328
WINOGRANDE
0.534 0.557 0.549 0.561
PIQA
0.680 0.686 0.695 0.720
BOOLQ
0.480 0.533 0.612 0.608
WSC273
0.586 0.608 0.667 0.667
HELLASWAG OSHOT
0.395 0.437 0.471 0.502
ARC CHALLENGE
0.253 0.269 0.298 0.307
ARC EASY
0.547 0.580 0.608 0.616
COPA
0.700 0.740 0.720 0.730
FINAL C4 LOSS
3.249 3.158 3.097 3.040
Table 9: Evaluation Performance for Mars, Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.513
0.548
0.576
0.601
OPENBOOKQA
0.330 0.336 0.366 0.378
WINOGRANDE
0.548 0.559 0.586 0.601
PIQA
0.704 0.717 0.725 0.735
BOOLQ
0.559 0.598 0.625 0.631
WSC273
0.601 0.696 0.736 0.780
HELLASWAG OSHOT
0.462 0.513 0.546 0.579
ARC CHALLENGE
0.294 0.317 0.333 0.351
ARC EASY
0.611 0.648 0.665 0.676
COPA
0.730 0.760 0.750 0.750
FINAL C4 LOSS
3.101 3.015 2.955 2.906
30

---

Published as a conference paper at ICLR 2026
Table 10: Evaluation Performance for Muon, Model Size
130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.358
0.396
0.431
0.456
OPENBOOKQA
0,286
0.288 0.318 0.300
WINOGRANDE
0.515 0.518 0.530 0.541
PIQA
0.640
0.660 0.670 0.678
BOOLQ
0.571 0.504 0.583 0.452
WSC273
0,553 0.564 0.579 0.582
HELLASWAG OSHOT
0.330 0.354 0.381 0.407
ARC CHALLENGE
0.231 0.244 0.269 0.276
ARC EASY
0,493 0.511 0.543 0.559
COPA
0.650 0.720 0.730 0.660
FINAL C4 LOSS
3.464 3.369 3.296 3.240
Table 11: Evaluation Performance for Muon, Model Size
300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.459
0.492 0.529 0.550
OPENBOOKQA
0.302 0.310 0.330 0.346
WINOGRANDE
0.504 0.534 0.548 0.570
PIQA
0.679 0.689 0.695 0.714
BOOLQ
0.482 0.555 0.599 0.570
WSC273
0.615 0.641 0.659 0.656
HELLASWAG OSHOT
0.406 0.447 0.481 0.509
ARC CHALLENGE
0.270 0.289 0.290 0.316
ARC EASY
0.564 0.591 0.620 0.638
COPA
0,670 0.730 0,700 0.700
FINAL C4 LOSS
3.224 3.143 3.079 3.032
Table 12: Evaluation Performance for Muon, Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.520
0.559
0.597
0.611
OPENBOOKQA
0.328 0.340 0.342 0.368
WINOGRANDE
0.560 0.567 0.594 0.590
PIQA
0.713 0.718 0.730 0.736
BOOLQ
0.590 0.555 0.633 0.613
WSC273
0.659 0.667 0.762 0.736
HELLASWAG OSHOT
0.482 0.525 0.554 0.587
ARC CHALLENGE
0.300 0.318 0.351 0.348
ARC EASY
0.623 0.640 0.671 0.674
COPA
0.730 0.760 0.730 0.780
FINAL C4 LOSS
3.073 3.002 2.945 2.900
31

---

Published as a conference paper at ICLR 2026
Table 13: Evaluation Performance for Lion, Model Size 130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.316 0.380 0.417 0.450
OPENBOOKQA
0,280 0.296 0.312 0.292
WINOGRANDE
0.503 0.518 0.519 0.521
PIQA
0.629 0.649 0.663 0.680
BOOLQ
0.603 0.507 0.419 0.489
WSC273
0.564 0.538 0.590 0.623
HELLASWAG OSHOT
0.304 0.343 0.366 0.402
ARC CHALLENGE
0.229 0.272 0.262 0.272
ARC EASY
0.464 0.500 0.536 0.539
COPA
0.700 0.670 0.730 0.730
FINAL C4 LOSS
3.552 3.409
3.331 3.252
Table 14: Evaluation Performance for Lion, Model Size 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.428 0.471 0.509 0.535
OPENBOOKQA
0.306 0.320 0.326 0.346
WINOGRANDE
0.506 0.530 0.532 0.564
PIQA
0.676 0.686 0.699 0.713
BOOLQ
0.486 0.490 0.609 0.593
WSC273
0.579 0.615 0.608 0.667
HELLASWAG OSHOT
0.386 0.428 0.466 0.502
ARC CHALLENGE
0.267 0.296 0.294 0.322
ARC EASY
0.535 0.586 0.600 0.618
COPA
0,690 0.730 0,710 0.760
FINAL C4 LOSS
3.268 3.170 3.100 3.046
Table 15: Evaluation Performance for Lion, Model Size 520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.507 0.543 0.569
0.597
OPENBOOKQA
0.336 0.334 0.356 0.358
WINOGRANDE
0.560 0.548 0.581 0.590
PIQA
0.699 0.713 0.726 0.740
BOOLQ
0,600 0.546 0.612 0.631
WSC273
0.608 0.667 0.685 0.700
HELLASWAG OSHOT
0.463 0.509 0.545 0.571
ARC CHALLENGE
0.303 0.320 0.340 0.371
ARC EASY
0.603 0.638 0.664 0.673
COPA
0.710 0.750 0.740 0.770
FINAL C4 LOSS
3.108 3.029 2.965 2.915
32

---

Published as a conference paper at ICLR 2026
Table 16: Evaluation Performance for NAdamW, W. Model Size 130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.325
0.383
0.413 0.446
OPENBOOKQA
0,284
0.286
0.308 0.322
WINOGRANDE
0.515 0.507 0.546 0.507
PIQA
0.633
0.652 0.667 0.682
BOOLQ
0.591 0.569 0.459 0.557
WSC273
0.535 0.560 0.612 0.597
HELLASWAG OSHOT
0.312 0.348 0.368 0.399
ARC CHALLENGE
0.239 0.245 0.242 0.270
ARC EASY
0,465 0.503 0.518 0.547
COPA
0.680 0.700 0.650 0.720
FINAL C4 LOSS
3.531 3.394 3.319 3.251
Table 17: Evaluation Performance for NAdamW W. Model Size = 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.437
0.477
0.514 0.543
OPENBOOKQA
0.310
0.332 0.316 0.348
WINOGRANDE
0.520
0.536 0.549 0.564
PIQA
0.681 0.693 0.708 0.712
BOOLQ
0.539 0.564 0.520 0.612
WSC273
0.582 0.637 0.645 0.700
HELLASWAG OSHOT
0.397 0.434 0.475 0.505
ARC CHALLENGE
0,264 0.275 0.292 0.307
ARC EASY
0.553 0.589 0.605 0.632
COPA
0.710 0.730 0,700 0.750
FINAL C4 LOSS
3.248 3.160 3.090 3.039
Table 18: Evaluation Performance NAdamW. W. Model Size = 520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.500
0.551
0.577
0.599
OPENBOOKQA
0.332 0.350
0.358 0.342
WINOGRANDE
0.553 0.571 0.594 0.597
PIQA
0.695 0.719 0.731 0.738
BOOLQ
0.613 0.606 0.639 0.618
WSC273
0.637 0.689 0.718 0.714
HELLASWAG OSHOT
0.469 0.513 0.550 0.580
ARC CHALLENGE
0.294 0.322 0.342 0.354
ARC EASY
0.605 0.646 0.657 0.674
COPA
0.700 0.720 0.760 0.780
FINAL C4 LOSS
3.100 3.013 2.955 2.907
33

---

Published as a conference paper at ICLR 2026
Table 19: Evaluation Performance for Kron, Model Size
130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.341
0.387
0.426 0.451
OPENBOOKQA
0,268 0.278 0.296 0.314
WINOGRANDE
0.520 0.522 0.515 0.543
PIQA
0.636 0.662 0.662 0.683
BOOLQ
0.533 0.530 0.575 0.557
WSC273
0.564 0.564 0.553 0.604
HELLASWAG OSHOT
0.323 0.347 0.374 0.397
ARC CHALLENGE
0.235 0.259 0.256 0.281
ARC EASY
0.487 0.519 0.543 0.572
COPA
0.700 0.690 0.710 0.710
FINAL C4 LOSS
3.492 3.389 3.307 3.239
Table 20: Evaluation Performance for Kron, Model Size = 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.429
0.485
0.516 0.548
OPENBOOKQA
0.310
0.314 0.314 0.326
WINOGRANDE
0.527
0.534 0.542 0.571
PIQA
0.675
0.690 0.706 0.709
BOOLQ
0.487 0.475 0.610 0.611
WSC273
0.546 0.619 0.659 0.718
HELLASWAG OSHOT
0.396 0.438 0.479 0.501
ARC CHALLENGE
0.262 0.285 0,298 0.323
ARC EASY
0.551 0.586 0.618 0.632
COPA
0,670 0.720 0.720 0.750
FINAL C4 LOSS
3.244 3.151 3.083 3.031
Table 21: Evaluation Performance for Kron, Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.512
0.549
0.569
0.603
OPENBOOKQA
0.338
0.332 0.350 0.352
WINOGRANDE
0.536
0.567 0.575 0.601
PIQA
0.721
0.713 0.724 0.739
BOOLQ
0.529 0.560 0.537 0.559
WSC273
0.648 0.663 0.696 0.762
HELLASWAG OSHOT
0.474 0.516 0.556 0.582
ARC CHALLENGE
0.287 0.312 0.344 0.353
ARC EASY
0.616 0.648 0.662 0.681
COPA
0.710 0.720 0.790 0.750
FINAL C4 LOSS
3.084 3.009 2.946 2.900
34

---

Published as a conference paper at ICLR 2026
Table 22: Evaluation Performance for Scion, Model Size 130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.355 0.382
0.427 0.446
OPENBOOKQA
0,282 0.300 0.288 0.314
WINOGRANDE
0.515 0.502 0.502 0.535
PIQA
0.641
0.655 0.675 0.684
BOOLQ
0.574 0.518 0.532 0.474
WSC273
0.505 0.531 0.553 0.608
HELLASWAG OSHOT
0.323 0.354 0.374 0.401
ARC CHALLENGE
0.247 0.241 0.276 0.264
ARC EASY
0.501 0.513 0.541 0.557
COPA
0.650 0.660
0.660 0.680
FINAL C4 LOSS
3.477
3.379
3.302 3.246
Table 23: Evaluation Performance for Scion, Model Size
300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.445
0.489
0.511 0.540
OPENBOOKQA
0.306
0.308 0.326 0.358
WINOGRANDE
0.531
0.550 0.555 0.566
PIQA
0.684 0.689 0.701 0.712
BOOLQ
0.576 0.610 0.574 0.585
WSC273
0.597 0.612 0.685 0.696
HELLASWAG OSHOT
0.406 0.442 0.478 0.506
ARC CHALLENGE
0.272 0.285 0.298 0.316
ARC EASY
0.564 0.592 0.625 0.633
COPA
0,680 0.730 0,700 0.740
FINAL C4 LOSS
3.232 3.152 3.086 3.039
Table Evaluation Performance for Scion, Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.521
0.552
0.575
0.601
OPENBOOKQA
0.338
0.362
0.366 0.374
WINOGRANDE
0.569
0.551 0.605 0.586
PIQA
0.704
0.717 0.736 0.743
BOOLQ
0.575
0.615 0.620 0.641
WSC273
0.674
0.692 0.736 0.736
HELLASWAG OSHOT
0.481
0.520 0.549 0.581
ARC CHALLENGE
0.289
0.314 0.346 0.354
ARC EASY
0.625
0.647 0.667 0.686
COPA
0.730
0.730 0.750 0.770
FINAL C4 LOSS
3.080
3.007 2.952 2.904
35

---

Published as a conference paper at ICLR 2026
Table 25: Evaluation Performance for Cautious, Model Size 130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.314 0.385
0.420 0.458
OPENBOOKQA
0.298 0.280
0.308 0.318
WINOGRANDE
0.515 0.520 0.519 0.508
PIQA
0.638 0.652 0.664 0.669
BOOLQ
0.580 0.512 0.512 0.550
WSC273
0,560 0.527 0.593 0.604
HELLASWAG OSHOT
0.314 0.347 0.371 0.401
ARC CHALLENGE
0.239 0.244 0.261 0.270
ARC EASY
0.476 0.508 0.532 0.544
COPA
0.690 0.720 0.710 0.690
FINAL C4 LOSS
3.535 3.403 3.334 3.253
Table 26: Evaluation Performance for Cautious, Model Size 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.437
0.479
0.510 0.544
OPENBOOKQA
0.316
0.314 0.326 0.328
WINOGRANDE
0.515
0.533 0.530 0.576
PIQA
0.677 0.699 0.707 0.711
BOOLQ
0.547 0.492 0.584 0.574
WSC273
0.568 0.608 0.634 0.667
HELLASWAG OSHOT
0.397 0.432 0.477 0.505
ARC CHALLENGE
0.267 0.280 0.292 0.308
ARC EASY
0.541 0.569 0.606 0.618
COPA
0,670 0.730 0,760 0.720
FINAL C4 LOSS
3.260 3.165 3.094 3.043
Table 27: Evaluation Performance for Cautious, Model Size 520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.509
0.549
0.578
0.604
OPENBOOKQA
0.318 0.338 0.342 0.364
WINOGRANDE
0.548 0.575 0.571 0.605
PIQA
0.705 0.715 0.733 0.739
BOOLQ
0.601 0.611 0.582 0.606
WSC273
0.619 0.718 0.659 0.736
HELLASWAG OSHOT
0.467 0.514 0.550 0.583
ARC CHALLENGE
0.305 0.315 0.344 0.358
ARC EASY
0.609 0.649 0.676 0.689
COPA
0.730 0.760 0.770 0.770
FINAL C4 LOSS
3.100 3.017 2.956 2.910
36

---

Published as a conference paper at ICLR 2026
Table 28: Evaluation Performance for SOAP. Model Size
130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.344 0.404 0.434 0.458
OPENBOOKQA
0,280 0.270 0.300 0.318
WINOGRANDE
0.511 0.510 0.523 0.517
PIQA
0.643
0.652 0.666 0.676
BOOLQ
0.569 0.491 0.519 0.559
WSC273
0.513 0.568 0.597 0.612
HELLASWAG OSHOT
0.321 0.351 0.376 0.401
ARC CHALLENGE
0.247 0.255 0.270 0.270
ARC EASY
0.475 0.519 0.537 0.571
COPA
0.690 0.710 0.720 0.710
FINAL C4 LOSS
3.487
3.376
3.295 3,240
Table 29: Evaluation Performance for SOAP, Model Size 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.001 0.492
0.518 0.541
OPENBOOKQA
0.256 0.324 0.324 0.334
WINOGRANDE
0.506 0.534 0.546 0.555
PIQA
0.546 0.689 0.706 0.711
BOOLQ
0.402 0.593 0.565 0.593
WSC273
0.498 0.619 0.703 0.670
HELLASWAG OSHOT
0.257 0.443 0.478 0.506
ARC CHALLENGE
0,219 0.281 0.305 0.324
ARC EASY
0.309 0.590 0.612 0.632
COPA
0.590 0.740 0.720 0.700
FINAL C4 LOSS
5.437 3.147 3.082 3.030
Table 30: Evaluation Performance for SOAP, Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.513
0.555
0.582
0.606
OPENBOOKQA
0.308 0.312 0.368 0.350
WINOGRANDE
0.548 0.562 0.594 0.583
PIQA
0.695 0.714 0.729 0.739
BOOLQ
0.563 0.623 0.596 0.643
WSC273
0.648 0.703 0.722 0.755
HELLASWAG OSHOT
0.478 0.523 0.553 0.586
ARC CHALLENGE
0.292 0.345 0.334 0.356
ARC EASY
0.610 0.656 0.655 0.678
COPA
0.740 0.760 0.740 0.770
FINAL C4 LOSS
3.079 3.004 2.957 2.899
37

---

Published as a conference paper at ICLR 2026
Table 31: Evaluation Performance for AdamW, Model Size
130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.322 0.380
0.410 0.442
OPENBOOKQA
0.306 0.290 0.298 0.292
WINOGRANDE
0.506 0.519 0.515 0.530
PIQA
0.632 0.663 0.665 0.674
BOOLQ
0.568 0.415 0.553 0.576
WSC273
0.582 0.553 0.564 0.623
HELLASWAG OSHOT
0.312 0.340 0.365 0.391
ARC CHALLENGE
0.229 0.255 0.262 0.270
ARC EASY
0,463 0.497 0.523 0.544
COPA
0.660 0.650 0.720 0.730
FINAL C4 LOSS
3.529 3.409 3.322 3.262
Table 32: Evaluation Performance for Adam W. Model Size = 300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.435
0.484 0.500 0.537
OPENBOOKQA
0.300 0.312 0.328 0.318
WINOGRANDE
0.519 0.530 0.542 0.552
PIQA
0.673 0.691 0.697 0.713
BOOLQ
0.454 0.496 0.508 0.602
WSC273
0.586 0.626 0.667 0.692
HELLASWAG OSHOT
0.386 0.432 0.471 0.501
ARC CHALLENGE
0.269 0.295 0,304 0.310
ARC EASY
0.535 0.581 0.612 0.616
COPA
0.700 0.700 0.690 0.710
FINAL C4 LOSS
3.264 3.166 3.094 3.043
Table 33: Evaluation Performance for Adam W. Model Size
520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.500
0.540
0.570
0.591
OPENBOOKQA
0.320 0.336 0.342 0.356
WINOGRANDE
0.539 0.586 0.583 0.601
PIQA
0.705 0.719 0.732 0.739
BOOLQ
0.555 0.615 0.596 0.609
WSC273
0.604 0.656 0.703 0.729
HELLASWAG OSHOT
0.456 0.507 0.543 0.578
ARC CHALLENGE
0.299 0.311 0.323 0.349
ARC EASY
0.613 0.639 0.669 0.688
COPA
0.740 0.680 0.710 0.780
FINAL C4 LOSS
3.110 3.023 2.958 2.913
38

---

Published as a conference paper at ICLR 2026
Table 34: Evaluation Performance for Adam Adam-Mini, Model Size 130m
DATA SIZE
5B
10B
21B
PERFORMANCE METRIC
LAMBADA OPENAI
0.317 0.369
0.412 0.444
OPENBOOKQA
0,286 0.282 0.282 0.302
WINOGRANDE
0.525 0.500 0.515 0.528
PIQA
0.640 0.655 0.662 0.670
BOOLQ
0.506 0.496 0.445 0.583
WSC273
0.571 0.557 0.546 0.634
HELLASWAG OSHOT
0.310 0.337 0.364 0.390
ARC CHALLENGE
0.241 0.247 0.255 0.264
ARC EASY
0,465 0.495 0.525 0.555
COPA
0.690 0.710 0.730 0.700
FINAL C4 LOSS
3.542 3.416 3.328 3.266
Table 35: Evaluation Performance for Adam-Mini, Model Size
300m
DATA SIZE
6B
12B
24B
48B
PERFORMANCE METRIC
LAMBADA OPENAI
0.429 0.468
0.509 0.535
OPENBOOKQA
0.300 0.318 0.318 0.350
WINOGRANDE
0.502 0.528 0.545 0.586
PIQA
0.688 0.693 0.683 0.701
BOOLQ
0.480 0.546 0.609 0.573
WSC273
0.586 0.546 0.593 0.689
HELLASWAG OSHOT
0.381 0.424 0.464 0.492
ARC CHALLENGE
0,270 0.292 0.294 0.300
ARC EASY
0.545 0.579 0.607 0.625
COPA
0.700 0.720 0.740 0.730
FINAL C4 LOSS
3.272 3.178 3.103 3.049
Table 36: Evaluation Performance for Adam-Ming Model Size = 520m
DATA SIZE
10B
21B
42B
85B
PERFORMANCE METRIC
LAMBADA OPENAI
0.500
0.538
0.576
0.604
OPENBOOKQA
0.306 0.326 0.330 0.368
WINOGRANDE
0.534 0.561 0.571 0.594
PIQA
0.707 0.721 0.723 0.733
BOOLQ
0.543 0.626 0.619 0.536
WSC273
0.619 0.700 0.733 0.744
HELLASWAG OSHOT
0.459 0.505 0.541 0.576
ARC CHALLENGE
0,288 0.327 0.332 0.352
ARC EASY
0.613 0.638 0.654 0.671
COPA
0,710 0.720 0,760 0.760
FINAL C4 LOSS
3.112 3.027 2.966 2.912
D HYPERPARAMETER ABLATION IN PHASE I
We reported the results for the optimizers we swept in Phase I. The result is formulated as follows: the first row shows the approximately best configuration found and the following rows show the results for the 1-dimensional ablations centered around the found configuration. The loss presented here is the final loss on the C4/EN validation set.
39

---

Published as a conference paper at ICLR 2026
D.1 SWEEPING RESULTS FOR A
Table 37: Hyperparameter ablation for AdamW on 130m on 1x Chinchilla Data
81
Inorm Imnin
BSZ
warmup
X
Loss
Link
0.9
0.98
le-20 0.008
128
2000
0.1
3.529
0.95
3,539
0.98
3.882
2
0.9
3.545
3
0.95
3,535
le-25
3.529
5
le-15
3,531
lc-10
3,531
7
0.004
3,550
0.016
3.538
9
0.032
7.781
3,534
11
2.0
3,534
12
256
3.611
13
500
7.452
14
1000
3.532
4000
3.575
16
O
3,545
0.2
3.536
18
Table 38: Hyperparameter ablation for AdamW 130m on 2x Chinchilla Data
B1
82
Inorm Mmin
BSZ
warmup
A
Loss
Link
0.9
0.98
le-20
0.008
0 128
2000
0.1
3.409
0
0.95
3.417
0.98
7.557
2
0.9
3.423
0.95
3,413
4
le-25
3,409
le-15
3,409
6
le-10
3,410
0.004
3,420
0.016
3.419
9
0.032
7.840
10
O
3,410
11
2.0
3,408
12
256
3.437
13
512
3.527
500
7.277
15
1000
3,413
4000
3.415
17
0
3,436
18
0.2
3.415
19
40

---

Published as a conference paper at ICLR 2026
Table 39: Hyperparameter ablation for AdamW on 130m on 4x Chinchilla Data
81
B2
e
Inorm Mmmi
BSZ
warmup
Loss
Link
0.9
0.98
le-20 0.008
128
2000
0.1
3.322
0.95
3.330
0.98
3,416
0.9
3.338
0.95
3.329
le-25
3.322
le-15
3.323
le-10
3.324
0.004
3.329
0.016
3.337
0.032
7.562
10
3.327
2.0
3.324
12
256
3.331
13
512
3.373
14
1024
3.480
15
500
7.262
16
1000
3.327
17
4000
3.325
0.
3.359
19
0.2 3.335
Table 40: Hyperparameter ablation for AdamW on 130m on 8x Chinchilla Data
31
B2
7
Jnorm Mmin
BSZ
warmup
Loss
Link
0.9
0.98
le-20
0.008
256
1000
0.1
3.262
0.95
3.273
0.98
3.430
0.9
3.272
0.95
3.266
le-25
3.262
le-15
3.263
le-10
3.261
0.004
3,270
0.016
7.435
0.032
7.658
10
3.263
11
2.0
3,264
12
128
3,264
13
512
3.286
14
1024
3.328
500
3.278
16
2000
3.263
4000
3.262
18
3.310
19
0.2
3.269
20
41

---

Published as a conference paper at ICLR 2026
Table 41: Hyperparameter ablation for AdamW on 300m on 1x Chinchilla Data
81
B2
9horm Mmin
BSZ
warmup
X
Loss
Link
0.9
0.98
le-10 0.008
128
2000
0.1
3.264
0
0.95
3.271
0.98
7.351
2
0.9
3,280
0.95
3.269
4
le-25
3,265
le-20
3,265
le-15
3,263
0.004
3,272
8
0.016
7.760
0.032
7.784
10
3,263
2.0
3.263
12
256
3.282
13
512
3.367
14
500
7.704
15
1000
7.759
4000
3,270
17
0
3.303
0.2 3,275
19
Table 42: Hyperparameter ablation for AdamW on 520m on 1x Chinchilla Data
81
B2
7
Jaorm Mmin
BSZ
warmup
Loss
Link
0.9
0.98
le-10
0.004
256
1000
0.2
3.110
0.95
3.112
0.98
3.229
0.9
3.116
0.95
3.111
le-25
3,116
le-20
3,116
le-15
3.115
0.008
7.837
0.016
7.756
0.032
7.680
10
3.114
11
2.0
3.118
12
128
7,630
13
512
3.169
14
1024
3.302
500
3.165
16
2000
3.113
4000
3.126
18
7.270
19
0.1
3.135
20
42

---

Published as a conference paper at ICLR 2026
D.2
SWEEPING RESULTS FOR CAUTIOUS
Table 43: Hyperparameter ablation for Cautious on 130m on 1x Chinchilla Data
81
Inorm Imnn
BSZ
warmup
X
Loss
Link
0.95
0.98
le-15 0.008
128
2000
0.1
3.535
0.8
6.698
0.9
3.549
2
0.98
3,534
3
0.9
3,551
0.95
3.543
5
le-25
3.537
lc-20
3,537
7
1e-10
3.536
0.016
3.539
9
0.032
7.802
3,534
11
2.0
3,532
12
256
3.610
13
500
3,572
14
1000
3,535
4000
3.582
16
0
3,552
0.2
3,537
18
Table 44: Hyperparameter ablation for Cautious 130m 2x Chinchilla Data
81
82
Inorm
Mmin
BSZ
warmup
A
Loss
Link
0.98
0.98
le-15
0.008
2
0
128
2000
0.1
3.403
0
0.8
7.417
0.9
3.427
2
0.95
3.413
0.9
>10
4
0.95
3,408
le-25
3,404
6
le-20
3,404
le-10
3,404
0.016
3.416
9
0.032
3.513
10
3,415
11
1.0
3,401
12
256
3.422
13
512
3.499
500
3.453
15
1000
3,412
4000
3.410
17
0
3,436
18
0.2
3.405
19
43

---

Published as a conference paper at ICLR 2026
Table 45: Hyperparameter ablation for Cautious on 130m 8x Chinchilla Data
31
B2
e
Inorm
Immn
BSZ
warmup
Loss
Link
0.98
0.98
le-15
0.008
2
256
2000
0.1
3.253
0.8
7.395
0.9
3.271
0.95
3.259
0.9
>10
0.95
3.253
le-25
3.251
le-20
3.251
le-10
3.250
0.016
3,257
0.032
3,383
10
3.254
1.0
3.251
12
128
3.265
13
512
3.261
14
1024
3.294
15
500
3.276
16
1000
3,253
17
4000
3.255
0
3.291
19
0.2 3.253
Table 46: Hyperparameter ablation for Cautious on 300m on Chinchilla Data
31
B2
Juorm
Mmin
BSZ
warmup
Loss
Link
0.98
0.98
le-25
0.008
128
2000
0.1
3.260
0.9
3.286
0.95
3.271
2
0.9
>10
3
0.95
>10
4
le-25
3.260
5
le-15
3,260
3.274
7
3,259
256
3.270
9
1000
7.352
4000
3.264
11
0.0
3.310
12
0.2 3.266
13
44

---

Published as a conference paper at ICLR 2026
Table 47: Hyperparameter ablation for Cautious on 520m 1x Chinchilla Data
B1
B2
9horm Mmin
BSZ
warmup
X
Loss
Link
0.98
0.98
1e-25 0.008
256
2000
0.1
3.100
0
0.8
>10
0.9
7.264
2
0.95
3.108
0.9
>10
4
0.95
3.105
le-25
3,100
le-20
3,100
le-15
3.101
8
10-10
3.101
0.016
3,125
10
0.032
7.662
0
3.123
12
2.0
3.102
13
128
>10
14
512
3.123
15
500
>10
1000
3.119
17
4000
3.107
O
3.131
19
0.2 3.105
20
45

---

Published as a conference paper at ICLR 2026
D.3 SWEEPING RESULTS FOR LION
Table 48: Hyperparameter ablation for Lion on 133mm on Ix Chinchilla Data
B1
32
"
Inorm Mmni
BSZ
warmup
Loss
Link
0.9
0.95
0.002
0 128
2000
0.7
3.552
0.8
3.575
0.95
3.557
0.98
7.644
0.9
3,550
0.98
3.630
0.0005
3.579
0.001
3.549
0.004
7.806
0.008
7.828
0
3,559
10
2.0
3.557
256
3.643
12
500
7.840
1000
7.739
14
4000
3.601
15
3.570
16
0.1 3.562
17
0.2 3.557
18
0.3 3.555
19
0,4 3.553
0.5 3.551
21
0.6 3.549
0.8 3.554
23
0.9 3.556
3.557
25
46

---

Published as a conference paper at ICLR 2026
Table 49: Hyperparameter ablation for Lion 130m 2x
Chinchilla Data
81
B2
"
Inorm
Vmmin.
BSZ
warmuu
Loss
Link
0.9
0.95
0.001
0
128
2000
0.7
3.414
0.8
3.436
0,95
3,418
0.98
7.313
0.9
3,433
0.98
3.409
0,0005
3.435
0.002
3.414
0.004
3.489
0.008
7.826
0
3,414
10
2.0
3.413
256
3.447
12
512
3.540
500
3.435
14
1000
3.415
15
4000
3.423
16
0.4
3.419
17
0.5
3.418
0.6 3.414
19
0.8 3.414
0.9 3.413
21
3.413
Table 50: Hyperparameter ablation for Lion 130m 4x Chinchilla Data
81
82
9uorm
Tman
BSZ
warmup
Loss
Link
0.9
0.95
0.001
0
128
2000
0.7
3.331
0
0.8
3.345
0.95
3.335
2
0.98
7.330
0.9
3.345
4
0.98
3.340
0.0005
3.338
6
0.002
3.342
0.004
7.406
8
0.008
>0
0
3.333
10
2.0
3.334
11
256
3.346
12
512
3.386
13
1024
3.492
14
500
3.364
15
1000
3.336
16
4000
3.333
17
0.4 3.335
18
0.5 3.333
19
3.329
0.8 3.332
21
0.9 3.332
22
3.335
23
47

---

Published as a conference paper at ICLR 2026
Table 51: Hyperparameter ablation for Lion 130m 8x Chinchilla Data
31
B2
fhorm
Mmmin
BSZ
warmup
Loss
Link
0.9
0.98
0.001
128
2000
0.7
3.252
0
0.8
3.287
0,95
3.264
2
0.98
3.286
0.9
3.277
4
0.95
3.263
0.0005
3.254
6
0.002
3.310
0.004
7.829
8
0.008
NaN
0
3.329
10
2.0
3.265
11
256
3.260
12
512
3.287
13
1024
3.342
14
500
3.336
15
1000
3.273
16
4000
3.258
17
0.4 3.256
18
0.5 3.251
19
0.6 3.252
20
0.8 3.258
21
0.9 3.259
3.261
23
Table 52: Hyperparameter ablation for Lion on 300m on 1x Chinchilla Data
B1
32
"
Juorm
Thmin
BSZ
warmup
Loss
Link
0.9
0.95
0.001
1
0
128
2000
0.6
3.268
0.8
3.283
0.95
3.271
0.98
7.690
0.9
3.283
0.98
3.271
0.0005
3.286
0,002
3.283
0.004
7,817
0.008
7,863
0
3.274
10
2.0
3.269
11
256
3.295
12
512
3.367
13
500
7.607
14
1000
3,278
15
4000
3.277
16
0.4
3.272
0.5 3.271
18
0.7 3.268
0.8 3.268
20
0.9 3.269
21
3.269
22
48

---

Published as a conference paper at ICLR 2026
Table 53: Hyperparameter ablation for Lion 520m 1x Chinchilla Data
31
B2
fhorm
Mmmin
BSZ
warmup
Loss
Link
0.9
0.95
0.001
128
2000
0.7
3.108
0
0.8
3.117
0,95
3.121
2
0.98
7.577
0.9
3.119
4
0.98
3.208
0.0005
3.113
6
0.002
7.734
0.004
8.046
8
0.008
NaN
0
3.128
10
2.0
3.109
11
256
3.117
12
512
3.151
13
1024
3.252
14
500
7.403
15
1000
7.119
16
4000
3.115
17
0.4 3.109
18
0.5 3.108
19
0.6 3.108
20
0.8 3.109
21
0.9 3.112
22
3.115
23
49

---

Published as a conference paper at ICLR 2026
D.4 SWEEPING RESULTS FOR MARS
Table 54: Hyperparameter ablation for Mars on 130m on 1x Chinchilla Data
31
B2
Y
!
Inorm
Tmmi
BSZ
warmup
Loss
Link
0.9
0.95
le-25
0.025
0.016
0 128
2000
0.1 3.537
0.8
3.568
0,95
3.548
2
0.98
3.586
0.9
3.548
4
0.98
3.536
0.99
3.562
le-30
3.537
le-25
3.537
1c-20
3.537
9
le-15
3.537
10
1e-10
3.537
11
0.0125
3.540
12
0.05
3.542
13
0.1
3.553
14
0.008
3.538
15
0.032
7.574
16
256
3.616
17
500
>10
18
1000
>10
19
4000
3.596
20
3.552
0.2 3.561
22
Table 55: Hyperparameter ablation for Mars on 130m on 2x Chinchilla Data
B1
82
7
Ioorm
Tmmn
BSZ
warmup
Loss
Link
0.95
0.98
le-25
0.025
0.008
0
128
2000
0.1 3.396
0.8
3.421
0.9
3.402
0.98
3.401
0.9
3.409
0.95
3.400
0.99
3.398
le-30
3.396
le-20
3.396
le-15
3.398
1e-10
3.397
10
0.0125
3.398
11
0.05
3.404
12
0.1
3.415
13
0.016
3,402
14
0.032
3.441
15
256
3.427
16
512
3.524
17
500
3.400
18
1000
3.395
19
4000
3.409
20
3.430
0.2
3.404
22
50

---

Published as a conference paper at ICLR 2026
Table 56: Hyperparameter ablation for Mars 130m on 4x Chinchilla Data
B1
82
Inorm
Mmmin
BSZ
warmup
Loss
Link
0.9
0.98
le-25
0.025
0.008
0 128
2000
0.1 3.323
0
0.8
3.336
1
0.95
3.326
2
0.98
3.350
3
0.9
3.337
4
0.95
3.330
5
0.99
3.324
6
le-30
3.323
le-25
3.323
8
lc-20
3.323
9
le-15
3.321
10
le-10
3.322
11
0.0125
3.323
12
0.05
3.324
13
0.1
3.330
14
0.016
3.337
15
0.032
8.642
16
256
3.336
17
512
3.384
18
1024
3.525
19
500
3.328
20
1000
3.321
21
4000
3.327
22
0
3.359
23
0.2 3.333
24
51

---

Published as a conference paper at ICLR 2026
Table 57: Hyperparameter ablation for Mars 130m on 8x Chinchilla Data
B1
82
Inorm
Mmmin
BSZ
warmup
Loss
Link
0.98
0.99
le-25
0.025
0.008
0 128
2000
0.1
3.247
0
0.8
3.272
1
0,9
3,252
2
0.95
3.256
3
0.9
NaN
4
0.95
3.257
5
0.98
3.249
6
le-30
3.247
le-25
3.247
8
lc-20
3.247
9
le-15
3,247
10
le-10
3.248
11
0.0125
3.255
12
0.05
3.252
13
0.1
3,410
14
0.016
3.262
15
0.032
3.322
16
256
3,250
17
512
3.277
18
1024
3.339
19
500
3.250
20
1000
3.248
21
4000
3.251
22
O
3.292
23
0.2 3.265
24
52

---

Published as a conference paper at ICLR 2026
Table 58: Hyperparameter ablation for Mars 300m on 1x Chinchilla Data
B1
82
Inorm
Mmmin
BSZ
warmup
Loss
Link
0.98
0.98
le-25
0.05
0.008
0 128
1000
0.1
3.249
0
0.8
7.899
1
0,9
3,259
2
0.95
3.247
3
0.9
NaN
4
0.95
3.256
5
0.99
3.247
6
le-30
3.249
le-20
3.250
8
le-15
3.250
9
lc-10
3,252
10
0.0125
3,259
11
0.025
3.249
12
0.1
3.280
13
0.016
3.265
14
0.032
3.410
15
256
3.288
16
512
3.389
17
1024
3.624
18
500
3.254
19
2000
3.254
20
4000
3.269
21
3.312
22
0.2
3.262
23
Table 59: Hyperparameter ablation for Mars 520m on Ix Chinchilla Data
B1
82
Inorm
Imin
BSZ
warmup
À
Loss
Link
0.95
0.95
le-25
0.025
0.008
1
0 256
2000
0.1
3.101
0
0.8
>10
1
0.9
3.108
2
0.98
3.107
3
0.9
3.108
4
0.98
3.099
5
0.99
3.102
6
le-30
3.101
le-20
3.101
8
le-15
3.101
9
le-10
3.101
10
0.0125
3.100
11
0.05
3.103
12
0.1
3.113
13
0.016
3.111
14
0.032
NaN
15
128
3.102
16
512
3.130
17
1024
3.224
18
500
3.128
19
1000
3.105
20
4000
3.112
21
0
3.131
22
0.2
3.103
23
53

---

Published as a conference paper at ICLR 2026
D.5 SWEEPING RESULTS FOR NADAMW
Table 60: Hyperparameter ablation for NAdam W on 130m on 1x Chinchilla Data
31
Iuorm Imnin
BSZ
warmup
X
Loss
Link
0.95
0.98
le-25 0.008
128
2000
0.1
3.531
0.8
4.764
0.9
3.552
2
0.98
3.585
3
0.9
3.552
0.95
3.535
5
le-20
3,531
lc-15
3,533
7
1e-10
3.531
8
0.016
3.545
9
0.032
>10
10
3,537
11
2.0
3.539
12
256
3.624
13
500
3.646
14
1000
3.545
4000
3.577
16
O
3,547
0.2
3,535
18
Table 61: Hyperparameter ablation for NAdam on 130m 2x Chinchilla Data
81
82
Inorm Imin
BSZ
warmup
A
Loss
Link
0.95
0.98
le-10
0.008
1
0
128
2000
0.1
3.394
0
0.8
3,452
0.9
3.408
2
0.98
3.402
0.9
3,403
4
0.95
3.399
le-25
3,394
6
le-20
3.394
le-15
3.393
0.016
3.406
9
0.032
7.675
10
0
3,400
11
2.0
3.396
12
256
3.423
13
512
3.520
500
3.424
15
1000
3,400
4000
3.405
17
0
3,421
18
0.2
3.398
19
54

---

Published as a conference paper at ICLR 2026
Table 62: Hyperparameter ablation for NAdam on 130m on 4x Chinchilla Data
31
B2
e
Inorm Immin
BSZ
warmuu
Loss
Link
0.95
0.98
le-10
0.008
128
2000
0.1
3.319
0.8
7.085
0.9
3.331
0.98
3.349
0.9
3,332
0.95
3.327
le-25
3.321
le-20
3.321
le-15
3.323
0.016
3.343
0.032
7.733
10
3.324
2.0
3.323
12
256
3.332
13
512
3.372
14
1024
3.496
15
500
6.917
16
1000
3.328
17
4000
3.321
0
3.359
19
0.2 3.330
Table 63: Hyperparameter ablation for NAdamW on 130m on 8x Chinchilla Data
31
B2
7)
Jaorm Mmin
BSZ
warmup
Loss
Link
0.95
0.98
le-10
0.008
128
2000
0.1
3.251
0.8
3.282
0.9
3.257
0.98
3.253
0.9
3.260
0.95
3.255
le-25
3.251
le-20
3.251
1e-15
3,250
0.016
3.274
0.032
7.670
10
3.253
11
2.0
3,250
12
256
3,249
13
512
3.270
14
1024
3.321
500
3.274
16
1000
3.255
4000
3.252
18
3,286
19
0.2
3.265
20
55

---

Published as a conference paper at ICLR 2026
Table 64: Hyperparameter ablation for NAdam on 300m 1x Chinchilla Data
B1
$2
9horm Mmin
BSZ
warmup
X Loss
Link
0.95
0.98
le-10 0.008
128
2000
0.1
3.248
0
0.8
7.542
0.9
7.054
2
0.98
3,254
0.9
3,263
4
0.95
3,256
le-25
3,250
le-20
3,250
le-15
3,250
8
0.016
7.538
0.032
7.695
10
3,260
2.0
3,250
12
256
3,272
13
512
3.344
14
500
7.744
15
1000
7.421
4000
3,256
17
0
3.283
0.2 3,257
19
D.6 SWEEPING RESULTS FOR ADAM-MINI
Table 65: Hyperparameter ablation for Adam-Mini on 130m on Ix Chinchilla Data
81
82
1)
Inorm Mmin
BSZ
warmup
X
Loss
Link
0.9
0.98
le-15
0.008
128
2000
0.1
3.542
0
0.95
3.560
0.98
7.733
0.9
3.554
3
0.95
3.545
le-25
3.546
5
le-20
3.546
le-10
3.548
7
0.004
3.558
0.016
7.800
9
0.032
7.825
10
3.542
11
2.0
3.542
12
256
3.785
13
500
7.725
14
1000
7.729
4000
3.587
16
0
3.566
17
0.2
3.589
18
56

---

Published as a conference paper at ICLR 2026
Table 66: Hyperparameterr ablation for Adam-Mini on 130m 2x Chinchilla Data
81
B2
9horm
Mmin
BSZ
warmup
X Loss
Link
0.9
0.98
1e-20 0.008
2
128
2000
0.1
3.416
0
0.95
3.429
0.98
7.520
2
0.9
3,422
0.95
3,419
4
le-25
3,416
le-15
3,416
le-10
3,415
0.004
3,425
8
0.016
7.796
0.032
7.721
10
3,416
11
1.0
3,416
12
256
3.487
13
512
3.758
14
500
7.617
15
1000
7.445
16
4000
3,424
17
0
3,447
0.2 3.426
19
Table 67: Hyperparameter ablation for Adam-Mini on 130m on 4x Chinchilla Data
81
B2
7)
Jnorm
Mmin
BSZ
warmup
Loss
Link
0.9
0.98
le-10
0.008
128
2000
0.1
3.328
0.95
3.360
0.98
7.771
0.9
3.337
0.95
3.331
le-25
3.331
le-20
3,331
le-15
3.334
0.004
3.334
0.016
7.717
0.032
7.652
10
3.329
11
2.0
3,329
12
256
3,363
13
512
3.447
14
1024
3.784
500
7.855
16
1000
7.411
4000
3.331
18
3.364
19
0.2
3.365
20
57

---

Published as a conference paper at ICLR 2026
Table 68:
Hyperparameterr ablation for Adam-Mini on 130m 8x
Chinchilla Data
31
B2
e
Inorm Mmmi
BSZ
warmup
Loss
Link
0.9
0.98
le-10 0.008
128
2000
0.1
3.266
0.95
3.290
0.98
7.552
0.9
3.281
0.95
3.267
le-25
3.267
le-20
3.267
le-15
3.266
0.004
3.264
0.016
7.614
0.032
7.773
10
3.268
2.0
3,270
12
256
3,281
13
512
3.324
14
1024
3.426
15
500
7.868
16
1000
7.022
17
4000
3.266
0 3.304
19
0.2 3.291
Table 69: Hyperparameter ablation for Adam-Mini on 300m on Ix Chinchilla Data
81
Ba
Juorm
Mmin
BSZ
warmup
À
Loss
Link
0.9
0.98
le-25
0.004
2
128
2000
0.2
3.272
0.95
3.276
0.98
8.024
2
0.9
3.279
3
0.95
3.277
le-25
3,272
5
le-20
3.272
le-15
3,273
7
le-10
3.273
0.008
7.859
9
0.016
7.990
10
0.032
8.581
11
3.272
12
1.0
3.272
13
256
5.691
14
512
3.629
15
500
7.020
16
1000
7.042
4000
3.280
18
O
3.336
19
0.1
3.280
20
58

---

Published as a conference paper at ICLR 2026
Table 70: Hyperparameterr ablation for Adam-Mini on 520m Ix Chinchilla Data
81
B2
9horm
Mmin
BSZ
warmup
X Loss
Link
0.9
0.98
le-10 0.004
0
128
4000
0.1
3.112
0
0.95
3.113
0.98
7.459
2
0.9
3,120
0.95
3,115
4
le-25
3.115
le-20
3,115
6
le-15
3.112
0.008
7.771
8
0.016
7.746
0.032
7.778
10
1.0
3.112
11
2.0
3.112
12
256
3.133
13
512
3,200
14
500
7.457
15
1000
7.274
2000
7.289
17
0
7.792
0.2 3.115
19
D.7
SWEEPING RESULTS FOR KRON
Table 71: Hyperparameter ablation for Kron on 130m on Ix Chinchilla Data
B1
blocksize
77
Inorm
Tmin
NormGrad Blocking Initpe
Upe
Ppe
BSZ
Steppe
warmuu
À
Loss
Link
0.95
256
0.002
1
0
True
True
0.2
0.05
128
2000
1000
0.7
3,492
0
0.9
3.500
1
0.98
3.497
2
128
3.492
3
512
3.494
4
0.0005
3.528
5
0.001
3.501
6
0.004
3.514
7
0.008
7.838
8
0.0
3.492
9
2.0
3.492
10
False
3.493
11
0.1
3.498
12
0.1
3.493
13
256
3.525
14
512
3.632
15
500
3.503
16
1000
3.498
17
2000
3.507
18
4000
3.583
19
0.0 3.519
20
0.5 3.491
21
0.9 3.495
22
59

---

Published as a conference paper at ICLR 2026
Table 72: Hyperparameter ablation for Kron 130m 2x Chinchilla Data
81
blocksize
Inorm
Mmin
NormGrad Blocking Inilpe
Upe
Ppe
BSZ.
Sleppe
warmup
À
Loss
Link
0.95
256
0.002
1
0
True
True
0.2
0.05
128
2000
1000
0.5 3.389
0.9
3.393
0.98
3.391
512
3.390
0,0005
3.409
0.001
3.391
0.004
3.410
0.008
7.458
0.0
3.389
2.0
3.389
False
3,419
10
0.1
3.390
0.1
3.388
12
256
3,402
512
3.438
14
1024
3.532
15
500
3.393
1000
3.391
17
2000
3.392
4000
3.407
19
0.0
3.420
20
0.7 3.392
21
0.9 3.401
22
Table 73: Hyperparameter ablation for Kron 130m on 4x Chinchilla Data
81
blocksize
Inorm
Tmin
NormGrad Blocking Initpe
Tpe
Ppe
BSZ
Steppe
warmup
À
Loss
Link
0.95
256
0.002
1
0
Truc
True
0.2 0.05
128
2000
1000
0.5
3.307
0.9
3.316
0.98
3.310
128
3.308
512
3.313
0.0005
3.316
0.001
3.303
0.004
6.703
0.008
7.492
0.0
3.307
2.0
3.307
10
False
5.676
0.1
3.311
12
0.1
3.304
13
256
3.307
14
512
3.327
15
1024
3.370
16
500
3.311
17
1000
3.310
2000
3.310
19
4000
3.317
0.0
3.341
21
0.7 3.326
22
0.9
3.340
23
60

---

Published as a conference paper at ICLR 2026
Table 74: Hyperparameter ablation for Kron 130m 8x Chinchilla Data
81
blocksize
Jnorm
Mmin
NormGrad Blocking Inilpe
Ape
Ppe
BSZ
Steppe
warmup
À
Loss
Link
0.95
256
0.001
0
True
True
0.2
0.1
128
2000
1000
0.5 3.239
0.9
3.243
0.98
3.239
512
3.239
0,0005
3.245
0.002
3.252
0.004
6.279
0.008
7.504
0.0
3.239
8
2.0
3.239
False
3.247
10
False
3.240
0.1
3.242
12
0.05
3.240
256
3.247
14
512
3.264
15
1024
3.294
500
3.240
17
1000
3.239
2000
3.239
19
4000
3.240
20
0.0 3.282
21
0.7 3.241
0.9 3.246
23
Table 75: Hyperparameter ablation for Kron on 300m on Ix Chinchilla Data
81
blocksize
7
Inorm
Tmin
NormGrad Blocking
Initpe
Upe
Ppe
BSZ
Steppe
warmup
Loss
Link
0.95
256
0.001
0
True
True
0.2
0.1
128
2000
1000
0.5 3.244
0.9
3.248
0.98
3.244
512
3.243
0.0005
3.263
0.002
3.245
5
0.004
7.636
0.008
6.923
7
0.0
3.244
2.0
3.244
False
3.253
10
0.1
3.247
11
0.05
3.245
12
256
3.268
13
512
3.309
14
1024
3.393
15
500
3.248
16
1000
3.245
2000
3.249
18
4000
3.255
0.0
3.280
20
0.7 3.241
21
0.9 3.243
22
61

---

Published as a conference paper at ICLR 2026
Table 76: Hyperparameter ablation for Kron 520m on 1x Chinchilla Data
81
blocksize
7
Jnorm
Mmin
NormGrad Blocking Inilpe
Ape
Ppe
BS7.
Steppe
warmup
X
Loss
Link
0.95
256
0.001
1
0
True
True
0.2
0.1
128
2000
1000
0.5
3.084
0
0.9
3.088
1
0.98
6.663
2
512
3.087
3
0,0005
3.095
4
0.002
6.412
5
0.004
6.927
6
0.008
7.018
7
0.0
3.084
8
2.0
3.084
9
False
5.487
10
0.1
3.088
11
0.05
3.093
12
256
3.099
13
512
3.126
14
500
3.087
15
1000
3.085
16
2000
3.090
17
4000
3.094
18
0.0 3.127
19
0.7 3.088
20
0.9
3.089
21
D.8
SWEEPING RESULTS FOR SOAP
Table 77: Hyperparameter ablation for Soap 130m 1x Chinchilla Data
81
B2
blocksize
Juorm Mmin
Blocking
Spc Bshampoo
BSZ
warmup
X
Loss
Link
0.95
0.98
256
le-15
0.016
O
True
1
0.95
128
1000
0.1
3.483
0
0.8
4.868
0.9
4.547
2
0.9
3.496
0.95
3.491
4
0.99
3.482
128
3,489
6
512
3.487
le-20
3.513
8
le-10
3.483
0.004
3.509
10
0.008
3.491
11
3.488
12
0.9
3.488
13
0.98
3.489
14
0.99
3.491
15
256
3.523
512
3.611
17
3.483
2000
3.507
19
0
3.508
20
0.2 3.501
21
62

---

Published as a conference paper at ICLR 2026
Table 78: Hyperparameter ablation for Soap 130m 2x Chinchilla Data
81
82
blocksize
1
Inorm
Mmii
Blocking Syc Bshampoo
BSZ
warmup
Loss
Link
0.95
0.99
256
le-15
0.016
O False
0.98
128
500
0.1
3.376
0.8
5.163
0.9
3.397
2
0.9
3.393
0.95
3.384
4
0.98
3.380
128
3.376
6
512
3.376
le-20
3.384
8
1c-10
3.375
0.004
3.388
10
0.008
3.374
11
10
3.381
12
0.9
3.378
13
0.95
3.376
14
0.99
3.379
15
256
3.385
16
512
3.414
17
1024
3.479
18
1000
3.379
19
2000
3.385
20
0
3.437
21
0.2 3.395
22
Table 79: Hyperparameter ablation for Soap on 130m on 4x Chinchilla Data
81
82
blocksize
7
Jnorm
Mman Blocking
fpc Behempoo
BSZ
warmup
Loss
Link
0.95
0.99
256
le-15
0.008
False
0.98
128
500
0.1
3.295
0.8
3.328
0.9
3.302
0.9
3.312
0.95
3.303
0.98
3.298
5
128
3.295
512
3.295
1c-20
3.297
le-10
3.295
0.004
3.303
10
0.016
3.303
11
10
3.299
12
0.9
3.296
13
0.95
3.294
14
0.99
3.297
15
256
3.305
16
512
3.325
17
1024
3.358
18
1000
3.298
19
2000
3.300
20
3.370
0.2
3.304
22
63

---

Published as a conference paper at ICLR 2026
Table 80: Hyperparameter ablation for Soap 130m 8x Chinchilla Data
81
82
blocksize
"
Inorm
Mmiin
Blocking Syc Bshampoo
BSZ
warmup
Loss
Link
0.95
0.99
512
le-15
0.008
O True 10
0.98
256
1000
0.1
3.239
0.8
3.298
0.9
3.250
2
0.9
3.251
0.95
3.244
4
0.98
3.241
128
3.240
6
256
3.242
le-20
3.239
8
1c-10
3.238
0.004
3.248
10
0.016
3.249
0.9
3.239
12
0.95
3.239
13
0.99
3.241
14
128
3.242
15
512
3.250
16
1024
3.276
17
500
3.240
18
2000
3.240
19
3.308
20
0.2 3.243
21
Table 81: Hyperparameter ablation for Soap 300m 1x Chinchilla Data
81
B2
blocksize
Juorm
Mmin
Blocking
Spc Sshampoo
BSZ
warmup
X
Loss
Link
0.95
0.99
512
le-10
0.008
O
True 10
0.9
128
1000
0.1
3.231
0
0.8
4.544
0.9
3.251
2
0.9
3.247
0.95
3.240
4
0.98
3.234
128
3,239
6
256
3.235
le-20
3.233
le-15
3.233
0.004
3.239
10
0.016
5.559
11
0.95
3.233
12
0.98
3.235
13
0.99
3.237
14
256
3.248
15
512
3.283
500
3.231
17
2000
3.235
0
3.268
19
0.2
3.238
20
0.3 3.249
21
64

---

Published as a conference paper at ICLR 2026
Table 82: Hyperparameter ablation for Soap 520m on 1x Chinchilla Data
81
82
blocksize
"
Inorm
Mmiin
Blocking Syc Bshampoo
BSZ warmuu
X
Loss
Link
0.95
0.99
512
le-10
0.008
O
True 10
0.95
128
1000
0.1
3.079
0
0.8
4.630
0.9
4.316
2
0.9
3.097
0.95
3.090
4
0.98
3.085
128
5.395
256
4.392
le-20
3.082
8
1c-15
3.081
0.004
3.079
10
0.016
5.762
0.9
3.080
12
0.98
3.082
13
0.99
3.083
14
256
3.085
15
512
4.215
500
4.163
17
2000
3.081
0
6.106
19
0.2
3.527
20
0.3 3.118
21
65

---

Published as a conference paper at ICLR 2026
D.9 SWEEPING RESULTS FOR MUON
Table 83: Hyperparameter ablation for Muon 130m Ix Chinchilla Data
Qadam
B1
82
Decay(WSD)
"
Schedule Fnorm
Mmnin
Buuon
Emoon
BSZ
warmup
Loss
Link
0.0032
0.8
0.98
0.8
le-15 0.016
linear
0
0.95
1e-05
128
0
0.1
3.464
0.0016
3.474
0.0048
3.463
0.9
3.468
0.95
3.467
4
0.98
3.470
0.9
3.481
0.95
3.469
0.2
3.516
0.4
3.482
0.6
3.468
10
1.0
3.467
11
le-25
3.465
12
1e-20
3.465
13
le-10
3.465
14
0.008
3.491
15
0.032
3.484
0,064
3.535
17
0.128
3,688
3.464
19
2.0
3.464
20
0.8
3.503
21
0.9
3.473
22
0.98
3.474
23
le-25
3.465
24
le-20
3.465
25
le-15
3.465
26
le-10
3.465
27
256
3.504
28
512
3.601
1024
3.811
30
3.532
31
0.2
3.483
32
66

---

Published as a conference paper at ICLR 2026
Table 84: Hyperparameter ablation for Muon 130m 2x Chinchilla Data
Madam
B1
32
Decay(WSi)
"
Schedule Jnorm Mmmin Amnon
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8
0.98
0.8
le-15 0.008
linear
0.98
1e-05
128
0.1 3.369
0.0008
3.385
0.0016
3.373
0.9
3.372
0.95
3.373
0.98
3.375
0.9
3.390
0.95
3.377
0.2
3.404
0.4
3.380
0.6
3.371
10
1.0
3.374
le-25
3.370
12
1e-20
3.370
le-10
3.369
14
0.016
3.373
15
0.032
3.406
16
0.064
3.549
17
0.128
5.932
0
3.371
19
2.0
3.370
20
0.8
3.417
21
0.9
3.385
0.95
3.371
23
le-25
3.370
1e-20
3.370
25
le-15
3.370
26
lc-10
3.370
27
256
3.400
28
512
3.465
29
1024
3.608
30
3.431
31
0.2 3.378
32
67

---

Published as a conference paper at ICLR 2026
Table 85: Hyperparameter ablation for Muon 130m 4x Chinchilla Data
Madam
B1
32
Decay(WSD)
"
Schedule Jnorm Mmmin Amuon
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8
0.98
0.8
le-15 0.008
linear
0.95
1e-05
128
0.1 3.296
0.0008
3.309
0.0016
3.300
0.9
3.297
0.95
3.299
0.98
3.299
0.9
3.317
0.95
3.305
0.2
3.334
0.4
3.310
0.6
3.301
10
1.0
3.297
le-25
3.297
12
1e-20
3.296
13
le-10
3.297
14
0.016
3.306
15
0.032
3.347
16
0.064
3.431
17
0.128
4.807
0
3.296
19
2.0
3.296
20
0.8
3.327
21
0.9
3.307
0.98
3.295
23
le-25
3.296
1e-20
3.296
25
le-15
3.296
26
lc-10
3.297
27
256
3.308
28
512
3.346
29
1024
3.420
30
3.352
31
0.2 3.316
32
68

---

Published as a conference paper at ICLR 2026
Table 86: Hyperparameter ablation for Muon 130m 8x Chinchilla Data
Madam
B1
32
Decay(WSD) (WSD)
"
Schedule Jnorm Mmmin Amuon
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8
0.98
le-15 0.008
linear
0.98
1e-05
128
0.1 3.240
0.0008
3.249
0.0016
3.243
0.9
3.240
0.95
3.240
0.98
3.241
0.9
3.260
0.95
3.247
0.2
3.280
0.4
3,257
0.6
3.248
10
0.8
3.242
le-25
3.238
12
1e-20
3.238
le-10
3.241
14
0.016
3.253
15
0.032
3.298
16
0.064
3.407
17
0.128
5.568
0
3.240
19
2.0
3.239
20
0.8
3.265
21
0.9
3.250
0.95
3.243
23
le-25
3.239
1e-20
3.239
25
le-15
3.239
26
lc-10
3.239
27
256
3.240
28
512
3.257
29
1024
3.302
30
3.310
31
0.2 3.256
32
69

---

Published as a conference paper at ICLR 2026
Table 87: Hyperparameter ablation for Muon 300m Ix Chinchilla Data
Madam
B1
32
Decay(WSD)
"
Schedule Jnorm Mmmin Amuon
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8
0.98
0.8
le-15 0.008
linear
0.98
1e-05
128
0.1 3.224
0.0008
3.232
0.0016
3.225
0.9
3.226
0.95
3.226
0.98
3.227
0.9
3.241
0.95
3.231
0.2
3.271
0.4
3,240
0.6
3.229
10
1.0
3.225
le-25
3.224
12
1e-20
3.224
le-10
3.225
14
0.016
3.236
15
0.032
3.281
16
0.064
4.695
17
0.128
5.976
0
3.224
19
2.0
3.225
20
0.8
3.257
21
0.9
3.235
0.95
3.224
23
le-25
3,226
1e-20
3.226
25
le-15
3.226
26
lc-10
3.226
27
256
3.244
28
512
3.301
29
1024
3.419
30
3.285
31
0.2 3.234
32
70

---

Published as a conference paper at ICLR 2026
Table 88: Hyperparameter ablation for Muon 520m Ix Chinchilla Data
Madam
B1
32
Decay (WSD)
"
Schedule Jnorm Mmmin Amnon
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8
0.98
le-25 0.008
linear
0.98
1e-05
128
0.1
3.073
0.0008
3.076
0.0016
3,072
0.9
3,073
0.95
3,074
0.98
3.073
0.9
3.090
0.95
3.080
0.2
3.134
0.4
3.101
0.6
3,086
10
0.8
3,076
1e-20
3,074
12
le-15
3.073
13
le-10
3,073
14
0.016
3.095
15
0.032
3.147
16
0.064
7.886
17
0.128
7.900
0
3.072
19
2.0
3.071
20
0.8
3.098
21
0.9
3.080
0.95
3,073
23
le-25
3,072
1e-20
3.072
25
le-15
3.072
26
lc-10
3,072
27
256
3,080
28
512
3.115
29
1024
3.188
30
3.134
31
0.2 3.091
32
71

---

Published as a conference paper at ICLR 2026
D.10 SWEEPING RESULTS FOR SCION
Table 89: Hyperparameter ablation for Scion on 130m on Ix Chinchilla Data
Nadam
B1
Decay(WSD)
7
Schedule Inorm
Mmin
Bmunn
Eacion
BSZ
warmup
À
Loss
Link
0.0032
0.95
0.8
0.016
linear
0.95
1e-15
128
0.1
3.477
0
0.0016
3.477
1
0.0048
3,481
2
0.0064
3.485
3
0.008
3.488
4
0.0096
3.490
5
0.8
3.493
6
0.9
3.479
7
0.98
3.479
8
0.0
3.779
9
0.2
3.525
10
0.4
3.493
11
0.6
3.480
12
1.0
3.477
13
0.008
3.490
14
0.032
3.509
15
0.064
5.195
16
0.128
6.308
17
3.478
18
2.0
3.478
19
0.8
3.480
20
0.9
3.474
21
0.98
3.496
22
1c-20
3.477
23
1e-10
3.477
24
1e-05
3.477
25
256
3.505
26
512
3.583
27
1024
3.761
28
0
3.537
29
0.2
3.495
30
72

---

Published as a conference paper at ICLR 2026
Table 90: Hyperparameter ablation for Scion on 130m 2x Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm Immin
Amuoo
Eacion
BSZ
warmup
X
Loss
Link
0.0032
0.95
0.016
linear
0.9
1e-15
128
0
0.1
3.379
0
0.0016
3.378
1
0,0048
3.382
2
0.0064
3.384
3
0.008
3.386
4
0.0096
3.388
5
0.8
3.400
6
0.9
3.383
7
0.98
3.379
8
0.0
3.738
9
0.2
3.437
10
0.4
3.405
11
0.6
3.390
12
0.8
3.383
13
0.008
3.390
14
0.032
3.411
15
0.064
3.513
16
0.128
6.193
17
3.379
18
2.0
3.378
19
0.8
3.385
20
0.95
3.380
21
0.98
3.388
22
1e-20
3.379
23
lc-10
3,380
24
1e-05
3.380
25
256
3.391
26
512
3.436
27
1024
3.532
28
0
3.443
29
0.2
3.401
30
73

---

Published as a conference paper at ICLR 2026
Table 91: Hyperparameter ablation for Scion on 130m 4x Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Immin
Amuoo
Eacion
BSZ
warmup
X
Loss
Link
0.0016
0.98
0.008
linear
2
0.9
1e-15
128
0
0.1
3.302
0
0.0008
3.302
1
0,0024
3.306
2
0.0032
3.307
3
0.004
3.306
4
0.0048
3.307
5
0.8
3.331
6
0.9
3.314
7
0.95
3.306
8
0.0
3.584
9
0.2
3.340
10
0.4
3.317
11
0.6
3.307
12
0.8
3.303
13
0.016
3.310
14
0.032
3.361
15
0.064
5.152
16
0.128
5.894
17
3.303
18
1.0
3.303
19
0.8
3.307
20
0.95
3.302
21
0.98
3.303
22
1e-20
3.302
23
lc-10
3,303
24
1e-05
3.302
25
256
3.323
26
512
3.365
27
1024
3.447
28
0
3.375
29
0.2
3.309
30

---

Published as a conference paper at ICLR 2026
Table 92: Hyperparameter ablation for Scion on 130m 8x Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Immin
Amuoo
Eacion
BSZ
warmup
X
Loss
Link
0.0016
0.98
0.008
linear
2
0.9
1e-05
128
0
0.1
3.246
0
0.0008
3.245
1
0,0024
3.250
2
0.0032
3.249
3
0.004
3.251
4
0.0048
3.252
5
0.8
3.275
6
0.9
3.257
7
0.95
3.249
8
0.0
3.558
9
0.2
3,288
10
0.4
3,266
11
0.6
3.255
12
0.8
3.249
13
0.016
3.265
14
0.032
3.325
15
0.064
5.264
16
0.128
6.174
17
3.247
18
1.0
3.247
19
0.8
3.249
20
0.95
3.245
21
0.98
3.247
22
le-20
3.246
23
le-15
3,246
24
1e-10
3.247
25
256
3.250
26
512
3.272
27
1024
3.319
28
0
3.308
29
0.2
3.261
30
75

---

Published as a conference paper at ICLR 2026
Table 93: Hyperparameter ablation for Scion 300m Ix Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Immin
Omuon
Eacion
BSZ
warmup
X
Loss
Link
0.0008
0.98
0.8
0.008
linear
2
0.9
1e-05
128
0
0.1
3.232
0
0.0016
3.237
1
0,0024
3.242
2
0.0032
3.242
3
0.004
3.241
4
0.0048
3.243
5
0.8
3.256
6
0.9
3.240
7
0.95
3.236
8
0.0
3.497
9
0.2
3.271
10
0.4
3,245
11
0.6
3.234
12
1.0
3.237
13
0.016
3.242
14
0.032
3.301
15
0.064
3.400
16
0.128
5.627
17
3.233
18
1.0
3.233
19
0.8
3.240
20
0.95
3.231
21
0.98
3.236
22
le-20
3.232
23
le-15
3,232
24
1e-10
3.233
25
256
3.259
26
512
3.319
27
1024
3,432
28
0
3.303
29
0.2
3.241
30
76

---

Published as a conference paper at ICLR 2026
Table 94: Hyperparameter ablation for Scion on 520m Ix Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Immin
Eacion
BSZ
warmup
X
Loss
Link
0.0008
0.98
0.008
linear
2
0.9
1e-05
128
0.1
3.080
0
0.0016
3.089
1
0,0024
3.090
2
0.0032
3.090
3
0.004
3.090
4
0.0048
3.091
5
0.8
3.104
6
0.9
3.090
7
0.95
3.081
8
0.0
3.400
9
0.2
3.136
10
0.4
3.106
11
0.6
3.091
12
0.8
3.083
13
0.016
3.105
14
0.032
3.173
15
0.064
4.420
16
0.128
7.335
17
3.082
18
1.0
3.081
19
0.8
3.084
20
0.95
3.079
21
0.98
3.086
22
le-20
3.081
23
le-15
3.081
24
1e-10
3.080
25
256
3.093
26
512
3.134
27
1024
3.215
28
0
3.150
29
0.2
3.097
30
77

---

Published as a conference paper at ICLR 2026
D.11 SWEEPING RESULTS FOR SOPHIA
Table 95: Hyperparameter ablation for Sophia on 130mm on 1x Chinchilla Data
B
B2
17
BSZ
warmup
À
Loss
Link
0.95
0.9
le-07
0.0125
0.004
128
4000
0
3.544
0.8
3.581
0.9
3.546
0.98
3.624
0.95
3.546
0.98
3.553
2
0.99
3.563
0.995
3.573
4
le-17
3,559
le-12
3.559
2
0.00625
3.546
0.025
3.546
2
0.05
3.554
3
0.002
3,551
0.008
3,576
2
0.016
7.769
0.032
7.745
4
500
7.754
1000
7.824
2
2000
3.576
0.1 3.553
0.2 3.566
0.3 3.579
3
78

---

Published as a conference paper at ICLR 2026
Table 96:
Hyperparameter ablation for Sophia on 130m on 2x Chinchilla Data
B1
82
BSZ
warmup
À
Loss
Link
0.95
0.9
le-07
0.0125
0.004
128
4000
0.1
3.414
0
0.8
3,439
0.9
3,416
2
0.98
3,469
0.95
3,414
0.98
3,418
0.99
3,421
0.995
3,428
le-17
3.426
le-12
3,424
0.00625
3,416
0.025
3,415
0.05
3.423
3
0.002
3,417
0.008
3,438
0.016
7.282
0.032
6.938
256
3,446
500
7.811
1000
7.549
2
2000
3,436
0
3,431
0.2 3,415
0.3
3.419
3
Table 97: Hyperparameter ablation for Sophia on 130m on 4x Chinchilla Data
81
B2
BSZ
warmup
À Loss Link
0.95
0.99
le-07
0.0125
0.004
128
4000
0.2
3.330
0.8
3.351
0.9
3.332
0.98
3.371
0.9
3.328
0.95
3.328
0.98
3.328
lc-17
3.337
le-12
3.338
0.00625
3.330
0.025
3.330
0.002
3.329
0.016
7.059
0.032
6,664
256
3.340
512
3.390
500
7.345
1000
7.022
2000
3.349
0
3.367
0.1 3.330
0.3 3.332
79

---

Published as a conference paper at ICLR 2026
Table 98:
Hyperparameter ablation for Sophia on 130m on 8x Chinchilla Data
81
B2
BSZ
warmuu
Loss
Link
0.95
0.95
le-07 0.0125
0.002
128
4000
0.2
3.259
0
0.8
3.291
0.9
3.265
2
0.9
3.259
0.98
3.260
2
0.99
3.260
le-17
3.266
le-12
3.265
0.004
3.265
0.008
3.308
256
3.265
1000
3.277
2000
3.260
2
3.300
Table 99; Hyperparameter ablation for Sophia on 300m on 1x Chinchilla Data
81
B2
BSZ
warmuu
À Loss
Link
0.9
0.9
le-07
0.0125
0.004
128
4000
0.1
3.267
0
0.8
3.288
0.95
7.390
2
0.98
7.525
0.95
3,270
0.98
3,275
0.99
3,280
3
0.995
3.280
le-17
3,289
le-12
3,289
0.00625
3,273
0.025
3,271
0.05
3.274
3
0.002
3.276
0.008
6.966
0.016
7.142
0.032
6.811
256
3.298
500
7.149
1000
7.093
2
2000
7.382
0
3.288
0.2 3,274
0.3 3.281
3
80

---

Published as a conference paper at ICLR 2026
Table 100:
Hyperparameter ablation for Sophia 520m on 1x Chinchilla Data
B1
B2
BSZ
warmuu
Loss
Link
0.95
0.9
le-07 0.0125
0.002
128
4000
0.3
3.106
0.8
3.125
0.9
3.109
0.98
3.133
0.95
3.111
0.98
3.111
0.99
6.571
le-17
3.116
le-12
3.116
0.00625
3.107
0.025
3.107
0.004
6.940
1000
6.908
2000
6.823
3.148
0.1
3.113
0.2
3.105
E HYPERPARAMETER ABLATION IN PHASE
We reported the results for the optimizers we swept in Phase The result formulated in the same way as in Phase I.
E.1 SWEEPING RESULTS ADAMW
Table 101: Hyperparameter ablation for AdamW 300m 2x Chinchilla Data
81 82
Juorm
Tmin
BSZ
warmup
À Loss
Link
0.9
0.98
le-15
0.008
1
0
128
2000
0.1
3.166
0.004
3.167
256
3.170
0.2
3.183
Table 102: Hyperparameter ablation for Adam W on 300m on 4x Chinchilla Data
3
82
Juorm
Tmin
BSZ
warmup
Loss
Link
0.9
0.98
le-10
0.008
2
0
256
2000
0.1
3.094
0.004
3.101
128
3.103
0.2
3.103
81

---

Published as a conference paper at ICLR 2026
Table 103: Hyperparameter ablation for AdamW on 300m on 8x Chinchilla Data
81 88
Shorm
Mmmi
BSZ
warmuu
Loss
Link
0.9 0.98
le-10
0.008
2
0
256
2000
0.1 3.043
0.004
3.042
128
3,057
0.2 3.059
Table 104: Hyperparameter ablation for A on 520m on 2x Chinchilla Data
81 82
Shorm
Mmin
BSZ
warmup
Loss
Link
0.9 0.98 le-10
0.004
0 256
1000
0.2 3.023
128
6.654
0.1 3.025
Table 105: Hyperparameter ablation for AdamW on 520m on 4x Chinchilla Data
81 82
Inorm
Mmmn
BSZ
warmup
Loss
Link
0.9 0.98
le-10
0.004
1
0
256
1000
0.1 2.958
0.008
7.075
128
7.139
0.2 2.962
Table 106: Hyperparameter ablation for AdamW on 520m on 8x Chinchilla Data
31 82
Shorm
Ummn
BSZ
warmup
Loss
Link
0.9 0.98 le-10 0.004
0 256
1000
0.1 2.913
0.008
7.183
128
6.932
0.2 2.921
E.2 SWEEPING RESULTS FOR CAUTIOUS
Table 107:
Hyperparameter ablation for Cautious 300m 2x Chinchilla Data
81
B2
Inorm
Mmin
BSZ
warmup
À
Loss
Link
0.98
0.98
le-25
0.008
1
0
256
2000
0.1
3.165
0
0.004
3.175
128
3.171
2
Table 108: Hyperparameter ablation for Cautious on 300m on 4x Chinchilla Data
31 82
Iuorm Mmin
BSZ
warmup
X Loss
Link
0.98
0.98
le-25
0.008
0
256
2000
0.1
3.094
0.004
3.098
128
3.114
2
82

---

Published as a conference paper at ICLR 2026
Table 109: Hyperparameter ablation for Cautious on 300m on 8x Chinchilla Data
81 B2
9horm
Mmin
BSZ
warmup
X
Loss
Link
0.98
0.98
le-25
0.008
0
256
2000
0.1
3.043
0
0.004
3.041
128
3.071
2
Table 110: Hyperparameter ablation for Cautious on 520m on 2x Chinchilla Data
81
B2
fhorm
Imin
BS7
warmup
X
Loss Link
0.98
0.98
1e-25
0.008
1
0
256
2000
0.1
3.017
0
0.004
3.019
128
>10
2
Table 111: Hyperparameter ablation for Cautious on 520m on 4x Chinchilla Data
81 82
Inorm
Mmin
BS7
warmup
X
Loss Link
0.98
0.98
1e-25
0.004
0 256
2000
0.1
2.956
0
0.008
2.959
128
2.971
2
Table 112: Hyperparameter ablation for Cautious on 520m on 8x Chinchilla Data
81
B2
Juorm
Mmin
BSZ
warmup
À
Loss
Link
0.98
0.98
le-25
0.004
1
0
256
2000
0.1 2.910
0
0.008
2.919
128
2.931
2
E.3 SWEEPING RESULTS FOR LION
Table 113: Hyperparameter ablation for Lion on 300m on Chinchilla Data
8
B2
7)
Inorm
Tmmin
BSZ
warmup
Loss
Link
0.9
0.98
0.001
0
128
2000
0.7
3.170
0.9
3.189
0.95
3.175
2
0.0005
- 3.172
256
3.183
4
83

---

Published as a conference paper at ICLR 2026
Table 114: Hyperparameter ablation for Lion 300m 4x Chinchilla Data
B1
B2
1
Inorm
Jmmin
BSZ
warmup
Loss
Link
0.9
0.98
0.001
0
256
2000
0.7
3.100
0
0.9
3.114
0.95
3.103
0.0005
3.105
128
3.104
4
Table 115: Hyperparameter ablation for Lion 300m 8x Chinchilla Data
B1
B2
1
Inorm
Tmmin
BSZ
warmup
À Loss
Link
0.9
0.98
0.001
0
256
2000
0.7
3.046
0
0.9
3.058
0.95
3,050
0.0005
-3.43
128
3.061
4
Table 116: Hyperparameter ablation for Lion 520m 2x Chinchilla Data
8
B2
1)
Inorm
Tmmn
BSZ
warmup
À
Loss
Link
0.9
0.95
0.001
1
0
128
2000
0.6
3.029
0
0.9
3.045
0.98
7.779
0.0005
3.028
256
3.030
4
Table 117: Hyperparameter ablation for Lion on 520m 4x Chinchilla Data
B
B2
?)
Inorm Tmin
BSZ
warmup
À
Loss
Link
0.9
0.95
0.001
0
256
2000
0.6
2.965
0
0.9
2.971
0.98
2.965
2
0.0005
2.966
128
2.975
4
Table 118: Hyperparameter ablation for Lion on 520m on Chinchilla Data
8
B2
7
Inorm
Tmin
BSZ
warmup
Loss
Link
0.9
0.95
0.0005
0
256
2000
0.6
2.915
0.9
2.922
0.98
2.915
2
0.001
2.920
128
2.922
4
84

---

Published as a conference paper at ICLR 2026
E.4
SWEEPING RESULTS FOR MARS
Table 119; Hyperparameter ablation for Mars on 300m on 2x Chinchilla Data
31
82
7
7
Ihorm
Mmin
BSZ
warmup
À
Loss
Link
0.98
0.99
le-25
0.05
0.008
128
1000
0.1
3.158
0
0.9
3.174
1
0.95
3.156
2
0.004
3.168
3
256
3.171
4
2000
3.159
5
4000
3.167
6
Table 120: Hyperparameter ablation for Mars 300m 4x Chinchilla Data
B1
B2
Inorm Mmii
BSZ
warmup
À Loss
Link
0.98
0.99
le-25
0.05
0.008
128
1000
0.1
3.097
0
0.9
3.111
1
0.95
3.096
2
0.004
3.095
3
256
3.098
4
2000
3.097
5
4000
3.101
6
Table 121: Hyperparameter ablation for Mars on 300m on 8x Chinchilla Data
B1
82
7
7
Inorm Mman
BSZ
warmup
À
Loss
Link
0.98
0.99
le-25
0.05
0.008
256
1000
0.1
3.040
0
0.9
3.049
1
0.95
3.038
2
0.004
3.046
3
128
3.050
4
2000
3.049
5
4000
3.043
6
Table 122: Hyperparameter ablation for Mars 520m on 2x Chinchilla Data
B1
7)
Jnorm Mmin
BSZ
warmup
X
Loss
Link
0.95
0.98
1e-25
0.025
0.008
0
256
2000
0.1
3.015
0
0.9
3.019
0.98
3.014
2
0.004
3.019
128
3.025
4
1000
3.023
5
4000
-3019
6
85

---

Published as a conference paper at ICLR 2026
Table 123: Hyperparameter ablation for Mars 520m 4x Chinchilla Data
B1
82
7
7
Jnarm 1/min
BSZ
warmup
À Loss
Link
0.95
0.98
le-25
0.025
0.008
256
2000
0.1
2.955
0
0.9
2.960
1
0.98
2.953
2
0.004
2.953
3
128
2.974
4
1000
2.964
5
4000
2.956
6
Table 124: Hyperparameter ablation for Mars 520m 8x Chinchilla Data
81
82
7)
Inorm M/min
BSZ
warmup
À Loss Link
0.95
0.98
le-25
0.025
0.004
0
256
2000
0.1
2.906
0
0.9
2.908
1
0.98
2.906
2
0.008
2.917
3
128
2.916
4
1000
2.906
4000
2.907
6
E.5 SWEEPING RESULTS FOR NADAMW
Table 125: Hyperparameter ablation for NAdam on 300m on 2x Chinchilla Data
81
B2
7)
Inorm Imin
BSZ
warmup
À Loss
Link
0.95
0.98
le-
0.008
128
2000
0.1
3.160
0.004
3.060
256
3.165
2
Table 126: Hyperparameter ablation for NAdam on 300m on 4x Chinchilla Data
81
82
Inorm Umin
BSZ
warmup
À
Loss
Link
0.95
0.98
le-
0.008
256
2000
0.1
3.090
0.004
3.097
128
3.098
2
Table 127: Hyperparameter ablation for NAdam W on 300m on 8x Chinchilla Data
81
82
Iuorm Mmin
BSZ
warmup
X Loss
Link
0.95
0.98
le-
0.008
256
2000
0.1
3.039
0.004
3.039
128
3.055
2
86

---

Published as a conference paper at ICLR 2026
Table 128:
Hyperparameter ablation for NAdamW on 520m on 2x Chinchilla Data
B1
B2
9horm
Mmin
BSZ
warmup
X Loss
Link
0.98 0.98
1e-10
0.004
0
128
4000
0.1 3.013
0
0.008
3.023
256
3.020
2
Table 129: Hyperparameter ablation for NAdamW on 520m on 4x Chinchilla Data
81 B2
9horm Mmin
BSZ
warmup
À Loss
Link
0.98
0.98
lc- 0.004
128
4000
0.1 2.955
0.008
2.971
256
2.954
2
Table 130: Hyperparameter ablation for NAdam on 520m on 8x Chinchilla Data
81
B2
77
Inorm
7min
BSZ
warmup
À
Loss
Link
0.98
0.98
le-
0.004
256
4000
0.1
2.907
0.008
2.910
128
2.913
2
E.6 SWEEPING RESULTS FOR ADAM-MINI
Table 131: Hyperparameter ablation for Adam-Mini on 300m on 2x Chinchilla Data
81 82 Inorm
Mmin
BS7
warmup
Loss
Link
0.9 0.98 1e-25
0.004
2
0
128
2000
0.2
3.178
0.002
3.180
256
6.960
4000
3.183
0.1 3.179
Table 132: Hyperparameter ablation for Adam- on 300m on 4x Chinchilla Data
81 82
Inorm
Tmin
BSZ
warmup
À Loss
Link
0.9
0.98
le-25
0.004
2
0
128
2000
0.1
3.103
0.002
3.111
256
3.109
2
4000
3.104
0.2 3.111
87

---

Published as a conference paper at ICLR 2026
Table 133: Hyperparameter ablation for Adam- on 300m on 8x Chinchilla Data
81 82
Shorm
Mmmn
BSZ
warmuu
Loss
Link
0.9 0.98
le-25
0.002
2
0
128
2000
0.2 3.049
0.004
3.064
256
3,052
4000
3,050
0.1 3,051
Table 134: Hyperparameter ablation for Adam-Mini on 520m on 2x Chinchilla Data
81 82
Inorm
Imin
BSZ
warmup
À Loss
Link
0.9 0.98 le-10
0.004
1
0
128
4000
0.1
3.027
0.002
3.031
256
3.032
2000
7.359
0.2 3.037
Table 135: Hyperparameter ablation for Adam- on 520m on 4x Chinchilla Data
81
Inorm
Mmin
BSZ
warmup
Loss
Link
0.9
0.98
le-10
0.004
1
0
128
4000
0.1 2.966
0.002
2.963
225
256
2.963
2000
7.529
0.2
2.981
Table 136: Hyperparameter ablation for Adam- on 520m on 8x Chinchilla Data
81
Iuorm
Tmin
BSZ
warmup
À Loss
Link
0.9 0.98 1e-10
0.004
1
0
256
4000
0.1
2.912
0.002
2.918
128
2.921
2000
7.449
0.2
3.025
E.7 SWEEPING RESULIS FOR MUON
Table 137: Hyperparameter ablation Muon on 300m on 2x Chinchilla Data
Madam
31
32
Decay(WSD)
7
Schedule Inorm Mmin Buaon
Cyuon
BSZ
warmup
À
Loss
Link
0.0024
0.8
0.98
0.8
le-15 0.008
linear
0.98
1e-05
128
0.1
3.143
0.004
3.144
256
3.145
2
88

---

Published as a conference paper at ICLR 2026
Table 138: Hyperparameter ablation for Muon on 300m on 4x Chinchilla Data
Madam
81
82
Decay(WSD)
Schedule Ihorm
Mmin
Amton
6muon
BSZ
warmup
Loss
Link
0.0012
0.8
0.98
0.8
le-15 0.004
linear
0
0.98
1e-05
128
0.1 3.079
0.008
3.088
256
3.083
Table 139: Hyperparameter ablation for Muon on 300m on 8x Chinchilla Data
Nadam
81 32
Decay(WSD)
7
Schedule Inorm
Tmin
Amuon
Emuon
BSZ warmmp
À
Loss
Link
0.0024
0.8
0.98
0.8
le-15 0.008
linear
0.98
le-05 256
0.1
3.032
0.004
3.029
128
3.049
2
Table 140: Hyperparameter ablation for Muon on 520m on 2x Chinchilla Data
Madam
31 32
Decay(WSD)
7
Schedule Inorm
Imin
Amuon
Emuon
BSZ
warmup
Loss
Link
0.0012
0.8 0.98
le-25 0.004
linear
2
0.98
le-05
128
0.1
3.002
0.008
3.008
256
3.009
Table 141: Hyperparameter ablation for Muon on 520m on 4x Chinchilla Data
Nadam
31 B2
Decay(WSD)
Schedule 9uorm
Imin
Bmunn
Emuon
BSZ
warmup
Loss
Link
0.0024
0.8 0.98
le-25 0.008
linear
2 0
0.98
le-05
256
0.1
2.945
0.004
2.944
128
2.963
Table 142: Hyperparameter ablation for Muon on 520m 8x Chinchilla Data
Qadam
8 32
Decay(WSD)
Schedule Inorm
Mmin
Aunon
Cyuon
BSZ
warmup
À
Loss
Link
0.0024
0.8
0.98
le-25 0.008
linear
2
0
0.98
1e-05
256
0
0.1
2.906
0.004
2.900
128
2.930
E.8 SWEEPING RESULTS FOR SCION
Table 143:
Hyperparameter ablation Scion on 300m on 2x Chinchilla Data
Jadam
B1
Decay(WSD)
"
Schedule
Inorm Mmii
Sunon
Cacion
BSZ
warmup
Loss
Link
0.0008
0.98
0.8
0.008
linear
0.95
1c-05
128
0
0.1
3.152
0
0.004
3.153
_
256
3.154
2
89

---

Published as a conference paper at ICLR 2026
Table 144: Hyperparameter ablation for Scion on 300m on 4x Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Immin
Smuon
Eacion
BSZ
warmuu
Loss
Link
0.0008
0.98
0.008
linear
2
0.95
1e-05
256
0
0.1
3.086
0
0.004
3.099
128
3.090
2
Table 145: Hyperparameter ablation for Scion on 300m on 8x Chinchilla Data
Nadam
B1
Decay(WSD)
"
Schodule Inorm
Mmin
muon
Cacion
BSZ
warmup
Loss
Link
0.0004
0.98
0.8
0.004
linear
2
0.95
le-05
128
0
0.1
3.039
0
0.008
3.057
1
256
3.037
2
Table 146: Hyperparameter ablation for Scion on 520m on 2x Chinchilla Data
Madam
B1
Decay (WSD)
"
Schedule Inorm
Imin
Amunn
Excion
BSZ
warmup
Loss
Link
0.0004
0.98
0.004
linear
2
0.95
le-05
128
0
0.1
3.007
0
0.008
3.015
1
256
3.020
2
Table 147: Hyperparameter ablation for Scion on 520m on 4x Chinchilla Data
Madam
B1
Decay (WSD)
7
Schodule Inorm
Mmin
Smuon
Cscion
BSZ
warmup
Loss
Link
0.0008
0.98
0.008
linear
2
0.95
le-05
256
0
0.1
2.952
0
0.004
2.952
1
128
2.970
Table 148: Hyperparameter ablation for Scion on 520m on 8x Chinchilla Data
Nadam
B1
Bocay(WSD) "
Schedule Inorm
Tmnn
Buuon
Eacion
BSZ
warmup
Loss
Link
0.0004
0.98
0.004
linear
2
0.95
le-05
256
0
0.1
2.904
0
0.008
2.913
1
128
2.913
E.9
SWEEPING RESULTS FOR KRON
Table 149:
Hyperparameter ablation for Kron 300m 2x Chinchilla Data
3
blocksize
77
Inorm
Imin
NormGrad Blocking
Initpe Tpe Ppc
BSZ
Steppe warmup
Loss
Link
0.95
256
0.001
1
0
True
True
1 0.2 0.1 128
2000
1000
0.7
3.151
0
0.0005
3.157
90

---

Published as a conference paper at ICLR 2026
Table 150:
Hyperparameter ablation for Kron 300m 4x Chinchilla Data
81
blocksize
7)
Jnorm
Imin
NormGrad Blocking Inilpe
Upe Ppc.
BSZ Sleppe
warmup
Loss
Link
0.95
256
0.0005
0
True
True 0.2 0.1 128 2000
1000
0.7
3.083
0.001
3.090
Table 151: Hyperparameter ablation for Kron 300m 8x Chinchilla Data
81
blocksize
77
Inorm
Mmin
NormGrad Blocking
Initpe
1ppc Ppc
BSZ
Steppe
warmup
Loss
Link
0.95
256
0.0005
1
0
True
True 0.2 0.1 128 2000
1000
0.7
3.031
0
0.001
3.074
Table 152: Hyperparameter ablation for Kron 520m 2x Chinchilla Data
3
blocksize
7
Inorm
Mmin
NormGrad Blocking
Init
The Ppc BSZ Steppe warmup
Loss
Link
0.95
256
0.0005
1
0
True
True 1 0.2 0.1 128 2000
1000
0.5
3.009
0
0.001
3.009
Table 153: Hyperparameter ablation for Kron on 520m on 4x Chinchilla Data
81
blocksize
Inorm
Tmin
NormGrad Blocking Initpe
The
Ppc
BSZ
Steppe warmup
Loss
Link
0.95
256
0.0005
1
0
True
True 0.2 0.1 128 2000
1000
0.5 2.946
0.001
2.950
Table 154: Hyperparameter ablation for Kron on 520m on 8x Chinchilla Data
31
blocksize
7)
Juorm
Mmin
NormGrad Blocking
Initpe
Upc
Pyc
BSZ
Steppe warmup
Loss
Link
0.95
256
0.0005
1
0
True
True
0.2 0.1
128
2000
1000
0.5
2.900
0.001
2.909
256
2.902
2
E.10
SWEEPING RESULTS FOR SOAP
Table 155: Hyperparameter ablation for Soap on 300m on 2x Chinchilla Data
81
82
blocksize
1)
Inorm Imin
Blocking fnc Behampoo
BSZ
warmup
À
Loss
Link
0.95
0.99
512
le-10 0.008
0
True 10
0.9
128
1000
0.1
3.147
0
128
3.154
256
3.150
2
0.004
3.147
256
3.153
4
91

---

Published as a conference paper at ICLR 2026
Table 156: Hyperparameter ablation for Soap 300m 4x Chinchilla Data
81
B2
blocksize
"
Inorm
Mmiin
Blocking Syc Bshampoo
BSZ
warmup
X
Loss
Link
0.95
0.99
512
le-10
0.008
O
True 10
0.9
256
1000
0.1
3.084
0
128
3.086
256
3.084
2
0.004
3,086
128
3.091
4
Table 157: Hyperparameter ablation for Soap on 300m on 8x Chinchilla Data
81
B2
blocksize
7
Inorm
Imin
Blocking
Spc B Sshampoo
BSZ
warmnup
X
Loss
Link
0.95
0.99
512
le-10
0.008
0
True 10
0.9
256
1000
0.1
3.030
0
128
3.034
256
3.032
2
0.004
3.031
128
3.043
4
Table 158: Hyperparameter ablation for Soap on 520m on 2x Chinchilla Data
81
B2
blocksize
7)
Juorm Mmin
Blocking
Spc
Ishampoo
BSZ warmup
À
Loss
Link
0.95
0.99
512
le-10
0.008
True 10
0.95
256
1000
0.1
3.004
128
3.013
256
3,010
2
0.004
3,008
3
128
3.011
Table 159; Hyperparameter ablation for Soap 520m 4x Chinchilla Data
31
82
blocksize
n
Auorm Mmin
Blocking Spc B Ashampoo
BSZ warmup
Loss
Link
0.95
0.99
512
le-10
0.004
O
True 10
0.95
256
1000
0.1
2.944
0
128
2.948
256
2.945
2
0.008
2.949
3
128
2.946
4
Table 160: Hyperparameter ablation for Soap on 520m on 8x Chinchilla Data
81
B2
blocksize
")
Unorm Imin
Blocking fpc B 8shampoo
BSZ
warmuu
À
Loss
Link
0.95
0.99
512
le-10
0.004
O True 10
0.95
256
1000
0.1
2.899
0
0.008
2.906
F
HYPERPARAMETER ABLATION IN PHASE III 1.2B EXPERIMENTS
We reported the results for the optimizers we swept in Phase The result is formulated the same way in
Phase I.
92

---

Published as a conference paper at ICLR 2026
F.1
SWEEPING RESULTS FOR ADAMW
Table 161: Hyperparameter ablation for A 1.2b on 1x Chinchilla Data
81
B2
Inorm
Mmin
BSZ
warmmp
Loss
Link
0.9
0.98
le-10 0.002
256
2000
0.2
2.905
0.95
2.905
0.98
2.909
0.9
2.914
0.95
2.909
le-25
2.907
le-20
2.907
le-15
2.907
0.004
2.916
0.008
7.347
2.909
10
1.0
2.908
11
128
2.904
12
512
2.928
13
1024
2.985
14
500
2.917
15
1000
2.910
16
4000
2.912
17
2.946
0.1 2.916
19
Table 162: Hyperparameter ablation for AdamW 1.2b on 2x Chinchilla Data
81
82 Inorm
Tmin
BSZ
warmup
À Loss
Link
0.9 0.98
lc-10
0.002
1
0
256
1000
0.2 2.836
Table 163: Hyperparameter ablation for Adam on 1.2b on 4x Chinchilla Data
81 82
Iuorm Mmin
BSZ
warmup
Loss
Link
0.9 0.98 le-10 0.002
1
0 256
1000
0.2 2.787
Table 164: Hyperparameter ablation for AdamW on 1.2b on 8x Chinchilla Data
31 82
Juorm
Imin
BSZ
warmup
Loss
Link
0.9 0.98 le-10 0.002
256
1000
0.2 2.752
E2 SWEEPING RESULTS FOR NADAMW
Table 165: Hyperparameter ablation for NAdamW on 1.2b on 1x Chinchilla Data
81 82 Iuorm
Tmin
BSZ
warmup
À
Loss
Link
0.98 0.98
le-10
0.004
256
4000
0.1 2.902
0
93

---

Published as a conference paper at ICLR 2026
Table 166: Hyperparameter ablation for NAdamW 1.2b on 2x Chinchilla Data 81
82
9horm
Mmin
BSZ
warmup
X Loss
Link
0.98 0.98 le-10 0.004
256
4000
0.1 2.833
0
Table 167: Hyperparameter ablation for NAdamW on 1.2b on 4x Chinchilla Data
81 B2
Juorm
'mmn
BSZ
warmup
Loss
Link
0.98 0.98
le-10 0.004
256
4000
0.1 2.785
0
Table 168: Hyperparameter ablation for NAdamW on 1.2b on 8x Chinchilla Data
81 B2
Iuorm
Mmin
BSZ
warmup
À Loss Link
0.98 0.98 le- 0.004
256
4000
0.1 2.749
F33 SWEEPING RESULTS FOR SOAP
Table 169: Hyperparameter ablation for Soap 1.2b on 1x Chinchilla Data
81
82
blocksize
Inorm Iman
Blocking fee Behempoo
BSZ warmuu
X
Loss
Link
0.95
0.99
512
le-10 0.004
O True 10
0.9
256
1000
0.1
2.940
0
Table 170: Hyperparameter ablation for Soap on 1.2b on 2x Chinchilla Data
31
B2
blocksize
7
Juorm Imin
Blocking f.c
3shampoo
BSZ warmup
X
Loss
Link
0.95
0.99
512
le-10 0.004
O True 10
0.9
256
1000
0.1
2.829
Table 171: Hyperparameter ablation for Soap on 1.2b on 4x Chinchilla Data
81
B2
blocksize
7)
Juorm Mmin Blocking Syc. Ashampoo
BSZ warmup
X Loss
Link
0.95
0.99
512
le-10 0.004
O True 10
0.9
256
1000
0.1
2.783
Table 172: Hyperparameter ablation for Soap on 1.2b on 8x Chinchilla Data
81
82
blocksize
"
Ihorm
Mmii Blocking Spc. Ashampoo
BSZ warmup
À
Loss
Link
0.95
0.99
512
le-10 0.004
True 10
0.9
256
1000
0.1
2.749
E4 SWEEPING RESULTS FOR MUoN
Table 173: Hyperparameter ablation for Muon on 1.2b on 1x Chinchilla Data
Nadam
3 32
Decay(WSD)
e
Schedule Inorm
Imin
Bmuon
Cynon.
BSZ
warmup
Loss
Link
0.0012
0.8
0.98
le-15
0.004
linear
2 0
0.98
1e-05
256
0.1
2.891
0.008
2.886
94

---

Published as a conference paper at ICLR 2026
Table 174: Hyperparameter ablation for Muon on 1.2b 2x Chinchilla Data
Madam
81 B2
Decay(WSD)
Schedule Ihorm
Mmin
Amaon
6muon
BSZ
warmup
Loss
Link
0.0012
0.8
0.98
le-15 0.004
linear
2
0
0.98
1e-05
256
0.1
2.827
0.008
2.833
Table 175: Hyperparameter ablation for Muon on 1.2b 4x Chinchilla Data
Madam
31
32
Decay(WSD)
Schedule Inorm
Imin
Punon
Cmuon
BSZ
warmmp
À
Loss
Link
0.0012
0.8 0.98
le-15 0.004
linear
2
0
0.98
1e-05
256
0
0.1
2.780
0.008
2.793
Table 176: Hyperparameter ablation for Muon on 1.2b Chinchilla Data
Madam
31
B2
Decay(WSD)
" Schedule Ihorm
Imin
Ainon
6ynaon
BSZ warmup
À
Loss
Link
0.0012
0.8 0.98
1e-15 0.004
linear
2
0.98
1e-05 256
0.1
2.748
G HYPERPARAMETER ABLATION IN PHASE III 16x CHINCHILLA EXPERIMENTS
G.1 SWEEPING RESULTS FOR ADAMW
Table 177: Hyperparameter ablation for AdamW on 130m on 16x Chinchilla Data
81 82 " Inorm Tmmn
BSZ
warmup
Loss
Link
0.9 0.98 le-10 0.008
0 256
1000
0.1 3.207
Table 178: Hyperparameter ablation for AdamW on 300m on 16x Chinchilla Data
81 82
Juorm
Imin
BSZ
warmup
À
Loss
Link
0.9 0.98 le-10 0.004
2
0
256
2000
0.1
3.001
G.2 SWEEPING RESULTS FOR NADAMW
Table 179: Hyperparameter ablation for NAdam W on 130m on 16x Chinchilla Data
8 B2
Juorm Mmin
BSZ
warmup
À
Loss
Link
0.95 0.98
le- 0.008
256
2000
0.1 3.200
Table 180: Hyperparameter ablation for NAdam W on 300m on 16x Chinchilla Data
81
82
Iuorm
Mmin
BSZ
warmup
À Loss
Link
0.95
0.98
le- 10
0.004
256
2000
0.1 2.998
95

---

Published as a conference paper at ICLR 2026
G.3
SWEEPING RESULTS FOR SOAP
Table 181: Hyperparameter ablation for Soap on 130m on 16x Chinchilla Data
81
82
blocksize
7
Ihorm
Mmmin
Blocking fNc Sshampoo
BSZ
warmnup
X
Loss
Link
0.95
0.99
512
le-10 0.008
0 True 10
0.98
256
1000
0.1
3.191
0
Table 182: Hyperparameter ablation for Soap on 300m on 16x Chinchilla Data
81
B2
blocksize
"
Ihorm
Mmii
Blocking
Sec Sahampoo
BSZ warmuu
X Loss
Link
0.95
0.99
512
le-10 0.004
True 10
0.9
256
1000
0.1 2.990
G.4
SWEEPING RESULTS FOR MUON
Table 183: Hyperparameter ablation for Muon 130m 16x Chinchilla Data
Madam
31
82
Decay(WSD)
Schedule Inorm
Imin
Emuon
BSZ
warmup
À
Loss
Link
0.0012
0.8
0.98
le-25 0.004
linear
1
0
0.98
lc-05
128
0
0.1
3.192
0.008
3.202
Table 184: Hyperparameter ablation for Muon on 300m on 16x Chinchilla Data
Nadam
B1
32
Decay(WSD)
"
Schedule Gnorm Mmin
Amuon
Emwon
BSZ
warmup
Loss
Link
0.0012 0.8 0.98
0.8
le-15 0.004
linear
0.98
1e-05 256
0.1
2.991
H COMPARISON WITH PRIOR WORK
This paper benchmarks the performance of 11 proposed optimizers and show vastly different speed-up ratio than prior works reported. In this section, we will compare the setup our experiments with prior works with the hope of understanding the difference.
1. Sophia Liu et (2024a) (2x) This work utilizes a small peak learning rate of learning rate smaller than 6e-4 (similar to the one shown in Figure Top Left). The reason for the small peak learning rate is likely 2-fold: (i) the authors are training on a pretraining dataset PILE with lower quality compared to the current pretraining dataset and (ii) in the implementation of Levanter that the authors used, the data shuffling is not completely random and instead is correlated on every compute node. Upon reproducing the results, we note that this difference significantly impact the stability of the training process and complete random shuffling is crucial for the usage of a large learning rate.
2. MARS Yuan et al. (2025) (2x) This papers considers similar setup Sophia and uses similar Adam W baseline. We note that in the first version of the paper, the authors also reported that increasing the learning rate of AdamW to 3e-3 can significantly improve the performance of AdamW (see Figure 6 of Yuan et al. (2025) arxiv version 1).
3. Soap Vyas et al. (2025) (1.4x) The actual speedup of Soap on 300M and 520M models are 1.2 to 1.3 which is only slightly lower than the claimed 1.4x speedup. We that our implementation of Soap is slightly different from the one used in the paper that we performs blocking of weight in order to reduce the memory footprint and further uses b for the momentum in the 1.2B experiments. Both modifications may lead to slightly lower dte-wwisc performance.
96

---

Published as a conference paper at ICLR 2026
4. Muon Jordan et al. (2024); Liu et al. (2025a) (2x) The speedup of Muon reported in different works are vastly different. In the original Nanogpt speedrun, Muon achieves 1.3x speedup over AdamW. Later the reproduction of Kimi reported much higher speedup of 2x. We note that the Kimi version utilizes notably low learning rate for AdamW (8e-4 to 9e-4 for model between 400M to 1.5B) in the scaling experiments. We also note that the smaller learning rate is important for hyperparameter transfer from Adam W when roughly matching the update norm and Adam can perform better higher learning rate in experiments. Further, their comparison of Muon and Adam W the MoE experiments compare two models with not fully decayed learning rate and this may significantly favors Muon, as shown in Figure It is later shown independently in the work of Essential AI AT al. (2025) that Muon's 's token efficiency compared to AdamW is only 1.1 to 1.2x.
5. Cautious Liang et al. (2025), Block-wise Learning Rate Adam Wang et al. (2025), FOCUS Liu et al. (2025c) report 2x speedup over AdamW. These papers use similar baseline as Sophia and MARS.
6. SWAN Ma et al. (2025) and DION Ahn et al. (2025) report 2-3x speedup over A.am The comparison between these two optimizers and AdamW is carried out on smaller than 1x Chinchilla regime, where the speed-up of matrix-based optimizer may be larger. We also note that we didn't consider the communication cost, which is the main focus of DION.
7. SPlus Frans et al. (2025) reports 2x speedup over Adam W. This work considers atypical setup where the model trained with a constant learning rate.
VALIDITY OF COORDINATE DESCENT
L.1 THEORETICAL GUARANTEES FOR COORDINATE DESCENT
Here, we provide theoretical result showing that, assuming (i) each hyperparameter is discretized densely enough, and (ii) the hyperparameter-to- function contains no saddle points, classical results from block coordinate descent theory guarantee CD convergence to a locally optimal configuration.
Definition 1 (Hyperparameter Search Space). The hyperparameter search space S is the set of all possible hyperparameter configurations h = (h1...., hha) where all 14 E [L,, R.) some lower and upper bounds Ls and R4
Definition 2 (Coordinate Descent). The coordinate descent (CD) algorithm with grid size 8 is an iterative algorithm that minimizes a loss function C(h) through the following steps:
1. Discretize the search space S into grid of size ó to a set Ss (h Sh h, L, +, for some e
{0, R.-L, }}.
2. Initialize (0) (h1 ...,,h4)) (0) randomly on the grid So.
3. Update h (1+1) to the value in Sş that minimizes ((+(++))) and at most one coordinate of h (k+1) can be different from (1))
Run this procedure until the loss value converges. Assumption 3 (Regularity of the hyperparameter-to-loss function). The hyperparraeeterttolless function L: S + R satisfies the following conditions:
1. All stationary points are either strict local minimimizers or strict local maximizers.
VL(gg) = = <2 = À or V2 1 -X.
2. The hyperparameter on the boundary is not coordinate optimal in the sense that, if h E Ss and h4 L,, then (Chh), < Similarly, h Sā and h4 R4 then VL(h);
3. It is C22 smooth function, i.e.. [[V VC(h'))2 < h h' and
M|h h for some constants L,, M > O and all h, h1 E Sz.
Theorem For any loss function L(h) that satisfies Theorem 3, there exists constant C, such that the converged hyperparameter hī, found by the CD algorithm with grid size 5 (Theorem 2), there exists corresponding local minimum ht' of C(h) such that the regret on the loss value is bounded by C82
C(h3) C(h') < C8?
97

---

Published as a conference paper at ICLR 2026
Proof. By Assumption 3, we know that the
Lemma S. liz satisfies that V (V23))))22 S dLō.
Proof. The proof is by contradiction. Suppose that |V > dLo. Then, there exists a dimension i such that [V((gg)l > Lő. By symmetry, we can assume that VC(h)); Lő.
We now discuss by cases, assume is the i-th standard basis vector.
If ht de, E So, because smooth, we have the following inequality:
C(h δe.) - ((h3) + Vi(hh)"((ée)) L8
1le:12
L8
< ((hh) - L8 +
2
< ((h)).
This contradicts the assumption that h is a converging point for the CD algorithm.
If hiz dc, & Ss, then let ææ the boundary S satisfy that | hīll2 5 and (hB,)); (h3)s is parallel to e,. By smoothness,
VLLhbs.)) vL(33)ll2 < T.IlbB,8 - hīll2 S T.5.
Meanwhile, due to Assumption 3. know that VL(hB,5) <.
2 Vi(h)), (V((h)))VV((()ll
(VL(hB,s)) > L8.
This leads to a contradiction.
The proof is complete.
Further, we can show that the converged hyperparameter hi not close to any of the local maximizers of L(h).
Lemma 6. For every minimizer he of f(h), there exists radius R=
2M O around ht, the loss T, is
locally 2/2 strongly convex and satisfies the Tolyak-Lojasiend condition at the ball B(h",R)
- -(h") < 2A [V
Symmetrically, for every maximizer h², the negated loss L is locally A/2 strongly convex and satisfies the Polyak-Lojasiewizz condition at the ball (hh", R).
Proof. Because hi is minimizer, we have that V VL(h") = 0 and by Theorem 3, we know that V²C((")) 2 A.
Further as the loss is C2 smooth, we have that
vh E B(h", R), [() v² < M| h MR² A 2
Therefore, Yh B(h", R), (LC()) A/2. Therefore, the Polyak- condittiiiiinnnnccciiiiiiin loss is strongly convex when restricted to the ball B(h", RR).
condition is satisfied as the
We will first show that the converged hyperparameter h3 is not close to any of the local maximizers of C(h).
Lemma There exists a radius R R= R=V
> 0.1f hz the converged hyperparameter found by the CD
algorithm with grid size 8, for any local maximizer h' of [(h), have that || h'll2 R/2.
98

---

Published as a conference paper at ICLR 2026
Proof. We will prove by contradiction. Suppose that ||4 12 < R/2.. By Lemma 6, we know that the negated loss - is 2/2 strongly convex. And by Lemma we know that [VV[()))ll S dL.. Therefore, we know that hg will be closed to h as strong convexity of - -. implies
2 hilla - A IVC((i)l S dLS.
2
Smoothness then implies that ((hz) is close to L(h").
22
[(h3) L(N) 2 2 11n 1112 2
A2 5².
Further, can show that for any h (B(N', R) B(h' Ld()/)55/2 n S, the loss value will be smaller than C(hū) due to the local strong concavity of L.
[(h) (()) s - ||| 11112
As
8,
|/ 1 11112 2
x 8².
Then we have that,
L(h3) 2 L(h') L3
X2 8²
> L(h') 2 11 h' 2 L(h).
As Lhī)) is a converged point for the CD algorithm, we know that it has the smallest loss value in the following set:
[hž- - ide, : : Z} ns.
Let N = [4d( + 1], when ō is small enough, we know either li Noe; or hi + Noe is in the set [hg idei: : nS and the distance between either of them to ht' is smaller than It and larger (/(LA))//25 This is a contradiction. The proof is then complete.
Now can prove the main theorem, showing that the regret on the loss value is bounded by C82. Now consider the set of all stationary points of (hh) to be S* = [h- VL(h") = 0} and define the neighborhood
of lt" to be N(S)) = {h E S 3h E S", |/ h- 12 S V 237 J. As the gradient of L is continuous and S is compact, we know that there exists lower bound on the gradient norm in S NS*)), i.c., there exists constant y > 0 such that
vh S N(S)), IVC())) 27.
For all d < 2 by Lemma 5, we know that the converged hyperparameter h; found by the CD algorithm with grid size 8 satisfies that [/V2()))]lz S dLS < y. Therefore, hj N((*). By Theorem 7, we know that hū is not close to any of the local maximizers of C(h). Therefore, there exists a corresponding local minimizer h'
of Ch) such that ||h/ hille S V
2M
Further by Lemma 6, we know that
1 - I 21
dLL (c
82
2A
Therefore, the regret the loss value is bounded by C6 with C'
&t
99

---

Published as a conference paper at ICLR 2026
I.2 BENIGNITY OF THE HYPERPARAMETER OPTIMIZATION FROM PRIOR WORKS
Our theoretical result relies on the following benignity condition on the hyperparameter optimization problems: all stationary points are strict local minima. While is not computationally feasible check this condition for all hyperparameters, we investigate some prior works studying the interactions between two hyperparameters and find that this condition is often satisfied when constrained to the 2-dimensional hyperparameter search space.
You et al. (2017; 2019); Marek et al. (2025); Li et al. (2025a) study the interactions between batch size and learning rate. As shown in the Figure 1 of Li et al. (2025a), the loss surface is near quasi-convex with respect to learning rate and batch size and only contains one basin.
2. Bergsma et al. (2025) study the interactions between weight decay and batch size. As shown in the Figure 4 of Bergsma et al. (2025), the loss surface is near quasi-convex with respect to weight decay and batch size and only contains one basin.
3. Marek et al. (2025) study the interactions hetween learning rate and $2. As shown in the Figure of Marek et al. (2025), the loss surface is near quasi-convex with respect to learning rate and Ba is near quasi-convex and only contains one basin.
I.3 VALIDATION or THE BENIGNITY OF THE HYPERPARAMETER OPTIMIZATION IN OUR EXPERIMENTS
We validate the benignity of the hyperparameter optimization in our experiments by performing 2 dimensional sweeping for the hyperparameters in Muon 130M 1x Chinchilla regime. We consider the following three pairs of hyperparameters: (learning rate, batch size), (learning rate, weight decay), and (learning rate, WSD decay ratio). The results shown in the Figure 10.
Heatmap: Loss by LR and OZ
Heatmap: Locs by and WD
Heatmap: Less by Ll and Dit
01
3,011
J001
5.0e-22
6, 6.0e-33
2:04
1,04
1:3+002
1.0e-33
1,81
120
2017
0
2.
Aneng
80050
Lix02
deot
Leaming Rote
Jeam Rate
Figure 10: Validation of the benignity of the hyperparameter optimization in our experiments settings. We observe that the loss surface is near quasi-convex with respect to the learning rate and other hyperparameters. The blue star marks the optimal hyperparameter configuration in these sweeping and it is the one we found by our coordinate descent procedure.
J ABLATIONS ON THE EFFECT OF BATCH SIZE
We conduct sweep across different model sizes, training budgets, and batch sizes to understand how the speedup between AdamW and Muon changes with the batch size. Due to time constraints, we adopt the hyperparameter found in experiments and only sweep the learning rate when change the batch size.
Our experiments cover 130M and 300M parameter models trained with varying Chinchilla ratios using both AdamW and Muon optimizers. We also scale up 1.2B parameter model in the Ix Chinchilla ratio to further confirm the results. We first show the aggregated results on the best performance of the two optimizers across different model sizes, training budgets, and batch sizes in Figure 11. We observe two phenomena: (i) the loss values increase as the batch size increases for both optimizers; (ii) Muon's 's loss increases slower than AdamW's. leading to a larger speedup between the optimizers the batch size increases.
J.1 RESULTS LEARNING RATE SWEEP
We make the following observations how the learning rate scale with the batch size:
100

---

Published as a conference paper at ICLR 2026
A0
1.0
123
3.0
330
400
1100
102.e boo Soa 1204 Commbr 221
102 Sw St Dontattraa11
3.50
314
3.00
1.0
341
540
Figure 11: Minimum achievable loss function of batch size for different model sizes and Chinchilla ratios. We observe a general upward trend in the loss values as the batch size increases for both optimizers but Muon's loss decreases slower than Adam W'
3.25
adarmn bom125
adame to=120
195
volrmer 1x=328
atame tam25
tx=22e
3.035
1.000
3.75
MMge b0m253
377
tom2be
3.852
muzn 51-512
2.7755
muan Sa
3.70
muon ba-512
3.50
3.800
5
3702
1675
3,50
1.75
1030
2+01 Fest 3nt6 EDI E:42
46:01
6336 EX1T
3762
L1 -Busicles
01 (S60stcus
Uix Sate-s0e
L055 11.7550 1130, Chacc da Roto 88
Los B00thas SOON Caicc Roto 3.01
0055
(128
3.712
ssama b 25
atamn bo
3.50
Some 256
soune
8:25
scame
Journ 23
acumm 23m 126
muze
3325
muee
3.75
3.50
3,075
bym312
3:50
3.65
3.400
3.50
5425
3.200
3400
2375
2,550
JX. SMIETAE
Figure 12: Learning rate sweep results. Each subplot shows evaluation loss C4 function of learning rate (scaled by / BBttchSize/ (aaiiiee/122) 128) for different training budgets. Different colors represent different batch sizes, with blue for Muon and red for AdamW.
Square root scaling until critical batch size. We confirm prior works You al. (2017; 2019); Marek et al. (2025) that the learning rate scales with the square root of the batch size until a certain batch size is reached, where the optimal learning rate tends to a constant. We observe this phenomenon in our experiments for both Muon and AdamW.
Training budget effeets: As the training budget increases (higher Chinchilla ratios), the optimal learning rate tends to decrease, and the loss curves become less sensitive to learning rate selection.
Based on experiments, we provide the following recommendatioos for practitioners:
1. Batch Size Scaling:
Use LR LR x /BatchSizenew// BaatchSineeee/BBacchiiieee. pattchSizeeasse
Larger batch sizes (up to 512) can be used without performance loss when learning rate is properly scaled
2. Optimizer Choice:
101

---

Published as a conference paper at ICLR 2026
If the goal is to reach decent loss as quickly as possible, use Muon with the largest batch size can be potentially beneficial as this can combine the benefits of Muon larger batch size setting and Muon's s memory efficiency. This also reduces the step-wise overhead of Muon over Adam W. If the goal is to reach the optimal loss given fixed amount of data, Muon with the smallest batch size will likely lead to the best performance.
K
ADDITIONAL EXPERIMENTS ON THE LEARNING RATE SCHEDULE USED FOR MUON
In response to the reviewer's problem on the linear learning rate schedule used for Muon, we performed additional experiments investigate the effect of the learning schedule. We consider swapping the linear learning rate schedule with the cosine learning rate schedule in the Muon 130M Ix Chinchilla regime. The results are shown in the Figure 13. We observe very minimal difference in the loss values between the two learning rate schedules (less than 2e-3 on the loss value).
Muon Cosine vs Linear Learning Rate Schedule
Cosine
3.8
3.7
CA/EN Loss
3.6
3.2
5000
10000
15000
20000
Step
Figure 13: Validation of the effect of the cosine learning rate schedule in the Muon 130M 1x Chinchilla regime.
L COMPUTE BREAKDOWN
L.1 SUMMARY
Total unique tuples: 150
Total runs: 4,281
Total tracked hours: 7542.52
Total GFLOPs: 186.42T
H100 GPU HOURS (USING 989 TFLOPs As 1 H100 THROUGHPUT)
0100 Equivalent Hours: 52,359.48
H100 Equivalent Days: 2,181.64
H100 Equivalent Years: 5.98
102

---

Published as a conference paper at ICLR 2026
L.2 PHASE I - 61 UNIQUE TUPLES
All 130m (except 42B and 48B), All 300m 6B, All 520m 10B
PHASE SUMMARY
Total GFLOPs: 79.31T
Total Runs: 3,772
Total Hours: 5,456.01
H100 Equivalent Hours: 22,275.52
H100 Equivalent Years: 2.54
# Model
Data
Optimizer GFLOPs
Runs
Hours
520m
10B
sophia
6.0T
129
704.55
2 520m
10B
scion
4.5T
76
110.03
520m
10B
mini
4.5T
90
150.58
4 130m
21B
soap
4.4T
130
383.35
5 130m
21B
sophia
4.0T
135
807.47
6 130m
21B
mars
2.9T
73
111.81
7 130m
21B
muon
2.6T
61
46.06
8 520m
10B
mars
2.4T
83
52.30
9 520m
10B
kron
2.4T
53
36.88
10 520m
10B
muon
2.3T
40
64.85
11 520m
10B
soap
2.2T
60
233.62
12 520m
10B
nadam
2.1T
37
33.01
13 520m
10B
adam
2.0T
46
33.66
14 130m
scion
1.7T
81
79.11
15 130m
10B
scion
1.7T
95
73.27
16 130m
10B
soap
1.5T
126
283.55
17 520m
10B
cautious
1.5T
30
18.11
18 130m
21B
cautious
1.4T
38
57.20
19 300m
6B
sophia
1.4T
95
239.74
20 130m
10B
muon
1.3T
102
113.12
21 300m
6B
mars
1.2T
67
56.56
22 300m
6B
mini
1.2T
61
46.15
23 300m
6B
scion
1.2T
133
59.51
24 130m
10B
adam
1.1T
53
45.03
25 130m
21B
kron
1.0T
28
9.60
26 300m
6B
soap
1.TT
62
114.65
27 130m
10B
sophia
1.0T
72
133.75
28 130m
10B
mars
1.0T
48
54.55
29 300m
6B
cautious
993.4B
55
35.11
30 300m
6B
adam
991.9B
47
29.86
31 130m
21B
nadam
951.5B
22
10.37
32 130m
10B
kron
936.9B
54
64.66
33 130m
21B
adam
908.3B
21
8.98
34 130m
21B
mini
874.6B
21
30.64
35 130m
10B
mini
802.5B
38
19.16
36 300m
6B
muon
766.3B
35
17.28
37 130m
5B
soap
745.4B
76
186.13
38 130m
kron
706.
141
94.43
39 130m
2B
muon
671.0B
134
81.45
40 130m
2B
soap
565.3B
124
102.08
41
130m
2B
scion
544.8B
114
61.91
42 130m
5B
scion
543.0B
52
25.65
Continued on next page
103

---

Published as a conference paper at ICLR 2026
Table 185 continued from previous page
#
Model
Data
Optimizer GFLOPs
Runs
Hours
43
130m
5B
mars
532.6B
71
29.89
44
300m
6B
kron
503.6B
36
9.28
45 130m
2B
sophia
476.6B
103
133.70
46 130m
10B
nadam
475.8B
22
31.92
47 130m
10B
cautious
475.8B
22
37.10
48 300m
6B
nadam
459.8B
21
13.76
49 130m
5B
sophia
428.3B
50
87.75
50 130m
5B
cautious
400.1B
37
15.60
51 130m
5B
muon
378.4B
35
13.23
52 130m
2B
cautious
354.2B
66
38.16
53 130m
2B
adam
298.2B
61
31.99
54 130m
198B
adam
281.2B
14
8.39
55 130m
5B
kron
248.7B
25
8.03
56 130m
5B
adam
248.7B
23
11.60
57 130m
5B
nadam
227.1B
21
7.10
58 130m
5B
mini
227.1B
21
8.03
59 130m
2B
nadam
195.7B
43
21.50
60 130m
2B
mini
156.8B
32
13.67
61 130m
2B
mars
145.1B
31
15.52
104

---

Published as a conference paper at ICLR 2026
L.3 PHASE II - 54 UNIQUE TUPLES
All other 300m and 520m (except 96B and 171B)
PHASE SUMMARY
Total GFLOPs: 70.98T
Total Runs: 407
Total Hours: 1,749.34
H100 Equivalent Hours: 19,936.05
H100 Equivalent Years: 2.28
# Model Data Optimizer GFLOPs
Runs
Hours
520m
85B
mini
6.8T
14
12.14
2 520m
85B
mars
6.2T
13
42.15
3 520m
85B
scion
4.8T
10
117.56
4 520m
85B
soap
4.2T
14
115.94
5 520m
42B
soap
3,2T
17
199.24
6 300m
48B
mini
2.8T
16
159.03
7 520m
85B
adam
2.8T
24.80
8 520m
85B
nadam
2.1T
5
22.25
9 300m
48B
mars
2.0T
12
83.16
10 520m
85B
muon
1.9T
6.14
11 520m
85B
cautious
1.9T
4
1.56
520m
42B
mini
1.9T
15
7.24
13 520m
42B
scion
1.TT
8
70.32
520m
42B
mars
1.7T
2.53
15 300m
48B
soap
1.6T
64.89
520m
85B
kron
1.5T
103.75
17 520m
42B
adam
1.4T
35.14
300m
48B
scion
1.4T
21.83
19 300m
24B
mini
1.3T
15
13.11
20 300m
12B
adam
1.2T
29
19.60
21 520m
21B
mini
1.1T
10
20.59
22 520m
21B
soap
1.0T
10
73.19
23 300m
48B
adam
1.0T
26.27
24 520m
42B
muon
1.0T
5
22.76
25 520m
42B
cautious
930.2B
27.42
26 520m
21B
scion
924.4B
8
9.76
300m
24B
soap
913.9B
12
63.52
28 520m
21B
mars
847.6B
7
7.57
29 300m
48B
muon
804.4B
29.87
30 520m
42B
nadam
726.5B
3
23.79
31 300m
24B
scion
706.7B
10
22.70
32 300m
48B
nadam
620.5B
4
21.27
33 300m
48B
cautious
618.1B
23.69
34 300m
24B
mars
613.0B
8.48
35 300m
48B
kron
525,5B
19.21
300m
24B
adam
493.5B
22.71
37 520m
21B
adam
484,4B
16.16
520m
42B
kron
484.4B
16.26
39 520m
21B
cautious
408.4B
4
2.95
40 300m
12B
mini
394.1B
16.15
41
520m
21B
nadam
363.3B
3
11.28
42 520m
21B
muon
363.3B
1.22
Continued on next page
105

---

Published as a conference paper at ICLR 2026
Table 186 continued from previous page
#
Model
Data
Optimizer
GFLOPs
Runs
Hours
43
300m
12B
mars
357.8B
12
16.32
300m
24B
cautious
350.3B
13.28
45 300m
24B
nadam
350.3B
4
15.42
46 300m
12B
soap
262.8B
12
41.53
47 300m
24B
kron
262.7B
16.24
48 300m
24B
muon
262.7B
7.06
49 520m
21B
kron
242.2B
11.80
50 300m
12B
scion
218.9B
5
1.96
51 300m
12B
cautious
175.2B
3.43
52 300m
12B
nadam
131,4B
3
1.31
300m
12B
muon
131.4B
5.98
54
300m
12B
kron
131.4B
3
5.81
106

---

Published as a conference paper at ICLR 2026
L.4 PHASE III - 32 UNIQUE TUPLES
1.2B, 130m 42B, 520m 171B, 300m 96B
PHASE SUMMARY
Total GFLOPs: 36.13T
Total Runs: 102
Total Hours: 337.17
H100 Equivalent Hours: 10, 147.90
H100 Equivalent Years: 1.16
# Model
Data
Optimizer GFLOPs
Runs
Hours
1.2b
24B
adam
8.9T
41
135.65
2 1.2b
193B
muon
3.2T
3
11.88
1.2b
193B
soap
2.9T
2
15.39
4 1.2b
193B
adam
2.1T
2
1.13
5 1.2b
96B
muon
2.1T
2
6.75
6 1.2b
193B
nadam
2.1T
1
0.96
7 1.2b
96B
soap
1.3T
2
19.93
8 1.2b
96B
nadam
1.1T
2
1.13
9 1.2b
96B
adam
1.TT
2
8.42
10 1.2b
48B
muon
1.IT
2
1.03
11 1.2b
48B
soap
1.0T
3
26.58
12 520m
171B
soap
968.7B
12.07
13 520m
171B
kron
968.7B
0.37
14 520m
171B
nadam
968.7B
9.34
15 1.2b
24B
soap
693.8B
10
21.04
16 520m
171B
adam
645.9B
12.08
17 520m
171B
muon
600.7B
1
7.22
18 1.2b
48B
nadam
528.5B
2
0.91
19 1.2b
24B
muon
525.4B
2
1.37
20 1.2b
48B
adam
525.4B
0.78
21 300m
96B
adam
366.4B
2
0.79
22 300m
96B
nadam
350.3B
11.23
23 300m
96B
soap
350.3B
1.38
24 300m
96B
kron
350.3B
0.39
25 300m
96B
muon
350.3B
0.77
26
130m
42B
muon
346.0B
8
11.42
27 1.2b
24B
kron
262.7B
7.45
28 1.2b
24B
nadam
262.7B
0.70
29
130m
42B
soap
86.5B
7.71
30
130m
42B
nadam
86.5B
0.39
31
130m
42B
kron
86.5B
0.49
32
130m
42B
adam
86.5B
0.40
Total Unique Tuples (normalized): 147
Total GFLOPs: 186.42T
Total Runs: 4,281
Total Hours: 7.542.52
H100 Equivalent Hours: 52,359.48
H100 Equivalent Days: 2,181.64
H100 Equivalent Years: 5.98
107


Reviews:
[{"review_id": "o9zv4D8UpC", "reviewer": "Reviewer_xiBh", "summary": "The paper benchmarks popular optimizers on LLM pretraining, including AdamW as a baseline. It tunes their hyperparameters using a three phased methodology. In phase 1 and phase 2 the hyperparameters are searched using coordinate descent i.e., at every iteration only a single hyperparameter is searched, while all others are held fixed at their current best values, and this is repeated until convergence. In phase 2, which involves larger scale experiments, only a subset of the hyperparameters are tuned, based on if they are determined scale sensitive using phase 1 results. Finally, phase 3 fits scaling laws from the hyperparameters found in phase 2. The benchmark trains Llama 2 architectures of four sizes, from 0.1B to 1.2B parameters and reports validation loss on C4-EN and on downstream tasks. The paper concludes that, while newer optimizers show non-negligible speedups over AdamW, the speedups are more moderate than those stated in existing literature and decrease as model size increases.", "questions": "Is Coordinate Descent (CD) a common methodology for tunning optimizer hyperparameters? Could the authors cite relevant works (preferably in the setting of LLM pretraining) to support this? Could the authors provide supporting experimental evidence or analysis for the claim of \"near-optimal hyperparameters\" and also discuss potential limitations of CD (if any) for the given setting?\n\nIt would be useful if the authors could provide some statistics about the number of runs conducted per setting. For example, for each unique triplet (Optimizer, Model size, Chinchilla Ratio) how many pretraining experiments where performed in each phase? Also, could the authors provide how long each experiment took approximately (if possible translated in H100 GPU hours). This would help me get a better understanding of the scale of the benchmark.\n\nIt is mentioned in the paper that the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing to only 1.1x for 1.2B parameter models. Could the authors discuss possible reasons for this trend? Additionally, for models larger than 1.2B parameters, do the authors expect the speedup to decrease further?\n\nIn Table 5 (Scaling-Sensitive Hyperparameters), some optimizers appear to have more scale-sensitive hyperparameters than others. For example, AdamW appears scale-sensitive to learning rate, warmup, weight decay and batch size, whereas Muon only to learning rate. Also, Adam-Mini is not scale-sensitive to batch size, while AdamW is. Could the authors provide any further insights on these differences", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important and timely topic. Pretraining is indeed the most resource-demanding stage in the LLM training process and thus increasing the optimizer's speed is of great research interest. I also agree with the authors that many works that propose new optimizers lack fair and rigorous evaluation, leading to unrealistic speedups (e.g., 2-3x speedups over AdamW). \n\nA wide range of popular optimizers is evaluated, from different representative categories.\n\nThe experimental settings are clearly stated, and the accompanying code with the sweep configs ensures reproducibility of the results.", "weaknesses": "**Suitability of Coordinate Descent for hyperparameter tuning:**\n\nMy main concern is the use of Coordinate Descent (CD) for hyperparameter tuning. The paper's primary contribution, as stated in the introduction, is the claim that \"the hyperparameters are near-optimal for each setting\", and this is reiterated in the Phase 2 results section: \"Combined with the results in Phase I, we obtain a set of near-optimal hyperparameters\".\n\nHowever, it is well established in the literature that many optimizer hyperparameters are coupled. For example, the learning rate and batch size are often considered to follow  a  linear or square root relationship [1,2]. Recent studies suggest that the weight decay also shares a similar coupling with the batch size [2]. [3] also demonstrates a strong relationship between the Adam beta2 parameter and batch size. Similar couplings are likely to exist for other hyperparameters, such as learning rate and warmup steps (fewer warmup steps could likely benefit from smaller learning rates) or learning rate and gradient norm clipping threshold.   \n\nIn short, the objective (validation loss) is far from separable across coordinates (hyperparameters), making it uncertain whether CD can truly obtain \"near-optimal\" hyperparameters rather than getting stuck at \"bad\" local minima. In addition, the three phased methodology increases the likelihood of errors aggregating from one phase to another.\n\nThe paper includes no ablations or error analysis (e.g., as in [4]) to validate the tuning methodology and support the claim of \"near-optimal hyperparameters\". The ablations in the Appendix (\"1-dimensional ablations centered around the found configuration\") do not address such possible limitations of CD, since each step varies only a single parameter at a time. For example, the batch size is increased from 128 to 256 and 512, while the learning rate as well as all other hyperparameters remain fixed. \n\nIn short, since all of the paper's results rely on the robustness of the evaluation methodology, it is critical that this methodology is fully justified and potential limitations of CD in this setting be discussed. The paper makes a strong claim of \"near-optimal hyperparameters\", but absence of such analysis make this claim uncertain and this represents a major limitation of the work.\n\n**Speedup Results Comparability:** \n\nThe paper defines the speedups based on the number of tokens needed to reach a given loss, without considering the time each optimizer needs to reach that loss. However, since the batch size is treated as a tunable hyperparameter, some optimizers use larger batch size than others in certain settings. For example, on 300M model for 4x Chinchilla Data, AdamW uses batch size 256 (Table 102) and Adam-Mini 128 (Table 132). As a result, it is unclear if the reported results are directly comparable, since optimizers with larger batch size may have increased throughput (assuming sufficient hardware), which could lead to a faster convergence in wall-clock time. \n\nIn short, it is debatable if the batch size should be treated as a regular hyperparameter, given that it influences the throughput and can affect the speedup in terms of wall-clock time. Furthermore, it is common for users to want to choose the best optimizer for a given batch size constrained by memory limitations, and the results provided do not help with this decision-making process.\n\n**Single Architecture Type Used:**\n\nThe paper only tests the optimizers on four models of the same family (LLaMA 2). Including a broader range of model families would have been ideal to ensure the results are generalizable. For example, including more recent models (e.g., Gemma 2 or Qwen 3) or exploring different model types such as Mixture-of-Experts. Additionally, while scaling up to larger models (7B+) might be resource-prohibitive, testing on models at this range would more closely approximate production-level model sizes.\n\n[1] LARGE BATCH OPTIMIZATION FOR DEEP LEARNING: TRAINING BERT IN 76 MINUTES, 2020.\n\n[2] Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training, 2025.\n\n[3] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful, 2025.\n\n[4] Practical Efficiency of Muon for Pretraining, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Mtwstng39", "reviewer": "Reviewer_3kks", "summary": "A thorough pretraining evaluation study is proposed, comparing and contrasting recent matrix-based optimizers such as Muon, SOAP, etc., with scalar-based optimizers such as AdamW. Different model sizes (130M–1.2B) and data budgets in terms of Chinchilla optimal (1x–8x) with the Llama 2 architecture are considered for benchmarking. Key findings include, among others, a decrease in the performance of matrix-based optimizers relative to AdamW as model size increases, and the relative performance differences between different matrix-based optimizers (Muon, SOAP, Kron) disappear in the over-trained regime.", "questions": "- Can authors please elaborate on the search space of hyperparameters used for each optimizer in their sweeps and how was it chosen?\n- During coordinate descent - how are hyperparameters sampled and what scale is used (uniform or log-scale)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Optimizers are benchmarked under strong settings: diverse data mixtures (code, text, math), language, chinchilla-optimal data budgets (1-8x), various model scales (130m-1.2B) and modern matrix-based baselines (muon, scion, soap).\n- Tuning framework to identify scale-sensitive hyperparameters for each optimizer in small-scale settings with sweeps and a scaling law to predict optimal scale-sensitive hyperparameter values for large-scale experiments based on model size and data.", "weaknesses": "- Optimizer optimal hyperparameters are chosen with a one-at-time sweep over hyperparameters which may miss correlation among hyperparameters as obtained from a dense grid search. This could have non-trivial downstream impact on the findings of the study, since optimal hyperparameters from a grid search could be quite different. The coordinate descent is done over too many hyperparameters including non-trivial ones with a few runs (~10-20 which is way less compared to prior work e.g. [1]), and I'm not sure if the final hyperparameters obtained from this one-at-a-time coordinate descent are actually optimal compared to a standard grid search over hyperparameters. \n- Schedule for Muon in appendix tables (Table 83-88) is mentioned as \"linear\" -- does it mean just a linear decay? On the other hand, figure 6 shows cosine decay with warmup for different optimizers. Can authors confirm the choice of scheduler and if it's the standard cosine decay?\n- It should be noted that benchmark is highly specific -- llama 2 architecture with text-only datasets, fixed sequence length (4096) and upto 1.2B scale. Hence, findings such as matrix-shaped optimizers decrease in speedup with model size, may change in multi-modal setup with vision tokens and different/modern architectures such as MoEs which are realistically used in the pretraining of modern LLMs.\n\n[1] Small Batch Size Training for Language Models (NeurIPS 2025), https://arxiv.org/abs/2507.07101", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEyOInirXU", "reviewer": "Reviewer_pAs8", "summary": "The paper presents a comprehensive study of eleven deep learning optimizers to find the pareto frontier across four model sizes (0.1B–1.2B parameters) and data-to-model ratios (1–8× the Chinchilla optimum). It provides valuable practical insights and methodological guidance to find the optimal pretraining optimizer, though it is primarily an empirical study with limited theoretical contributions.", "questions": "Like mentioned in the scope for improvement section, I am curious what the authors think about:\n- a receipe to evaluate optimizers on SOTA models with even higher data ratios.\n- theoretical understanding about some of the trends are missing. For instance, in Figure 1, the last line says \"the three matrix-based optimizers converge to a similar loss in an overtrained setting.\" A more in depth theoretical analysis would be very helpful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- The paper’s deep dive into evaluating multiple optimizers by individually tuning hyper-parameters is very interesting.\n- The finding that speedups in newer optimizers over AdamW come down to 1.1X at 1.2B model scale at 8X Chinchilla is intriguing.\n- The paper is methodical and analyses the optimizers in an under explored regime (of 1.2B model size and 8X Chinchilla data size) in terms of both model and data size. It has the potential to serve as a good reference for the research community while making decisions on pre-training large language models.\n- The authors look at a range of benchmarks to evaluate the optimizers.", "weaknesses": "Scope for improvement:\n- The largest models the authors experiment with is 1.2B parameters, while SOTA models are at-least 100X larger. This might not be directly applicable to SOTA open source models, because even the authors claim that ranking of optimizers changes with data / model scale.\n- On a similar note, a recipe to evaluate optimizers on bigger models and higher data ratios would be very interesting to look at.\n- The paper focusses a lot on empirical results, but doesn’t add any theoretical contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oCZjZu6FWy", "reviewer": "Reviewer_DRRU", "summary": "The paper benchmarks 11 pretraining optimizers (e.g., AdamW, Muon, Soap, Mars, Kron) across model scales up to 1.2B parameters and multiple Chinchilla ratios. It introduces a three-phase tuning protocol to ensure fair comparison and finds that most reported speedups over AdamW shrink to ~1.1–1.4× after proper tuning, especially at larger scales. Matrix-based optimizers perform slightly better on small models but lose advantage as size grows.", "questions": "pls see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper presents a comprehensive optimizer benchmark for pretraining, with consistent tuning and scaling analysis.\n* The proposed three-stage hyperparameter search ensures that each optimizer is fairly tuned, reducing the common bias of under-optimized baselines in prior studies.\n* The work clarifies that many claimed gains in prior work are overstated and provides actionable guidance for future optimizers' usage.", "weaknesses": "* The work is entirely empirical. Although this is acceptable for benchmarking, it lacks a theoretical analysis explaining why matrix-based optimizers lose advantage with scale or high data-to-model ratios. Even basic intuition about curvature estimation or variance scaling would enrich interpretation.\n* Despite the emphasis on “LLM pretraining,” the largest model tested has 1.2B parameters. It remains unclear whether the diminishing returns of matrix-based optimizers persist at realistic scales (10B–70B+).\n* The paper normalizes speedup by “tokens to reach a loss,” assuming step overhead is <10%. Please also report wall-clock time, throughput, optimizer step latency and memory footprint. This will show whether matrix preconditioning’s modest token gains translate to real training gains at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)