Beyond Outliers: A Study of Optimizers Under Quantization
Authors: Georgios Vlassis
Saleh Ashkboos
Alexandra Volkova
Torsten Hoefler
Dan Alistarh
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
Quantization
Optimizers
post-training quantization
quantization-aware training
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a systematic study of the effect of optimizer choice on quantization, both during and after training

Abstract: As new optimizers gain traction and model quantization becomes standard for efficient deployment, a key question arises: how does the choice of optimizer affect model performance in the presence of quantization? Despite progress in both areas, systematic evidence on optimizer–quantization interactions remains limited. To fill this gap, we study the impact of optimizer choice on model robustness under quantization, considering both post-training quantization (PTQ), and quantization-aware training (QAT). We first train full-precision models, ranging from 50M to 1.5B parameters, with six optimizers, to explore the hyperparameter landscape, and establish well-tuned baselines.
We then apply PTQ to evaluate how model performance degrades when trained with different optimizers. We find that outlier-related metrics, such as the max-to-mean ratio (MMR) and Kurtosis, fail to predict the PTQ performance across different optimizers. We show analytically that this is due to the MMR capturing only isolated layer errors, while ignoring how quantization errors accumulate and propagate through the network. To study the QAT degradation, we train quantized models from scratch and compare them to our original-precision baselines. We find that optimizers performing well in the original pretraining setup may not remain optimal under QAT, and that models trained with Shampoo show the lowest accuracy degradation. Finally, we derive scaling laws for quantization-aware training under different optimizers, showing that Shampoo achieves the highest parameter efficiency of all tested optimizers.

Published as a conference paper at ICLR 2026
BEYOND OUTLIERS: A STUDY OF OPTIMIZERS UNDER QUANTIZATION
Georgios Vlassis'
Saleh Ashkboos
ETH Zurich gylassis@ethh.cht
ETH Zurich saleh. walehh.shhboos@iif..tt
Alexandra Volkova
Torsten Hoefler ETH Zurich htor@ethz
Dan Alistarh ISTA & Red Hat AI dan anw.aalsstrristt..
ISTA
avolkovaGist.ac.
ABSTRACT
As new optimizers gain traction and model quantization becomes standard for efficient deployment, key question arises: how does the choice of optimizer affect model performance in the presence of quantization? Despite progress both areas, systematic evidence optimizer-quantie limited. To fill this gap, we study the impact of optimizer choice on model robust- ness under quantization, considering both post-training quantization (PTQ), and quantization-aware training (QAT). We first train full-precision models, ranging from 50M 1.5B parameters, with six optimizers, explore the hyperparameter landscape, and establish well-tuned baselines. We then apply PTQ to evaluate how model performance degrades when trained with different optimizers. We find that outlier-related metrics, such as the max-to-mean ratio (MMR) and Kurtosis, fail to predict the PTQ performance across different optimizers. We show analytically that this is due to the MMR capturing only isolated layer errors, while ignoring how quantization errors accumulate and propagate through the network. To study train quantized models from scratch and compare them the QAT degradation, baselines. We find that optimizers performing well in the to our original pretraining setup may not remain optimal under QAT, and that models trained with Shampoo show the lowest accuracy degradation. Finally, we derive scaling laws for quantization-awan training under different optimizers, showing that Shampoo achieves the highest parameter efficiency of all tested optimizers.
interactions remains
INTRODUCTION
Large language models (LLMs) based on the Generative Pretrained Transformer (GPT) architec- ture have billions of parameters, and this number continues to grow, making both training and de- ployment increasingly challenging. Quantization is a common technique to make the training and deployment of large-scale neural networks more efficient. In general, LLM quantization can be categorized into two types: Post-Training Quantization (PTQ), where model is trained in full precision and then quantized, and Quantization-Awar Training (QAT), where quantization is in- corporated directly during training. The effectiveness of both methods ultimately depends on how well models can maintain accuracy the face of muantizationinnduce degradation.
Post-training quantization (PTQ) is one of the most important techniques to address the challenges of LLM deployment. To accelerate the inference, both weights and inputs (also known as "activation") quantized, enabling most computations to be performed in low precision. This technique is known as joint quantization (Ashkboos et al., 2024; 2023; Xiao et al., 2023; Liu et al., 2024). Several studies (Dettmers et al., 2022; Wei al., 2022; Bondarenko al., 2023) have shown that joint quantization is challenging due to the presence of outlier features (or OF) in the input matrices. To mitigate OF, various approaches have been proposed, such as reducing kurtosis (Nrusimha et al.,
"Equal contribution.

---

Published as a conference paper at ICLR 2026
0.82
0.70 Arger
60:0
Avg. Zero Shot Aee (%)
07.5-
00.0-
PRGD
PSCD
80.0
Maoo
Moo
47.8-
SQ4P
S0AP
Figure 1: Accuracy correlation with different metries for the 760M model. Traditional outlier- sensitive metrics like MMR (Left) and kurtosis (Center) show little to no correlation (measured by p) with model accuracy, whereas proposed metric (Right) correlates strongly with the model's zero-shot performance. MMR and kurtosis are computed row-wise, on the output of the last trans- former block.
2024; Akhondzadeh et al., 2025) or the Max-Median Ratio (MMR) (He et al., 2024) prior to PTQ, often through techniques like rotations (Ashkboos et al., 2024) or architectural modifications (He et al., 2024). However, it remains unclear how the optimization process during pretraining affeets the presence of such outlier features in the resulting model.
Quantization-awarr training (QAT) is another important technique for reducing the deployment cost of large language models (LLMs). Unlike post-training quantization, QAT applies quantization during training, allowing the model to adapt Ashkboos et al., 2025; Xi et al., 2024). Typically, the forward pass is quantized while the back- ward pass remains in high precision, with the gradient of the rounding function estimated using the Straight-Through Estimator (Bengio et al., 2013). Most existing works adopt the same optimization process as full-pree full-precision training, and Adam remains the most commonly used optimizer. Yet, the impact of using different optimizers on the final performance of QAT remains unclear.
-ww-pcciiionn computations (Panferov et al., 2025;
Over the last decade, Adam (Kingma & Ba, 2015) and Adam (Loshchilov & Hutter, 2017) have become the default optimizers for LLM training due to their simplicity and stability. At the same time, several alternative optimizers, such as PSGD (Li, 2015), Shampoo (Gupta et al., 2018), Muon (Jordan, 2024), SOAP (Vyas et al., 2025), and Scion (Pethick et al., 2025), have been pro- posed. Recently, concurrent studies (Wen et al., 2025; Semenov et al., 2025) have benchmarked optimizers in practice by training models with different hyperparameter settings to evaluate their empirical benefits. These studies are restricted to high-prec nigh-precision models, and despite advances in both optimization and quantization techniques, their interaction remains underexplored.
In this work, we investigate the gap between optimization and quantization of LLMs for efficient de- ployment. For post-iraining quantization (PTQ), investigate the question: "Do two models with the same validation loss but trained with different optimizers perform similarly under PTQ?"" To answer this, we train the same model to the same validation loss using different optimizers and compare the accuracy drop after quantization. We then evaluate the impact of different optimizers during quantization-awwe -waree training (QAT), asking: "How sensitive is QAT to the choice of opti- doier--dess the optimizer that performs best in full precision maintain its advantage under QAT?" Finally, we explore "How well do our findings transfer to larger models?" and provide scaling law for QAT performance across different optimizers. Our contributions are as follows:
1. We present the first systematic study of the interaction between optimization and quanti- zation, by training models ranging from 50M to 1.5B parameters with six different opti- mizers, and evaluating full-pre full-precision training (FP), post-training quantization (PTQ), and quantization-awarl training (QAT) schemes.
2. For FP training, we sweep over different hyperparameters for each optimizer-model com- bination, and train each using the Chinchilla-optiaal data-to-model ratio. We find that Muon outperforms other optimizers across nearly all model sizes.
3. For PTQ. we find that full- full-precision accuracy does not correlate with quantized recovery, nor with standard metrics outlier quantification, challenging current understanding. In- stead, provide first theoretical analysis of error propagation during quantization, and
2

---

Published as a conference paper at ICLR 2026
propose a metric for predicting the accuracy of quantized models. In particular, demonstrate that models trained with Shampoo are more robust to quantization.
4. For QAT, we again observe that optimizers performing best full precision are not neces- sarily optimal under QAT. Among all optimizers, the model trained with Shampoo yields the lowest accuracy degradation compared to its high-precision counterpart almost all of the model sizes. We further provide scaling law for different optimizers under 4-bit QAT and show that Shampoo achieves the highest parameter efficiency, supporting the transfer- ability of results to larger scales.
2 BACKGROUND AND EXPERIMENTAL SETUP
In this section, we first present the model architecture used in experiments and the optimiz- ers applied during training. We then describe the hyperparameter tuning strategy employed in our experiments. Finally, we provide the training and evaluation details for both PTQ and QAT.
Model Architecture. We use the OLMo2 architecture (OLMo et al., 2024), which integrates several recent architectural improvements. particular, OLMo2 combines no biases, rotary positional em- beddings (Su et al., 2024), RMSNorm (Zhang & Sennrich, 2019), reordered pre-normalization (Liu et al., 2022; OLMo et al., 2024) and QKNorm (Henry et al., 2020). Our only modifications on top of OLMo2 are the tying of the input and output weights, and the use of the ReLU² activation func- tion (So et al., 2022). We train models with 50M, 125M, 350M, 500M, 760M and 1.5B parameters, and we list the exact architectural details in Table 1.
Optimizers. We evaluate six optimizers. AdamW serves the standard baseline for training LLMs. To capture curvature-aware methods, we include PSGD and Shampoo. addition, we also consider Muon and Scion, as methods with motivations rooted in feature learning (Yang al., 2023). Lastly, we include SOAP, which operates by rotating gradients from and to a different eigenspace. We provide complete analysis of the computational complexity and memory overhead of each optimizer for a linear layer, with details provided in Appendix A.5.
Experimental Setup and els in 6 with the following protocol for hyper-parameter selection:
Protocol. As a full- full-precision baseline, we train mod-
1. Optimizer Parameters. We initialize each optimizer with hyperparameters recommended in its original paper and perform sequential one-dimensional sweeps to determine the op- timal settings. These sweeps conducted on the smallest 50M model, and the resulting configurations are applied across all experiments.
2. Learning Rate Selection. After fixing the optimizer hyperparameters, we sweep over eight different learning rate values and fully train each model-optiminee pair to find the optimal learning rate. For the 1.5B model, we select the optimal learning rate for each optimizer based results from the 760M model, and then test four additional values (two higher and two lower) to identify the best learning rate. We train our 1.5B model using Adam (as the baseline), Muon (the best-performing optimizer without quantization), and Shampoo (the most effective optimizer for preserving accuracy under quantization).
Datasets, Training Steps, and Evaluation Taks. We train all models on the ClimbMix dataset (Diao et al., 2025), a high-quality mixture of 400B tokens, following the Chinchilla training regime. For each model size, we define the Common Loss (or CL) as the lowest evaluation loss achieved by all optimizers using their best hyperparameters with at most a 20x token-to- parameter ratio. This CL checkpoint then used for applying the PTQ scheme. For each model podeel-ottimizer pair, we select the best hyperparameters from high-precision experiments and perform quantized training with fixed 20x token-to-pann -arrmmetrr ratio. We evaluate our models on three standard zero-shot benchmarks that offer early signal: PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), and ARC-Easy (Clark et al., 2018). We report the average accuracy across these tasks. All evaluations are performed using the LM Evaluation Harness (Gao et al., 2021) with default parameter settings. We always use 8196 samples from the test set ClimbMix dataset to calculate the statistics, each with 1024 tokens.
token-to-parameter
3

---

Published as a conference paper at ICLR 2026
Model
Exact Num. Paramelers
Blocks (L)
Q
heads
KV heads
Width (4)
Toks
Iters.
CL
50M
49,748,760
2
768
18
2,000
3.73
121,813,686
3.32
350M
325,335.892
12
4
1536
6B
12,000
2.97
500M
477.024,972
12
4
1536
10B
20,000
2,80
760M
729.974.655
15
5
1920
148
28.000
2.75
1.5B
1,489,423,554
25
18
6
2304
30B
60,000
2.57
Table I: Training configurations used for our experiments. CL denotes the common loss, which is the lowest validation loss that all optimizers can achieve for a given model size and number of tokens (used for the PTQ experiments).
Quantization. We apply the round-to-nearest-ineee scheme for PTQ experiments. For each ten- sor, every row is normalized by its largest magnitude value and then scaled by the largest rep- resentable number, a method known as symmetric AbsMax row-wise quantization. We use 4-bit quantization for both the weights and inputs of all linear layers, while keeping the remaining mod- ules in their original precision. For QAT, we train our models using QuEST (Panferov et al., 2025), tatee-oftthearr stable quantization-awan the forward pass, QuEST applies the Hadamard transform both the inputs and weights of linear layers, followed by the selection of an optimal clipping ratio to minimize MSE before quantization. During the backward pass, masks gradients that deviate significantly from the clipping value. We keep the backward pass (and non-linear modules) in the original precision following standard QAT.
training scheme for very low precisions (e.g., 4-bit). In
RESULTS AND ANALYSIS
In this section, we present our main results, analyses, and findings from the experiments. We begin with full-precision training, which serves as the baseline for our study and show that the outlier- related metric MMR correlates with learning rates across different optimizers. Next, we report the results of post-training quantization (PTQ) and provide a theoretical analysis error propagation using our ABC decomposition framework. Finally, present the results of our quantization-awan training (QAT) experiments, along with the corresponding scaling laws.
3.1 FULL-PRECISION
Table 2 shows the zero-shot accu- racies of our (unquantized) models across different optimized learning- rates. Except for the 50M and 125M models, Muon consistently outper- forms other optimizers in zero-shot accuracy, aligning with the concur- rent study of Wen et al. (2025). The performance gap widens for larger models (from 0.01% % in the 350M to 1.03% in I.SB model), indicating that Muon is the best optimizer in high precision, when hyperparameters are tuned. We provide additional re- sults Appendix A.I.
Model Size
Optimizer
50M
125M
350M
500M
760M
1.5B
AdamW
43.75
48.64
56.58
60.39
63.90
67.93
Muon
45.03
49.62
58.08
61.86
64.63
69.19
PSGD
45.08
49.95
55.99
60.85
61.47
N/A
Scion
45.41
49.83
57.53
61.68
63.82
N/A
Shampoo SOAP
44.81
49.53
56,51
61.03
63.05
68.16
45.73
50.24
58.07
61.34
62.27
N/A
Table 2: Average zero-shot accuracy (t) for the full- precision models, trained with BFloat16. Results for the test loss, which show similar trends, can be found in Table 5.
We also investigate how varying the learning rate impacts both the validation loss and the MMR, which is used to study the outlier patterns, in Figure 2. Our analysis reveals clear trend: increasing the learning rate consistently leads to higher MMR values, independent of the optimizer used. This suggests that larger learning rates tend to amplify the relative difference between the maximum and median value in the input tensor, potentially indicating larger outliers. Furthermore, across all optimizers in our experiments, Muon consistently achieves the lowest MMR. We provide results for other models in Appendix A.1.
4

---

Published as a conference paper at ICLR 2026
Adarn
Muon
x367
PSGD
X10
Vilideioon Loss
3.0
1100
4.0
2.9-
0.00
56-
-30
MMRR
2,82
2.8
0.75
0.70
0.11
2.7
288
101
10
10
10
10'1
10
Scion
Shampoo
SOAP
Vakkaaion Less
0.15
3.0
244 2.775
1.0
1.6
3.00
2.9
1. 2.700
MMR
2.85
12
2.8
0.5 2725
0.0
2.7
2.00
Loarning Rate (x)
Leurnnng Rate (1)
Loarning Rate (2)
Figure 2: The effect of changing learning-rate 7 on the validation loss and MMR in different opti- mizers for 760M model. We average the MMR over the rows in the input tensor of the last linear layer (before head). 3.2 POST-TRAINING QUANTIZATION
In the previous section, showed that models trained with different optimizers yield substantially different MMR values. We would thus expect models with higher MMR (larger outlier features) to exhibit severe degradation than networks with lower MMR. Here, examine what extent this intuition holds in practice. To facilitate comparison, we train all models to a common loss (CL), defined as the lowest validation loss achievable by all optimizers, given the same number of tokens. Since all the networks were trained to the same CL, the downstream performance before quantization very similar. Hence, instead of measuring the performance degradation, we can equivalently look at the final downstream performance.
According to our results, presented in Table 3, the optimizer that consistently leads to the least degraded networks, at least for model sizes above 125M, is Shampoo. This is counter-intuitive, since Shampoo's 's networks are also characterized by the highest MMR. Following folklore knowledge, this would imply that it would deteriorate the most under quantization. Also noteworthy is the fact that optimizers with low MMR (e.g. Muon), that would be expected to perform near the top, show a significant decline in accuracy. Plotting the row-wise MMR of the output of the last transformer block against the average downstream accuracy after PTQ, as in Figure 1 1-left, reveals that the two metrics are uncorrelated. Changing the metric use to quantify outliers from MMR to kurtosis, as in Figure 1-center, does not change the pattern.
To understand this discrepancy, introduce a new metric, which not only correlates strongly with the PTQ performance, but also allows us to track the quantization error through the network, as well as decompose into independent and interpretable terms.
Model Size
Optimizer
50M
125M
350M
500M
760M
1.5B
AdamW
40.23
45.15
49.23
55.17
59.22
62.51
Muon
41.88
45.23
47.42
50.60
50.00
47.75
PSGD
42.69
47.39
50.09
54.01
52.11
N/A
Scion
42.44
46.04
49.80
52.14
53.74
N/A
Shampoo
40.84
45.68
53.93
55.65
59.26
63.88
Theoretical analysis. We start with a brief description of the setup for which our analysis works, and present a concise summary of the main results. A full derivation can be found in Appendix A.4.
SOAP
43.36
46.35
49.08
50.91
46.22
N/A
Table 3: Average zero-shot accuracy (f) after we applied PTQ (row-wise W4A4) to the networks with the same common-loss (CL). The pre-quantization models used here have the same validation loss (shown in Table 1). Thus, the post-quantization performance is indicative of the degrada- tion due to quantization. The equivalent table for test loss is in Appendix A.2. GPTQ (Table 7) and Llama 2 (Table 8) have similar results.
We consider a Feed-Forward Net- work (FFN) with L modules fe(-), for é E (1....,J.. We denote the ac- tivations of the network with he E R"e, for which h fe(he-1), with
ho being the imput of the network. Notably, this construction allows for any arbitrary transformation fe(.) (e.g. linear layers, convolutions, self-attention etc.), at any level of granularity.
5

---

Published as a conference paper at ICLR 2026
R;
Ad
Muuor 0.56))
PSGD R
1.5
1.0
1.0
0.50
0.0
0.0
100
100
50
100
Sciuni Rg-0.2000
Shamppoo Ri 168)
COAP(R;=0099))
6-
1.0
4-
0..0
Figure 3: ABC decomposition for the 760M models. The x-axis shows the module index f. To use the stacked bar plot, and because Cy be negative, actually visualize CeJ. If Rg, which should track Ay Be + Cz. is below the green bar. it means Ce is negative, and the green bar only shows the magnitude. In the majority of cases, the biggest component A (accumulated error), followed by BN (layerwise error), followed by Ce (error interaction). The regular dips in the plots correspond RMSNorm layers, which attenuate the quantization error instead of amplifying
Now assume that we quantize the modules of the network. We can choose to quantize part or all of the modules, with any arbitrary PTQ scheme. If we denote the quantized modules with f? the quantized activations become he = f (h Thus, there is both a change in the input, from he-1 to 1 well change in the function, from fe(-) to (.) The change in the input h comes from the propagation of the quantization error through the previous - 1) modules, while the change in the function fe(-) is the newly introduced layerwise perturbation.
We aim to precisely quantify these changes. To do so, we study the difference in the activations
A A == hō hg. We prove (Appendix A.4) that this difference can be written as Ah; = ag + bb. with:
Change in the input under 12(-)
Change in the input under fe(-)
a₂ (52 (he-
(fe(h( fe(he-1))
Change in the functtin under h
Change the function h -
b. (h) fe
+ (f2 (h)-1)) (arni)))ff((aahh fe(he-1))
2
From the definitions, we see that ae the average of the change in the input under f (-). and the change in the input under f-(.). On the other side, by is the average of the change in the function under hg_,- and the change in the function under h Hence, a, captures the effect of the change in the input, while b, encodes the change in the function.
In order to reduce Ahe an interpretable number, we compute the Lo-norm. Since || A should be interpreted relative to the scale of the initial unquantized activations, we also normalize by |[hell- The norm of the relative change the inputs is then:
Ahell
la + bell
heel
To be able to use the Law of Cosines (which we use to prove the following decomposition), we instead study the square of this quantity Re == r?. Intuitively, this quantity tracks the deviation of the quantized network from the original one across its layers. Hence, we would expect R₁, at the output of the network' to correlate with loss degradation.
'Calculating Rr. requires four forward passes (one for each of fr((h,.)), (t. (h...))), f (h?_,)) and is thus more computationally demanding than MMR. However, also much more pre-

---

Published as a conference paper at ICLR 2026
Adamn
PSGD
Shampoo
Muoon
Seiio
SOAP
G: [won-linesr)
Gr (lineur)
G₁
1.0
10-
0.8
0.0
20
60
1000
10
30
10
20
Cos
Dos V/o
Gg1
0..75
5
0.25
0.00
0-
Figure 4: Gain decomposition for the 760M models with different optimizers. The x-axis shows the module index f. Reassuringly, the dips coincide with those of Figure 3.
For any module of the network, the square of the norm of the relative change in the activations R can be decomposed exactly into three terms (see Appendix A.4):
Re A
BB
+ Ce
a |
2
2
(ar: by)
A. =
Be = 12e
C=
||A,/112
Here Ae reflects the quantization eггоr accumulated from the previous (e- 1) modules, By expresses the quantization error introduced in the ê-th layer, and C₁ captures the interaction between the two.
To understand how a module transforms the quantization error of the previous layer from Re-1 to Ae, we can define the gain:
A
G.
R
In general, without assumptions about the functional form of the module, cannot further lyze G2 (even though we can use A, and R to calculate it). However, for a linear layer under weight and activation quantization with h (W, E/ >h² + ef, where ew and e are quan- tization noise introduced by weight and activation quantization respectively, there is a very intuitive decomposition of Ge. Specifically, we can write:
(
(
[I + 11.
2
G2. == cos Φe
G. GI. GI,e== =
Well
cos the
where de is the angle² between the change in activations A and the weight matrix after quan- tization, and bhy is the angle between the original activations and the unquantized weights.
The term GLa, which we call the "spectral ratio", reflects the change in the spectral norm of the weights due to quantization. We call the term G2,8 the "alignment ratio", since cos Φe E [0. 1] is the alignment between the change in the activations and the weights after quantization, while cos the E [0, 1] is the alignment between the incoming activations and the original weights.
To validate our analysis, we monitor Rg along the residual path of the transformer. We analyze the models at their most granular level, treating layer normalizations, residual connections, linear layers and activation functions separately, with the exception of the multi-head self-attention (MHSA) module, which consider single unit', In our framework,
Empirical Validation.
dictive of quantization degradation (see Figure 1). In our code, we include an optimized implementation which closely follows mathematical derivation of R;.
[Az]]] ). This non-standard "Here, define the angle between matrix A and vector arccos( [A||-|[>]]]
definition, that nonetheless find useful quantify how close maximizing |Ar|]. Notably, this angle
between and g. contrast to angles between vectors, which between 0
We do this simply hecause MHSA internally has three branches, keys, queries and values, making non- obvious what considered the residual path.
7

---

Published as a conference paper at ICLR 2026
the input hy Ro was assumed to be a vector. Because transformers process multiple vectors (corresponding to tokens), we need to use a summary statistic. As long as the statistic is linear, the ABC decomposition still holds with Stat(R,) otherwise noted, we use the average as our summary statistic
Stat(Ar) + Stat + Stat(Cz). Unless
First, we confirm our intuition that R2 should predict the degradation due to quantization in Fig- ure (rright). This suggests that Rg will be an informative quantity about how the quantization error propagates through the network.
Figure 3 shows the ABC decomposition for the 760M models. Here, we make two observations. First, in nearly all the cases, Rs is dominated by Ae, while Bg and C2 play minor role. That is, even if MMR is a good predictor of the quantization error introduced by quantizing a specific layer, the total quantization error, represented by Rg. mostly determined by the amplified error from the previous module, represented by A Second, networks trained with different optimizers exhibit very different error propagation behaviors. Even though all the networks ex- hibit oscillatory patterns with the Rs generally increasing with depth, some optimizers (e.g. Adam W, Scion) lead to networks with prominent spikes toward the end, while others (e.g. PSGD, Muon) have a "flatter" profile.
Finally, we measure the gain and its related quantities in Figure We separately show the gain of non-linear modules, and the gain of the linear layers, which we analyze to G1, and G2.-- In addition, we show the alignment factors cos Φe and cos t/y. We notice that, in the non-linear modules, different optimizers produce very similar G2 profiles, with the exception of Shampoo, which exhibits not only two upward spikes but also a sharp downward drop. In contrast, the patterns observed for linear layers are more distinguishable for different optimizers. Notably, Muon has the highest gain for linear layers, possibly explaining its sharp quality degradation despite that it has relatively low MMR. On the other hand, AdamW, along with Shampoo, have the lowest gains.
As for the decomposition of the gain for linear layers, we see that the spectral ratio is close to 1 for all optimizers. That is, the spectral norm of the quantized weights is not significantly different than that of the original weights. Hence, G2 is almost entirely determined by G2./. Looking at cos dz, we see that for Shampoo, and even more so for Adam the change in the activations less aligned with the quantized weights. The other optimizers mostly follow the same trends. Lastly, the factor cos the is similar across most optimizers with the exception of Adam W, for which the input activations are even less aligned with the weights.
3.3 QUANTIZATION-AWARE TRAINING AND SCALING LAW
Finally, we study quantization-awee aware training (QAT), where we empirically investigate the extent of performance degradation under QAT compared to full- cull-precision training, and how strongly this degradation depends on the choice of optimizer. We then provide a scaling law for QAT experiments to examine the transferability of our results to larger-scale models.
We use the tite-ooftth-ar 4-bit QAT QuEST scheme (Panferov et al., 2025), where weights and activations are quantized to 4-bits on the forward pass using row-wise symmetric quantization. We train each model-optimizer pair under the same compute budget, following the Chinchilla-optimal ratio (Hoffmann et al., 2022), using the best fuel-precision (FP) hyperparame- ters. We report the QAT evaluation losses across model sizes and optimizers in Table 9, as well as averaged zero-shot accuracy results (together with relative differences to their FP-baseline) in Ta- ble 4. First, we see that the ranking of the optimizers differs from the full-precision regime in almost all of the model sizes: no single best-performing optimizer could be identified for quantization- aware training. However, Shampoo remains the most effective optimizer in minimizing accu- racy degradation during quantization-awar training. Optimizers differ in how much accuracy they lose compared to their full-precision baseline, which is why promising results of optimizer in FP do not always lead to its strong performance under QAT. Overall, we find that full-precision results do not reliably predict QAT behavior.
Empirical Results.
'Figure is the only exception where we use the truncated average (average after we ignore the top I 1% of the values) as summary statistic. Trunc (Re) and Avg( Ra) only differ non-negligibly for a single intermediate layer for Shampoo, where a spike in Avg( Re) would force us to use log-scale.
8

---

Published as a conference paper at ICLR 2026
Model Size
Optimizer
50M
125M
350M
500M
760M
1.5B
AdamW
43.37 (- (-0.87)
48.08 (- 1.15)
54.64 (-3443
60.07 (-0.53)
62.22 (-2.63)
66.82 (-1.63)
Muon
44.07 (-2.13)
48.33 (-2.60)
55.19 (-4.98)
61.05 (-0.73)
62.32 (-3.57)
67.08 (-2.11)
PSGD
44.33 (-1.66)
48.82 (-2.26)
51.25 (-8.47)
60.32 (-0.87)
60.50 (-1.58)
N/A
Scion
44.86 (-1.21)
48.20 (-3.27)
56.01 (-2.64)
60.72 (-1.56)
62.30 (-2.38)
N/A
Shampoo
43.80 222))
48.31 (-2.6))
53.02 2.6.44
60.80 (- 0.38)
62.76 (- 0.46)
67.34 (-1.20)
SOAP
44.41 (-2.89)
49.38 1.711)
56.61 (- 2.51) 60.58 (- 1.24)
61.79 (-777)
N/A
Table 4: Average zero-shot accuracy (1) for the models trained with 4-bit QAT via QUEST. We report the difference in accuracy relative to the full full-precision baseline in the brackets. We bold the best accuracy and the smallest degradation for each model size. The corresponding results for test loss are presented in Table Results for smaller data-to-model ratio can be found in Table 10.
Adam
PSGD
Shampon Pin =0 -0.89 0,879
SOAP
=0.863
Pen 0 0.852
#07399
=0.856
Puu 0.822
4.0
1816
Tost Loss
W4A4
3.5
3.0
2.5
10'
10a
10"
10
10a
10a
10a
10
10a
10a
10a
10a
N
Figure 5: Scaling Laws for each optimizer, for full precision (BF16) and QAT (W4A4). For each optimizer, we report the parameter efficiency of 4-bit QAT in the subplot title. Shampoo has the highest parameter efficiency, followed by AdamW.
Scaling Laws. To provide a comprehensive comparison of QAT performance, we employ scaling laws. For each optimizer, the scaling law takes the form of Equation (1) from (Hoffmann et al., 2022), where L is final test loss (cross-entropy), N model parameter count and is number of training tokens:
B D
L pN
+
+.
(1)
We fit distinct sets of hyperparameters for each optimizer. To account for the QAT results the per-optimizer parameter efficiency p. introduced by (Kumar et al., 2024), as an extra hyperpa- rameter in the law. By convention, we p 1 for full-precision training, and we denote pabit the parameter efficiency in QAT W4A4 regime. Intuitively, an optimizer with higher p has higher parameter efficiency under QAT. For example, a model of size N trained in 4 bits QAT performs equivalently to the model of size Paba N trained in full-precision.
Since our experiments use fixed - 20, the full form of the law would be under-determined, and we report the coefficients of a iso-compute law:
A'
p)a E.
Here, the parameters A' and a describe the scale and the curvature of the power law, and E corre- sponds to the irreducible error. We estimate parameters of the law with robust non-linear Huber loss (Ō 10-3) on (log N, log D, log L.) manifold, and report the values with confidence intervals for each optimizer, obtained by leave-one-out validaiaii,,,on, in Appendix A.6.
Figure shows our fitted curves for different optimizers. The single scalar Pabat serves a quantitative measure of robustness to quantization: a higher value means the quantized model retains a larger effective parameter capacity. Similar to our QAT results, we can see that among the optimizers evaluated in this study, Shampoo yields the highest Piba, indicating that it is the most resilient to quantization and preserves model performance more effectively. Also, the top optimizers in terms of parameter efficiency (Shampoo, AdamW, Scion) are also leading PTQ recovery (Table 3).

---

Published as a conference paper at ICLR 2026
4 CONCLUSION AND FUTURE WORK
In this paper, we present a systematic study of the role of optimizer choice in the accuracy drop during quantization. First, we establish full-precision baseline by training a single model with multiple optimizers in the original precision, tuned over eight different learning rates and optimized hyperparameters. We then apply post-training quantization, showing that existing outlier-centric metrics like MMR or Kurtosis, used in previous works, are not predictive of PTQ accuracy, and introducing a new metric that correlates well with the accuracy of the quantized model. Finally, we examine the impact of optimizer choice for quantization-awy aware training (QAT), and demonstrate that the optimizer performing best in higher precision may not remain optimal the quantized setting. We further support finding by deriving scaling laws for each optimizer under QAT.
There are still opportunities to further extend our study. We plan to investigate additional bitwidths (e.g., 8-bit and 6-bit) for QAT, as well alternative PTQ schemes, and to study the error propa- gation of these methods using our ABC decomposition framework from Section 3.2. Studying the error propagation of other 4-bit data-types, like micro-scaling format, is also interesting. More- over, deriving the gain decomposition analysis for other modules in the network (like self-attention) would be very natural extension of our work. Lastly, given that many optimizers (c.g., PSGD, Muon) internally solve a constrained optimization problem, by adapting the problem to also include a condition about the gain G3, so that the final weights propagate quantization error only mildly, one might be able to obtain quantization-frient optimizer.
ACKNOWLEDGEMEENT
This work supported by the ERC PSAP project (grant agreement No. 101002047), and un- der project ID "FlexLM: Efficient Targeted LLM Compression" " as part of the Swiss Al Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.
This work was supported under project ID A140 as part of the Swiss AI Initiative, through grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.
REFERENCES
Mohammad Sadegh Akhondzadeh, Aleksandar Bojchevski. Evangelos Eleftheriou, and Martino Dazzi. Kurtail: cantosis-based Ilm quantization. arXiv preprint arXiv:2503.0142 2025.
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang. Jie Ren, Torsten Hoefler, and Dan Alistarh. Quik: Towards end-to-end 4-bit inference generative large language models. Xixiv preprint arXiv:2310.09259, 2023.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 57:100223-100244 2024.
Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto Castro, Torsten Hoefler, and Dan Alistarh. Halo: iiltaamard-asssitee lower-precision optimization for 1lms. arXiv preprint arXiv:2501.02625, 2025.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirt-Fouuthh AAAI Conference Artificial Intelligence, 2020.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Remov- ing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 3675507-776755 2023.
10

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, tbs/1803.05457,, 2018. URL httht://ppppp sewanticschooooeooee org/ CorpusID. 3922816 6.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:
30318-30332 2022.
Shizhe Diao, Yu Yang. Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustcring- iterative data mixture bootstrapping for language model pre-training. arXiv preprint arXiv:2504. 2025.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor op- timization. In International Conference on Machine Learning, 2018. URL htttps://apt semant iww.nnicsschol icscholar.......o.o Boatholar..rg/Copppcscoo Iorgg//oorstD:::: 35850668..
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in transformer training. Advances in Neural Information Process- ing Systems, 37:83786-88378 838466 2024.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.04245, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henni- gan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2022.
Keller Jordan. Muon: An optimizer for the hidden layers of neural networks. https:// kel mellerjordan gi i wttpoot//////ttoottoo///s////o/t /mwon// 2024.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. In Proceedings of the 3rd International Conference Learning Representations (ICLR), 2015. URL https: // //arxiv. org/ abs 1112..1112 6980. Preprint available at arXiv:1412.6980.
Tanishq Kumar, Zachary Ankner, Benjamin Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, 2024. URL https: //tp:v/.a.iivvaa/aa org/ abs /2411 04330.
Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization the modular norm. Advances in Neural Information Processing Systems, 37: 73501-73548, 2024.
Xi-Lin Li. Preconditioned stochastic gradient descent, IEEE Transactions on Neural Networks and Learning Systems, 29: 1454444466, 2015. URL https: hess://pppi /api. semanticscholar.ooo CorpusID: 5011801.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12009-12019, 2022.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantiza- tion with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.

---

Published as a conference paper at ICLR 2026
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017. URL https: wemaattcsscoolar.ccoomm CorpusID: 53592270.
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.03605, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.0066 2024.
Andrei Panferov, Jiale Chen, Soroush Tabesh, Mahdi Nikdan, and Dan Alistarh. QuEST: Stable training of LLMs with 1-bit weights and activations. In Forty-second International Conference Machine Learning, 2025. URL /tt.tttttto/eeeeeeeeeetee
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Forty-second International Conference on Machine Learning, 2025. URL forum? 20mmmmdddd20mm22222
net/
Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv:2509.01440 2025.
David R. So, Wojeiech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoe V. Le. Primer: Searching for efficient transformers for language modeling, 2022. URL brg/abs/2100 08668.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Nikhil Vyas, Depen Morwani. Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //openreview.... net./ feree/ffrrie forum? i d= d=
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:1 75:12-2--1744 2022.
Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02044. 2025.
Haocheng Xi, Yuxiang Chen, Kang Zhao, Kai Jun Teh, Jianfei Chen, and Jun Zhu. Jetfire: Effi cient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403. 12422, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099, PMLR, 2023.
Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning, arXiv preprint ar 13..:2310.78//.. 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint ar500:190.007830 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation P Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
ADDITIONAL FULL-PRECISION RESULTS
Avg. Zero-Shot Acc.(1) TTest Loss(
Optimizer
50M
125M
350M
500M
760M
1.5B
A
43.75
3.695
48.64
3.318
56.58
2.961
60.39
2.834
63.9
2.744
67.93
2.627
Muon
45.03
3.632
49.62
3.263
58.08
2.915
61.86
2.804
64.63
2.719
69.19
2.612
PSGD
45.08
3.615
49.95
3.283
55.99
2.978
60.85
2.841
61.47
2.799
N/A
Scion
45.41
3.615
49.83
3.269
57.53
2.932
61.68
2.805
63.82
2.726
N/A
Shampoo
44.81
3.648
49.53
3.311
56.51
2.959
61.03
2.831
63.05
2.741
68.16
2.622
SOAP
45.73
3.589
50.24
3.241
58.07
2.916
61.34
2.803
62.27
2.754
N/A
Table 5: Results for the best models trained with B6 For each model, we show the average zero-shot accuracy (f) the left, and the test loss (4) on the right. As expected, the track each other closely. While SOAP does better for small sizes, Muon is consistently the best for bigger models.
A.2 ADDITIONAL POST-TRAINING QUANTIZATION RESULTS
Model Size
Optimizer
50M
125M
350M
500M
760M
1.5B
A
5,445
4.233
4.274
3.730
3,480
3.769
Muon
4.305
4.145
4.941
4.819
4.928
9.725
PSGD
4.102
3.704
3.948
3.878
3.776
N/A
Scion
4.116
3.803
3.951
4.471
3.759
N/A
Shampoo 4.731
4.786
3.246
5.219
3.217
3.639
SOAP
4.027
3.790
4.222
3.987
6.320
N/A
Table 6: Test loss (1) after applied PTQ (row-wise W4A4) the networks with the same common-loss (CL). The results mostly mirror Table 3, with Shampoo generally showing the mildest degradation. The exception 500M, where Shampoo exhibits noticeable degradation in loss, despite outperforming other optimizers in downstream tasks.
Model Size (GPTQ)
Optimizer
500M
AdamW
55.47
Muon
50.22
PSGD
53.73
Scion
52.21
Shampoo 55.84
SOAP
51.16
Table 7: Average zero-shot accuracy (f) after we applied GPTQ (W4A4) to the networks with the same common-loss (CL). The trends are similar to Table 3 (RTN).
13

---

Published as a conference paper at ICLR 2026
Model Size (Llama 2)
Optimizer
350M
AdamW
47.35
Muon
45,20
PSGD
48.01
Scion
47.77
Shampoo
50.86
SOAP
48.13
Table 8: Average zero-shot accuracy (1) after we applied PTQ (row-wise W4A4) to Llama net- works with common-loss 2.74. We see the same trends as with OLMo (Table 3).
A.3 ADDITIONAL. QUANTIZATION-AWARY TRAINING RESULTS
Model Size
Optimizer
50M
125M
350M
500M
760M
1.5B
AdamW W
3.757
3.375
3,009
2.905
2.787
2.655
Muon
3.698
3.340
2.971
2.868
2.765
2.651
PSGD
3.696
3.339
3.19
2.976
2.844
N/A
Scion
3.703
3.335
2.980
2.850
2.763
N/A
Shampoo 3.735
3.341
2.999
2.849
2.782
2.640
SOAP 3.682
3.302
2.981
2.842
2.797
N/A
Table 9: Final test loss (1) for the W4A4 QAT models. Comparing with Table 5, we see that Shampoo shows the smallest degradation relative to the full-precision baselines.
Model Size ( 5)
Optimizer
350M
AdamW
51.30 (-3.69)
Muon
52.75 (-5.05)
PSGD
49.68 (-7.88)
Scion
53.23 (-2.71)
Shampoo
52.51 (-2.54)
SOAP
53.45 (-2.66)
Table 10: Average zero-shot accuracy (f) for the models trained with 4-bit QAT via QuEST, and data-to-model ratio instead the Chinchilla-opiimal 20. Shampoo shows the smallest quanti- zation degradation, though it does not lead the best model at this scale (similarly to Table 4).
14

---

Published as a conference paper at ICLR 2026
A.4 ABC DECOMPOSITION
Consider a Feed-Forward Network (FFN) with L modules fe(:), for & {l,..., , L}. The output of the network [(x)) E IR for input R is:
hg = 2
h fe(he-1) for & f1....,,L)
f(x) = h2
where h2 R are the activations of the network.
The modules Si(-) can any arbitrary transformation, including linear layers, convolutions, acti- vation functions, layer normalizations and self-attention. Thus, the class of networks consider is quite general, and includes some of the most popular architectures, like Convolutional Networks (CNNs) and Transformers. Morcover, this framework is also flexible in terms of the granularity of the module partitioning. Even a module also consists itself of submodules, we can arbitrarily choose consider it as whole. Therefore, desired.
study the network in any level of granularity
Assume now that we "quantize" the modules, changing them from fe(-) to (.)()- There is again flexibility on what "quantize" can mean in this context. For example, we can arbitrarily choose to apply weight and activation quantization on linear layers, activation quantization on activation functions, and no quantization on layer normalizations. Other choices are also possible, as we treat "quantization" as a general perturbation in the function space.
After quantizing, the activations of the network become:
h
h (2 for E (I....,L}
(*(x) h2
By comparing the activations pre and post-quantization:
h; = fe(he-1) h2 = f2
see that the change in the output he is caused by the change in the input h (activation space), as well as the change in the function fe(.) (function space). The change in the input h comes from the propagation of the quantization eтror through the previous (E 1) modules, while the change in the function fe(-) is the newly introduced layerwise perturbation.
We wish to precisely quantify these changes. In order to do this, study the difference in the activations:
Ah: = h2 he (h fe(he-1)
By adding and subtracting either f (ha- or fe(h)-1). we can get:
Ah; (12 (2:)))(((hh0) (2(6))1)) (J2( - fe(he-1))
Ah; (13(h) fe(h2_1)) (Je(h) fe(he-1))
instructive to understand what the four terms on the right sides represent. The top left term stands for the change in the input under the function f-(-), the top right term for the change in the function under the input h2-1, the bottom left for the change in the function under the input h-_ and the last term for the change in the input under the function fe(-)-
Thus, the top equation implies that we first perturb fe(-) (-),,, and then h h On the other hand, the bottom equation implies that we first perturb the input, and then the function. Both equations valid, but reflect different assumptions about the order of the perturbations. Arbitrarily choosing one can lead to attribution bias, and thus, following the Shapley principle we average both
AAccording to the Shapley principle, the unigue way to distribute contributions fairly across two players (here: input function) is averaging the possible orderings. This is the only allocation scheme that satisfics the three axioms of symmetry, efficiency and lincarity.
15

---

Published as a conference paper at ICLR 2026
as follows:
Ah (1 ( (he--))) + +1Z((1 - fe(hee1))
(2 5102211) ((chh-))-e(hh--l))
2 (7 (h +))+
(f fe(h +ff((he11))) (h(-1)))
2
ar
with:
a₂ = ( 18(he-1)) (fe(h fe(he-1))
2 b 07 - f((h .)) + (ff(he-1)) - fc(he-1))
Hence, the change in the output Ah, == h h can be exactly decomposed into two terms. The first term, ae entirely captures the effect of the change in the input from h h h2-1 while the second term by completely encodes the change in the function from fe(-) f2)
However, Ahs, az, by are all still vectors. In order to reduce them to interpretable numbers, we compute the Lg-norm of Ahe. Specifically, since || should be interpreted relative to the scale of the initial unquantized activations, we normalize with respect to |hell. Thus, we study the L2 -nom of the relative change the activations:
A
|a, + bell
|a, bell
=
r
heel
hell
121 1
By setting R² = r and using the Law of Cosines:
||ae|2 + 2(as, be) + |1bel
R₂
hhel?
||pell 2
+ (ap, bb)
|
+
A + Br + Cr.
where:
|ar|]
A₂
|
||bel| 2
Bj
|
(aa, by)
C
||/2||12
with A standing for the quantization error accumulated from the previous (e (6-1) 1) modules, By for the new quantization error, introduced the f-th layer, and C₁ for the interaction between the two.
A quantity of interest that helps us understand how module amplifies or attenuates the quantization error from the previous layer the gain:
A R
G
'We can also use the "more natural" RMS-nom norm (as in Large et al. (2024)), which is normalized with respect dimension, allowing us to compare numbers across networks with different widths. In the end, these two choices are equivalent, because ultimately investigate ratios of Tg-norms, which are equal to ratios of RMS-norms.
16

---

Published as a conference paper at ICLR 2026
Ge determines how the quantization enror of the previous layer, expressed by RE-1, propagates through the E-th module to appear as Aq.
In general, if we treat the module though we can calculate it from A₃ and Ry-1). However, for specific functional forms of fe(-) we exactly recover G in closed-form. The most straightforward example is the case of linear layer under weight and activation quantization.
black-box, cannot further analyze the gain Ge (even
Specifically, for linear layer' under joint quantization:
h fe(he-1) W,he-
h2 = (2 (We+ e 1 E
where e\" and af are quantization noise introduced by weight and activation quantization', respec- tively.
Moreover:
(he-)))) (We+ + + 1he-1 + E fe(h)_1)= Wch
By using the last four equations:
(1))1 (ha-1 (Wr +E() Ah
fe(h)_1) fe(he-1) W
19(h-_)) Se(h2_1 = h221 + E
(hee-) fe(he-1) = h + ex
In turn, by replacing those into the definitions of ag and be we obtain:
1
(We + =5((
)
(hr-1 12-1
bt eW
E
2
Finally, by using the definition of As:
|az||
A
heel
(W
+ Je ah
) )
Wheee-ll
We + _ |+ Abs cos de
2 (ll ||+ denotes the spectral norm)
[Well||heezll cos the
W, JeW 11. 1b)
(1
(1Ahe-ill) |
2
cos di cos vhs
Well
Gr,
G. = Gr,
We assume no bias for ease exposition (this also coincides with our architecture). 'Here, without loss of generality, assume quantize the activations after the matrix multiplication. Quantizing before the matrix multiplication would mean h (Wz+ (h²_1 (We+el" }hū (We + E ) which would lead equivalent analysis, since
the last single quantity.
17

---

Published as a conference paper at ICLR 2026
where we defined:
I(W; + jE, )
Cos Φt =
We SE, |1 1h/-1
(Wehe-ll
the
Well- Vell-[[he-11
I + 2e:"1l1 [1:))
Gi,e
("Spectral ratio")
[Welll
COS tE
G2 =
("Alignment ratio")
cos thy
The angles Oi and the are between matrices and vectors. This is a non-standard definition, but quite intuitive nonetheless. The angles lie between 0 (cos /e / 1) and 90 (cos / 0), and indicate how close is the direction of the vector to the direction of the top right singular vector of the matrix. An angle of 0 means that the vector points towards direction that is maximally stretched by the matrix. In other words, the vector is colinear with the top right singular vector. In contrast, an angle of 90" indicates that the vector and top right singular vector are orthogonal, and that the vector is being shrunk to O.
The angle de is between the change in the activations Ah;-1 and the weight matrix after quantiza- tion (Wr }= (the 1/2 comes from Shapley). The angle t is between the incoming activations h and the unquantized weight matrix W. and denotes how aligned are the incoming activations with the weights. The "spectral ratio" G1 is the (squared) ratio of the spectral norms of weights before and after quantization. Finally, the "alignment ratio" G2.8 just encodes the (squared) ratio of the alignment between the change in the activations and the weights after quantization, and the alignment between the incoming activations and the original weights.
To sum up, for linear layer under joint quantization, further factored the gain G. of quantization error into product of two interpretable terms, the spectral ratio G1,t. and the "alignment ratio" G2.6- We can do a similar analysis for the terms Be and Cz. which we omit here because we found that Re is dominated by As (see Figure 3). A similar-style analysis can also be done for other modules, like layer normalizations and self-attention, by using Taylor's theorem, but we leave this for future work.
A.S OPTIMIZERS' MEMORY AND COMPUTATIONAL COMPLEXITIES
Table 11 shows the asymptotic memory (including gradients) and computational complexities for one step of gradient update of hidden layer with weight matrix W E R
Optimizer
Memory Overhead
Computational Overhead
A
3mn
(mm) O 2nm2² m²)), m S 0 m +n + 2(m + n)mn) ( + m²)), m S n ((m² + n² n²) + () (m n))m) O(mm + + )1 + }) + 2(m + m)mm)
Muon
2mn
PSGD
mn + m² + n
Scion
2mn
Shampoo
3mn + m² + n2
SOAP
3mn + 2 + n?
Asymptotic memory (including gradients) and computational complexities per one opti- mizer step for linear layer of size m x n. Here denotes the number of Newton-Schulz iterations in Muon algorithm, and f is the preconditioner update frequency for PSGD, Shampoo, and SOAP.
Table 11:
Recall that multiplication of 2 matrices AB, where A Rmnk B E R requires mnk multi- plications and mn(k- 1) additions, which totals in mn(k k 1) 2mnk operations.
18

---

Published as a conference paper at ICLR 2026
Iteration time (s)
Memory (GB)
Optimizer
50M (128)
125M (64)
350M (64)
500M (32)
760M (32)
1.5B (16)
A
0.66
63
1.37
41
3.07
66
4.51
48
6.28
57
12.89 66
Muon
0.70
63
1.41
3.11
66
4.59
48
6.35
57
13.09 61
PSGD
0.80
63
1.70
41
3.49
66
5.12
50
7.08
63
N/A
Scion
0.71
63
1.51
3.14
4.63
48
6.55
57
N/A
Shampoo
1.30
67
2.18
48
4.34
83
6.43
72
8.38
88
16.38 91
SOAP
0.82
63
1.52
42
3.34
66
5.02
58
7.02
77
N/A
Table 12: Measurements for full-precision training GH200 96GB GPU. Inside the parentheses next to each model size, we list the maximum batch size that works with all optimizers, also used for the measurements. AdamW and Muon have similar requirements, despite Muon leading to the strongest models (see Table 2). Even though models trained with Shampoo show the mildest degradation (see Table 3), they also take the longest to train, while also requiring the most memory.
1. AdamW
Gradients + and moments 3mn memory. The number of moments and weight update operations is linear with respect to mn, the number of elements in the weight matrix.
2. Muon
Gradients (mn) and the first moment (mn) give the total memory overhead of 2mm.
Suppose < n, one step of the Newton-Schulz iteration for matrix X E R=xx then becomes
X' = aX ax b(Xx) +x
(3)
which gives O + 2m2n + 2m + 22m2n) = (22m2 m²) time complexity, For the total of T iterations it then becomes O + m²)).
3. PSGD
Gradients (mm) + preconditioning matrices (m m and n x n) give mn + m2 + n² total memory.
Applying preconditioners PL E Rmxm and PR E o both sides of the gradi- ent Pr.9PR uses 2m²n. + 2mn² operations, and the preconditioner update has computa- tional complexity (mm + n²). Amortized by update frequency f. the total complexity is O22m2n 2mn² + m m²²n")
4. Scion
Gradients (mn) and the first moment (mn) give the total memory overhead of 2mn. Scion uses the same Spectral LMO as Muon, which results in O(T(2m²n + m²)) com- plexity overhead.
5. Shampoo
Memory overhead consists gradients, 1st and 2nd moments (3mn), left and right pre- conditioners and eigenvector matrices (2m² 2n²).
Updating the preconditioners takes O(m² + n²)), and applying them additional O(m²n + mn²) operations. Preconditioner change with frequency f adds extra O m m complex- ity. In practice 10 100, so other terms dominate.
6. SOAP
The same as for Shampoo, the total memory complexity is 3mm + 2m + 2n²
The time complexity same as for Shampoo with extra O(m2n + mn²) for projecting back from the eigen-space. Recomputing the eigenbasis Qz. Qn with frequency adds extra O t ? complexity. In practice f 10 100, other terms dominate.
19

---

Published as a conference paper at ICLR 2026
A.6 SCALING LAWS FITTING PROCEDURE
To estimate the parameters of scaling law for each optimizer. we perform nonlinear least-squares regression using scipy. wooyyleasee lewsstssuaasss. To improve robustness against outliers, we employ the Huber loss with parameter = 10-3. We minimize the residuals between the predicted and observed log-loss values, fitting flog N,, D.. Pb.) to log Li, where N; is the model size, D, the dataset size, and b, is the precision. The loss values are transformed into logarithmic scale prior to fitting for numerical stability. For the total parameter count N, we include embedding parameters, which we found improves the fit. We report the values with confidence intervals, obtained by leave- one-out vross-validatio,, Table 13.
We estimated the confidence intervals of the fitted hyperparameters by computing their standard deviations across folds in a leave-one-out cross-validation procedure. We found that excluding the loss value corresponding to the largest model size provides unstable fit, retained the lowest loss point for all the folds.
Optimizer
A'
E
Pabit
AdamW
79
0.20+0.01
1.40=0.05
0..03
Muon
208
0.27 40.01
1.85
00.02
PSGD
77
0.1840.05
1.39-0.41
0.739+0.010
Scion
148
0.25
1.75
0.856.1..00
Shampoo
142422
0.2440.01
1.72=0.09
0.879
SOAP
706
0.35
2.22=0.03
0.822+0.010
Table 13: Scaling law coefficients for different optimizers.
20


Reviews:
[{"review_id": "kQBaGHo7p1", "reviewer": "Reviewer_f8Mk", "summary": "This paper addresses a critical gap in the intersection of large language model (LLM) optimization and quantization: how optimizer choice impacts model performance when quantization (a key technique for efficient LLM deployment) is applied. It systematically investigates two major quantization paradigms—Post-Training Quantization (PTQ) (quantizing a fully trained model) and Quantization-Aware Training (QAT) (integrating quantization during training)—across LLMs of varying sizes (50M to 1.5B parameters) trained with six optimizers (AdamW, Muon, PSGD, Scion, Shampoo, SOAP).\n\nThe study first establishes well-tuned full-precision (FP) baselines using the OLMo2 architecture (modified with tied input-output weights and ReLU² activation) and the Chinchilla-optimal data-to-model ratio. It then evaluates how these FP models degrade under PTQ, and how optimizers perform when training quantized models from scratch via QAT. A key theoretical contribution is the ABC decomposition framework to analyze quantization error propagation, which reveals limitations of traditional outlier-centric metrics (e.g., Max-to-Median Ratio (MMR), Kurtosis) in predicting PTQ performance.", "questions": "In your full-precision training, you use the ClimbMix dataset (400B tokens) but do not analyze how data domain diversity (e.g., mixing technical vs. conversational text) interacts with optimizer-quantization performance. For instance, if a subset of ClimbMix has more outlier-prone features, might this disproportionately favor Shampoo? Did you conduct any ablation on dataset subsets to rule out data bias in your optimizer comparisons?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. As the first work to rigorously explore optimizer-quantization interactions, it covers a broad scope—models from 50M to 1.5B parameters, 6 optimizers (AdamW, Muon, etc.), and both PTQ/QAT paradigms. This breadth avoids narrow conclusions and ensures results are tested across diverse scenarios.\n\n2. It establishes robust full-precision baselines using the Chinchilla-optimal data-to-model ratio, detailed hyperparameter tuning (e.g., sequential sweeps on 50M models), and clear architectural modifications (OLMo2 with tied weights/ReLU²). Evaluation uses standard zero-shot benchmarks (PIQA, HellaSwag) and open tools (LM Evaluation Harness), ensuring reproducibility.", "weaknesses": "1. Your paper focuses on validating optimizer-quantization interactions using the OLMo2 architecture (with modifications limited to tied input-output weights and the ReLU² activation function), which provides a clear and controlled experimental base. We’re curious, have you considered extending these experiments to more mainstream LLM architectures like Llama or GPT-2 (or even Llama 3, which is widely used in industry)? We’d also be interested to hear the reasoning behind choosing OLMo2, and how you anticipate Shampoo’s quantization advantages might hold (or adapt) when tested on these other architectures, to further strengthen the generalizability of your conclusions.\n\n2. The proposed \\(R_\\ell\\) metric effectively addresses the limitations of traditional metrics like MMR and Kurtosis by capturing layer-wise activation error propagation, which is a valuable theoretical contribution. We noted that computing \\(R_\\ell\\) for large models (e.g., 1.5B parameters) requires iterating through all layers’ activation tensors, leading to higher computational complexity than MMR/Kurtosis. In real-world deployment, users often need fast quantization outcome predictions (e.g., during preprocessing). We’re wondering if you have considered ways to optimize \\(R_\\ell\\)’s computational cost, such as approximating it with key layers rather than all layers, and how you balance the metric’s predictive accuracy with the industry’s demand for low-cost tools?\n\n3. Your QAT experiments use the QuEST scheme, which is a strong choice for stable low-precision training. Given that industry frequently adopts other QAT variants like GPTQ and AWQ (which differ in handling quantization noise and model adaptation), we’re curious if you’ve thought about how Shampoo’s performance might compare to other optimizers under these alternative schemes? We also wonder if testing a broader set of QAT methods could further validate whether Shampoo’s advantages are specific to QuEST or generalizable across common QAT paradigms.\n\n4. Your scaling laws leverage Chinchilla’s full-precision optimal data-to-model ratio, which provides a consistent baseline for comparison. We note that quantization can alter a model’s information capacity, for example, 4-bit models may require more data to offset precision-related accuracy loss. We’re interested to hear your thoughts: how do you think reusing the full-precision data ratio might influence the assessment of Shampoo’s parameter efficiency? And would recalibrating the data ratio specifically for QAT be a valuable next step to confirm if Shampoo’s efficiency stems from inherent quantization robustness, rather than compatibility with full-precision settings?\n\n5. You rightly point out that MMR and Kurtosis are limited by their focus on single-layer errors, which motivates the need for \\(R_\\ell\\). We’re curious, have you explored the possibility of refining these traditional metrics (e.g., a weighted MMR that assigns different importance to attention vs. FFN layers) to better capture cross-layer error effects? If an improved MMR showed stronger correlation with quantization performance, how do you think it might complement (or compare to) \\(R_\\ell\\), and could there be synergies between refining existing metrics and advancing new ones like yours?\n\n6. Your conclusion thoughtfully highlights that optimizers performing well in full precision may not retain that advantage under QAT, a key insight for practical LLM deployment. We’re wondering if you’ve considered discussing the potential design direction of optimizers that excel in both full precision and QAT (e.g., quantization-adapted variants like a modified Lion optimizer)? We’d be interested to hear how such dual-excellence optimizers might align with your current findings, and whether you see this as a valuable direction for future work to build on your conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ly58K6ZDLm", "reviewer": "Reviewer_7Mqb", "summary": "This work explores the impact of quantization on various optimizers. This work presents a comprehensive empirical analysis of how optimizers fare before and after quantization. The authors dispute that using outlier-related metrics like max-to-mean ratio (MMR) of input tensors is a useful indicator of quantization performance. Instead, this work focuses on studying the relative change in activations as a more expressive quantity to observe. The experiments are performed on recent class of transformer-based models, using a wide array of recent/popular optimizers. This work concludes that based on their experiments Shampoo optimizer fares best with the lowest accuracy degradation.", "questions": "See points under weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The stated goals of the work, of assessing the impact of quantization on different optimizers, and formalizing the accuracy degradation using better metrics than MMR is interesting. \n* The work makes a compelling case showing that the focus on outliers is not useful to predict accuracy degradation. The formalism of relative change in activations $R_\\ell$ as a proxy for total quantization error is convincing.\n* Experiments are comprehensive; provide solid empirical evidence for the claims in the paper.", "weaknesses": "* **Outlier-centric motivation:** One of the main motivation of this empirical study is to show that outlier-based metrics are not predictive of accuracy degradation due to quantization. However, the authors do not point to any relevant literature or the general consensus in the community that argues this is the case. In my opinion, the consensus is on straight-through-estimator (STE) when using quantization aware training (QAT) [1], and uniform quantization that are the more widely cited reasons [2]. Can the authors point to the specific literature that claims outliers as the problem? \n\n* **Motivation for pursuing gain**: The point above also leads to my next point. While empirical results show that relative change of activation, formulated as spectral ratio and alignment ratio, are more indicative of performance degradation, what is the theoretical justification for this? For instance, QAT has mechanisms to overcome these activation rescaling as it is performed during training, and has the opportunity to compensate for these fluctuations. Are there other subtle mechanisms that this work is identifying?\n\n* **Oscillations in Figures 3 & 4:** The oscillations in Figure 3 and more pronounced in Figure 4 warrant some attention. What is going on in Fig. 4? Why are $G_\\ell, R_\\ell$ oscillating between regular values between successive layers? This seems to be the case for all optimizers. \n* **Point of Scaling laws** While the analysis and the fit for the scaling laws are interesting, I was confused as to how this ties to the rest of the paper. Is it to show that $\\rho$ for Shampoo is slightly higher than AdamW? What else do we learn from these scaling curves? \n\n* **Use of OLMo2 architecture:** The experiments are all performed on the same architecture family. The experiments are across different number of parameters, but the results and observations in this paper might be overfitting to a single architecture family. What is the justification? And would the authors expect these observations to generalize to other architecture families?\n\n* **Optimizer hyperparameters:** It is mentioned that all hyperparameters for the optimizer were chosen based on the 50M model. Were these done in any consistent manner? For instance, following the muP recommendations?[3] If not, what is the argument for doing this?\n\n### Other comments\n\n* Two versions of the same paper, Panferov et al. 2025, are referenced\n* Difficult to see what is going on with the components of $R_\\ell$ i.e, the ABC decomposed terms. \n* MMR abbreviation is introduced several times.\n* Column 2 in Table 1 is redundant, given the first column.\n\n### References\n\n[1] Huh, Minyoung, et al. \"Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Oh, Sangyun, et al. \"Non-uniform step size quantization for accurate post-training quantization.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhoc0KKHWI", "reviewer": "Reviewer_Ag8F", "summary": "The authors study the interaction between quantization and choice of optimizers for both PTQ and QAT scenarios. The authors train BF16 models with various optimizers and perform PTQ to measure the degration. The authors then propose a new metric which correlates well with the PTQ performance compared to commonly used metrics like MMD and Kurtosis. The authors then study the impact of optimizer choice during W4A4 QAT and show that the choice of optimizer does not transfer directly from high precision to low precision.", "questions": "- Why is the OLMo2 architecture changed to incorporate $\\text{ReLU}^2$?\n- Any reason for not using MuP for hyperparameter selection?\n- Is it reasonable to expect the optimal hyperparameters to be same for both the full-precision and low-precision runs?\n- What happens if we use fancier PTQ schemes compared to Round-to-nearest? Is the result still the same?\n- \"Shampoo’s networks are also characterized by the highest MMR\". Figure 2 suggests it's PSGD?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is a great paper. Extremely well written and addresses important questions still unresolved in the community.\n- Compares models using downstream performance rather than a proxy metric like validation loss.\n- The differences in error propagation through the network with various optimizers was an interesting observation.", "weaknesses": "- Only RTN PTQ scheme is studied, which is not usually used in practice.\n- For QAT, microscaling formats like MXFP4/ NVFP4 would perhaps be a better choice\n- Due to lack of using hyper parameter selection techniques like MuP, the current hyperparameter choices might be suboptimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)