Training Dynamics Impact Post-Training Quantization Robustness
Authors: Albert Catalan-Tatjer
Niccolò Ajroldi
Jonas Geiping
Venue: ICLR 2026 Poster (2026)
Keywords: Efficiency
quantization
optimization
Primary area: optimization
TL;DR: Post-training quantization robustness is tightly coupled to learning-rate dynamics rather than data scale. We suggest practical ways to modulate quantization performance outcomes.

Abstract: While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct a comprehensive analysis of quantization degradation across open-source language model training trajectories up to 32B parameters and 15T training tokens to accurately assess the relationship between training dynamics and quantization performance. Our key finding is that quantization errors in large-scale training runs are driven by a complex interplay between learning rate and other training hyperparameters. Specifically, once learning rates decay, validation loss and quantization error diverge, largely independent of training data scale. To investigate interventions on the training dynamics and identify specific configurations that can modulate quantization robustness favorably, we train our own models in controlled experiments up to 100B tokens. Our results challenge the assumption that increasing dataset scale inherently compromises quantization effectiveness, demonstrating instead that strategic training hyperparameter interventions can improve quantization quality at scale.

Published as a conference paper at ICLR 2026
TRAINING DYNAMICS IMPACT POST-TRAINING QUANTIZATION ROBUSTNESS
Albert Catalan- -
Niccolò Ajroldits
Jonas Geiping
'ELIS Institute Tübingen
Max Planck Institute for Intelligent Systems & Tübingen Al Center
$OpenEuroLLLM
ABSTRACT
While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct comprehensive analysis of quantization degradation across open-source language model training trajectories up to 32B parameters and 15T training tokens to accurately assess the relationship between training dy- namics and quantization performance. Our key finding is that quantization errors in large-scale training runs are driven by a complex interplay between learning rate and other training hyperparameters. Specifically, once learning rates decay, validation loss and quantization error diverge, largely independent of training data scale. To investigate interventions on the training dynamics and identify specific configurations that can modulate quantization robustness favorably, we train our own models in controlled experiments up to 100B tokens, and analyze how the loss curvature evolves and interacts with the learning rate during training. Our results challenge the assumption that increasing dataset scale inherently compromises quantization effectiveness, demonstrating instead that strategic training hyperparameter interventions can improve quantization quality at scale.
INTRODUCTION
Deep learning has already entered the low-bit era (NVIDIA, 2025). This transition has been enabled by specialized hardware support and algorithmic innovations, with quantization serving as the core technology driving low-precision workloads. Modern neural networks surprisingly quantizable, and even modern large language models (LLMs) trained over trillions of tokens in 16 and 32 bits of precision be quantized into zoo of low-bit formats, leading widespread adoption through- out the entire model deployment workflow, and large interest from both hobbyists and model service providers. In the following we will denote this workflow as post-training quantization (PTQ).
Generally, quantization maps models trained with high-precision formats to lower-precision repre- sentations. Common strategies to preserve performance involve scaling (Xiao al., 2024), rotating (Ashkboos et al., 2024), grouping (Lin et al., 2024), or indexing in codebooks (Tseng et al., 2024). GPTQ and AWQ (Frantar et al., 2023; Lin et al., 2024; Tseng et al., 2024) unlock low-bit primitive throughput and memory gains during inference not only through strong quantization strategies, but also through specialized kernels that support fast inference on quantized models. However, despite the widespread use of PTQ in all layers of the community, from model providers to practitioners, there is still a limited understanding of the principles that govern the brittleness of quantization, i.e. the ease with which different models can be quantized and what error rates to expect. Recent efforts to study quantization in Kumar et al. (2024) and Ouyang et al. (2024) suggest that PTQ becomes less effective for LLMs training progresses, arguing that the number of training tokens relative to model size is a central factor in quantization sensitivity. Consequently, as datasets inevitably grow severe, ultimately questioning
larger (Brown al., 2020), they expect degradation to become whether post-training quantization remains viable for future models. However, we find these results overlook key piece of the puzzle: the influence of training dynamics on the ease of quantization.
"a www.er..ctaa aebert.ccaallnnttaajrrrttttt
su

---

Published as a conference paper at ICLR 2026
1e
1e-6 4
1e-4
1.75
2.0
2.9
2.0
Quantization error
1.50
1.5
Leariin rate
Velidation loss
1.5
Learning rate
1.25
2.88
1.0
0.5
2.7
0.5
0.75
Stable Decay
0.0
PTQ decay
0.50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number training tokens
1e13
Number of training tokens
1e13
(a) 4-bit quantization error vs training tokens.
(b) Validation loss vs training tokens.
Figure 1: Evolution of quantization error and validation loss during training of Sm (Bakouch and validation loss throughout training under both the constant (r) = 2e up to 10T tokens) and annealing phases the learning rate schedule (whose evolution is shown as dotted lines). As the learning rate decays, validation loss consistently decreases, whereas quantization error rises sharply and to much greater extent than at any carlier point in training.
al., 2025). We report quantization
While Ahmadian et al. (2023) showed that large activation outliers can be controlled with weight de- cay to improve PTQ performance, the effect of training hyperparameters on quantization quality has been difficult to study, since open-weight, releases typically provided only single checkpoint (Tou- vron et al., 2023), offering no insight into training details or into the trajectory of quantization error during training. However, with the recent surge of open-source large language models (LLMs) (Biderman et al., 2023; Groeneveld et al., 2024; OLMo et al., 2025; Bakouch et al., 2025), which vary substantially training design and learning rate configurations, we now have access to much richer data to study this question in detail. Open-source model training runs document number of hyperparameter choices, but how these choices affect quantization rarely discussed.
In this work we provide a systematic study of the post-training quantization error across training stages for six modern, open-source LLM training efforts. While previous work has studied quantization degradation in controlled settings or for short training runs below 300B tokens, we include trajectories of open-source L.LMs of up to billion parameters trained on up to 15 trillion find that the actual hyperparameter choices taken by model
tokens. Through this investigation, trainers play a larger role in quantization error than previously expected. Training our own models, verify the effect of learning rate scheduling and weight averaging PTQ error in controlled studies, and provide actionable suggestions to intervene on quantization. In summary,
We measure quantization error across hundreds of intermediate training checkpoints from major open-source LLM families and correlate quantization eπrо trajectories with training stages and learning rate schedules in Section
In controlled experiments in Section 4, we verify that quantization error is modulated by learning rate schedule. Maintaining larger learning rates, all else being equal, reduces quantization error. Informed by these findings, we show in Section 5, that, for own training runs, lower quantiza- tion error can be achieved by optimized learning rate schedules, and how weight averaging along training trajectories can be used to improve quantization performance.
Finally, in Section 6, we analyze the geometric properties of the loss suggesting that the proposed interventions interact with quantization performance via the promotion of flatter minima.
Through systematic investigation and concrete examples, we highlight that training hyperpa- rameters, and the resulting training dynamies significantly change how easy it is quantize modern LLMs. We argue that studying PTQ continuously during pretraining, and especially during hyperparameter selections before large-scale runs, should be an essential step, as we identify several cases, in which, for example two learning rate choices seemed equally promising, but choosing the smaller one, did lead to an increased quantization error down the line.
BACKGROUND AND RELATED WORK
2.1 POST-TRAINING QUANTIZATION
Post-training guantization methods reduce the memory required run large neural networks by reducing their numerical precision. However, as LLM inference is dominated by auto-regressive decoding, which is in turn limited by memory bandwidth (the rate at which model weights can be
2

---

Published as a conference paper at ICLR 2026
transferred to accelerator's 's compute units, e.g. streaming multiprocessors on GPUs), quantization can often improves the speed of the model.
The most naive quantization method is to simply cast all floating-point parameters of the model to the desired precision. More advanced algorithms, such BNB, AWQ, GPTQ (Frantar et al., 2023), optimize which parts of the model to quantize and by what approach to minimize errors, when quantizing weights, activations and VV-ccche. In particular, for linear layer with weights W, let X denote the input and We the quantized low-pree oo--recision weights derived from W by some method. During inference, Wo is loaded onto the GPU and the matrix multiplication (GEMMs) is performed with the dequantized weights W such as XW.T. For weight and activation quantization, the input X is also quantized. Modern mixed-precisio kernels fuse the dequantization and multiplication steps for efficiency. Initially, quantization methods would aim to minimize the weight error ||W - WIll (Courbariaux et al., 2016); however, more recent approaches minimize the reconstruction error ||XT - X- The latter methods require a calibration dataset to compute X at quantization time, several other variants exist (Frantar et al., 2023; Lin al., 2024; Tseng et al., 2025)
Most quantization approaches build upon variations of these core concepts (Vanhoucke & Senior; Jacob et al., 2017; Tseng et al., 2024; Dettmers et al., 2022: Ashkboos al., 2024): high-precision auxiliary states, such as scaling factors, to map between the dynamic range of original tensors and that representable in low-precision; dividing the quantization problem into smaller groups of typically 128 weights; processing outliers that would affect the dynamic range of the group with different strategies. While numerous quantization techniques exist in the literature, we focus our analysis on GPTQ (Frantar et al., 2023) quantization at 3- and 4-bit precision levels. However, our supplementary experiments demonstrate that AWQ (Lin et al., 2024) and AndBens (BNB) Dettmers et al. (2022) quantization methods exhibit analogous trends, as detailed in Appendix A.
2.2 LLM TRAINING HYPERPARAMETERS
Large-scale pretraining of neural networks, such as language models, is dependent on a large number of hyperparameter choices. We review here some fundamental elements of the pretraining pipeline, as we later show they are linked to quantization error and can be exploited to modulate it.
A key aspect of optimization is the choice of learning rate schedule. Whereas earlier language model training largely relied on cosine decay schedules (Loshchilov & Hutter, 201 more recently model builders have shown increasing interest in the trapezoidal schedule (Zhai al., 2022; Hu et al., 2024), also known as Warmup- Staale-Daacee (WSD). This scheme splits training into a constant learning rate phase followed by a linear-decay stage, enabling training across different compute budgets with significantly fewer resources (Haegele et al., 2024) and has hence seen growing adoption (Bakouch et al., 2025; Nezhurina et al., 2025; Apertus Team, 2025). Alongside the scheduler shape, the peak learning rate (LR) itself is arguably one of the most important parameters for final model performance (Tissue et al., 2024) and training stability (Wortsman et al., 2023). Together with the peak LR value, the value after annealing can also impact perfor- mance (Bergsma et al., 2025), scaling law derivation (Li et al., 2025) and adaptability to supervised finetuning (Singh et al., 2025). Overall, many design choices remain somewhat arbitrary, frequently guided by heuristics (OLMo et al., 2025) and often yielding equivalent results when sufficiently tuned (Haegele al., 2024). In this work, we argue that one additional line of analysis should be robustness to quantization, as the interplay between these variables and PTQ degradation reveals underexplored design decisions and a path for guiding future choices.
2.3 MODEL BRITTLENESS TO POST-TRAINING QUANTIZATION
How well will a certain quantization algorithm work for given, already trained, LLM, and does this depend on the size of the model, or the amount of training data? Recently Kumar et al. (2024) and Ouyang et al. (2024) developed scaling laws for quantization error, in which they relate the scale of training dataset with the degradation induced by quantization. In summary, they reach a similar conclusion, models are trained more data, they exhibit higher quantization induced degradation. However, scaling up the training dataset is one of the primary levers to improve model performance, and small overtrained models are becoming increasingly popular (Gadre et al., 2024).
3

---

Published as a conference paper at ICLR 2026
1
0.10
OLMo-18
Base
Quantization Error
0.08
OLMo-7B
Base (Model Soup)
Learning Rate
OLMo-138
Base (Model Soup)
0.06
OLM
Base (Model Soup)
0.04
0.02
3 Number of training tokens
5
6
le12
Figure 3-bit quantization error along the training trajectories of OLM22 models. Error grows gradually during cosine decay but spikes under the steep linear decay phase. Model souping (x) reduces degradation, achieving lower PTQ error than any individual run.
Yet, these studies overlook the role of the training dynamics in model robustness to post-training quantization. fact, we find that on open sourced LLMs, quantization degradation abruptly increases as learning rates decays, regardless of training data size. In Section 4 we investigate these contradicting results and we find that their characterization of the effect of training dataset scale and quantization performance is mostly confounded by the learning rate hyperparameters used in their experiments. Overall, we identify this gap in the literature and address this crucial question: what is the relationship between the training dynamics and quantization performance?
3 POST-TRAINING QUANTIZATION OF MODELS IN THE WILD
In this section, we amalyze training trajectories of the following models: OLMo model family (IB, 7B parameters; trained on 2.5T-3T tokens) (Groeneveld et al., 2024); OLMo2 family suite (IB, 7B, 13B, 32B; 4TT-6TTT (OLMo al., 2025); SmolLM3 (3B, 1 11TT) (Bakouch et al., 2025); Apertus (8B, 15TT) (Apertus Team, 2025); Open-science (1.3B, ITT) (Nezhurina et al., 2025), for which we consider the Nemotron-cc release (Su al., 2025); and Amber (7B, 1.3TT) (Liu al., 2023). We use GPTQ (Frantar et al., 2023) to post-train quantize them to 3 and 4 bits. We detail the quantization process in Appendix A, and share the complete set of results for all model families in Appendix B.
We evaluate PTQ robustness by first examining quantization error in validation loss and later by assessing its impact on downstream tasks.
3.1 QUANTIZATION-
DEGRADATION ON VALIDATION LOSS
To more accurately represent the intuition that increases in cross-entropy loss are more expensive the lower the cross-entropy is (as loss decrease is roughly logarithmic in compute), we show relative
CE(W) cross-entropy loss, defined as (CE(W)
We decouple the effect of learning rate decay from the amount of training data consumed, we first focus on models trained with a Warm up-Stable-Decay schedule. We begin by examining Figure 1a, which shows quantization error alongside the learning rate during the training trajectory of SmolLM3. We observe that, while quantization eтror increases rapidly in the beginning of training. stays relatively constant during the 11 trillion tokens of stable phase, and only as the learning rate decays does quantization error spike. Figure 1b shows how the validation loss follows albeit inverse-curve than that of the quantization error. Similarly, OpenSci training runs from Nezhurina et al. (2025) in Figure 11 display analogous pattern: quantization error surges sharply as the learning rate decreases, for the different models on vastly different token budgets.
Next, we consider the OLMo2 model family, which includes four language models with 1, 7, 13, and 32 billion parameters, all developed using a consistent training methodology. Training occurs in two phases: initial general pretraining phase using 4-6 trillion tokens with cosine learning rate decay. followed by a second phase that applies a short and sheer linear decay schedule across different orders of high-quality data configurations, also referred "ingredients". The final
4

---

Published as a conference paper at ICLR 2026
18-4
Validation Loss Degradation
2.0
2,0
1.50
Accuracy Degradation
Decay
1.5
1.5
1.25
Long Context
Mid Training
1.00
1.0
SFT
0.75
APO
1.0
0.5
Main
0.50
0.0
0.5
0.0
0.25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
of Training Tokens
1e13
Number Training Tokens
1e13
(a) 3-bit validation loss degradation.
(b) 3-bit accuracy degradation.
Figure 3: 3-bit quantization effects across SMoILM3 post-training stages. Degradation in validation loss (left) and downstream accuracy (right) show that PTQ effects differ across stages and appear sensitive post- training interventions. The final model. weighted average mid-training and APO, shows better robustness than both individual components.
model weights are obtained through model souping (Wortsman et al., 2022), averaging models trained with different ingredients, except for the 1B parameter model, which retains weights from single decay trajectory. Figure 2 presents quantization eтror and learning rate trajectories for the four models. The quantization error shows different trend across the two phases, increasing gradually during slow cosine decay, but rising sharply under steep linear annealing. Although the learning rate itself may not directly cause this degradation, this observation once again suggests deeper connection between optimization dynamics and quantization performance. Finally, report the quantization eгтоr for the model soup, and find that averaging substantially reduces degradation, with the model soup achieving lower PTQ error than any of the individual ingredients. We will return to this observation later in Section and 5.
3.2 QUANTIZATION-INDAE INDUCED DEGRADATION ON DOWNSTREAM TASKS
While cross-entropy loss serves convenient proxy for model performance, downstream eval- uation better reflects the practical utility model. Following OLMo et al. (2025), we evaluate performance across 12 established benchmarks and report the average 5-shot accuracy across all tasks (see Appendix for additional details on the evaluation pipeline).
we show the performance degradation induced by 3-bit quantization SmolLM3. Alongside the validation loss (Figure 3a), we present the relative accuracy drop, defined as Acc(W)-Aco(Ht> 1-Acc(W) (Figure 3b). Despite fluctuations, similar pattern can be identified both curves: performance degradation increases as the learning rate decays. We observe similar results across individual tasks and report them in Appendix D (Figure 17, Figure 18).
In Figure 3
Modern LLMs are optimized beyond general pretraining stages to promote alignment, extend con- text, incorporate supervised fine-tuning, and perform instruction tuning (Tie al., 2025). Here, we study the effect of quantization across post-pretraiiing stages. In SmolLM3, these include long context training, a mid-training phase incorporate general reasoning capabilities, super- vised fine-tuming (SFT) for domain-specific skills, and anchored preference optimization (APO) (D' O'Oosterlinck et al., 2024) to promote alignment. Finally, the released (main) model linear merge with weights of 0.9 and 0.1 of the APO model and a middtrriinng checkpoint. Figure 3 reports the performance degradation under 3-bit quantization after each stage in SmolLM3. Interestingly, context extension sensibly reduces quantization degradation, while mid-training largely amplifies it. PTQ degradation then decreases through SFT and APO. Remarkably, although the main model obtained by averaging the mid-training and APO weights, it exhibits lower quantization degradation than either of them individually. We recall similar results from the previous analysis on OLMo-2 data mixtures exhibited lower quantization degradation than
(Figure 2), where model soups any of the individual components. These results suggest that averaging benefits quantization, a novel finding investigate further in Section
5

---

Published as a conference paper at ICLR 2026
1e-3
1e-3
3.0
Decay
2.5
4.0
Stable
2.5
2.0
Learning rake
Validation loss
Quartization
6-
1.5
3.8
1.5
4.
1.0
3.6
0.5
0.5
0.0
3.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
D.4
0.6
0.8
1.0
Number training takens
1el1
Number of training takens
lel1
(a) 4-bit quantization
training tokens.
(b) Validation loss vs training tokens.
Figure 4: 4-bit quantization error at different training durations. We use WSD, training a 160M-parameter
transformer up 100B tokens and performing additional cooldowns at 12B, 28B, 46B, 64B, 82B tokens. Fig- 4a shows quantization error during training with different token budgets, and Figure 5b the corresponding validation loss. Despite varying the amount of training data, runs show comparable quantization error after cooldown, highlighting that error spikes are associated with training dynamics rather than token budget.
4
CONTROLLED EXPERIMENTS
4.1 REPLICATING THE OBSERVED PHENOMENA
To understand the insights from Section 3, we conduct pretraining experiments with transformer models on smaller scale, varying token budget, learning rate, LR schedule, and weight decay one time. We follow Biderman al. (2023) for model specifications, and FineWe (Penedo et al., 2024) as pretraining corpus (see Appendix C for details on the training procedure and hyper- parameters). We use GPTQ, and discuss results for additional quantization methods in Appendix A.
In Figure 4 we show quantization error and validation loss across range of token budgets, which we obtain by decaying the learning rate at different steps during training. We observe that the constant learning rate stage is not immune to PTQ degradation, showing a slight increase in quantization error. At the same time, despite training durations ranging from 10B to 100B tokens, models achieve comparable quantization error after decay, which spikes learning rate decays and validation loss drops, regardless of than token count. In Figure 21 we replicate the experiment using a cosine decay schedule, where model performance (Figure 21b) and quantization robustness (Figure 21a) vary with the training horizon. However, changes in the peak learning rate, and thus the scheduler shape, have larger impact, in some cases yielding improved quantization error at lower validation loss.
In conclusion, this evidence suggests that the phenomena observed in Section are not merely serendipitous outcomes of complex model interactions, but strongly shaped by training dynamics, with factors such as learning rate decay playing a key role in quantization performance.
4.2 SCALING TRENDS PRIOR WORK ARE DOMINATED BY LEARNING RATE SCHEDULES
In an effort to explain the rise of quantization eтror during training, previous studies attributed this phenomenon dataset size or training duration, concluding that PTQ degradation increases as models are trained on more data (Kumar et al., 2024), and hence that quantized undertrained models scale more favorably (Ouyang et al., 2024). We argue that these works did not sufficiently control for a key confounder, namely the optimization dynamics induced by the learning rate schedule, which we find to be the primary driver of their observed degradation.
Specifically, we replicate analyses from Kumar et al. (2024) in Figure 5, training models at dif- ferent token budgets under both original cosine schedule and WSD schedule. While cosine results (bluc) suggest that åPTq increases noticeably with token budget, we show that comparable WSD schedule (brown) can yield lower validation loss, with degradation growing more slowly (70M) or remaining stable (160M), indicating that the effect cannot be ascribed to data alone (see also Fig- ure 21 for a similar conclusion).
Finally, argue for additional caution when collecting checkpoints at different token counts, as done in Ouyang et al. (2024). We recall that similar considerations have been discussed in the scaling law literature: Hoffmann al. (2022) suggested that their power law discrepancy with Kaplan et al.

---

Published as a conference paper at ICLR 2026
70M parameters
160M parameters
70M parameters
160M parameters
Validation Loss
4.25
bit WSD
decay
Degradation Byro
4.00
bit Cosine decay
10-1
WSD
3.75
1
3.50
10"
10a 10
10
10"
16
10" Tokem/Parameter ratio
Token/Praameter
Token/Paramette
TakeroParameterrato ratio
(a) Validation loss,
(b) 0p CE(W) - CE(WW.
Figure 5: Learning rate affects quantization scaling trends. Following Kumar et al. (2024), we train 70M and 160M transformer models with cosine decay across different token budgets, and a WSD schedule under the same model configurations. Cosine decay replicates prior results, with êpTQ increasing at larger token budgets, while WSD shows slower growth at 70M and increase at 160M, hinting that other factors beyond data volume influence quantization scaling.
(2020) arose from differences in learning rate schedules, and further works validate the importance of collecting checkpoints only after learning rate annealing (Haegele et al., 2024). We suggest that the same discretion necessary when deriving scaling laws for quantized models, as optimization dynamics influence observed robustness (Figure 4).
INTERVENTIONS ON THE TRAINING DYNAMICS
Having explored the connection between training dynamics and quantization degradation we inves- tigate how simple interventions can modulate PTQ robustness and achieve better quantized models.
5.1 LEARNING RATE
In Figure 6, we demonstrate how different peak learning rates impact quantization. Figure 6a shows that higher learning rates consistently lead to smaller errors, with curves inversely ordered by rate magnitude. Figures 6b and 6c report full-precision versus 4-bit and 3-bit quantized validation losses. These parametric curves capture quantization error relative total validation loss: perfect quantization would lie on the x = y bisector, with deviations measuring the error. Comparing curves with LR 1 and 3e-3 shows that, at similar validation loss, the larger rate achieves better low-bit quantization, at no apparent cost. This suggests that, for comparable full full-precision performance, em- ploying a larger learning rate might be preferable, as it enhances low-bit quantization performance. We replicate this experiment on a 300B token pretraining run of OLMo2-7B in Figure 23.
Learning rate schedules designate the magnitude of the learning rate throughout training, repre- sented as dotted lines in Figure 22a. On one hand, while the cosine schedule (green) has much higher peak learning rate, profile is dominated by the one of WSD decay phase (yellow and blue). Despite this rapid decay. the cosine schedule still achieves lower quantization error and better validation loss than the WSD schedule. This indicates that quantization performance depends on training dynamics beyond just the learning rate magnitude at any single point. On the other hand, examining 3-bit quantization in Figure 22c reveals that cosine schedules experience sharp upward curvature near the end of training. likely due to very small learning rates in the final steps. This suggests that cosine schedules' inability to control end-of-training learning rates, where the rate becomes small regardless of the initial peak, may hurt quantization performance compared to schedules like WSD that maintain better control throughout training.
Ie
1e 2
1.0
3.60
3.9
Quantization Errer
1.5
0.8
Val. Los Quantized
3.8
3,50
3.7
1.0
3.6
3.45
Lielc 5
0.5
3.5
3.40
3.4
0.25
3.40
3.45
3.60
1el1
Val.
Baseline
(a) 4-bit quantization error.
(b) FP to 4-bit validation loss.
(c) FP to 3-bit validation loss.
Figure 6: Larger learning rates lead to lower quantization error. Figure 6a displays the quantization error achieved by fixing the training recipe and varying the learning rate. We observe that quantization error decreases when employing higher learning rates. Furthermore, Figure 6b and 6c show that, similar validation loss, larger learning rates achieve better low-bit quantization, at no apparent cost.
7

---

Published as a conference paper at ICLR 2026
1.0
1.0
1,0
Velication Los
4.0
LAWA
Stable
Valication Locd
4.0
PTQ LAWA
PTO Stable
Quantizatio Eree
3 b in
Decay
PTO Decay
3.8
0.6
0.6
0.6
0..4
3.8
0.4
0.4
0.2
3.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Number tokens
1el1
Number tokens
1611
Number tolkens
Iel1
(a) FP validation loss.
(b) 3-bit validation loss.
(c) 3-bit quantization error.
Figure 7: Weight averaging as an alternative to LR decay for PTQ. Validation performance and quantization error for 160M model trained on 100B tokens at constant learning rate. We compare intermediate learning cooldowns with weight averaging of checkpoints collected from the stable phase. We report the validation performance of frel-precision model (Figure 7a), the 3-bit quantized model (Figure 7b), and their difference (Figure 7c). Whereas LAWA falls short of reaning-rrtee decay in the full- full-precision setting, its 3-bit PTQ performance yields lower validation loss than all cooldowns, demonstrating successful setting for LAWA.
5.2 WEIGHT AVERAGING
Given the encouraging results quantizing model soups in Section 3.1, and the detrimental effect of learning rate decay on quantization performance, a natural question is whether weight averaging could serve as an alternative and mitigate its negative impact'. Intuitively, averaging parameters along the training trajectory reduces noise and can approximate the effect of learning rate decay. Prior work derived equivalent averaging schemes for common LR schedules under SGD (Sandler et al., 2023), and later studies showed that averaging improves performance over constant learning rate training (Haegele al., 2024), though still falling short of LR decay. Nevertheless, its effect PTQ robustness remains unexplored, despite its simplicity, and compatibility with existing pipelines.
Therefore, pretrain 160M-parameter transformer on 100B tokens with constant learning rate and compare LAtest Weight Averaging (LAWA) (Kaddour, 2022) against several intermediate learning rate cooldowns, with averaging configuration described in Appendix C. As observed in prior work (Ajroldi et al., 2025), in the full- full-precision setting (Figure 7a), LAWA yields better check- points than constant learning rate but does not reach the performance of intermediate cooldowns. In contrast, for 3-bit quantized models (Figure 7b), we find that checkpoints obtained through weight averaging can match or even surpass the performance of those trained with learning rate decay.
Finally, we apply the same technique to training trajectories of open-source models. Specifically, we consider OLMo-1B (Groeneveld et al., 2024), averaging checkpoints during training and using LAWA aggregation scheme (Figure 24). Despite the lack of control over checkpoint saving fre- quency, the averaged model still improves upon the final one, performing better both in full- alllprecission and after quantization, confirming averaging promising direction improve PTQ robustness.
5.3 WEIGHT DECAY
Learning rate and weight decay are coupled popular Adam W implementations (Paszke et al., 2019). We analyze the impact of changing the weight decay À on the quantization error for fixed training recipe, with an implementation where learning rate and weight decay À are decoupled (Schaipp, 2024). In Figures 19b and 19c we observe that among models that achieve comparable performance (seen in the x-axis) in full-precision quantized validation loss, those with larger weight decay À exhibit lower 4- and 3-bit quantization error. This shows that, for À configurations that achieve comparable loss, higher values are preferable to reduce PTQ errors, which confirms Ah- madian et (2023) observations. Moreover, compared to Figure we see that changes have smaller effect on quantization enror than learning rate.
GEOMETRIC PROPERTIES OF THE LOSS
The findings presented in Section 5 reveal several important relationships between interventions and downstream performance, but is there an underlying, unifying mechanism? To investigate, we
We distinguish between model soups (Wortsman et al., 2022), which average models from different training runs, and weight averaging (Izmailov al., 2018), which aggregates checkpoints along single trajectory.
8

---

Published as a conference paper at ICLR 2026
analyze the geometric properties of the loss landscape to illustrate the interaction between these seemingly disconnected phenomena.
6.1 Loss LANDSCAPE
We visualize a 2D slice of the loss landscape (Goodfellow et al., 2015; et al., 2018) defined by three checkpoints of interest, Øx the model at the end of training, O the model at previous step of training, and θx, the model the end of training quantized. We refer to Section F for additional details.
Our goal is to analyze how hyperparameter decisions during pretraining result in different local neighborhoods around Øµ and Ô in the landscape of the loss via the 21) slice they span. In Fig- ure 8 we present four different landscapes, corresponding to pretraining our usual 160M parameter model with different learning rates, shown in Figure 6. In Figure 8, θx is the result of 4-bit GPTQ quantization, we refer to Figure 25 for analogous results on 3-bit GPTQ quantization. We begin by observing that, expected, the smaller the learning rate, the closer θx-1 and O are. Perhaps more interestingly, the distance between θx and θ follows the same trend, it is larger for larger learning rates. All the slices depict a local minimum around θx.
What is interesting is that y-axis, the quantization direction, to the x-axis, the direction to the previous optimization step. In this sense, the geometry of the quantized model seems closely related to the geometry induced by training. Furthermore, the learning rate magnitude is proportional to the flatness of the basin of the loss, where, even though θş and θх are closer for smaller learning rates, the sharpness of the basin is such that θx falls in a higher loss level, phenomenon which is exacerbated further for larger weight perturbations e.g. for even lower bit quantization Figure 25.
that in all examples, the landscape is structured similarly in the
6.2 CURVATURE
To better understand the topology of the loss landscape and the dramatic effect of learning rate decay quantization robustness, we further examine the second order information of the loss. We estimate the trace of the Hessian via Hutchinson estimator (Hutchinson, 1989), and the sharpness (maximum eigenvalue) via power iterations, using PyHess sieessiar (Yao et al., 2019). We refer to Appendix G for details on the estimation procedure and additional results.
In Figure we report the sharpness and trace evolution during the stable and decay phases when training a 160M transformer on 100BT. The maximum eigenvalue shows a consistent rapid surge whenever the learning rate decays. Although we also observe an initial increase in sharpness under a constant step size, a more detailed analysis shows a clear distinction between the two regimes: in the stable phase, only the top eigenvalue initially rises while the others remain small, whereas in the decay phase all eigenvalues increase, underscoring a notable difference between these training dynamics. The trace presents similar pattern, remaining stable under constant learning rate, and rising abruptly as it decays, remarkably mirroring the evolution of quantization error in Figure 4.
We visualize checkpoints that trained for 100 billion tokens during K 190000 steps. We the checkpoints every 2000 tokens, therefore 1 188000.
LR=300e-04
(iw1 00e-03
1R=3300e-33
(Rw 0,0000 02
Final
Ox-:
stop
6g quantized
Figure& Landscape the visualize the landscape of the plane spanned by the weights (Ox, Ox-1, 1, θx) for learning rates correspondigg the experiment in Figure 6. We observe that flatness of the loss basin proportional learning rate magnitude.

---

Published as a conference paper at ICLR 2026
Top Hessian eigenvalue
Estimated Hessian Trace
25
10-
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Number of tokens
1e11
Number of tokens
lel1
Eigenvalus rank
3e9 Tkns
5e9 Tkns
749 Tkns
9e9 Tkns
10e9 Tkns
11e9
13e9 Tkns
0 Gi
Stable
Decay
LAWA
20
20
20
20
20
20
Figure 9: Sharpness (top left), Hessian trace (top right) and first 25 eigenvalues (bottom) estimated on the training trajectory 160M transformer model (training runs in Figure 4). Sharpness consistently increases when the learning rate decays. Under a constant icarning rate, only the top cigenvalue briefly increases while the rest of the spectrum remains low; the second row shows the distribution during this early increase. The trace shows clearer trend, although it is confounded by being the sum of all eigenvalues.
Although learning rate dynamics are known to affect the Hessian spectrum in simpler settings (Co- hen et al., 2025), there is limited understanding of any causal structure in more complex training setups. Based on the observed phenomena, we hypothesize that, as the learning rate decays, the model traverses sharper region of the loss landscape, making more sensitive perturbations such as quantization.
Our analysis also indicates that averaging weights during training leads to wider minima, in line with Izmailov et al. (2019). Such improved conditioning of the Hessian might explain the superior quantization robustness of LAWA in Figure 7, but also offers new perspective on weight averaging: whereas prior work linked it theoretically and empirically to learning rate decay (Sandler al., 2023), we show that the two methods produce solutions with substantially different curvature properties. We believe that the improved quantization robustness of model soups in Figure 2 may be explained by similar curvature properties induced by souping.
Finally, the benefit of larger learning rates on stochastic gradient descent is well documented (Barrett & Dherin, 2020; Lewkowyez et al., 2020; Gilmer al., 2022), and it has been suggested that the additional noise leads to flatter minima, which should generalize better (Hochreiter & Schmidhuber, 1997; Chaudhari al., 2017), and require fewer bits be specified (Hochreiter & Schmidhuber, 1994). When considering training trajectories under different maximum LR (Figure 6), we indeed find that larger ones produce lower sharpness (Figure 26a) and smaller trace estimates (Figure 26b), suggesting the presence of flatter minima, yet interestingly also leading to lower quantization error.
DISCUSSION
We conduct systematic investigation of how training interventions affect quantization degradation in language models under controlled experimental configurations. First, we observe that the magni- tude of the learning rate determines quantization robustness when all other hyperparameters remain fixed. Second, we identify that averaging checkpoints, either across different data configurations via model souping or along the training trajectory. promotes robustness to quantization. These concrete examples, where quantization degradation noticeably shifts with training dynamics, lead us to advocate studying quantization robustness during routine hyperparameter tuning. We then study geometric properties of the loss to investigate how learning rate and weight averaging affect quantization performance, finding that these interventions coincide with convergence to flatter minima, which argue might benefit quantization robustness.
Overall, we end on an optimistic note. Our findings indicate that quantization degradation stems from an intricate relationship between training dynamics alluding to general model robustness. As a result, we find that, rather than being an unavoidable consequence of training data scale, can be acted upon with existing tools, which especially beneficial for low-bit quantization.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
JG acknowledges the support of the Hector foundation. JG and ACT acknowledge the support of the Amazon Science Hub Tübingen. This research was partially supported by the European Commission under the grant No. 101195233 (OpenEuroLLM)
REFERENCES
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization scale, 2023. URL https: / arxii.ivvvv...va org/ abs 2aass/3305 2305. 19268.
Niccolò Ajroldi. plainim: Language model pretraining in pytorch. https::/gitte com/ Niccolo-Ajrooddippla.nn 2024.
Niccolò Ajroldi, Antonio Orvieto, and Jonas Geiping. When, where and why to average weights?
In Forty-second International Conference on Machine Learning, 2025. URL https://
apenreview.nete wtt.://rrmmmiiieae (oorumiid--N80011222
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Concel-Kedziorsee Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v//N19 12245.... URL https: / www.lannhhoog..rrgooooooooooo org 9 1245/.
Apertus Team. Apertus: Democratizing Open and Compliant LLMMs for Global Language Environ- ments. https:://///. htt////gggggngggggeeeee co/ swiss ai/Apertus 70B- 2509, 2025.
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Mar- tin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-Free 4-Bit In- ference Rotated LLMs, October 2024. URL http:://ttttet httt:::/aax.v..or//bbb////////// 2ergg/bb//244.00 2404. 00456. arXiv:2404.00456 [es].
Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clementine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long- long-context reasoner, 2025. URL https:/ htt.h/ggggggaceeeeeo
ct/blog/bbaaaccccccc stmm/mmlllm 1m33.
David G. T. Barrett and Benoit Dherin. Implicit gradient regularization. CoRR, abs/2009.1116 2020. URL https:///// arxiv..vvvv wwt/aas//////eaeo/..//.///eea abt////........////////// 11162.
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate works best for llms, 2025. URL.
https: http::///xxxvvvvroee wwwp:/s/////oooo.oooooooooooss abss 2592..
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika. and Oskar van der Wal. Pythia: A Suite for Analyzing Large Lan- guage Models Across Training and Scaling, May 2023. URL http:://axat org/ a 2304. 01373. arXiv:2304.01373 [cs].
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
11

---

Published as a conference paper at ICLR 2026
Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020. URL http:://arxett org/ abs /000 14165. arXiv:2005. 14165 [es].
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys, 2017. URL https: /wtpxvaaivvvv//aa//////eaa/aawva org/o//////... aaaaabss1111... 111i//188..
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 2018. URL https: /ttp://///aiii.aa/ wttp:/vv.rrg/a///////aa/o
05457.
Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, J. Zico Kolter, and Jason D. Lee. Understanding optimization in deep learning with central flows, 2025. URL
/bs/
2410.24206.
Matthieu Courbariaux, Yoshua Bengio, and lean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations, 2016. URL
abs/
1511. 00363.
Aaron Defazio. Why Gradients Rapidly Increase Near the End of Training, June 2025. URLhttp: wwaxxii.org/abbe..... 2506. 02285. arXiv:2506.02285 [es].
Aaron Defazio, Xingyu, Yang. Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The Road Less Scheduled, May 2024. arXiv:2405. 15682 [cs, math, stat).
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int80: 8-bit Matrix Multiplication for Transformers at Scale, November 2022. URL httr:ttt//////vvorrg/////// 2208. 07339. MrXiv:2208.07335 [cs].
Karel D' D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christo- pher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment, 2024. URL https:/ http://aavvvvvv///e/eeeeaae.. org/ abs/2400 06266.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post- Post-Training Quantization for Generative Pre-trained Transformers, March 2023. URL http: /arxiv. org/abs/2210. 17323. MxXiv:2210.17325 [cs].
Samir Yitzhak Gadre, Georgios Smymiss, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilbarco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muen- nighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on down- stream tasks, 2024. URL https: aaxiv.iii org/ abs/ / 2403 08540.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black. Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL https: 5371628.
Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Car- doze, George Edward Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instabilities of deep learning models. In International Conference Learning Represen- tations, 2022. URL https:////.. /ttp////peeeeeeeeeeeee/eeeeeeeee net/ http://fommmmmmddommmee 36vUs.
Ian J. Goodfellow, Oriol Vinyals. and Andrew M. Saxe. Qualitatively characterizing neural net- work optimization problems, May 2015. URL http: /ttpiv/aaivvvv//aa www./ogg/......oo...........o... haag//bb/144 /112, 6544. 4116 [cs].
12

---

Published as a conference paper at ICLR 2026
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lam- bert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannanch Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.
URL hwtps:tpp///omm /ttp:///axxvvvvv org/ abs/ 2212. 00752.
Alexander Haegele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations, 2024. URL
https: /ttp///aivvvv//vv///////......vv org/ abs// 2405. 18392.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence Learning Representations, 2021. URL https: http///pppereveieeeeeeeeeeeeeeee net
wwwp://wwwww..eeeooomu...... forum?id=
d77BBjmI 3GmQ.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1 1997.
Sepp Hochreiter and Jürgen Schmidhuber. SIMPLIFYING NEURAL NETS BY DISCOVER- ING FLAT MINIMA. In Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL https: ptttps://ppoceedinng neurips. wwtpppppppppaeeeeeee httr//9994//haa// Absttrrccbb htt.raatt..m.......
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hen- nigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training, In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https://////// /openeevieew net ///ooooo forum?io did= iBBCRUl OAPR.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL httpp:://arxi.....t//// org/ /
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communication in Statistics- Simulation and Computation, 18:1059-1077 1076, 01 1989. doi: 10.1080/036 0919008812866
Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018. URL https:/ http::://a..semaaiiii.o..oom semanticschol. org/ / 3833416.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization, 2019.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anogerrarithmetic-onlyy inference, 2017. URL https: axxiv. org/ abs/1712. 05877.
13

---

Published as a conference paper at ICLR 2026
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP, pp. 2567-2577, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/DD) 1EE3////DD1--22// URL http::s:s//aaaatthooooooooo / 019-1259/.
Jean Kaddour. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging, October 2022. urXiv:2209.4998 [cs, stat).
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. org.ooooula..//oouuuuutt
CoRR, abs/1412.6980, 2014. URL https: /app. 6628106.
Tanishq Kumar, Zachary Ankner, Benjamin Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling Laws for Precision, November 2024. URL
abs/ 2411. 04330, DXXiv:2411.043300
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism, 2020. URL https://arxiv. org/abs/ 2203. 02218.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape Neural Nets, November 2018. URL httthtt://aa/te wtt..//orgg/ab///...99909// arXiv:1712.09913 [cs].
Houyi Li, Wenzhen Zheng, Qiufeng Wang. Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin J'ang. Predictable scale: Part i, step law optimal hyperparameter scaling law in large language model pretraining, 2025. URL https:
https///aawvvw..ee.eeeeee /
/2503. 04715.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, July 2024. URL http:///aa/ org/ abs/ 2306. 00978. arXiv:2306.00978.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He. Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. Llm360: Towards fully transparent open-source Ilms, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https::/arre orgo alos/ 608. 03983.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https: / //t.aiii.og//aaaaaaaaaoa/aa//aaa http:/aabs/11111110 05101.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018. URL https://arxiv: org/abs/ /1809 02789.
ModelCloud.ai and qubitium @ modelcloud.ai. modelcloudh ceteelclllo////eemmeeeee 2024.
Gptqmodel.
14

---

Published as a conference paper at ICLR 2026
Mariannaa Nezhurina, Jörg Franke, Taishi Nakamura, Timur Carstensen, Niccolò Ajroldi, Ville Komulainen, David Salinas, and Jenia Jitsev. -pen-sci-ref-0.0:: open and reproducible ref- erence baselines for language model and dataset comparison, September 2025. URL http:
//arxiv. orgeorgggaao /abs///a//aa 2509.000 09009. arXiv:2509.0900 099009 [cs].
NVIDIA. precision inference.
Introducing
NVFP4 for efficient and accurate low-
https://developee www.:oopr..iiiiiiioo www.ea.r.iiiiiiii......... htw.::/wwwooo.oooeo..eeeeaaeeaee
June 2025. NVIDIA Technical Blog.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi. Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wil- helm, Michael Wilson, Luke Zeutlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Ha- jishirzi. 2 OLMo 2 Furious, January 2025. URL http::/ http:://axxiv...oo.e wttp:/v.orrg/bb/////000 00666. arXiv:2501.0065 [es].
Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens, November 2024. URL Utthttp: httttp://axxi...oo// wwwpriiwwrrgbbb//eeeee /tabs/241110 17691. arXiv:2411.17691 [cs].
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL htths::: /ttttpp://rxxxxv /wwxiv..iviv/v/////////aa////iva org/abs 911 01703.
Guilherme Penedo, Hynek Kydlīček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. httpttss://ppent httpp//ppneeee.ee/oe/pp net forum? id=n6SCkn2Qem QaG.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https::///// /ttp///aavvv/////////////////aaa org/ arrg/al aaaaab//999 10683.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale, 2019.
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Nolan Miller. Training trajectories, mini- batch losses and the curious role the learning rate, February 2023. arXiv:2301.02312 [cs].
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialIQA: Common- sense reasoning about social interactions. In EMNLP, 2019.
Fabian Schaipp. How to jointly tune learning rate and weight decay for AdamW, https:// // 2024.
fabian- sp gi github. io htt/pstss/22//2///eee///
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https::/aaaaaeeeee wwwp:iiiiorrr/aaaa 0110/1101111611
Vaibhav Singh, Paul Janson, Paria Mehrbod, Adam Ibrahim, Irina Rish, Eugene Belilovsky, and Benjamin Thérien. Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre-training, 2025. URL https: //ta:///aiiiii http::::://..xiooooo//bbbbb/s/ org/abs 2503. 02844.
15

---

Published as a conference paper at ICLR 2026
Dan Su, Kezhi Kong, Ying Lin, Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into refined long-horizon pretraining dataset, 2025. URL https://arxiv..rr//aaatt 0.595.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 4159-4158, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: (0.18653/11/1191144 1421. URL https: hat///aaaaanttooyyy...o... org// N1 0119-141//
Guiyao Tie, Zeli Zhao. Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang. Jiangyue Yan, Yao Su, Zhenhan Dai, Yifeng Xie, Yihan Cao, Lichao Sun, Pan Zhou, Lifang He, Hechang Chen, Yu Zhang, Qingsong Wen, Tiamming Liu, Neil Zhenqiang Gong, Jiliang Tang, Caiming Xiong. Heng Ji, Philip S. Yu, and Jianfeng Gao. A survey on post-training of large language models, 2025. URL https: :ttpppp::///iiiii/// wttp/wawwrgg////aaaaa/e/../ooo/t alss///////// /2503. 06072.
Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing, 2024. URL https: //tppiv////xvv.////////s//..v.v/ org/ als// /wt./////000 2408 11029.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1..02.13977 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, June 2024. URL http: //t::v/axivvvaa www.///g//........... abs/ / 220224422222 04396. arXiv:2402.0453 [es].
Albert Tseng, Zhaofeng Sun, and Christopher De Sa. Model Model-preserving adaptive rounding, 2025.
URL httpt::t///// /atp:ivaaiivvvv org/ abs/2505 22988.
Vincent Vanhoucke and Andrew Senior. Improving the speed of neural networks on CPUs.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: org/abs/ 1106. 03762.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science ques- tions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User--generated Text. pp. 94-106, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10. 10.18653/1//11-4411.. URL hattps://aclanthollee
001p//1174413//
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https:/ http://wwwww.b..././/....p.t acwweb. org.org///oooooo htr/gnnthooog// 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves ac- curacy without increasing inference time, 2022. URL https: /ttp//aavvv.///////////////..avl org/ ab2/2233. 05482.
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities, 2023. URL https: //atss://///ii/// http://wwwwwwwww///////////ooos/
14322.
16

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https: / wttp://rrr/abbb/...eeee.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Pyhessian: works through the lens of the hessian. 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590, 2019. URL https: /api. semanticscholat wttg/oopos/oooo/o/ooooooo 209376531.
Neural net-
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. URL https: / arxiv org/ abs/1905 07830.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022. URL https:: /ttpss///aaaii../eeeeeeeeeeeeeea wwwp:/wwwwrgg/////////// / 04560.
17

---

Published as a conference paper at ICLR 2026
A QUANTIZATION PROTOCOL
Alternative quantization methods. Our results are centered around GPTQ Frantar et al. (2023) popular and accessible quantization method that works off-the-shelf for new models with minimal engineering overhead. To assess whether the phenomena we observe are specific to GPTQ or reflect broader trends in PTQ, we replicate Figure with ALM.int8() Dettmers et al. (2022) and AWQ Lin et al. (2024), As shown in Figure 10, we observe a consistent association between learning rate driven training dynamics and quantization error.
3.5
Qusrtradtin
1.5
1.
1.0
0.0
044 6.6. tratting
baittig
Mamber brsttin
1032
(a) GPTQ
AWQ
(c) LLM.int8
Figure 10: Quantization error on different 4-bit quantization backends. We replicate results from Sec- tion 4.1, training 160M- caapaereeeter transformer with different quantization backends, and recover similar trends quantization error during both constant and cooldown phases of the learning rate schedule.
Quantization details. For each model, quantize the linear layers following the default set- tings of GPTQModel (ModelCloud.ai & qubitium @ modelcloud.at 2024) and HuggingFace inter- nal quantization backend. For GPTQ, we follow common practice (Wolf et al., 2020) and C4 (Raffel et al., 2023) as the calibration dataset, with a group size of 128. For AWQ (Lin et al., 2024), use Kwon al. (2023).Final) Finally, for LLM.nt88) Dettmers al. (2022) we follow HuggingFace Wolf et al. (2020) implementation.
B PTQ ROBUSTNESS ON ADDITIONAL MODELS IN THE WILD
In this section we report the quantization degradation for additional model families. Although most models follow a regular pattem. some exhibit unpredictable behaviors. Amber (Liu al., 2023) in Figure 12 displays brief spike in full-precisin validation loss, while the full-precision model recovers, 4-bit PTQ degradation rises sharply, hinting change the training dynamics whose cause cannot identify. Additionally, Apertus (Apertus Team, 2025) in Figure 15 exhibits very large, fluctuating quantization errors from the beginning, which may indicate numerical issues either in the quantization process or in the weights. However, note that, for these models, quanti- zation degradation increases as the learning rates decays, consistent with our previous findings.
le-3
Quantization Error
4.0
3.00
3
Learnin Rate
Validation Loss
Learnit Rate
2.95
3.5
2
2.90
2
2.5
2.80
Stable
PTO Stable
0
Decay
Decay
0.2
0.4
0.6
0.8
1.0
0.2
1.0
Number training tokens 1e12
Number training tokens
1e12
(a) 4-bit PTQ emror vs training tokens.
(b) Validation loss vs training tokens.
Figure 11: Evolution of quantization error and validation loss OpenSci-1.3B model (Nezhurina et al., 2025) trained on IT tokens from Nemotron-ce (Su al., 2025).
18

---

Published as a conference paper at ICLR 2026
3-bit Quartiza tion Error
4-bit Quantization Erro
Validation loss
1c
1e
1e-2
3.0
Amber 78 3-bit
2.35
Amber 78 4-bit
2.5
Quarttration Error
Quantization Error
Lo33
2.30
2.0
2.25
1.5
2.20
1.0
0.5
2.15
0.5
1.0
0.5
1.0
0.5
1.0
Number of tokens
1c12
Number tokens
1c12
Number of tokers 1e12
Figure 12: Quantization degradation for Amber-7B. 3 and 4-bit quantization with GPTQ.
Apertus-88
Quantiration Error
bit Quantization Error
Validation
Apertus-88 3-bit
1.5
2.5
Apertus-88 4-bit
Apertus-88 88 FP
Quantization Enror oa
2.9
2.0
8
1.5
2.6
1.2
2.5
1.1
2.4
1
Number
1e13
Number
1e13
Figure 13: Quantization degradation for Apertus-SB. 3 and 4-bit quantization with GPTQ.
OLMo-1
Velidation loss
1e-4
1e-2 3-bit Quantization Error
1 Quantization Error
GLMp 1 18 3 bit
3.0
Quantizatin conooottoon Error
Quantrzation tiom Error
3.0
OLMo-1 1B
2.0
2.8
2.5
1.0
-0.050
-0.025
0.000
0.025
0.050
Number of tokens
1e12
Number of tokers
Number of tokers
1e12
Figure 14: Quantization degradation for OLMo-1 1B. and 4-bit quantization with GPTQ.
OLMo-1 78
Quantization
Quantization Error
Validation loss
3.0
1.25
3.50
OLMo:1 78 4-bit
Duanttaation Error
3.25
DLMo-1 78
2
Quantiizaion
0.75
3.00
2.0
0.50
2.75
1..5
0.00
tokens
Figure 15: Quantization degradation for OLMo-1 7B. and 4-bit quantization with GPTQ.
19

---

Published as a conference paper at ICLR 2026
C PRETRAINING HYPERPARAMETERY AND SETUP
Hyperparameter details. We use the open source codebase from Ajroldi (2024) to pretrain Pythia-160M parameter transformer models (Vaswani al., 2023; Biderman al., 2023) on causal language modeling, training up to 100 billion tokens of FineWebEde (Penedo et al., 2024) on up RLA100-80GB GPUs. We employ a sequence length of 2048 and batch size of 0.5M tokens. We use cross cross-entropy loss and employ Adam (Kingma & Ba, 2014) with decoupled weight decay (Loshchilov & Hutter, 2019) of 0.1 and gradient clipping of I, and B1 0.9, 32 0.95. For the periments in Figure 4 we use a WSD learning rate schedule with peak learning rate of 3e-3, warmup of 1900 steps (1 ()), and a cooldown duration of 1900 steps (10% of total duration), decaying the learning rate to zero (Bergsma al., 2025).
Weight Averaging. For the analysis in Section 5.2 and Figure 7 we use LAtest Weight Averaging (Kaddour, 2022), collecting checkpoints every 500 optimization steps, and maintaining rolling window of length over which weights are uniformly averaged. For the analysis Figure 24 where checkpoints are only available at fixed release intervals, we instead average the consecutive released checkpoints, reporting results for different window lengths.
D EVALUATION
Evaluating model performance is influenced by many factors, and quantization methods add an- other: the calibration dataset. For example, model quantized using web data for calibration, may perform better on web-based tasks. In general, interactions between training data, calibration sets, and validation sets may create complex effects that affect the reliability of results.
To address this problem, we evaluate using two approaches:
A held-out split of Ref (Penedo et al., 2024), to gather validation loss performance.
Downstream performance on the following tasks:
ARC-Challenge (ARC_C) (Clark al., 2018)
ARC-Easy (ARC E) (Clark et al., 2018)
OpenbookQA (OBQA) (Mihaylov et al., 2018)
PIQA (Bisk et al., 2020)
HellaSwagg (HSwag) (Zellers et al., 2019)
WinoGrande (WinoG) (Sakaguchi al., 2019)
MathQA (Amini et al., 2019)
PubMedQA (Jin et al., 2019)
SciQ (Welbl et al., 2017)
Social IQa (SIQA) (Sap et al., 2019)
CommonsenseQA (CSQA) (Talmor et al., 2019)
MMLU (Hendrycks et al., 2021)
We evaluate models using Sok--vval-haneee (Gao et al., 2021) and vLLM (Kwon al., 2023).
We report per- par-task accuracy of SmolLM3 in Figures 16, 17, ?? for the full-precision, 3-bit GPTQ quantzied and 4-bit GPTQ quantized weights respectively.
20

---

Published as a conference paper at ICLR 2026
Hellaswag
Q4
Openbock
Accuisi
0.55
Accurccy
Accuncy
Accuney
0.350
0.4
0.325
0.50
0.300
Pige
Pubmed QA
Seig
Social IGA
0.97
Accursry
0.78
accuracy
0.7
Accuracy
Accurayy
0.50
0.96
0.76
0.95
Tokens Arc Challenge
Winegrande
Arc Easy
Commonsense QA
0.55
Accuraty
0.70
Accuract
0.50
Accuracy
0.00
Accuracy
0..45
0.4
0.75
0.2
Figure SmolLM3 per per-task full full- precision accuracy, measured throughout training.
Hellaswag
Math
MMLU
Openbook
68l Emat
Rei Enror
0.075
0.2
0.050
0.05
0.025
Num: Tokkns
0.0
Num. Tokens
Num. Tokens
0.000
Mum. Tokens
0.15
Pubmed
Scla
0.075
Social
M /
6al Etgar
Rel Error
0.050
eel
0.025
0.0
0.000
Num. Tolonns
Num. T5 kons
Kum, Tokens
Mum, Takens Commonsense
0.2
0.3
0.6
Rell Enror
Rel. Enrer
0.15
Rel. Enror
0.2
0.4
0.10
0.1
Num Tokens
Num. Tokens
Numn Tolenss
Mum, Tokens
Figure 17: SmolLM3 per-task relative accuracy degradation under 3-bit GPTQ, measured throughout training.
Hellaswag
Math QA
0.075
MMLU
Openbook QA
Del. Eroor
0.
Rel. Enor
Eroor
Del. Error
0..01
0.02 02
0.025
0.00
0.00
0.000
Tokons
Num.
0.
Piga
Pubmed QA
Sola
Secial IQA
Bol. Ermor
Rel. Enor
0.1
0.02
0.07
0.0
80.0
-01
Nom Tokenns
Num. Tikens
Num. Tokens
Num Tokens
Winogrande
Arc Challenge
Arc Easy
Commonsense QA
Rei. Enor
0.025
W
0.06
Eiror
0.05
M
0.02
#1
0.00
0.025
0.00
Nom TokenS
Nom. Takens
Nom. Toxens
Num. Tokens
Figure 18: SmolLM3 per-task accuracy degradation under 4-bit GPTQ, measured throughout training.
21

---

Published as a conference paper at ICLR 2026
1d-2
3.75
Quentizztion Eroo
3,70
Loos Quanticd
5.8
1-0
1
Los
3.60
3.7
0.5
@ 3.55
3.66
Su-4
10.3
3,50
3.35
Number tokens
1c9
Loss Baseline
Val Baseline
(a) b-bit quantization error.
(b) FP 4-bit validation loss
(c) FP 3-bit validation loss.
Weight decay promotes PTQ robustness. With fixed learning rate 3e and WSD we train several models changing the weight decay parameter only. We observe that larger parameters lead to models with higher PTQ robustness. The dashed line represents the parameter chosen for all prior experiments.
Figure 19:
E ADDITIONAL RESULTS
In this section we provide additional figures for Section 5.
E.I WEIGIT DECAY
We show Figure 19.
E.2 GRADIENT OF THE LOSS
Recent work has shown that the gradient of the loss increases during the end of training (Defazio, 2025). We have observed that this phenomenon coincides with the decay phase of WSD, to this end, we analyze whether this change in the training dynamics is driving quantization degradation in Figure 20. Fixing all other hyperparameters (more details in Appendix C) we train with AdamW (Loshchilov & Hutter, 2019) (in cyan), and AdamC (Defazio, 2025) (in orange) which aims to correct this behavior. We observe that AdamC reduces the spike of the norm of the loss gradient in Figure 20b while simultancously changing the norm of the weights in Figure 20c. However, despite modulating different actors of the training dynamics, both optimizers demonstrate almost identical quantization degradation in Figure 20b, suggesting that the of the gradient of the loss does not impact quantization performance as a standalone factor, indicating more complex relationship.
le-3
AdamC
1.2
Quantizatinn Emro
Adami
Learning Rate
2.0
Norm
1.1
Learning Rate
Weightt NeeigghtNoon
Learning
Learning
1.5
1.0
0.6
0.9
0.4
0,8
0.00
0.25
0.50
0.75
1.00 0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1,00
Number of tokens lell
Number of tokens 1ell
Number of tokens 1ell
(a) 4-hit quantization error.
(b) T.2 norm of the loss gradient.
(c) T.2 norm of the weights.
Figure Loss gradient norm does not directly modulate quantization error. Quantization error, La norm of the loss gradient, and L2 norm of the weights for a 160M model trained with AdamW (Loshchilov & Hutter, 2019) (in cyan) and AdamC (Defazio, 2025) In Figure 20b we observe that the gradient of the loss spikes during the later iterations when using AdamW, whereas AdamC reduces the spike at the end of training. Furthermore, Figure 20c we observe that AdamC affects the norm of the weights.
E.3 COSINE DECAY VS WSD
In Figure 21 present the quantization error and validation loss for 160M parameter models trained on different token budgets with the same learning rate with cosine decay and with WSD learning rate schedules. We observe that though quantization error appears be related to training data
22

---

Published as a conference paper at ICLR 2026
budget for cosine decay learning rate schedule, on WSD quantization and training data budget appear to be less entangled.
le
Ir=3e-3
Quantization Error
2.5
Ir=6e-4
2.5
2.0
3.8
2.0
+-
1.5
3.7
1.5
3-
3.6
1.0
0.5
0.5
1
0.0
3.4
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.6 Number tokens
0.4
0.8
Number of tokens
lel1
1el1
(a) Quantization error vs training tokens.
(b) Validation loss vs training tokens.
Figure 21: PTQ error at different training durations with cosine decay. We repeat the experiment in 4.1 and Figure with a cosine learning rate schedule. PTQ error (left) varies with training horizon, but peak learning rate and scheduler shape have larger impact.
E.4 LEARNING RATE
We repeat the experiment in Section 5.1 on a larger scale, using OLMo2-7B evaluating quantization error during a learning rate annealing run 50B tokens after model was pretrained for 250B tokens on 4 different learning rate values. In Figure 23 we observe that, even though the quantization degradation lower, the same patter arises, where larger learning rates lead to lower quantization degradation, even at the same validation loss.
F ADDITIONAL DETAILS AND RESULTS FOR LOSS LANDSCAPES
Given parametric model θ E TR a set D = [(x;, Ui) m of feature vectors with corresponding labels pairs, and a loss function L()) = L Σm E(Zi, yi; θ). we adapt Goodfellow et al. (2015); Li al. (2018) visualize 2D slice of the loss. Our aim is interpolate the loss between three checkpoints of particular interest, O₂ the model at the end of training, Ox-1 the model at a previous step of training', and θx, the model at the end of training quantized. Setting and as the direction vectors from O θK-1 and Ox to θx respectively, and the validation set D, we care about
fla B) L(D; θµ av + Bu)
To populate the contour plots simply sample 1000 points regular grid contained by largest bound from the set that we are comparing, and then reconstruct a model from the vectorized defini- tion that sampled.
To vectorize a quantizaed model, we first "dequantize" by explicitly multiplying the scales and low- bit primitives, and we retrieve high-precision approximation of the quantized model that we can use.
3-bit GPTQ Loss Landscape Analogous Figure 8, we show the loss landscape for 3-bit GPTQ quantization on Figure 25. We observe that the same pattern occurs, with larger weight perturbations, where the flatness of the basin of the loss is more relevant.
G SECOND ORDER STATISTICS
Trace. In order to approximate the Hessian trace, we exploit the following result. Let A E Raxe be a symmetric matrix, let be a multivariate random variable in R with mean
Wee visualize 160M parameter models where n = 1.6e
WWe visualize checkpoints that are trained for 100 billion tokens during K 190000 steps. We save the checkpoints every 2000 tokens, therefore K 188000.
23

---

Published as a conference paper at ICLR 2026
3.55
Quantizaton Liren
Vol. Loss (Quantaadd)
(Quantreg
33.8
3.7
3.45
Cosine (Rm6e-4
3.40
WSD
LR=3e3
WSD
LRm3e-4
0.4
0.6
0.8
1.0
3.35
3.40
3.55
3.6
37
3.8
3.9
Number of tokeons
lell
Vol. Loss (FF)
Val. Loss (FF)
(a) 4-bit quantization error.
(b) FP 4-bit validation loss.
(c) FP 3-bit validation loss.
Warm up-Stable-Decay and Cosine decay. Figure 22a shows the quantization degradation that results from changing the learning rate magnitude and schedule. We observe that learning rate modulates quantization enror regardless of the schedule. Finally, in Figure 22c observe that cosine schedules have sharper trade-off the validation loss of the full precision the quantized weights.
Figure 22:
1e-3
2.64
2.70
5.0
Quantized
2.63
Quantization
2.62
2.66
12e-4
8 2.61
9
Ir=9e-4
3.5
2.60
2.62
Ir-6e-4
3.0
2.60
Ir-3e-4
2.59
2.6
2.7
2.8
2.9
3.0
2.60
2.62 Val. Loss Baseline
2.64
2.600
2.625 2.650 2.675 2.700
Number of tokens lell
Val. Loss Baseline
(a) 4-bit quantization error.
(b) FP 4-bit validation loss. (c) FP 3-bit validation loss.
Figure 23: Larger learning rates lead to lower quantization error. Figure 23a displays the quantization error achieved by fixing the training recipe and varying the learning rate of OLMo2-7B. Wc observe that quantization error decreases when employing higher learning rates. Furthermore, Figure 23b and 23c show that, similar validation loss, larger learning rates achieve better low-bit quantization at no apparent cost.
and covariance Σ, then:
(r(AE) + ET
where E indicates the expectation and tr the trace operator. Therefore, for a random vector & with zero-mean and identity covariance matrix, =" Az is unbiased estimator of Ir(A). Hutchinson (1989) showed that when is distributed accordingly to multivariate Rademacher distribution, the estimator achieves lower variance than choosing be multivariate Gaussian random vector.
We can leverage this property to estimate the Hessian trace of the loss function by drawing samples from Rademacher distribution and computing Hessian vector products, which be easily com-
2.85
Val. Loss Quantized
2.8
OLMo-18
2.75
WA-2
WA=4
WA-8
2.7
WA=10
2.75 Val. Loss Baseline
2.8
2.85
Figure 24: Weight Averaging improves OLMo performance before and after quantization. We use LAWA, averaging weights along the OLMo-1B training trajectory. We measure and report validation loss in full pre- cision and after 4-bit quantization. Compared to individual checkpoints the full trajectory, LAWA yields lower validation loss both before and after quantization, with larger averaging windows performing best.
24

---

Published as a conference paper at ICLR 2026
Lite 3.00e-04
Uiei 10e 63
Uk 3, 3,00e-03
Ute1 00e-02
-10
-.5
-20
205
Bc-1 step
8, quantized
Figure 25: Landscape of the loss. We visualize the landscape of the loss in the plane spanned by the weights
{θK, Ox-1, Ox} } for learning rates corresponding to the experiment in Figure 6. We observe that flatness of the loss basin is proportional to learning rate magnitude.
1e 2
Top Hess eigenvalue
LR=3e-4
LK=Ye2
1.0
LR-le3
LR=3e-3
LR=3ed
20
LR=1e:3
0.8
LR=1e-2
LR-3e-4
0.6
Estimated
0.4
10a
10
0.0
0.4
0.6
0.8
0.25 Number of tokens lell
0.75
1.00
Number of tokens 1ell
(a) Sharpness.
(b) Hessian trace.
Figure 26: Second order statistics across learning rates. We train using WSD, varying the maximum learn- ing rate, but always decaying zero. Higher learning rates lead to lower sharpness and smaller trace esti- mates, suggesting that the model may have converged to wider minima. Interestingly, larger learning rate also lead to lower quantization error (Figure 6).
puted with extra pass over the computational graph. We use PyHessian (Yao al., 2019) for such Monte Carlo estimation in PoTorch.
Sharpness and spectrum. Furthermore, we measure the largest eigenvalue Amaz the Hessian, also referred to sharpness. In order to estimate Amaz power iterations, once again lever- aging Hessian vector products computation in PyHessian. In some cases we further compute the first 25 hessian eigenvalues.
We measure both summary statistics on house trained Pythia-160M models. We compute the trace and sharpness of the validation loss, computed held-out set of 100 text sequences from Fine WedEd,, each of length 2048.
H LIMITATIONS
Our analysis focuses primarily on the effect of learning rate, schedules, and weight decay leaving other parts of the optimization pipeline unexplored. Factors such optimizer choice may also affect quantization performance, and we leave the exploration of schedule-fres methods (Defazio al., 2024) to follow-up work. Moreover, although limit our analysis dense quadratic model, we expect similar conclusions for sparse (Shazeer et al., 2017) and sub-quadratic architectures (Gu Dao, 2024).
25

---

Published as a conference paper at ICLR 2026
DISCLAIMER FOR USE OF LLMS
We primarily used LLMs in coding auto-completion applications to facilitate experimentation. LLMs were also used writing tools assist in refining the paper. However, the final version was carefully reviewed and finalized by the authors. No LLMs were used in ideation and experi- mental design.
26


Reviews:
[{"review_id": "FEoK5CE8dx", "reviewer": "Reviewer_Po58", "summary": "This paper examines the interplay between post training quantization (PTQ) performance and variables related to training dynamics, specifically learning rate schedule and model averaging. The authors challenge the conclusion from prior work that PTQ error is primarily driven by training duration. The paper presents evidence that observations about training length duration are confounded by learning rate cooldown and that it's the cooldown period which primarily drives PTQ error growth. The authors demonstrated their claims against a large suite of open source models.", "questions": "* While potentially out of scope for this work, it seems important to understand whether the same phenomenon occurs with quantization aware training (QAT). Did the authors run any experiments with QAT?\n* From looking at Figure 3, it seems like the slope of the quantization error decreases as a function of the model size examined. It would be good to disentangle whether lr cooldown impacts quantization error less at larger scales or whether this is the interplay between overtraining and lr cooldowns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The question that the paper studies is a very interesting one and disentangling learning rate cooldown from training duration is a subtle but important distinction that can inform practitioners in their pretraining choices.\n* The investigation into model souping provides practical information to guide practitioners in reducing PTQ error.\n* The effect observed is quite convincing in terms of home prominent of a phase transition there is\n* The empirical results are very thorough and replicate over a large suite of models and dataset sizes.", "weaknesses": "* While the phenomenon observed is quite interesting, the paper is missing a predictive model of the effect of learning rate on PTQ. Similar to previous works (Kumar et al.), providing some scaling analysis that incorporates the relevant LR parameters would strengthen the paper greatly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q1BscNJO0G", "reviewer": "Reviewer_dMzB", "summary": "The paper explains recent work on post-train quantization scaling finding that degradation increases with data. The authors attribute it instead of learning rate annealing effects, and propose two new ways to mitigate this degradation.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is clearly divided into sections, where each one has a clear claim and empirical evidence for it. \n- They replicate and explain past work on PTQ scaling, reassuring the reader that their baselines are well-tuned. \n- They propose interventions to mitigate the identified \"mechanism\" for degradation. \n- Fig5 is particularly strong evidence that the annealing itself is causal for degradation.", "weaknesses": "- I'm not entirely sure this paper has enough \"meat\" to be a conference paper. There is a lot of redundancy in the plots, and the contributions/main claim can be summarized as \"higher LR and model averaging can partially mitigate PTQ degradations on long training runs, though we don't know why.\" It's not clear this is enough to comprise a conference paper? \n- Even the core claim that \"these data effects may actually be LR effects\" is actually not novel: [1] make a very similar claim in reference to the same literature, but in the related setting of finetuning instead of quantization. \n- The authors also do not posit any conceptual model explaining their findings, or interpret their findings. Even if there is no theory (which is fine), having a mental model with experimental ablations would be helpful. What exactly is going on here -- the fact that WSD and cosine both end up giving the same PTQ-induced loss but over timescales makes it feel like there is some \"degradation potential\" interpretation in the spirit or flavor of [2]. There is no actual scientific model presented with experiments. \n\n[1] Overtrained Language Models Are Harder to Fine-Tune. Springer et al, 2025. https://arxiv.org/pdf/2503.19206\n\n[2] Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. Wen et al., 2025. https://openreview.net/pdf?id=m51BgoqvbP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzkEyLTUke", "reviewer": "Reviewer_2yxu", "summary": "The authors study the impact of training dynamics (learning rate schedule) on post-training quantization Error. Through the study of various open source model families, the authors find that the learning rate schedule is the primary driver of PTQ error contrary to the popular belief of over-training leading to higher PTQ error. Based on this insight, the authors propose ideas to mitigate this discrepancy.", "questions": "- Do the observations hold for recent quantization techniques like QuaRot etc?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is an excellent paper. The authors analyse multiple different families to show that their insights hold.\n- The interventions (at a smaller scale) also demonstrate their findings.\n- PTQ degradation remaining near flat with WSD LR with higher tokens/params is a great observation.\n- The authors study both PTQ error and downstream performance degradation", "weaknesses": "- The authors claim the effect of training hyperparameters on quantization quality hasn't been well studied, yet the authors don't cite Intriguing Properties of Quantization at Scale [1] by Ahmadian et al. \n\n- The fact that smaller learning rates lead to larger PTQ errors hints at the manifold geometry (sharpness etc.) playing a key role in determining the degradation yet there is no discussion about this. It would be nice to relate PTQ errors to the geometry of the loss basin (albeit at a smaller scale). I think this would considerably strengthen the paper.\n\n[1] Intriguing Properties of Quantization at Scale : https://openreview.net/pdf?id=IYe8j7Gy8f", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBKLMmETU1", "reviewer": "Reviewer_9xqL", "summary": "This paper is an empirical study which explores the relationship between training dynamics and quantization (PTQ) performance, providing interesting observations that quantization errors are related to training hyperparameters like learning rate and scheduler settings, challenging the previous assumption that quantization errors are inherently related to dataset scale. The authors also experiment to intervene the training dynamics to identify specific configurations that modulate quantization robustness favorably, providing practical insights for related studies.", "questions": "- Could the authors provide a theoretical or intuitive explanation for why learning rate decay leads to increased quantization error? A deeper understanding of the underlying mechanism would significantly enhance the paper’s conceptual contribution and long-term impact.\n- The study focuses on the relationship between training dynamics and PTQ. However, given that PTQ is primarily applied when retraining is infeasible or training details are unavailable, what is the practical motivation for analyzing training-phase interventions on PTQ performance? If the authors have the capability to conduct large-scale pretraining & fine-tuning, why not extend the study to Quantization-Aware Training (QAT) and investigate how training dynamics influence QAT outcomes, which may offer more actionable insights for future model development?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The insights provided by this paper are very interesting and original. It discusses the relationship between quantization errors and training hyperparameters like learning rate and scheduler settings in detail, and shares some empirical results like the divergence of quantization error and validation loss with the decay of learning rates. I believe those insights could benefit future related studies.\n- The efforts of the authors trying to reduce quantization error by intervening training dynamics are more applicable for practical usage, compared with previous works' focuses on training data scale.\n- The paper is well presented with sufficient experiments and corresponding figures, making it easy to visualize the key points of the observations.", "weaknesses": "- While the empirical results provided by this study are abundant and interesting, the paper fails to provide more in-depth explanations on the reasons that lead to such phenomena, as it's not very explicit to relate factors like learning rate with quantization errors. This might limit the interpretability of the provided results.\n- The selection of evaluated models lacks representativeness, as several of the most widely used and influential model families (e.g., LLaMA, Qwen) are not included. So it remains unclear whether the observed correlations between training dynamics and quantization robustness hold for mainstream architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)