Oscillations Make Neural Networks Robust to Quantization
Authors: Jonathan Wenshøj
Bob Pepin
Raghavendra Selvan
Venue: Accepted by TMLR ()

Abstract: We challenge the prevailing view that weight oscillations observed during Quantization Aware Training (QAT) are merely undesirable side-effects and argue instead that they are an essential part of QAT. We show in a univariate linear model that QAT results in an additional loss term that causes oscillations by pushing weights away from their nearest quantization level. Based on the mechanism from the analysis, we then derive a regularizer that induces oscillations in the weights of neural networks during training. Our empirical results on ResNet-18 and Tiny Vision Transformer, evaluated on CIFAR-10 and Tiny ImageNet datasets, demonstrate across a range of quantization levels that training with oscillations followed by post-training quantization (PTQ) is sufficient to recover the performance of QAT in most cases. With this work we provide further insight into the dynamics of QAT and contribute a novel insight into explaining the role of oscillations in QAT which until now have been considered to have a primarily negative effect on quantization.

Published in Transactions on Machine Learning Research (11/2025)
Oscillations Make Neural Networks Robust to Quantization
Jonathan Wenshøj
jowe Rowedddi.co ku, dk bope di. ku, dk raghav&di di. ku, dk
Bob Pepin
Raghavendra Selvan
Deportment of Computer Science, University of Copenhagen, Denmark
Reviewed on OpenReview: https: // openreview. net/ por//frmm?? id= bOwkDOnDDC
Abstract
We challenge the prevailing view that weight oscillations observed during Quantization Aware Training (QAT) are merely undesirable side-effects and argue instead that they are an essential part of QAT. We show in a univariate linear model that QAT results in an additional loss term that causes oscillations by pushing weights away from their nearest quantization level. Based the mechanism from the analysis, then derive regularizer that induces oscillations in the weights of neural networks during training. Our empiri- cal results on ResNet-18 and Tiny Vision Transformer, evaluated on CIFAR-10 and Tiny ImageNet datasets, demonstrate across a range of quantization levels that training with oscillations followed by post-training quantization (PTQ) is sufficient to recover the perfor- mance of QAT most cases. With this work we provide further insight into the dynamics of QAT and contribute a novel insight into explaining the role of oscillations in QAT which until now have been considered to have primarily negative effect on quantization.
1 Introduction
Deep neural networks have grown increasingly successful at solving difficult modelling tasks, but are also increasingly becoming more expensive to use. As model sizes balloon into the hundreds of millions of parameters, the cost of inference has become a significant. bottleneck, particularly for deployment on edge devices or usage in large-scale services (Sevilla et al., 2022). To reduce this cost, quantization of model weights has emerged as prominent strategy (Nagel et al., 2021).
Weight quantization, however, comes with its own cost, namely drop in accuracy. Reducing precision introduces quantization error, and naive approaches like Post-Training Quantization (PTQ) often lead to sharp degradations in model performance at low bit widths. To combat. this degradation in accuracy, many methods have been proposed, usually centered around minimizing the quantization error (Hung et al., 2015; Hirose et al., 2017; Li et al., 2020; Choi et al., 2020; Han et al., 2021; Zhong et al., 2025).
Yet these approaches fall short of Quantization-Awant Training (QAT) (Jacob et al., 2018; Krishnamoorthi, 2018), which integrates quantization directly into the training loop, yielding models that remain robust even under low-precision constraints. However, despite its empirical success, the underlying behavior of QAT remains poorly understood. In particular, QAT often exhibits oscillations in the weights during optimization, while regular training does not. Rather than settling into a stable state, quantized weights fluctuate between adjacent quantization levels. These oscillations are widely seen as undesirable artifacts. Several works have tried to suppress these weight oscillations mainly through dampening or weight freezing (Défossez et al., 2022; Nagel et al., 2022; Huang et al., 2024; Gupta & Asthana, 2024; Liu et al., 2023).
This paper poses the question: Have we misunderstood the nature of oscillations in QAT? To the best of our knowledge, we the first to argue that weight oscillations in QAT are not flaw, but rather essential feature of the training dynamics. Using toy model, we show that oscillations arise from
Source code is available

---

Published in Transactions on Machine Learning Research (11/2025)
Oscillations during Quantization Aware Training
x
0.5
q(w)
Threshold
VL(w)
0.0
x
x
x
V8L
10
20
Epoch
Figure 1: Oscillatory behavior during QAT in a one weight linear model g = ((w),, with weight w, quantized weight g(w), imput æ = 1 and target y = 0.75. The gradient of the loss term of this model during QAT can be decomposed into two terms: VL(w) and Vóz = q(w) w, where the latter term what differentiates QAT from just optimizing the full-precision loss I(wr). During QAT, VL(w) always points towards y, while Vog introduces dynamic which pushes towards the nearest bin threshold. This causes w to oscillate when y is not exactly a quantization level. In the above case this makes (ww) alternate between 0 and 1. Note the frequency of oscillations of q(w) lets the quantized weight on average to converge to 0.75.
gradient component in QAT which pushes weights towards the nearest quantization threshold - this stands in contrast to aligning weights at the quantization level, which would minimize the quantization error. We then induce such oscillations to push the weights towards the nearest thresholds deliberately during the model training. This is realized simple regularization term that pushes weights towards their nearest quantization threshold. Surprisingly, this leads to models that robust to quantization and in most cases we find that these eggularizationiinnuee oscillations recover the accuracy obtained with QAT. Additionally, in many cases, oscillations outperform QAT under cross-bit evaluation (i.e. testing at precision levels not simulated during training). These findings further deepen our understanding of QAT and point to a more nuanced role for weight oscillations; suggesting they may be beneficial to QAT rather than being solely detrimental as argued in most existing literature.
Our primary contributions supporting this claim are:
1. Using univariate model we illustrate how the Straight Through Estimator (STE) leads to oscillations and clustering of model weights during QAT (Sec. 3);
2. We show experimentally that by using mechanism inspired by the toy model, we can induce oscillations and clustering during training in neural networks (Sec. 4).
3. We empirically confirm using the CIFAR-10 (Krizhevsky & Hinton, 2009) and Timy-ImageNet (Le & Yang, 2015) datasets on a multi-layer perceptron (MLP), ResNet-18 (He et al., 2016) and Tiny Vision transformer (Tiny VIT) (Wu et al., 2022) that introducing oscillations through regularization most cases recovers the accuracy of QAT (Sec. 5).
2 Preliminaries and Related Work
Quantization: A quantizer divides a continuous input range into quantization bins, where each bin is represented by a specific quantization level. The boundaries between bins are called quantization throsholds. During quantization, any value within a bin is mapped to that bin's quantization level. With a uniform quantizer, the step size (the distance between two adjacent quantization levels) is equal to the scale factor
2

---

Published in Transactions on Machine Learning Research (11/2025)
We consider uniform symmetric, scalar quantizer, q()), that can then be expressed as:
w
((w) = 8
(1)
Here, 8 represents the scale factor and [.] denotes the rounding operation.
The scale factor is set cover the range of w as this removes the need for the clamping operation clamp(q(w):0) () in the quantizer. The function clamp(-) restricts greater than is set to and respectively. Additionally set the number of positive and negative quantization levels to be 2 1, we have symmelric nunber of levels around 0:
q(w;) S and any values less
max (lwl)
(2)
26
where b the number of bits the quantizer can use.
The quantization process introduces quantization error A, defined the difference between the original and quantized values:
- w w ((w).
(3)
Due to the uniform quantizer each component of the absolute error (due to quantizing w:) is bounded bet ween 0 A S 8/2, for all the bins. This maximized at quantization thresholds and is at quantization levels.
Straight- Straight-Through Through Estimator (STE): STE is heuristic gradient approximation technique commonly used to enable backpropagation through functions f(-) whose gradients are zero almost everywhere (Bengio et al., 2013). In the forward pass, the function f(x) is applied as usual. During the backward pass, however, its gradient is approximated by replacing the derivative of f with the constant 1. Formally, STE defines the approximate gradient of function f as
0f df 0x
(4)
Quantization-Awe Training (QAT): In QAT, the network maintains full-precision weights for learn- ing but quantizes them during the forward pass before computation. This allows training to proceed with gradients applied to full-precision weights, while the network effectively operates on quantized values. While there exist many variants of QAT, the typical forward pass is performed using the quantized weights q(w) (Ja- cob et al., 2018; Krishnamoorthi, 2018). The gradient for the weights during QAT is given by
@l(g(w)) @l(a(w)) @g(w) Ow Öq(w) dw
(5)
A problem with the above formulation is that the gradient of the quantizer 0q(w) zero almost everywhere, Sw
causing the last term to interrupt gradient-based learning. A widely used solution to this problem to use the STE (Eq. (4)).
Weight Oscillations: We use the definition from Nagel et al. (2022) which defines a weight oscillation during QAT to occur in itcration t if it satisfies both the following conditions:
The quantized value of the weight needs to change between iterations i.c. g(w,) + q(w;-1).
The dircction of the change in the quantizod domain needs to be the opposite than that of its previous change i.c. (T, # sign(T,), where is the iteration of the last change, and I q(w;) q ((w)1)) is the direction of the change.
3

---

Published in Transactions on Machine Learning Research (11/2025)
Related Work: Minimizing the quantization error is the most commonly used strategy to reduce the impact of quantization model accuracy. Extensive research has been dedicated to developing techniques that explicitly minimize the quantization error during optimization (Hung et al., 2015; Hirose et al., 2017; Li et al., 2020; Choi et al., 2020; Han et al., 2021; Zhong et al., 2025). Despite these efforts, the aforementioned strategies often fall short of the accuracy obtained with QAT (Jacob et. al., 2018) al individual bits or indiroctly rely upon QAT themselves.
However, there is limited understanding of how QAT affects model optimization and why it outperforms other methods. One phenomenon observed during QAT is weight oscillations (Défossez et al., 2022; Nagel et al., 2022), which are periodic changes in the value of the quantized weight between two adjacent quantization levels. It is speculated in these works that the abrupt changes in values caused by oscillations can interfere negatively with optimization. Oscillations are assumed to be undesirable side effects caused by the use of the STE during backpropagation, as the STE allows gradients to pass through the rounding operation the quantizer, which has a gradient of zero almost everywhere (Défosscz et al., 2022; Nagel et al., 2022).
Several approaches have been suggested to mitigate oscillations by either freezing dampening (Défossez et al., 2022; Nagel et al., 2022; Huang et al., 2024; Gupta & Asthana, 2024; Liu et al., 2023). However, the reported accuracy gains are sometimes marginal, and these methods may inadvertently also hinder the optimization process. For instance, Nagel et al. (2022) notes that freezing or dampening weights too early during training can hurt optimization, indicating that oscillations might contribute to finding better quantized minima of the loss. Liu et al. (2023) propose that weights with low oscillation frequency should frozen, where as high-freque frequecyy ones should be left unfrozen, under the rationale that high frequency means the network has little confidence in what value to quantize the weight to, whereas low frequency means the optimal weight lies close to quantization level.
3 Oscillations in QAT
Previous studies have explored linear models to analyze the behavior of QAT and the phenomenon of weight oscillations (Défossez et al., 2022; Nagel et al., 2022; Liu et al., 2023; Gupta & Asthana, 2024). Inspired by these works, we also analyze a linear regression model to gain insights into the optimization dynamics during QAT.
3.1 Toy Model
Consider a linear model with single weight w, input x and target E R. The quantized version of this model defined as Ô = q(w)x, where q(-) the quantizer from Eq. (1). The quadratic loss for the quantized model given by
1 [(q(w)) -0 y)² (9(())= - 2
1
(6)
Our goal in this section is to understand how QAT affects the full-precision optimization process. For a given loss function C(-) with quantized weights, we have
L(g(w)) ((w) + f(g(u)) L(w).
(7)
We can then expand the difference in loss caused by quantization as follows
1 82- = L(q(w)) L(w) = ((a(w)x y)² - (wx y)²)
(8)
= 2 (0² ( (yx(w g(w)))-
This expression decomposes the loss difference & into a quadratic term (g(22 w²) and linear term yo(w - q(w))).
4

---

Published in Transactions on Machine Learning Research (11/2025)
Next derive the gradient of òc wrt.
06c = dw [(q(w)) -[(w)
(1
= dw 2 (q(w)) - - + yr(w - g(w))
(10)
dw
Oq(w)
+ yo (1- Ōg(w)
= r ((w))
w
(11)
Ow
Ow
Using the STE and recalling that
8q = 1 the expression of the STE gradient simplifies to
dw
dōc =² -w) ->²A(w). dw
(12)
3.2 Oscillation Mechanism
To see how the observations in Sec. 3.1 give rise to oscillations, let wn denote the upper quantization threshold for an arbitrary weight w, defined as we ((w) + s/2. For e E s0,s, note that we have g(wo 5) g(w)
and g(ur + +) (ww) + 8, so that
A Wi + + ( + = + +s -s/2 E
(13)
A (wo <) g(w) + x/2 ((w) -8/2- - E.
(14)
Assuming x # 0, the negative STE gradient "flips" from - s/2 to +s/2 2 as the weight w passes the quantization threshold Wo, pushing the weight back towards the threshold. We note that the STE gradient is zero at the special value w = q(w), but the preceding argument shows that this is an unstable critical point and gradient noise will immediately cause the weights to move away from it. When combined with (stochastic) gradient descent and finite discretization timestep we can identify this as the driving mechanism behind oscillations during training with QAT (Fig. 1).
3.3 Weight Clustering
We can also see how the dynamics described above can lead to weight clustering around quantization thresh- olds by looking at the sign of A for different values of w. For weight let diow (w) and dup(w) denote the distance from to the upper and lower thresholds, dwo(ww)
(a(w) ) Aww) + & and (w) (a(w) = w - Aww) respectively. If is closest to the upper threshold we have
dup diow
(15)
While if is closest the lower threshold
dhow
dup
A
A<O
(16)
We emphasize that this mechanism causes the weight to move Lowards the quantization thresholds (the edges of quantization "bins") opposed to the quantization levels (the centers of the quantization "bins"). As shown above, the magnitude of the pull towards the threshold increases as the weight approaches it. The weight will therefore eventually cross the threshold and start oscillating, unless L(w) and ôr exactly cancel out, which is unlikely to happen with a finite gradient descent step size.
4 Regularization Method
Our thcoretical observations in the linear model in Sec. 3, show that the oscillation component is the only part that differentiates QAT from standard full-precision training. We now confirm empirically that the
Note that there clamping the quantizer because of the scale factor Eq. (2).
5

---

Published in Transactions on Machine Learning Research (11/2025)
mechanism in Eq. (12) is sufficient to introduce weight oscillations during training of neural networks, and study if this also results in QAT-like behavior with respect to the quantization noise.
From the quantization difference in Eq. (9) and the STE gradient derived in Eq. (12), we have al(g(w))
al(w) x² (w),
(17)
dw where the first lerm is the gradient of the original full-precision loss, and the second term causes the quan- tization oscillations in QAT.
dw
In order to emulate the effects described in Section 3, we propose regularization term so that the training objective becomes
[(w) + Rx(w),
(18)
where we let the regularization term be similar to the quadratic term in Eq. (9):
(w(w) A Σーー
Σ (wx)))
(19)
Here À 2 0 is a hyperparameter that controls the amount of regularization, é ranges over the layers in the model and over the weights in each layer. In this term, replaced the factor x² by hyperparameter X, since the precise expression of x² is specific to the model studied in Sec. 3. We empirically find that this regularizer is sufficient to induce oscillations. The exploration of the design space of oecillation-inducingg regularizers, including deyerddependent and/or adaptive scale factors, is left to future work.
Using the STE,
dy Ow 1, we have the following expression for the gradient:
A i RA R.(w)= (q(w)) wf) - Aw!)) Ow
A
(20)
By the same reasoning as in Sec. 3 this pulls the weight wf towards the quantization threshold and causes the gradient to "flip" as w crosses the threshold. We expect this to lead to oscillations based on the same mechanism as in the model from Sec.
Figures 2 and show the results of an experiment where we observe the weight distributions, and measured the oscillations, during training of neural network (ResNet-18) with varying degroes of regularization, respectively. For comparison, the figures also show the weight distributions and oscillations observed during training with QAT.
Our first observation is that QAT displays more oscillations Fig. 3a(a) than a baseline model without QAT or regularization (corresponding to 0 Fig. 3-b)). As we increase we observe that the number of oscillations as well as the clustering increases. This confirms that the regularizer from Eq. (19) can indeed induce oscillations similar to QAT during the training of deep neural networks. At A A= 1 (Fig. 3-(c)) the number of oscil- lations observed with regularization is similar to the behavior of QAT, lending support to our hypothesis that the mechanism in Eq. (13) is indeed at the root of the oscillations observed when training neural networks with QAT. We use Welch's t-test to test if the oscillation counts in Fig. 3, which is the per-weight oscilla- X 10
Comparison t-score
p-value
Ao
va.
A1
25.13
< 0.001
Ao
vs. A1o
38.86
0.001
Ao vs. QAT
18.83
< 0.001
Table 1: Welch's t-test comparing the per-weight oscillation count after 50 epochs of training, for the different cases in Fig. 3. We note how the baseline = 0 differs significantly from QAT and regularization induced oscillations with
tion count after 50 epochs, are significantly different to the À = 0 bascline, reported in Table 1. Each of the pairwise comparison shows that the distribution of oscillations are significantly different.
5 Experiments and Results
In this section, we address the question: Is inducing weight oscillations during training sufficient to obtaán the benefits of QAT? We answer this question affirmatively using empirical evidence based on the results

---

Published in Transactions on Machine Learning Research (11/2025)
weight datribation
weight distibation
weightt dieribction
weight ducribution
Thrahs
Threhhold
Thraheld
Thramed
30
10
-0.2
-011
5.0
9.1
1.2
-0.2
-03
0.5
011
0.2
-03
0.5
0.5
0. -535
-530
-005
0.03
0.75
0.50
0.55
wwwee
(a)
(b)
(c)
(d)
Figure 2: Weight distribution analysis of ResNet-18's first convolutional layer after 50 epochs of training from scratch. a) Weight distribution under QAT with 3-bit quantizer. b)-d) Our proposed regulariza- tion approach with 3-bit quantizer at varying regularization strengths (A -A= 0, 1, 10, from left to right). When À = 0, the training reduces to standard optimization. The QAT distribution (leftmost) exhibits the characteristic threshold clustering behavior. As À increases, we observe progressively stronger clustering of weights around quantization thresholds, illustrating the relationship between regularization strength and weight clustering.
9AT
0.20
Jed
0.20
I-d
0.20
1-10
1.10
6.50
0.10
00000
4.05
6:55
6:25
0.35
Numbe o1 cac Adboro
Number of caci
Munber of aoclattaa
Mamber axcleiore
(b)
(c)
(d)
Figure Distribution of weight oscillations. The plots show the distribution of weights with oscillation counts 0 when training with a) QAT and b)-d) the regularizer for different values of X. Here À 0 corresponds to full-prccision model where the regularizer has influence training. The y-axis represents the percentage of total weights in the first convolutional layer of ResNet-18 trained from scratch for 50 epochs, while the x-axis shows the oscillation count after 50 epochs. Following the oscillation definition from (Nagel et al., 2022), we count oscillations at each epoch during training. The results demonstrate that QAT produces a significantly higher proportion of oscillating weights compared to = 0. Furthermore, we observe that as we increase greater percentage of weights oscillates.
of training ResNet-18 and Tiny Vit on the CIFAR-10 and Tiny-ImageNet datasets. This is both in the training-fromt fro-sscraccm (FS) setting and when fine-tuning (FT) pretrained models.
In the following subsections first describe the experimental sctup, present accuracy results in FS and FT settings for models trained with different quantization levels for the quantizer in Rr or QAT and finally, and report the cross-bit bit accuracy of the fine-tuned models. We train models at ternary (three possible values: -1, 0, 1), 3-bit and 4-bit. This is in line with contemporary research, where the emphasis lies on quantization at 4-bit and below, since the challenges of maintaining accuracy are more significant compared to quantization at higher bit widths.
5.1 Experimental Setup
We conducted our experiments using the CIFAR-10 (Krizhevsky & Hinton, 2009) and Tiny-ImageNel (Le & Yang, 2015) datasets. We evaluated three architectures; multi-layer perceptron with five hidden layers and 256 neurons per layer (MLP5), ResNet-18 (He et al., 2016) and Tiny ViT (Wu et al., 2022).
For each architecture we used the Adam optimizer (Kingma, 2014) and tested multiple configurations: a baseline model to establish optimal floating-point (FP) accuracy and PTQ performance, a model with QAT, and a model with regularization using Eq. (19). The two latter configurations are trained using 3-bit and 4-bit quantizers. In all our experiments we use the regularizer Rx defined in Eq. (19) to induce oscillations (marked as "Oscillations" in the results table).

---

Published in Transactions on Machine Learning Research (11/2025)
Quantization
MLP5 (FS)
ResNet-18 (FS)
Tiny VIT (FT)
ResNet-18 (FT)
Baseline FP32
51.43 +: 0.39
83.26 1.07
96.11 0.31
88.50 + 0.64
3-bit PTQ
5.64
11.56
3-bit QAT
50.53 + 1.43
82.51 = 2.14
88.13 + 0.60
85.69 + 1.83
3-bit Oscillations
48.48 + 0.29
81.77 0.46
88.68 1.08
84.94 + 1.59
4-bit PTQ
46.50 +: 0.76
82.11 + 1.21
21.37 + 5.33
35.56 +: 9.05
4-bit QAT
51.39 + 0.60
82.00 2.57
94.96 0.33
87.71 + 1.14
4-bit Oscillations
50.72 + 0.17
83.74 = 0.59
94.82 + 0.51
87.08 + 0.72
Table 2: Performance comparison CIFAR-10 dataset. Results show classification accuracy for MLP5, ResNet-18, and Tiny ViT across different quantization approaches and bit-widths. Models trained from scratch are marked FS, and fine-tuning experiments are marked FT. FT experiments are based on models pre-trained on ImageNet-1k. In all cases oscillations followed by quantization of the weights matches QAT accuracy. Results are means and standard deviations over random seeds. PTQ results are from the FP32 baseline.
Beadning-Fro-SScraa (FS): For the MLP5 architecture, we used learning rate of 10-3 and regulariza- tion parameter A=1. The ResNet-18 was trained with a learning rate of 10-3 and X=0.75 (sce Appx. A.2 for our hyperparameter selection). We modified the ResNet-18 architecture by replacing the input layer with smaller 3 x kernel and adapting the final layer for 10-class classification of both ResNet-18 and Tiny VIT. Training proceeded for maximun of 100 epochs with early stopping triggered after 10 epochs with out improvement in validation performance. For quantized models, monitored the quantized validation accuracy at the larget bit precision, while for the baseline, we tracked floating-poin poinn accuracy.
Fine-tuning (FT): We fine-tuned two ImageNet-Ik (Deng et al., 2009) pre-trained models on CIFAR- 10 and Tmny-ImageNet: a Tiny ViT (learning rate: 10-4, À E (170.75.00 0.5) depending on the bit) and a R8sNet-18 (learning rate: 10-3, E (1,0.75, 0.5) depending on the bit). To maintain compatibility with the pre-trained architectures, we up-sampled both CIFAR-10 and Imagentt images to 224 224 pixels. The parameter selection process for Tiny ViT is detailed in Appendix A.2. Fine-tuning continued for up to 200 epochs CIFAR-10 10 and 50 epochs for Tiny- ImageNet, with early stopping after 30 epochs without improvement, using the same accuracy metrics as training from scratch.
Quantization: We implemented weight quantization using a per-tensor uniform symmetric quantizer as defined in Eq. (1). PTQ is applied in its most minimal form, by simply quantizing the weights without any calibration. QAT is used as defined (5). The quantization range was determined by computing minimum and maximum values per layer. In our implementation of ResNet-18 (11M paramcters) all layers except batch normalization were quantized, covering 99.96% of parameters. For Tiny VIT (5.5M parame- applied to MLP, Self- Attentttnn and key-query-valie projection layers, encompassing
ters) quantization 97.18% of parameters. And lastly for the MLP5 model all layers were quantized. For Imag-ImageNet models are trained at 3 and 4-bit precision only.
5.2 Results
The performance on the two datascts in training-frome www.---rrocchhhcce and fine-tuning settings is presented in the following sections, along with the observations about cross-bit generalization.
5.2.1
Performance on CIFAR-10
Training from Scratch: Table 2 (A) shows the results from training MLP and ResNet-18 from scratch
on the CIFAR-10 dataset. Doing only regularization with Eq. (19) demonstrates improvements compared to the PTQ baseline. More importantly, it also matches the performance of QAT at bit widths of 3 and 4.
For both models we see that at 3-bit and 4- bibi using the R^ regularizer from Eq. (19) exhibits similar performance as QAT but with less variability. With both models, QAT and R^ regularization are competitive with the full-precision baseline. Notably, both R2 regularization and QAT significantly outperform PTQ when applied to the full-precision baseline.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Model
Train bit 4 Eval. bit
FP32
Ternary
3-bit
4-bit
8-bit
Baseline (PTQ)
88.50 + 0.64
10.01 + 0.01
10.28 1: 0.48
35.56
9.05
88.45 0.64
3-bit QAT
10.01 i 0.04
17.42
4.96
16.56
3-bit Oscillations
87.86 + 0.42
20.19 + 10.74
84.94 + 1.50
87.56 = 0.38
87.86 + 0.42
ResNet-18
4-bit QAT
87.76 1.13
10.13 + 0.29
82.08 + 6.25
87.71 = 1.14
87.76 + 1.12
4-hit Oscillations
87.85 0.49
11.91 + 0.87
85.57 + 1.10
87.08 +: 0.72
87.87 : 0.49
Baseline (PTQQ
96.11 0.31
9.39 + 1.11
11.56 1: 1.99
21.37 1: 5.33
96.03 1: 0.34
3-bit QAT
86.94 # 0.91
19.78 + 6.04
88.13 + 0.60
86.69 + 0.62
86.95 + 0.89
3-bit Oscillations
96.47 0.11
9.48
1.64
88.68 + 1.08
95.35 0.18
96.50 0.11
Tiny Vir
4-hit QAT
95.14 0.29
11.11 + 1.84
59.86 70.23
19.95
94.96 + 0.33
95.13 +: 0.28
4-bit Oscillations
96.54 0.09
11.90 + 1.29
12.75
94.82
0.51
96.55 0.09
Table 3: Cross-bit evaluation of pre-trained ImageNet-1l models fine-tuned on CTFAR-10. Grey background is the target-bit accuracy. Models are trained using different quantization methods (QAT and ours) and bit- widths (3-bit, and 4-bit), then evaluated across various bit-widths ranging from lernary to FP32. Results are means and standard deviations over 5 random seeds. All significant differences between QAT and Oscillations are shown in bold face.
Fine-tuning: Table 2 (B) summarizes the lest accuracies for fine-tuning using our RA regularization and QAT ResNet-18 and Tiny VIT architectures with CIFAR-10 and Tiny-ImageNet. The observations are roughly in line with the results observed for training from scratch in the previous section.
For CIFAR-10 as with the case for training from scratch, with both ResNet-18 and Tiny Vit, we see an increase performance compared to PTQ when using the regularization in Eq. (19). Regularization with R^ and QAT show comparable performance when quantized at 3 bits and bits, while achieving test accuracy close to the full- full-precision model at 4-bits.
Performance comparison and related discussions for ternary quantization are presented in Appendix A.3.
Robustness to Cross-bit Quantization: In this experiment we evaluated the robustness of oscillations- only and QAT lowards quantization at bit widths different from the ones used during training.
When using the RA regularization approach, we applied the regularization term with the training bit width during training and applied PTQ after training at a different quantization level. For QAT we trainod using the training bit width and afterwards applied PTQ to the latent weights. For each method we also evaluated the corresponding model without PTQ, directly using the latent weights for inference (reported as FP32).
In Table the results from the experiment are reported. A first observation is that the models produced by RA regularization consistently achieve nearly full-precision accuracy when quantized at 8-bit or when used without quantization, irrespective of the quantization level used during training. This contrasts with QAT, which produces a viable 8-bit or full-precision model only when trained with at least 4-bit.
Furthermore we see that regularizing using Eq. (19) mostly maintains performance when trained at 3 or 4-bit and quantized at bit level of 3 or 4-bit. QAT also achieves this for Tiny ViT but for ResNet, the accuracy of QAT trained at 3-bit and quantized other bit widths is barely above random guessing.
Regarding training with ternary quantization, we see that R^ regularization produces models that achieve near full-prccision performance for ResNet when quantized at 3-bit or higher. Ternary training for ViT is somewhat peculiar in that it fails to produce model that is viable when quantized to lernary, whereas the performance of the resulting models starts to show a high level of variability at 4-bit and finally reaches close to full-precision accuracy at 8-bit. In contrast, for both ResNet and ViT, the performance of QAT degrades completely random guessing when trained with ternary quantization and evaluated at any other quantization level.
9

---

Published in Transactions on Machine Learning Research (11/2025)
Quantization
Tiny VIT (FT)
ReaNet-18 (FT)
Baseline FP32
67.17 +: 0.67
62.93 0.55
PTQ
0.38
0.51 1
3-hit QAT
44.29 + 0.49
54.08 + 0.52
3-bit Oscillations
44.62 = 2.47
49.34 0.76
4-hit. PTQ
11.02 +: 2.11
20.02 1: 4,80
4-bit QAT
60.61 = 0.16
68.31 0.19
b-bit Oscillations
60.54 # 0.37
57.26 + 0.33
Table 4: Accuracy on Tiny-ImageNee datasct. Mcan and standard deviation over runs. The models is fine-tuned for 50 epochs on the pretrained ImageNet models. PTQ results is from the FP32 baseline. In both the 3 and 4 bit case, oscillations followed by quantization of the weights matches QAT and is noticeably above the PTQ baseline which has neither oscillations nor QAT.
Model
Method / Eval.
Ternary
3-bit
4-bit
8-bit
Baseline PTQ
62.93
0.35
0.50 + 0.00
0.51 4: 0.03
20.02 +: 4.80
62.83 :: 0.43
3-bit QAT
50.81 +: 1.85
4.51 +: 1.00
54.08.1 2.39
49.76.1 1.82
50.85.1: 1.85
ResNet-18 (FT)
3-bit Oscillations
56.67 0.01
1.48# 0.10
49.34 0.76
55.904 0.18
56.684 0.03
QAT 4-bit Oscillations
56.57 1.59
0.654
39.65
58.311 0.19
56.6611
61.58= 0.57
0.53# 0.02
30.10 = 4.04
57.26# 0.33
0.52 61.584
Baseline PTQ
67.17.L 0.67
0.49: 0.05
0.58.L 0.16
11.02.1 2.11
67.06.1 0.69
3-bit QAT
19: 1.18
1.73# 0.05
44.29 0.16
36.024: 2.11
39.18 0.69
Tiny VIT (FT)
3-bit Oscillations
56.75 4.11
1.51 0.86
44.62 2.47
56.221 4.00
56.78.1 4.07
4-bit QAT
59.75# 0.73
0.49# 0.02
34.42# 1.93
60.61 0.16
59.73+ 0.80
4-bit Oscillations
65.58= 0.29
0.541 0.09
22.26#
1.76
60.54 0.37
65.60+ 0.31
Table 5: Cross-bit bit evaluation of pre-trained ImageNet-1k models fine-tuned on Timy-ImageNet.. Grey back- ground is the target-bit. accuracy. Models are trained using different quantization methods (QAT and ours) and bit-widths (3-bit, and 4-bit), then evaluated across various bit-widths ranging from ternary to FP32. Results are means and standard deviations over random sceds. All significant differences between QAT and Oscillations are shown in bold face.
5.2.2 Performance on Tiny-ImageNet
Fine-tuning: Table 4 summarizes the test accuracies for the T'iny-ImageNet dataset. Here we observe the same tendency as with CIFAR-10; oscillations provides a significant increase in accuracy compared to the PTQ baseline. While for the Tiny ViT model R, regularization sufficient. to recover the quantized accuracy of QAT in both the and 4-bit case, for ResNet18 there is a degradation in accuracy at 3-bit.
Robustness to Cross-bit Quantization: In Table 5 we see the cross-bit results from the Tmag-ImageNet experiments. As with CIFAR-10 we note that the models produced by RA regularization achieves better cross-bit performance at bits higher than the target bit. Though we do note change in the cross-bir bit behavior; the cross-bit results for 3 and 4-bit is generally lower and not as close tot he FP baseline as in the CIFAR-10 case, yet still there is a significant difference between QAT and Rx regularization.
6 Discussion
We have shown that training with weight oscillations induced via R2 regularization is sufficient in most cases to maintain performance after quantization for ResNet and Tiny VIT. The primary effect of our regularizer is to push weights toward quantization thresholds, which results in weights that are robust to quantization. This "Ibreshold-pushing" dynamic naturally leads to two observable phenomena: weight, clustering and oscillations. This begs the question whether weight oscillations are also a necessary part the QAT training process. Indoed, some previous work already points towards this. There are examples claiming that both dampening and/or or freezing of oscillations too early in the training process is detrimental to performance after quantization (Nagel et al., 2022; Han et al., 2021). And in other case presented in Liu et al. (2023), freezing
10

---

Published in Transactions on Machine Learning Research (11/2025)
only the low frequency oscillating weights improves performance. This suggests that weight oscillations are both a necessary and sulficient part of QAT, at least in the early phases of the training process. This further supports our hypothesis that oscillations in QAT have a positive effect on quantization robustness overall.
Yet we do note deviations from QAT when regularizing with Eq. (19): QAT outperforms R^ regularization at ternary quantization (Appendix A.3), whereas R^ regularization outperforms QAT cross-bit accuracy for the ternary and 3-bit case. In A.5, we see how it seems that the cross-bit bit performance for QAT is upper- bounded by the target-bit performance, which might explain the subpar QAT performance at cross-bit compared to R, regularization which seems bounded by the full-precision accuracy.
Limitations and Future Work Our theoretical analysis was performed using the toy model in Section 3, and the regularization term is motivated using this analysis. We expect other effects that are not entirely captured by this analysis to be part of QAT. This is explored further in Appendix A.1, where we show how the second term in Eq. (9) is not. zero in the gradient when there are multiple layers.
The second term in Eq. (9) is closely related to the dscilatiios-dampening methods such as the one presented in Equation 6 in (Nagel et al., 2022), which works by adding linear pull towards the quantization level. In Appendix A.1 we show how for the linear case with two weights, the second term is no longer zero in the gradient and as such QAT then consists of two components; one that creates oscillations and one that dampens them. From this we can see that oscillations alone do not capture the full dynamics of QAT as analyzed in linear models.
The work presented here investigales the oscillations that arise when preparing deep neural networks for quantization using QAT. By characterizing these oscillations, we gain deeper insight into the dynamics of QAT, which can be leveraged to enhance its effectiveness and, consequently, the efficiency of quantized models. Improved efficiency translates into reductions in memory, computation, and energy demands, facilitating the deployment of high-performing quantized networks on low-power edge devices. Some of these applications can have broader positive impact (for example, medical electronics) and also negative impact (for example, surveillance systems or weapons).
Broader Impact:
7 Conclusions
Based on the analysis of linear model we hypothesized that weight oscillations during training in deep neural networks make the model robust to quantization akin to QAT. In Sections 3 and 4 we explain how training with QAT and STE leads to oscillations and propose a regularizer that encourages this oscillating behavior. We confirm that as we increase the strength of the regularization, and empirically observe the appearance of clustering together with oscillations.
Finally, experimentally confirm that the regularizer indced leads to consistent robustness towards quan- tization for 3-bit and 4-bit quantization levels. Our oscillations by regularization approach achieves com- parable performance to QAT above ternary quantization when quantizing to the target-bit seen during the optimization. Furthermore, we also observe that it shows increased robustness compared to QAT in cross-bit quantization with quantization levels higher than the target-bit used in the quantizer during training. All this is evidence favor of our hypothesis.
Our insights on weight oscillations and their role in quantization robustness open new horizons for model quantization approaches which usually build on the idea of aligning weights at quantization levels the opposite of what seems to be the core dynamic in QAT. The regularization approach especially creates interesting possibilities for bross-bit robustness, potentially making the regularization method more appcaling than QAT when the goal is to deploy or release a single set of model weights that can work across different bit widths or maybe even quantizers. While the regularizer used in our experiments should be viewed as an initial step, we expect that quantization robustness could be further improved by developing oscillation- inducing methods that are adaptive to different learning rates, layer statisties or phases of the training process.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgments: Authors thank Tong Chen, Jákup Svødstein, Sophia Wilson, Sebastian Hammer Eliassen, Shivam Adarsh and members of SAINTS Lab for useful discussions throughout.
JW, BP and RS are partly funded by European Union's Horizon Europe Research and Innovation Action pro- gramme under grant agreements No. 101070284, No. 101070408 and No. 101189771. RS also acknowledges funding received under Independent Research Fund Denmark (DFF) under grant 4307-00143B.
References
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprinl arXin:1308.3432 2013.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Learning sparse low-precision neural networks with
learnable regularization. TEFE Access, 8, 2020.
Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo quan-
tization noise. Transactions Machine Learning Reseorch, 2022. ISSN 2835-8856. URL https:
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition (CVPR), pp. 148-255. Icco, 2009.
Kartik Gupta and Akshay Asthana. Reducing the side-effects of oscillations in training of quantized yolo networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2452-2461, 2024.
Tiantian Han, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Improving low-precision network quantization via bin regularization. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 5261 5270, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEFE conference on computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Kazutoshi Hirose, Kota Ando, Kodai Ueyoshi, Masayuki Ikebe, Tetsuya Asai, Masato Motomura, and Shinya Takamaeda-Yamasn Quantization error-based regularization in neural networks In International Con- ference on Innovative Techniques and Applications of Artificial Intelligence. Springer, 2017.
Xijie Huang, Zhiqiang Shen, Pingcheng Dong, and Kwang-Tiiee Ting Cheng. Quantization variation: A new perspective training transformers with low-bit precision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL sttps://openreviess not:/formmid--Hioooooo
Pei-Hen Hung Chia-Han Lee, Shao Wen Yang, V Srinivasa Somayazulu, Yen-Kuang Chen, and Shao- Yi Chien. Bridge deep learning to the physical world: An efficient. method to quantize network. 2015 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1 IEEE, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient. aateger-rritmmetic only inference. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2018.
Diederik P Kingma. Adam: A method for stochastic optimization. arXio preprint arXiv:1412.0000 2014.
Raghuraman Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint Xrxi::1866.0888 08342, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https://api. semanticscholar org/ Corpus 16664790.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient. non-uniform
discrctization for neural networks. In International Conference on Learning Representations, 2020. URL https://openreviier nww/ffrummmeeeee -dt.rrr2nndd--gKX224DD
Shih-Yang Liu, Zechun Liu, and Kwang-Ting Cheng. Oscillation-free quantization for low-bit vision trans- formers. In International Conference on Machine Learning, pp. 11813-21822 PMLR, 2023.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yclysci Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. while paper on neural network quantization. arviv preprinl urXiv:2106. 08295,, 2021.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-wware training. In International Conference on Machine Learning, pp. 16318-1633/ PMLR, 2022.
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJONN), pp. 1 8. IEEE, 2022.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision, pp. 68 85. Springer, 2022.
Yunshan Zhong, Yuyao Zhou, Fei Chao, and Rongrong Ji. Mbquant: A novel multi branch topology method for arbitrary bit-width net work quantization. Pallern Recogrilion, 158:111061, 2025.
A Appendix
A.1 Two-layer with Single Weights
Consider a linear model f(x) = wyW11, with weights wi, w2, input x, and target E R. The quantized version of this model defined as (x) = q(w2))((1)), where q(-) the quantizer from Eq. Equation (1).
The quadratic loss for the model is given by
c(f())) [w2wrr= 2)²
The difference compared to full-precision optimization then given
5r = (()(())) L(((x))
(21)
[(((w)))(w2)) - 2)2 (w2www - v)²1
(22)
(((w)))(w1<<) (wzwir)² 22((((2))(w1)) -w2w111))) w2w(x)
(23)
Ia (((2))) ((( wrzwl + yo w2w0 ((w2)g(m))
(24)
The loss difference decomposes into:
5 (q(w2)2 uz
+
yo www - (((2))((u))
(25)
Quadratic term (Oscillator)
Linear (Oscillation Dampener)
13

---

Published in Transactions on Machine Learning Research (11/2025)
Taking the derivative of L w.r.t Wi:
0ôc a -
(26)
owu owi
a
[
)²((2=)² (q(w)))
(q(wz)
-wsw))) yr wywi q )a((
(27)
Owi
2
oq(wi) w2wu/ + yot w2 ((
3 (1))
x²² (2((( ((w))
(28)
Owi
Swe
Using the STE approximation from Eq. 5, get:
a8c x² (q(u))) (((1)) wzwr]] + yz wo g
(29)
Ow1
We note that the linear term is no longer zero in the gradient and thus for a model consisting of two single weight layers we see that there are additional effects from QAT other than oscillations. Furthermore, we would like clarify that even though the two layered model under consideration does not have explicit non- linear activation functions, it cannot be reduced single layer model. This is because of the non-linearity of the rounding operalion.
23.0
12.0
Figure 4: We repeat the toy model experiments, but this time with two weights, taking into account that the linear term is no longer zero in the gradient. We notice at epoch 15 and 18 where the prediction of the quantized model is greater than y, the effect of the terms flip for w2.
A.2 Hyperparameters
A.2.1 ResNet-18
In Fig. and Fig. 6 we see the results of the hyperparameter search over different learning rates for a ResNet-18 model. There clear trend of seeing the best performance at a learning rate of 10-3 We note that interestingly there is a comparable performance for wide range of As, indicating that is the presence of oscillations which is important for quantization robustness, and not the exact frequency of oscillations.
A.2.2 Tiny ViT
In Fig. 7 we note how also the Tiny Vit seems allow for a wide range of As even though we this time note that X 1 performs significantly better than the others.
A.3 Ternary Quantization
Performance comparison for the ternary quantization for different models on CIFAR-10 is reported in Table
While both QAT and oscillations improve the PTQ baseline significantly, oscillations degrade compared QAT, especially for the Tiny VIT. This is in line with previous literature (Liu et al., 2023), where transformers
14

---

Published in Transactions on Machine Learning Research (11/2025)
Accuracy Regularization Lambda
80
70
Test Aucuracy (%)
60
50
20
10
10-1
10-1
10a
10a
10
Regularization Lambda (log scale)
Figure 5: Mean over 3 runs of the best test accuracy for different lambdas. Fine-tuning pretrained ResNet- 18 on CIFAR-10 10 for 50 epochs. Quantizer is set to 3-bit and 10-3 learning rate and 100% of the training data is used.
Learning rate comparison 3-bit regularization
60
X
3-bit (%)
Ternary (%)
daiidaion Accuracy
50
0.25
68.77 1 0.19
47.85 1. 5.51
0,50
4.83
0.75
70.08
= 0.40
46.86 1 3.01
1.00
66.20
= 4.05
17.33 = 2.06
30
1.25
09.31
0.32
43.14 + 0.02
1.50
68.96
0.30
46.73
1 3.91
20
1.75
69.92 - 0.11
47.02
: 4.19
LR: 0.0001
0.001
0-
30
Epoch
Figure 6: Mean over 3 runs of the best validation accuracy for different lambdas. Training ResNet-18 from scratch. Both ternary and 3-bit is at 10-3 learning rate and 50% of the data used for training. The plot shows three learning rates, where we evaluate with the As for each learning rate (shown in the table on right). The colored background covers the range between the maximum and minimum value of the quantized validation accuracy with the given λs.
have been identified as especially sensitive to oscillations. If oscillations are fundamental to QAT, this still leaves the question of why QAT achieves superior ternary performance. Appendix A.1 offers an indication the gradient of the two-layer toy model, in Eq. 25 we note that the
of the underlying reason. Looking second term is no longer zero in the gradient. This term bears close resemblance to existing formulations of oscillation dampeners, such as in Eq. 6 of Nagel et al. (2022). These works by pushing weights towards their nearest quantization level, thereby dampening or mullifying the effect of any oscillator (which pushes weights lowards their nearest threshold). We therefore speculate that the main component of QAT still oscillations, but that QAT also has an inherent dampening mechanism. Given that in a uniform quantizer, the quantization error incrcases exponentially as we decrease the bits the quantizer, ternary quantization' 's large error magnitude makes the absence of dampening particularly detrimental, resulting in sub-optimal quantized performance compared to QAT.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Different
80
0.75
Accuracy
3-bit (%)
Ternary (%)
60-
0.01
18.85
0.5
85,21
15.10
0.75
87.68
40
2.0
89.31
14.16
2.5
13.70
m
20 40
60
80
100
120
140
Epoch
Figure 7: Validation accuracy at different values and the corresponding best validation accuracies for 3-bit and 2-bit configurations for single run. Learning rate is set 10 for fine-tuning. For the 2-bit we test higher À but still for the low of 10
improvement in accuracy. We note how all the As lies close to each other, except
Quantization
MLP5 (FS)
ResNet-18 (FS)
ResNet-18 (FT)
Tiny (FT)
Baseline FP32
51.43
0.39
83.26
1.07
83.26
1.07
96.11
0.31
Ternary PTQ
10.00
0.02
10.00
0.01
10.00
0.01
9.39
1.11
Ternary QAT
49.20
1.34
79.62
6.42
77.02
7.57
73.53
0.77
Ternary Oscillations
36.49 + 0.51
61.50 + 1.82
44.59 + 3.30
13.51
1.32
Table 6: Performance comparison with ternary quantization on CIFAR-10 dataset. Mcan and standard deviation is over 3 runs. The models is fine-tuned for 50 epochs on the pretrained ImageNet models. PTQ results is from the FP32 baseline. For both oscillations only and QAT the PTQ baseline. Yet oscillations degrade significantly compared to QAT, especially for the Tiny Vit. FS: Train from scratch. FT: Fine-tuned.
significant improvement over
A.4 Epochs and cross-bit robustness
There an interesting interaction between number of epochs trained and robustness both of our method and QAT. We note how QAT converges first for the target-bit and then over time also converges for the 4 and 8-bit. Additionally not the case for oscillations only shown in Fig. 8.
that QAT seems upper-bounded by the Larget-bil, performance, while this is
A.5 Convergence behavior during oscillations-oni optimization
Fig. 9 shows the convergence behavior of the full precision weights and the quantized weights target-bit. We note how the Tiny ViT displays peculiar convergence behavior, where the accuracy breaks, only to go up again. In the Tiny Vit model regularize the self-attention layers. It is already noted in Liu et al. (2023) that ViTs are especially vulnerable to oscillations in the query and key of self-attention layers, which might be related to the convergence behavior observed when regularizing with Eq. 19.
A.6 Weight Distribution and Oscillation Frequency
16

---

Published in Transactions on Machine Learning Research (11/2025)
100
100
90
80
Accuary (%6)
Accuracy (9%
60
D-DI
6-D2
3-2
50
Tummury
Temory
v MW
30
20-
20-
17
Epoch
Epoch
Figure 8: Left is the validation accuracy during training of a VIT with QAT at different bits, right is for our regularization. Both QAT and regularization is trained with a 3-bit quantizer. We note how the order of convergences for cross-bit changes between QAT and our model and that QAT cross-bit robustness especially depends on number of epochs trained.
Quant Val accuracy
100
23
space
Figure 9: (right) Convergence behavior of ResNet-18. (left) Convergence behavior of a Tiny VIT with regularization with a 3-bit quantizer. Note the peculiar behavior of the orange line, which the validation accuracy on the target-bit bit performance. The performances cycles between 90% and 10%, while the full precision accuracy (the model evaluated without quantized weights) stays some-what stable.
QAT
10
Oscillation
Weights
Oscillation
Weights
Oscillation
Weights
c_nvv
2_0___ __0__onnv
Figure 10: Oscillations count and weight distribution side by side across layers and regularization settings.
Each pair (left: oscillation, right: weights) corresponds the same configuration. Note that weight distribution plots do not share and y axis.
17


Reviews:
[{"review_id": "56gx1H96FV", "reviewer": "Reviewer_qnLH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper argues that the performance improvements seen with Quantization Aware Training (QAT) result from oscillations that QAT induces in the training dynamics. This is argued through analyzing the difference in gradients, in a 1D linear model, that one observes with versus without QAT. This is further supported by experiments where the authors introduce a regularization term during regular training (i.e. without QAT) that causes similar oscillations to occur and that leads to similar performance results as QAT. The authors also note that the oscillations appear, in certain cases, to lead to more robust generalization than QAT across a range of different quantizations that the models were not trained on.\n\n## Strengths\nI find the main strengths of this paper to be the results, and presentation. The result that oscillations seem to reproduce some key features of quantization is quite interesting, and the approach of using regularization to show this in experiments seems novel and informative. Similarly, the result that oscillations alone lead to better cross-bit generalization in many cases is interesting and seems like it could have useful applications. In terms of presentation, the structuring of the paper is largely easy to follow, and most of the figures are easy to understand.\n\n## Weaknesses\nI find four main weaknesses with the paper. I think that they should be able to be resolved with edits and a little bit more work.\n1. My main concern with the work is that certain aspects of the results seem to be explored in a cursory fashion. In particular, it would be nice to see more than just the input layer data in Fig 1, 3. See Requested changes “elaboration on certain results”, below, for more details.\n2. It would be nice to have some measures of statistical significance for a subset of the results. See “statistical significance” section below for more details.\n3. Portions of the background material could be presented a little more clearly. See “need for elaborated background” section below for more details.\n4. There are some issues of clarity with the writing, including grammar, spelling, and weird wording issues, at a number of points. This is important to resolve, but I think this should be fairly easily addressed. See “clarity” section below.\n\nAdditionally: there are some more minor issues with tables and figures that I mention in the “extra table and figure issues” section.", "claims_and_evidence_yn": "No", "claims_explanation": "I believe that most of the paper's claims are well supported. I have only answered \"no\" here because of two reasons that are mentioned in the requested changes, below. Specifically, I would (1) like to see data in Fig 2 and 3 on more than just the first layer of the ResNet, and (2) like to see statistical tests on Fig 3 showing that the difference in oscillations is significant.", "audience_yn": "Yes", "audience_explanation": "Quantization is an exciting frontier for improving computational benefits of machine learning models, which has potential for high impact particularly in light of growing model size. This paper provides some exciting results about quantization aware training with the straight through estimator. For this reason I think the paper is of interest to a subset of the TMLR community.", "requested_changes": "I have organized the requested changes into sections based on the main weaknesses. I have added “**C**” at the end of changes that I view as critical, for my review, and “**S**” after ones that I view more as suggestions.\n\n# Elaboration on Certain Results\n- In Fig.2, 3 the plotted data is from the first convolutional layer of a ResNet. Why the first layer? Do you only find these results to hold in the first layer? If so, perhaps it is because of the single layer analysis in the 1D model. Perhaps comment on this, and, critically, please also show data from other layers too. **C**\n- Second last paragraph of Discussion: “where we show how the second term is not zero”. It’s not clear here what the second term you are referring to is. Could be useful to elaborate. **C**\n- Appendix: “Additionally because of the non-linearity of the rounding operation, even with the absence of a non-linear activation function, we can no longer reduce the model to a single weight.” What do you mean by this? **C**\n- Appendix, Figure 7: why does validation accuracy drop at around 120 epochs? Does oscillation make learning unstable? Should discuss this **C**\n\n# Statistical Significance\n- Fig 2, Fig 3: would be nice to include some test of statistical significance to compare whether the displayed distributions are significantly different from each other. Otherwise cannot say that, e.g., QAT leads to significantly here oscillations than other conditions **C**\n- Table 2: where are the SDs in this table? Could be useful to include statistically significant differences. At the very least, SDs should be shown **C**\n\n# Need for Elaborated Background\n- (intro, paragraph 3): Straight-Through Estimator (STE) mentioned for the first time. Could be worth explaining in brief. **S**\n- (section 2): could you explain whether the floating point weight value is maintained during training with QAT, or whether, after each weight update, the current weights are set to the closest quanta? **C**\n- (section 2 paragraph 2): what is a max-range scale factor? If this is defined in Eq.2, maybe mention eq.2 earlier? **S**\n- (section 2 paragraph 3): what is the “usual clamping operation in the quantizer”? **S**\n- (3rd paragraph of “Related work” section in section 2): what does “at individual bits” mean here? **C**\n- (footnote on page 4): could you elaborate on what clamping means here? **C**\n\n# Clarity\n- weird wording (intro, paragraph 1): “grown increasingly powerful at many tasks” => grown increasingly successful at solving difficult modelling tasks” **S**\n- grammar/wording (intro, paragraph 2): the first sentence in this paragraph seems grammatically suspect. Could maybe solve with a colon perhaps **C**\n- word choice (intro, paragraph 2): “degradations” => “reductions” **C**\n- writing flow: (intro, paragraph 3): “quantization” mentioned 3 times in first sentence; suggest replacing with another word for better flow of text **S**\n- redundancy: (intro, paragraph 3): “Unfortunately, QAT brings its own mysteries” seems redundant. For example, could be removed and replaced by starting the next sentence with “however”. **C**\n- redundancy: (intro, paragraph 3): “In particular, QAT often exhibits a seemingly strange phenomenon: oscillations in the weights.” => “In particular, QAT often exhibits oscillations in the weights during optimization, while regular training does not.” **C**\n- vague wording (intro, paragraph 4): “What if we had this view on oscillations in QAT backwards?” => “what if QAT oscillations are actually beneficial for training”. If this change was made, one would likely want to re-write the paragraph a bit **S** \n- writing issues (section 2 paragraph with “Quantization-Aware Training” heading): by “fundamentally” do you mean “typically”? Also, in the same paragraph, you don’t need to say “variants of QAT” twice in the first sentence. **C**\n- redundancy (3rd paragraph of “Related work” section in section 2): this paragraph seems to repeat stuff mentioned earlier in the intro. Could be worth checking to see if redundancy can be removed **S**\n- redundancy (3rd paragraph of “Related work” section in section 2): “In short, …” => remove. **S**\n- redundancy (equation 6): this can be removed **C**\n- (third last paragraph of section 5.2.1): it is written “In Table 6 shows the results from the experiments”. Note the grammar error. **C**\n- Second last paragraph of Discussion: “is not zero in the gradient, when there are multiple layers” => remove comma. **C**\n- Second last paragraph of Conclusions: “all this being evidence of our hypothesis” => “all this is evidence in favour of our hypothesis”. **S**\n- Acknowledgements section: you will probably want to update this section. **S**\n- Appendix: “we see that there is additional effects from QAT” => “we see that there are additional effects from QAT” **C**\n- spelling issue (figure 6 caption): “for each learning ratue evaluate” **C**\n- spelling (figure 7 caption): “but still see no improvemenet in accuracy” **C**\n\n# Extra Table and Figure Issues\n- Third last paragraph of section 5.2.1: it is written “In Table 6 shows the results from the experiments”. Shouldn’t this be “table 2? Regardless, table 2 does not seem to be referenced in the body of the paper. **C**\n- Last paragraph of section 5.2.1: should ref table 6 here I think. **C**\n- Appendix, Figure 4: is \"target\" just \"y\"? and \"q pred\" the predicted value? If so, maybe label them \"y\" and \"y predicted\" or something like this. **S**\n- Appendix: “In Fig. 5 and Fig. 6 we see the results of the λ hyperparameter search over different learning rates for a ResNet-18 model. There is a clear trend of seeing the best performance at a learning rate of 10−3.” Here, it would be nice for the order of figures to match the order of mention in the text. I.g. should switch the order of the figures or should mention regularization tuning before learning rate. **S**\n- Should put tables in appendix that are not referenced in the appendix either all at top or at bottom, to avoid mixing them up with the appendix text, in my opinion. **S**", "broader_impact_concerns": "It could be useful to touch on both positive and negative impacts of this work that stem from potential improvements to quantization of neural networks. To give some examples: in terms of benefits, quantization improvements seem likely to improve computational efficiency, potentially reducing energy footprints. This could be useful given the huge energy consumption of large deep learning models. In terms of downsides, improvements in computational efficiency could result in larger models being implemented on smaller devices. This could be applied, for example, to drones used for military and surveillance purposes, resulting in human rights infringements and violence.", "additional_comments": "**Important:** Note that, while I have a strong deep learning background, I am not very familiar with quantization. For this reason I cannot comment on how this work fits into the broader literature, but only on the soundness of the results and the clarity of the exposition. Please take this into account when considering my review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OQIhGqmNC7", "reviewer": "Reviewer_xy6A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies quantization aware training (QAT), and in particular investigates the empirical evidence that weights tend to \"oscillate\" around a quantization threshold during training. The purpose of this paper is to give a sense to these oscillations, and to stand with the perspective that they are a natural (or even beneficial) aspect of QAT. To do so, the Authors analyze a univariate linear regression model to explain this behaviour analytically. Then, from the derivation they semi-heuristically define a regularization term that could be added to the loss to recreate the bias of QAT. Finally, the authors investigate empirically the effects of such regularizer, finding positive results on a discrete variety of tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is well written, self contained, and the technical results are correct. The numerical evidence seems solid.", "audience_yn": "Yes", "audience_explanation": "This work look into QAT, which is a primary sub-area of research in deep learning. This work brings solid evidence to clarify the empirical phenomenon of oscillations during such training methods, and I therefore presume many individuals in the community would appreciate this reading.", "requested_changes": "**Minor**\n\n- STE as it is introduced in the first section might be a bit obscure to people not in the field (it was to me). Maybe it is worth it to just skip mentioning it (it does not seem so necessary at this point) or to add few words to describe what the algorithm look like, adding a cross reference to its formal definition later in the paper (same comment goes for the mention at point 1 of the contributions).\n\n- Right after Eq. (3), $\\Delta$ is natural to be interpreted as a vector, if one thinks about a model with more than one parameter being quantized. Then the notation $| \\Delta |$ after this equation is a bit \"sloppy\". Maybe the Authors can replace it with $| \\Delta_i |$ to mean each entry of $\\Delta$, and remark that $q$ is meant to be taken component-wise?\n\n- As somebody that was not aware of STE, I struggled a bit at first in understanding the algorithm. In particular, the fact that at every iteration $w$ is not a quantized parameter, as it is stored as \"continuous\" all along, and the quantization is used only in the forward passes during training and to release the final model (is this correct?). Can the Authors perhaps add few more line of explanation about this?\n\n- Maybe the authors can remark at the level of the contributions that the toy model is univariate. This is a quite simplifying setting, and maybe it is fair to stress it before.\n\n-  The proposed regularizer is in form different from what would give rise to the gradient difference in your toy model (Eq. (18) does not come directly from Eq. (8) if I understood directly). If this is indeed the case, can the Authors remark that this definition is heuristic?\n\n---\n\n**Major**\n\nIn the course of the paper, it seems there is remarked some sort of causal connection between \"oscillations\" and successful quantization. For example, see the sentence in the first paragraph of Section 6 \"weight oscillations are both a ncessary and sufficient part of QAT\", or (more directly) the title of the paper.\n\nPossibly, my concern stems from the fact that an \"oscillation\" is not exactly defined in the paper. For this reason, the empirical connection between successful quantization, clustering around the thresholds, and oscillations (as measured for the plots in Figure 3) remains qualitative, and it is not obvious to state that \"oscillations are good for quantization\", as there are other latent variables here.\n\nI am talking from the perspective of somebody that does not have much experience in this sub-field of research, so it might be possible that this nomenclature of things is simply well accepted by the community. However, I still believe the Authors could be more careful in drawing this connection, or at least to elaborate on it in the Section 6.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sAYLxsgquo", "reviewer": "Reviewer_pwtX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work shows that the oscillation effects in QAT training, contrary to conventional wisdom, can be beneficial", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments are rather extensive\n\nThe derivations are informal and to-the-point and are easy to understand", "audience_yn": "Yes", "audience_explanation": "This is a standard  machine learning / deep learning problem, so yes", "requested_changes": "Nothing that I spot", "broader_impact_concerns": "NA", "additional_comments": "This review is short because I do not see major problems with this paper, even the figures are pretty nicely done", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)