Learning under Quantization for High-Dimensional Linear Regression
Authors: Dechen Zhang
Junwei Su
Difan Zou
Venue: ICLR 2026 Poster (2026)
Keywords: quantization
generalization
linear regression
Primary area: learning theory
TL;DR: We provide a refined analysis on the excess risk of finite-step stochastic gradient descent for high-dimensional linear regression under a comprehensive range of quantization.

Abstract: The use of low-bit quantization has emerged as an indispensable technique for enabling the efficient training of large-scale models. Despite its widespread empirical success, a rigorous theoretical understanding of its impact on learning performance remains notably absent, even in the simplest linear regression setting. We present the first systematic theoretical study of this fundamental question, analyzing finite-step stochastic gradient descent (SGD) for high-dimensional linear regression under a comprehensive range of quantization targets: data, label, parameter, activation, and gradient. Our novel analytical framework establishes precise algorithm-dependent and data-dependent excess risk bounds that characterize how different quantization affects learning: parameter, activation, and gradient quantization amplify noise during training; data quantization distorts the data spectrum and introduces additional approximation error. Crucially, we distinguish the effects of two quantization schemes: we prove that for additive quantization (with constant quantization steps), the noise amplification benefits from a suppression effect scaled by the batch size, while multiplicative quantization (with input-dependent quantization steps) largely preserves the spectral structure, thereby reducing the spectral distortion. Furthermore, under common polynomial-decay data spectra, we quantitatively compare the risks of multiplicative and additive quantization, drawing a parallel to the comparison between FP and integer quantization methods. Our theory provides a powerful lens to characterize how quantization shapes the learning dynamics of optimization algorithms, paving the way to further explore learning theory under practical hardware constraints.

Published as a conference paper at ICLR 2026
LEARNING UNDER QUANTIZATION DIMENSIONAL LINEAR REGRESSION
FOR HIGH-
Dechen Zhang
Junwei Su
Institute of Data Science The University of Hong Kong dechenzhang@cconnet: hku. hk
School of Computing & Data Science The University of Hong Kong
junwei su@conneceeeeeeeeeeeeeoee hku. hk
Difan Zou
School of Computing & Data Science and Institute of Data Science
The University of Hong Kong
dzou Shku.
ABSTRACT
The use of low-bit quantization has emerged as an indispensable technique for enabling the efficient training of large-scale models. Despite its widespread em- pirical success, a rigorous theoretical understanding of its impact on learning per- formance remains notably absent, even the simplest linear regression setting, We present the first systematic theoretical study of this fundamental question, alyzing finite-step stochastic gradient descent (SGD) for high high-dimensional lin- ear regression under comprehensive range of quantization targets: data, label, parameter, activation, and gradient. Our novel analytical framework establishes precise algorithm-dependee terize how different quantization affects learning: parameter, activation, and gra- dient quantization amplify noise during training; data quantization distorts the data spectrum and introduces additional approximation error. Crucially, we dis- tinguish the effects of two quantization schemes: we prove that for additive quan- tization (with constant quantization steps), the noise amplification benefits from a suppression effect scaled by the batch size, while multiplicative quantization (with input-dependent quantization steps) largely preserves the spectral structure, thereby reducing the spectral distortion. Furthermore, under common polynomial- decay data spectra, quantitatively compare the risks of multiplicative and ad- ditive quantization, drawing a parallel to the comparison between FP and integer quantization methods. Our theory provides powerful lens characterize how quantization shapes the learning dynamics of optimization algorithms, paving the way further explore learning theory under practical hardware constraints.
and data-dependent excess risk bounds that charac-
INTRODUCTION
Quantization has garnered widespread attention an essential technique for deploying large-scale deep learning models, particularly large language models (LLMs) (Lang et al.] 2024 Shen et al. 2024). In line with this low-precision paradigm, a new frontier of research has emerged: quanti- zation scaling laws. which seek to formalize the trade-off's between model size, dataset size, and computational bit-width. Seminal work by Kumar al. (2024) treated bit- hit-width sure of precision. This was extended by (Sun et al. (2025), who established a more comprehensive scaling law for floating-point (FP) quantization (Kuzmin et al. 2022) by separately accounting for the distinct roles of exponent and mantissa bits. Going further, Chen et al. (2025) proposed uni- fied scaling law that models quantized error as a function of model size, training data volume, and quantization group size. Collectively, these studies provide rigorous understandings to guide the joint allocation of fixed compute or memory budgets across data size, model size and precision (quantization bit-width).
discrete mea-

---

Published as a conference paper at ICLR 2026
The empirical understanding of low- low-precision training has advanced rapidly, yet significant theory- practice gup persists. Theoretical research remains predominantly restricted to analyzing conver- gence guarantees on the training loss for quantized optimizers (Nadiradze et al. 202T Liu et al. 2023 Markov et al. [023 Xin et al. 2025). For example, Markov et al. (2023) derived convergence guarantee for the communication-effciiee variant of Fully-Shared PatalParallee distributed training under parameter and gradient quantization. While these studies offer crucial insights into optimiza- tion, they overlook a more fundamental question: how does quantization affect the model's 's learning performance? ? Specifically, a rigorous characterization of the interplay between quantization, model dimension, dataset size, and their joint effect on the population risk remains largely unexplored. A notable step in this direction is Zhang et al. (2022) which analyzed the generalization of quantized two-layer networks through the lens of neural tangent kernel (NTK). However, their work limited in three key aspects: it only considers parameter quantization; its analysis is confined to the lazy- training regime; and it fails to provide explicit generalization bounds terms of core parameters like sample size, dimension, and quantization error. These limitations restrict its applicability to modern low-precision training practices.
Motivated by recent theoretical advances in scaling laws (Lin et al. 2024 2025 Li et al.
2025
analyze the learning performance of quantized training using a nigh--imensional linear model. This model serves
powerful and well-established testbed for isolating phenomena like learning rate
and batch size effects (Kunstner & Bach 2025 Luo et al. [025] ZZhng et al., 2024b Xiao| 2024]
Ren al. 2025; Bordelon et al.) 2025). Its simplicity provides the analytical fiexibility necessary to derive precise relationships between generalization error and critical parameters such as dimension, sample size, and quantization error (or bit-width).
Our setting. In this paper, we consider SGD for linear regression under quantization. We first iterate the standard linear regression problem as follows:
1 min L(w), where I(w) 2 Fx,y (y - (w, x))²
w
Here H is the feature vector, H is some (finite d-dimensional countably infinite dimensional) Hilbert space, y E IR is the response, 'D is an unknown distribution over x and and w E H. is the weight vector to be optimized. We consider the constant step size SGD under quantization:
each
iteration t, an i.i.d. batch (with batch size B3 of examples (X,yye) Vr) E R Bxd x R is observed, and the weight w, [Rd is updated according to following quantized SGD algorithm.
1 1 W; = W.-1 + 7 (a(x.) Te
(e 2. (Q,,())))
1= 1=
(quantized SGD)
where y >> 0 is a constant stepsize, N is the number of sample batches observed, the master weights initialized at Wo. and Qa: Q1, Qp., Qa, Q. are independent general quantization operations for data feature, label, model parameter, activation and output gradient respectively. Notably, for theoretical simplicity, we assume all matrix operations (c.g., addition and multiplication) are computed in full precision, with quantization applied subsequently to obtain low-precision values. Then, we consider the average iterate the algorithm output, i.e., WN N EN-- w.. Without loss of generality, we assume the initial parameter is wo O.
The goal of this work is to characterize the learning performance of the quantized SGD via evaluating the population risk L(Wx), and more importantly, its relationship with the quantization error. Let w" = arg min L(W), we define the following excess risk as a surrogate of the population risk:
E(wn) L(W L(w").
(Excess Risk)
Our contributions. We perform novel theoretical study the learnability of the quantized SGD algorithm for high-dimensional linear regression problems. Our contributions are summarized as follows:
We perform systematic analysis and establish theoretical bound for the excess risk of quantized SGD. This bound is explicitly formulated as a function of the full eigen-spectrum of the quantized data feature covariance, sample size, and quantization errors (see Theorem 4. for details). Our results precisely reveal how quantization applied to different model components impacts learning performance: quantization of data distorts the spectrum of effective data covariance and introduces
2

---

Published as a conference paper at ICLR 2026
additional approximation error; while the quantization of parameter, activation, and output gradient amplify noise throughout the training process on the quantized feature space.
We analyze two standard quantization error models: additive and multiplicative, which conceptu- ally relate to the integer and FP quantization techniques. For additive quantization, our theoretical bounds indicate that the noise amplification stemming from activation and output gradient quanti- zation diminishes the batch size increases, whereas the spectrum of the effective data covariance is distorted by a constant noise floor (see Corollary 4.1 for details). Conversely, for multiplicative quantization, our results demonstrate that data quantization preserves the intrinsic spectral struc- ture of the effective covariance, thereby reducing spectral distortion; however, the resulting noise amplification remains independent of the batch size (see Theorem 4.2 for details).
We further derive the conditions the quantization errors such that the learning performance of the full-precision SGD can be maintained (in orders). Our results indicate that compared with multiplicative quantization, additive quantization necessitates stricter spectral constraints on data quantization but allows for more relaxed conditions on activation and output gradient quantization, benefiting from the batch-averaging effect (see Corollary 4.2 for details). By applying our excess risk bounds to polynomial decay spectrum, we show that multiplicative quantization is applicable even in high-dimensional settings, whereas additive quantization not (see Corollary 4.3 for details). These simplified theoretical results also draw implications for comparing integer and FP quantization, allowing us to identify the conditions under which each type is likely to yield superior performance.
Notations. For two positive-valuce functions fxx and g(x), we write f(x) S ((x) or f(x) 2 g(x) if [(x) S cg(x) or (x) 2 cg(x) holds for some absolute (if not otherwise specified) constant e >> 0 respectively. We write [(x E ( if (xx S g(x)) ك (xx) For two vectors and in Hilbert space, we denote their inner product by (u, v) or u v. For two matrices A and B of appropriate dimensions, we define their inner product by (A, B) := tr (A (A We use || to denote the operator norm for matrices and fy-norm for vectors. For positive semi- (PSD) matrix A and a vector v of appropriate dimension, we write || ||A v Av.
2 RELATED WORKS
High-dimensional linear regression via SGD. Theoretical guarantees for the generalization prop- erty have garnered significant attention in machine learning and deep learning. Seminal work by Bartlett et al.] (2020); Tsigler & Bartlett (2023) derived nearly tight upper and lower excess risk bounds in linear (ridge) regression for general regularization schemes. With regards to the classi- cal underparameterizee regime, a large number of works studied the learnability of iterate averaged
SGD in linear regression (Polyak & Juditsky| 1992, Défossez Bach, 2015 |Bach & Moulines) Dieuleveut et al., |017] Jain al. 2018 2017). With regards modern overparameterized
2013
setting, one-pass SGD in linear regression has also been extensively studied (Dieuleveut & Bach
2013 Berthier et al. 2020: Varre al. 1021 Zou et al.) 2023 Wu et al. 2022a b Zhang et al.
2024a). providing a framework to characterize how the optimization algorithm affects the general- ization performance for various data distributions. Another line of work analyzed the behavior of multi-pass SGD on a high-dimensional e --regularized least-squares problem, characterizing excess risk bounds (Lei al. 2021 Zou et al.) (2022) and the exact dynamics of excess risk (Paquette al.) 2024a). From technical perspective, our work builds on the sharp finite-sample and dimension free analysis of SGD developed by Zou et al. (2023). However, these works did not concern the practical quantization operations. It remains unclear how quantization error affects the learning behavior of SGD for linear regression.
Theoretical analysis for quantization. As a powerful technique for deploying large-scale deep learning models, quantization has attracted significant attention. From the theoretical perspective, a line of works focus on the convergence guarantee in both quantized training (SGD) algorithms
(De Sa et al.) 20155] Alistarh et al. 2017 Faghri et al., 2020 Gorbunov et al., 2020 Gandikota
et al. [202]] Markov et al. 2023 Xin et al. 2025) and post-training quantization methods (Lybrand
& Saab, |201]2221]]] Zhang & Saab/ 2023 Zhang et al. 2023 2025). For low-precision training (SGD), De Sa et al. 2015) was the first to consider the convergence guarantees. Assuming unbiased stochas- tie quantization, convexity, and gradient sparsity, they gave upper bounds on the error probability of SGD. Alistarh et al. (2017) refined these results by focusing on the tra between commu-
3

---

Published as a conference paper at ICLR 2026
nication and convergence and proposed Quantized SGD (QSGD). Faghri al. (2020) extended the fixed quantization scheme (Alistarh et al., 2017) to two adaptive quantization schemes, providing a more general convergence guarantee for quantized training. For post-training quantization, Lyaiiiiaatii,,,Lanndd Lybrand & Saab] ) derived an error bound for ternary weight quantization under independent Gaussian data distribution. /Zhang et al. (2023) extended these results more general quantization grids and a wider range of data distributions using different proof technique. More recently, Zhang et al. (2025) presented the first quantitative error bounds for OPTQ post-training algorithm framework. However, no prior work provides explicit generalization bounds.
Linear models for theory of scaling law. Several recent studies have sought formalize and explain the empirical scaling laws using conceptually simplified linear models (Bahri et al. [2024]
Atanasov al., 2024, Paquette al. 2024b; Bordelon et al. 2024 [in al. 2024; 2025). Among them, |Bahii et al. (2024) considered a linear teacher-student model with power-law spectrum and showed that the test loss of the ordinary least square estimator decreases following a power law in sample size N (or model size M) when the other parameter goes to infinity. Bordelon et al. (2024) analyzed the test error the solution found by gradient flow linear random feature model and established power-law scaling in one of N, .M and training time T while the other two parameters go to infinity. Building on the technique in Zou et al. (2023), Lin et al. (2024) analyzed the test error of the last iterate of one-pass SGD in sketched linear model. They presented the first systematic study to establish a finite-sample joint scaling law (in M and N) for linear models that aligns with empirical observations (Kaplan et al. 2020). More recently, [Lin et al. (2025) extended the scaling law analysis to the setting with data reuse (i.e., multi-pass SGD) in data-constrained regimes.
PRELIMINARY
3.1 QUANTIZATION OPERATIONS
For all quantization operations in (quantized SGD), employ the stochastic quantization method (Markov et al., 2023), which unbiasedly rounds values using randomly adjusted probabilities. This stochastic quantization is widely used in both empirical and theoretical analysis of quantization (oodoaanu et al. 2024 Ozkara et al. 2025). We summarize this in the following assumption.
Assumption 3.1. Let ii E [d,l.p. a, o} be the coordinate-we wise quantization operation for data feature, label, model parameter, activation, and output gradient, respectively. We assume that the quantization operation is unbiased, i.e,, for any u,
E (u)uuu
Furthermore, to better uncover the effect of quantization, we consider the following two types of quantization error: multiplicative quantization and additive quantization, which are motivated by abstracting the behavior of prevalent numerical formats used in practice.
Definition 3.1. Let Q be an unbiased quantization operation. We categorize it based on the structure of its error variance:
Multiplicative quantization. We call the quantization is c-multiplicative if the conditional second moment of quantization error is proportional to the outer product of raw data itself, i.e.,
E [[((x)) - (x)
=xxxx =XXT.
Additive quantization. We call the quantization is c-additive if the conditional second moment of quantization error proportional to identity, i.e.,
E (x(x))x)(0)) - x) x = cI.
This theoretical distinction is grounded in practical quantization schemes. For instance, integer quantization (c.g., INT8, INT16) uses fixed bin length, resulting in an error that is largely indepen- dent of the value's ' magnitude. This characteristic aligns with our definition of additive quantization, where the error variance is uniform across coordinates. Conversely, floating-poin quantization (e.g..
4

---

Published as a conference paper at ICLR 2026
FP8, FP32) employs value-aware bin length via its exponent and mantissa bits (e.g.. the E4M3 for- mat FP8). This structure causes the quantization error to scale with the magnitude of the value itself, corresponding to the model of multiplicatiie quantization.
To precisely capture the quantization error, we further introduce some relevant notations on quanti- zation errors during training. Denote the activation and output gradient time
a, (
o, = Q.(y:) - Qa (Qa (Q.((.)),(ww-))))
Then we are ready to define quantization errors.
Definition 3.2. The quantization ervor on data e(d), on label e on parameter et at time I, on activation (a) at time and on output gradient (o) at time t are defined as follows.
(d) = Qa(x) -x, e = -Qi(y) - - Et = =2p(w)) Wc.
E : (ar) ar: 61 = 20(0r) - Oz.
3.2
DATA MODEL
We then state the regularity assumptions the data distribution, which align with those common in prior works (Zou et al. 2023 Lin et al. 2024). A key distinction in our setting is that the training process is performed on quantized data, i.e., Qa(x) and (v)) Consequently, assumptions directly on the quantized data rather than the full-precision versions.
formulate these
Assumption 3.2 (Data covariance). Let H = E xx
be the data covariance matrix and
H == (x.
D E[(Qs(x) (xxx - 1,
be the covariance matrices of the quantized data feature and quantization error of data covariance, respectively. Then we assume that (r( and (r) are finite.
Further let H Avv. be the eigen-decomposition of H. where { are the eigenvalues of H sorted in non-increasing order and v, are the corresponding eigenvectors. As in Zou et al. 22033 we denote
Ho:k Σ A.V. 1=1
H = A.v. Toak = Σ viv.
I = Σ v.v. i>k
i>k
i=1
Similarly, we denote the cigen-decomposition of H as H Σ,
and correspond-
ingly obtain H paks
(q)
(q)
(q)
(g)
We then extend the fourth moment and noise assumptions in to the low-precision setting.
Zou et al. 2023 Lin et al. (202))
Assumption 3.3 (Fourth-order moment). Let x(2) = =a((x. Then for any PSD mairix A, there exists a constant on > such that
E
19 (9)
(99)
ag (
To extend the model noise assumption in Zou al. 2023) the low-precision setting, define the optimal model weight regarding the quantized data feature and label:
(w) * argminn E (( - (w, (x))))
Then ready make the assumption the model noise Qi(v) (w(") (x))) Assumption 3.4. Assume there exists positive constant > 0 such that
E (x) (9)
In fact. Assumptions 3.3 and 3.4 can be directly inferred from the standard assumptions the full-
precision data (Assumptions 2.T and 2.2 in Zou et al. (2023)) under specific quantization schemes. We defer the discussion to Section
5

---

Published as a conference paper at ICLR 2026
4 MAIN THEORETICAL RESULTS
We first derive excess risk upper bounds for quantized SGD in Section 4.1 then compare these rates with the full-precision SGD (in orders) in Section 4.2 and perform specific case study in Section 4.3
4.1 EXCESS RISK BOUNDS
We now provide excess risk bounds under general quantization, multiplicative quantization and additive quantization. Denote the effective dimension for H A -max {K: x
2 No }.
Theorem 4.1 (General quantization). Consider general quantization. Denote DH D(H +
H( + D) D, D H(H N7 (I- )N (HH Under
Assumption 3.1 3.2 3.3
and 3.4 if the stepsize Y ogtt(Hooy then holds,
B[E(WN) S 2arrEr + 2BiasErr + ApproxErr,
where
[|w()) (1
+ to (g) G
20B
+
Nv
10:000
k-
N + N22 Ea
VarErr
1 (-ranir(H!a)
12
BiasErr N2 1|w(1)"1
112 (AT b:k-)
) + //
10:0000
ApproxEIr S 1|w" || 1p + 1/w" I
with o (g) o² oa { L (o))
Jos
(c)
lac
a sup, E tr H HHW 2
B
Theorem establishes the first excess risk bound for quantized SGD under a general quantiza- tion paradigm. The excess risk is decomposed into three components: variance error, bias error, and approximation error. Notably, the variance and bias errors mirror those full-precision SGD (Zou et al., 2023) and exact equivalence is recovered when the quantization error vanishes. The key role that quantization plays is two-fold: data quantization significantly influences the effec- tive (quantized) data covariance ((9), while activation, output gradient and parameter quantization amplify the effective noise variance (4) (which will be further characterized in the subsequent theo- rems when given specific quantization type). Specifically, the quantized data covariance arises from performing SGI) in quantized data feature space and the quantized noise variance corresponds to additional quantization error introduced in the parameter update rule. We also note that the addi- tional approximation error, resulting from quantization of data, can be interpreted as the discrepancy between the global optimum in full-precision data space and quantized data feature space.
Crucially, in the absence of quantization, our excess risk bound reduces exactly to the standard results presented Zou et al. (2023). It is also worth noting that under the unbiased quantization assumption, the quantization of parameter, output gradient, and activation do not affect bias error
To further clucidate the effects of quantization, we examine two specific schemes: multiplicative and additive guantization. The result for additive quantization can be derived directly from Theorem and is summarized below.
and if there exist Corollary 4.1 (Additive quantization). Under Assumption 3.1 3.2
(p, la and C, such that for any i E [d,1,p, 0), quantization Q is G-additive, and the stepsize satisfies y < ag(uu(H)+dea) then
B[E(wN)] S ApproxErr + VarErr + BiasErr,
For theoretical tractability and simplicity, our framework employs the unbiased quantization assumption (Assumption 3.1). Without this assumption, the conditional expectations of the parameter, output gradient, and activation quantization emoors (i.e., quantization bias) would contribute to the bias error. We believe our framework is readily extendable this general biased quantization setting.

---

Published as a conference paper at ICLR 2026
where
ApproxErr
cd
||w* H
BiasErr X -2N2
/|
112
(H
+
+//w(o)*
da + ed
aB (Bl|w" lA + o² te + a (tr(H) (H) + deal h:" Nop E(d+tca)) i>
VarErr x
+
- 1- + deal
N
Corollary 4.1 explicitly demonstrates how data quantization distorts effective data covariance spec- trum and how parameter, activation and output gradient quantization amplify noise during training under additive quantization scheme. A key observation concerns the scaling with respect to the batch size B. Consistent with the label noise o², the noise amplification from activation and output gradient quantization (Ca, Co) are scaled by factor of 1 1/B. In contrast, the noise amplification from parameter quantization (ep,) scales with the trace of the quantized data covariance and is independent of batch size.
The interpretation is that additive quantization imposes a constant bound on the conditional second moment of the quantization error. Consequently, the underlying data structure inherent within the ac- tivation quantization error c!a) and output gradient quantization error ()) is effectively neutralized. T
Formally, the noise amplification from these terms is characterized as 1z R. X 6E X²). Under ad- ditive quantization, since the error variance bounded by a constant, the dependency on data within vanishes. However, the noise amplification from parameter quantization, which is characterized
B as 1 B[X (P)(P))(()) X X9), preserves the underlying dependency on data, even the error
variance itself is constant.
Moreover, a critical consequence of additive quantization is the distortion of the data covariance spectrum H Specifically, fixed constant 6a is added across the entire spectrum, effectively im- posing noise floor that prevents the tail eigenvalues from decaying. This spectral flattening severely impedes learnability, as leads substantial risk accumulation within the high-dimensional tail subspace.
We next examine the multiplicative quantization scheme. Unlike additive quantization, multiplica- tive quantization exhibits inherent structural alignment with the full-precision dynamics, as the error scales relative to the signal magnitude. Exploiting this property allows us to derive refined ex- cess risk bound through direct analysis, rather than relying generic application of the general result in Theorem [4.1] Our theoretical findings are summarized below.
Theorem 4.2 (Multiplicative quantization). Under Assumption 3.7 3.2 and 3.4 if there exist Ea, Ea Ep. Ea and e, such that for any i E [d, a, o), quantization Q, is e;-multiplicative, and the stepsize satisfies y as follows.
then the excess risk can be upper bounded
E[E(wN)] ApproxErr + VarErr + BiasEir,
where
ApproxErr
S
Ed ed
||
BiasErr S N2 1|w 112 ( )-1 + l| H (H
(1+e))o" + +aB + co)
co(1 ep)l |
VarErr S N + Ed)² N
x?
7an(1 + 6.) + Ep + (a ep]] + ea)tr (H)
Theorem 4.2 characterizes the spectrum distortion and noise amplification effects induced by multi- plicative quantization. Notably, in stark contrast to additive quantization, which severely flattens the tail spectrum by imposing constant floor, multiplicative quantization largely preserves the intrinsic spectral structure. Specifically, it acts linear transformation that scales the entire spectrum by factor of (1 + ea) without altering the relative distribution of cigenvalues. This preservation of the spectral decay property ensures superior learnability compared to the additive quantization scheme.
Regarding noise amplification, Theorem 4.2 reveals a critical divergence from the additive quanti- zation scheme. While the contribution from intrinsic label noise (a²) is still suppressed by the batch size factor 1/ B. the quantization noise stemming from activation and output gradients (c,, c,) is coupled with the model parameter ||w*||] [| and does not scale with 1 B. This phenomenon arises
7

---

Published as a conference paper at ICLR 2026
because multiplicative quantization error (scales proportionally with the signal strength) is inher- ently signal-dependent and is intrinsically tied to the data structure.
We provide further analysis of quantized SGD with quantized master weights in Section F Train- ing with quantized master weights necessitates stricter step size conditions to ensure convergence and introduces additional error terms into the excess risk bounds, thereby degrading generalization performance.
4.2 COMPARISONS WITH STANDARD EXCESS RISK BOUND
In this part, we will provide detailed comparison with standard excess risk bounds and identify the conditions on the quantization error such that the excess risk bound will not be largely affected. First, let kj max 1: Al Z No. we recall the standard excess risk bound (Zou et 2023):
+ |
a²
hi N
0B
N7
Ro
N22
X
1 gr (H)
N2 ||w" (H
||w"
The following corollary derives the conditions on the quantization errors such that the learning performance of the full-precision SGD can be maintained (in orders).
Corollary 4.2. To ensure that E (W(ww)) & Ro. conditions on the quantization evror are as follows:
For multiplicative quantization, under the conditions in Theorem
4.2
require
Ro W H
Ny
w"
Loigg
+
||
cd S
1A
Cos Car Cp
A
BaBl|w+IIl
[
For additive quantization, under the conditions in Corollary
4.
require
5
h
N
N22. Ei>k;
X?
RoAs I 12
Ear 02 o² + Bag
||w 12anng
+
||
N22(d - kg)
Ny
|| || Ny
+ || "Il
Eo S Banntr(H) deal
tr(H) dea
Corollary 4.2 identifies the conditions under which the quantized excess risk matches the full- precision baseline Rg. Regarding data quantization (E.), the additive scheme imposes stringent spectrum-ddependet constraints compared to the multiplicative quantization scheme. Specifically, the precision requirements notably strict prevent the constant quantization noise floor from overwhelming weak spectral components. Conversely, for activation and output gradient quantiza- tion (fa; ea), the additive scheme exhibits favorable dependence by the scaling with in the bounds for 6a and c.. larger batch sizes effectively relax the preci- sion requirements for these components. In contrast, larger batch sizes may essentially tighten the requirements under the multiplicative quantization scheme.
the batch size. As indicated
These findings validate our core insights: (1) multiplicative data quantization is superior in main- taining the spectral structure of H. thus tolerating larger data quantization errors; (2) additive quan- tization benefits from the fact that the noise variance in activation and output gradient is independent of the signal magnitude, allowing these errors to be effectively suppressed by increasing the batch size.
4.3 CASE STUDY ON DATA DISTRIBUTION WITH POLYNOMIAL-DECAM SPECTRUM
study the excess risk bounds assuming optimal parameter prior and the power-law law spectrum for more concise theoretical results. In particular, we make the following assumption.
Following Lin al. (204 2025
8

---

Published as a conference paper at ICLR 2026
Assumption 4.1. There exists 1 such that the eigenvalues of H satisfy X io 0. We
also assume that E w wo and o² S 1.
Corollary 4.3. Taking expectation on w", under Assumption 4. we have:
For multiplicative quantization, under the conditions in Theorem 4.2
E [[((wN)] For additive quantization, under the conditions in Corollary 4.1
ed d' N +1 + +ep + eal1 + tp)](1 11001+000 tea) +
I
[[E(WN)] 2
6o t
ep + dea)) (1+
(d) 1 + dued
d=
dd
dacd
1 Eo + co 6p(1 + dea) (1+ d' daa)
Our findings in polynomial-decay data spectrum scenarios reveal distinct scaling behaviors under multiplicative and additive quantization. Specifically, the excess risk induced by additive data quan- tization exhibits a detrimental dependency on data dimension d, whereas the risk under multiplica- tive data quantization remains clinension-indpenne for learnability: in high-dimensional regimes (d - oo), the risk bound for additive quantization diverges, rendering the generalization guarantee vacuous. In contrast, the dimension-free nature of multiplicative quantization ensures its applicability even in infiniteddimensional settings.
This dependence has critical implications
Intuitively, this disparity stems from how each scheme interacts with the spectral structure. Mul- tiplicative quantization preserves the intrinsic spectral decay, thereby retaining the utility of the effective dimension (k*) cut-off. This allows the learning complexity to be controlled by the in- trinsic data properties rather than the data dimension. Conversely, additive quantization employs a uniform quantization strength across all dimensions. This constant noise floor prevents the tail spec- trum from decaying effectively and accumulates across the entire high-dimensional tail, rendering the effective dimension mechanism failed.
Implications for integer and FP quantization. These simplified theoretical results (Corollary 4.3)
draw critical implications for comparing integer and floating-poit ppiti (FP) quantization, allowing to identify the conditions under which each type yields superior performance. Specifically, in practical integer quantization with bit-width b and FP quantization with mantissa bit-width m, the quantiza- tion step size for value are approximately ((z) ~ 2 and 8(x) = |x|2 m respectively. Since the conditional second moment of quantization error [[(((z) x) is roughly proportional to the square of the quantization step size (8(z)²), the quantization error parameters in our bounds can be characterized as Cadd 2-26 for the additive (integer) quantization scheme and Gmul 2-2m
2m for
the multiplicative (FP) quantization scheme.
Equipped with this mapping, practitioners can directly apply Corollary 4.3 to determine the optimal quantization scheme for specific scenarios. A notable observation concerns the distinct role of the dimension d in data quantization. Roughly, FP quantization becomes preferable when my 2 by logg d whereas integer quantization favored when ba 2 m. + _ log, d 3 This means FP quantization can outperform integer quantization even when its mantissa bit-width is smaller than the integer bidhw by log, d, highlighting the advantage of FP quantization in high-dimensional settings.
Numerical experiments. We evaluate constant-stepsize SGD with iterate averaging a Gaus- sian least-squares model. The feature distribution has covariance matrix with eigenvalues X; = i The ground-truth parameter is w with entries w w*] [i] 1, and the observation noise variance is o² = 1. This study answers two questions: Q1: How do additive vs. multiplicative quantization errors affect learning? Q2: How does dimension d interact with these two quantization types?
We assume the exponent hits in FP quantization can cover the scaling of æ. For integer quantization, we assume the dynamic range (i.c., Tmax - Fmin) is normalized to constant level. 'Here by and ma are the bit-width for integer data quantization and the mantissa bit-width for FP data quantization respectively.

---

Published as a conference paper at ICLR 2026
050
682
Roa
1.75-
Seoonko Rol
1,75
Eroowh Rol
2.5
1.50
1.50
1.00
20
20
Semple Blcce5d))
Bunds (lac)(93)
Bample Blac)da))
Sample Sioole50)
(a) Multiplicative (FP-like) (b) Additive (INT-likk) (c) Multiplicative (FP-like) (d) Additive (INT-like)
Figure I: Generalization under quantization. Population risk (Ex.pl(y [(v (w, )) for quantized SGD with iterate averaging under multiplicative (FP-like) vs. additive (INT-like) quantization. (a) and (b): vary the quantization level at fixed dimension. (c) and (d): vary dimension at fixed quanti- zation level.
Q1 (Quantization level). We fix d 200 and B = and vary the quantization error level e E {0.001, 0.005, 0.01} for each scheme. Results are shown in Fig. [a.b). Under multiplicative quantization, quantized SGD largely retains the generalization performance of full-precision SGD across a wide range of quantization levels. Conversely, under additive quantization, performance degrades the quantization level increases. These empirical observations validate our theoreti- cal findings: with batch size of B = 1, additive quantization requires stricter conditions (lower quantization level) match the performance of full-precision SGD.
Q2 (Dimension). We fix the quantization level at s = 0.01 and B = 1, and vary d E {50, 100, 200, 400). Results are shown in Fig. [..d). Under multiplicative quantization, gener- alization performance is preserved even in high- digennimensional settings; conversely, under additive quantization, performance deteriorates as the data dimension increases. These empirical results cor- roborate our theoretical findings: multiplicative quantization remains effective in high-dimensional contexts, whereas additive quantization is ill-suited for such scenarios.
Furthermore, we conduct additional experiments on the real-world Communities and Crime dataset, as well as settings with larger batch sizes and exponential-decay spectra. These results, presented in Section G consistently align with our theoretical analysis.
CONCLUSION AND LIMITATIONS
In this work, we presented comprehensive theoretical framework analyze the excess risk of quan- tized SGD in high-dimensional linear regression. Our analysis disentangles the distinct impacts of various quantization targets: while parameter, activation, and gradient quantization primarily serve as noise amplifiers, data quantization fundamentally distorts the effective feature covariance spec- trum. Crucially, we show that multiplicative quantization excels at preserving the spectral structure of the data, thereby maintaining learnability even in high-dimensiioal settings. In contrast, additive quantization leverages the independence of noise variance from signal magnitude, allowing activa- tion and gradient noise to be effectively suppressed by large batch sizes. Furthermore, our theory establishes the conditions on quantization errors required maintain full- precision SGD perfor- mance, and identifies the scenarios under which FP and integer quantization are each likely to yield superior performance under polynomial decay spectrum.
Future work. Our work lays solid foundation for several promising research avenues. Firstly, de- veloping lower bound analysis for the excess risk of quantized SGD. Secondly, extending single- pass SGD to more practical training configurations, such as data reuse (i.e., multi-pass SGD), learn- ing rate scheduling, momentum, and preconditioning. Thirdly, extending training the linear models to the training of pwerpearamteriieed neural networks. Fourthly, deriving scaling laws for low- precision training.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and area chairs for their helpful comments. We acknowledge the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 1024A1515012444 5044444, and the central fund from HKU IDS.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have carefully reviewed the Code of Ethics and find that our work does not raise any significant ethical concerns. Our research does not involve human subjects, sensitive data, or potentially harm- ful applications. We believe our methodology and contributions align with principles of faimess, transparency, and research integrity.
REPRODUCIBILITY STATEMENT
The paper fully discloses all the information, including training and testing details, needed to repro- duce the main experimental results of the paper to the extent that it affects the main conclusions of the paper, as described in Section 4.3 All data are synthetically generated with detailed description.
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovie. Qsgd: Communication-fficient sgd via gradient quantization and encoding. Advances in neural in- formation processing systems, 30, 2017.
Alexander Atanasov, Jacob Zavatone- Veth, and Cengiz Pehlevan, Scaling and renormalization in nigh-dimensional regression. arXiv preprint arXiv:2405.00592, 2024.
Francis Bach and Eric Moulines. Non-strongly smooth stochastic approximation with con-
vergence rate (1/n). Advances in neural information processing systems, 26, 2013.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings the National Academy of Sciences, 121(27):e231187111, 2024.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063330077) 2020.
Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for
stochastic gradient descent under the noiseless linear model. Advances in Neural Information Processing Systems, 33:2576-255 2586, 2020.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01099. 2024.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. Journal of Statistical Mechanics: Theory and Experiment, 2025(8),084002, 2025.
Mengzhao Chen, Chaoyi Zhang, Jing Liu, Yutao Zeng, Zeyue Xue, Zhiheng Liu, Yunshui Li, Jin Ma, Jie Huang, Xun Zhou, et al. Scaling law for quantization-ane aware training. arXiv preprint arXiv:2505.14300 2025.
Christopher M De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré. Taming the wild: A unified analysis hogwild-styy algorithms. Advances in neural information processing systems, 28, 2015.
Alexandre Défossez and Francis Bach. Averaged Seast-mennsqqareee Bias-variance trade-offs and optimal sampling distributions. In Artificial Intelligence and Statistics, pp. 205-213, PMLR, 2015.
Aymeric Dieuleveut and Francis Bach. Non Non-parametric stochastic approximation with large step sizes. Annals of Statistics, 44(4), 2015.
Aymeric Dieuleveut, Nicolas Flammarion, and Francis Bach. Harder, better, faster, stronger gence rates for least-squares regression. Journal of Machine Learning Research, 18(101):1-51, 2017.
11

---

Published as a conference paper at ICLR 2026
Fartash Faghri, Iman Tabrizian, Ilia Markov, Dan Alistarh, Daniel M Roy, and Ali Ramezani- Kebrya. Adaptive gradient quantization for data-paree aata-parallel sgd. Advances in neural information processing systems, 33:3174-3183 3185, 2020.
Venkata Gandikota, Daniel Kane, Raj Kumar Maity, and Arya Mazumdar. vqsgd: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, pp. 2197-2205. PMLR, 2021.
Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. A unified theory of sgd: Variance reduc- tion, sampling, quantization and coordinate descent. In International Conference on Artificial Intelligence and Statistics, pp. 680-690. PMLR, 2020.
Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, Venkata Krishna Pillutla, and Aaron Sidford. A markov chain theory approach to characterizing the minimax optimality of stochastic gradient descent (for least squares). arXiv preprint drX1v:1710.09433, 2017.
Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Paralleliz- averaging, and model
ing stochastic gradient descent for least squares regression: misspecification. Journal of machine learning research, 18(223):1-422 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0836/ 2020.
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:2411.044 2024.
Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under zipf's 's law. arXiv preprint arXiv:2505. 19227, 2025.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Sys- tems, 35:14651-14662. 2022.
Jiedong Lang, Zhehao Guo, and Shuyu Huang. comprehensive study on quantization tech- niques for large language models. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pp. 224-231. IEEE, 2024.
Yunwen Lei, Ting Hu, and Ke Tang. Generalization performance of multi-pass stochastic gradient descent with convex loss functions. Journal of Machine Learning Research, 22(25): 1-11, 2021.
Binghui Li, Fengling Chen, Zixun Huang, Lean Wang. and Lei Wu. Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules. arXiv preprint arXiv:2509.19188 19189, 2025.
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter I. Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. Advances Neural Information Processing Systems, 37:60556-60666 2024.
Licong Lin, Jingfeng Wu, and Peter I. Bartlett. Improved scaling laws in linear regression via data reuse. arXiv preprint S00i:22006084115, 2025.
Lumin Liu, Jun Zhang, Shenghui Song, and Khaled B. Letaief. Hierarchical federated learning with quantization: Convergence analysis and system design. IEEE Transactions on Wireless Communications. 22(1):2-18, 2023, doi: 10.11099TW.220223399955
Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun. Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. A multi law for loss curve prediction across learning rate schedules. In The Thirteenth International Conference on Learning Representations, 2025.
Eric Lybrand and Rayan Saab. A greedy algorithm for quantizing neural networks. Journal of Machine Learning Research, 22(156): 2021.
12

---

Published as a conference paper at ICLR 2026
Ilia Markov, Adrian Vladu, Qi Guo, and Dan Alistarh. Quantized distributed training of large models with convergence guarantees. In International Conference on Machine Learning, pp. 24020- 24044, PMLR, 2023.
Ionut- Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtić, Thomas Robert, Peter Richtárik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space over- head and provable convergence. Advances in Newral Information Processing Systems, 37:1-43, 2024.
Giorgi Nadiradze, Amirmojtaha Sabour, Peter Davies, Shigang Li, and Dan Alistarh. Asynchronous decentralized sgd with quantized and local updates. Advances in Neural Information Processing Systems, 086829-6622,, 2021.
Kaan Ozkara, Tao Yu, and Youngsuk Park. Stochastic rounding for Ilm training: Theory and prac- tice. arXiv preprint arXiv:2502.20566, 2025.
Courtney Paquette, Elliot Paquette, Ben Adlam, and Jeffrey Pennington. Homogenization of sgd in high-dimensions: Exact dynamics and generalization properties. Mathematical Programming, pp. 1-90, 2024a.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute- optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459- 16537, 2024b.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-8))) -55,55 1992.
Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D Lee. Emergence and scaling laws in sgd
learning of shallow neural networks. arXiv preprint arXiv:2504. S04iv:2504./98... 19983, 2025.
Shen, Zhiquan Lai, and Dongsheng Li. Exploring quantization techniques for large-scale lan- guage models: Methods, challenges and future directions. In Proceedings of the 2024 9th Inter- national Conference on Cyber Security and Information Engineering, pp. 783-790, 2024.
Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, et al. Scaling laws for floating point quantization training. arXiv preprint arXiv:2501.02423, 2025.
Alexander Tsigler and Peter Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24(123):1-76, 2023.
Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Last iterate convergence of sgd for least-squares in the interpolation regime. Advances in Neural Information Processing Systems, 14:21581-21599 2021.
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression. In International conference on machine learning, pp. 24280-2444 14. PMLR, 2022a.
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of pvetraining-finetuniin for linear regression under covariate shift. Advances in Neural Information Processing Systems, 55:33041-33053, 2022b.
Lechao Xiao. Rethinking conventional wisdom in machine learning: From generalization scaling.
arXiv preprint arXiv:2409. 15156, 2024.
Jihao Xin, Marco Canini, Peter Richtárik, and Samuel Horváth. Global-qsgd: Allreduce-comee compaabile quantization for distributed learning with theoretical guarantees. In Proceedings of the Sth Work- shop on Machine Learning and Systems, pp. 216-229, 2025.
Haihan Zhang, Yuanshi Liu, Qianwen Chen, and Cong Fang. The optimality of (accelerated) sgd for high-dimensional quadratic optimization. arXiv preprint 0007944409 2024a.
13

---

Published as a conference paper at ICLR 2026
Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Fos- ter. and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676. 2024b.
Haoyu Zhang, Shihao Zhang, Ian Colbert, and Rayan Saab. Provable post-training quantization: Theoretical analysis of optq and qronos. arXiv preprint arXiv:2508. 048.23338,04,3, 2025.
Jinjie Zhang and Rayan Saab. Spfq: A stochastic algorithm and its error analysis for neural network quantization. arXiv preprint arXiv:2309. 10975, 2023.
Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. SIAM journal mathematics of data science, 5(2):373-33) 2023.
Kaiqi Zhang, Ming Yin, and Yu-Xiang Wang. Why quantization improves generalization: Ntk of binary weight neural networks. arXiv preprint arXiv:2206.05916, 2022.
Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Risk bounds of multi-pass sgd for least squares the interpolation regime. Advances in Neural Information Processing Systems, 35:12909-12920, 2022.
Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Benign over- fitting of constant-stepsize sgd for linear regression. Journal of Machine Learning Research, 24 (326):1-58, 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Initial Study
17
A.I Deviation of the Update Rule
17
A.2 Decomposition of the Excess Risk
18
B Analysis of Approximation Error
21
Analysis of Rx
22
C.1 Preliminary
22
Initial Study
22
Bias- Variance Decomposition
27
Bounding the Bias Error
29
C.S Bounding the Variance Error
35
D Deferring Proofs
41
D.I Proof for Theorem 4.
41
D.2 Proof for Theorem
D33 Proof for Corollary 4.1
42
D.4
Proof for the Multiplicative Statement in Corollary 4.2
43
D.5 Proof for the Additive Statement in Corollary 4.2
45
D.6
Proof for the Multiplicative Statement in Corollary
46
D.7
Proof for the Additive Statement in Corollary 4.3
48
Discussion of Assumptions
49
E.1
Discussion of Assumption 3.3
50
Discussion of Assumption 3.4
Extension to Quantized Master Weights
53
General Quantization
54
Additive Quantization
58
Multiplicative Quantization
58
F.3.1
Analysis of Variance Error
59
F.3.2
Analysis of Bias Error
62
Details of Additional Experiments
67
Additional Datasets
67
G.2
Experimental Settings and Results
67
The Use of LLMs
68
15

---

Published as a conference paper at ICLR 2026
The appendix is organized follows. In Section A
begin the analysis of excess risk bounds for the iteratively averaged quantized SGD by firstly deriving the update rule for the parameter deviation We- We-w(9)" (detailed in Section A.1) 1) and secondly performing an excess risk decomposition (detailed in Section A.2):
[[E(wN)] 2
(H. E & 7xl) ApproxErr.
R
We then conduct refined analysis for ApproxErr Sections B For Rx. we extend techniques
from Zou al. (2023) in Section C In particular, first introduce useful notations in Section
C.I and then present comprehensive analysis of the update rule for E[nem in Section C.2 This analysis is crucial for adapting previous proof techniques the quantized SGD setting. Based
these results, we perform a
decomposition in Section C.3 and analyze the bias and
variance errors separately in Section C.4 and C.5 weight is quantized.
In Section
we include bounds when master
The following proof dependency graph visually encapsulates the logical structure and organizational architecture of the theoretical results in our paper. In particular, the arrow from element X to element means the proof of Y relies X.
Corollary 4.3
Theorem 4.2
Corollary 4.2
Corollary 4.1
Theorem D.1
Corollary
Lemma A.3
Theorem 4.1
Lemma C.17
Lemma B.1
Lemma B.2
Lemma C.16
Lemma C.15
Lemma C.11
Lemma C.10
Lemma C.14
Lemma C.13
Lemma C.7
Lemma C.9
Lemma C.8
Lemma C.6
Lemma C.12
Lemma C.5
Lemma C.4
Lemma C.3
Lemma C.1
Lemma C.2
Lemma A.1
16

---

Published as a conference paper at ICLR 2026
A INITIAL STUDY
For simplicity, we denote y( = (1(y), w; = (p(w)). = Qx(x). For convenience, we assume that H is strictly positive definite and that L(w) admits unique global optimum Zou al. 2023). We first recall the definition of the global minima w and w
w argminw w (y - (w, x))]
w( = argmin E [Q - (w, 2a(x)))
The first order optimality shows that
BI(y xx)x O, ([((r()) (w(x), Qd(x)) =
(A.1)
which implies that
H [(xx))Dlyx]] w(*)* (H (1) - (2) (s) = -
(H
Hence, by denoting H(4) = H + D, we can characterize the difference between w(o) and w as:
w- w" ())
E [xx,)>=/x>]
-(H(w) -1 H H
9)
H HI'EE [(xx))DDlxx]
-(H (9) -1 H H()) w'
(A.2)
H 'Dw"
(H + D) Dw
A.1 DEVIATION OF THE UPDATE RULE
In this section, we derive the evolution of parameter deviation n, W, w(0) Lemma A.1 (Error propagation).
T I -.(())) (x.))) 9:-1 + 11 ex(x))
5: + 6 (0)) -((-) - -x(x))
where the quantization errors are
(0) Q. (Qi(r.) 2. (Q((()))))
Q.
( : (2,,(x))) - (,(
6- - WW--1
&, (v.)) 2d(X))
Proof. The lemma can be proved directly by the parameter update rule. By definition and the update rule of w, (quantized SGD).
n. -w w())"
W..1 w()) + 77 O -. (w
1 -- 7 - (x.)) Q (Qr().) e. (2,
We then introduce quantization errors to better characterize each quantization operation @(.). In particular, define quantization erros:
(o) 2. (Qi(y.) - Qa
(Q) - 2.
(a) = (2,((.)),, Q2
(p) = =Qp(w(-1) W1-1
c-
&. (i(r)) - -,((x))(''',
17

---

Published as a conference paper at ICLR 2026
Then the update rule for the parameter deviation can be expressed as:
7h - 1
- (X,,) T 1 [Q((.)) Q. (O. +7 + ( c(
B
T 1
1 2a (X.) (c, (0) -c( ) (0))
-h- + )X,(
- 2w
B
- + 7 B (x,), (c (6)) -4,0) (x)))))(X))'
1
(R((v)) - ex(X, ( (00 -Qa(X))
+
- + 7Xx(X)) (E) (0) +7)x)) 1
B
[Q(())) - -a((x:)((1" (0)" - -Qa(X,) - 2x(X))= (n)
1 Qa(X.) (E, - -c,") -70(11)(X)) I
1 Qa 1
= ? B
B
1 [ (1-()() [ Mh-1 7 eu(x)) c -c,")) -
A.2 DECOMPOSITION OF THE EXCESS RISK
In this section, take the initial step analyze the excess risk of averaged SGD iterate W
In particular, we define the deviation of the averaged SGD iterate as Ty == tV4 E Th We decompose the excess risk as follows. Lemma A.2 (Excess risk decomposition). Under Assumption 3.7 and Assumption 3.2
[[E(wx)) = R₁ + RR + R3 + Rs,
where
R1
E (WN:, 2x(x) - x)²1,
2
R2
(9), Eiii x nN])
2
Rg E [[w\0 2x(x) -x)²]
R4 H. - ww()" @ (w* w(()
1
2
Proof. By the definition of the excess risk
Excess Risk). )
1
B[E(W,)
E
(y (WN, x
E
(v (w*,x))² x))²
2
2
E (u (W., x)) x]
E [(Or(v) - (WNN (x(x)))
2
2
+
E
[(2)()) - (WN, (x()))
2 (2 (w()) 2x(x)))
(())()) - (w(*)" =2(x))))
( (w/()) ) 27
+
E
E
2
2
Es
(v- (v-(w(o)) (wwl)))))))
+
E
(y (w*, xx)]]
2
Ex
18

---

Published as a conference paper at ICLR 2026
where E1 captures the gap of the averaged SGD iterate between the full-precision and quantized domains, Eg characterizes the distance from the averaged SGD iterate to the quantized optimal solution within the quantized domain, E3 represents the mismatch of the quantized optimal solution in full-precision data space and quantized data space and E4 defines the discrepancy between the averaged SGD iterate and the quantized optimal solution in the full-precision domain.
We would like to remark that the quantization operations 21(y) and (x(x)) introduced in excess risk decomposition are independent of those quantization operators introduced in the training stage, i.e. w.. Next, we analyze E1, E2, Eg and Es respectively. These computations are mainly based on the first order optimality condition (A.1) and the unbiased quantization Assumption 3.11 For Es,
E4
E
y/
(w(o) xx)
1
E (w - (w* x))²
2
2
(w* w(1)* xx) (2y (w" - wl)))* xx)
E
2
E
(w> w(1))* xx)2
(A.3)
1
W' w-(0)* H -w()))
1 (w" - w(0)) @ (w" - w(n))) )}
2
where the third equality uses the first order optimality condition that E (xxx)) (V - ((",,)) x))xx = 0.
For Eg,, similarly by the first order optimality condition (A.1) with respect to w(0)", it holds
E2 E [(Qr()) (WN. [a(x))) _ [Q((v)) (w()), (x))))
E (wrl)* WN. (2Q((v) (wll)* + WWN, 2d(x)))
(A.4)
E (w/w Oa()))))
2
(H(9), , 8 7,1)
2
For Es.
E (Q((v) - (w(") -x)x1) 2 E [vv ( (w(l))
; )
27
Eg =
2
- E (0.(0) - y - --(w()) 2x(x) - x)) (Di(e) + y- (ww()) -a(xx) + xx)
R Q(()) -²² +w(l)) Qx(x) - x)(wl()) Qx(x) + xx)
2
where the last equality utilizes the unbiased quantization Assumption 3.1
For E1, similarly by the unbiased quantization Assumption 3.1 it holds
E1 (y (W.. (W,,))) x _ (()()) (WN. (x))))
2
E [(y (v(2) (WN. (x(x))) (v+ ( + Qa(x)))
2
B
1022 2 + E Bl ex(x)) X Qa(x))].
Hence,
E1 + F. E. (w(v)) (0) (x) x)²]
1
E [[[(wN, x- 2x(x))
(A.5)
2
Therefore, combining
AA.))
(A44 and A.5 we have
E[E(Wx)
HH (q) E
E
2 >> +
1
HH, (w" - w())') @ (w" - -(()")))
2
+ +
E [w/()) (x(x) - -x)
1
E (WN.x- - 21
2
19

---

Published as a conference paper at ICLR 2026
Lemma A.3 (Refine excess risk decomposition). Under Assumption 3.1 and Assumption 3.2 stepsize y < x
if the
1 (H. E @ 7Nl) + ApproxErr,
B[E(wN)]
2
Ry
where
ApproxErr 2
1
(H, (w" -w(n)") 2 (w" - w())'))
+ ((wl0)))
T
Ny (I -- (I (H (H()) - H) wln)
1
Proof. By Lemma A.2
[[E(wN)] (H E.TT & 7N])
+ HH (w" w())) @ -w(0")
2
E (wwo" 2x(x) - -x)² -x(())xx²}
1
2 [(Wxxx-Qu(xx)) 2x(x))
2
We then focus E [WNN, [x(x)) Recall that
W = Wy - (0)) + w()) TIx + w(a)"
have
1
E WNN. x-
2
E
w
H
(0) H
WN
2
2
1 E
(wll)") T (H()) H w(n))
2
77
(
H
7N
+ 2
+E
MN
H
(1)
H
w(n)"
Hence,
[[E(WN)]
(H,
E 7
2 nxl)
+ 2 (,, (w---(0)" @ (w" - w())'))
2
(A.6)
4 H w(r) I
-E
NN
Noticing that by Lemma A.I
7:: (1- 1
Ou(x,) (x(x)) mh-1 + 1 7 Qu(x))
[ (()+ -((") - -2(x))))))()))
B
follows by Assumption 3.1 that
E [n]] = E TE
(n 1 E
[I-> (
= m E [mcc1] -11-
17-1
1(1==111--1 (I-,H(0) no-
Hence,
N-1
E
(4) T
H w(0)']
= 7a
I - H (HW) - H) wl))
(HW-
N
1=
N-1
=(w(o))
I (I- (H()) -0 H w()))
N 1=U
= =(wll)))
I - [I - H())))
(H())- (H()) H w
Ny
Combining (A6)) completes the proof.
20

---

Published as a conference paper at ICLR 2026
B ANALYSIS OF APPROXIMATION ERROR
In this section, we analyze ApproxErr under multiplicative quantization and additive quantization, respectively. We first apply the definition of w())"
(())* = ( H'w.
We first handle 1/2 (H. (w" - -w())) 20 (w* - w(())) )} Recall D = H() H. we have
H. (w" wl9))) (w" wl)))))
E tr H(H + +D "Dw' w D(H + +D)+ +D--1)
E tr w D(H D) HHH
(D(H+)))HH(++ D) "D
|| DDB
(B.1)
We then handle (w())*)
Ny (I -I- (HH (H( (H H) wlo)
(w()))) T =w" =w HHH 1 Ny (I- (()))(())) (H())-1 (H()) H
[I -I (I-HH(0)M) (H( H(4) w(
Ny
(B.2)
(H -Hw'
Lemma B.1 (Approximation error under multiplicative quantization). If there exists ea such that Os is ca-muliiplicative, under the assumptions and notations in Lemma A.3
à 2(1
ApproxErr S 2(1 + ca)2
11 +
ed
||
(1 ca)
Proof. Under multiplicative quantization,
H HH) D = eH
follows by B.I that
1 (H, (w" w((0)) & (w" (wi'-
= 1 w 1 D(H+D) (H(HD)) ID 2(1 ea)²
|w 11.
2 Similarly, by (B.2p.
(w(O)*) T
Ny [ (I -I- -H(()))) (HO))-1 H(() H w(7)"
HHH
[ - (HHH(()))))))))) (H H(w) H (H(0))-11E HHw
Ny
ed
H(
I (I H (H HHw'
Ny(1 + Ea)
ed
| 11 (H 1/2 II - (I H (H(0)
Ny(1 + c2)
min (1, N² (0)
Ed
||w"
max
NT + 6a)2
(9)
(1+)))) ||
Cd
Lemma B.2 (Approximation eroor under additive quantization). If there exists ca such that Qa is ca-adiitive, under the assumptions and notations in Lemma A.3
23
Aiea
ApproxErr - 2(A4 + E
w 1 + (ha + Ea) +
|
21

---

Published as a conference paper at ICLR 2026
Proof. Under additive quantization,
H = +H+
D = cal.
It follows by B.I that
1 H. (w" w(s)) @ -w-"-(s!")) = 1
=5/ w HAADD-
2(Aa + cx) |w w'llit--
2
Similarly, by B.2
((()') T
[ (I 11-I-~HH'' (-H((1)) (H( H H) w-
Ny
-w"T((H -1
Ny II- 1-((-- (HHHH))) (H()))- (H(())) (H()) - H (HO)) (H( 'Hw'
cd
H() -1 I (I - { (H(1))-11)) -'W' 'Hw"
w"
Ny(Aa Ed)
Aiea
|| || (H 1122 (I - (I (1- (H(2))-1/2)
1/2
Ny(Aa + (2) + [d)
min (1, No1
|w'l| max
Ny(da + ca) + Ed)
AlEd
||
Ea) + Ea)
C ANALYSIS OF RN
C.I PRELIMINARY
We first define the following linear operators as Zou et al. 2023)
I=10L I, M(4) E[x (9)
x
(a)
x
(0)
@ xx(1)
M - & H())
T ( II+ & ( 7M
7
H III ( H H
For a symmetric matrix A, the above definitions result in:
ZoA=A M o E T Ax ((())
J. M A = (4) (H
(T A [III xx)) A 7x 7x x(1) }.. (I (I-
( (
A (I - - ) ((I-)H(1),
Further, generalize the linear operators from Zou et al. (023) account for batch size effects. For a symmetric matrix A, we define
M (g)
A
E
x
(x) X(o)
B
x(1)) A ((I I 1-777 7
1 x -
)
(I
-
91)
A
=E
11
I
B
C.2 INITIAL STUDY
To analyze Rv, we firstly utilize the fact that
(H()), Ennnonnll < 2 (HO, 1 H. E m [l)) S umax (H(H())))) 2
Ry 2
M,EN & nxl)
(C.1)
We secondly substitute 7. with the summation of n. This step mainly based the propagation in Lemma A.I] the unbiased quantization Assumption 3.1 and the first order optimality condition A.I. We summarize the following lemma.
22

---

Published as a conference paper at ICLR 2026
Lemma C.1. Under Assumption 3.1
and Assumption 3.2
N
N-1
R
(I
H (4)
E n. @ n.l
N
N
t=0
k=t
Proof. Recall that by
C.1
R (H ET & nvl)
then focus on VTT
7yyl By definition
N Ei=o Th
1 N-1
E T & nx
Σ
Elm @ m +
Σ
Elm @ ml
N2
OSkSigN-1
OSAcksN-1
(C.2)
Σ
E E R mm/ll +
Σ
E llimona mm/mill
N22
0SASISN-1
USISKEN-1
Note that by the unbiased Assumption 3.1
(x.))
(5) Et
(a)
- -e(()))
7/-1 0.
Further, by the optimality A.1
E
2a(()) &, M--
=E [ox(X.)) - Mh-1 0.
Hence, by Lemma A.I
E [mm(mc11] (I-2H(0) m-11.
(C.3)
Therefore, by C22 and C.3
E
nN
Σ 0SKSISN-1
E Em & mxlmm]]+ +
Σ
E [[ @ mx/n]]
N
0SISkEN-1
1
Σ (I- H @ nd +
Σ
Eim @ m011I ((())
N2
0SKStEN-1
OSISkSN-1
N-11
1
Σ
(1)
EEm X E/m @ mi([ (1))))))))
N2
1=0
k=t
(C.4)
Applying C.4) into R2 we have
R (0)
(H)
E & <Nd
N-1 N-1
H(g), H(0),(I--HH) (I->) (()))>-EE(v) @ ml + EE @ mm (I ((([--(()))))))
2N2
t= k=
N- N-1
Σ
(2) 'H E & nil) (1)
N2
A=
where the last equality holds since (w) and (I (() commute. This completes the proof. 1
23

---

Published as a conference paper at ICLR 2026
Lemma C.1 A.1
implies that, to bound Ryy the main goal is bound E(m,, @ m.]. Recall that by Lemma
I (I
1 B
(u((x)) n--1 + oa(xi))
E: + c
7)
Denote
bias
(1_)))x(x)) ( ea(xxi)
bias 4-
bias 770,
n
770
1 (x() ld(xi)
(d((ii)
Mi-1 + t1B 2d (x.)
E1 + c - _ c. - -x(())
n
B
with var
77
0. Then
7) 77 yar
+ 77
and
E B mml 32
E [mbias
bias
E [mt
yar
& m
(C.5)
B.
C.
Regarding BG,
1
B I (x()) (x(x)) 2x(x))
B E
(C.6)
Regarding Ct, by the unbiased quantization Assumption 3.1 and no = 0, it holds.
C c E ( (T- 1/ 2a (x)x ex(x) C--1 I [I-?
1 (exx)))(x)
1 Qa [ exxx) +,ZI,.
(C.7)
where
Σ, = R Qa(X,)
1
(5 + tc( -c(0 2xx,,))) (p)
S1 + - Qa(x,)c) Qa(x))
B
E
Qa(X)) EIET Qa(Xi)
+
E
Qa(X)) T " ea(X.)
B
B2
x
x
(( (x),))
(x (x) TT(x.)Te.(xx)) (x, Tox
E
E
B
We then summarize the update rule for Em & nd as follows.
Lemma C.2 (Update rule under general quantization). Under Assumption
3.1
Assumption
3.2
Assumption 3.3 and Assumption 3.4
C E [
I 7 Ou (x) T.(xx) C-11 I (I-B ex(x)) [xxx)x]exxxx + t
1 B
(0) I (I-1 Qu(x) T
H(#)
[I (1-7 Qa(X) T 2a(x))
1
1
B = E
B
where
sup,
R
c
(0) (0)
ct
lo:
+
R
(a) (a)
E
|ar
2 G
B
[ 1 11]
r²
+ag sup E...
tr
H
c
(p)
G--1
(p)
+
B²
with a, = Q.((..),, o, = -e(()) Q. (Qa (X,) (w)-1) and
denoting the
spectral norm.
24

---

Published as a conference paper at ICLR 2026
Proof. We cope with each term in E, to provide an upper bound. For 2; (0)
1
E
Qx(X,)
Qa(X)) (P)
(p) ox(X,) (oxx,))(((()))) Qa(xi)
B
(p) T /m--)) H
Zaz sup Bw.-1
tr H E c
where the incquality holds by Assumption
3.3 For Σ
Σ
1
E
Qx(X,)
EiEl Qx(x))
B
B B
1
ΣΣ (x,))) & (Qu(x,)) Et
E
B²
1
Σ (a(xx)) E (Ru(xx)) Ei) {
B
(C.8)
E [Qx(x)E(2((x)3)]
B
E
(x)))(())
B
0²
H()))
B
where the third equality holds as samples are independent and data quantization is applied to each sample independently, the inequality holds by Assumption 3.4 For E "
(0)
+
(m)
1
E T (E)
(x,,,)
(o) (o)
+
(a)
{((a) (x,)))
B2
E
(x(),)
T
E
c
clo) T
lo
+E
el cl(a)
T 12 1) eucss)
B2
1 B
E
E
(0) (0)
10:
E
(a)
(a)
lag
(x. Tex(x)) exx
B2
sup
E
(0)
(0)
0
E
(a)
6
(a)
B [u(x.) Te(Xxi)
1
sup
E
0
+
E
(a)
(a)
aa
H(9))
B
where || represents the matrix spectral norm.
Combining the upper bounds for " the proof.
Σ
E
(0)
+
Σ
(a)
C.66)
and
C.7 immediately completes
For multiplicative quantization, the explicit dependence of the conditional expectations on w, ren- ders Lemma C.2 inapplicable to the update rule for Ein, mi). We thus propose the following alternative update rule.
Lemma C.3 (Update rule under multiplicative quantization). there exist ca, Ct Cp. Ca and 6, such that for any i E {d, o). quantization 2, is e;muliilicative, then under Assumption 3.1 Assumption 3.2 Assumption 3.3 and Assumption 3.4 holds
(I
1 B
1 [x(( Txxx()) C--1 1 [1
C, <
2 Qa(x) T (a(X) + C.-1)
Qx(( (x((x))e((x)) + +220 (g) H())
+aB
B
B
B, =E
(I- " (x()) 2x(x)) Tx(x)) B
I
1
[-
( 2
1
25

---

Published as a conference paper at ICLR 2026
where
= 81,(1 + Ep)(1 + ea) + 4ep dea (1 tp),
(1) MM
(1+ )²
|w
l
+
an (4ea + e1 + ea) + 11 + 2ea + tp) + 2tp).
B
1 + ed
Proof. To complete the proof, we merely need derive the upper bound for E; ΣS
+ E; (n)
+
Dg + +Ee ( Regarding Σ,, by the computation in the proof of Lemma C.2 i.e., (C.8)
Σ 3 H (o)
(C.9)
B
Regarding ";""
1
(p)
B Qx(,,) (.
E
9--18-1 wi-1 Rx(X,) (((Xx,)@(xx)))) eu(xi))
(p) (p) T
cp
Qx(X.)) D.CK ((((x) T ed(x,)
B
(C.10)
26p
1.X Ta((x)) (a(Xx)
B
26p
(((X)))((X,))W w(o) 2a(X,) ea(x,) (x(())) (o,(x)
Ba
Regarding (=)
1
(x(x)) od(x.))
122
ca
(x(x)) T 2 Qx(x.))
Ba
+ (p)
E
T
OO(X)//-W- To,(x) Te.(X)) Qd(()))
(C.11)
B
2(1 (
E
Q.(X))) J
B
2(1 <p)ce
(x,) T.(X)) (E (
ea(1))
B2
Regarding D
similar to De holds
())
1
(x) & (0) Cx(x,))
B
Co
(a(X,) o.o," ex(x))
B²
2E.
(a((.)) (1y.)) Qx(x)) B
2e0 g T (a()) (a. rx(X.))
E
B2
For the second term,
20
Qa(X,) TT Qa)))) e.(x,)
Ba
2(1
Ea)
(d(x))
ara Tx()))
B
4(1 + (1 + Ea)Ko
(a(x)) (m(()))m To((X.) (xx,))
B2
4(1 + 6p) + ea ea
(x(X,) 2, wTo (x(,))
T.,x) To,Xx.)
B²
26

---

Published as a conference paper at ICLR 2026
For the first term,
2 R (a((x)) 2(00)
4ea
B
(x(X,)
E 2x(X.))
Ba
B
4ea
w-())*
Qa(X,) ea(xx)
(Q(Xx)QQ(xX))
E
where we use & = Qi(y.) - 2a(X, (x(()) Further, by the bound for D
C.8 we have
1
E Qa(Xi))
(dxx))) (xx
o² H
1
B
it follows that
E 46 H
4(1 + Ep)(1 + to [a(x.)) 2d mm Qx(x,) 2xxx
B²
+ 1e (1 ea) (
Qa(X,)w(o) w(())*
2x(X,) ed(x.)) T
B
(C.12)
Further, by Assumption 3.3 holds
E
(x.)))
2d
(X,
w9)))
(9)
T (x(X,) (x)) 3 abtr ((O w())-7 H(a)
B
then together with C.9
and C.12 it holds
(1 + 02
(4c + ep) e.) +1+++1++++ +] + 6p) + 2cp) tr H(0 wlee H(
E,
H
B
4e 41 + cp) + co) + 2ep + 2e, (1 + (p) (x(x)) "mm ed(xi)
B
Note that by the definition of multiplicative quantization,
(9) w()) w [ tr
| 2
1 1+6, Ed
then
(1 + 4co)
||w"|
Σ, <
+
ag (4co((1 + cp) + +ca)+11
+ +1]+ 2e,2 + tp) + 2cp) ((0)
B
1t
4eo (1 + 6p) + Ea)
+ 2ep 2e, (1 + 6p) (x.)))) - (xx,,)
B²
(C.13)
Hence, by
CC.33
C.7
and E
I & ml = 2(B, + Cc), we have
C. 1E E (1 B
I
2x( 2x(X)) C.-1 I
B
(x(X) (x) + C.-1) 1 I (xx) [
+ +8 + tp) tea) + dep + 4ea + tp)
B
B
(1 + 4co)o2
| ag (4e + tp) t0) + +1] 2e +tp) + 2pp) H(1)
B
1 1
Equipped with Lemma C.1
we are ready to derive bounds for R(O) As shown in Zou et al. (2023). we first perform bias-variance decomposition.
Lemma C.2 and Lemma C.3
C.3 BIAS- VARIANE DECOMPOSITION
2023), we perform bias-variance for excess risk, which is summarized as the following lemma. Here slightly abuse the notations of B, and C,.
As Zou et al.
27

---

Published as a conference paper at ICLR 2026
Lemma C.4 (4ias-aavvaaan variance decomposition under general quantization). Under Assumption 3.1 As- and Assumption 3.4
sumption 3.2
Assumption 3.3
N-1N-1
N-1N- 1-N-1
RN
/ <
(I (
B
(1-
(I
C.),
N2
k=t
k=
variance
where
B, = (I -7T Bo.
Bo = 77 & nol
C = - (I- C2 + 2 (0) H
Co=0. = 0.
Proof. By Lemma C.1
N-1N-1
R
(I- Eln @ ml)
N
The proof is immediately completed by Lemma C.2 and E[n @nd 3 2 C.).
For multiplicative quantization, we can directly deduce from Lemma C.4 by the update rule under multiplicative quantization (Lemma
Lemma C.5 variinar decomposition under multiplicative quantization). Under Assumption 3.1 Assumption 3.2] Assumption 3.3 and Assumption 3.4 if there exist Ed, EJ Ep. ca and co such that for any i {d, L,p, a, o), quantization Qi is e-muuliplicative, then
N-IN-1
N-IN-1
RV 2
Σ
Σ
(I- (I-
k-))))kHHH (M)
(9))
B
Σ
Σ
c
N2
N2
t=o
k=t
t=0
bias
variance
where
B (I -TT + (
B( E
Emo no]
c(M) = (I - 7T(4) + + C.
(M)
7 H
co 0.
Proof. By Lemma C.3
I B (x( [ (x(x)) C--1 II- 1 [e0x())
1
C =E
+ 7 Oa(()) (x C. C.-1) en(()) +300 (0) ((9)) H())
B, =E (I- 1
11-3)a(())0axx B [1- (I- ()((() (u(x))
Hence,
E Em nd <2 B, C.)
32 [I 7- + +z M (B.-1+ + +C-1) + (0) (g) H(()
32 (B(M)
(M) + (.M)
Applying Lemma C. completes the proof.
28

---

Published as a conference paper at ICLR 2026
C.4 BOUNDING THE BIAS ERROR
By Lemma C.4
N-1N-1
bias-
(1- ))) k- (9()) B,)
N2
t= k=t
N-1
I 1- (I ( B.
(C.14)
7 NN2
t=0
N-1
NN2 (I H 1) N
B
For S n S N, let - Ei= B, ( = Σm= B (M)
then we only need to bound Sy and s(M) to bound bias term under general quantization and multiplicative quantization, respectively. We first derive the update rule for S, and s(²)
Lemma C.6 (Initial study of S.). For 1 < L N.
S: = (I - - S + 72 22 o Sw + Bo.
Proof. By definition.
t-1
S. Σ II a Bo
k=0
(C.15)
-I 7T(T)
Σ (I O 1--1 Bn + Bo
10)
-I TTB B
S Bo.
Then convert T
(g)
to (1), By C.15
S, -I (7/1)
S,1 Bo
= II- 7(()) S,- + + - -T(9)
(4) o S + Bo
-II > F(() o S ( M((1) S + Bp
<II (1)) 1.11 2 SN + Bo,
where the third equality holds by the definition of linear operators.
Lemma C.7 (Initial study of (( For StsN
S (M) (I- x S MM (1 (1+2)) (1+0)²MM (0) s Bo
Proof. The proof is similar to the proof for Lemma C.6
s(MM) (I- TT((9) + + (M) + Bo
=(I - 77(1)
(((()) - -Tg o s(M + 2y M s(M)) + BB
=(I 7())
(72((1 M ((M) + Bo
3(I (1) +(1+ 2) 2 (M( B
Lemma C88 (A bound for M (4) B
St). For 1 < t < N, under Assumption 3.1
Assumption 3.2
Assumption 3.3 and Assumption
3.4 is?
H then
I (I - + BB)
QB tr
M S 5
(
7(1 YaB (H()))
29

---

Published as a conference paper at ICLR 2026
Proof. We prove by deriving a crude bound for S, and applying M this crude bound. Take summation via the update rule, we have
S. E 1 Bo 7
I- (I )7 Bo-
Note that
I - xT 3 - TTB²)
(1)
( )) 3 (I (I (I- (I- (T-
and further note that _ is a PSD mapping and [I - (I - ( Bo is PSD matrix, we obtain
S: 3 7 -T 'T (I (I ( (T))) Bo-
For simplicity, we denote A = (I - (I- Bp. We then tackle T -1 apply ((9)
o A. To be specific,
F()) T 1
A MM
THO AA+A.
Therefore,
A (7
e M
0 T
o A + (7 A.
Then we apply M (g) on both sides.
M
A) 3 M (T(1)) -1 MM
(1) T(1) -1
o A + M (4) o (7
B
( (n) B o (f() (M o (f o A) (By recursion).
(C.16)
By Assumption 3.3
M B
( A on tr (H (7())) A)H())
H (I - H A H H())
agytr
agtr H (H())) A) H())
W ())
where the first equality holds by the definition of 7 and the last inequality requires the condi-
tion that y < ( Hence, by (C.16). and further by (T())) ((9) I I M I =
og tr (H ( (9),, we obtain
M (g) B o (T AA) 3 Σ
(T (g)
B o (F()) ( (M o (T -1 A)
t=
aggtt (A)
(70B tr (H (())
ag tr())
H())
aa tr ((
Therefore,
H(4) ag tr [I - (I - 7 o Bu
((0) S 3, 0g tr(A)
H())
1 708 tr(H (H))
711 Ya- tr H (v))
yaB
*T(00= -1 is PSD mapping under the condition that 7 |.. which be directly deduced by
Lemma B.1 in Zou 2023). We omit the proof here for simplicity.
30

---

Published as a conference paper at ICLR 2026
Lemma C.9 (A bound for M (0)
S
(
For 1 S N, under Assumption 3.1
Assumption 3.2
Assumption 3.3 and Assumption
3.4 ? (148)ngt((HH0))
QB QB.tr tr [ (I - ) Bb) 7(1 - (1 + Ehan (H)))))
M
o S
MM)
H
Proof
The first step is to derive a crude bound for S Take summation via the update rule, we have
1-1
S(M)
(I- (II-NTh +2 BB = - - (0
+ cBo
K=
Note that
I I->
I-77(1) I - -T (I (I ( +
+ -
(I- +,22
obtain
( 1 7 (0) EyM (1) 11 (I - (I- f (1) + + (0) Bu.
B
Denote ; = (I - (I - 77(1)) + Ey M ( ) Bo. then applying 7(1)
7 o (T 8y ( A < (1 + 3M o (TH Ey -1 o A
Therefore
の - 2n - A A. (T 9) &M A 3 (1+ -1 M o (T
A
Then we undertake the second step, applying g
(n) on both sides.
M B o (T -ExMM - 3 Σ (1++)M ( - " o ( (000-"o00 -A). (C.17)
By Assumption 3.3
M o (T())
o A + an ( (T())) o A)H(
= agytr
H (1 H H A(I - H( H())
(C.18)
= agtr
H (2H() - (H))))) -A) (())
3 (agtr(A)H())
where the last inequality requires the condition that y H Hence, by C.17), C.18 and
further by (T())) ()) = and (
oI ag tr ( H()), we obtain
M ((T (q) ) A)
Σ (11 (1+ ( -1 ' (M (7(0)) A)
3 Santr(A) Σ ((1 + Than H))
a tr())
1 1- (1+ H(
Therefore,
( 3 aB tr [ - (I (( o Bb)
On tr(A)
(M) -1 1 ( M (
H
711 (1+ 2)ya tr (H()))
*(T(() و is PSD mapping under the condition that 7 [1+ (1( which be directly deduced by Lemma B.1 in 7ou al. (002).. We omit the proof here for simplicity.
31

---

Published as a conference paper at ICLR 2026
By Lemma C.6 Lemma
C.7
Lemma C.8 and Lemma C.9
provide refined bound for St
and S [M) Then we are ready bound the bias error.
Lemma C.10 (A bound for bias under general quantization), Under Assumption 3.7
Assumption
3.2
Assumption 3.3 and Assumption 3.4 if the stepsize satisfies < antr(Hoo) then
20B
l|wo w(1)" 11 Dik
+ +Ny #Nylwo
wlo))
112
H
k*
Σ
bias 1
Ny - yag tr HH()))
7 N2 ||wo w(n)" 1| (H + Iwo w(n)"
12
H
Proof. Recalling Lemma C.6.
we can derive a refined upper bound for Sy by Lemma C.8
S, -= - (イ)) o S + 7 2 (g)
o SN + Bo
B
yaB tr
[ - (I 7 о BBu)
3(I )(() S +
HH()) + Bu
- (100BB (r(H())))
1-1
yag tr
I (I - -
BB
3 Σ II +(
.H + Bo
(1- tr(H(s)))
k=0
t-1
7aB- .tr tr (Bo (I 7H N Bo N
Σ II - (()
H( + +Bu
(I- (())
(1 B tr iH(H))))
km0
(C.19)
Before providing our upper bound for the bias error, we denote
Babb == B. - (I 7H( Ba (I -2((0))0-a
Then by CC.14) and C.19
bias
[I
HH()) Σ B²)
222
1
1 1- (I - - ( (I (())+
yaB . (Bo.N)
H(9)) + BB (1 -1- (1)))))))
N2
1-700 (r (H())))
k=
N-1
(I (I-7 (I (I->H(1) ) yan . ir (Bo.N) 1- ya0 (H)))
H(() + +Bo) Bo))
N222
k=
Note that
(I- (1- (I -H(1)) N+2k (I (I- H (I-)))(1)* -1--)) N+#
3 (I (I- )( - -1-
we obtain
1 N-1
) (I- (0) N+ JaB tr (Bn,N)
bias <
[I
H(O) Bo
722
- aan (H))))
k=0
Therefore, suffices to upper bound the following two terms
I az (ro,.
(a- (I (I- H1 N+* H())
N2 - (()
N-1
I = N2
Σ
()) Bo)
-I- N+*
0=0
32

---

Published as a conference paper at ICLR 2026
Regarding 4, since H()) and I - yH(() can be diagonalized simultaneously,
aB tr Bo..)
N
2219)/ - - ( 1
1 = No - yaB ya tr (9))
a tr B Bo.N)
- (1 (1-128)) -
7/ 7a tr (0))
0B tr Bo.N
min
7 7BB tr (H))))
0. tr (Bo.N)
h-*
+ 2
Σ
7(1 yaB tr (H(1)))
N2
where 1-' = maxfk: : 1 2 NN
1 }. Then we tackle tr tr(Bu.v).
tr(Bo, tr (Bo- - (I (I- - (0) - Bo )
Σ (1 (1 -1- --o-))) (wwo - wlo)" , w(0o)e 2
(C.20)
22 Σ min ;)) (1w6- - w()) -000-w0) w,(o)) 2
s (lwo- - w(0) 1R + Ny/lwo - w(0) 11
Hence,
2ag |Wo - w(0) 12 1/300.
+ N71lwo w(()) 112
1..
(
k2 N
+ +Ny² Σ 112 Caf(0122)
h S
Ny(1-7an (r(H(o)))
Regarding 12, decompose H = V(s) ( (() then
1= N2 (I->A ()) - (I- A v(o) T
Bo Boo))
Note that Bo- 7070 it can be shown that the diagonal entries of v(0) Bo ( are wf wf, where (9)
wi= (()) 770 =
(wo w(())). Hence,
1
(1 (:)00) (00)))((111) -(1- (10))(())) wp
1=
NN2
k=0
Σ
w/
[- (1 - -
22
{(a)
w
(1)
min
N2
< N2
Wa
X w?
(g)
?2
||wo - (4)
1|2
(1)
(4)
1-1
+
||wo
-w() 1
u.e.
33

---

Published as a conference paper at ICLR 2026
In conclusion, if the stepsize satisfies y agtr(H())
bias 5/ + 12
20B (|wo - w(0)"|p
+ H w(o) 11 (Tw
(N
hst
Eaa 13
No(1 - yag yaB tr ((())
22
Iwo
11 w(1)
(H
+ Iwo w(()) HO
Lemma C.11 (A bound for bias under multiplicative quantization). Under Assumption 3.1
Assump-
tion 3.2 Assumption 3.3 and Assumption 3.4 if the stepsize satisfies y < (1+E)agtr(Ho)) if there exist ca; Et, Epr ca and co such that for any (d a, o), quantization Qi is e-multiplicativ, then
(IwO- (Iwo-w!lo) wl + +7/l|wo - w(0)*
2(1 + E)aB
H
(
k* N
+ Nx² Easme
bias <
Ny(1 -
1>
22 |/o w/(0"11 (1 1aago (H + l/ -w(1) 1|2 (
$0:00
Proof.
Recalling Lemma C.7 we can derive an upper bound for St by Lemma C.9
S. <I 77()) S. + (1+ +1++))22 M SN + Bo
(1 + E)TaB tr ([I- (I F
oBo)
3(I 7()) o S
H( Bo
(1- (1++))aBt(HH*))
(1 (1+8)aB tr ([z I (I T
1-1
o BB
II >
7()))
H (0) + Bo
(1- (1 E) tr (H)))
k=0
1-1 = aa
(1+ - (Bo - 0 Bo H
H(() + Bo
(I - H
(1 Zyy tr H H())
k=0
Repeat the same computation the proof Lemma C.10 we obtaín
N-1
II H()) (I -I- ) N (1+ 8)aB (Bo.N)
bias <
H(() + B,
7/22
1 1 (1 E) (H(())
Am0
(C.21)
Therefore, it suffices to upper bound the following two terms
(1+ tr (B.,N)
N-1
(a (H- - - (1) (9)
4 = N2 (1 + (r(())
k=0
1 N2
(I->HOy (I H (0) N+
N+ Bo)
Repeating the computation in the proof of Lemma C.10
2(1 + 7)BB
(lwo - w(0)=1p 12.0 + NNyll w(o)*
(((9) $9100
k*
Σ (2(00)3)
I <
N +
NN2
N7(1 + E)aa tr(H())))
T2 < ?2N2 ||wo - (())"
1|
(
J-1
+ ||
w(o)"112
(1(0))
:0:00
34

---

Published as a conference paper at ICLR 2026
In conclusion, if the stepsize satisfies y < (1+))
2(1 2)aB ||wo w/ 1 1,70
+N w(o)"
22
(*
k-
bias S
N
+
N2²²
Σ
Nº(1 - (1 + E3700 tr(H(()))
N2 l/wu w(9))11
(H 0.6.)
+ Ww
w/9)
112
H 10:00
C.5 BOUNDING THE VARIANCE ERROR
Recalling Lemma C.4 and Lemma C.5 the key part of bounding the variance error is to derive an upper bound for C, and C where
C=:= (I TTB + + (1)
H Co= 0.
( = (I
+ (T 9))
TTt
(0)
(M)
+ + (1)
H
c 0.
We first estimate C, by converting _ to T())
C. + 12 (g) 2 H(9)
C (I 7TT
=(I (T
C,-1 1 T 41)
C.-1 + (1)
H
(C.22)
=II + 7(())
+1 M 0 C, + (0) H
3(I 7
C - C. 200 (1) H(9)
Similarly,
c(M =(I 7T (9) + C + 7 10
(M) (9) H(1)
=(I - ((()) ((M) (T()) T iM (4) (M))
+ +7
(4) H())
M
(C.23)
=(I 7 (M) (M -MO + (0)
(M)
+ + (
M H(
30I - 7) (M) + (1 + aM C. + 0 M
(4)
H
The following two lemmas provide upper bounds for M o C. and M o c(M) (g) B o C.). For 2 1, under Assumption 3.1 Assumption 3.2
Lemma C.12 (A bound for M
Assump-
tion 3.3 and Assumption 3.4 if the stepsize 7 S |ptr(Hor.) then
agtr(H no (g) Mge C. - (a)))((()))
H
Proof. The main goal is to derive a crude upper bound for C, Denote Σ = o(g) H()), Step 1: C, is increasing. By definition,
C, = (I TT()) C. --1+
2 Σ I - E
(solving the recursion)
= C + 2 - -T (2) yr
Σ
2 C,-1- (since I
TT
is PSD mapping).
B
35

---

Published as a conference paper at ICLR 2026
Step 2: Coc exists. It suffices to show that tr(C)) is uniformly upper bounded. To be specific, for any 2 1,
1-1
C₁ = 2
TTB
(0))
o
3
E
I - Σ
y
k=0
=
Then
tr(C,) 2
Σ
00
(I- = = Σ
00
tr((E)
yrr
oc,
1
(1)
where the second inequality holds by the iteration:
tr(E,) = tr(E,-1) - 2y H E,-1) + 2r
E
x(1)
x(1)x(1) x x(x
B
S (r((E_1)) - (2 2 (agtr(H()))
tr(
< br (I-
< (1 (1-778)) (4)) Lr (E
where the first inequality holds by Assumption 3.3 and the second inequality holds if y s Hgtr(HHoo))
Step 3: upper bound C By the update rule for C₁,
Cx= (I 77-BBB 91) C.c +
which immediately implies
C - O.
(C.24)
We provide the upper bound by applying 7(1)
T(t) o C.s = T Coo 7M o C.o - -7M(1) C.o
= m0 B
(0) Co ( Coo
+ (n) C.o.
T(() and ( and the second equality holds by where the first equality holds by the definition of C.24). Hence,
(g) 7(0) C.. 3 ro H + C.e.
Therefore, by applying (()))-1 have
(g)2 C.e 1 70
-7W0- H + ( MM o C..
3 300
(
Σ
x
(T(()) o ) o (1 (0), (solving the recursion) (C.25)
t=0
We first deal with (7())) (())
8
(T())-- H (1) Σ
[I - 7 (
1=0
8
7
[I H
(1)
H (4)
H9)
(C.26)
1=0
II
H
[g)
(9) (9)
1=0
=1,
36

---

Published as a conference paper at ICLR 2026
where the second equality uses the definition of ((1), Hence, by (C.25) and C.26
Co 1 3706
(g)²
(T(()))) :M o
G
(T)) -1 o
MM o
t=
0c
G00 (0)2
(T((1)) -1 M 7antr(H())
(C.27)
3 30
(0)2
G0G (q)
(0m)r((()))
where the second inequality holds by the fact that M o I3 ag tr(H())) H
B
Here complete deriving crude upper bound for C₁:
20 (
C, 1 Cx = 1- vastr (H()) I.
Then by M (g) oI ag tr(H H again,
2
agtr(H()) hog G H
M C. (-)))t((H+))
(g)
Lemma C.13 (A bound for (9) c (. For 1 2 1, under Assumption 3.1 if the stepsize 7 S (14 then
Assumption 3.2
Assumption 3.3 and Assumption 3.4
Øgtr H (0 (
2 - (1+ (H) M o ((M)
H
Proof. The proof idea is similar the proof of Lemma C. upper bound for ((M) We deduce from the proof of Lemma C.12 that
while the main goal derive crude
C (M) CLM) (T (4) ETM(
OoM H
(g)
(C.28)
We provide the upper bound for c! by applying 7()
7(0) (M)
(T(e)
M
(4)
((M) (1 + M (g)
C(M) -M C(M)
B
(4) (
( 11+ (g) C(M) MM ((M)
1 3 M
H
4 + 7> ((M)
where the first equality holds by the definition of (1)) and the second equality holds by the definition of (M) C.28). Therefore, applying (T(+)) have
(M)
= 10 M
(g)2 (+())-- H (0) + (1 + 1 M c(M)
2 3 70 M
(11 (1 ( (1+
M (7())-1 H())) (solving the recursion)
(C.29)
((T(") (o)) exists under the condition that 7 (H) which can be directly deduced
by Lemma B.1 /ou et al. 2023). We omit the proof here for simplicity.
37

---

Published as a conference paper at ICLR 2026
By the computation (C.26 in the proof for Lemma C.12
(T(9))-'oHC)) H()) 3I
(C.30)
Hence, by
C.29 and
C.30
C
3 70 Ad
(4) 2
(1 (1)))(((()))))) y
t=0
= 70 M
(q) 2
(1+08 (1
M aI
3 M
(
Σ (1 o (1 8) I
( 3 7a M
Σ ( (1+t)aatt(HH²))
t=0
70 (1)
1 - (1 + ()
where the second inequality holds by the fact that M o 1 ag (H H Therefore, we complete the proof by
( (1) 1- (1 +() M e c(M)
H
By C.22 CC.23
Lemma C.12 and Lemma C.13 we can provide a refined bound for C, and C.M)
Then we are ready to bound the variance error.
Lemma C.14 (A bound for variance under general quantization). Under Assumption
Assump-
tion 3.2
Assumption 3.3 and Assumption 3.4 the stepsize satisfies aBtr(H
then
k* N
E 1>
variance 1 yo (H())
Proof. We first provide refined upper bound for C,
C.22
C. - ((l)) C. 2 MM (g)
C +
20
((
(g)
<(I - 7 C.-1+
2
(99
+
7 (9)
o H
yantr(H
23006 (g)
=(I - -71))
C.-1+
H
(
200 2
[-1
- o H(4)
-
(solving the recursion)
(C.31)
120(00
H
(9) I H9)
(agtr(H())
22000) (g)
(g)
H H 44)
1 (a
A=0
2
Yo
(I (I -I- ((-))
1 1- (a
38

---

Published as a conference paper at ICLR 2026
where the second inequality holds by Lemma C.12 and the second equality holds by the definition
of T(.).
After providing a refined bound for C₁, we are ready to bound the variance. By Lemma C.4
--1
[I [I-~ H H( C.)
variance
N
N-1
- (I - H H N Ci)
N t=0
I (I- (I- W I - (I (I- (HHH()))))
2 1 yagtr (H
1 - (1 - - 1 (1 - )
N2 1- (aglr(H())
P0
1 (1-1- M (1 2 [1
(C.32)
2 1 (agtr(H())
20 (g)2
1
- (1 (1-
N2 yaptr(H(()
22 (
1
11,2
min
N
(agtr(H())
220000 (g)2
1
+² N2g2
Σ (2/0112
2N
()))r(H(()
1>k
G
h- N2
Σ (2
Btgtr ())
N
where the first inequality holds by (C.31 and the last inequality holds by the definition of ks
max k ((0) 2 No .. This immediately completes the proof.
Lemma C.15 (A bound for variance under multiplicative quantization). Under Assumption 3.1 As-
sumption 3.2 Assumption 3.3 and Assumption 3.4 if the stepsize satisfies Y < (I-2)ogtr(HH))
if there exist ea; e1, ta and to such that for any i E (d,l.p,a,o). quantization Q e:- multiplicative, then
2 variance S 1 (1 g)
k" N
+ NN2
Proof. Applying C.23 and repeating the computation in the proof of Lemma C.14
M 2(I - (()) (M) + 7 ( ((M)
2
(g)
H())
M
I(I - o CLM
+ 7 (1 + é)
&gtr
H +
+220 (g)
MM H())
1 (1
(agtr H())
200000 (g) H())
=(I (T) o (.M)
1-(1+))aatire H(()
( M
70
I-
(I-II-H())
(1 ((())
where the second inequality holds by Lemma C.13 and the last inequality repeats the proof in
CC.3
39

---

Published as a conference paper at ICLR 2026
Therefore, repeating the procedure in the proof for Lemma
C.14
directly deduce that
(K- N
k2 Nyp.
variance 1 (1 + a)
1>
which immediately completes the proof.
Lemma C.16 (A bound for Ryy under general quantization). Under Assumption 3.1 1
Assumption
3.2
Assumption 3.3 and Assumption 3.4 if the stepsize satisfies y < agtr(Hoyy then
x/ /2 <
20B ||wo w(0) 11
0.
+
N
Wo wl0)"112
H 6":00
K*
N + N₂
Ea
Ny(1- - yaB (r(H(()))
|Wo w( 11 (HI j-1 + llwo w(
HP(.)
y2 N2
(g)
k-
N + Na2 E((e))2))
- HaBtr(H(4)
i>
where K- max k 1(0)) 2 Ny
and
sup: {
[o] T
(a) (a)
E
6,
6
0z
+
E
6 6,
lag
(g)
0
B
Wr 1 1)]
o²
+ap sup Ew.-1
tr
(())g
E
(p)
(p)
B
Proof.
The proof is immediately completed by Lemma C.4 Lemma C.10 and Lemma C.14
Lemma C.17 (A bound for Rx under multiplicative quantization). Under Assumption 3.1 As- sumption 3.2 Assumption 3.3 and Assumption 3.4 if the stepsize satisfies y < (T-e)ogt(HHo))
if there exist ed: El, Sp. E and e, such that for any E ad, quantization Q, is et- multiplicative, then
2(1 + t)BB [Iwo w10)"1p2 12:00 + N Wo w(O) 112 H(O)
(2 /2 <
N 12
k-
Nº² Σ E((00)))
N^(1 - (1 E) tr (r(H()))
|w w(0)
(HT + |Wo - w( 1|2 (P(() (
+
7 N2
2
10 N N222 +
M
1 11 ())))r((H))
1>k
where K- max k ((1) 2 Nv and
8 + t1) + ea) + 4ep 4ea(1 (1++)) tp),
(q) (1+ 20
||w*||i, an (4ca((1 + (1 + 6a) + 1] + 2ea + tp) + 2ep)
0
+ 1 ca
B
Proof. The proof is immediately completed by Lemma C.5 Lemma C.11 and Lemma C.I5
40

---

Published as a conference paper at ICLR 2026
D DEFERRING PROOFS
D.1 PROOF FOR THEEORM 4.1
Proof. By Lemma A.3
B
and Lemma C.16 we have
(.E(wN) x)] VarErr + 2BiasErr + ApproxÉπr,
where
| [Wo-- 11:00
20B
+ l|wo w
+o(
(g)
VarErr
Ny
H
k*
+ Nx2.
(1(10)2)
1 1-7antr(H(a))
N
BiasErr
N2 l|wo (w-10=
w-(o)
1|2
(H
)-
+ l/
w(1)
1
Osk
k9:00
ApproxErr= = -//
///rrrl|/0111 DM 2 ||
with D = H(
H. K- = max k: (0) 2 No
and
D H(H
(I I- -I-(1- (I-
D (
H,
(o)
(a)
sup;
E
c c
+ E
a
(g)2
B
o²
+a sup Ew
E
(p)
4-1
B
Let the initialization wo = completes the proof.
D.2 PROOF FOR THEOREM
We prove tighter excess risk bound under multiplicative quantization in this subsection:
Theorem D.1 (Multiplicative quantization). Under Assumption 3.7 3.2 3.3 and a.4 there exist ed, Et, Ea and e, such that for any i E {d,1,p, 0}, quantization Q, is e-multiplicative and the stepsize satisfies y < +ol as follows.
then the excess risk can be upper bounded
[[E(ww)] $ ApproxErr VarErr + BiasErr,
where
ApproxErr
S lted
|| l
BiasErr X
1
/l/((1)
1|2
(H 0.k )-1 + /|wwl))
N2
H
k0:00
K3 (K. N Ny (a) Σ x3 70.(1+
(1tc,)
B + aBo
VarEi
Ep + (1 (p)](1 + ea)tr (H)
with
2h = = (Ltte))))) 11"" [1 ||w(o)*
0ie-
||
"10 1[2a5 (
Ny
Proof. By Lemma
A.3
Lemma B.1 C.1 and Lemma C.17 we have
E[((w) ) 2arrE + 2BiasErr + ApproxErr,
41

---

Published as a conference paper at ICLR 2026
where
[wo (2) 11
OM 2
+ + + 2
l/wo
w(o)
H ktino 11
Ny
k"
VarErr
+
+Ng2
1 - (1 + ()
13k
BiasErr
l|wo -w/--w
w/
1|
[H
+ llwo -w(0)
1|2
N2
'H
0,k
og
ApproxErr 2(1 ca)
||
ed
||w*
||
H
(1 + cd)
I+ cd
with k- =ax max k :1 1 2 Nr f and
8= 8c + (p)(1 + (a) + 4G + 4c, + 5 (1+ ca(1 +p)),
(a) M
(1 + 4c
+ |w'lli aB (1 + 2c +2) + 2cp)
B
et
S (1
+ co)
+ Let initialization Wo = 0. Regarding VarErr, noticing that + + (1 + ca]l t + ca + tp),
+ || a (E + (1+ (1())
B
have
||w(0) Ii
(1) M
dikt + || H
+ 2(1 + +jaB
Ny
(1 + tn)o2 B
+ on | (co + (1 + col(sp + ca(1 + (p)
||
taB(1 60 + ca +p)1)))))
+ *||w(1)"
Ny
H
Then the proof is completed by (H())))) = (1 + (a)tr(HH)))))) and 1 = (1 + ca)
Theorem 4.2 can deduced from Theorem D.I by noticing that
2 S + (1+ + co + ca(1 + []]]][w"]]
where we use
[w(x))
((o)
|w" 1 1 1 Ea
||w(10"11
12 1
S /w())*
S || 1|
H(e)
NY
D.3 PROOF FOR COROLLARY
We provide a tighter excess risk bound under additive quantization in this subsection:
Corollary D.1 (Additive quantization). Under Assumption 3.1 3.2 3.3 and 3.4 if there exist
ca, and 6n such that for any i E [d,1,p,0, quantizatiion Q is é:-additive, and the stepsize satisfies y < an [de) then
B[E(wN)] ApproxErr VarErr + BiasErr,
where
ApproxErr
S da + Ed
|
BiasErr X
ll/w(o)' "1w [1a + (|
120000
N2N2
60000
|w(w)' '11
|w(("" 112 H(w
o² ocotoa ausopptt(( (T()) + dea)
aB
Ny
kk
VarErr S
+ NN22 Σ (dz (di+tcaf ca)) cca)
1- yay (tr (H) + deal
N
1>k
42

---

Published as a conference paper at ICLR 2026
Proof. By Theorem 4.1
[[E(wN)] < 2arrEr + 2riasErr + ApproxErr,
where
|| 1/00
+ to (g)2
2
Ny
+
//
H
k"
N +Ng7 Eamop) 1
VarErr
I
1-7agtr(H(9)
ApproxEIr I || I (H + //
N2 (2)* (2)*
BiasErr
S
H
+ ||
DH
with (g)2 G
o² d'tsup | jele (0) fot
1o
+E
(4) (a)
Ja
E tr H (10)e())e()) (p) ']
+ a sup,
and
DH HHH Ny
HWW) (DH)
Under additive quantization, it holds
(r) = tr(H) + ded,
E
Σ (): +ea)3
isk"
and
(g)
0² + Co + co
G
B
+
(tr)) + deal.
Then we have
(w)
||
ww())
1|2
+ o tc,ta + aBepltr(HH) + dea]
VarErr X
Ny
H
k-
+ N22.
Ede +(a))
1- y-yan (H) + dea)
N
15k"
The proof is completed by Lemma B.2
2
Aiea
ApproxErr s 2(Aa + ea)²
| 1
|w =
|w"]|
(Aa + ca)( + ea)
Ao + ea
Corollary can be deduced from Corollary D.I by noticing that
|w(0) (1) (10)
Ny (1]
1
[| (1)* 2 Ho = w H(H Hw" < || 1
'HW'
D.4 PROOF FOR THE MULTIPLICATIVE STATEMENT IN COROLLARY Proof.
Recall that kj = maxfk: do 2 Ng
kg
O
Nr
w
+
|| H + o²
75
Ro
Nº²
X
N
-aggtr (H)
EffectiveVar
|| x [[Howng -- +
|w" 2 Hegin
22N2
EffectiveBias
and by Theorem D.I
B[E(wN)) ApproxFrr VarErr + BiasErr,
43

---

Published as a conference paper at ICLR 2026
where
ed ApproxErr S
|w"]
BiasErr v
1
-I|w())
112
(H )-1 + ||wlo))1
cd
?2N2
k2 N
(1te,l0²
B + an
VarErr S
Ny + Ea)²
Σ
1 7a + E
+ Ep + ea + Ep]](1 Ea)tr (H)
with
//w(o)) 1|2
2 = (Lot (L+t))))))) ([+e))) (Lltee)lltstteaaeee
0.6-
||
1| 1155
Ny
We then compare the upper bound of [[][ with Ro. Regarding VarErr, we first analyze 1
N
+
21 + (a)² Σi>k- X2. Note that for kü < i S k", .xx(+ea) S a < Ny. we have
k* + (1 + cca) Σ X
N
i>
k*
kū
+ (1))(++
Σ
N
Σ x
N
N
kicisk
k*
kō
N22 + <d)²(k - ki)
1 22 ca)
+ +NNg c2)
Σ X
N
N
i>1g
k
+
(1+ Ea)2
Σ X²
112
((9)
We then analyze
+ l|w(1))
112 H1
Similarly,
Ny
|| "1
+ ||lw(0)"
1
Ny
l|w(())
12,00 4014"
0146
||w(0)
+ +llw00)
(() kg
Ny
||w(s)
+ [|
1
Ny
H(9))
8:00
||
||w 11
Ny
Therefore, the sufficient conditions for VarErr S Effective are
BAB w 12 |w
o²
Ny
long
+ [|l
Hagi
6a S 1, Cas Eas S
^ 1,
[w"
Secondly, we analyze BiasErr. Similarly,
1
|/w(2)
(x)
112
(H )-1 || 1
N²
1
(Iw(0M12
+ l//(())* 1
(2 bg -llw(s)"
+ -l/w(())
1
N2
H(0)
10:00
(D.I)
1
||w(")*
112
3-1 + ||w(s)" 112
N2
(TT())
(1))))
bikg
1
||w*
||
(( + ||w" 11
1i EffectiveBias.
N2
44

---

Published as a conference paper at ICLR 2026
Thirdly, the sufficient condition for ApproxErr X Ro is cd S
Ro
Overall, require
w
||w
ll
Ro
ed 2
1A
for Ea: Ep X
A 1.
W
2
Ba wx | H
w
l
H
D.5 PROOF FOR THE ADDITIVE STATEMENT IN COROLLARY 4.2
Proof. Recall that kj = maxfk: d 2 Nyt.
Ro
kig N
i k> N2² 2
x
aB
N7
2
+
|
Magioe
o²
1 yantr (H)
EilectiveVar
|| 1 (Hoagg) )-
2 N2
1:000
EffectiveBlas
and by Corollary D.1
E[ ApproxErr + VarErr + BiasEIr,
where
ApproxErr S
ed
|
BiasErr X
N2N2 ||w[o)"||2
([TT)0) 1- )-1 + ||
da cd
21:00
[w(a)*
11:00) D.k"
VarErr
aB
Ny
+
|w
H
+ trattn + aBBepptr(HH + deal
B
k-
+N22
Ed [(N++e))))
yaB (tr(H + deal
N
i>k*
We then compare the upper bound of ([[((w)) w)] with Ro Regarding VarErr, we first analyze + N222 Ei>k- (As + ca)2. Recall that for kg < <i S k", Ny cd
S
di < Ny.
N
+ Nº² Σ (Ax + Ex)
kü N
+
k
ki
N2²
Σ
(A: (Ai+ea) + N₂² 1(i+
N
N
1>
kj Ajcish
+
N2²
(Ai + E2)
N
(w) 11,
We then analyze
+ ||
112
Similarly,
Ny
H k0000
(g))) Ip3
|| Loag
||w (4) 1
w 11
(D.2)
Ny
H 1000
Therefore, the sufficient conditions for VarErr S Effective are
l/w 11 +
N
+ Na2. Σi>kg X²
o²
Ny
+ 1| H.g. 11
Ed S
co S
Nr2(d- kg)
(ay(tr(H) + deal
tr(H) des
|| 11
Ca, Co S 0² + Bay
+ || lit
Ny
45

---

Published as a conference paper at ICLR 2026
Secondly, analyze BiasErr. Similarly,
//w(())
( (1I6)) 1-1 + l/ (
N2
N2
(Iww
112
(HOS )-1 + //
(H
1g )-1)
//w(
"
+
1//w(1)
//w/(1)"
1|2 (1H(0) 1-1 + 1|w(())
1
N2
H
0000
l|w"
EffectiveBias.
?2
(g)
Thirdly, the sufficient condition for Apr S Ro is ca S T
R.
Ad.
Overall, we require
H
Ca
N
NN2 Ei>nz
X²
A
RoAd
Car S 0² + BaB Ny
||w*lBaan
+
|| l
Ny2(d kg)
w
2
H
Iw' 11piigg
+ || -lagg
Ny
cp (r HH) + deal
+
tr(H) + ded
D.6 PROOF FOR THE MULTIPLICATIVE STATEMENT IN COROLLARY
Proof.
We prove by applying Theorem
B[E(WN)] & ApproxErr + VarErr + BiasEIr,
where
ApproxFrr S
Ed
[ BiasErr
///(()) || bi )-1 + 1/w/(o)
cd
N2
(H
H
k* VarErr $ N + N
(1tc,)o² + an + c. sp + co + ep)i 1|
(1(1+ +e)²
Σ x2
7a + 6.]l + + ea + Ep + Ea)tr (H)
We first deal with VarErr under power-law spectrum Assumption 4.1 zation, can estimate k- by
Under multiplicative quanti-
1
(1 + ka)
Ny¹
that is
k² ~ [Ny(1 + ca)
(D.3)
Further, the power-law Assumption 4.1 also implies that for any positive k
(D.4)
By D.3) and (D.4).
min fd, [N N ex) (NY(++
k.
N + N1 + ea)2
Σ k>
N
min (N c=)
N
Moreover, under polynomial spectrum Assumption
(r(H) マ 1, [ =
46

---

Published as a conference paper at ICLR 2026
Therefore, under Assumption 4.1 by applying stepsize y < and taking expectation on w It holds that
1
min a (N + ca)l=
Ew VarErr
(1 + eo)l t totttt ea + te)s,
(D.5)
N
where we use 0² $ We secondly deal with BiasErr. Under Assumption
4.1 using D.1
1
Ew BiasErr <lww
l (oong)
+ |
2
22
Ap
A
227
i=1
kg
(D.6)
X
NY
E>kg
kg
+ (k,)1-
N7
max { d' (Nv) 1/a-1
Therefore, together with D.5 and (.6), and taking expectation on w we have
+ max { (d'= ( (N7))/a-1)+ min {d, [N + ea)l+
B[E(WN)] Ed
(1+ + + ca(1 +c(1+())) +
cd
N
Denote R = [B[[[[
Itea
d > [1 + ca)] In this case,
R R(N(7)) [Ny(1 +z)l
(1+ co)l1 + + ep)
N SN (1+ t c +l "o.
(Ny) <d [Na(1 In this case,
R RS((N) + (1+ + +ea(1 d S(N7)
N
N7 + ea))
+ aall + 1
N
SN1/0-1 (1+ (1 + ca + (1) + 1//
d S 2Na this case,
R 5d + (1 co][1 + ca (ca (p) N
Sal +11 + [.)++)))N
Overall,
[[E(wN)] S
6d d-a N1//- c + ca(1 + ())(1 + t/.
1 ea
47

---

Published as a conference paper at ICLR 2026
D.7 PROOF FOR THE ADDITIVE STATEMENT IN COROLLARY 4.3
Proof. We prove by applying Corollary 4.1
BlE(wN)) S ApproxErr + VarErr + BiasEIr,
where
ApproxErr
Ed
||w" H BiasErr X 22N2
(H (H )- + //w(2))
$122
Aa ed
H 10
on wxww I +
tcata + 00 (t((H))))))))))) + ped) B
h
VarErr S
N NN². Σ (4= + Ea)2
(I + pea)
We first deal with BiasErr. Under Assumption
using D.2).
Ew BrasErr IEw-
N2
w 1
(Hoigg)
+
w
d
X
22
i=1
i> Kig
hg
d
(D.7)
A
N7
i>
kg
+ (ko))
Ny
max { d dd (Nv))
We then analyze VarErr. If ca + d 2 Ny. then
k* N
EAi++ea) d
+ N2².
N
i>k
Otherwise,
+ N22.
Σ()=
(A: + ca)²
N
12k"
Ny
Ed
-1/0
+22y
( 1
Ny -cd
-(1-2a)/a
+ e22222
d
Ny cd)
11/a
N
-1/0
-1/a
N7
1
(d)
+
N222222
d
Nr
cd
N
Id-a v ( (1_
NT -dd) ] -1/a
Denote koff
follows that
Ea++ hell + egN keit)
N + N2?.
N
1>*
Hence, under Assumption 4.1
taking expectation on w and applying stepsize 7 < 20 (PH)
Ew VarErr S 1 6o + (p + dea) kif N3 - kott)
(D.8)
N
Therefore, together with D.7 and D.8), and taking expectation on w", we have
B[E(WN). S -- + ed
Ed
maxx d (N-) + (1
Eo + Ea
+ (1 + dea)
bett + =eNN222(d
keet)
B
N
Denote R = [[[[(N))] g=n+ed
48

---

Published as a conference paper at ICLR 2026
d = 1/ this case, let e'3 = ea. Then d' d 1/([N>) - -'" That is, d a NY(1+e))
(N² -ca) 11/a + &2g222 d (N,, -ca) cd) -1/a)
(c))
R (S((N))//0-14
1+
+ Ea
+ tpp(1 dea))
B
1/a (1 2222222 + -
NNy- -
=(N7))/a-1
1 +
co
+ Ea
+ tzp(1 + dea)
B
N
(1 (1+c))(a(N))) - E223? + GDN2Pd
S(N7))
1 1
co c
ca
+ cpp(1 + -dea)
B
N
[ [1+ + N2
S(N7)
1
t
+ t dea))
B
We then focus on Nd.. By No S
de Itej.
we have
2 dlta (22)
Nydd x
dl-o
1+ E/
1 1 c
Therefore,
R R 11/a-11111 + (14//-a111ddaa dd l/ (1 11
Eo t
+ cp(1 + dea) + (1+
to + ea
+ Ep(1 + dea)
(d)) 11-0
B
B
1 + do
1 Ed < do < 1 Ny
this case, Ny -
1
d
E < a that is, d < Ny(1 + E2) Consequently,
R RS((N)) +
1 Eo tc
( + dea)
B
N
S(N9) +
1+
Eo t
+ t + dea)
(1 + 2/
B
6 tc ca
+ ep(1 + dea)) (1 + d' (2
1
B
d= N
In this case,
R Sil aa +
1
E t
+ ( + dea)) NV
B
Sai a +
1+
6 t
+ ep(1 + des)
N 11/a-1
B
Overall,
deEad + 1+ Eo t
Ep + dea) ) | [ deea)= + (1 (+ (d))
B[E(WN)] S
1 dded
1 + doed
E DISCUSSION OF ASSUMPTIONS
In this section, we verify Assumption 3.3 and Assumption 3.4 under the standard fourth moment and noise assumptions made on the full-precision data (Zou et al. 2023).
Assumption E.1. Assume there exists positive constant ag > 0, such that for any PSD matrix A, holds that
E xx
Axx 1 oo tr(H))
49

---

Published as a conference paper at ICLR 2026
Assumption E.2. Assume there exists constant og such that
E (vv - (w" x)) 1 of
We consider specific quantization schemes.
Example E.1 (Strong multiplicative quantization), We consider a strong multiplicative quantiza- fion. In this case, there exist constants Cas Caa such that
(1 (d)
|x cxxx",
(())
A (d)
xx 1 , Axx..
Example E.2 (Strong additive quantization). We consider strong additive quantization. In this case, there exist constants Cd, C'a such that
E
B ( (d) |x]
eal, E ( (d) T [x] Ed)r((A)I
(E.1)
E. DISCUSSION OF ASSUMPTION 3.3
Under Assumption E.1
ready to verify if Assumption 3.3
can be satisfied. We begin by:
x9) Ax
(())
=E
(2)
Ax(1)
x ( x((1))
(x(2)
Ax (xx T + tl())(1))
22
x Ax + (d)) Ac (xxx + c c(0)
<4E
(E.2)
xx Axx " 4E (()) (e(
=E
+4E
(xx Ax) e ((0) c(d) + 4E
(eld)] Aela) xx
Lemma E.1. Under strong multiplicative quantization E. and Assumption
E.
E x
(4)
S on(1 + (t)((())))) A)H( Ax (9) (
Proof. We proof by (E.2p. From Assumption
E
xx
Axx
I on tr(HA)H.
(E.3)
Under strong multiplicative quantization E.1
we have
E
(E(()) Ae
T
xx
egE
xx
Axx
_| 3 edon (r(H)),,
(E.4)
E (x T Ax) ((d) ((0) T]
cB xx
Axx
3 ed00 tr(HA)H,
(E.5)
and
(10) Ac (d) (d)
E
XX
Axx
1 G0D trHA)
(E.6)
Therefore, together with E.2
and E.6) we have
E x (0) Ax
S ova + + (r)))(HH)HH < on + + ())
That is, under strong multiplicative quantization Example R. and fourth moment Assumption on full-precision data, Assumption 3.3 is verified.
Lemma F.2. Under strong additive quantization
E.2 and Assumption E.
(x 1 S (1+ ap) (1+
E (1))
H ()) 9)
2
50

---

Published as a conference paper at ICLR 2026
Proof. We proof by
E.2
Under strong additive quantization E.2
B ( ((()) Ae
Xx
1 EctrrAA)
(E.7)
(x Ax) c(u) c( ] ccatr(HA)I,
E
(E.8)
and
E
( (()) (e tT((L
(E.9)
Therefore, together with
E.8
and E.9 have
9
(9)
9)
(e)
(1 + ao)
(1+ (H(
A)H
That is, under strong additive quantization Example full- full-precision data, Assumption 3.3 is verified.
E.2 and fourth moment Assumption E. on
E.2
DISCUSSION OF ASSUMPTION
3.4
Under Assumption
E.2 we are ready verify if Assumption 3.4 can be satisfied. We begin by:
(y/ (1) (w
x ( ) (1)) T
E
(y()) y + (w", x) + (w", x) - (w(s)) (x(0)))2x(l) x(())(())
=B
(v - v) (0) (() "] 3E (w (w", (x"",)))xx (1))
33
(E.10)
((w", x) (w(a)) xx())) (0) x(1)) ]
+3E
<3 (w/)) 2) x(1) 3E (v (w", x)) 2 (1) 7
+6E (ww()) w", x) xx (0) _ + 6E (w(()*
c x x(1)
Lemma E.3. Under strong multiplicative quantization
E.
Assumption E.1 and Assumption E.2
(v)0) (w(n)" (x()))) (x00)))
3 of +
1 ca + c' an || 11 H
Proof. Regarding E (yr - (w", xx)
(y (w", (w"'x)> xx)x)(1))))))))11 x0)) |]
-2E [(y (w" x)) xxx 7 +2E (y
21E
(w", x (d)
=2 + EE) (1y - (w", xxx 1
<2(1 + H,
(E.11)
where the second inequality holds by the definition of Example E.1 1 and the last inequality holds by
Assumption E.2 Regarding (w(x)" , (x)
E0 e (1))))(())(1))
(a)
((d) (()) (( ((0)
=E
(E.12)
<2E
((d) w w()) (d)) Xxx
2E (d)
wlo)* w( ((d)) (d)) T]
=2esaotr(w(e)) TT H)H + 26, 26 -T H)H,
51

---

Published as a conference paper at ICLR 2026
where the last inequality holds by the definition of (Example E.1 and Assumption E. Regarding the term
(wwl)) w. x/2
32E (w(l) w , xX + 2E (wl()* w x) c(c) T
<2 + ea)ontr ((w(o) w") - H
(E.13)
where the last inequality holds by the definition of Example E.1 and Assumption E. Re-
garding (y/a) y) x() T. if we further assume that there exists a constant C such that
xx 3 CH, then
E (y)l) y) xx)xx())x(x) (7) T
32B (1(() y)"xx" T ] 2B (g/(0 (4)
32(1 + e2)lR (w)()) -))2xx² T
(E.14)
-2 + +2)e,BEly2xxx
32 + CH.
Therefore, together with E..0
E.
E.
EE.3
and E.14 we have
E (1)0) (w(9)* (1)))) (0)
3(1 C + (ca + 22)00 w(9))
(1 (a)aa)
ca)an w(() w
H +1+
H
H
3 a1l+ea)(e + +e) + (1 + + <')oollw"
H
H
1 <a c' w
o +a
H
1 cd
That is, under strong multiplicative quantization Example E.1 and fourth moment Assumption E.2 on full-precision data, Assumption 3.4 is verified.
Lemma E.4. Under strong additive quantization
Assumption E. and Assumption E.2
E (1) (w(1)* x2 ']
z 0 + a eal1
|| + (1 0o)
|| H H(a)
Proof. Regarding E (yy (w", xx x1) I if we further assume that E [(v - (w", xw'x))) I
oi then
(u x))² (9)
<2F [Vy - (w", xx)xxxx [ + 2TE (u (w",)) xx)22 (c))
-20 + I
(B.15)
Regarding E (2)l)) -1)3x(0)(()) J.
(v(() - -2)=x(())()))) CacI
E
(E.16)
Regarding
E (w(o)" _((0))2((0)
(w (w))) (x))) (1)
E
(d)) (1)) ( (00)))))
=E
(E.17)
<2E
((d) w()) w())* (d) Xxx
2E
(d))
w(o)* =T ((d) (()) T]
(1))
=2eatr(w(() (6)* H + 2c
TT
I.
52

---

Published as a conference paper at ICLR 2026
Regarding (wln) -w, x
E (w(2)* w x),x) x(1)
=E x (wll)* ww)(w(()* w") xxx (2) x(1)
-2E x (w(n)* - -w')(w((0) - -w') xXx + 23 x T <T(wl()* - -w")(w( -w") (x0) c(d)
((w(()) (w)(((() w + 2eatr ((w()))
-w") (ww())--w") -w")
=2antr
(E.18)
Therefore, together with
E.15
E. and E.18) we have
E (v/()) (w<()* xx(() x()))) ) xx(())))) (2)x(0) (0) T
300 (H) cd(1 + 1 |
H(a) + (1 ax) |w/n) w will H(a)
$ oo a + ca(1 1|w'|| + (1 + an) [| H(a)
That is, under strong additive quantization Example E.2 and noise Assumption E.2 on full-precision data, Assumption 3.4 is verified.
F EXTENSION TO QUANTIZED MASTER WEIGHTS
In the quantized SGD algorithm (quantized SGD), the master weight maintains full precision. In this section, we demonstrate that our theoretical framework can naturally extend to the setting where the master weight is also quantized. For simplicity, only discuss the bounds for R The theoret- ical bounds are presented in Theorem [. Theorem F2 and Theorem F.3 for general quantization, additive quantization and multiplicative quantization, respectively. These results demonstrate that when the master weights are quantized, quantized SGD requires stricter conditions on the step size ensure convergence. Furthermore, the final excess risk bounds incorporate additional error terms, which degrades generalization performance.
We first present the algorithm and the propagation of EE [nm]
Specifically, we consider
1 Wr = (w.-)) + +7 7 (x(,,) (Di(x)) - 2x (. (2.((X))) t= I= N. (EI)
When master weight is quantized, Lemma A.I changes to
I B (xRa(x,)eexxx)) n: n=-1t1 Qa(X,) [s. (( (0) - ci _ Qa(X))e + +
(F2)
In particular, the coefficient for parameter quantization error d changes from
Qa X.) (x,),) to (7,,( (x().). Therefore, we can rewrite Lemma
C.2
and Lemma C.3 as follows.
Lemma F1 (Update rule under general quantization with quantized master weight). Under Assump- 3.2 Assumption 3.3 and Assumption 3.4
tion 3.1 Assumption
E nmn. _1 1 =((B, + C.).
where
C. < E (I (I-9 [ C.1 ( T
H() 2B
(p)
(===((->7_0x))) ei(x)) B.- (1-x ex(x))
B, B==B B
53

---

Published as a conference paper at ICLR 2026
with Co 0, Bu
E
Iooool and
T
(a)
a }
supt
E
Et
Jo
+ E
Ci
(g)²
G
B
+20B sup Ew
H
c-
(p)
(p) T
| 1)
o²
B
Proof. Noticing that
2x)a( 2x(,)) ( 6-14-1 (1- (u((i))
T
(F.3
32 E
2
@a(X,)
ex(X)) e Qx(X,) ex(x,) (x(x))
+ 2E
(p) (p) 12-1c-1
B
together with F2) and Lemma C.2 complete the proof.
Lemma F2 (Update rule under multiplicative quantization with quantized master weight). If there
exist ed: El; Eps En and e, such that for any {d,, 1, a, o). quantization Q, is e;-ultiplicaatiee
then under Assumption 3.1 Assumption 3.2
Assumption 3.3 and Assumption 3.4 it holds
n 3( + C.).
where
[1-1 1
- [)ex(( C--1 [ (1- (x(x)) 8cp (C,-1 + -B--1)
C, =E
7 B
Qa(X) T (B(X) + C. )a(x) [x(x)) + +>20 (g) H(O) + 4e,, (w (w())))²
+
[I- II-
1 (x) ( ex(x))
B. =E
Bi-1 I-
B
with
E= 8 + (l+ + Se, dea +
2 (1 40,
|| ||w'|l as (4c. () + ca) + + 2ea (1 cp) 4ep).
+
B
1t
Proof. Under multiplicative quantization,
E
(p)
e, [W-IW = 2E,1E [7-197]]] 2e, (w
(w(o)*)T.
(F4)
By and Lemma C.3 the proof is immediately completed.
F.1 GENERAL QUANTIZATION
this section, we derive upper bounds for R under general quantization. We first perform bias- variance decomposition under general quantization. By Lemma F.]. we have
nmm
2 + C.),
where
B, = (I (== Bo,
Bo = 1777 T
and
C = (I -T((()
91) o C,11 + 7
20
(g)
H
2
4-15--1
T
C. 0.
54

---

Published as a conference paper at ICLR 2026
Then by Lemma C.1
N-1N-1
N- N-1
RN 1
(I-
B,
(I HO H C,
N
N
bias
variance
(F.5)
Noticing that the bias error when master weight is quantized is the same as the bias error when master weight maintains full precision, we then only need to derive bounds for variance error. Similar to
C.22).
(g) H(w) C. 3I o C o C.-1 + 220
(p) (p) 7. 61-16(-1
2E
(F6)
Lemma F.3 (A bound for M (g)
C, with quantized master weight). For l 2 1, under Assumption if the stepsize y S antr(Haa))) then
Assumption 3.2 Assumption 3.3 and Assumption 3.4
aBtr(H(()) (7O
(g)
2du
M o C. 1
H
1-
Proof.
We first derive a crude bound for C, Denote E -= (9) = H(w) + Pr sup, E
( By
(p) (p) T)
C.24)
C, Coo =TT(1)= ...
Applying T(4), we have
7(0) Cx = =T(0) C.c + o Cx - -mM(1) C..
=7Σ + 7M o Cx - 7M Coo
3 + 7M o Cxo
= =70 ((9) sup E
(
2
(p) (p) T 1--16--1
+ + (g) C
Applying (7(4)--1 and by solving recursion, we have
C.o 3 106 2 (T()) 1 H( + ( o (o) (p)
6-116-1 ]
( oM! o C.o
3 (T( - M o (2o ( (7 o H(1) + 2 (7(1))- o sup E
2
(p) (p) T)
4-14-11
(F.7)
By C.27
(g)
1 o MnD) :M())) ogo00² (T())- ( H(
G
I.
(F8)
Noticing that
(T(0)-1 o E
(p) (p) 6
[I- 7((1) sup
E
(p) (p)
(=0
I 7 H(1) 4 sup
(p) (p)
(I - H(
1=0
supp
E
1-1--
H = p. -H(H)))
55

---

Published as a conference paper at ICLR 2026
have
(T(())) ) ( o B
T
11-16--1
(7(())) - M 21 (H()))-
(7(7(6) M (7(6)) M
24 (H
(>(7(0) -1 M (((6)) antr(I)
24 H()) (1)
(F99
(T(()) M Mg0 2uard-
32duaa
(aBtr(H()) I
2dp
())
()r(H(())
2dp
()r(H(()) - yagtr (H())
Therefore, together with
F.8 and F.9
we have
(g) 2
2da
70
C, 1 0 1
yagtr H())
follows that
(nlr(H(()) (70 (g) C.
+ [tr(H
M B
H(9)
(aBtt(((())
Together with (..)). can provide refined bound for C, and ready to bound the variance eтrоr.
Lemma F.4 (A bound for variance under general quantization with quantized master weight). Under
Assumption 3. Assumption
3.2
Assumption 3.3 and Assumption 3.4 if the stepsize satisfies 7 <
then
ogtr(H
(g) 0
+ 2andy
variance
< (agtr(Ho) (H)))
-
k" + N22.
+22
2p
Σ iSk
+ N,7 N77(1))
Σ Ex 1>*"
200
N
[p] (p) T
where μr = suu:;
- 6-1
56

---

Published as a conference paper at ICLR 2026
Proof. We first provide refined upper bound for C,. By (.6 and Lemma H33
C - c C Ct-1 + g
(g) 2
H
+ 2E
y PaBtr(H()) (70 (g)
2du
3(I- - 7 C+-1+
H(w) + 2-2 020 (1) 2
0 H(9) 2
1 1 1-yaatr(H")))
( Po(0)²
+ 2ya
-(I - - c C
H(1) + 2
(anl((H()))
200/0 (g)2 2yaBdu
1-1
L-1
[I )T0)
H + 2p
Σ
(I - 7 01
1-magtr(H())
k=0
"000""
+ 2ya
1-1
HI-HHO0)H'H (I~H 2/
Σ (I 7 7H )2)
(-gott(HH))
k=
(1)2 + 22 G
H (0) H ④)) + 2p Σ [I - H
1 1-nagtr(H()
k=0
G + 20
(I- (I ((-)
+ 24
I I (I - ())) (H(
By C.32
N-1N-1 N2
(I V1 <
0
(n) 2ex
-/v N
k* + N22.
EEa(o)3)
yastr H(()
1=
k=t
(F.10)
Regarding V2,
N- N-1 Σ
(1- V2)
N2
(=0 kmt
2p
N-1
Σ
I (I - H N- (I- (I - ((())) (H()))-2)
N2
N-1
2p
Σ
1 - (1 (1 (0) N -1 [1(0)0] 1
(F.11)
N2
1=
2p
100 N2 (1)
min
2
2p
)
+ Nr
N7y(11)
(g)
ISk
Together with (F.)).
F. and F..1
we have
(g) G
+ 2andw
variance 5
k-
N2²
+2
Σ iSk- Ny
E +
+N²
Σ
Ed
(eBBt((H))
N
(p) (p)
where μ ===1m) =us,
10-10-1
Theorem F.1 (A bound for R
under general quantization with quantized master weight). Under and Assumption 3.4 if the stepsize satisfies 7
Assumption 3.7
Assumption
3.2
Assumption 3.3
57

---

Published as a conference paper at ICLR 2026
oatr(Ho)) then
20 ||wD- 11 p(u) + NN - w(1)"
RN 12 s
10:0000
(N
k-
N,? Ea(0)s
Ny(1 - y0B
-78 ir (H()))
+ N2
|wo w(1)' 1
((H U.k
)-1 + l|wo 1r(0)112
H
(9) pad
+ yagtr (H(w))
G
k"
+ Nop.
Σ En((0)e))
+
2/4
+ +Ny Σ (00)
N
Ny1
i>
1 }. where k- = maxx k: 11:1 2 No f. 1== sup;
(m)
(m) 4-1
sup;
E
and
c (1))
(a) c,))
sup;
E
10
+
E
E
aa
2
B
1)
o²
+ 2ag sup Ew
H
E
(p)
(p)
We-1
B
Proof.
The proof is completed by (F.)) Lemma C.10 and Lemma
F.4
Next, we deduce an upper bound for R under additive quantization from Theorem F.1
F.2 ADDITIVE QUANTIZATION
Theorem F.2 (A bound for R0 under additive quantization with quantized master weight), Under
Assumption 3.7 Assumption 3.2 Assumption 3.3 and Assumption 3.4 if there exist cdi Qt Cp. co
and 6o such that for any E {d, L.P. o}, quantization Q; is E--additive, if the stepsize satisfies
7 ag (H)(u(H)))e)) then
20B
||wo w(0)"
112 p!u
+ +N/l| wo - w(s) 112
FP( 10:000
K- N N2²
RN /2
Σ (A: taa?)
N7 - yonn|Lr(H + dea))
1>
1
|wo w(s)" [| lik ) l/ w(0)) 1120011
N2N2
k0:000
0² te, 260 (4 tr(H) + dea)
(5+- k* +N72
E(i++ea)
1 1- (- + deal
20p
Σ NT(A++ea)) + NY Σ (di E(ii+
iSk
i>
Proof. The proof is completed by Theorem
F.3 MULTIPLICATIVE QUANTIZATION
In this section, we derive upper bounds for R under multiplicative quantization, i.e., there exist ea, Ed, 6a and 6, such that for any E {d, L,P, o). quantization Q, is m-multipliaative. We first perform bias-variance decomposition under multiplicative quantization. Denote
B.M (M) == ((1 + 8c,) - TT + +
M
(g) (M)
((M) =E [nond]]
cM ((1 (11+8,))Z-TT +
(M) +2 (g) ( 4e, ((w(())) cgM)) 0.
58

---

Published as a conference paper at ICLR 2026
By Lemma F2
we have
E mmn | 2
(B(BB(M) (B(m)) + c
Then by Lemma C.I
N-1
RN /2
(I O (M)
+ N2
H()) (1) H c
N
bias
variance
(F.12)
F.3.1 ANALYSIS VARIANCE ERROR
Similar to (.23)),
c(M) =(11 SE,)I 7Th²) + M ) +700 (2) H + dep (1)" (wol0)²s (w
3(11+ 8cp)I - (()) C.M + +1+ c
(g)2 + M
H( + de,, (0)' (w (w(1)))
w())
Recall that 7()) H I H HH H Denote T
H & I+ x
H(w) 7)2HH( & H Before proceeding, find conditions for step size such that SepI 1
TT(1) (7( It suffices to restrict:
86p 5 31
3
222
(F.13)
4
Equipped with (F.13)
((M) <I 7 o
C(M + +1 + +
clM + +2
(g) H(4) + (ww))))
(wlx)))
We would like remark that, in the analysis of variance, simplify w(0)' (wlz)*)T, we assume the parameter prior
E
I,
and take expectation over w on variance error. follows that 7
c (dd)
1 (I
77())
c! + (1 + M) o (. + +200 (g)
H()) + 4egI.
(F.14)
In subsequent analysis, first derive crude bound for c(M) then establish a refined bound for ((M) via (F.14).
Lemma F.5 (A bound for M cC (MM with quantized master weight). Fort t 2 1, under Assumption
(M)
3.1
Assumption 3.2 Assumption 3.3 and Assumption 3.4 if the slep size satisfies
>> 2(1 + ( ()),
8ep S 72
3
2/0) ())
then
( 2yo
M c.
(M)
1 1-2(1 2(1 + () (gtr(H H
M
[(t))
Proof. By F114
MM x C(M) 39
(M)
(7(())
71 27(1--11+00)) 711+7M
(1)
(
Ep
2
Acttallyy (F.14) holds under the expectation of w". Slightly abusing notations, we omit Teyy
59

---

Published as a conference paper at ICLR 2026
Applying )2 T o C( - (72") - 2((1 + +)++M cla
+ +1 + +.m o cls
3 + 2(1 + 8) C
Applying (T - we have
C(M < M
(4)
(7 H (1) + 86 (T( + 2(1 (1 o MM
c
Σ ((1 + 2) (1) - M (T ) (2
(4)2 H(a)
+ 8E2T
t=0
Noticing that (T) H (v) and (T (T(1)) o I (H(1))-1,
-1 we have
C
+ ((1+))(()))000000))) M (T 0
(1) 2 M
H BEPT)
(F.15)
+ [2(1+
[2(1
((1+))((T())))
(1)) -0 "
M
(270
( M
8cp (H)) I+
Firstly,
2(1 - M M 01 ())
(F.16)
Secondly,
2(1 + +0+ M
B (H =2(1 + ()))
(F.17)
Therefore, together with (.155..
F.16)
and E.17
we have
CMM
211 + ((1++)((T2))
(10)
M
oI. [411 2 (g) agtr(H()) + 16(1 + B)
1=
27 (1) ( M
8dep
2( (1 o M 4-1
cI. .I + ()
)tr(H(#))
127 2
8dep
2(1 ()())
>l)(H())
240
(q) M
+ HH)
8dc
I,
2(1 + Br ((()))
where the second inequality uses (F.16). At last,
(q) 8dtp 2 M
M c.
(M)
- 2(1 + ) 1 (Btr (())
(t
Lemma F66 (A bound for variance under multiplicative quantization with quantized master weight). Assumption 3.3 and Assumption 3.4 if the step size satisfies
Under Assumption 3.7 Assumption 3.2
3 202
>> 2(1 + ())
8ep S 72
then
40 ( + 32(11 + add h.* N22
Ew variance <
1 1- 2(1 + E)aatt((H())
166g +
Σ
+
No
Σ ((0)
Nyy
(2)
isk.
60

---

Published as a conference paper at ICLR 2026
Proof. We first provide refined upper bound for C,. By (F144 and Lemma F5
3(I ((M)
2 T 0
(M + (1 +
2) c. + 7 o M H (0) + T
(4)
3(I ] (
1-1 + + M H + 4cp1
(M)
(1)
2
2g M
(4)² 2
+
8de
+12 + 8)
1 2(1 ()pptt(HH)) @gtri H(s) H())
/-0'0/ (q)
-(I - - c/1
81 aB H(1) + 4ep1
1- + ()aatr(HH
P0M + +8 + 2)dC
(1)2 2
t-1 I 7
t-1
H + 4cp Σ
[I
T
01
1 2(1 + + (H)))
2
2
k=0
(1)² 7 MM + +8 + d)
t-1
t-1
2 H
01)
H (1)
+ 4cp
(I
1)
2(1 + B H)))
2
k=
k=
? 2 270 M
+ 16( + d)depa
2 H)))))
8ep
(I
2
H
)
1 2(1 + yt (H)))
2
?
(F.18)
Recall (F.12).
N-1N-1
E variance S N2
Σ Σ
V1 + V22
k=t
Regarding V1,
N-1
1
(I- (O)) V1)
N
N N-1
< N2
[I
(4)) k- (((0) Vz)
2
1= kutt
2ga (g)²
+ 16(1 + d)dG N-1
2
I (I 7 (( N-t I (I 7 ((())())
22y 1 - 2(1 + ()
2 270 (q)2
M + 16(1 + a)
(1 (1 2 21
(0) N (1 - (1 - ? (000)
N2, 1- 2(1 (r
2
(F.19)
270 (
+ 16(1 + d)
N-1
1 - (1 -1-3 3
< 22 1- 2(1 + ()
(g)² < 2 2y0 M + 16(1 E)depaB Σ min
{1, 1, [
?2N2 (0)
Ny 2(1 + ())))(((H())
(g) 2 M
2 240
+ 16(1 + d)
min
Nv 1 1- 2(1 + ()ya)t((H())
4a ( 2
+ +32(1 + ddee,on/7
N22.
Σ
1 - 2(1(1 + (H(1))
N
61

---

Published as a conference paper at ICLR 2026
Regarding V2,
N N-
2))
H(9) H( H2))
N2
t=0
k=t
N-1
2
I --
(I
H H
N2y
t=
N-1
Stp
I
2
H
(I 2HO))
7 N27
(F.20)
8cp
N-1
Σ
1 -
(1
(00) N -
| - (1 - [
N27
2
1=0
166p
(1) N332\(00
$ N2²
min
16Ep
Σ 1001
+ N2
No)(1)
iSk
i>k
Together with 1212
F.18.
F.19
and
F.20
40 (g) 2
Ew variance S
1 1- 2(1 + ()))))((HH())
M + 32 + Addaaa//
(N
[K- + N22. E(((0)))
k.
+ 166g
Σ
Ny
(1)
+
N2
Σ
(.00
12k
ish
1>k"
F.3.2 ANALYSIS OF BIAS ERROR
Recall that
B (M) : ((1 + 8T,)I - - + + (OB() (M)
(( = -Bo = E [oo
From C. we deduce that
N- 1
B (M) ()
bias S NN2
I (I
(F.21)
1=
Denote (M)
21 1 (M)
B(M) Motivated by Lemma C.7
S( =((1 + 8c,) I - -TT(0 +
+ (0) (M) + B,
B
=((1 + 8t,) - T799) ( (T(1) - ) (M) + 2 (g) . + Bo
-(11 8c,) 77(1) S + (11+) +0) M( (1) S (M)
(M)
+Bo
(M) ((1 + 8c,) - - o S + (1 + + S
(M) B
Similar to the analysis of variance error, we consider
3 2100
7 < 3x 8ep 5
7
It follows that Se,I 1 (7(())
- - (1) and hence
s((0) 1 (I
2 7(1)) (1)) + +1+
s + +B.
(F.22)
2
62

---

Published as a conference paper at ICLR 2026
Lemma E.7 (A bound for M Si (M) with quantized master weight). For 1 S l S N, under
Assumption 3.7 Assumption 3.2
Assumption 3.3 and Assumption 3.4 if the step size satisfies
3 222200²
1> 2(1 hagtt H ) 8cp S 72
(4)
4
then
2on tr ([I - (I (I- T7 y
BB
M
o s(ar) B
H
111 221 + E) tr tr(H()))
Proof. From
F:22
s(ar)
Σ
t-1
I
7(0)) 2 2
+(1+ ) k Bo
k=0
1 7())
(1 + 11+7\M50)
I I 7
(1 M (0)
o Bu
γ 2
A
Applying 7, we have
(7(0) 7at) 2 - (1+ -1+7)),M AA
22 2(1 + + (7/-00 -(1 -1+ (1+ -1
o..
Applying (T( -1 we have
(7(0) (1 (1+ AA
2 _
-2(T o A + 2(T -1 (1+7) (17,0) (1 + 1+777 -1
oA.
Applying M we have
M (17 - + 1) A
-2 B
(T (1) o (1+ + 7)
1 (770)
- -(1+3hM
o oA + 2M o (T A.
2
follows that
(77000 - (1 + + -1
M(g
2 77
A
(F:23)
Σ
[ 221+ (7. 2M ( - A.
1=U
By Assumption 3.3
M B 0 (T ())
A 1 aa ( -1 ()
7 ir =aB
Σ H [I ( A(I H ((9)
2
=gtt (H(*(2H())
- ( -A) (())
3 (
63

---

Published as a conference paper at ICLR 2026
together with F223
(T2 (T -1l H(9)
< and M I an aa (H(H))))H))) we have
2 T (1 (1+7),88 oA
M (0)
2(1 + M o ) 2agtr(A)H
1=0
2(1+3)m ( H
=2attr(A)
<2antr(A)
[2( + ((1++))al(((HH))) H(
AA)tr((A
1 1 2(1 + ((1+)))))((HH)) H
Therefore, recall that
A I -
I
17(() 2
+ +1++))²'M
Bp I - ------21 1 )
o Bo.
we have
20B tr (I- Bo
M
(1))
-1
2 tr(
H(()
H())
1 2(1 + y tr (H()))
(1-2(11) + E) tr (H))))
Together with F.22) and Lemma F+7 master weight.
we are now ready to bound the bias error with quantized
Lemma F88 (A bound for bias under multiplicative quantization with quantized master weight).
Under Assumption 3. Assumption 3.2
Assumption 3.3 and Assumption 3.4 if the stepsize satisfies
1> 2(1 + H
8cp S 72
202
4
then
[/wo-w(o) 11 + l/wo w(o) 1 112
8(1 + E)aB
N7
(R= + +Na² Σ Ex(e2
2(00 1
bias
1- 2[2 + 2)) ((()))
N
4
||
w
(...) )- + 4//woo -w 1/2
N2
Proof. By 1:22 and Lemma
S ((M)
-1-
1[I - 2 2 o (M)
+ 21+2) (g) ((A)) + +B
20g tr ([I - (I- (T(0)) y>|
Bb)
3(I - (1)
s.S.M.
+ (17
.H + Bo
2
71 2(1 +2)00 tr (H(1))
2ag tr [I [I- -I- - ) N.
1-1
BB))
Σ (I 2
(1 i)7
(H()) + Bo
7(1 - 2 + 8) tr (H()))
k=
2aB 1r ([I- (I 37. N
(-1
BB
-
7 H 00)+
(1 (1+
H (4) + Bo
(I H ())y
2
7(1 2(1 E) tr (H))))
2
0=
64

---

Published as a conference paper at ICLR 2026
Before providing our upper bound for the bias error, we denote
7 (Hll))-BB... (I 2 HH
Ba,b == B (I
2
Recall from F.21) that
N 1
bias S
1
I-
-I
7
Σ
B B(M)
7222
t=0
we have
N-1 Σ
bias
<
2
I- (I
B (M)
7222
2
1=0
2
I- (I
2
H
N-1
Σ
[I
-
H ( 2(1 + +))an Ir (Bo.v)
H(() + Bo (I- ()))
72V2
2
2
1 21-2(1- + 2 tr ())
2
k=0
2
N-1
(I
7 H
(1) 2*
(I
2
H N+2k 2 tr (Bo.N)
H(O) + Bo
N2
2
1 2(1 + E)yaB ())
k=
Note that
(I
H H())2
(I
7 H())) 2- 2k
1 (I
H - (I 7 ) 2
2
2
2
have
bias S
2
N-1
[I
H 1 (I-2 HOOY N+R
+ E)an Lr (Bo,n)
H(() Bo
7N2
2
'1 2(1 + 8) tr (H))
Therefore it suffices to bound the following two terms:
11
2
-1
H(
(1
7 H (1) N+* 2(1 (11+t)yap tr (Bo.n)
1 2(1 + + tr((()) H(())
NN2
2
N-1
2
-H & (I
_
7 H .B)))
12
N2
2
k=0
Regarding J,
N-
())) - -I- 7 H ( N+E H()) (0)
2
2
k=
N-1
Σ Σ (1 - 2
(00) -(1 -11-3 2 (0)))) 2.(0)
2
Σ [1- -1 - [(1
2 2000 - -(1-2 [0)
2
2
2
Σ
Σ min {1. 100
< ks N222 Σ 1/002
2
Hence,
4 2(1 + 7) tr (Bo.n)
N²
Σ 2(002) k
h S
2(1 + Than (r()))))
65

---

Published as a conference paper at ICLR 2026
Then we tackle tr (r(...))
Lr(Bo.n) =1r ir (Bo (Bo-(1
-01--1 -(I -II 2 H1 Bo(1 - - (
Σ (1 1 (1- 7 ((o))ee)) (lww
wlod v/(0))
(F:24)
Σ min NrA (two- -w(")" .v.0)) 2
1 wo w(()" 1 + Ny -
-w w( Iire....
Therefore,
(Iwb (woo-w(07 w(0) 1..00 + Ny/lwo - w-() (100 ((p)..)))
4
2(1 + 2)yB
(K*+ N2+22
Σ 1902). 8sk
N2
2(1 + E) 1r (H()))))
Regarding /2.
N-1
2 H() - -I- 2 H N+ Bo)
1 =
N
2
k=0
N-1
2
2 1000 -(1 -1- 2 (o0 N+1 w/
NN2
2
k=0
4
(1 - 2 No) w/
N²
1100
4
w
min 11, (1(0)))
N2
[g)
4
||wo- w 2
(H 1 + 4/ 4o
w(0)
N2
Overall,
bias S 11 + 12
(10 Bi
8(1 + 2)BB
+ l|wo- w(1)" w(0)11 1iter
Ny
(
kit
Nº²
Σ
1-2(1 + E)yaB tr((()))
+ ?²NN2" -|wo
4
+ 4 4
w(() 12 (2
Based on the analysis for bias and variance error, we are ready to present the bounds for R(O) under multiplicative quantization with quantized master weight.
Theorem F.3 (A bound for R under multiplicative quantization with quantized master weight), Suppose there exist Ed, E,, Epr En and e, such that for any i E {d, I, p, a, o}, quantization Q, is e Assumption and Assumption 3.4 if the
multiplicative. Under Assumption 3.1 Assumption 3.2. step size satisfies
8cp I 77 2 2(00
1 > 2(1 + "))
4
66

---

Published as a conference paper at ICLR 2026
then taking expectation over w' variance
R 12 40 (9) 32(1 + 2))cpa/
- 2(1 + ())attr((HH))
1
p +
+Ny Σ 1.00
7
isk No
Wwo (wo-w(w)'
"
8(1 E)aB
l|wo w(()) He
Ny
ht
+ Ny2 Σ 1502))
1- 2(1 E)yagttrHHo) tr(H(())
N
13
4
||wo - w(0'|1 [[H160) )- + +4/l w(1) H(00 100
+ N2
where
ē= + (l
+ 8ep + tea (1 Ep),
(n) M
(1 + 4eoo0
Ew [wwll (4e, + tp (a) +1 + 2ea (1 + ep) + 4ep)
+
B
1 l
Proof. The proof is completed by (F.12) Lemma F.6 and Lemma F88
G DETAILS OF ADDITIONAL EXPERIMENTS
G.1 ADDITIONAL DATASETS
For the supplementary experiments, we consider both synthetic and real-world datasets.
Synthetic dataset. We construct a synthetic regression dataset whose covariance spectrum follows an exponential decay. Specifically, the eigenvalues are given by
x =
i = 1,2....... dd
This allows us to examine the behavior our method under rapidly decaying spectral structures, complementing the polynomial-decay setting used in the main paper.
Real-world dataset: Communities and Crime. We additionally evaluate on the publicly available Communt ties and Crime dataset, which contains community-level statisties from across the United States. The features integrate socio-economic indicators from the 1990 U.S. Cen- sus, law-enforcement statistics from the 1990 LEMAS survey, and crime records from the 1995 FBI Uniform Crime Reporting (UCR) program. The task is standard regression problem: predicting the per-capita violent crime rate from community attributes. The dataset contains about 2000 instances with 122 features.
G.2 EXPERIMENTAL SETTINGS AND RESULTS
We describe below the protocol for each additional experiment and corresponding results. In both real-world dataset and synthetic datasets, examine how do additive vs. multiplicative quantiza- tion affect learning (generalization) performance.
Real-world regression (Communities and Crime). We apply both additive and mudtiplicative quantization schemes (with fixed quantization error level e 0.01) to the regression task on Communities and Crime dataset. For each quantization method, we evaluate the resulting
population risk Ex.y (y- (w, {)]]. As illustrated in Figure 2(a) the results demonstrate that, unlike additive quantization, the multiplicative scheme successtully maintains the performance of full-precision SGD. This aligns with our theoretical finding that multiplicative quantization exhibits greater tolerance to quantization error level.
HHre assume that Fww" "w"" T.
67

---

Published as a conference paper at ICLR 2026
2.00
Additive Quantization
Excessive Risk
1.75
multiplicative additive
Eigenvalue
Multiplicative Quantization
Original Spectrum
1.50
0.5
1.25
1.00
60
40
Sample Size(x50)
Eigenvalue Index (sorted)
(a) Real-world regression
(b) Effects on spectrum
Figure 2: Comparison between multiplicative quantization and additive quantization.
(a):
Real-world regression (Communities and Crime). (b): Effect of quantization on data spectrum.
Exseave Mac
Beccoire Rax
1e3
1e-3
10-4
2.--
162
1.2
1.0
1.0
Skule50)
Slaz)63)
Sixo85))
(a) Multiplicative (FP-like) Additive (INT-like) (c) Multiplicative (FP-like) (d) Additive (INT-like)
Figure 3: Generalization under quantization. Test risk for SGD with iterate averaging under multiplicative (FP-like) vs. additive (INT--like) quantization. (a) and (b): vary the quantization level at fixed B 10. (c) and (d): vary the quantization level under exponential decay.
Effect of quantization on data spectrum. Using the same settings on Communities and Crime dataset, we record the resulting empirical covariance spectra study how each quanti- zation type perturbs the underlying eigenvalue structure. Results are shown in Figure 2(b) shown that additive eπrors errors dramatically distort the spectrum of effective data covariance while multiplicative quantization errors largely preserve the spectral structure. This visualization corroborates the specific mechanism by which additive and multiplicative quantization lead to distinct generalization behaviors.
Sensitivity analysis on batch size and spectral decay. To demonstrate the robustness of our findings, we conduct additional experiments varying the batch size and data spectrum. First, we extend the batch size to
10 (with = 200) and vary the quantization error level e E (0.001, 0.005, 0.01}. Second, replace the polynomial-decay spectrum with exponential- decay synthetic dataset while keeping other settings identical. The results, shown in Figures [[a] 3(b) (batch size) and Figures 3(c) 3(d) (spectral decay), consistently mirror the findings in the main paper: multiplicative quantization preserves the generalization performance of full-precision SGD across various quantization error levels, whereas additive quantization suffers from perfor- mance degradation as the error level increases.
H THE USE OF LLMS
The use of large language models (LLMs) in this work was limited to linguistic polishing of the text (e.g., grammar, clarity, and readability) and was not involved in any research phases, from conceptualization and proofing to experimentation and interpretation.
68


Reviews:
[{"review_id": "pttqx5Jnmg", "reviewer": "Reviewer_fw7v", "summary": "This paper develops a theoretical framework for understanding how low-precision quantization affects the learning performance of stochastic algorithm descent (SGD) for the problem of high-dimensional linear regression. This paper specifically focuses on the impact of quantization on the generalization (population) risk (not just the training loss/sample risk). Quantization is applied to five components of training: Data (both features and labels), model parameters, activations and gradients. Subsequently, the iterate-averaged SGD is analyzed using two quantization schemes:\n\n1. Additive quantization, where the absolute quantization error is constant -- capturing the effect of fixed-point precision formats (e.g., INT8).\n\n2. Multiplicative quantization, where the relative quantization error is constant (in other words, quantization error scales with the magnitude of the quantization input) -- capturing the effect of floating-point precision formats (e.g., FP8).\n\nThe key contributions of the paper include:\n\n1. Derivation of explicit bounds on the generalization risk showing contributions from variance and bias (which are prevalent even for full-precision SGD, i.e., no quantization error), approximation error from the quantization of data/labels, and additional quantization error, arising from accumulation of quantization error from the SGD iterates.\n\n2. Study of how the distortion is input data spectrum due to quantizing the features/labels affects the generalization risk. More specifically, the authors propose that multiplication quantization does not lead to significant spectral distortion and matches the sample complexity of full-precision SGD under mild conditions. On the other hand, for additive quantization, spectral distortion increases with dimension, and leads to worse sample complexity. \n\n3. The work also considers a polynomial eigenvalue decay of the data spectrum, and proposes when additive vs. multiplicative quantization is preferred on the other.", "questions": "I have a few questions/suggestions are would appreciate it if they are addressed:\n\n1. Why is $Q_o$ referred to as the quantizer for *output gradient*? Shouldn't the input to $Q_o$ also include the term $Q_d(\\mathbf{X}_t)^\\top$, and not just the second factor?\n\n2. Recent techniques for quantized SGD like error feedback (https://arxiv.org/abs/1909.05350) are not considered in the analysis. This somewhat limits the applicability of the results, since error feedback has been around for a few years now and is pretty widely used for optimization with compressed gradients. This should be explicitly highlighted in the limitations section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The author positions this work in the context of LLMs and low-precision training, precision-scaling laws, and benign overfitting and overparameterization theory -- which is pretty relevant in today's ML landscape where bigger models and more data are preferred for better performance. The paper tackles a pretty fundamental question: How does quantization affect the generation performance of SGD for linear regression, in contrast to prior works that focus on convergence of optimization algorithms under quantization. The systematic study of quantizing data, label, parameter, activation and gradient is a major contribution as well -- prior works usually isolate one or two of these. Modeling quantization schemes as multiplicative or additive also provides an useful abstraction between this work's theoretical analysis and the data formats generally used in practice (e.g., FP8 and INT8).", "weaknesses": "Firstly, I have a semi-major concern: The paper implicitly assumes quantization is purely detrimental and focuses on bounded degradation relative to full precision. But quantization introduces stochasticity that also plays a role analogous to implicit regularization similar to SGD noise, weight-decay, etc. (Ref: https://arxiv.org/abs/2101.12176). There are also some empirical works in deep-learning where low-precision improves generalization slightly (e.g., https://arxiv.org/abs/2206.12372). The theory for linear regression is exactly the setting where we would want to understand whether/when this can occur. In my opinion, this is a key conceptual gap, which is not apparent from the main body of the paper (pl. correct me if I have missed anything). It would be highly appreciated if the authors could comment on the following:\n\n1. Discuss whether their analysis explicitly take into account this implicit regularization? If yes, where exactly? If not, is it a limitation of the current analysis, or is studying the implicit regularization not relevant?\n2. If the authors agree with the potential benign regularization effects of quantization, under what conditions might they emerge?\n\nSecondly, I think the writing of the paper can be significantly improved. Currently, the theorems statements in the main paper have pretty long expressions. If it is possible to state simplified/informal versions of the theorem and/or corollaries wherever relevant to extract the essence of the theorem, I think it would make the paper highly approachable.\n\nReal world linear regression experiments would also be appreciated, where the assumptions on the eigen-decay of the data matrix are justified. I believe this is important because the section on **Implications to integer and FP quantization** is quite relevant to practitioners in determining the choice of precision formats to be used for quantization. However, in practice, INT8 and FP8 perform for or less similar to each other (perhaps because for most recent models, dimension is high?) Sometimes, INT8 is even preferred because INT8 matmuls are faster than FP8 matmuls.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bc9bNbFiMX", "reviewer": "Reviewer_WYgy", "summary": "The paper provides theoretical foundation for understanding how quantization impacts generalization in SGD. The authors bridge the gap between optimization convergence analyses and generalization theory under hardware-limited precision. The paper specifies precise conditions on quantization errors needed for quantized SGD to match the learning performance of full-precision SGD. The authors also show an analytical framework that quantifies how quantization on various components.", "questions": "1. How do non-uniform or mixed precision quantization methods fit into the framework here?\n2. Can it be generalized to multiple epochs and adaptive optimizers, would the excess risk decomposition change qualitatively?\n3. Since the case study is on polynomials decaying, I am curious how would the results change for exponentially decaying or heavy tail spectra which also happen in real world embeddings.\n4. In Fig 1, how sensitive are the results to batch size or step size, can authors also show some results other than B=1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a unified framework that decomposes the excess risk of quantized SGD into interpretable components.\n2. The paper is mathematically solid and demonstrates excess risk bounds explicitly with clear decomposition and scaling behavior under both additive and multiplicative quantization.\n3. The authors provide interesting insights into precision and generalization trade off, and also provide theoretical link between quantization type, like FP vs. INT, can be beneficial for scaling law and safe precision reduction.", "weaknesses": "1. The experimental section is only on synthetic Gaussian data, it could be more convincing if validated on real world dataset.\n2. The analysis relies on idealized assumptions, such as unbiased stochastic quantization, but these may not hold in practical low-precision systems. Discussion or relaxation of these assumptions would strengthen the generality.\n3. While motivated by scaling-law literature, the link between derived quantization effects and empirical scaling behaviors remains largely qualitative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdainWsQ94", "reviewer": "Reviewer_D5UN", "summary": "The paper provides a theoretical analysis of Linear Regression with diverse quantization targets: data, labels, parameters, activations, and gradients. The paper analyzes two types of unbiased compression methods: multiplicative and additive quantization, and derives convergence guarantees for SGD under these schemes. The paper also includes a small empirical evaluation on synthetic data.", "questions": "* It is not clear why quantizing data would outperform quantizing parameters or gradients. Could the authors provide a concrete example or theoretical justification where data quantization provably yields better results than parameter/gradient quantization?\n* Regarding the quantization of the model weights in the (Quantized SGD) update step. Wouldn’t it make more sense to quantize both $w_{t-1}$? If only one of them is quantized, it seems we still need to maintain a full-precision copy of the parameters, which weakens the memory motivation.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "* The paper brings a theoretical analysis of different quantization targets.\n* The definition, assumptions, and notations are easy to follow.", "weaknesses": "* The statement of Theorem 4.1 is difficult to analyze and to derive guidance on what to quantize to maximize performance within a limited resource budget; the paper would benefit from organizing and grouping the results and from plugging in an optimal learning rate.\n* The bounds in Theorem 4.1 do not improve with the number of samples $N$, which is atypical for SGD analyses (even in more general convex or non-convex settings). This issue remains even without compression (i.e., when $\\varepsilon=0)$.\n* The paper uses a simple setup of unbiased compressions (where biased compressions are more practical and common) over a simple linear regression objective for empirical risk minimization (i.e., not guaranteeing generalization).\n* Empirical results are limited to a simple synthetic dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viRjMPwbkF", "reviewer": "Reviewer_MJwg", "summary": "The paper presents a systematic theoretical analysis of the generalization performance of quantized Stochastic Gradient Descent (SGD) in the context of high-dimensional linear regression. It studies the effects of quantizing five key components: data, labels, parameters, activations, and output gradients. The main result is a precise, decomposed bound on the excess risk, which clearly separates the approximation error, bias error, variance error, and the unique quantization error. The authors contrast two major quantization error models: additive (related to INT) and multiplicative (related to FP). They use their bounds to provide conditions under which quantized SGD achieves performance comparable to its full-precision counterpart.", "questions": "1.\tCan the author comment on what they think would be the effect of the quantization error in nonlinear models and other algorithm designs, such as multipass SGD and momentum? Do they think that the implication of the result regarding multiplicative quantization will also carry in these settings? \n2.\tWhat does the notation $≲$ mean? \n3.\tLines 355-361: Please indicate where the fourth term appears. It would be helpful to provide a proof sketch that explains all the pieces in the proof. I understand the analysis is similar to Zou et al (2023). However, it would be nice to have a summary here.  \n4.\tLine 336: Is it clear that these errors remain O(1)? Please provide more intuition on why, rather than just directing to that appendix, as this is central to the analysis. \n5.\tLines 393-397: I am not sure I understand this. The errors are not completely decoupled, as $\\epsilon_a$, $\\epsilon_o$, depend on $\\epsilon_d$, $\\epsilon_p$. \n6.\tLine 463: It would be nice to provide a clearer explanation and maybe plot the Hessian spectrum as well in both cases, to illustrate the effect.  \nMinor: \n1.\tLine 293 remove either “plays” or “relies”\n2.\tLine 394 \\epsilon_0 should be \\epsilon _o.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper tackles an important and novel theoretical problem with significant practical implications. The systematic analysis of five quantization targets is comprehensive. The derived generalization bounds for quantized SGD are the first of their kind to my knowledge. The comparison between additive and multiplicative quantization, particularly the insight into spectral distortion, is a major contribution. The paper is well-written, and the theoretical results are intuitively explained and justified.", "weaknesses": "1. The analysis is restricted to high-dimensional linear regression, leaving the extension to non-linear models (like neural networks) as a critical open question. \n2. More significantly, the paper provides only an upper bound on the excess risk. As noted, this bound, especially for additive quantization (scaling with d and N), appears non-tight and is not compared directly against the numerical simulations, undermining the conclusiveness of the theoretical comparison between additive and multiplicative models. Proving a matching lower bound or providing a tighter analysis for the additive case is necessary to solidify the theoretical claims.\n\nOverall, the paper makes a foundational contribution to the theoretical understanding of quantization's impact on learning performance. The novel bounds and the systematic approach to different quantization targets (data, parameters, etc.) provide invaluable, actionable theoretical guidance for the design of efficient learning systems. While the lack of a tighter lower bound and the restriction to the linear model are limitations, the complexity and significance of the problem tackled, combined with the depth of the current analysis, make this work a major theoretical advance worthy of publication. The authors should be encouraged to pursue a tighter analysis of the additive and multiplicative quantization errors in the revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)