Robust Training of Neural Networks at Arbitrary Precision and Sparsity
Authors: Chengxi Ye
Grace Chu
Yanfeng Liu
Yichi Zhang
Lukasz Lew
Li Zhang
Mark Sandler
Andrew G. Howard
Venue: ICLR 2026 Poster (2026)
Keywords: Quantization
Sparsity
Primary area: optimization
TL;DR: The paper introduces a principled dequantization transform that creates an error-aware gradient path, solving the instability of the straight-through estimator to enable stable training of ultra-low-bit and sparse networks.

Abstract: The discontinuous operations inherent in quantization and sparsification introduce a long-standing obstacle to backpropagation, particularly in ultra-low precision and sparse regimes. While the community has long viewed quantization as unfriendly to gradient descent due to its lack of smoothness, we pinpoint—for the first time—that the key issue is the absence of a proper gradient path that allows training to learn robustness to quantization noise. The standard Straight-Through Estimator (STE) exacerbates this with its well-understood mismatch: a quantization-aware forward pass but oblivious backward pass, leading to unmanaged error and instability. We solve this by explicitly modeling quantization as additive noise, making the full forward-backward path well-defined without heuristic gradient estimation. As one natural solution, we introduce a denoising dequantization transform derived from a principled ridge regression objective, creating an explicit, corrective gradient path that makes learning robust to the noise STE bypasses. We extend this to sparsification by treating it as a special form of quantization that zeros out small values. Our unified framework trains models at arbitrary precisions and sparsity levels with off-the-shelf recipes, enabling stable A1W1 and sub-1-bit networks where others falter. It yields state-of-the-art results, mapping efficiency frontiers for modern LLMs and providing a theoretically grounded path to hyper-efficient neural networks.

Published as a conference paper at ICLR 2026
ROBUST TRAINING OF NEURAL NETWORKS AT ARBI- TRARY PRECISION AND SPARSITY
Chengxi Ye
Grace Chu
Yanfeng Liu Mark Sandler
Yichi Zhang
Lukasz Lew
Li Zhang
Andrew Howard
Google DeepMind
(yex, exy, yanfenglu,, yichizh, 1ew, zh1, sandler, howarda
ABSTRACT
The discontinuous operations inherent in quantization and sparsification introduce a long-standing obstacle to backpropagation, partieularly ultra-low precision and sparse regimes. While the community has long viewed quantization as un- friendly to gradient descent due to its lack of smoothness, we pinpoint- for the first
the key issue is the absence of a proper gradient path that al- lows training to learn robustness to quantization noise. The standard Straight- Through Estimator (STE) exacerbates this with its well-understood mismatch: quantization-ave aware forward pass but oblivious backward pass, leading to unman- aged error and instability. We solve this by explicitly modeling quantization ad- path well-defined without heuristic
ditive noise, making the full horward-backwer gradient estimation. As one natural solution, we introduce denoising dequanti- zation transform derived from a principled ridge regression objective, creating an explicit, corrective gradient path that makes learning robust to the noise STE by- passes. We extend this to sparsification by treating it as special form of quantiza- tion that zeros out small values. Our unified framework trains models at arbitrary precisions and sparsity levels with off-the-shelf recipes, enabling stable AIWI and sub-l-bit networks where others falter. It yields state-of- of-tthf--aattt art results, map- ping efficiency frontiers for modern LLMs and providing theoretically grounded path to hyper-efficient neural networks.
INTRODUCTION
To deploy complex Al models on resource- consurcee-onttrieeeed devices, techniques such as quantization and sparsification are essential. However, their non-differentiable nature poses long-standing challenge for gradient-basee based training. For years, the community has relied on the Straight-Throug Estimator
(STE) (Bengio et al. 2013 Hubara al., 2018). surrogate gradient that has been instrumental in enabling Quantization
Training (QAT). While STE has enabled progress, its use often
leads unpredictable and unstable convergence, particularly in demanding low-precision settings. This training instability is a well-documented obstacle; the estimator's fragility is often concealed in large, wver-parameterized models (Wang et al.) (223) but becomes apparent on smaller models that are more sensitive to quantization еrтor, where STE-based methods frequently diverge. Such insta- bility has necessitated numerous heuristic modifications, including additional normalization (Zhang et al., 2022). learning rate adjustments Wang et al. 2023 optimizer changes (Liu et al. 202 1 and fine-tuning (McKinstry et al., 2018). These ad- solutions underscore the urgent need for more principled, robust training framework. Our work addresses this long- standing challenge with theoretically grounded solution that does not rely on surrogate gradient estimation, instead deriving well-defined, differentiable gradients from ridge regression objective.
Rather than layering on more empirical fixes, we propose a new QAT framework that addresses the source of this instability. The core mechanism of UTE--apprxximating the derivative of the round- ing function as an identity creates a well-understood mismatch: the forward pass affected by quantization error, while the backward pass is not. This lack of explicit, corrective gradient path for the guantization error causes poor convergence in ultra-low- bit QAT. Our non-stran non-straight thhroggh method incorporates quantization enror into the backward pass via novel, data-dependent dequan-

---

Published as a conference paper at ICLR 2026
Quastization
Comporison
Our (5MM31 (A.M)) 3T6 (A38L)
2211
Steze
(a) Training Stability Shakespeare Dataset)
(b) Affine vs. Linear Quantization Performance
Figure 1: Training Stability and Quantization Robustness Analysis. (a) Comparison of training loss on the Shakespeare dataset with 1-bit weights and activations (AIWI). Standard STE and BitNet fail to stabilize in this extreme regime, exhibiting divergence or high loss. In contrast, our approach converges smoothly, matching the stability of pecheerrpeiiionn baselines. (b) Comparison of Linear vs. Affine quantization schemes AI Standard STE (right hars) fails utilize the additional expressivity of affine parameters, showing no improvement over linear quantization. Our method (left hars) robustly learns the affine parameters (scale and bias), achieving a significant accuracy jump (see Table [ Appendix A.1.2.
tization step (simple to implement; Code Snippets 12. provides well-defined gradients without estimation, creating perturbation-resilene forward pass and error-aware backward gradients.
Our framework's 's robustness enables a more general approach to training efficient models. Whereas recent state-of-the-art methods often rely on bespoke recipe changes (Wang et al.) (023), architec- tural modifications, or complex, bit-specific recipe tuning (Liu et al. 2025), our method provides a universal "drop-in" solution that is effective across wide range of precisions and standard archi- tectures. It also allows us to unlock the full potential of the theoretically superior but rarely used technique of affine quantization. The computational cost of a naive affine implementation has tra- ditionally made it impractical; we overcome this barrier with novel shortcut formula for affine quantized matrix multiplication that both robust and efficient (Figure lb)).
By treating sparsification form of quantization, our unified framework provides stable founda- tion to explore the trade-offs between storage, energy, and quality. On a Gemma3 IB model (Team) al. 2025), map the corage-accuracy Pareto frontier (Figure 2). revealing that asymmetric pre- cision (c.g.. b-bit activations 1-bit weights (A4W)) 1)) optimal for storage. Furthermore, we analyze approximate energy-
the trade-off between accuracy and computational efficiency by mapping accuracy frontier (Figure 3). For this, we use a hardware-agnostic cost metric that estimates the arithmetic computation effort Zhang et al. (2022). a dominant factor in the energy consumption of modern accelerators. This analysis reveals that structured sparsity can simultancously reduce this approximate computational cost and improve accuracy. These findings reinforce the viability of ex- tremely low-bit precision for high-capacity models, suggesting that aggressive quantization is key enabler for maximizing model performance within strict hardware constraints. Our key contribu- tions are:
Pinpointing STE's "quanizaaiion-obiivouss backward pass the key instability source- revicue-eeval.nn... for the first time, that proper gradients enable learning to handle quan- tization noise.
A simple, robust denoising dequantization transform from ridge regression, enabling for the first time stanle A III and sub- 1-it training with standard recipes.
A novel shorteut formula to make affine quantized matrix multiplication computationally efficient, reducing its overhead to few cheap, low-rank matrix operations.
State Sfate-of-the of-the-ar art results across range of challenging, ultra-low precision models, mapping the storage and energy efficiency frontiers for modern LLMs.
2

---

Published as a conference paper at ICLR 2026
2
MOTIVATIONS
2.1 GHOSTS OF DEPARTED QUANTITIES: THE STE BLIND SPOT
The stability of Quantization-Aaan Training (QAT) is critically dependent on its foundational algo- rithm: the Straight-Through Estimator (STE). For over a decade. STE has been the de facto solution to the non-differentiable nature of quantization, but its reliance on surrogate gradient estimation has perpetuated training instability. A central element of our analysis is reformulating the stan- dard quantization, y = 8 round = as an additive perturbation. By defining the rounding error 5 = round( ) _, a term that is critically detached and receives no gradient, the forward pass can 5
be expressed as simple addition: y ( + 5)
The issue arises in the backward pass. STE addresses the non non- differentiable rounding operation by replacing its true derivative with surrogate (Hubara et al. 2018 Yin et al., 2019). In the most common case, this surrogate is the identity function, yielding
1. The gradient of the loss with
respect to x is then computed as dL dl dv dL As is clear from this equation, the quantization error term 5 is completely absent from the gradient computation.
da dy de dy
In essence, STE creates system where the forward pass is quantization-awart but the backward pass is quantization-oblivee -obliviioon.o This is a "critical blind spot": the quantization error 6, despite af- fecting the forward pass, vanishes from the gradient, becoming, in the words of Bishop Berkeley's critique calculus, a "ghost of a departed quantity""Berkeley (1754). Because the update signal is blind to this error, due the simplistic gradient approximation, preceding layers have no opportu- nity to learn to handle the perturbation. This unmanaged error corrupts the learning signal, leading training divergence FFigure[aa)
2.2 THE AFFINE QUANTIZATION DILEMMA
A robust framework must also handle the asymmetric data distributions common in neural networks, which can degrade performance if the quantization grid is misaligned (e.g., aligning non-negative outputs from a ReLU function an integer grid of principled solution is the affine quantization s s-q+b, but it suffers from two major drawbacks directly linked STE's blind spot. First, STE's error-oblivious gradient struggles to optimize the sensitive bias term b, often yielding little to no quality gain (Figure (b)). Second, its naive implementation computationally expensive, undermining the goal of efficiency. A core motivation of our work is to unlock the benefits of affine quantization without these prohibitive costs.
1} whose mean is negative). The
2.3 THE PROMISE OF BITWISE COMPUTATION: A WI AND BEYOND
Solving these foundational issues facilitates the robust training of networks at extremely low preci- sion, such as 1-bit for activations and weights (AIWI). At this level, expensive floating-point matrix multiplications can be replaced by highly efficient bitwise operations, specifically XNOR and pop- count (bit-counting). This capability opens the door hardware architectures that are significantly simpler, faster, and more power-efficient. Furthermore, while Spiking Neural Networks (SNNs) have long sought achieve similar efficiency by mimicking discrete biological impulses (Hodgkin & Huxley 1952] Dayan & Abboti 2005). their non-differentiable nature poses significant training challenges Yamazaai et al. 2022 Lee et al.) 2016). As this paper demonstrates, our robust train- ing algorithm for 1-bit networks offers a practical alternative, achieving comparable computational sparsity and efficiency within a standard gradient- gradient-based framework.
3 A THREE-STAGE METHOD FOR ROBUST QUANTIZATION
Our framework conducts quantization-awan training using three- stree-stage process. This method ap- plies to both quantization and sparsification, which we treat as a special form of quantization that maps insignificant values to zero.
3

---

Published as a conference paper at ICLR 2026
3.1 STAGE 1: PREQUANTIZATIOS TRANSFORM (f)
First, we apply prequantization transform f, to map the high-precisinn precision input tensor into range suitable for integer or low-precision float rounding. For zero-centered data like weights or for easy- to-quantize regimes, we may use a simple linear transform f(z) = o,f |] For activations or in setra-low-precisiig regimes where asymmetry is critical (e.g., 2-bit values of {-2, -1. 11). we use an affine transform f() 2-b,-1 to optimally align the data with the quantization grid.
81
3.2 STAGE 2: QUANTIZATION ERROR INJECTION (6) AND THE STE BLIND SPOT
The second stage of our framework models the quantization step as an additive quantization error, 8. The quantized low-precision vector, q, defined as the of the scaled transform and this new error term:
= f(x) +
(1)
Here, ò = rf f f(x) represents the error introduced by the rounding operation. This formulation general: for integer quantization, rounding is to the nearest integer; for low- low-precision float formats like FP4, it is to the nearest representable float value. Crucially, 5 is intentionally detached from the computation graph so that receives no gradient in the backward pass.
This detached error, 5, is the central problem that causes training instability. The standard Straight- Through Estimator (STE) navigates the con-differentiable rounding function by approximating its local derivative as an identity (T(() I) or other surrogates (Bengio et al. 2013 Hubara et al,
2018 Yin et al. 2019). The gradient calculation is completely blind to the error that was introduced, and this unmanaged perturbation corrupts the learning signal that propagates to the preceding layers.
3.3 STAGE 3: DEQUANTIZATION WITH A DENOISING TRANSFORM (g)
A core innovation of our work lies in the dequantization step which maps the quantized data back to the original floating- Noating-ppintt range to approximate the unquantized data. While typical methods simply invert the scaling from Stage 1, our approach introduces denoising dequantization transform, approh
g. which is explicitly designed to solve the problem of the unmanaged quantization error, 5. Un- like heuristic gradient estimators, we formulate this dequantization as a principled ridge regression problem, yielding well-defined gradients that ensure numerical stability against low-variance data without any surrogate approximations.
3.3. FORMULATION: GENERAL AND SYMMETRIC TRANSFORMS
General (Affine) Dequantization for Uncentered Data For uncentered data, such as activations,
we use full affine transform, g(q) Sg. q + by. We find the optimal scale sy and offset by by solving the ridge regression objective:
1 2N
min bg
||s + bbb 1- x1| xxx 2
(2)
2
where À is regularization factor. The closed-form solution for the dequantized vector, is:
(وq) = Covag
Var, A (q q) +
(3)
The regularization parameter acts as a "denoising" knob. As À + 0o, the scale sg - O, and the dequantization collapses to the mean: T. forcing the transform to ignore the "noisy" quantized vector q and fall back to the most stable component of the original signal: its mean. This provides mechanism balance signal fidelity against noise suppression, which is critical for stabilizing the backward pass. A single value of À = 0.01 proved sufficient to ensure stable training across all our diverse experimental settings.
'The parameter for the linear quantization is 8y max(
"The parameters for the affine be computed as 8
Amax Emin and by Tuii 88 qmin-
Vmax - Ymin
4

---

Published as a conference paper at ICLR 2026
Symmetric (Linear) Dequantization for Centered Data For data that is naturally centered around zero, the dequantization simplifies to a more efficient linear transform, g(q) = $g q.
The optimal scale sy is found by solving simplified, bias-free objective:
A
min Bg 2N
|sg
x
(4)
2
This yields simpler closed-form solution for the scaling factor: 8g
(q.z)
+9.9)+X
3.3.2 How THIS SOLVES THE STE BLIND SPOT
Our method rectifies the information-flow problem in STE through a two-part mechanism:
The Forward Pass: The Error is Included. The input the dequantization transform g is the clean value (a)); it is the noisy, quantized vector = f(x) + 6. This means the quantization error is fundamentally part of the input to the dequantization step.
1.
2. The Backward Pass: The Gradient is Error-Aware.
During backpropagation, the gradient with respect to the quantized vector is computed via the chain rule: dه dL dg(g). Because the dg Zg dq
parameters of g (its scale and offset) are calculated from the statistics of q (Eq. 3). its derivative is directly shaped by the values within q- Since q contains the error ò, this local derivative da(g) be- comes an explicit function of that error (Sec. (Sec.[A.4). By forcing the quantization error to participate in the backward pass, transform provides the learning signal that STE discards, allowing preceding layers to adapt their weights and become robust to the error.
dg
3.4 SPARSIFICATION A SPECIAL FORM OF QUANTIZATION
Our framework seamlessly extends to network sparsification by treating it as a special form of quan- tization that maps only insignificant values to zero. Our framework unifies these techniques by modeling them as sequential, additive error injections.
First, hard-thresholding operation applied the full-precision tensor enforce specific sparsity pattern, such as 2:4 structured sparsity. This non-differentiable step introduces the first source of error, the sparsity error ôş threshold(z))-as x. The resulting sparse tensor is Ø5 = - Next, this sparse tensor 2s is fed into the quantization pipeline described in Section 3.2 This stage introduces the second source of error, the quantization error &G. resulting in the final oowpreccsion, sparse tensor q = f(zs) ōq.
The power our unified framework lies in the final dequantization stage. The denoising transform p(q) is applied to this doubly-perturbed tensor with the objective of reconstructing the original, dense, high-precision tensor x. Because the parameters of y are derived from the statisties of q. transform inherently learns to correct for the combined error distribution from both d5 and 8q. Since the input to the dequantization q = f fxs) + 8Q is the tensor after both sparsity and quantization errors have been injected, the ridge regression objective (Eqs. 2 implicitly minimizes the loss to the unperturbed æ with respect to the combined perturbation. The backward pass is aware of the total perturbation, allowing the network become robust both compression techniques.
4 THE DEQUANTIZATION TRANSFORM AS A NORMALIZATION LAYER
Our framework can be further understood through powerful analogy: the denoising dequantization transform also acts as normalization layer applied directly to the noisy, quantized vector q. This perspective highlights two complementary benefits: the normalization structure creates a corrective gradient pathway, and the term provides numerical stability akin to the in LayerNorm. This Normalization-basee view explains both the stability and efficiency of our method.
Comparing our transform standard LayerNorm: 7 g- 3. reveals the structural similarity:
5

---

Published as a conference paper at ICLR 2026
Cov
q
Our Denoising Transform g:
Var,
Var, +
Analogous
Analogous
Analogous ta M-
This formulation reveals that our transform normalizes the noisy quantized vector q and then rescales and shifts to best approximate the statistics the original tional cost of these operations is par with LayerNorm. When the bias term is not included, the transform simplifies and its complexity falls back to being on par with an RMSNorm.
vector æ. The computa-
5 EFFICIENT AFFINE QUANTIZED MATRIX MULTIPLICATION
two-sided affine transform provides the most robust quantization, and applying it on a per-channel basis critical for preserving quality. A naive implementation this operation, = X-W, expands into sum of four terms, making it complex to implement efficiently. We introduce novel shortcut that proves this qugh-tuality approach is both elegant and fast.
5.1 NOVEL SHORTCUT ENABLED BY THE L2 FORMULATION
Our method built upon a mean mean-centered and mean ceen-commponet interactions:
mean-centeree centering identity, which decomposes a matrix product into its
= X W = (X - E 1") 11-1-
(5)
where a and w denote mean vectors, and n represents the inner product dimension size.
This identity provides the key structure to dramatically simplify the affine dequantization process, as formalized in the following theorem.
Theorem 1, The result of a two-sided, channel-wise affine dequantization, V, can be expressed as:
= (8x @ (Q* QQ - 4x -ww) + x. 20" n
(6)
where n is the inner dimension size, the bur notation (-) denotes the mean, variables with X are col- umn vectors (row-wise statistics), and variables with W are column vectors (column-wise statistics) that are transposed where appropriate.
Proof. By substituting equation 3 iinto the quantized matrix multiplication:
XW = ((sx 1²) @ (Q* Tx 1²) + +
- -100-11-))001111- +1 w.
((sx 17) o (Qx - 17)) (1Q" -
(8x st o (Q"Q" Tx T -
Interpreting the Shortcut Formula Our theorem's 's efficiency comes from recasting complex expansion into simple structure: a standard linear term plus two cheap, rank-1 corrections. The main term, (sx wW @ (Qx . Q") is the standard computation for linearly quantized ma- trix multiplication. This is supplemented by two offset corrections: novel subtraction term that centers the product based on the means of the quantized data -x - n. and an addition that re- constructs the output's 's correct mean using the original high-precision data H w" n. This efficient structure proves that robust, channel-wise affine dequantization is nearly fast standard linear quantization, reducing the computational overhead from four matrix terms to single integer matrix multiplication and two low-rank corrections (Code Snippet 3).
EXPERIMENTS
We conduct comprehensive set of experiments to validate the robustness and efficiency of our quantization-awe -waree training framework. Our evaluation spans a range of model scales and architec- tures, from small-scale transformers to state-of-the-art Gemma L.MMs (Team et al. 2025). A key

---

Published as a conference paper at ICLR 2026
finding is that while many standard quantization methods benchmarked large, overparame- terized models where their flaws can be masked, these methods often falter on smaller models that have less redundancy. Our framework, in contrast, demonstrates superior stability and performance across all model scales.
While this section focuses on our primary results modern transformer architectures, the appendix provides a comprehensive set of supplementary experiments and analyses. These materials offer deeper validation of our framework's components. We analyze the consistent superiority of our method over the standard STE (Sec. A.I.I). quantify the benefits of affine quantization unlocked by our approach (Sec. (.1.2). and explore the application of advanced techniques, including our cen- tered Hadamard transform (Sec A.1.3), low-precision FP4 formats (Sec. Bec.(A...4).. and the trade-offs in structured sparsity (Sec. A.1.5) Furthermore, to demonstrate the broad applicability of our method, the appendix presents results on established benchmarks beyond large-scale language modeling, in- cluding ResNet-50 on ImageNet and Transformers on WMT machine translation tasks, providing a comprehensive validation of our framework's ' robustness and versatility.
6.1 EXPERIMENTAL SETUP
To test robustness, hyperparameters matched BF16 baselines. Our formulation eliminates need for tumed schedules. All use channel-wise or sub-channel quantization (SCQ. block 128) for fine- grained quantization. Notation: "AxWy" for x-bit activations, y-bit weights. For our quantization- only experiments (1, 2, and 4-bit), distributions. In contrast, for all experiments involving ternary quantization (1.5-bit) or structured sparsity, we intentionally use the simpler linear quantization, which naturally produces representa- tions favorable for hardware simplicity. The implementation frameworks are tailored to the exper- iment scale: small-scale experiments on nanoGPT models are conducted in PyTorch on NVIDIA A100 GPUs, whereas scage-scale pre-training on Gemma LLMs is implemented in JAX and trained on up to 64 TPUs.
affine quantization to effectively model asymmetric data
6.2 CORE METHOD VALIDATION ON NANOGPT
We begin by validating our method's 's stability on small-scale GPT models from the nanoGPT repos- itory (Karpathy) 2022). This low-resource setting effectively highlights the fragility of standard methods. We use the Shakespeare dataset, consisting of approximately one million characters, to train a transformer model with 6 layers, 6 attention heads, and 11 million parameters. We apply channel-wise quantization on the model. As shown in Figure [a), our method, using its robust affine quantization, converges smoothly even when both activations and weights are quantized to 1-bit (AIWI). In direct comparison, the affine version of STE exhibits significant instability the same precision an issue that could not be resolved by simply reducing the learning rate. These re- sults validate our theoretically grounded solution, which succeeds without gradient estimation where long-standing methods fail. For baseline comparisons, we utilize the publicly available symmetric implementations of BitNet (Wang et al.) 2023) and ParetoQ (Liu al., (2025), as they lack native support for affine quantization. Our results indicate that our method's 's inherent stability enables it to effectively leverage the superior modeling capacity of affine quantization, regime where other methods are either demonstrably unstable or lack native support.
To validate these findings on a more realistic task, we trained a GPT-2 small model (124M parame- ters) on the Open SeenWeeeText Text dataset for 25k steps. The results, shown in Figure 8 confirm our initial findings. Our method again demonstrates highly stable training, while both BitNet and STE show unstable training, with their validation losses becoming erratic or resulting in NaNs. While ParetoQ also trains, it converges to significantly worse loss.
6.3 SCALING TO STATE-OF-THE-AN LLMS: A GEMMA 1B CASE STUDY
To validate that our framework scales effectively modern architectures, we performed a compre- hensive analysis on the Gemma 1B model. We conducted a miniature pre-training run, training the model for 25k steps 6.5B tokens with context length of 512 from the C4 dataset. This setup serves as a rigorous testbed to map the Pareto frontiers for storage and computational efficiency, allowing us identify the optimal strategies for quantizing -fate-of-the-art language models.
7

---

Published as a conference paper at ICLR 2026
6.3.1 THE STORAGE EFFICIENCY FRONTIER: ASYMMETRIC QUANTIZATION KEY
Our first analysis examines the trade-off between model accuracy and the storage cost of the quan- tized weights, measured in effective bits per element (BPE). As illustrated in Figure 2 our method maps out a clear and compelling Pareto frontier, demonstrating a practical path to extreme model compression without significant performance degradation.
The central finding is that the optimal storage frontier is achieved not through a symmetric quan- tization scheme (e.g., A2W2), but through distinctly asymmetric strategy: pairing robust 4-bit activations (A4) with aggressively quantized, ultra-low-bit weights (e.g., W1). This approach is effective because it allocates the precision budget intelligently, preserving activation precision to maintain information flow while leveraging the static nature of weights for extreme compression. By integrating this principle with structured sparsity, our framework can push weights into the sub- 1 1-bit regime while preserving the model's 's core capabilities.
central dilemma in quantization is how handle outliers. While some methods clip these values, this risks discarding valuable information. A more recent approach, the Hadamard transform (Tseng 2024 Ashkboos et al. 2024 Liu et al. 2024; Panferov et al., 2023). acknowledges their potential value by blending outliers into other coordinate axes.
Our framework proposes third, more direct approach. We also acknowledge outliers may be features and, like some coordinate rotation methods, we "blend" them; however, we blend their influence into the computation of the statistical dequantization parameters (Eq. 3 rather than the coordinates. Both outliers and normal samples participate in this computation, ensuring all samples contribute to the information flow. This is made robust through subchannel quantization (SCQ), which localizes this statistical blending. An outlier's ss corrupting influence is contained within its own moderate-siee sized block, allowing its information to be preserved without distorting the quantiza- tion for the rest of the channel. Our results confirm that this strategy- local statistical blending via SCQ a superior Pareto frontier, establishing a more direct and practical path to state-of- the-art storage efficiency than methods relying on complex transforms like Hadamard.
Storage Efficiency Pareto Frontier (Gemma3 1B)
3F16 Baseline
A4W4 SCO128
Hadamard AEEI
0.42
A4W2 SCQ128
A4W1. SCQ128
A4W1 SC0128
A 2201929122500 5(1:4)
C4 Acturacy
0.38
0.36
0.34
BF16 Baseline
AL5A15
Pareto Frontier
1.0
2.0
2.5
3.0
3.5
4.0
Effective Bits Per Element (BPE) for Weights
We map the trade-off between validation accuracy and effective bits-per-element (BPE). The frontier reveals that asymmetric quantization (e.g. A4W1) provides superior storage-accuracy trade-off compared symmetric settings (e.g., A2W2), as preserves activation information while aggressively compressing static weights.
Figure 2: Storage Efficiency Frontiers:
6.3.2 THE ENERGY EFFICIENCY FRONTIER: THE SYNERGISTIC POWER OF SPARSITY
The energy efficiency analysis, shown in Figure B reinforces these findings and reveals a powerful synergy with structured sparsity. We use hardware-agnoste proxy for arithmetic energy cost Zhang et al. 22022,, defined as (Sparsity Factor) x (Activation Bits) x (Weight Bits). x (Total Operations). It is important note that this metric is first-order approximation and deliberately omits the dom-
8

---

Published as a conference paper at ICLR 2026
Energy Efficiency Pareto Frontier (Gemma3 1B)
Baseline
A4W4 SCQ128
Pareto Frontier
A4W2 SCQ128
BF16
0.42
A4W SCQ128
A4W1+ 2C S(2:4)
50012
S4W1+$C0128+511.
0.40
C4 Accuracy
ALOMLD SOLDSONG
0.38
ADAL
0.36
ADNL
0.34
256
Approx. Energy Cost (Sparsity A_Bits W__Operational Bits) [Log Scale]
Figure 3: Approximate Energy Efficiency Frontier: Estimating compute cost (Activation Bits Weight Bits x Sparsity). The results demonstrate synergy between our estimator and structured sparsity: the (1:4 and 2:4) structured sparse A4W1 models reduce the compute cost of the dense equivalent while maintaining high accuracy. defining the efficiency frontier.
inant costs data movement (Neseem et al., 2024), quantization overhead, or other non-arithmetic operations, but serves principled tool for comparing the theoretical arithmetic efficiency of different bit allocations. This score represents a principled lower bound on the true energy cost.
Our analysis identifies two critical results: First, for any given computational budget, asymmetric bit allocation that prioritizes activation precision remains the superior strategy. Second, and most notably, introducing 2:4 sparsity the A4W1 model simultaneously cuts the computational cost in half while also increasing model accuracy (from 0.4068 to 0.4080). This outcome confirms that the most effective path to efficiency is through the synergistic combination of our core denoising transform, asymmetric bit allocation, sub-channel quantization, and structured sparsity.
6.4 SCALING TO LARGER MODELS: A 4B PARAMETER CASE STUDY
To investigate whether larger model, when quantized to a similar footprint, can outperform smaller model, we conducted 26B token pre-training run comparing the 700a-errameter Gemma3 1B against the 3.2B --B-paraeeter Gemma3 4B.
6.4.1 STORAGE AND COMPUTE EFFICIENCY AT SCALE
Our results, visualized in Figure demonstrate that a larger model aggressively quantized with our method can be superior model, quantized to A4W1 with 2:4 sparsity, achieves higher accuracy (0.4517) than both the BF16 Gemma3 IB (0.4494) and the quantized A4W4 Gemma3 1B (0.4443). The energy efficiency anal- ysis reveals an clearer advantage: the sparse, quantized 4B model is not only more accurate than a quantized IB model but achieves this with significantly lower total computational cost. For detailed tabular comparison across all 1B and 4B configurations, please refer to Table 6 in Appendix A.1.6
smaller model, even one at higher precision. The Gemma3
6.4.2 SCALING PROPERTIES AND HARDWARE IMPLICATIONS
These experiments confirm that for a fixed budget, aggressively quantizing a larger model yields superior performance compared smaller, high-precision model. This trade-off offers distinct advantages for hardware implementation. Ultra-low-precisiin models, particularly those operating the 1-bit ternary level, be executed significantly simplified hardware architectures.

---

Published as a conference paper at ICLR 2026
Storage vs. Accuracy: Quantized Gemma 4B vs. Gemma 1B
6,400
0,455
Gennee (AW)
AdVS SCOLER 91211)
(595
0.400
Canva ARWO Scou.8
Garora BEAR
C.455
C4 Accuras
s-00 aCO128
Genvaa 48 A1 AWLLS 60g12
6.400
0.435
5.400
0.48 12 AL 200 SC0128
A26
1323
1200
1850
Total Weight Storage (MB)
Total Energy Efficiency. Q Quantized Gemma 4B vs. Gemma 1B
0.455
0.003
(Tetal 681
0.443
(Tetal Corke
C4 Accuracy
0.442
0.422
0.430
Approx, Total Energy Cost (Params A_Bits W_op_Bits) _pp_ttttt [Log Scale)
Figure 4: (a) Storage vs. Accuracy comparison between Gemma3 IB and 4B models. The quantized 4B model achieves higher accuracy than both BF16 and quantized versions of the IB model. (b) Total Energy Cost vs. Accuracy. The quantized and sparse 4B model is both more accurate and more computationally efficient than quantized 1B model.
By replacing complex floating-point units with efficient bitwise operation circuits, we can achieve substantial reductions in power consumption, silicon area, and manufacturing costs. This efficiency paves the way for deploying high-capacity models on edge devices and specialized accelerators where energy constraints are paramount.
7 SUMMARY
The non-differemtiable nature of quantization has long challenged efficient model training, as the community has viewed it as inherently unfriendly to gradient descent's reliance on smoothness. We pinpoint-for for the first time-thar the core issue is the lack of a proper gradient path allowing train- ing to learn robustness to quantization noise, which STE exacerbates by ignoring error in the back- ward pass. Our work provides a theoretically sound resolution through ridge regression-derived denoising dequantization transform, ensuring well-defined gradients without heuristic estimation. This unified framework enables robust training at arbitrary precisions and sparsity with standard recipes and no ad-hoe fixes, unlocking the full potential of low-precision computing for resource- constrained applications.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
AmirAli Abdolrashidi, Lisa Wang, Shivani Agrawal, Jonathan Malmaud, Oleg Rybakov. Chas Le- ichner, and Lukasz Lew. Pareto-optimal quantized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3091 2091-3099, 2021.
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213-10022 2024.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv:1611.01700 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 2013.
George Berkeley. The Analyst, Or, A Discourse Addressed to an Infidel Mathematician. J. and R. Tonson and S. Draper, 1754.
Yoonho Boo and Wonyong Sung. Fixed-point optimization of transformer neural network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1753-1757. IEEE, 2020.
Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Logarithmic un- biased quantization: Simple 4-bit training in deep learning. arXiv preprint arXiv:2112.10769, 2021.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 2016.
or-1. arXiv preprint
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. Vs-quant: Per-vector scaled quantization for accurate low- oow-precision neural network inference. Proceedings of Machine Learning and Systems, 3:873-884, 2021.
Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005.
Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo quantization noise. arXiv preprint arXiv:2104.09987, 2021.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk-
boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03077 2023.
Peijie Dong, Lujun Li, Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, et al. Stbllm: Breaking the 1-bit barrier with structured binary llms. arXiv preprint wrXh:22408.01803, 2024.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint arXiv: 2019.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010000412, 2020.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15 2015.
Alan L Hodgkin and Andrew Huxley. A quantitative description of membrane current and its application to conduction and excitation nerve. The Journal of physiology. 117(4):500, 1952.
11

---

Published as a conference paper at ICLR 2026
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations, journal of machine learning research, 18(187):1300 2018.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 27104 2018.
efficient
Andrej Karpathy, NanoGPT. httt::///g.tuub..co//aaaaaaa
2022.
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. arXiv preprint arXiv:1911.1240 2019.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00098. 2023.
Chunlei Liu, Peng Chen, Bohan Zhuang, Chunhua Shen, Baochang Zhang, and Wenrui Ding. Sa- bnn: State-aware binary neural network. Pruceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2091-2099, 2021a.
Jing Liu, Jianfei Cai, and Bohan Zhuang. Sharpness-aware quantization for deep neural networks. arXiv preprint 21.x:21111/2221 2021b.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp-
722-737 -37, 2018
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng- How do adam and training strategies help bnns optimization. In International conference on machine learning, pp. 6936-6946. PMLR, 2021c
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantiza- tion with learned rotations. arXiv preprint arXiv:2405. 240.64405.66666, 2024.
Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. Paretoq: Scaling laws in extremely low-bit 11m quantization. arXiv preprint arXiv:2502.02631, 2025.
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani. John V Arthur, Izzet B Yildiz, and Dharmendra S Modha. Discovering low- low-precision networks close to full- precision networks for efficient embedded inference. arXiv preprint
(e0::0099999999 2018.
Marina Neseem, Conor McCullough, Randy Hsin, Chas Leichner, Shan Li, In Suk Chong. Andrew Howard, Lukasz Lew, Sherief Reda, Ville-Mikko Rautio, et al. Pikelpn: Mitigating overlooked inefficiencies of low-precision neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15996-16005, 2024.
Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto Castro, Mahdi Nikdan, and Dan Al- istarh. Quest: Stable training of ilms with 1-bit weights and activations. arXiv preprint arXiv:2502.050003 2025.
Jun-Hyung Park, Kang-Min Kim, and Sangkeun Lee. Quantized sparse training: A unified train- able framework for joint pruning and quantization in dnns. ACM Transactions on Embedded Computing Systems (TECS), 1-):1112 2022.
12

---

Published as a conference paper at ICLR 2026
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542- Springer, 2016.
Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrish- nan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 00511796-107,, 2020.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint arXiv:2503. 2025.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402. 04396, 2024.
Hongyu Wang. Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arxiv preprint arXiv:2310. 11453, 2023.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6612-8620, 2019.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. arXiv preprint arXiv:2304. 13013, 2023.
Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. arXiv preprint arXiv:2306. 11987, 2023.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences, 12(7):863, 2022.
Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by sparsity-quantizatien quantization joint learning: A constrained optimization cadiizzation-based approach. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2178-2181111 2288,2 2020.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing ttraight- estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662. 2019.
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12475-12485, 2022.
Yichi Zhang, Ankush Garg, Yuan Cao, Lukasz Lew, Behrooz Ghorbani, Zhiru Zhang, and Orhan Fi- rat. Binarized neural machine translation. 2023. UKL/atppss//ppeneeviiee..n forum?
id d= fmWpu
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
This appendix presents supplementary experiments and analyses that further validate our frame- work's robustness and versatility. We begin with detailed ablation studies: we analyze our method's consistent superiority over standard STE (Sec. A.I.Ip, quantify the gains unlocked by affine quan- tization (Sec. A.1.2). and explore advanced techniques including our centered Hadamard transform (See. (Sec.|A.1.3) A.1.3). low-precision FP4 formats (Sec. (.I.44) and structured sparsity trade-offs (Sec..A.1.5) Additionally, we outline energy consumption methodology in Sec. A.1.6 To demonstrate appli- cability beyond LLM pre-training, we present results on ResNet-50 (Sec. A.1.7 and WMT machine translation tasks (Sec. A.1.8).
On the implementation side, we detail the computational flow of quantized matrix multiplication in Sec. A.2 analyze the complexity of the affine shortcut in See. Sec. and and provide rigorous gradient analysis in Sec. AA.4. Finally, we provide an extended discussion of related work in Sec. A.5 and include reference code in Sec. A.6
A.1 ABLATION STUDIES AND ANALYSIS
A.1.1 OUR METHOD CONSISTENTLY OUTPERFORMS STE
Across all tested configurations, our denoising reconstruction method consistently outperforms the standard Saraight-Througe Estimator (STE). While the improvement is present at all bit-widths, the performance gap widens significantly at lower precisions. highlighting method's superior stability and accuracy the most challenging regimes (Table Figure
a higher precision like A4W4 (Affine, SCQ 128), our method achieves an accuracy of 0.4274 compared to STE's 0.4254.
The advantage grows at A2W2 (Affine, SCQ 128), where our method scores 0.4027 while STE only reaches 0.3794.
Most critically, in the A1.5W1.5 channel-wise setting, STE fails to converge, whereas method remains stable and achieves a respectable accuracy 0.3297, demonstrating its fundamental robustness.
Our Method vs. Standard STE
2,474
1.4
3.4
3.44
Figure 5: Comparison of our denoising reconstruction method against STE across all experiment configura- tions, sorted by method's 's performance. Our approach consistently yields higher accuracy, and the improve- ment is most pronounced lower bit-widths. Notably. in the A1.5W1.5 channel-wise setting, STE fails converge entirely.
14

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of our denoising reconstruction method against STE across various configurations. Our method consistently yields higher accuracy. with the most significant gains at lower bit-widths, The STE run for A1.5W1.5 channel-wis Linear quantization failed to converge.
Configuration
Quantization
STE Ace.
Our Method Acc.
Improvement
AIWI
Linear, Channel
0.3235
0.3335
+0.0100
AIWI
Affine, Channel
0.3045
0.3442
+0.0397
AIWI
Linear, SCQ 128
0.3399
0.3547
+0.0148
AIW1
Affine, SCQ 128
0.3397
0.3751
+0.0354
A1.5W1.5
Linear, Channel
NaN
0.3297
A1.5W1.5
Linear, SCQ 128
0.3722
0.3835
+0.0113
A2W2
Affine, Channel
0.3449
0.3831
+0.0382
A2W2
Affine, SCQ 128
0.3794
0.4027
+0.0233
A4W4
Linear, Channel
0.4167
0.4219
+0.0052
A4W4
Affine, Channel
0.4149
0.4218
+0.0069
A4W4
Linear, SCQ 128
0.4245
0,4265
+0.0020
A4W4
Affine, SCQ 128
0.4254
0.4274
+0.0020
A.1.2 AFFINE QUANTIZATION GAINS ARE UNLOCKED BY OUR METHOD
Our results show that while affine quantization is theoretically superior for handling asymmetric data, only our method can reliably unlock its benefits, especially at low precision. In contrast, STE often fails to gain any advantage from an affine transform and can sometimes even perform worse (Table Fi
With our method at A1w1 (SCQ 128), Affine (0.3751) significantly outperforms Linear (0.3547), demonstrating clear benefit. However, with the standard STE baseline A1W1 (channel-wise), Affine STE (0.3045) is surprisingly worse than Linear STE (0.3235).
This supports our conjecture that the learnable bias term in affine quantization is highly sensitive to outliers. STE's "quamization-oblivoou"" gradient cannot optimize this sensitive parameter effec- tively, leading to a misaligned quantization grid. The ability our framework to correctly handle this error is therefore critical advantage.
Conversely, at higher precisions such as 4-bit, the benefits affine quantization diminish signifi- cantly. Our A4W4 experiments show that simple linear transform achieves nearly identical results to the more complex affine version (e.g.. 0.4265 for Linear with our method vs. 0.4274 for Affine with method, both with SCQ 128). This suggests that while correctly modeling asymmetries is critical in ultra-low-bit regimes, its importance is less pronounced when more bits are available.
Table 2: Comparison of quantization schemes. For each configuration, the highest accuracy for each method (Our Method vs. STE) is in bold. Our method consistently benefits from affine transform, whereas STE's performance is erratic, often favoring simpler schemes.
Our Method Accuracy
STE Accuracy
Configuration
Linear
Affine x Lin W
Affine
Linear
Affine x Lin W
Affine
A1W1, Channel
0.3335
0.3429
0.3442
0.3235
0.3085 0.3403
0.3045
AIWI, SCQ 128
0.3547
0.3706
0.3751
0.3399
0.3397
A4W1, Channel
0.4016
0.4007
0.4000
0.3986
0.3971
0.3908
A4W1, SCQ 128
0.4056
0.4066
0.4068
0.4042
0.4042
0.4036
A4W4, Channel
0.4219
0.4216
0.4218
0.4167
0.4142
0.4149
A4W4, SCQ 128
0.4265
0,4258
0.4274
0.4245
0.4242
0.4254
15

---

uses subchannel Published as a conference paper at ICLR 2026
Comparisnn Comparison Quantization Schemes:
Performance with Extinater
Parformance with STE Extimator
Figure 6: Side-by-side comparison of quantization schemes (Linear, Aftine X Linear W, and full Affine). (Left) With our denoising reconstruction method. full Affine scheme provides the best performance especially at low precision. (Right) With the STE baseline, the benefit of Affine scheme inconsistent and sometimes detrimental, particularly at low precision.
A.1.3 CENTERED HADAMARD TRANSFORM FOR OUTLIER MITIGATION
One effective strategy for mitigating outliers during quantization is to apply change of coordinates that rotates the feature space, a technique designed to prevent large values in a single dimension from dominating the quantization grid. This approach, which we call outlier blending, distributes the impact of an outlier across multiple new dimensions. This stands in contrast to the outlier localization philosophy of main framework, which, discussed in Section 6 quantization (SCQ) to confine an outlier's influence to a small, local block.
While early approaches considered random rotations Tseng al.) 2024 Ashkboos al., 2024 Liu et al.] 2024), the randomness can be removed through the deterministic and computationally efficient Fast Hadamard Transform (Panferov al., (025). However, the deterministic structure of the Hadamard transform introduces a new challenge. Its first basis vector is constant vector of all ones, creating a sigmificant DC component. Real-world data neural networks is often uncentered; when such data is transformed, this DC component can create new, massive outlier in the rotated space, disrupting the subsequent quantization process.
To address this, we propose a novel application of our mean-correction technique to the Fast Hadamard Transform. We pre-center the data before applying the transform and add the mean back as low-rank correction term afterward. The computation becomes:
( x 1 (W - 1 (.-ww) + w
In our framework, we treat the Hadamard transform and Sub-Channel Quantization (SCQ) as stan- dalone techniques be compared against each other for improving quantization quality. For achiev- ing ultimate quality, these techniques can be used together, but at a higher computational cost. Our analysis of the Pareto frontier curve revealed that SCQ with block size of 128 consistently outper- forms the quality gain from a standalone Hadamard transform. Furthermore, the Hadamard trans- form leads extra energy costs not fully captured by our primary efficiency metric. Given these findings, we concluded that SCQ offers more direct and efficient path to state-of-the-art perfor- mance. Therefore, while we validate our centered Hadamard approach here, we feature our storage efficiency study and prioritize SCQ for the main energy efficiency analysis.
This analysis highlights two distinct philosophies for handling the outlier dilemma (noise vs. fea- ture). The rotation-base approach blends outliers into other coordinates, distributing their magni- tude. Our framework, in contrast, blends outliers into the statistical parameters (scale and offset) of the dequantization transform having all samples participate in the ridge regression (Eq. 3) This is made robust by SCQ, which ensures the statistical blending is a local effect. While our cen- tered transform improves the Hadamard implementation (Table Figure 7) main experiments
16

---

Published as a conference paper at ICLR 2026
(Figure show that the localized statistical blending via moderate- block-size SCQ consistently out- performs the gains from the global poordinate-blending of the Hadamard transform, confirming it as more direct and effective solution.
moderate-block-sszz
Centered Uncentered Hadamard Transform Performance
2.4
Centerel
04M4
140
3.2
C4 AeAcuree
C.404
02494
Figure 7: Comparison of applying the Hadamard transform to centered versus uncentered data. Pre-centering the data before the transform consistently improves accuracy across different bit-widths and for both method and the STE haseline, confirming the benefit of mitigating the DC-induced outlier.
Table 3:
Comparison applying Hadamard transform to centered versus uncentered data. Pre-centering the data consistently improves accuracy across different bit-widths for both our method and the STE baseline.
Configuration
Method
Uncentered Acc.
Centered Acc.
Improvement
AIWI
STE
0.3541
0.3560
+0.0019
A1W1
Our Method
0.3605
0.3630
+0.0025
A1.5W1.5
STE
0.3476
0.3507
+0.0031
A1.5W1.5
Our Method
0.3635
0.3656
+0.0021
A4W4
STE
0.4243 0.4250
0.4246
+0.0003
A4W4
Our Method
0.4253
+0.0003
A.1.4 QUANTIZATION TO LOw-PRECISION FLOATS (FP4)
Our framework is limited integer quantization and also supports low-precision float formats. In this scenario, the quantization vector can represent values from grids like FP4 or FP8. Fur- thermore, our reconstruction parameters (scale and bias) are themselves resilient low-precision storage; empirical evidence confirms that storing them in float8 (E5M2) does not adversely affect accuracy.
This flexibility allows for interesting comparisons between formats. We conducted experiment to compare standard 4-bit integer quantization (INT4) against the E2M1 variant of 4-bit floating-point (FP4). We find that FP4 quantization yields slightly better result than its INT4 counterpart. As shown in Table 4 the A4W4 model with our linear quantization achieves an accuracy of 0.4273 in FP4, compared to 0.4265 in INT4. We conjecture this is because FP4's format, which includes exponent bits, provides a wider dynamic range, making it inherently more tolerant to outliers than a uniform integer grid.
However, it is important to note that the best overall result in this configuration is still achieved by using an affine transform with integer quantization, which reaches accuracy of 0.4274. This
17

---

Published as a conference paper at ICLR 2026
suggests that while FP4 is robust to outliers, explicitly modeling the data's asymmetry with an affine transform on uniform grid can be an even more effective strategy.
Table 4: Comparison of b-bit integer (INT4) and 4-bit floating-point (FP4) quantization using our method. FP4 slightly outperforms the standard linear INT4. However, the best performance is achieved by using affine transform on the integer grid.
Configuration
Quantization Scheme
Accuracy
A4W4, SCQ 128
Linear (INT4)
0.4265
A4W4, SCQ 128
Linear (FP4 E2M1)
0.4273
A4W4, SCQ 128
Affine (INT4)
0.4274
A.1.5 TRADE-OFFS IN STRUCTURED SPARSITY AND QUANTIZATION
Our framework enables class of ultra-low precision models by combining M:N structured sparsity with quantization. This process creates ternary weights (f-1, 0, 1)) by first enforcing sparsity mask and then quantizing the remaining non-zero values. For these approximately zero-centered ternary weights, we use the efficient linear reconstruction for the dequantization transform.
Analysis of Efficiency Metrics When evaluating sparse models, it is crucial to consider not just storage but also computational energy.
The storage cost for M:N sparse formats must include metadata. For a block of elements and 1-bit values, 1:4 sparsity requires 2 bits for metadata (a 2-bit index for the single non-zero location), yielding effective BPE of (1 2)/4 0.75. In contrast, 2:4 sparsity requires 4-bit mask (c.g., to encode specific 2/4 pattern). resulting in higher BPE of (2 4)/4 1.5.
Storage (Effective BPE):
Computation (Energy Score):
The benefit of structured sparsity lies in computational savings. Modern hardware can exploit N:M patterns to reduce the number of multiply- accumulate operations. We estimate this with an energy score (Sparsity Factor Act. Bits x Weight Bits). For 2:4 sparsity, this factor is 2/ = 0.5, halving the computational cost.
Results and Analysis As shown in Table combining structured sparsity with our method reveals important trade-offs between accuracy, storage, and energy.
Our sub-1 1-bit model (1:4 sparsity) provides a compelling option for memory-constrained environ- ments. It achieves 25% reduction in weight storage (0.75 BPE) and 75% reduction in computa- tional energy, with only minor and graceful degradation in accuracy compared to the dense A4W1 baseline.
Most notably, the 2:4 sparse model achieves the highest accuracy of all (0.4080), slightly outper- forming the dense baseline. While its storage footprint is higher (1.5 BPE), its primary advantage is 50% reduction in energy and compute cost. This demonstrates that 2:4 structured sparsity in- troduces a powerful representation that can simultancously boost model accuracy and dramatically lower inference energy consumption.
Table 5: Comparison of dense and sparse weights on Gemma IB (A4W1). Structured sparsity provides signif- icant energy savings. The 2:4 model halves the energy cost while slightly improving accuracy.
Weight Type
Sparsity
Accuracy
Effective BPE
Energy Score
Dense Baseline
None
0.4056
1.00
4.0
Sparse Ternary
1:4
0.3979
0.75
1.0
Sparse Ternary
2:4
0.4080
1.50
2.0
18

---

Published as a conference paper at ICLR 2026
A.1.6 THE ENERGY EFFICIENCY FRONTIER: METHODOLOGY
To meaningfully evaluate the benefits of our quantization and sparsification framework, we must move beyond conventional metrics like FLOPs and instead estimate the total computational cost. We adopt bardware-inspired, anergy-aware metric motivated by the principle that on modern ML hardware, the energy consumed by a multiply-accumulate (MAC) operation is roughly proportional to the product of the bit-widths of its operands (Zhang al. 2022). This is because the fundamental cost of digital multiplier scales quadratically with the precision of the numbers processes.
Following this principle, we define Approximate Total Energy Cost score for each model con- figuration. This score serves as a hardware- hardware-agnostie proxy for the total energy consumed by the arithmetic operations that dominate the computational workload.
The score is calculated as follows:
Total Energy Cost R (Sparsity Factor) x (Activation Bits)
(Weight Bits) (Total Operations)
Where:
Activation Bits and Weight Bits are the precisions used for the activations and weights, respectively. The product, Activation Bits x Weight Bits, represents the relative energy cost a single MAC operation. For a standard BF16 operation, this baseline cost is 16 x 16 = 26, whereas A4W1 operation has relative cost of approximate energy per operation.
4-a 64x reduction in
Sparsity Factor accounts for the reduction in MAC operations due to structured sparsity. For M:N sparsity (e.g., 2:4), this factor is For 2:4 sparse model, the computational cost is effectively halved, so the sparsity factor is 0.5. For dense model, this factor is 1.0.
Total Operations is the total number of multiply accuuulae (MAC) operations in the model. This factor scales the per-operation cost reflect the model's overall size and com- putational density. When comparing models of different scales, we use the total number of parameters as a proxy for total operations.
It is worth noting that some techniques, such as the Hadamard transform, introduce additional com- putational costs not captured by this MAC-centric metric. While the Hadamard transform has complexity of (n Iog n), for simplicity, we omit detailed analysis of its energy contribution in this study. This metric nevertheless provides unified and principled foundation to systematically map the trade-off's between total energy efficiency and model accuracy. By using this score, can construct the energy efficiency Pareto frontier shown in Figures 3 4 revealing the synergistic power of combining robust training framework with asymmetric quantization and structured sparsity.
Table 6: Comparison of Gemma IB
4B models. A larger model (4B) aggressively quantized with method surpasses the performance a smaller (IB) full- precision model.
Model
Act Bits (A)
Weight Bits (W)
SCQ
Sparsity
Method
Accuracy
Gemma3 IB
16
None None
Baseline
0.4494
Gemma3 IB
4
128
Affine
0.4443
Gemma3 4B
16
16
None
Baseline
0.4706
Gemma3 4B
1
128
None
Affine
0.4221
Gemma3 4B
1.5
1.5
128
None
Linear
0.4298
Gemma3 4B
1.5
1.5
32
None
Linear
0.4393
Gemma3 4B
2
128
None
Affine
0.4465
Gemma3 4B
I
128
None
Linear
0.4496
Gemma3 4B
128
None
Affine
0.4510
Gemma3 4B
1
128
2:4
Linear
0.4517
19

---

Published as a conference paper at ICLR 2026
Table 7: Top-1 Validation Accuracy of ResNet-50 the ImageNet Dataset.
Precision
100 Epochs
400 Epochs
A32W32
76,41
A4W4
76.45
A4W2
75.12
75.59
A4W1
72.04
73.97
A.1.7 RESNET-50 ON IMAGENET
We utilized the Flax framework to train ResNet-50 from scratch ImageNet, employing stochastic gradient descent with an initial learning rate of 0.1, training the model for 100 epochs with weight decay of 0.0001. As shown in T the top-1 accuracy from the A4W4 configuration (76.45) sur- passes the baseline (76.41) without any hyperparameter tuning. This demonstrates the effectiveness of method in achieving competitive performance without requiring extensive optimization. We compared our results to previously reported A4W4 quantization results for ResNet-50 trained on Im- ageNet. Our results compare favorably to existing work, without the need for additional operations such as parameter search, fine-tuning, calibration, clipping, gradient estimation, or reinforcement learning (Table
Table 8: Comparison top-1 accuracy for A4W4 ResNet-50. The columns represent: FP32: Full precision baseline. A4W4: Quantizing both activations and weights to 4 bits. GE: Whether gradient estimation is involved. PT: Peeraiiiig/inetunnng/lllbbttoooo required. Clip: Clipping required. LB: The lowest bitwidth
reported in corresponding paper. EEstimated from Fig. (Abdolrashidi
2021
Method
FP32
A4W4
GE
PT
Clip
LB
AQT
Abdolrashidi et al. 2021
76.65
76.4"
Y
Y
Y
VS-Quant (Dar et al. 202
76.16
75.28
Y
Y
Y
3
FAQ
McKinstry et al. 2018)
76.15
76.25
Y
Y
Y
HAQ
Wang et al. 2019)
76.15
76.14
N
Y
Y
4
Ours
76.41
76.45
N
N
N
1.1.8 TRANSFORMER ON WMT
To evaluate the effectiveness of our method on transformer models, we employed the Flax frame- work to train the transformer model two WMT2017 datasets (EN-DE, DE-EN) and subsequently assessed performance on the corresponding WMT2014 datasets. The training process utilized 0.1 and cosine scheduler for 25,000 steps, employing
AdamW optimizer with weight decay a batch size 1024. Recognizing the known slow convergence of transformer models, we extended the training duration to 100,000 steps (Table 9). Remarkably, low-precision results consistently surpass the full-precision baseline.
Given the prevalence of transformers in large language models, extensive research has been dedi- cated to quantizing transformer models. We compare our findings to other works, and ours stands as the only method that can surpass the full-precision baseline (Table 10 This achievement highlights the unique strength of our formulation, which not only preserves signal fidelity but also benefits from regularization effects. Several recent works have explored alternative quantization ap- proaches using different datasets, which are not included in this table. One such method is AWQ, a weight-only quantization 4-bit quantization method (Lin al.) 2023), requires retaining 1% of salient weights and all activations unquantized. Their method also involves searching for an opti- mal scaling factor and a calibration set. Additionally, BitNet (Wang et al. 2023), presents 1-bit quantization method for transformers. The lowest activation precision achieved in their work is 8 bits, exceeding the highest activation bit in our method. Their method also necessitates clipping, additional normalization, and recipe changes.
Binary Transformers
20

---

Published as a conference paper at ICLR 2026
Table 9: BLEU Score of training low oww-prccision Transformers the WMT datasets. Models trained for 100k
steps consistently outperform their counterparts. and low-precision settings like A4W4 and A4W2 achieve results competitive with or even exceeding the full-precisio baseline.
DE-EN
EN-DE
Precision
25k Steps
100k Steps
25k Steps
100k Steps
A32W32
33.5
33.9
29.49
29.8
A4W4
33.78
33.64
29.71
30.17
A4W2
33.45
34.04
28.58
30.03
A4W1
32.76
33.66
27.06
28.32
A2W2
32.32
33,51
27.56
28.61
A2W1
31.39
32.51
26
27.4
AIW1
27.4
28.27
21.42
23.64
Table 10: BLEU score comparison of A4W4 transformers. Columns are defined as in Table
Method
FP32
A4W4
GE
PT
Clip
LB
LSQ+LUQ Xi et al. 2023)
27.5
27.17
Y
N
Y
4
Fixed-Point [Bo Sung 2020)
28.48
26.94
GradScale (unn et al. 2020)
27.5
25.9
Y
N
N
4
L.Q++MM (Chmiel et 2021
27.5
27.25
Ours
29.49
29.71
N
Table 11: BLEU Score of Transformers with binary activations on the WMT datasets. For the last column we record the drop from full precision models.
DE-EN
EN-DE
Precision
25k Steps
100k Steps
25k Steps
100k Steps (Drop %
A1W4
29.74
30.74
24.07
26.28 (-11.81 %)
A1W2
28.81
29.81
23.4
25 (-16. 16..11% %)
A1W1
27.4
28.27
21.42
23.64 (-20.67%)
A1W1
Zhang et al. 2023)
17.87 (-32.18%)
Inspired by the temporal nature of the transformer model, we reduced the activation precision to 1-bit for all linear layers, transforming into temporal binary network, akin to quasi-spiking neural network. However, unlike traditional spiking neural networks, our model doesn't 't simulate spike generation or consider spiking frequency. To evaluate our approach, we assigned 1, 2, and 4 bits to the weights (Table [T] Our results showcase that these converted binary transformers remain highly competitive with full-precision counterparts
recent study attempted binarize transformers (Zhang et al. 2023). Their approach included extra normalization layers, clipping, and progressive quantization during training. We compared our method to their A IW1 configuration, achieving significant improvements (Table 11 last column).
1. SUBCHANNEL BLOCK SIZE
In addition to precision and sparsity, subchannel block size also presents a trade-off between aceu- racy and efficiency. We observe that its influence is more pronounced at extremely low precision levels, such as 1-bit. While using a smaller block size can improve performance, the effective bits element can easily exceed the original design. We provide some comparisons here. Consider- ing this trade-off, lower precision models may not always be more efficient (Table 12] Firstly, the accuracy achieved with smaller blocks in AIW1 models may not surpass the accuracy achieved using A2W2. The perturbations introduced during lower precision training often remain high, hin- dering the achievement of high quality. Optimal selection needs to be made based the underlying
21

---

Published as a conference paper at ICLR 2026
hardware support and the problem of interest. Our work facilitates comprehensive study of this trade-off.
Table 12: BLEU Score comparison of adjusting the block size when training the AIWI Transformers for 100k
steps on WMT DE-EN Dataset.
Block Size
A1W1 BLEU (100k)
32
29.71
128
28.27
512
27.14
A.1.10 THE DUAL ROLE OF THE DENOISING TRANSFORM AND REGULARIZATION (A)
The stability of our framework arises from the synergy between two components: the novel gradient path created by our dequantization transform and the regularization term A. This section aims to disentangle these two effects and clarify their respective roles.
The primary innovation of our method is the new gradient path, deig), which makes the backward dg
pass "quantization-awarc This path is the fundamental mechanism that allows the network to learn to be robust to quantization error by forcing the error term å to participate in the gradient computa- tion. This mechanism exists even when À = O. However, this new gradient path can be numerically unstable. The denominator of the scaling factor $y in our transform (Eq, 3 and its linear counter- part) contains the term g'arg- In common scenarios where block of data has zero or near-zero variance-for example, if all activations after a ReLU are positive and get quantized to the same integer-this denominator collapses, leading to division by zero and training divergence. This is precisely the behavior observed in Table 13 when X = 0.
The regularization term provides the principled solution to this instability. It is not an ad-hoc fix but an integral part of the ridge regression objective (Eq. 2 and 4 from which our transform is derived. Ridge regression specifically designed to provide stable solutions to such ill-conditioned problems. The À term ensures the denominator is always bounded away from zero, guaranteeing stable and well-defined solution for the scaling factor Sg. While simple heuristic like clipping the variance (c.g., mar(V ( argg e)) could also prevent NaNs, use of À is more theoretically sound as smoothly discounts the influence noisy, low-variance blocks.
As described in Section 3.3 also acts "denoising" knob. A larger À suppresses the impact of perturbations by reducing the scale 8g forcing the transform to rely more on the stable mean of the original signal, 26.
In our experiments, we observe that a wide range of À values yield satisfactory results (Table 13 For higher precision (2 bits), where quantization noise is lower, smaller À values (e.g.. 0.000T) can be safely used. However, to ensure stability across all settings, especially during the transition 1-bit quantization, we our preference on the safer side and use default of 0.01.
In summary, the stability of our framework comes from two-part design: the new gradient path provides the mechanism for error-aware learning, while the regularization provides the principled robustness that makes this mechanism numerically stable and viable in practice.
Table 13: BLEU Score comparison of adjusting the À when training the AIWI Transformers for 25k steps the WMT EN-DE Dataset.
X
A1W1 BLEU (25k)
1.0
5.83
0.01
21.42
0.0001
20.08
0
NaN
22

---

Published as a conference paper at ICLR 2026
A.1.11 ADDITIONAL EXPERIMENTAL VALIDATION
In this section, provide further empirical evidence substantiate the robustness of our method across different scales and distinct quantization schemes.
Scalability to Tex (GPT-2 Small). To validate that our stability findings generalize to standard language modeling benchmarks, we trained GPT-2 Small model on the WebnWe dataset. Figure 8 illustrates the training loss trajectories in the challenging AIWI regime. Our method demonstrates smooth and stable convergence throughout the training process. In contrast, the standard STE baseline and BitNet exhibit significant instability or failure to converge to com- petitive loss. This confirms that the robustness of our denoising estimator holds as model capacity and dataset complexity increase.
Training Loss Comparison OpenWebText (GPT-2 Small)
12
NaN
Ours (RQ A1W1)
STE (A1W1)
ParetoQ (A1W1)
BitNet (AIW1)
10
Loss
7
5
40
5000
10000
15000
20000
25000
Training Steps
Figure 8: Scalability: GPT-2 Small on WpenWeTText (A1W1). Our method (Blue) converges smoothly, whereas STE (Orange) and BitNet (Red Cross) exhibit instability divergence.
Isolating Estimator Gains: Linear vs. Linear Comparison. To rigorously isolate the contri- bution of our denoising estimator from the benefits of affine quantization (addressing Reviewer BVSw's inquiry), we reproduced the Shakespeare A1WI experiment with additional "linear-only" configurations. We compared our method against the standard STE, BitNet, and ParetoQ, restricting all methods to identical symmetric linear quantization schemes where applicable.
The results, presented in Figure 9 demonstrate three critical findings:
1. Estimator Stability (Linear vs. Linear): When restricted to linear quantization, our method converges stably low validation loss (== 2.1). In stark contrast, the stan- dard STE with linear quantization exhibits significant instability and diverges to high loss (> 5.0). This confirms that our estimator provides fundamental stability advantage independent of the quantization mapping.
2. Comparison with Strong Baselines:
Our linear implementation outperforms both BitNet and ParetoQ in terms convergence speed and final validation loss.
3. The Affine Advantage: While our linear method robust, enabling our full affine quan- tization yields the lowest overall loss (&s 1.9). This reinforces that our estimator is robust enough to unlock the additional expressivity of affine parameters, which other methods fail to utilize effectively.
23

---

Published as a conference paper at ICLR 2026
A1W1 Loss Comparison on Shakespeare Dataset
Durs (Allnee AIWI
(Lnnear A1W1]
BRteet (AIW1)
ParetorQ (AIW))
LLinear A1W1)
Validation Loss
2-
1000
2000
3000
4000
5000
Training Steps
Figure 9: Mechanism Isolation: Sinear-vs-Linext Ablation (Shakespeare). We isolate the estimator's effect by running "Ours" and "STE" identical linear schemes. Ours (Linear) remains stable and outperforms STE (Linear), confirming the estimator's 's intrinsic value. Ours (Affine) achieves the best overall performance.
A.2 COMPUTATIONAL FLOW FOR QUANTIZED MATRIX MULTIPLICATION
The practical application of method during training and inference follows of two computa- tional paths, depending on the availability specialized hardware for low-precision arithmetic.
Simulation on Standard Hardware (Fake Quantization). In the absence of native hardware sup- port for low-precision integer matrix multiplication (e.g., on standard CPUs or GPUs), the benefits of quantization are simulated. This "fake quantization" flow is beneficial for Quantization-Awar Training (QAT) and for validating model accuracy. The process involves quantizing the high- precision input tensors (weights and activations) and then immediately dequantizing them back to floating-poin ppint format. This injects the quantization error into the tensors, which are then multiplied using standard floating-pon point arithmetic. While this approach correctly models the accuracy impact of quantization, it does not yield the computational or energy savings of true low-precision matrix multiplication.
Native Low-Precision Inference with Hardware Support. When specialized hardware is available TPUs TPUs or modern GPUs with integer matrix-multiply units), the full efficiency our framework can be realized. In this flow, the input tensors quantized low-precision integers and the matrix multiplication, Qx Qw. is performed directly using highly efficient integer arithmetic. The result, still in low-precision integer format, is then dequantized back the floating-poin domain using our shortcut formula. This final dequantization step, which includes the low-rank corrections and scaling, performed in floating-point but is computationally inexpensive compared the main matrix multiplication. This two-stage process-a fast, low-precision multiply followed by cheap, high-precision dequantization-nnlocks the full performance and energy benefits of method.
A.3 COMPLEXITY ANALYSIS OF THE AFFINE SHORTCUT
The primary benefit the novel affine shortcut (Theorem 1, Section 55 is the reduction of compu- tational overhead from the naive expansion of the two-sided affine quantized matrix multiplication. Here, we analyze the complexity of the shortcut formula:
- (8x s xQx Q" ax (hrn _0
24

---

Published as a conference paper at ICLR 2026
Let the dimensions of the imput matrix X be . x N and the weight matrix W be N P. The output matrix Y is M x P. The variables 8x, xx E are row-wise statistics (M x 1), and sw, qw: w are column- wise statistics (P 1).
1. Main Computation: The core term is the low-precision integer matrix multiplication (Int- MM): Q QQ".
PMMNP). This is the cost of the standard matrix multiplication, per- formed in high-speed, low-precision arithmetic.
Complexity:
2. Correction Terms and Element- Wise Operations: The remaining operations constitute the overhead:
Statistical Overhead: Calculating the mean (qx: Qw: I. w) and scale (sx, Sw) terms requires computing the statistics of X and W. This involves O(MN + NP) complexity, a negligible cost compared to the main matrix multiplication.
Rank-1 Corrections: The two terms 4x Tu and x=w""n n are rank- outer products, which are then scaled. Their complexity is (M
Element- Wise Scaling: The final element-wise (Hadamard) multiplication @ and the additions/subbractioons all have complexity OMMP).
Conclusion on Overhead The total overhead introduced by the shortcut is dominated by the P(MP) complexity of the low-rank corrections and the OMM + NP) complexity of the statisti- cal caleulations. Since the main matrix multiplication has complexity of (MMNP); the relative overhead of the affine shorteut is negligible the typical setting where the depth N large (i.c., N > M and N P).
The shorteut's ' complexity is asymptotically equivalent to a standard low-pree precision linear quantized matrix multiplication, making the robust, high-quality affine quantization practically efficient.
A.4 GRADIENT ANALYSIS: EXPLICIT DEPENDENCE QUANTIZATION ERROR
In Section 3.3.2 posit that our denoising transform establishes backward pass that is explicitly aware of the quantization error 8, addressing the "blind spot" of the standard Straight-Through Estimator (STE). Here, we provide rigorous derivation of the local gradient (Jacobian) to validate this claim.
Consider the symmetric (linear) dequantization case for simplicity (Eq, 4h. The transform is defined as:
g(q) (g) -q
(10)
where the scale factor s, is a constant parameter, but scalar function of the quantized vector q
2 derived from the ridge regression solution: sg(q) q a+
To analyze the backward pass, we compute the Jacobian matrix J rule (q)) yields:
8g(q) 8g
Applying the product
Ōg(g) sn(g) I+ (V)s{(g))
(11)
0q
where I is the identity matrix.
The first term, 8, (g) corresponds to standard scaling operation. The second term, however, introduces data-dependent rank-1 correction. We derive (g) using the quotient rule:
Tq q A) (x q)2q
VgZ
(12)
(q
2)
Substituting this result back into the Jacobian expression reveals that the gradient complex, non-linear function of q.
The Role of Quantization Error. Since q f(x) + 6, the local derivative 0g 8g is explicitly
parameterized by the quantization emorr 5.
25

---

Published as a conference paper at ICLR 2026
Comparison with STE. In standard STE, the backward pass typically assumes an identity (or constant scalar) derivative oa(a) I, effectively rendering the gradient independent of the specific Bi value of 8. In contrast, our method computes the gradient of the loss with respect to æ via the chain rule:
ac Oc Og(a)
θg(g)
0q
(13)
dx
3q
Ōx
Error-Aware
Because the term og(g) Bg varies based on the magnitude and direction of the noise å (embedded within ②), the learning signal flowing back to the weights is dynamically modulated by the instantaneous quantization error. This mechanism is mathematically analogous to the stabilizing gradients found in Normalization layers, forcing the network to adapt to the noisy quantization surface.
A.5 RELATED WORK
Neural network quantization has become widely adopted technique for reducing memory foot- print and accelerating inference time, enabling efficient deployment on resource-constrained de-
vices (Gholami et al. (2022). While full-precision models typically store weights in floating-poin ppint
format, quantized weights are represented as integers, typically using bits (Dai et al.
2021 Worts-
man et al. 2023 Jacob et al.
2018), 3-4 bits (Dettmers et al. 2023 Liu et al.) bb Abdolrashidi
ct al. 2021; ct al. 2021). ). or even bit (Zhang et al. 2022
Liu et al., 2021a Wang et al.
2023
Zhang et al. 2023 Courbariaux et al. [0166] Rastegari al. 2016). In addition to quantizing
weights tt, model activations can also be quantized to further enhance computational efficiency (Dai
al. 2021 Jacob et al., 20118 Esser al. 2019).
Although 8-bit quantization is commonly used as a standard practice in industry (Jacob et al. 2018). achieving lower-bit quantization remains challenging and requires specialized techniques to ensure robust training. Several common techniques include: 1. Mixed precision quantization: This ap- proach selectively assigns different bit levels to different weights. aiming to optimize the trade-off between model size and accuracy (Wang et al., 2019, Lin et al.] 2023 Han et al., 2015 Défossez al., 2021). 2. Training recipes: These techniques compensate for the discontinuities introduced by quantization by employing strategies such as sharpness-aware minimization (Liu et al. 2022bb] Foret et al., 2020). state-aware optimization (Liu et al.) 2021a), knowledge distillation (Kim et al, 2019), and multi-phase training (Liu et al. (222cc). 3. Quantization-frienll network architectures: This approach involves replacing original network layers with alternatives that are more amenable to quantization (Zhang et al. 2022).For For example. Bi-Real Net enhances 1-bit CNNs by adding an identity shortcut to preserve some real-valued information in the forward pass Liu et al. (2018).
In contrast to prior work, our method explicitly models quantization discontinuities perturbations. We decompose the perturbed signal into clean and noisy components, then apply denoising to sup- press the noise. This approach leads a closed-form solution that guarantees training convergence even at extremely low bitwidths. While previous methods have also modeled quantization noise using continuous distributions (e.g.. Uniform or Gaussian) for gradient estimation (Défossez al.) 2021 Ballé et al. 2016). they do not optimize the reconstruction process itself to enhance training stability.
To further reduce model footprint, researchers have been combining sparsity/pruning and quanti- zation in unified formulation further compress neural networks (Park et al., 2022 Yang et al., 2020). In this paper, we extend our noise injection and denoising reconstruction theory to sparsity.
In the time since our method was first proposed, the community has made rapid advancements in bow-bit LLM training. One major thrust has been to improve the Quantization-Aware Training (QAT) process itself. Recent works like ParetoQ (Liu et al.] (2025) have developed unified frame- works systematically compare different bit-widths and highlight the importance of the quantiza- tion function design. By optimizing training schemes and refining quantization functions, ParetoQ surpasses methods tailored to specific bit-widths and reveals that ternary and 2-bit quantization can exceed 4-bit performance in the size-accuracy trade-off.
Concurrently, methods like QuEST (Panferov et al. (025) have sought improve upon the standard STE by designing more sophisticated backward passes. QuEST introduces a "trust gradient estima- tor" to stabilize training, coupled with Hadamard normalization in the forward pass to make weight
26

---

Published as a conference paper at ICLR 2026
distributions more suitable for quantization. Another direction has focused combining binariza- tion with structured sparsity to break the 1-bit barrier. Post-training methods like STBLLM achieve effective bit-widths low 0.55 by employing N:M sparsity techniques Dong et al. (024).
Our work addresses the core problem of training instability from a different, foundational perspec- tive. Instead of developing bit-specific heuristies or entirely gradient estimators, explicitly model the quantization discontinuity as an additive perturbation. Our primary contribution denoising dequantization transform, derived from a principled objective, that makes the training process inherently aware of and robust to this quantization error. By deriving well-defined gradients without surrogates, our approach offers a theoretically grounded alternative the heuristic, gradient estimation based methods.
A.6 REFERENCE CODE
We provide the reference code for our core contributions.
def affine queeeigaatteeeeeee (x, pits, axis, eps=le-8): """uaantizs a tensor to the range [0, 2++bits 1]."""
max_val max.max(e (x, axis=axis, keepdims-True) min_val inp.min axis=axis, keepdims-True)
Scale to the target integer range
scaled_x= = (x - min_val) / (max_val - min_val + eps) (2**bits - 1)
Detach rounding error from the gradient delta jax. lax, stop_
10
round scaled
11
12
Inject quantization error scaled_x + delta
Code Snippet I: JAX reference code for quantization: = (x) + 8.
def affine_dequant (q, x, axis, 1md=le-2)
a**ueuantties zes based on a ridge regression objective."""
E_q2 jnp. (q axis-axis, keepdims-True)
E_q jnp. (q,
axis=axis, keepdims=True)
__ jnp. x, axis=axis, keepdims=True)
x = jnp.mean
axis=axis, keepdims=True)
Var_c E_q2 E_q++2 Cov_cx E_qx E_q E_x
10
11
Solve for scale (s) and offset (o)
Cov_qx (Var_q + 1md)
Ex E_q
14
15
return + lq E q)
E x
Reconstruct: r s*q +
Code Snippet 2: The denoising dequantization transform r= r=
B ACKNOWLEDGEMENTS
We extend sincere gratitude to Abhijit Ogale, Dinghua Li, Jeff Dean, Jian Li. Kyuyeuan Kim, Rasmus Larsen, Sameer Agarwal, Sanjiv Kumar, Sergey loffe, Tammo Splank, Zhifeng Chen for their insights and supports.
27

---

Published as a conference paper at ICLR 2026
def affine quant ized matmul_shorteut (x, w, bits, 1md=le-2):
"""Computes an efficient affine quantized matmul.
Note: Assuncs a function affine_quant that returns (q, s, 0)
x_x, 8_x, = affine_quant (x, bits-bits, axis=-1) 9_w, s_w, affine_quant (w, bits-bits, axis=0)
# Decompose into linear term two rank-1 corrections
linear_term q_x q_w corr corr_term = mean (-1, keepdims-True) 8 sum keepdims=Truel
(-1, keepdins=True) w.sum (0, keepdims=True)
11
12
Combine terms as per Theorem 1
13
return (s_x A s_w) (linear_term corr_term_l) corr_term_?
Code Snippet The shortcut formula affine quantized matrix multiplication
28


Reviews:
[{"review_id": "lMXW71CnUH", "reviewer": "Reviewer_nwiN", "summary": "The paper tries to address the drawbacks of using Straight through estimator in QAT by proposing a denoising dequantization transform which models quantization error in both fprop and backprop. The proposed transform acts a normalization layer stabilizing gradients.\nThe efficacy of the proposed technique is measured on low bitwidth and sparse networks, and the emperical results on nanoGPT, Gemma-1B/4B models show the pareto frontier for storage and energy and demonstrates the outperformance compared to literature work on STE, BitNet etc.", "questions": "1.How sensitive is training stability and convergence to λ across architectures and datasets?\n2. Does the model remain robust during methods such as fine-tuning (dense or using LoRA)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper shows stable training of A1W1 and sparse sub-1-bit models where prior literature methods diverge.\n2. Demonstrates results on nanoGPT and Gemma-1B/4B, showing pareto efficiency on storage and energy frontiers.\n3. The method is simple, and needs no special hyperparameter tuning.", "weaknesses": "1. Lacks ablation on the sensitivity of λ (regularization) across architectures.\n2. Do you more experiments with longer running fine tuning or pre-training to ensure that convergence and generalization does not suffer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KCk9p2C7Md", "reviewer": "Reviewer_Rrwh", "summary": "This paper proposes an improved method for STE, where the backward pass is surrogated by that of naive linear mapping. Specifically, the authors claim that STE ignore the impact of quantization error during the backward pass, but only take into account of it for the forward pass, which is claimed to be the main cause of instability involved in training quantized models. The authors propose an algorithm based on a denoising dequantization transform derived from ridge regression, which is shown to be related to normalization layer, partially explain the reason of stablization. The experssion for the proposed quantization method is simplified to enable an efficient implementation with negligiblely introduced extra computation, only of the order of rank-1 corrections. The proposed method could unify binarization/quantization with sparsification, and experiemnts shows some improvements.", "questions": "- At line 100 of the original paper, it states that unmanaged error corrupts the learning signal, leading to training divergence. Could the author provide some further demonstrations to verify this statement, e.g., gradually introducing such quantization error in back-propagation, i.e., making it as a linear combination of the vanilla full-precision gradient (which include the quantization error) and the one from STE (which does not include it), to demonstrate that the error indeed causes the divergence issue?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method of using ridge regression to derive the quantization backward is novel.\n- The analysis and method could unify binarization/quantization with sparsification.", "weaknesses": "- The characters in the figures are too small and very difficult to read.\n- The final results are too difficult to read from Figure 2 and there are no table for comparison to illustrate the claimed advantage compared to previous results.\n- The authors claim that in STE, the forward pass is affected by quantization error while the backward pass is not (line 042 in the original paper). However, during the backward pass, the weight and activations involved are also quantized, so it is also different from full-precision model. This statement could be improved and should be corrected.\n- For asymmetric data distribution, as discussion in Section 2.2, the usual quantization process is to directly map it to some given quantized range, say from the full-precision interval of [0,1], to values of {0, 1/4, 2/4, 3/4} for 2-bit quantization, which does not introduce bias issue. Another way is to first shift and scale it to make it symmetric and apply the symmetric quantization, and finally rescale and shift back. For weight quantization, unbiased quantization is important as it impacts the training dynamics and is crucial for convergence, but for activation quantization, which is usually the asymmetric part, the symmetry requirement is not necessary and would not impact the training dynamics. Regarding training dynamics and symmetric weights, the authors could check previous work, such as [1].\n- In equation 3, the authors should explain what the values with bar above them are.\n- In Section 3.3.2, for the first part, for STE, the input to the dequantization of STE also includes the quantization error. For the second part, it is not clear or proved why the local derivative is an explicit function of the error. For example, suppose in equation 3, the final function after simplification become a linear function of q. The derivative would then be independent on q or delta, i.e. the quantization error.\n- In equation 5 and 6, the authors should explain what n represents.\n- Writing problems: The sentence at line 366 in the original paper is not complete nor correct in grammar. The sentence “This low resources setting quickly highlights the fragility of standard models.” at line 368 in the original paper is redundant and should be deleted.\n\n\n[1] Ben Poole, et al., Exponential expressivity in deep neural networks through transient chaos. NeurIPS 2016.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3lVNnY4cSW", "reviewer": "Reviewer_xFhy", "summary": "Quantization aware training (QAT) is a widely used method to prepare models for quantization. To overcome the rounding operation during backward pass, QAT uses straight through estimator (STE) which bypasses the gradients in the backward pass outside of the rounding operation. This work, characterizes this feature as the backward pass being quantization oblivious. To overcome this, the authors present a workaround that exposes the backward pass to the quantization error using a combination of ridge regression and affine quantization. Experiments on a wide array of models, and bit regimes shows superior performance of the proposed method when compared to plain STE-based QAT.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* This paper studies an interesting problem that is commonly taken as a given -- of using STE in QAT. \n* Interesting approach to alleviate STE gradient bypassing that allows error-aware gradient.\n* Sparsification as a form of quantization?\n* Impressive results on a wide-range of experiments; shows a clear benefit over QAT with STE.", "weaknesses": "* **QAT and error minimization:** The fundamental hypothesis of this work is that during the backward pass, QAT is oblivious to the quantization error. While this is possible, it has been shown by now that STE creates a different type of dynamic that results in weight oscillations. And these oscillations have factors that are trying to compensate the errors due to QAT [1,2]. Given this, what do the authors make of these explanations? And how does it alter their hypothesis, or not? If not, what is their argument?\n\n* **STE beyond 2 bits:** Even if one were to attribute STE for poor performance in extremely low-bit width, how is this not manifested to the same degree in higher bit regimes. What is the explanation? Is STE more problematic only in extremely low-bit regimes? \n\n* **Ternary quantization with STE:** There are several works that use STE for ternary quantization [3,4]. The critique that STE cannot be used for extremely low-bit regimes does not hold up. \n\n* **Presentation clarity:** The ideas, experiments, and results are quite compelling in this work. However, the presentation is unclear in many places. There are vague statements, unsubstantiated by evidence (discussions around biological neurons, intelligence), and presentation of results makes it difficult to parse them. Many of the interesting results are in the Appendix (Fig. 3, Table 1) whereas the main results in Fig. 1 and Fig. 2 are illegible, with no clear captions, legends, axes labels. This is unfortunate as it dilutes the impact of otherwise nice contribution. I would suggest improving these aspects.   \n\n* L-67: Very vague statement with exaggerated claims. It is by now common knowledge that large, quantized models outperform smaller ones. And also the claim of biological intelligence is extremely misplaced. \n\n### Other comments\n\n* Reference to Figure 1-a in L-37 is not useful as none of the concepts are fully introduced; consider dropping this reference or elaborating the caption so that it can independently explain the concepts in the figure.\n\n* L-38: What are the heuristic-based modifications authors are pointing to? No references to back this up. \n\n* L-54: Strange sentence; perhaps missing a preposition somewhere. Did the authors mean \"full potential of the theoretically...\"\n\n### References\n\n1. Wenshøj, Jonathan, Bob Pepin, and Raghavendra Selvan. \"Oscillations Make Neural Networks Robust to Quantization.\" arXiv preprint arXiv:2502.00490 (2025).\n2. Xie, Weiying, et al. \"Allowing Oscillation Quantization: Overcoming Solution Space Limitation in Low Bit-Width Quantization.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.\n3. Choi, Jungwook, et al. \"PACT: Parameterized Clipping Activation for Quantized Neural Networks.\" (2018).\n4. Wang, Jinheng, et al. \"1-bit ai infra: Part 1.1, fast and lossless bitnet b1. 58 inference on cpus.\" arXiv preprint arXiv:2410.16144 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HokCsxLliA", "reviewer": "Reviewer_BVSw", "summary": "The paper locates the root cause of QAT instability in the STE’s \"quantization-oblivious\" backward pass and replaces it with a non-STE framework. The core method is a denoising dequantization transform derived from a ridge-regression objective, which injects an explicit corrective gradient path for quantization error and extends naturally to sparsification. The authors also introduce a shortcut for efficient affine-quantized matmul. The framework supports training across a wide range of precisions, including fully binary (A1W1) and sparse sub-1-bit models, with standard recipes.", "questions": "1. Can you clarify whether, in the extreme A1W1 regime, memory bandwidth remains the dominant bottleneck or whether the additional “shortcut” terms shift execution to a compute-bound regime—and, if so, whether these computations can be overlapped (e.g., via pipelining or operator fusion) with the main integer GEMM/memory traffic to hide their latency?\n2. Can you provide ablations using linear (symmetric) quantization only, where your method and all baselines (STE, BitNet, ParetoQ) use the same quantization scheme and SCQ block size, so the estimator’s contribution is isolated from affine/SCQ effects?\n3. Can you report how your method scales beyond 1B and 4B (e.g., ≥7B/13B/30B) and what advantage—if any—it holds over BitNet and related A1W1 approaches?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A ridge-regularized dequantization transform creates an error-aware gradient path, stabilizing training at ultra-low precision and treating sparsification as quantization within one framework.\n2. The approach is compatible with standard autograd and includes a shortcut formula that makes affine-quantized matmul computationally viable in practice.\n3. At low precisions, the method achieves a new storage–energy Pareto frontier (demonstrated on Gemma-1B), offering clear, actionable trade-off guidance for model design.", "weaknesses": "1. Efficiency claims rely on a hardware-agnostic energy proxy rather than end-to-end measurements (latency/throughput/energy) on real accelerators.\n\n2. Using affine+SCQ for the proposed method versus symmetric implementations for BitNet/ParetoQ can over-attribute gains to the estimator; require matched schemes (linear vs. linear) to isolate estimator effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)