QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Authors: Andrei Panferov
Jiale Chen
Soroush Tabesh
Mahdi Nikdan
Dan Alistarh
Venue: ICML 2025 poster (2025)
Keywords: efficiency
LLMs
quantization
gradient estimation
Primary area: deep_learning->algorithms
TL;DR: We show that LLMs can be stably trained down to 1-bit, and are optimally trained at 4-bit weights and activations via a new quantized gradient estimation technique.

Abstract: One approach to reducing the massive costs of large language models (LLMs) is the use of quantized or sparse representations for training or deployment.
While post-training compression methods are very popular, the question of obtaining even more accurate compressed models by *directly training* over such representations, i.e., *Quantization-Aware Training (QAT)*, is still open: for example, a recent study put the "optimal" bit-width at which models can be trained using QAT, while staying accuracy-competitive with standard  FP16/BF16 precision, at 8-bits weights and activations. We advance this state-of-the-art via a new method called QuEST, for which we demonstrate optimality at 4-bits and stable convergence as low as 1-bit weights and activations. 
QuEST achieves this by improving two key aspects of QAT methods: 
(1) accurate and fast quantization of the (continuous) distributions of weights and activations via Hadamard normalization and MSE-optimal fitting; 
(2) a new *trust gradient estimator* based on the idea of explicitly minimizing the error between the noisy gradient computed over quantized states and the "true" (but unknown) full-precision gradient. 
Experiments on Llama-type architectures show that QuEST induces stable scaling laws across the entire range of hardware-supported precisions, and can be extended to sparse representations. 
We provide GPU kernel support showing that models produced by QuEST can be executed efficiently. Our code is available at [https://github.com/IST-DASLab/QuEST](https://github.com/IST-DASLab/QuEST).

QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Andrei Panferoy Jiale Chen' Soroush Tabesh'
Mahdi Nikdan
Dan Alistarh 12
Abstract
QUEST WIAI
One approach to reducing the massive costs of large language models (LLMs) is the use of quan- tized or sparse representations for training or deployment. While post-training compression methods are very popular, the question of obtain- ing even more accurate compressed models by directly truining over such representations, i.e., Quantization-Awan Training (QAT), still open: for example, a recent study (Kumar et al., 2024) put the "optimal" bit-width at which models can be trained using QAT, while staying accuracy- competitive with standard FP16/BF16 precision. at 8-bits weights and activations. We advance this saae--of-the-art via for which we demonstrate optimality at 4-bits and stable convergence as low as 1-bit weights and activations. QuEST achieves this by improving two key aspects of QAT methods: (I) accurate and fast quantization of the (continuous) distribu- tions of weights and activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust gradient estimator based on the idea of explicitly minimizing the eтror between the noisy gradient computed over quantized states and the "true" (but unknown) full-precision gradi- ent. Experiments on Llama-type architectures show that QuEST induces stable scaling laws across the entire range of hardware- narpoare-supported pre- cisions, and be extended to sparse represen- tations. We provide GPU kernel support show- ing that models produced by QuEST can be exe- cuted efficiently. Our code is available at ht tps: //github com/ IST DASLab SAUSS.///SS...... Q.EST.
3.8-
QUEST W2A2
QuEST W3A3
3.6
QUEST W4A4
calvall Loss
3.4
BF16
3.2
3.0
2.8-
2.6-
2.4
10'
10'
10'
Memory, Mbit
Figure 1. The scaling law induced by QuEST when training Llama- family models from 30 to 1.6B parameters on C4, with quan- tized weights and activations from kens/parameter regime (harder compression uses proportionally more data at fixed memory). QuEST allows for stable training at 1- bit weights and activations (WIAI) and the QuEST W4A4 model is Pareto-dominant relative to BF16. with lower loss at lower size.
method called QuEST,
bits, in the 100 to-
ficiency a critical challenge. One popular pathway to in- creased efficiency has been reducing numerical precision, usually done via post-training quantization (PTQ) methods for compressing weights (Frantar al., 2022; Lin al., 2024; Chee et al., 2024; Tseng et al., 2024) or both weights and activations (Ashkboos et al., 2023; 2024; Zhao et al., 2023). Quantizing both operands necessary to leverage hardware support for low-precision multiplications, which extends down 4-bit (NVIDIA, 2024). However, state- of-the-art PTQ methods are still far from recovering full accuracy for 4-bit precision (Ashkboos et al., 2024; Liu et al., 2024), leaving a gap between computational support and achievable accuracy.
One alternative is quantization training (QAT) (Raste- gari et al., 2016; Jacob et al., 2018) where models are trained from scratch with low-precision weights and acti- vations on the forward pass, but with a full-precision back- ward offferinn the potential for superior accuracy -ss- compression trade-offs, gradient optimization can correct compression errors. Despite promising results for weight- only quantization (Wang et al., 2023; Kaushal et al., 2024), it is currently not known whether QAT can produce accu- rate LLMs with bittiit weights and activations. Here,
1. Introduction
The massive computational demands of large language mod- els (LLMss, e.g. (Dubey et al., 2024), have made Al ef-
'ISTA 'Red Hat AI. Correspondence to: Dan Alistarh <dan.alia alintalistrrhee at.
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025, Copyright 2025 by the author(s).

---

Training Accurate LLMs with Low-Bit Weights and Activations
the key metric is the Pareto-optimal frontier, i.e., the mini- mal representation size (or inference cost) for the model to achieve certain accuracy under fixed data or training bud- get. Recently, Kumar et al. (2024) identified 8-bit precision as Pareto-optimal for QAT methods on LLMs.
kernels showing that models produced by QuEST can be run efficiently commodity hardware.
2. Background and Related Work
Hubara et al. (2016) and Rastegari et al. (2016) were among the first to consider training neural networks with highly- compressed internal states, focusing primarily on weight compression. Later work focused on quantization-awe training (QAT) (Jacob et al., 2018; Choi et al., 2018; Esser et al., 2019; Bhalgat et al., 2020) in the form considered here, where the model weights and activations (i.e. the forward pass) are quantized, but the backward pass is performed in full- full-pircis,,n, using variants of the straight-through esti- mator (STE) (Bengio et al., 2013). (The variant where all states, including gradients, are quantized (Wortsman et al., 2023; Xi et al., 2024) is beyond the scope of this paper.)
Contribution. We present QUEST, new QAT method that brings the Pareto-optimal frontier to around 4-bit weights and activations and enables stable training at 1-bit precision for both operands. As shown in Figure 1, when data and compute are scaled proportionally to model size, QuEST can train models with 4-bit weights and activations that have superior accuracy relative to BF16 models almost 4x in size.
We achieve this by re-thinking two key aspects of QAT meth- ods: 1) the "forward" step, in which continuous-to to-discrete tensor distribution fitting is performed on the forward pass, and 2) the "backward" step, in which gradient estimation is performed over the discrete representation. For the for- ward step, QuEST works by approximating the "optimal" continuous-to-discrette mapping by first applying a normal- izing Hadamard Transform, and then computing an MSE- optimal quantization for the resulting distribution. This replaces the prior "learned"' normalization approaches (Choi et al., 2018; Bhalgat et al., 2020).
Broadly, QAT considers the problem of finding quantized projection over a standard standard-precision tensor x. representing part of the weights or activations, minimizing output error. For symmetrie uniform quantization, the projection onto the quantized lensor & is defined as:
* = a clip(x, a)
The key remaining question is how to find an accurate gradi- ent estimator over a weight or activation tensor quantized as above. Here, prior work leverages the Straight-Throuu timator (STE) (Bengio et al., 2013), augmented with learn- able components, e.g. (Bhalgat et al., 2020). We propose a different approach called trust estimation, which seeks to minimize the difference between the "true" gradient (taken over high-precision weights) and its estimate taken over Lower-precision weights and activations. To do this, trust estimator diminishes the importance of the gradient for some components depending on their quantization error on the forward step, following the intuition that entries with large errors lead to significant deviations in the gradient.
Es-
where the clip function performs a clamping operation over the value distribution for all values above the clipping pa- rameter a, which also acts as a scaling factor, normalizing values to to [-1, and the function [-1 rounds each value to its nearest quantization point, defined as a uniform grid whose granularity depends on the number of available bits b (i.e., {-I...., 2 2--1..... 1)). Most QAT methods propose "learn " the factor a, for instance, via gradient- based optimization. For example, QAT methods usually keep a standard preccisonn version w of the weights; the STE gradient is computed over the quantized weights w, and then added to the full-precision accumulator, possibly also updating the clipping factor a.
1].
Next, we focus on the following question: assuming that training computation is not a limiting factor, what is the "'optimal" precision in terms of address this, we implement QuEST in Pytorch (Paszke et al., 2019) and train Llama-famil models (Dubey et al., 2024) of up to 1.6B parameters up to 160B tokens from the standard C4 dataset (Raffel et al., 2019), across precisions from INTI to INT8. Results show that QuEST provides stable and accurate convergence across model sizes and precisions down to 1-bit weights and activations. This in- duces new scaling laws, which we study across model sizes in the large-data (100 tokens/parameter) regime. QuEST leads INT4 weights and activations to be Pareto-optimal in terms of accuracy at a given model size and inference cost, suggesting that the limits of low precision training are lower than previously thought. In addition, we provide GPU
midel-size? To
Recent work such as BitNet (Wang et al., 2023; Ma et al., 2024) and Spectra (Kaushal et al., 2024) showed that weight- only quantization is viable for small- and medium-scall L.LMs. The concurrent work presents BitNet a4.8 (Wang et al., 2024), a hybrid scheme that combines ternary weights with mixed 4- and 8-bit activations, applied selectively to different matrices. In parallel, Kumar et al. (2024) inves- tigated scaling laws for GPT-type models with quantized states, concluding that the "Pareto-optimal" point for current QAT methods is around 8-bit weights and activations.
Prior work by Frantar et al. (2023); Jin al. (2025) stud- ied scaling laws specifically for sparse foundation models, establishing that the loss can be stably predicted across pa- rameter and data scales when the model weights are sparse.
2

---

Training Accurate LLMs with Low-Bit Weights and Activations
Recently, Frantar et al. (2025) generalized these laws to unify both sparsity and quantization, allowing to compare the "effective parameter count" for these two types of repre- sentations. Our work focuses on improved training methods for highty-compprssed representations, leading to improved scaling laws relative to standard dense training, and can be applied to both sparsity and quantization.
with the significant "outlier" errors incorporated by term (+*). For this, we propose efficient fitting mechanism that minimizes persistent errors, coupled with a "trust" gradient estimator step aimed at bounding outlier errors.
3.1. Step 1: Distribution Fitting
While optimizing the quantization grid to best fit the under- lying tensor is a core idea across all quantization meth- ods, PTQ methods traditionally use more complex and computationally heavy approaches (Dettmers et al., 2024; Malinovskii et al., 2024). In contrast, QAT methods rely on backpropagation through the scaling factor for error- correction (Esser et al., 2019; Bhalgat et al., 2020) while performing re-fitting. To avoid backpropagation errors im- pacting the forward pass, we do not use backpropagation for distribution fitting. Instead, we start from the empiri- cal observation that the distribution of weights and activa- tions during LLM training is sub-Gaue Gab-asssiinn but with long tails (Dettmers et al., 2022; 2023).
3. QuEST
Motivation. A simple way of describing current QAT meth- ods is that, given haandadd-precision tensor w, we first try to get an accurate discrete approximation w by optimizing parameters such as the clipping factor a in Equation 1 to minimize some loss target, such as the mean-square-enne (MSE), and then rely on STE to estimate V the gradient over w, by V L, the gradient taken w.r.t. the quantized weights W. Yet, the difference between these two gradients, which correlates to the gap in optimization trajectory, could be unbounded, specifically because of large errors in a small subset of entries.
Gaussian Fitting. Specifically, choose to optimize the grid to explicitly fit a Gaussian distribution with the same parametrization as the empirical distribution of the underly- ing tensor x. Concretely, we use root mean square (RMS) normalization to first align the empirical distribution of x with a (0, 1) Gaussian distribution (Frantar et al., 2025). We then perform the projection operation with the scale a* chosen to minimize the L22 error resulting from projecting (0, 1). Formally:
Instead, in this paper, we seek to minimize the "gradient bias," i.c. the difference between the true and discrete gradi- ents, measured, e.g. as
|| VWL--V// VoL
Prior work on gradient compression (Alistarh et al., 2017; Nadiradze et al., 2021) has identified this quantity as being critical for the convergence of gradient-based optimization algorithms.
clip (x/ (X))) a*)
X = 0² RMS(x)
Let us define the quantization error for each entry wg as eIr, -|w
a*
[ - و0x We can partition the weight indices & based on whether the quantization error err; is smaller or larger than some "trust factor" threshold T' Denote:
= proje. (x), where
clip(E, a)
2
= arg min RE-N(0.1) OER
$ --a a
Samall
= {k: : erry S <T;
Sturge = { k errr > 7}-
is the MSF-optimal scaling factor. If x were Gaussian- distributed, this would produce an MSE-optimal projection.
Then, the squared gradient difference in (2) decomposes as:
(V
V Lz)² + Σ (VwLs
Hadamard Preprocessing. Yer, the natural distribution of tensor values may not be Gaussian, especially given the emergence of outlier values (Dettmers et al., 2022: Nrusimha et al., 2024). To mitigate this, we add a Hadamard Transform (HTT) step before Gaussian Fitting. Thus, our for- ward pass projection becomes:
KESen
ke
&
(*N)
Assuming that the loss L is 7-smooth, the (x) "small er- ror" term would be upper bounded by ?212 S-malll- Intu- itively, this term is minimized in a standard QAT method's "distribution fitting" step. Yet, distribution fitting does not address the "large error" term (**): specifically, outlier en- tries clipped in the fitting step can lead to extremely large gradient estimation errors.
*A1 = proj HT(x).
(3)
In other words, we transform the target tensor via mul- tiplication with a Hadamard matrix of appropriate shape, applied along the matrix matttiiimultipliaation dimension, and then project to an MSF-optimal grid in the Hadamard domain.
QuEST takes this into account by balancing estimation er- rors due to minor but persistent quantization errors in (+),
3

---

Training Accurate LLMs with Low-Bit Weights and Activations
Here, we leverage 1) the fact that, roughly, multiplication of matrix with the Hadamard Transform leads the weight distribution to better match a Gaussian (Ailon & Chazelle, 2009; Suresh et al., 2017); 2) the existence of fast Hadamard multiplication kernels (Tri Dao), and 3) the fact that the HT orthogonal, so it can be easily inverted. While this HT effect has been utilized in PTQ (Tseng al., 2024; Ashk- boos et al., 2024; Malinovskii et al., 2024) and distributed optimization (Vargaftik et al., 2021; 2022), we believe we are the first to harness it for QAT.
Algorithm QuEST Training Forward
I: Input: Input activations x, row-major weight w
X HT(x)
3: Xn. = -proaa. Xa.
4: Wn = -HT(w)
5: Wn = -proj.- W/
y = Xnw,
Return: y, Xh. Who Mar (Xhi Kn). Mar (WAs Wh)
ing mask Ma- Ma- (x); onto the gradient w.r.t. quantized weights in the Hadamard domain.
3.2. Step 2: Trust Gradient Estimation
Next, we focus on the backward pass. For simplicity, we first describe the variant without the Hadamard Transform step and then integrate this component.
Gradient Effects. Notice that, in the absence of the HT or regularization effects (e.g., weight decay), the "untrusted" weights in Stagee would receive permanently removed from optimization. Yet, the addition of the HT means that the trust mask is no longer binary in the "standard" domain, allowing for gradient flow towards all model weights. We validated this effect empirically by observing that the HT reduced the final cardinality of the "untrusted" weights set Surge by z 4x, aligning it with the number of values we would expect to be outside the "trust set" at every step, for weights from a normal distribution. This is investigated in more depth in Appendix A.I.
gradient and may be
Trust Estimators for the Basic Projection. First, assume that x = proj.- (x). Since the projection operation [x], is not differentiable w.r.t. x, we need a robust way to estimate our gradient. Expressed as an operator, STE can be written as Ux R3 म| during the backward pass, allowing gradients to propagate through the network, but can lead to large errors due to components with large quantization error.
Ø 0
Specifically, the factor a", chosen to minimize the weight fitting error, acts as a natural scale for how far off their real value the majority of quantized values can be: for values below the scaling factor, this error is not larger than T To-I. the half-width of quantization interval. This gives natural bound for the (+) term in our analysis of Equation 2.
3.3. Discussion
Implementation. practice, we use identical Hadamard Transforms along the matrix-multipiication dimension for both the weights w and the activations x. Since the Hadamard Transform is unitary, the quantized matrix multi- plication output y = Xw" is aligned with the full precision output xw it approximates. The algorithm describes the forward pass over a linear layer actively quantized with QuEST for row-major weight representation.
To bound the second term (++), we choose to not trust the gradient estimations for weights with large errors (V :
E } Choosing T = -ー and masking gradients for elements in Starge we obtain the gradient operator:
a
0x ~ 1 ST O
= Ma (x; x)
0x
Ox
The algorithm 2 describes the backward pass over the same layer using the quantized weight and activations from the forward pass as well as error gradient w.r.t y. We note that, although the backward computation is performed w.r.t. the quantized weights and activations, the multiplications and gradient operands are performed in standard 16- bit preci- sion.
where I I|x is the standard indicator operator. We will refer to Ma. as the "trust mask"; this gradient estimation operator will be called the trust estimator.
Trust Estimators for the Hadamard Projection. We now interface the trust estimator with the Hadamard Transform (HT) and its inverse (IHT) to obtain the following forward scheme: X₁ = HT(x) and Rn = proj.- Xx. Then, the natural approach to perform trust estimation directly in the Hadamard domain, where quantization takes place:
Algorithm 2 QuEST Training Backward
1: Input: aL
Xr. Who Ma. (X(): Ma. (Wi:
By 2h. Wr.
8L
by
3: dx
aL
IHT
AL (Ma- (xx: 8h) O
2 ~ IHT Maa- (xa;(x); (x) 3 Jx, 0x
4: aw
aL
xT.
åL By
5: ow
aL
= IHT MMa (wa wh)
aL aw..
In other words, after deriving the trust mask w.r.t. distribu- tion fitting in the Hadamard domain, we apply the result-
6: Return: ox' ow
aL al
4

---

Training Accurate LLMs with Low-Bit Weights and Activations
weights and activations, QuEST will suffer from trust re- gions that extend out of the grid by whole a". To fix this, we reduce the size of the "outermost" trust regions, outside the clipping factor, by a scaling factor 8. Through small-scale experiments, we determined the optimal value of to be s 1.30. We use this scaling factor for all the 1-bit QuEST runs in this paper (unless stated otherwise). This modification is necessary (and leads to an improve- ment) only in the extreme 1-bit compression regime. This is discussed further in Appendix A.2.
Gradient Alignment
68-
0.6.
Trust Estimator w. Hadamard
Trust Estimater
Striight- Sitt-TTrroggh Estimator
0.4
60
0.5
1.6
1.5
2.0
255
3.0
4.0
4.5
5.0
Depth From Top, Blocks
4. Experimental Validation
Figure 2. Gradient alignment comparison for 30M Llama model
4.1. Implementation Details
after training on 2.7B tokens 8-bit precision.
Models and Hyperparameters. We tested our method on pre-training decoder-only Transformers (Vaswani, 2017) following the Llama architecture (Touvron et al., 2023), in the range of 30, 50, 100, 200, 430 and 800 million non-embedding parameters. Please see Appendix B.1 for architecture and hyper-parameter details. We trained all models on tokens from the C4 (Dodge et al., 2021) dataset, tokenized with the Ilama 2 tokenizer. We used the AdamW (Loshchilov & Hutter, 2019) optimizer with co- sine learning rate schedule and a 10% warmup period, with gradient clipping (1.0 threshold, decoupled weight decay of 0.1) We identified the learning rate optimally for a 50M FP16 model via learning-rate sweep. For other models, as standard, scale the learning rate iivers--ropportionaly to the number of non-embedding parameters. We reuse the exact learning rates for all QuEST training runs. Please see https: / git.ub....... com/ IST-DASLab/ QuEST for a reference implementation.
Training Complexity. In total, during training, for each original matrix multiplication (e.g., xw?"), we need only two Hadamard Transforms on the forward pass and two Inverse Hadamard transforms on the backward pass.
For a Transformer model (Vaswani, 2017) with d blocks and hidden dimension h, and a batch containing b tokens, the MatMul complexity of the forward pass can be estimated as: b x d x h2a Then, the asymptotic cost of the Hadamard Transform is the quantity x d x h x log h + d x h² log h, which is asymptotically negligible with 6 > log h.
Activation Effects. well-known (Choi et al., 2018) that activation quantization has major impact on training, possibly due compounding with model depth. To test the effect of different gradient estimators on backpropa- gation, we empirically examine "gradient quality" as fol- lows: we calculate intermediate gradients VarL with re- spect to activations after the 1-th Transformer block. For the same input, we disable activations quantization and calcu- late the "true" gradients V I.. We then define the "gradi- ent alignment" as the cosine similarity between gradients: L, L) = aV L, V Llla V (1)).
Unless stated otherwise, train every model on number of tokens equal to 100x its number of "free" parameters, e.g.. 10B tokens for Llama 100M model, regardless of precision. This allows us to explore the satursattration regime. We aim for comparisons that are iso-size: That is, to match the size / FLOPs of a 100M FP16 Llama model (trained on 10B parameters), we will train 4-bit weights and activations, using 40B total tokens. This allows us to explore accuracy for fixed model sizes, across compression ratios (see Figure 1). We discuss different D/N regimes in Appendix C.2.
a
While low similarity does not necessarily indicate poor gradient estimation (as the quantized forward pass might have utilized slightly different pathways, leading to discrep- ancy), high similarity clearly indicates that the estimator produces "high-quality" gradients relative to full precision. Figure compares the gradient alignment for the STE rela- tive QUEST, with and without the HT. QuEST leads to remarkably-high and well-concentrated alignment (2 0.8), even at larger depths. By contrast, standard trust estimation degrades alignment with depth but has good concentration, whereas the STE has poor alignment and high variance.
model with
4.2. Comparison to Prior QAT Methods
We compare QuEST STE; LSQ (Esser et al., 2019). widely used QAT baseline; QAT extension of QuaRot (Ashkboos et al., 2024), a method similar to QuEST but with AbsMax scaling instead of proper distribution matching; and AdaBin (Tu et al., 2022), a specialized WIAI training method. The results, presented in Table I, indicate that QuEST outperform all existing methods, including spe-
The 1-bit Case. In our original trust estimation formulation, we proposed to set the trust factor as half the quantization interval, T = a' 2=_T- Thus, the trust regions increase expo- nentially as the bitwidth decreases. In particular, for 1-bit
5

---

Training Accurate LLMs with Low-Bit Weights and Activations
Model size
Method
W4A4
W3A3
W2A2
W1A1
P
1
2
3
4
8
16
30M
STE
3.792
4.449
4.793
5.256
QuEST
0.02
0.16
0.43
0.70
1.02
1.00
QuaRot
3.338
3.612
4.481
4.932
LSQ
0.02
0.12
0.32
0.56
0.87
1.00
LSQ
3.315
3,410
3.598
3,991
AdaBin
3,988
QuEST
3.272
3.372
3.574
3.945
Table 2. Fitted scaling-law parameter efficiencies PP)
50M
STE
4.040
4.542
5.162
6.867
QuaRot
3,201
3.695
4.566
5,007
LSQ
3.240
3,290
3.501
3.862
AdaBin QuEST 3.135
3.843
3.226
3.441
3.791
2.5
2.0
Table C4 validation loss comparison across bit-widths and model sizes for STE, QAT extension of QuaRot, LSQ, AdaBin and QuEST. AdaBin is only defined in the binary case.
eff(P) 16
1.5
QuEST INT
1.0
QuEST 2:4 INT4
cialized ones, across all tested bitwidths. We perform more elaborate numerical comparison in the next section.
QuEST FP4
LSQ INT
STE INT
6.0
4.3. Scaling Laws
Background. Hoffmann et al. (2022) proposed to model loss scaling as a function of the number of parameters in the model N and the number of tokens D was trained on, in the form of parametric function:
Figure 3. Illustration of the efficiency factors eff( P)/P, arising from analysis, for different numerical precisions P, formats (INT, FP, INT+sparse) and methods. Higher is better. QuEST INT4 appears have the highest efficiency.
A
B D E.
L(N,
(4)
of our method described in Sections 5 and 4.6. Appendix Figure 12 illustrates the quality-of-fit.
No
where A, B, E, ar and 3 are the scaling law parameters that can be fit empirically. Following Frantar et al. (2025), we modify this formula assuming that the training precision p only affects the parameter count N multiplicative factor eff( P), which, for given quantization method, depends only on the training precision:
Results. The overall results were presented in Figure illustrating loss model size. First, observe that, remarkably, QuEST provides stable training down to 1-bit weights and activations, across model sizes, following a stable scaling law. Second, examining the Parcto frontier, we observe that 4-bit precision slightly superior to 3-bit, and consistently outperforms all higher precisions. Overall, these results show that QuEST can lead to stable scaling laws, which consistently improve upon prior results (Kumar et al., 2024), moving the Pareto-optimal line to around 4-bit.
A
B D
L(N,D,P) (N eff(P))
+
+E
(5)
If we take eff(16) 1.0, recover the law in Equation 4.
4.4. Finding the "Optimal" Precision
Fitting process. To estimate A, B, E. a, 8 and eff(P) for every quantization precision P we need, we fit this parametric function by minimizing the Huber loss (Huber, 1964) between the predicted and the observed log loss. Our process is detailed in the Appendix, and closely follows the setup of Hoffmaan et al. (2022), including the grid search and the loss hyper-parameters.
The Overtraining (OT) regime. The goal of a stan- dard scaling law (Equation 4) to determine the "opti- mal" model size N and training duration D under fixed pre-training compute C = GND. For instance, Hoffmann et al. (2022) estimated the "Chinchilla-optmma"" ratio to be around D/N zs 20. Yet, it now common to train (often smaller) models way beyond this ratio, effectively spend- ing additional training compute (relative to "optimal") to minimize deployment costs by executing a smaller model. For example, recent models are trained with D/N 2 1000 (Dubey et al., 2024: Team et al., 2024). With test-time com- pute (Snell al., 2024), there is an incentive increase
Specifically, we fit the model on the range of parameters P 2,, 3,4, 4, 16), N E (30, 50, 100, 200, 430, 800) x 10t and D = 100 x N. The resulting fit is presented Figure 1. To capture a larger range of D, we fit the model on additional runs with 10 and D/N E (25, 50). We additionally fit the extensions
(2, 3, 4). N E {30, 50, 100} x
6

---

Training Accurate LLMs with Low-Bit Weights and Activations
34-
3.6
W1A16
4.00
WIAL NOHT
W2A16
3.75
WIAI
3.4
3.2
W3A16
350
W2A2 NO
W4A16
W2A2
12-
BF16
QUEST W4A4
3,00
30-
2.8
QuEST 2:4 INT4
275-
QuEST FP4
25-
Memory, Mbit
Memory, Mbit
Momory, Mbit
Figure Additional scaling laws induced by QuEST: (a, left) compares INT, FP. and INT+sparse formats 4-bit precision, (b, middle) shows the scaling laws for weight-only quantization, where 2-bit appears to be Pareto-dominant, while (c, right) shows that trust estimation benefits significantly from Hadamard normalization.
this even further. we extrapolate and take D/N + oo, Equation takes the simplified form:
of QuEST across all bitwidths with gaps aroung 50% of baseline efficiency around the optimal precision.
4.5. Extensions Different Formats
(oT(N,P) = (N. eff(P))) +
(6)
The FP4 Format. We can use the same framework com- pare the "effective parameter count" for INT, INT sparse, and the lower-precision FP format supported by NVIDIA Blackwell (NVIDIA, 2024). QuEST can be extended to this data type by replacing the L-1 rounding operation with rounding to the FP4 grid [1] scaled fit the same [-1, 1] interval. The optimal scaling factor ofp4 would be defined by simply replacing 1-1 with [=] in the original definition. We choose the trust factor T for M (x; x) = I - as the largest half-interval of the FP4 grid.
We refer to this as the "overtraining" (OT) regime, where the training compute is less relevant, and is only bounded by factors such as the available amount of filtered training data. The focus on minimizing runtime/inference com- pute, measured for example by model latency. This problem can be formulated as finding the optimal model size N and precision P that minimizes certain runtime compute limit.
Runtime Cost Estimate. Since we focus on quantizing both weights and activations, the matrix multiplications can be performed directly in pwwer-preission, providing linear speedups in the precision P (AAdelkhhliik et al., 2022). As such, we can roughly estimate the runtime cost, up to con- number of basic operations
To determine the eff(P) parameter for FP4, we train 30, 50, 100, and 200M models with QuEST in FP4 precision and aggregate results in Figure 4(a), comparing them with the original uniform grid results. We observe that FP4 performs slightly worse than INT4. We also fit FP4 with the scaling law in Equation (5) and present the resulting eff (P) P in Figure 3 (red dot). The results show that, indeed, FP has lower parameter efficiency than INT at 4-bit precision. We hypothesize that this is correlated with the fact that, when clipping is allowed, FP4 has higher MSE than INT4 when data.
stants, as the seecision-weigheee (FLOPs) in a forward pass F NP. Then, the problem of minimizing loss while staying within certain runtime (FLOP) constraint can be re-written as:
A P)
fitting
min N.P
LNN,
E
F S F Fmax.
Extension to sparsity. QuEST can also be extended to sparsity. Then, the trust estimator will mask out sparsified elements with absolute value above the trust mask; specifi- cally, this covers the majority of sparsified elements, except for the small elements within - + 2 In practice, we still keep the whole weight matrix in full precision dur- ing training. On the forward pass, we first sparsify and then quantize. On the backward pass, we apply the trust mask as usual.
From this formulation, if we fix F < Fuaxx maximizing eM(()
P becomes the key factor that influences the "optimal" pre-training precision in the OT regime. Recall that we can estimate Pf) from the empirical scaling law (obtained in Section 4.3 and shown in Table 2). Thus, we can calculate -P for any precision. Figure 3 suggests that 4-bit ap- pears to be the optimal pre-training precision in this regime. Additionally fitting Pf for selected baselines and plot- ting them on the same figure, one can see the dominance
eM(()
Figure 4(a) illustrates the scaling law induced by the 50%
7

---

Training Accurate LLMs with Low-Bit Weights and Activations
1600 Medel
7B Model
3,5
1.75
5.0
HT
1.30
Speedup
2.0
0.75
0.50
0.5
NO
HT
QKV
Gate
Down
QKVV
Cate Up
Down
Batch Size (Sequence Length 512)
Figure 5. Per-layer speedups for QuEST INT4 vs BF16, on single RTX 4090 GPU. The results take into account quantiza- tion/dequantization costs for QuEST, and include the cost of the Hadamard transform (orange bar). present results for the 1.6B 4-bit QuEST model we trained, as well as inference speedups for proportional 7B- B--armmeetrr model.
Figure 6. End-to-end prefill speedups for QuEST INT4 BF16, across different batch sizes, using the 1.6B parameter model on a single RTX 4090 GPU. As expected, QuEST most effective for larger batch sizes, where the workload is more compute-bound.
trix multiplication on the quantized activations and weights, followed by dequantization of the result back to BF16.
sparse + INT4 of NVIDIA Ampere (Abdelkhalik et al., 2022), while Figure 3 (green dot) shows its parameter effi- ciency relative to INT and FP. With QuEST, this format can provide better scaling than FP4, but slightly inferior to INT4. (While this format is known as 2:4 sparsity, for INT4 + 2:4 requires a 4:8 mask with some additional constraints.)
For the first stage, utilize existing Hadamard ker- nel (Tri Dao). We developed a custom Triton kernel for the second stage to fuse the quantization and data formatting. This kernel computes MSE-optimal group scales and per- forms centered quantization on the activations. It also packs the INT4 elements into UINT8, with additional intermediate results prepared for matrix multiplication and dequantiza- tion. The third stage involves fused matrix multiplication and dequantization using our enhanced CUTLASS kernel. In this stage, both activations and weights are read and pro- cessed as integers to exploit the higher GPU throughput. The results are then dequantized back to BFI6 within the same kernel. We also apply CUDA Graph end-to-end to further reduce the kernel launching overhead.
4.6. Additional Experiments
Weight-only quantization. In addition to the comparison with the baseline presented in Section 4.2, we present full scaling for weight-only QuEST quantized training. We train models with 30, 50, 100, and 200 million parameters in 1,2,3, and bits in the same general setup as Figure 1. The results in Figure 4(b) show that our approach leads to stable scaling laws the weight-only case as well. Interestingly, here b-t weights appear to be Pareto-dominant, while 1-bit is surprisingly competitive with 3-bit weights.
To optimize GEMM performance, we carefully tuned the CUDA thread- and warp tile sizes and leveraged the high levels of the memory hierarchy to fuse the dequantiza- tion step before writing the results back to Global Memory in a custom CUTLASS epilogue. By performing dequan- tization at the register level, we minimize data movement, reduce GMEM memory access overhead, and minimize the number of kernel launches.
Hadamard ablation. Finally, we examine the impact of the Hadamard transform by removing it while maintaining the trust technique, described in Section 3.2. In Figure 4(c), present the results in the same setup Figure 1 for simplified trust scheme without the Hadamard Transform. Specifically, 1) training remains stable across all precisions, although WIA1 now inferior to BF16; 2) W4A4 remains Pareto-dominant, suggesting that the Hadamard transform improves the coefficients but does not alter the scaling laws.
Runtime Results. The per-layer speedups achievable using our kernel at 4-bit precision, relative 16-bit MatMuls, are illustrated in Figure 5. We provide breakdown across layers of the same shape, for 1.6B (which we have already trained), and a proportionally-scalee 7B model (which we plan to train in future work). These measurements include auxiliary overheads (e.g. quantization/dequantzaiiooo for QuEST; in addition, separate out the performance impact of the Hadamard transform.
5. GPU Execution Support for QuEST Models
Kernel Overview. Finally, we describe GPU kernel sup- port. Our forward-pass pipeline for the quantized linear layer in QuEST consists of three main stages: (1) applying the Hadamard transformation to the BF16 activations, (2) quantizing the BF16 activations into INT4 and packing them into the low-precti prepriciioon format, and (3) performing INT4 ma-
For the smaller 1.6B model, the per-layer speedups vary between 1.2x (on the smallest layers, with Hadamard) and 2.4x (largest projeepronctton layer, no Hadamard). The
8

---

Training Accurate LLMs with Low-Bit Weights and Activations
largest overhead of the Hadamard transform, of around 30%, is on the down-projection layer, which presents the largest dimension for the Hadamard. The speedups increase significantly (2.3-3.9x) when we move to the 7B-parameter model, as the MatMuls are much more expensive. Figure shows the end-to-end inference performance at 1.6B using our kernels vs. the BF16 baseline, showing speedups of 1.3-1.5x in the less memory--ount regime.
URL http::////a/r
/2088.22220022 11174.
Ailon, N. and Chazelle, B. The fast johnson johnson-lindenstrause transform and approximate nearest neighbors. SIAM Journal on Computing, 39(1)302-3222 2009. doi:
10.1137/060673096
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. Qsgd: Communication-efficci sgd via gradient quan- tization and encoding. Advances in neural information processing systems, 30, 2017.
6. Discussion and Future Work
We introduced QUEST, a new QAT method that achieves stable LLM training of in extremely low precision (down to 1 1-bit) weights and activations. Our results demonstrate that, if data and compute are appropriately scaled, 4-bit models can outperform standard- standard-precision baselines in terms of accuracy and inference cost, suggesting that the fundamental limits of low-preeiine wwwpprecision QAT much lower than previously thought. Further, our analysis provides new insights into the relationship between training precision and model efficiency, suggesting that low-precision may be good target for large- scale training runs in the overtrained regime. Third, we have shown that our approach can lead to inference speedups.
Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J.. Hoefler, T., and Alistarh, D. Towards end-to- end 4-bit inference on generative large language models. arXiv preprint arXiv:23100092259, 2023.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T.,, and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.0044 2024.
Bengio, Y., Léonard, N., and Courville, A. Estimating propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3433 2013.
Several promising directions emerge for future work. First, while we demonstrated QuEST's ' effectiveness up to 1.6B parameters, its scaling behavior for much larger models is an interesting direction we plan to pursue in future work. Second, our work focused primarily on decoder-only archi- tectures; extending QuEST to encoder-decoder models and other architectures could broaden its applicability.
Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., and Kwak, N. Lsq+: Improving low-bit quantization through learn- able offsets and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Bisk, Y., Zellers, R., Bras, R. L.,, Gao. J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https: http:://xxvvvvooo org.//ii.oor.....oooo. /bs/ 1911. 1911.11641.
Acknowledgements
The authors would like to thank Roberto Lopez Castro for key contributions to the GPU kernel implementations pre- sented in Section 5. We would also like to thank Elias Frantar for useful preliminary discussions. Further, the au- thors would like to thank Martin Jaggi for his generous support during the development of this project. This re- search was funded in part by the Austrian Science Fund (FWF) 10.55776/COE12.
Chee, J., Cai. Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. L.J.,, Srini- vasan, V., and Gopalakrishnan, K. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 1805. 2018
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https: / /aaaaaiivvv..////aaa/aaaa org/ aba/ 1803. 05457.
References
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 O): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 000018-3333,, 2022.
Abdelkhalik, H., Arafa, Y., Santhi, N., and Badawy, A.- H. Demystifying the nvidia ampere architecture through microbenchnarking and instruction-level analysis, 2022.

---

Training Accurate LLMs with Low-Bit Weights and Activations
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E.,, Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03077 2023.
Huber, P. J. Robust Estimation of Location Parameter,
The Annals of Mathemaatical Statistics, 35(1):73 101, 1964. doi: 10. 10.1114/amm//11777033333 URL https: / //doi.org
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H.. and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Dodge, J., Sap, M., Marasovié, A., Agnew, W., Ilharco, G. Groeneveld, D., Mitchell, M., and Gardner, M. Doc- umenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021. URL https: //arxiv. org/ abs/ / 2214211144.0 08758.
Jin, T. Humayun, A. I., Evci, U.. Subramanian, S., Yaz- danbakhsh, A., Alistarh, D., and Dziugaite, G. K. The journey matters: Average parameter count over pre- training unifies sparse and dense scaling laws, 2025. URL
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788, 2024.
https: / / arxii. wwwpwii.owg/......ooo//
Kaushal, A., Vaidhya, T., Mondal, A. K., Pandey, T., Bha- gat, A., and Rish, I. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint ar arXiv:2407.123277 2024.
Elfwing, S., Uchibe, E., and Doya, K. Weighid- linear units for neural network function approximation in reinforcement learning, 2017. URL https: //arxiv. org/ (tr//bs/1170 03118.
Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghu- nathan, A. Scaling laws for precision. arXiv preprint arXiv:2411.043. 2024.
Esser, S. K., McKinstry, J. I.., Bablani, D., Appuswamy, R., and Modha, S. Learned step size quantization. arXiv preprint 1xivv1902.088153 2019.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awarc weight quantization for on-device Ilm compression and accelcration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint MIXiv:2210.17323 2022.
Liu, Z., Zhao, C.,, Fedorov, I,, Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant Ilm quantization with learned rotations. arXiv preprint arXiv:2405.164400 2024.
Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., and Evei, U. Scaling laws for sparsely-connected foundation models, 2023. URL https://arxiv.ct org/ abs/ 2309 08520.
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization, 2019. URL 1 1711.05101
abs/
Frantar, E., Evei, U., Park, W., Houlsby, N., and Alistarh, D. Compression scaling laws: Unifying sparsity and quantization, 2025.
Ma, S., Wang, H., Ma, L., Wang, L., Wang. W., Huang, S., Dong, L., Wang. R., Xue, J., and Wei, F. The era of 1-bit Imms: All large language models are in 1.58 bits, 2024. URL https: arxiv. orgg abs / 12.42.17766
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K. van den Driessche, G., Damoe, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models, 2022. URL https::t///........ /ttp//aaxxiii www.i.ioorrrr.oo abs/ 2203. 15556.
Malinovskii, V., Panferov, A., Ilin, I., Guo, H. Richtárik. P., and Alistarh, D. Pushing the limits of large lan- guage model quantization via the linearity theorem. arXiv preprint arXiv:241 17525, 2024.
Nadiradze, G., Markov, I., Chatterjee, B., Kungurtsev, V., and Alistarh, D. Elastic consistency: A practical consis- tency model for distributed stochastic gradient descent. Proceedings of the AAAI Conference on Artificial In- telligence, volume 35, pp. 9037-9045, 2021.
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
10

---

Training Accurate LLMs with Low-Bit Weights and Activations
Nrusimha, A., Mishra, M., Wang, N., Alistarh, D., Panda, R., and Kim, Y. Mitigating the impact of outlier channels for language model quantization with activation regu- larization, 2024. URL https:/ http//axvvvvvoo/o www./bss//..././/. 2404. 03605.
Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y. Martinet, X., Mihaylov, T., Mishra, P., Moly- bog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M. Subra- manian, R., Tan, E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y.. Fan, A., Kambadur, M., Narang. S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: /tte:://aaiivv..i www.wwo.rgggooo. www//ogg//....oooooo... at///b/////30/ /207 09288.
NVIDIA.
brief. Nvidia blackwell architecture technical avidia. com/ 2024.
en-us-ble
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, pigh-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Tri Dao, Nikos Karampatziakis, H. C. Fast hadamard transform in cuda, with pytorch interface. URL https:
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M.. Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quipt:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04336, 2024.
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor-net: Imagenet classification using binary convo- lutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Tu, Z.., Chen, X., Ren, P., and Wang, Y. Adabin: Improving binary neural networks with adaptive binary sets, 2022. URL https:tp//// / arxiv, wwxiv.orrooooooooo orgg abs///////// 22002 08084.
Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G.,
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: /arxiv.ivvv..////i///aa////aaavv org/ abs/ 1907. 10641.
Ben-Itzhak, Y., and Mitzenmacher, M. Drive: One-bit distributed mean estimation, 2021. URL https:// arxiv. org/ abs 22011.1.... 08339.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint M.XXx:2408.03314 2024.
Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G., Ben- Itzhak, Y., and Mitzenmacher, M. Eden: Communication- efficient and robust distributed mean estimation for fed- erated learning, 2022. URL attps://arxiv.ort abs / 22 08842.
Su, J.. Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL httttttt//aaaati 21 1100... 09864.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
org/ abs/
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https: /ttp::///axivorrr/ abs 0bs/11066 03762.
Suresh, A. T., Yu, F. X., Kumar, S., and McMahan, H. B. Distributed mean estimation with limited communica- tion, 2017. URL https: //ta:///axivvi org/ /bs///b//////// /161. 00429.
Wang. H., Ma, S., Dong. L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., and Wei, F. Bitnet: Scaling 1- bit transformers for large language models. arXiv preprint arXiv:2310. 1453, 2023.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I.,, Rivière, M., Kale, M. S., Love, J. al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0888
2024. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshomn A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.,
Wang, H., Ma. S., and Wei, Bitnet a4. 8: 4-bit activations for 1-bit lms.. arXiv preprint arXiv:2411. 1.04655 2024.
Wortsman, M., Dettmers, T., Zettlemoyer, L., Morcos, A.,
Farhadi, A., and Schmidt, L. Stable and low-precision training for large-scale vision-language models. Ad- vances in Neural Information Processing Systems, 36: 10271-10298, 2023.

---

Training Accurate LLMs with Low-Bit Weights and Activations
Xi, H., Chen, Y., Zhao, K., Zheng, K., Chen, J., and Zhu, J. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403. 22022433.22222 12422, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sen- tence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhao. Y., Lin, C.-Y., Zhu, K., Yc, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. arXiv preprint arXiv:231.199102, 2023.
12

---

Training Accurate LLMs with Low-Bit Weights and Activations
A. Additional "Trust" Details
A.1. Trust Mask Analysis
For the purposes of weight trust masks interpretation, we trained a 30M model over 3B tokens (11,444 iterations at bs=512) with QuEST weights and activations quantization 8-bit with and without the Hadamard Transform (HT). We logged the trust masks every 500 iterations. Figure 7 shows the fraction of masked weights. We can that adding the HT leads to an 44 decrease in the amount of masked values, corresponding to the fraction of expected clipped weights for a standard normal distribution. We can also see that without the HTT the fraction deviates significantly from the expected fraction under the assumption of weights normality.
0.0005
Practiin of Mask Mask Elements
QuEST with HT
QuEST without HT
Expectation over M/ 1)
0.0010
0.0001
0.0000
4000
6000
10000
Iteration
Figure 7. Fraction of weights for which M.- 0 as function of number of training iterations for 30M model trained with QUEST.
Moreover, we looked at the percentage of masked elements at fixed iteration in the past, that remain masked at fixed later iteration. We plot these percentages in Figure 8. As we can see, for the run without the HT, around 69% of masked elements at iteration 6000 (roughly halfway through training) remain masked at iteration 10000 (towards the end of the training). This percentage is more than twice small for the run with the HT 30%. This implies that the HT makes masks less persistent, expected. In addition, we note that weight decay is applied on all weights (including masked ones). Thus, masked weight will slowly decay until it may "exit" the masked interval, obtaining gradient again.
Without Hadamard Transform
With Hadamard Transform
Old Mask Iteration
2016
w15
4012
037
025
025
2000
4000
19600
4000
10000
New
Musk Iteration
New Musk Itcration
Figure 8, Fraction of masked values retained from an old iteration to a new iteration for a 30M model trained with QuEST W8A8.
A.2. The 1-bit Case
To determine the optimal outer trust scaling factor s*, discussed in Section 3.3, conduct sweep s, varying the outer o size of the outermost trust regions as T = T The results for 1-bit, shown in Figure 9, indicate that s* = 1.30 for the standard QuEST setup and s* = 1.25 for the setup without the Hadamard Transform (HT), corresponding to exactly a quarter of the quantization interval.
13

---

Training Accurate LLMs with Low-Bit Weights and Activations
QuEST WIAL No HT
QuEST WIAL With HT
43
C4 Vol Loss
42-
4,0-
Figure 9. Performance of QuEST as function of the outer trust scaling factor for 30M model pretraining.
A.3. Zero-shot Evaluation of QuEST Models
To assess the effectiveness QuEST beyond perplexity, we conducted a comprehensive zero-shot evaluation on five established commonsense reasoning benchmarks: HellaSWAG (Zellers al., 2019), ARC (Fasy and Challenge) (Clark et al., 2018), PiQA (Bisk et al., 2019), and Winogrande (Sakaguchi et al., 2019). We compared multiple QuEST quantization settings against full-precision (BF16) baselines. All models were trained on 80B tokens unless otherwise noted.
Table 3 summarizes the zero-shot accuracy across these tasks. Overall, W4A4 QuEST closely matches its BF16 counterpart on HellaSWAG and PiQA, with minor degradation on ARC and Winogrande. Sparse quantization ("2:4 INT4") incurs larger drops.
Method, Model Size
HSWAG (%) f
ARC-e (%) f
ARC-c (%) f
PiQA (%) f
Winogrande (%) 1
BF16, 800M QuEST INT4, 800M
39.51
53.28
22.44
71.65
53.91
39.18
52.40
22.01
71.16
52.96
QuEST no HT INT4, 800M
38.03
52.44
22.70
71.11
51.38
QuEST 2:4 INT4, 800M
36.26
50.46
21.08
69.04
53.75
Table Zero-shot evaluation five commonsense reasoning benchmarks.
B. Additional Information about the Experimental Setup
B.1. Model Hyper-parameters
For our experiments, we chose use the Llama (Touvron et al., 2023) model as the base architecture. For the attention block, this architecture utilizes multi-head attention (Vaswani al., 2023) with rotary positional embeddings (Su et al., 2023). For the MLP block, it uses additional gate projection and SiLU (Elfwing et al., 2017) activation function. We kept the MLP intermediate dimension equal to 3/3gzl of the hidden size, padding it to 256 for increased kernel compatibility. For the AdamW optimizer, we used 31 0.90 and 32 = 0.95. We did not apply weight decay to any biases and layer normalizations, Table describes size- specifie models and optimizer
for all model sizes used in this work.
Model size
30M
50M
100M
200M
430M
800M
Num. Blocks
6
7
10
13
16
Hidden Size
640
768
1024
1280
1664
2048
Num. Attn. Heads
5
6
13
16
Learning Rate
0.0012
0.0012
0.0006
0.0003
0.00015
0.000075
Num. Tokens
3B
5B
10B
20B
43B
80B
Table 4. Hyper-parameters used for each model size.
14

---

Training Accurate LLMs with Low-Bit Weights and Activations
B.2. Training Stability and Convergence
Here we present the loss curves for BFI6, LSQ, PACT, and QuEST (ours) to analyze training stability and convergence, As shown in Figure 10(a), QuEST smoothly converges throughout training, closely tracking the BF16 baseline while consistently outperforming LSQ. Mcanwhile, PACT struggles with much higher loss, indicating poor convergence. To better highlight the differences between QuEST and LSQ in the later stages of training, Figure 10(b) focuses on steps after 1000, removing PACT for clarity. This zoomed-in view shows that QuEST maintains a consistently lower loss trajectory than LSQ, further reinforcing its superior stability and accuracy across training.
QURST
QuEST
LSQ
LSQ
PACT
29116
19166
3.6
3.4
3.2
3-
3.0
2000
4000
6300
ad00
10000
2000
4000
6005
10503
Step
Step
(a)
(b)
Figure 10. Training loss curves for a 30M model trained on 3B tokens with W4A4 bitwidth, comparing QuEST (ours), LSQ. PACT, and BF16. (a) Full training loss curves, showing that QuEST closely follows BF16 consistently outperforms LSQ. while PACT struggles with high loss. (b) Zoomed-in view of training steps after 1000, excluding PACT for clarity, highlighting that QuEST maintains lower loss than LSQ throughout training.
B.3.
Search for Baseline Methods
Weghh Deoi
Walda0o Loa
4.10
261
4.25
4.00
161
54-2
3.00
46-
3.35
262
Figure 11. Hyperparameter search for PACT on a 30M parameter model with 4-bit weights and activations, trained on 10% of the dataset. The search explores different values for learning rate scaling (LR Scale) and alpha weight decay, with validation loss indicated by the color gradient. Lower validation loss (darker colors) corresponds to hetter configurations.
To ensure fair comparisons between QuEST and prior QAT methods, we conducted hyperparameter searches for both PACT and LSQ. Given PACT's instability at lower bitwidths, we extensively tuned two key hyperparameters: weight decay and learning rate scaling for the quantization parameter a (i.e., 1). = X 17). Figure 11 shows the loss achieved across different weight decay and LR scale values.
For LSQ, we only tuned weight decay, the LSQ formulation already applies scaling internally to the gradient of making additional learning rate adjustments unnecessary. Table 5 summarizes the results of the weight decay search across 2-bit, 3-bit, and bit LSQ models, where the best persforerfoming configuration (highlighted in bold) was used for final model comparisons.
15

---

Training Accurate LLMs with Low-Bit Weights and Activations
Weight Decay
2-bit PPL.
3-bit PPL +
4-bit PPL. +
0.001
37.02
31.10
27.93
0.01
36.91
30.89
27.72
0.1
36.54
30.26
27.51
1.0
18.12
31.16
28.67
Table Weight decay hyperparameter search results for LSQ across different bitwidths of 30M model. The best-performing setting highlighted in bold.
Our hyperparameter search ensured that LSQ and PACT were tuned optimally before comparing against QuEST, leading fair evaluation of performance across all tested quantization methods.
C. Scaling Laws
C.1. Description of the Fitting Procedure
As described in Section 4.3, we closely follow the fitting procedure of Hoffmann al. (2022) for the scaling law (5) fiuting. Specifically, we copied their grid of initialization given by: a E {O., .., J. 3 {0., 0.5. 0..........,.... e e {-1., ......, I.), We also reuse their 10-3 for the Huber loss. addition, we
QuEST W3A3
QUEST W444
Loos
3.4
a E {0. (0.5..... 25), and b (0,5..... 25}.
3.2
fit the eff( P) coefficient for number of guantization schemes described below:
3.0
400
100
260
Memory,
Memory,
QuEST for (1,2,3,488)
Weight-only QuEST for P (1,2,3,4).
Figure 12. Scaling (5) and 4 QuEST with tokens/parameters ratios in (25, 50, 1003.
QuEST without the HT for P (1,2,3,4,8).
QuEST with FP4 grid.
QuEST with 2:4 INT4.
C.2. Analysis of the Transitory Data Regime
The results in Section 4.4 suggest that 4-bit training is optimal the D/N + oo regime. Here, we use the fitted scaling law (5) verify that 4 bit is also close to optimal for D/N ratios that reasonable in practice. We formulate the question follows: for fixed model size (e.g. in Gb), for which amount of compute is QuEST 4-bit the optimal precision?
QuEST INT
C4 Val Loss
Figure 14 demonstrates the (predicted) dependence of perfor- mance as function of D 16a N 7zz For BF16, this quantity becomes D/N. For other P, it ensures the same amount of training com- puted (~ D). As such, models there are compared at both the same size and the same training compute. We can that 4-bit quantization becomes optimal after it passes certain compute threshold that depends on model size. We can also see that the threshold value decreases as the model size (in Gb) grows. For 14.0Gb model (corresponding to 7B parameters in BF16), the threshold is around D/N 30, which is significantly below the amount of data that models of that size are currently trained on (see Section 4.3). For even larger models, the threshold eventually becomes less than the "hincciila-optimal" ratio of D/N x 20. This validates that the regime in which 4-bit pre-
2.70
2.65
2.
10
12
14
16
Figure 13. Comparison of different QuEST precisions P at fixed model size and training compute.
16

---

Training Accurate LLMs with Low-Bit Weights and Activations
1.6Gb Model
6.4Gb Model
39
BF16
BF16
29-
QuEST W8A8
2.5
QuEST W8A8
28.
QuEST W444
QuEST W4A4
24
C4 VSl Loss
27-
Loss
2.5
2.3
2.5
2.2
24-
2.1
23-
22
2.0
14.0Gb Model
L40.0Gb Model
BF16 QUEST QUEST W4A4
1.9
BF16
QUEST
188
QuEST W4A4
Val Loss
22-
Val Loss
2.1-
1.70
1.65
1.9
10
10'
10
10'
10
10'
10'
16
10'
10"
Figure Different QuEST precision performance as function of tokens-to-parameters ratio at fixed model memory footprint. The gray line indicates a 4-bit optimality threshold.
training is optimal can, in fact, be easily achieved in practice.
We validate this in practice by training set of models of approximately the same model size (1.6 Gb) and training compute
(30 exa-FLOP, 100B tokens for BF16 100M). The results, presented on Figure 13, show how P = 4 is optimal.
17


Reviews:
[{"review_id": "Q5pRKdDbxe", "reviewer": "Reviewer_zJfZ", "summary": "This paper explores how to improve quantization aware training.  Following recent work in post-training quantization, this paper proposes a combination of techniques it calls QuEST.  QuEST involves using a Hadamard Transform in the forward pass to improve the quantization process and introduces \"trust estimation\" to improve gradient estimation in the backward pass.  The evaluation shows QuEST improves the tradeoff between model performance at a given precision.  A new precision scaling law is fit to these results.  The evaluation also shows that at 4-bits the resulting model is faster in evaluating layers than when using BF16 on RTX 4090 hardware.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review! We address all your questions below. \n\n> I'm not convinced that the 'trust factor' approach to masking out gradients is well motivated. \n\nThank you for raising this. First, trust factor masking is motivated theoretically by directly targeting the source of error in the QAT iteration, relative to standard SGD.\n\nSpecifically, the “ideal” SGD iteration is $ x_{t+1} = x_t - \\nabla L( x_t) $. Instead, in QAT we execute $ x_{t+1} = x_t - \\nabla L( Q(x_t) ), $ where $Q(x_t)$ is quantization. The “error” between these iterations is precisely the $ \\| \\nabla L( Q(x_t) ) - \\nabla L( x_t ) \\|$ term we seek to minimize.\n\nThe theory of inconsistent SGD iterations shows that this “error” directly impacts SGD convergence: see the work of Nadiradze et al. (https://arxiv.org/pdf/2001.05918), who bounded this error by applying smoothness, and Lin et al. (https://arxiv.org/pdf/2006.07253) who investigate the same for sparse projections. In this context, our work investigates a fast heuristic for minimizing this error in the context of quantization.\n\nSecond, our practical results confirm that trust factors are key to good practical convergence.\nBesides Figure 2, we also illustrated this in Appendix Figure 10, which showed that properly tuned trust factors lead to much better loss than both clipping (s = 1) and STE (large s).\n\n> There does not seem to be sufficient evaluation to establish QuEST as a new SOTA for QAT.\n\nNote that, generally, QAT methods usually fall into two categories: \n1. General schemes that can be applied to any bitwidth (such as STE, PACT, LSQ, and QuEST).\n2. Schemes specialized to some bit-widths, e.g. binarization, such as AdaBin.\n\nFor general schemes, LSQ was SOTA: e.g., recent work from IBM & MIT (https://arxiv.org/abs/2404.03605) on outlier suppression still uses a variant of LSQ, whereas the recent work of Kumar et al. only uses STE. \nSince our work is focused on general scaling laws, we did not consider specific binarization schemes, as they do not directly “port” across bit-widths. \n\nTo fully address your concern, we have ported AdaBin to our setting and compared it with QuEST at 30-100M scale. \nRemarkably, we found that QuEST consistently outperforms AdaBin in the W1A1 setting (for which AdaBin is specifically designed). Moreover, the stripped-down QuEST *without the Hadamard* recovers the performance of AdaBin:\n\nhttps://github.com/QuEST2025/speedup/blob/main/AdaBin.md\n\nWe hope this clarifies our positioning: we believe QuEST is indeed the new SOTA for general QAT.  \n\n> The claim of improved efficiency seems somewhat orthogonal.\nThe goal of our kernels is showing that QuEST models can execute fast; this is not obvious since they require a Hadamard multiplication and dynamic clipping of the activations on the forward pass. \n\n> It was unclear to me how well optimized the baseline in Figure 6 is.\n\nThe baseline in Figure 6 is near-optimal for BF16; please see more details in our reply to Reviewer `HH36`.\n\n**Editorial comments and comments about supplementary material.**\n\nThank you for the detailed examination and useful editorial notes! We will address all these in the next revision, and add a more detailed README for the CUDA code structure.  \n\n> The use of Hadamard Transform seems similar to PTQ (e.g., QuIP#). \n\nPlease see “Regarding the Hadamard Transform (HT)” part of the reply to Reviewer `Xwq9`.\n\n> It seems incorrect to refer to \"Pareto-optimal frontier\" as a metric.\n\nPareto-optimality is defined in the introduction (page 1, col.2, l.49-50), and follows Frantar et al., ICLR24. We say that an approach X (e.g., QuEST INT4) is Pareto-superior to Y (e.g., STE INT8) if X provides better accuracy at the same model size, or, symmetrically, smaller size at the same accuracy. Figure 1 shows that QuEST INT2 is Pareto-superior to BF16 pretraining, but inferior to QuEST INT4. Thus, QuEST brings the “optimal” precision in terms of accuracy-vs-size to INT4, since no other method dominates QuEST INT4 across sizes. \n\nKumar et al. (2024) don’t use the same terminology, but their metrics are similar. E.g., in Section 4.3.2 of their paper, they solve for P*, the precision that yields minimal loss at some model size, which is the “Pareto-optimal” precision. \n\nWe used a similar setting to Kumar et al. (2024) for the scaling law study, but improved significantly on their findings in terms of optimal precision via a new method: QuEST brings down the “optimal” training precision to 4bit, down from the 7-8bit precision found to be “optimal” for STE by Kumar et al. (2024).\n\n> ...the sentence on lines 140-142 is true regardless of the smoothness of the loss function\n\nIndeed, we could obtain a bound of $T^2 |S_{small}|$ just by summing over indices $k$ in $S_{small}$. We used smoothness since if $\\gamma < 1$ we would get a better bound $\\gamma^2 T^2 |S_{small}|$. We thank the reviewer for this note and will simplify the derivation to not use smoothness.", "claims_and_evidence": "The claims of improved QAT results seem to rely mainly on a comparison to results from a five year old paper, LSQ, (Esser et al., 2019) as shown in Figure 3.   This does not seem to me to be sufficient evaluation to establish QuEST as a new SOTA for QAT.\n\nThe claim of improved efficiency seems somewhat orthogonal to the proposed approach (even if supported by empirical evaluation).  Using lower precision can be more efficient was, I thought, well known.  That said, it was unclear to me how well optimized the baseline in Figure 6 is (i.e., is it using cuDNN or the software flow described in Section 5, but on FP16?)", "methods_and_evaluation_criteria": "The benchmarks and backbone architecture employed make sense.", "theoretical_claims": "The paper does not make specific theoretical claims.", "experimental_designs_or_analyses": "One objection to the paper is lack of quantitative comparison with more recent QAT training proposals.", "relation_to_broader_scientific_literature": "The main questions in QAT are how to quantize and how to estimate and apply the gradient.  The paper is attempting to innovate on both fronts.    The use of Hadamard Transform to improve quantization proposed in the paper seems very similar to that in recent closely related works on post-training quantization (e.g., QuIP#).  The \"trust estimation\" approach may be somewhat more novel.", "essential_references_not_discussed": "I didn't see AdaBin https://arxiv.org/abs/2208.08084 (ECCV 2022) cited and I thought that work was also doing quantization aware training.\n\nI also was thinking some of the earlier works on binary neural networks ought to have been cited and discussed.  E.g., \n\nMatthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv, 2014. \n\nMatthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.\n\nMatthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv, 2016.", "other_strengths_and_weaknesses": "Section 5 provides fairly high level details of the GPU implementation.  Adding more documentation to the .zip or an appendix would make it easier to check how the provided code implements the high level algorithms presented in the paper.\n\nAdditional ablation studies would help the paper.  There is some data in Figure 5(c) showing the impact of the HT on QuEST, but this type of evaluation should be expanded.  More insight into how the 'trust factor' impacts the results could be provided through ablation studies.", "other_comments_or_suggestions": "Some of the writing could be improved.  For example:\n\n- In the abstract, what is meant by efficient?  how efficient?\n\n- Page 1: It seems incorrect to refer to \"Pareto-optimal frontier\" as a metric.  What does it mean for 8-bit precision to be pareto-optimal for QAT?   The word Pareto does not seem to appear in Kumar et al. (2024) and I'm not sure what the authors of the present articles mean.\n\n- The notation in Equation 1 seems non-standard.  Usually the argument to the rond to nearest operator is not between [-1,1].  Also, if there is some notion of quantization precision that is often typically made explicit in terms of number of quantization levels or number of bits used, but the formulation in Equation 1 seems to show neither.\n\n- Perhaps I'm missing something but to me it seems the $\\gamma$ in the sentence on lines 140-142 is irrelevant to the relationship in that sentence, which in any case would appear trivially true regardless of the smoothness of the loss function based only on the definition of $S_{small}$ on line 130.\n\n\n*** POST REBUTTAL COMMENTS ***\n\nThank you for the rebuttal response.  Unfortunately, I was unable to get to this before the too short timeline set by the conference organizers and it seems like I can no longer post a response below.  So, I'll just update the text of my review with my thoughts after your response. \n\nThanks for clarifying LSQ as SOTA and the particular taxonomic differences (reading through the Kumar  et al. 2024 reference was also helpful in this regard).\n\nI am still wondering where the 'trust factor' is implemented in your code.   If you are still able to respond (which I guess unfortunately you cannot), please point us to the files/lines.\n\nRegarding the \"inconsistent SGD iterations\" references cited, as I understood them they show a specific approximation (e.g., sparsity with certain properties) converges, but the relationship of what the process converges to with the original (local) optimum wasn't clear to me.   For example, the description of the example in Figure 2 in Lin et al. (https://arxiv.org/pdf/2006.07253) seems to make the case that gradients for the dense network point in a wrong direction when viewed from the sparse network.  So, 'trusting' those gradients might be misleading.\n\nKumar et al. (2024) in Section 4.3.2 middle plot of Figure 6, \"Empirical\", shows 6-bits doing better than 8-bit for \"Predicted\" on the leftmost plot for floating point.  I see the leftmost is for integer but the caption says \"predicted\" (I assuming from the scaling law fit) making it unclear to me whether the 8-bit optimum point would hold up in practice (i.e., \"empirically\").\n\nHowever, empirically, the results in your submission do seem to show an improvement to SOTA and going through the submission another time, I think I now get the intuition for why QuEST works, so I'm leaning towards raising my score.", "questions_for_authors": "I'm not convinced that the 'trust factor' approach to masking out gradients is well motivated.  The data in Figure 2 shows that this masking obviously makes the resulting direction closer to a full precision gradient, but this seems to counter the goal of QAT in that the idea is to perform gradient decent accounting for the noise introduced by quantization (so the optimizations paths should differ).  If you can clarify your motivation for the trust factor, and better back that up quantitatively or theoretically, that may help the paper.  Please comment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XLqPQSHXLK", "reviewer": "Reviewer_HH36", "summary": "In this paper, the authors propose QuEST, a low-bit quantization-aware training (QAT) method aimed at compressing models more accurately. Experiments demonstrate that QuEST outperforms the LSQ method in performance under various low-bit weight and activation quantization scenarios. Additionally, based on the designed INT4 kernel, the Linear computation per layer shows a better speedup ratio compared to BF16.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Detailed discussion on how its (QuEST’s) advantages manifest.\n\nThe fact that QuEST enables stable training across scales has the following implications:\n\n1. If using QuEST, INT4 is the “optimal” bit-width for weights and activations in terms of inference effectiveness, that is, the accuracy that can be obtained at a given model size. This is a strict improvement relative to STE, which was found in Kumar et al. (2024) and in our experiments to only provide competitive low-bit training at 7-8 bit weights and activations.\n\n2. As the reviewer remarked, this finding holds and transfers across all model scales: thus, future large-scale pre-training runs could leverage this technique to produce highly-accurate models with low precision.\n\n3. Our approach also enables a direct comparison between the “effectiveness” of different precisions P, namely the efficiency factor eff(P). Thus, based on runs on small models, the user can determine the “optimal” precision for a given model architecture and hardware target (which may influence the set of precisions supported).\n\nTo validate our findings at a larger scale, we trained a 1.2B-parameter model over 40B tokens (2’000 A100 GPU hours), which was the largest size we could train on our academic cluster. The accuracy findings, including C4 loss and 0-shot evaluations, confirm that our findings indeed scale to this larger model size: https://github.com/QuEST2025/speedup/blob/main/1200M.md \n\nWe are currently working towards a 3B-parameter run on 300B tokens (~40’000 GPU hours), but we hope the reviewer can appreciate that this is very computationally-intensive and does not fit within the scope of the rebuttal.\n\n> Detailed explanations of [GPU Kernel] optimizations.\n\nWe believe there may be a slight confusion here. As stated in the paper, we utilize the highly optimized CUTLASS operations for the “raw” matrix multiplications in both 16-bit (BF16) and 4-bit (INT4) precisions. Thus, these basic operations are heavily optimized by NVIDIA, the makers of the library and the hardware.  Our main focus is to reduce the inference overheads over the QuEST format: that is, quantization/dequantization, clipping and Hadamard multiplication. This is overviewed in Section 5, and we would be happy to describe it further in the discussion. \n\n> The paper only compares QuEST with BF16, which is insufficient to comprehensively evaluate the effectiveness of the method.\n\nOur speedup ratio experiments compare: the BF16 baseline, our approach (QuEST), and an idealized 4bit version which does not require the Hadamard multiplication (No HT).\n\nWe use BF16 data type in the baseline because\n\n1. Our experiments use smaller-scale Llama-3 models. The full-precision data type for Llama-3 models is BF16.\n\n1. BF16 and FP16 are common weight types in the popular open-source models (e.g., Llama, Qwen, etc.). BF16 and FP16 are equally supported by modern GPUs (e.g., Ampere and Hopper architectures). They have the same computational performance. The results for BF16 also apply to FP16.\n\nOur BF16 baseline is the official NVIDIA libraries (i.e., cuBLAS/cuDNN) that implement the GEMM routine used under-the-hood by e.g. PyTorch. These codes have been heavily optimized by NVIDIA to achieve near-optimal performance on their hardware.\n\nThe results show that: \n\n1. Our approach leads to significant inference speedups relative to the extremely competitive full-precision baseline, which is standard for all open models, especially on large matrices. Note that this occurs at the same or better accuracy (as per our results).\n\n1. The overheads of our format, including activation clipping and Hadamard transforms, are small (less than 10% on average, and 30% in the worst case).\n\n1. Our results are not surprising, since they track well with the expected speedups from low-precision GEMM for the corresponding matrix sizes.\n\nWe would be happy to run additional comparisons that the reviewer would find relevant in this context.\n\n> Tests should be conducted with different sequence lengths and batch sizes.\n\nWe agree; to address this, we conducted more tests on different sequence lengths and batch sizes. You can find the speedup results using this link https://github.com/QuEST2025/speedup/blob/main/tables.pdf\n\nIn addition, we conducted further experiments on the scalability of our CUDA kernels in isolation, by increasing the arithmetic intensity of the GEMM problems. The following table shows the speedup achieved with a fixed N = 4096 while varying the M (batch times sequence length) and K (hidden) dimensions.\n\n| m \\ k  | 2048  | 4096  | 8192  | 12288 | 16384 |\n|--------|-------|-------|-------|-------|-------|\n| 1024   | 1.84x | 2.87x | 4.31x | 5.00x | 5.55x |\n| 2048   | 1.94x | 3.23x | 4.83x | 5.58x | 6.07x |\n| 4096   | 2.17x | 3.78x | 5.44x | 5.86x | 6.27x |\n| 8192   | 2.24x | 4.04x | 5.86x | 6.22x | 6.28x |\n\nWe hope this clarifies your concerns, and are happy to continue the discussion if the reviewer finds it necessary.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes. The experimental design is not very reasonable.\nIn the speedup ratio experiments, the paper only compares QuEST with BF16, which is insufficient to comprehensively evaluate the effectiveness of the method. Additionally, tests should be conducted with different sequence lengths and batch sizes to observe the performance variations of QuEST under different scenarios. The current experimental analysis remains incomplete, and more comparative experiments are needed to support its efficiency claims.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "1. In Section 4.3, Scaling Laws are mentioned, and related experiments are conducted in Section 4.6. The stable Scaling Laws indicate that the QuEST method exhibits strong generalization capabilities across models of different scales, meaning its performance improvement patterns remain consistent across LLMs of varying sizes. This experiment may suggest that QuEST can still work effectively on larger models, but the paper does not provide a detailed discussion on how its advantages manifest in such scenarios.\n2. In Section 5, Kernel Overview, the paper mentions that the third stage utilizes an enhanced CUTLASS and optimizes GEMM operations, but it does not elaborate on the specific implementation. From an optimization perspective, this may involve the following aspects:\nGEMM kernel optimization: Potentially using more efficient INT4 computation layouts to improve Tensor Core utilization.\nMemory access optimization: Possibly reducing data movement and optimizing shared memory loading to enhance throughput.\nParallel computation optimization: Potentially improving warp-level scheduling or data flow optimization to boost computational efficiency. However, the paper does not provide detailed explanations of these optimizations, which need further clarification.\n3. In the speedup ratio experiments, the paper only compares QuEST with BF16, which is insufficient to comprehensively evaluate the effectiveness of the method. Additionally, tests should be conducted with different sequence lengths and batch sizes to observe the performance variations of QuEST under different scenarios. The current experimental analysis remains incomplete, and more comparative experiments are needed to support its efficiency claims.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. In Section 4.3, Scaling Laws are mentioned, and related experiments are conducted in Section 4.6. The stable Scaling Laws indicate that the QuEST method exhibits strong generalization capabilities across models of different scales, meaning its performance improvement patterns remain consistent across LLMs of varying sizes. This experiment may suggest that QuEST can still work effectively on larger models, but the paper does not provide a detailed discussion on how its advantages manifest in such scenarios.\n2. In Section 5, Kernel Overview, the paper mentions that the third stage utilizes an enhanced CUTLASS and optimizes GEMM operations, but it does not elaborate on the specific implementation. From an optimization perspective, this may involve the following aspects:\nGEMM kernel optimization: Potentially using more efficient INT4 computation layouts to improve Tensor Core utilization.\nMemory access optimization: Possibly reducing data movement and optimizing shared memory loading to enhance throughput.\nParallel computation optimization: Potentially improving warp-level scheduling or data flow optimization to boost computational efficiency. However, the paper does not provide detailed explanations of these optimizations, which need further clarification.\n3. In the speedup ratio experiments, the paper only compares QuEST with BF16, which is insufficient to comprehensively evaluate the effectiveness of the method. Additionally, tests should be conducted with different sequence lengths and batch sizes to observe the performance variations of QuEST under different scenarios. The current experimental analysis remains incomplete, and more comparative experiments are needed to support its efficiency claims.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFLOLmqHpW", "reviewer": "Reviewer_Xwq9", "summary": "The paper presents QuEST, a quantization-aware training (QAT) method that enables training LLMs with extremely low-precision weights and activations (both down to 1-bit). The key contributions are: (1) Hadamard normalization with MSE-optimal fitting for quantization, and (2) a trust gradient estimator that minimizes the difference between quantized and full-precision gradients. The method claims to achieve Pareto-competitive performance with FP16 while using 4-bit or lower precision.\n\n## update after rebuttal\n\nAfter reviewing the authors' rebuttal, I maintain my weak reject recommendation. While the authors made efforts to address several concerns, limitations remain:\n\n1. Scale and Evaluation:\n- The authors' testing remains limited to relatively small models (up to 800M parameters, with a mentioned 1.2B test). While they claim to be working on 3B, this still falls short of demonstrating effectiveness at production-relevant scales (7B+).\n- The authors did not address my concern about performance on reasoning tasks like GSM8k or AIME, only showing basic zero-shot evaluations on simpler tasks like HellaSwag.\n\n2. QAT vs PTQ Comparison:\n- The authors' comparison with PTQ methods at small scales (800M) is methodologically questionable, as PTQ methods typically show better performance with increased model scale due to greater parameter redundancy.\n- While they provided some results from QuaRot at 70B scale, this comparison doesn't fully address the fundamental differences between QAT and PTQ at scale.\n- Most concerning is the computational cost: modern PTQ methods can quantize 70B models in about an hour on a single A100, while the authors quote 1600 A100 hours for just training an 800M model with QuEST, and estimate 40,000 hours for a 3B model. This huge difference in resource requirements makes QuEST far less competitive compared to a PTQ+SFT flow in practical scenarios.\n\n3. Technical Novelty:\n- The use of Hadamard Transform (HT) for improving gradient flow isn't as novel as presented - though not directly equivalent, similar techniques appeared in FAT from 4 years ago.\n- QuaRot, a well known PTQ scheme, also uses Hadamard transform in both weights and activations. Thus I won't see Hadamard transform as a fundamental novelty in QuEST.\n\n4. Broader Impact:\n- Without demonstrating effectiveness on larger models and more challenging tasks (especially reasoning tasks), the practical impact of this work remains uncertain.\n- The authors' rebuttal suggests that reasoning tasks are \"outside the reach of 1B-parameter models without explicit instruction fine-tuning,\" but this sidesteps the important question of how their method performs after fine-tuning.\n\nWhile the paper presents some interesting technical contributions, these fundamental limitations in scale, evaluation breadth, novelty, and especially the prohibitive computational requirements make it difficult to recommend for acceptance at ICML. The work would benefit from more comprehensive large-scale evaluations, clearer differentiation from existing approaches, and better justification for its high computational costs versus existing PTQ solutions.\n\n## Additional post-rebuttal comments\n\nAfter further examining the materials, I would like to bring up two further issues which I believe will become viewable to the authors for improving their work:\n\n### 1. Mischaracterization of FAT vs. QuEST\n\nThe authors claim their use of Hadamard Transform (HT) is new & fundamentally different from FAT, but this is not accurate. While there are implementation differences, both methods transform weight representations through frequency domains to improve quantization:\n\n- In FAT (Fig. 3 and Supplementary Section 2.2), gradient flow explicitly passes through the Fourier domain during backpropagation with ∂Wt/∂W being a function of frequency components\n- Similarly, QuEST employs the Hadamard domain for gradient flow, and their \"trust estimator\" also operates in this transformed space\n- Both approaches use masking technique and both aim to achieve the same goal: improving gradient estimation by leveraging frequency-based transformations\n\nThe core innovation of \"transforming to a more quantization-friendly domain\" is shared between these approaches, though applied in different contexts.\n\n### 2. Critical limitations in practical applicability\n\nThe authors are encouraged to check out the recent \"Quantization Hurts Reasoning\" paper (https://arxiv.org/abs/2504.04823) which demonstrates why evaluating reasoning capability at low bitwidth is crucial for modern LLMs. This paper shows:\n\n- Severe degradation in complex reasoning tasks (e.g., AIME) when quantizing below 8 bits\n- Larger models (32B-70B) fare significantly better than smaller ones under aggressive quantization\n- Models' origins (distilled vs. RL-trained) substantially impact quantization tolerance\n\nQuEST's failure to demonstrate effectiveness beyond toy-scale models (800M-1.2B) leaves its practicality questionable for the most important use case: enabling efficient inference of large reasoning models (7B+). The computational cost of QuEST training (40,000 GPU hours estimated for just a 3B model) presents a prohibitive barrier to practical adoption.\n\nThis makes QuEST's contribution largely theoretical rather than practical, especially when PTQ methods can quantize 70B models in hours (my experience is <1hr) with acceptable reasoning performance degradation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why is the method named QuEST?\n\nWe apologize for the omission, QuEST stands for **Qu**antized, **E**fficient, and **S**table **T**raining. \n\n> Can you explain the overlapping W3A3 and W4A4 curves in Figure 1?\n\nThe decrease in training precision corresponds to fixed parameter count models shifting to the left. The performance degradation due to increased compression increases the loss and shifts the points upward. Between W3A3 and W4A4, those effects cancel out, landing the points on seemingly “the same” loss-to-model-size curve. In Section 4.4, we presented the “precision efficiency (eff(P))” metric to quantify this trade-off and accurately determine the optimal precision to be W4A4.\n\n> What challenges do you anticipate in scaling to larger models (7B+)?\n\nWe did not observe any training instabilities or unexpected performance degradation when scaling the model size. As such, the main challenge we anticipate is the need to commit hundreds of thousands of GPU-hours to test this novel approach for a production-scale pre-training run.\n\n> Comparison with recent works like ARB-LLM, QuaRot, and Kumar et al. (2024).\n\nWe did not include a comparison to PTQ methods (QuaRot or ARB-LLM) since they are not competitive with QAT: many PTQ methods work in one-shot, whereas QAT methods perform training or re-training by definition. \n\nTo fully address this concern, below we present a comparison of C4 validation PPL of the 800M models trained by us using BF16, QuEST INT4, and QuEST INT8. We then show numbers obtained relative to QuaRot PTQ (as suggested by the reviewer), and round-to-nearest quantization (RTN):\n\n\n| BF16  | QuEST W4A4 | RTN W4A4 | QuaRot W4A4 | RTN W8A8 | QuaRot W8A8 |\n|------|------------|-----------|-------------|-----------|------------|\n| 11.72  | 12.12               | 53.73         | 46.85               | 12.60          | 12.59               |\n\nThese results clearly show that PTQ isn’t competitive with QAT. Specifically, notice that our W4A4 model is _more accurate_ than the QuaRot model in W8A8. \nTo your question, ARB-LLM focuses on the simpler problem of weight-only quantization which is not the main focus of our work, whereas Feng et al. (2024) focuses on PTQ. As such, we largely omitted these comparisons from the paper.\n\nWe will provide additional background citations and clarification on this topic in the next version of the paper. \n\nPlease see the answer to [Reviewer HH36](https://openreview.net/forum?id=I0Ux2nAN6u&noteId=h10Gvw440m) for the relationship with Kumar et al. (2024). \n\n> Experiments only cover small models (30M-800M parameters). \n\nNotably, the largest models we trained (800M) closely match the size of the _Meta Llama-3.2-1B_ model (up to a smaller embedding layer). This puts us at the lower end of current standards in terms of model sizes. Importantly, our study considers slightly larger models than the concurrent work of Kumar et al. (2024). \n\nRuns on the 800M model require around \\~1600 A100 GPU hours, with the total experimental cost of our submission being of around 12000 GPU hours. This makes it hard to scale further in an academic setup. Nevertheless, we have confirmed out results at 1.2B-parameter scale in the answer to [Reviewer HH36](https://openreview.net/forum?id=I0Ux2nAN6u&noteId=h10Gvw440m), and are working towards a 3B-parameter run on 300B tokens (\\~40,000 A100 GPU hours). \n\nWe hope the reviewer can appreciate that this is very computationally-intensive and does not fit within the scope of the rebuttal.\n\n> Testing is limited to C4 dataset only, making all conclusions restrictive, e.g., lacking reasoning tasks like GSM8k or AIME performance.\n\nPlease note that our testing is not limited to C4: Appendix A.3 included the 0-shot evaluations of some of the models. To better address this issue, we present additional 0-shot evaluations (HellaSwag, ARC-Challenge, ARC-Easy, PiQA, Winogrande) of a broader set of models we trained: \n\nhttps://github.com/QuEST2025/speedup/blob/main/zero-shots.md\n\nThese results are again consistent with the C4 evaluations. As for the reasoning tasks like GSM8k or AIME, they are outside the reach of 1B-parameter models without an explicit instruction fine-tuning phase.\n\n> The success of the Hadamard transform in PTQ makes HT's effectiveness in QAT unsurprising.\n\n**Regarding the Hadamard Transform (HT):** We emphasize that the context in which we analyze HT is different from all the mentioned PTQ methods. Specifically, PTQ methods utilize the Hadamard transform to mitigate outliers in model weights and activations, and obtain better quantization grid fit. \nIn addition to this, we show novel effects of HT on:\n 1. Improving gradient estimator alignment, as discussed in Section 3.2 (line 205)\n 2. Circumventing the “dead weight problem” in gradient masking, as discussed in Section 3.3 (line 215) and Appendix A.1, by making sure that all weights get gradient. \n\nThese effects are unique to QAT; to the best of our knowledge, we are the first to explore them.", "claims_and_evidence": "The claims are partially supported by evidence, but with several limitations:\n\n- The experiments only cover relatively small models (30M-800M parameters). The authors didn't provide justification of not going to higher sizes.\n- Testing is limited to C4 dataset only, making all conclusions restrictive, e.g., lacking reasoning tasks like GSM8k or AIME performance\n- Improvements over baselines (PACT, LSQ) are modest (e.g., ~2.6 point improvement in W1A1 configuration)\n- The paper claims GPU speedups for 7B models but only projects them without actual training", "methods_and_evaluation_criteria": "The methods are sound but narrow in scope:\n\n- Use of C4 dataset alone is limiting\n- Model sizes tested (up to 800M) are not aligned with current community standards\n- Baseline comparisons (PACT, LSQ from 2018-2020) are quite outdated\n- Primary metrics (perplexity, validation loss) need broader validation", "theoretical_claims": "The theoretical framework appears sound, but several clarifications are needed:\n\n- The derivation of trust estimation is mathematically correct\n- The paper should clarify if α in equation (1) is per-output-channel scaling\n- The relationship between HT and gradient estimation could be better explained", "experimental_designs_or_analyses": "Experimental results appear valid. Can you explain why in Fig. 1 the W3A3 and W4A4 curves overlap?", "relation_to_broader_scientific_literature": "Several critical omissions in literature discussion:\n\n- Should discuss relationship to recent work on precision scaling laws (Kumar et al., 2024)\n- Should compare against recent work on numerical precision effects (Feng et al., 2024)\n- Missing comparison with state-of-the-art PTQ work like ARB-LLM\n- The meaning of \"QuEST\" is never explained\n\nSee below for more details.", "essential_references_not_discussed": "Several critical missing references and comparisons:\n\n1. Recent theoretical works on precision:\n- \"Scaling Laws for Precision\" (https://arxiv.org/abs/2411.04330) - Shows fundamental relationships between model precision and performance\n- \"How Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs\" (https://arxiv.org/abs/2410.13857) - Demonstrates theoretical limitations of low-precision training\n\n2. Recent PTQ advances:\n- \"ARB-LLM: Alternating Refined Binarizations for Large Language Models\" (https://github.com/zhitengli/arb-llm) - Achieves strong results with binary weights\n- \"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs\" (https://arxiv.org/abs/2404.00456) - Successfully applies Hadamard transform in PTQ setting\n\nThe omission of these works is particularly concerning because:\n\n- The use of Hadamard transform (HT) in QuEST is presented as novel, but QuaRot has already demonstrated its effectiveness in the more challenging PTQ setting, achieving strong results on much larger models (LLaMA2-70B). Their success in PTQ makes HT's effectiveness in QAT unsurprising.\n- Recent developments in HT optimization (as shown in PyTorch's HadaCore) demonstrate that the technique is becoming standard in quantization workflows. The paper should acknowledge this broader context.\n- The theoretical framework from Kumar et al. and Feng et al. provides important context about fundamental precision-performance trade-offs that should inform any QAT method.", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel integration of Hadamard Transform in QAT\n- Stable training achieved at extreme low precision (1-bit)\n- Practical GPU implementation with demonstrated speedups\n- Clear theoretical framework for trust estimation\n\nWeaknesses:\n\n- Limited model scale (only up to 800M parameters)\n- Narrow experimental validation (single dataset)\n- Marginal improvements over baselines\n- Some figures need better explanation (e.g., overlapping W3A3 and W4A4 curves in Figure 1)", "other_comments_or_suggestions": "- Editorial issues (repeated definitions of QAT, STE, etc., typos like \"Gaussiant\")\n\n- While the paper presents interesting ideas and achieves stable low-precision training, the limited scope of evaluation, modest improvements, and several important missing comparisons suggest the work needs substantial revision.", "questions_for_authors": "QUESTIONS FOR AUTHORS:\n\n- Why is the method named QuEST?\n- Can you explain the overlapping W3A3 and W4A4 curves in Figure 1?\n- What challenges do you anticipate in scaling to larger models (7B+)?\n- How does your method compare with recent works like ARB-LLM and QuaRot?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This is a paper on quantized model training. Reviewer Xwq9 provides a set of valid limitations for the work, and while I agree with their concerns, I am inclined to agree overall with zjfZ that this is a solid improvement. The community would be interested in this work, and any idea that can improve over QAT (which is long-established as SOTA) is something many people would like to read.
Decision: Accept (poster)