ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Authors: Utkarsh Saxena
Sayeh Sharify
Kaushik Roy
Xin Wang
Venue: ICML 2025 spotlightposter (2025)
Keywords: quantization
large language models
low precision computation
kv cache compression
efficient language models
Primary area: deep_learning->large_language_models
TL;DR: We perform weight/activation/kv cache quantization of large language models to 4-bit while keeping only 1/8 channels in 8-bit.

Abstract: Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, activation and key-value (KV) cache tensors to 4-bit without significantly degrading generalizability is challenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of-the-art. By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e.g.~8-bit, while quantizing the rest to 4-bit. Within each subspace, invariant random rotation is applied to further suppress outliers.  We show that this is a provably optimal mixed precision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of models, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks, achieving up to 33\% lower perplexity on Wikitext than the next best method SpinQuant, and upto 3X speedup over 16-bit baseline. Anonymous code repository available  at https://anonymous.4open.science/r/project-resq-2142.

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Utkarsh Saxena
Sayeh Sharify?
Kaushik Roy'
Xin Wang²
Abstract
state called KV (key-value) cache, and generation, where tokens are generated autoregressively. The prefilling stage is compute-bound, requiring trillions of floating-poin op- erations (FLOPs), whereas the generation stage is memory- bound due to iterative accesses and updates of the KV cache. These high computational costs are further amplified by modern LLMs' large sizes some exceeding 400 billion parameters and the increasingly long context lengths that necessitates large KV caches.
Post-training quantization (PTQ) of large lan- guage models (LLMs) holds the promise in re- ducing the prohibitive computational cost at infer- ence time. Quantization of all weight, activation and key-value (KV) cache tensors 4-bit without significantly degrading generalizability is chal- lenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of of-t By means of principal component analysis (PCA), it identifies low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are high- est, and keep the coefficients within this subspace in high precision, e.g. 8-bit, while quantizing the rest to 4-tit. Within each subspace, invariant ran- dom rotation is applied to further suppress outliers. We show that this a provably optimal mixed pre- cision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of mod- els, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on variety of benchmarks, achieving up to 33% lower perplexity on Wikitext than the next best method SpinQuant, and upto 5x speedup over 16-bit baseline. Code is available here.
Quantization algorithms are powerful and principled ap- proaches to address the immense computational demands of LLMs at both stages inference. Quantization of weights reduces parameter storage, KV cache quantization lowers memory usage of KV cache during generation, whereas activation quantization decreases the complexity of floating- point operation. However, effective low-precision quanti- zation is difficult due to large outliers in activations, which can be ~ 20 larger than other values (Dettmers et al., 2022). While post-training methods like KIVI (Liu et al., 2024c) and KVQuant (Hooper et al., 2024) achieve 2-bit KV cache quantization, and techniques like GPTQ (Frantar et al., 2023) and AWQ (Lin et al., 2024c) optimize very low-precision weights, quantizing activations below 8-bit precision remains an open challenge.
Recent LLM activation quantization methods feature two useful strategies: differential treatment of outliers retain out- lier channels high precision, leading to mixeddpreeisiee quantization (e.g.. Dettmers al. 2022: Zhao et al. 2024; Ashkboos et al. 2024b; Figure la), whereas invariant ran- dom rotation suppress outliers, leading to less difficult uniform low-precision quantization (e.g.. Ashkboos et al. 2024c; Liu et al. 2025; Figure lb). Both reduce quantization error and improve 1d,e) locally; yet a notable model performance gap persists from the 16-bit baseline. For example, Spin (Liu et al., 2025) at 4-bit, applied to Meta ama-3 3 88 (Meta, 2024b), exhibits 20% higher perplexity than the 16-bit floating point baseline, even after nontrivial optimization.
1. Introduction
Growing capabilities of large language models (LLMs) come with an increasing computational cost at inference time. LLM inference has two distinct stages: prefilling, which processes the input prompt and populates the internal
cotiull-o-quanizaattin noise ratio (Figure
Deparrment of Electrical and Computer Engineering, Purdue University, West Lafayette, USA 'd-Matrix, Santa Clara, USA. Correspondence to: Utkarsh Saxena <saxenau @ www.ueeuuuuuuuu>
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To mend this gap, we introduce ResQ, a novel PTQ method that combines the strengths of both aforementioned strate- gies and thereby improve model efficiency with aggressive bit quantization of activation, weight, and KV cache. Specif-
'tttps:/ //ttbbbb../tbt/t////// wtt..aaaaaaan/ppoeceeccce
esg

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Activation
Activation X
Quantized ctivation X
RTN
Hadamard Not ResQ
Q
Quantizer
Outlier
Outher+Hot.
(a) Outler channel detection and mixed mixed-precisionn precision quantization
Quairzttoo
Activation X
ctivation XR
Quantizee Activation X
10
Q Quintizer
Activation
10
25
30
[d)
Quantization SNR Atm. Layers
(b) Random rotation and uniform- prroorn-eeecsoo quantization
Quertizaion
16
Activation X
U=P R 0
Activation XU
Quantized Activation X
14
R2
Quantizer
A Artivattion projection
25 30
(c) Static subspace projection, random rotation and mixed-pin sexeedprecision quantization (ours)
(e) Quantization SNR FFN Layers
Figure I: (a)-(c) Different approaches to quantization including ResQ. Symbol sizes represent magnitudes of values and colors indicate precisions of quantization (blue: low precision, orange: high precision). (d)-(c) Quantization SNR comparison of ResQ with other baselines.
ically, by means of principal component analysis (PCA), we first identify low-rank subspace that captures highest vari- ances in activation, and mark the coefficients along this subspace for high-precision (8-bit) and the complement sub- space for low-precision (4-bit) quantization. Then, ResQ employs invariant random rotations within each subspace be- fore quantization to further suppress outliers (Figure Ic,d,e). We prove that the above treatment minimizes quantization error. Similar to SpinQuant, most projection matrices can be fused into adjacent weights, leading minimal runtime computational overhead (Section 4.3). Furthermore, ResQ can be applied to KV cache quantization as well, and can be combined with GPTQ (Frantar et al., 2023), resulting in even better generalizing L.M
We propose ResQ, a mixed precision weight, activa- tion, and KV cache quantization method by keeping low-rank, high-variance components in high precision, in combination with random ination-induced outlier suppression.
We theoretically analyze the projection matrices in ResQ and show that using PCA-based projections min- imizes quantization error.
We conduct extensive experiments on various models and language tasks and show that ResQ outperforms related state-of-the-art approaches.
4. We develop CUDA kernels and achieve runtime speedup on NVIDIA GPUs with our quantized models.
Outlier-based and rotation-based quantization methods can be combined. For example, high-precision outliers could be detected by -nom similar to QUIK (Ashkboos al., 2024b), and random rotations applied within both high- and low-precision quantization groups, in QuaRot (Ashkboos et al.. 2024c). These methods fare less well than ResQ (Figure 1d,e) in practice, in support of ResQ's provably optimal treatment of outlier quantization. When quantiz- ing weight, activation and KV cache to 4-bit with only I/s channels in 8-bit, ResQ achieves 4-33% lower perplexity on Wikitext and 0.1-5.4% 0-shot accuracy improvements over SpinQuant (Liu et al., 2025), the best in practice so far. Unlike SpinQuant, ResQ does not require -aadiett-based optimization, making a less demanding and faster PTQ solution. Furthermore, tuning the rank r of ResQ gives rise to Pareto-optimal solutions as a tradeoff between efficiency and accuracy. We claim the following contributions.
2. Prior Work
2.1. Quantization of LLMs
Quantization reduces model size and accelerates inference by lowering neural network bit precision (Choi et al., 2018; Hubara et al.. 2021; Yao et al., 2022; Gholami et al., 2022; Xi et al., 2023; Park et al., 2024). is broadly catego- rized into two categories: uniform precision quantization (UPQ) and mixed precision quantization (MPQ). Uniform precision quantization (UPQ) applies the same bit-width across all layers, simplifying implementation but neglecting layer-specific sensitivity to quantization. Weight-only UPQ methods reduce storage by compressing weights, using tech- niques like Hessian- -aiandd rounding (GPTQ, Frantar et al. 2023), adaptive rounding (QuIP, Chee et al. 2023), channel- wise scaling (AWQ. Lin et 2024c), and multi multi-codebook
2

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
quantization (AQLM, Egiazarian et al. 2024). However, these methods struggle with batch processing due sig- nificant activation memory overhead. Weight-activation UPQ compresses both weights and activations to address this. Methods such as SmoothQuant (Xiao et al., 2023) and OmniQuant (Shao et al., 2024) scale activations and weights to handle outliers, while RPTQ (Yuan et al., 2023a), QLLM (Liu al.. 2024a), and QServe (Lin et al.. 2025) em- play channel-level strategies like clustering and reordering. Rotation-based methods such as QuaRot (Ashkboos al., 2024c), SpinQuant (Liu et al., 2025) and DuQuant (Lin et al., 2024b) further enhance robustness in low oowprecision scenar- ios. KV cache UPQ reduces memory for large batches or long contexts. FlexGen (Sheng et al., 2023) employs 4-bit quantization and memory offloading, while KIVI (Liu et al., 2024c) uses asymmetric 2-bit quantization for compression, enabling efficient inference.
decomposition method that incorporates activation distri- butions into weight decomposition. Additionally, low-rank decomposition can be applied to reduce KV cache size, For example, Eigen Attention (Saxena et al., 2024) and ASVD (Yuan et al., 2023b) employ low-rank approxima- tions to reduce memory usage in KV caches during attention operations. PALU (Chang et al., 2024) introduces learnable projections to adaptively compress KV caches based on the compression budget. Finally, Matryoshka KV Cache (Lin et al., 2024a) refines this with hierarchical orthogonal pro- jections and knowledge distillation.
3. Quantization
Quantization of weight, activation or KV cache involves converting component elements to low precision so that they can be represented using fewer bits for more efficient compute and storage. The N-bit integer quantization and dequantization process on matrix X is given as
Mixed precision quantization (MPQ) optimizes bit- widths by adapting to the sensitivity of weights and ac- tivations, achieving better accuracy than UPQ at similar compression rates. Our proposed method, ResQ. follows the MPQ approach. Weight-only MPQ has advanced ef- ficiency for memory-bound applications with minimal ac- tivation demands. Methods like OWQ (Lee et al., 2024) and SpQR (Dettmers et al., 2024) mitigate activation out- liers' impact by retaining critical features in full precision, while Squeezel.LM (Kim et al., 2024) employs Dense-and- Sparse decomposition to efficiently store sensitive weights. Weight-activation MPQ enhances efficiency by address- ing activation outliers (e.g. (Guan et al., 2024; Zeng al., 2025)). Methods like LLM.int8( (Dettmers et al., 2022) and QUIK (Ashkboos et al., 2024b) preserve critical acti- vations with mixed or low-precision decompositions, while Atom (Zhao et al., 2024) and SliM-LLM (Huang et al., 2024) optimize quantization through channel reordering and sideenee-drven bit allocation. KV cache MPQ reduces memory usage while preserving precision for critical tokens using techniques like non-uniform quantization, importance- aware precision, and salient token compression (Hooper et al., 2024; Yang et al., 2024b; Dong et al., 2024; He et al., 2024). Alternatively, GEAR quantizes all tokens' KV cache and maintains low-rank quantization error (Kang al., 2024).
X 2x
QN Qx(X) =
Sx 2X,
(1)
SX
where [-] is round-and-clip function; xx and zx the scale and zero-point; zx = 0, sx
max( for symmetric 2
quantization or 2x = min(X), 8x = max (aax(X)=x -minn for asymmetric quantization.
2
4. ResQ
In this section, we introduce ResQ, a mixed-precision quan- tization approach that projects weights, activations, and the KV cache into an orthogonal space, retaining the low-rank components in high precision (8-bit) and the rest in low precision. We describe the quantization scheme, the genera- tion of the basis space, provide theoretical guarantees, and outline end-to-end LLM inference deployment procedure.
4.1. Quantization Scheme
Given input activation X E Rgdd and weight W E gaxd they are first projected onto an orthogonal basis defined by the vectors U E Rddd The coefficients of the projections along this basis are then subject to quantization. We seek to quantize some coefficients along certain bases at high precision while those remaining at low precision. Within Rd denote bases of a low-rank space of high-precision com- ponents by Un E kd and those of its complementary subspace of low-precision components by U, E (d-r), The rank r controls the amount of components in high pre- cision (in practice we typically choose r = d/s). We have U, + U,U,T = UUT T I because U is orthogonal. The quantized activation Xg is thusly
2.2. Low-rank Decomposition
Low-rank decomposition techniques have been widely used in model compression, reducing dimensionality while main- taining performance. For instance, SliceGPT (Ashkboos et al., 2024a) projects weight matrices onto principal com- ponents for sparsification, while ESPACE (Sakr& & Khailany, 2024) reduces activation dimensionality via pre-calibrated projections, achieving inference- time efficiency. Similarly, ASVD (Yuan et al., 2023b) introduces an activation-aware
X, = Q(XU) = (XU,) Qn (XUN).
(2)
3

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Qu(U[W)
XP
INT3
XU
NN
W
QL(X))
5.
INT4
INT4
0.
Low precision matmul (4-bit)
High precision
10
-20
-10
matmul (8-bit)
2000
4000
2000
4000
2000
4000
Charnel (a)
Charnel
Channel
Figure 2: operands
Matrix multiplication with mixed precision
(b)
(c)
Figure Activation distribution of the baseline and apply- ing the projection matrices.
Similarly, quantized weights W, is obtained by project- ing the inputs space of weights by UT and quantizing the coefficients,
To determine P. we minimize the activation quantization error || - X,I1r. For activations quantized according to Equation have,
(U, W) QU, w)
W₂ - QUU w)
| -X-x,ll - = (XUUIQQ((UIII)) + +|XUU - Q
And the output of the layer is,
(6)
X, W - QUU W) (U((XU.)Qn(U W).
(4)
Theorem 4.2. For any matrix X quantized to X, according to method described in Equation 2, assuming the values to quantized in X are normally distributed, we have
We make two observations due to orthogonality. First, the introduction of the projections do not alter the output of the model at infinite precision. This means that, quantization operation is removed from Equation 4, the layer output is numerically invariant. Second, multiplication between low- and high-precision components vanishes (Figure 2). This is efficient because only hardware kernels for quantized GEMM between operands of same precision are required.
VTlog(d
E B X,IIF S
EE X
26-1
Tlog(d -r)
VTllor
2b-1 -1
2H-1-1
E P
4.2. Projections and Optimality Thereof
Full proof of Theorem 4.2 is in Appendix Theorem 4.2 bounds the quantization error in Equation from above. To lower this upper bound of quantization error is thusly to maximize || Px|l F which happens when PA. comprises of eigenvectors of the covariance matrix XX with its largest eigenvalues. Therefore, the low-rank subspace for high- precision quantization can be obtained by means PCA, while the subspace for low-precision quantization can ob- tained using U, + U,U, P,P,, P.P." I (be- cause R& orthogonal). If we construct P by taking eigen- vectors of xxx arranged in increasing order of eigenval- ues, the last columns of such P would correspond to Pa, and the first d - r columns would correspond to P. The distribution of activation after applying different projec- tion matrices shown Figure 3. Projection of activation along P sorts the activation coefficients in increasing order of variance due to increasing eigenvalues of bases vectors. Consequently, the later channels of the projected acti- vations with higher variance are kept in higher precision. Projection along U = PR smoothes the activations along low precision and high precision groups further reducing quantization error (Figure 3) and improving quantization SNR (Figure 1(d,e)).
Intuitively, the orthogonal basis vectors U should have two properties: (1) the low-rank space for high-precision quanti- zation should capture the more important components, and (2) quantization error in both high- and low-precision groups should be minimized. We construct U as a combination of two rotation matrices serving both objectives respectively. We write U, P.R,, i (h, Therefore.
R O
0 R
- PM
(5)
where, Pi, R. E [dd(d-r), Ph, Rs Rddr, Inspired by prior work (Ashkboos al., 2024c: Chee et al., 2023), we make R2 R2 random orthogonal matrices because random rotation reduces outliers, making the rotated matrices easier to quantize. Furthermore, projection with a random orthogo- nal matrix increases Gaussianity of activations and weights within high- and low-precision groups, due to Lemma 4.1, conducive to the quantizations applied to these groups.
Lemma 4.1. By Central Limit Theorem, the distribution after multiplication with random orthogonal matrix is ap- proximately Gaussian (Tseng et al., 2024).

---

ResQ: Mixed-Preeiste Quantization of Large Language Models with Low-Rank Residuals
W embed
U
Altention Black
UA
Feed Forward Block
U
U¹
head
(b)
W
RaPE
Cc
proj
Softmax
U
W
Act
down proj
o_ o_pro]
U%
W x
RoPF
Uc
U
U
W
U
U
W O
U
v. T_prog
UX
W
U/
Wy
Up
ga to proppro
Mixed
Uniform precision quant
Merged and mixed precision weights
Mergod uniform precision weights
On-the-fly Hadamard projection
fly
Merged projections
precision
projectioon
quant
Figure 4: Model inference with ResQ incorporating the projection matrices. (a) UA modifies the inputs across blocks enabling better quantization. (b) UB, Uc enables mixed precision quantization of KV cache. (c) Up projects the activations and weights of down.proj layer.
4.3. Inference Computation with Optimized Projections
to project both the query and key using the same projection matrix Uc. The attention dot product remains invariant under projected inputs,
Once the projection matrices are obtained, the operation in Equation 4 requires multiplying the weights and acti- vations with U. Weights can be projected and quantized offline. The projection operation on an activation can be merged to the weight of previous linear layer. Based on the architecture of decoder based LLMs, we introduce four different kinds of projections (Figure 4) : U E IR dx xd,, UB. Uc E Re x xdeed, UD E RRders ddess where dy is hid- den dimension of LLM, dnead is the attention head dimen- sion and drex the hidden dimension of feedforward network (FFN).
Qproj qproj K (QUO)(UKK)= -
(8)
where and K are query and key after rotary embedding (RoPE), respectively. Because Uc cannot be merged into the previous linear layer due to presence of RoPE, the pro- jection is explicitly computed runtime, but made more efficient by applying uniform precision quantization Uc and corresponding input activations.
Projections within the feedforward block Un ensure im- proved quantization of activation within FFNs (Figure 4c), UJ is left-multiplied with weights of down_pro J, but due to the presence of activation functions within the block, UD cannot be merged to weights of preceding linear layers and is computed runtime. UD applied to the hidden di- mension of the FFNs (dres) which is typically 3x to 4x the embedding dimension in most LLMs. In this scenario, matrix multiplication with UD is extremely expensive in computation and storage. To minimize the overhead, we choose Un to be a hadamard matrix to leverage fast and ef- ficient hadamard transform kernel. And, we choose weights and activations for down.proo layer be uniformly quan- tized to low precision.
Projections at block boundaries Input activations to attention and FFN are projected via UA. Projection is handled by right-multiplying the weight matrix of fi- nal linear layer in each block (o-proj in attention and down_proj in FFN) by UA. Thus, projections of ac- tivations is handled at no additional inference cost. To maintain numerical invariance, the first linear layer of each block (q-proj k-proj vv-jroj in attention and up proj gate e proj in FFN) is pre multiplied with UJ (Figure 4a). Similarly, the weights of the embedding layer and the final head are modified to manage projection of the residual stream.
Projections within the attention block UB, Uc ensures that activations within attention block are projected (Fig- ure 4b). Post-multiplication of value projection layer by Un ensures that value vectors in KV cache are projected and quantized optimally. Consequently, the weights of j_proj layer need to be pre multiplied by UJ to ensure numerical invariance. Uc ensures that the quantization of key in KV cache is handled optimally. To achieve that, it is required
5. Experiments
5.1. Setup
Models, tasks, datasets and baselines We conduct exper- iments on Llama (Touvron et al., 2023), Llama (Meta, 2024b), and the recently released Ilama 3.2 (Meta, 2024a)
5

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table I: Comparison of perplexity score on Wikitext, average O-shot common sense reasoning accuracy and average O-shot MMLU accuracy. Results of all techniques were obtained using their official codebase. Our work ResQ and QUIK (Ashkboos et al., 2024b) keep 1/s channels in 8-bit and remaining 4-bit for W/A/KV = 4.5-bit. Other baselines are uniformly quantized W/A/KV 4-bit. All techniques except RTN employ GPTQ (Frantar al., 2023) for weight quantization. 1 higher is better, lower is better. Full results in Appendix D, Tables 10 and 11.
Met a LLaama - 3-
Met a 111 ama 70B
Family
Method
W/A/KV
Wiki (
Avg. 0-shot (1)
MMLU (T)
Wiki (I)
Avg. 0-shot (1)
MMLU (1)
16-bit bascline
16/16/16
6.1
67.1
63.1
2.9
73.1
75.9
RTN
4/4/4
218.9
39.3
23.6
452.7
45.5
23.2
GPTQ
4/4/4
166.3
39.8
23.3
11.6c3
34.9
25.5
SmoothQuant+
4/4/4
78.2
42.5
24.7
Llama
QUIK
5.5/4.5/4.5
14.2
51.6
32.7
8.0
58.2
51.1
QuaRot
4/4/4
7.8
62.1
53.2
67.6
65.3
SpinQuant
4/4/4
7.4
63.8
56.2
65.7
59.4
ResQ
4.5/4.5/4.
7.1
57.2
L1 ama-3 1amma-3.2-10 22-1B
L1 ama-3 3.2-3B
Family
Method
W/A/KV
Wiki (})
Avg, 0-shot (1)
MMLU (1)
Wiki (1)
Avg. 0-shot (1)
MMLU (1)
16-bit baseline
16/16/16
9.8
54.9
36.9
62.7
54.8
"IIN"
47474
329.1
38.1
23.8
268.8
38.7
23.7
GPTQ
4/4/4
108.9
38.0
24.9
178.3
40.3
24.8
SmoothQuant-
4/4/4
228.9
38.0
24.1
96.1
39.0
25.9
Llama
QUIK
4.5/4.5/4.5
21.8
44.3
25.1
15.8
48.8
31.1
QuaRot
4/4/4
14.3
49.0
25.5
10.1
56.1
42.0
SpinQuant
4/4/4
13.6
48.8
25.6
9.2
57.9
44,2
ResQ
4.5/4.5/4.5
12.4
50.1
29.4
8.8
59.0
49.8
Qwen2. 5-3B
Qwen2. S-72B
Family
Method
W/A/KV
Wiki ())
Avg. 0-shot (t)
MMLU (1)
Wiki (1)
Avg. 0-shot (1)
MMLU (f)
baseline
16/16/16
RIN
4747
39033.0
35.1
25.4
45412.7
34.3
24..0
GPTQ
4/4/4
9977.8
23.2
37967.2
34.5
23.3
Qwen2.5
SmoothQuant+
4/4/4
73306.7
34.8
23.9
QUIK
4.5/4.5/4.5
15.5
51.2
39.4
8.3
61.9
69.3
QuaRot
4/4/4
68.8
47.7
28.9
4.9
70.3
80.1
ResQ
5/5/
9.0
61.1
61.2
4.6
72.0
81.5
Table 2: Comparison of performance of quantization approaches generative tasks. Our work ResQ and QUIK (Ashkboos et al., 2024b) keep 1/s of channels in 8-bit and remaining in 4-bit for W/A/KV = bit. Other baselines are uniformly quantized to W/A/KV 4-bit.
GSM8K 5-shot (f)
LongBench (f)
Model
Method
W/A/KV
flexible
match
qmsum
repobench-p
16-hit baseline
16/16/16
$1.0
50.6
23.9
44.8
66.4
QUIK
4.3..
2.3
0.0
10.5
25.2
37.5
Kata-Llama L-ama-3
QuaRot
4/4/4
27.6
27.1
22.0
43.8
60.6
SpinQuant
4/4/4
29.8
29.6
23.0
43.9
62.6
ResQ)
4. 5.5/4.5/4.5
33.6
33.2
23.1
62.3
16-bit baseline
16/16/6
25.1
24.9
23.1
43.0
QUIK
3.3/4.3/4.5
2.3
15.9
.1amm-3.2-38
QuaRot
4/4/4
10.1
20.6
39.5
SpinQuant
4/4/4
11.6
ResQ
4.5/4.5/4.5
17.1
16.7
21.7
43.0
61.5
and Qwen2.5 (Yang et al., 2024a) models. We also include multi-modal language models belonging to Qwen2 VL. fam- ily (Wang et al., 2024) for our evaluations. We benchmark our approach against GPTQ (Frantar et al., 2023), QuaRot (Ashkboos al., 2024c), QUIK (Ashkboos al., 2024b), SpinQuant (Liu et al., 2025) and SmoothQuant+, stronger baseline created by combining SmoothQuant (Xiao et al., 2023) with GPTQ following Sharify et al. 2024. We evalu-
ate the quantization approaches on a range of tasks which measure the language modeling ability: perplexity on Wiki- text (Merity et al., 2017), common sense reasoning ability: average 0-shot accuracy on Arc-cle (Clark al., 2018). BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019). Openbook QA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021), language understanding: 0-shot accuracy on
6

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 3: MMMU accuracy (higher is better) of vision lan- guage models when quantized using various approaches. For 4-hit data structures, our work ResQ and QUIK (Ashk- boos al., 2024b) keep l/s of channels in 8-bit.
Table 4: Wikitext perplexity comparison of ResQ and base- line which keeps i/s channels with high 4 in 8-bit (and remaining in low precision) and uses rotation to reduce quantization error within high precision and low precision groups.
Model
WA/KV (bit)
Method
QWen2-VL
Qwen2-VL
Meta-II Perta-Llame
Llama
-2B-Iastruce
TB-Instruct
W/A/KV (bit)
Method
-3-80
-3.2-38
16/16/16
Bascline
39.6
51.6
KTN
25.0
26.7
4.5/4.5/4.5
outier+rot
7.2
9.0
4/4/4
GPTQ
27.7
24.9
ResQ
7.1
8.8
QuaRot
24.0
24.5
outlier+rot
6.6
8.3
4.5/8/4.5
4,5/4.5/4.5
QUIK
26.3
28.9
ResQ
6.5
8.2
ResQ
29.7
47.0
outliertrot
7.7
9,9
KIN
24.9
25.2
3.6/8/3.6
ResQ
7.5
9.8
4/8/4
GPTQ
23.4
24.3
outlier+rot
12.6
16.0
QuaRot
26.5
24.5
2.75/8/8
QUIK
28.4
26.4
ResQ
12.1
15.7
4.5/8/4.5
ResQ
34.0
48.8
3.6/3.6/3.6
outlicrtrot
14.7
18.7
ResQ
14.6
17.5
MMLU (Hendrycks et al., 2021), mathematical understand- ing: 5-shot GSM8K (Cobbe et al., 2021), dialogue summa- rization: samsum (Gliwa et al., 2019) and qmsum (Zhong al., 2021) from LongBench (Bai et al., 2024), code com- pletion: repobench-p (Liu al., 2024b) from LongBench, and multi-modal understanding: MMMU (Yue et al., 2024).
other baselines fail to achieve competitive results, and ResQ significantly outperforms them. Compared with QUIK, an- other mixed precision quantization approach, ResQ achieves 42-50 500 better Wikitext perplexity. 5.8-12.3% better average zero shot accuracy and 4.3-24.5% better MMLU accuracy over all models. Complete results on Llama and Qwen2.5 family of models are provided in Appendix D. Additionally, we provide comparison between baselines at W/A/KV 4/8/4 bits for Llama families in Appendix E. Among all the set of results, ResQ maintains superior performance.
Implementation details We implement ResQ using the HuggingFace Transformers library (Wolf et al., 2020) with PyTorch (Paszke et al., 2019). We share a single UA across all layers, while Up Uc and Un are generated per layer. Following SpinQuant (Liu al., 2025), we use per-token asymmetric quantization for activations, per-channel sym- metric quantization for weights, and per-head asymmetric quantization for the KV cache. We fuse the projection ma- trices UA UB, UD into weights and apply GPTQ (Frantar et al., 2023) for weight quantization. To efficiently imple- ment on-the-fly projections, Un is Hadamard matrix and Uc and its activations are quantized to 8-bit. The entire pro- cess, including obtaining projections and quantization, runs on a single NVIDIA A100 GPU; for Met BesttaLlaas LLama- -3 --BB it takes 35 minutes. Additional details are in Appendix C.
Generative tasks We also test ResQ on tasks that require auto-regressive token generation including the GSM8K mathematical understanding benchmark, dialogue summa- rization benchmarks (qmsum and samsum) and code com- pletion benchmark (repobench-p, Table 2). The goal of choosing these tasks is to evaluate the generation ability wide variety of domains. On the challenging GSM8K benchmark where QUIK fails to produce meaningful re- sults, ResQ outperforms SpinQuant by 3.8% and 5.5% on the 8B and 3B parameter model respectively, closing the gap to the 16-bit baseline. On LongBench evaluation tasks, ResQ demonstrates competitive performance and outper- forms SpinQuant without any additional training.
5.2. Main Results
Language modeling, understanding, and reasoning tasks We evaluate ResQ on tasks that test language modelling abil- ity (perplexity on Wikitext), common sense reasoning ability (average 0-shot accuracy on the eight tasks listed in section 5. and language understanding (average O-shot accuracy on MMLU). The results are presented in Table 1. We see that ResQ reduces the gap to 16-bit performance and outper- forms the quantization baselines across all tasks all mod- els. Particularly, on Llama 3/3.2 family of models, ResQ outperforms SpinQuant by achieving 4-33% lower Wikitext perplexity, 0.1.5.4% better average 0-shot accuracy and a 1-14.5% better accuracy on MMLU benchmark without any additional training. For the Qwen-2.5 model family, all
Multi-modi understanding We benchmark the quantiza- tion approaches on vision language models (VLMs) by quan- tizing Qwen2 VL family and evaluating their performance on MMMU (Table 3, Yue et al. 2024). Only the language model is quantized while the vision encoder remains 16- bit as the language model has many more parameters (over 10x for Dwenn-VVL-BBIInS. ruct). ResQ outperforms baselines on both 2B and 7B models, achieving superior accuracy and demonstrating its generalizability. Results for individual MMMU tasks are provided in Appendix F.
Comparison against outliers with rotation baseline A stronger baseline can be created combining existing quanti-
7

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 5: Comparison of perplexity score on Wikitext, aver- age 0-shot common sense reasoning accuracy and average MMLU accuracy at W/A/KV 4-bit. ResQ keeps 1/8 chan- nels with high eigen value in 6-bit, 1/s channels with low eigen value in 2-bit and the rest in 4-bit for average bit-width of 4-bit. Complete results provided in Appendix G.
formance of ResQ, the random rotation matrix R can be optimized to minimize final task loss similar SpinQuant (Liu et al., 2025) albeit at higher computational cost for quantizing the model. We keep identical training hyperpa- rameters as SpinQuant, and learn the rotation R for models upto 8B parameters. The evaluation results are provided in Table 15 Appendix H. We see upto 5% improvement in Wikitext perplexity, upto 0.6% increase in average O- shot reasoning accuracy and upto 1.1% increase in MMLU accuracy.
Model
Method
Wiki (L)
Avg. 0-shot (1)
MMLU(()
QuaRot
68.8
47.7
28.9
owen2.5-20
SpinQuant
70.6
48.6
32.8
ResQ
9.8
59.1
52.2
QuaRot
4e3
38.4
24.1
Qwen2. 5-7B
SpinQuant
3e3
38.6
24.3
ResQ
34.2
56.2
58.0
5.3. Hardware Performance
QuaRot
6.8
67.1
70.9
Qwen2.5 5-14B
SpinQuant
67.4
70.1
We implement the mixed-precision quantization using CUDA 11.8 and PyTorch. The weights are quantized into INT4 and INT8 components offline while online quanti- zation of activations into INT4 and INT8 components is handled in a single kernel call. We use CUTLASS (Thakkar et al., 2023) to perform INT4 and INTS GEMM oper- ations on TensorCore. Further, for KV cache compres- sion, implement online quantization and packing for memory efficiency. For efficient implementation of online hadamard transform involved in activation quantization in down-proj layer, we use the fast hadamard transform li- brary (fas, 2023).
ResQ)
71.3
QuaRot
67.8
77.0
Qwen2. 3-32B
SpinQuant
67.9
77.6
ResQ
5,9
69.1
77.9
QuaRot
4,9
70.3
80.1
Qwen2.5-2 5-728
ResQ
4.9
71.1
80.1
Table 6: Impact of different projections in ResQ. Evaluated by removing components and observing Wikitext perplexity.
Removed Projections
ResQ
Uc.
Up
U^
Up
Ue
Uy
Llama-2-20 2--b-hi
5.8
1550
2500
5.8
5.9
5.9
Prefill speedup On an NVIDIA RTX 3090 GPU, we achieve 1.61 3.03x speedup with ResQ over the 16-bit base- line for a single decoder block across various language models (Figure 5). Speedups are higher for larger models and shorter sequences. Compared to INT4, ResQ is only 14% slower on average, showing minimal overhead from mnixed-precisie and on-the-fly projections.
Met a a-lama-------- 3- 8B
7.1
1607
37.4
7.2
7.3
7.4
1llama3.22-33
8,8
279.2
39.0
9.0
9.2
9.4
zation approaches. Like QUIK, one can find channels which consistently contain outliers and keep them in 8-bit while keep the remaining channels in low precision. And, the quantization of high/low precision groups can be improved using random rotations introduced in QuaRot. Compared with such baseline which keeps channels with high 4x-- norm in 8-bit, ResQ's 's unique approach involves keeping coefficients along bases with high eigenvalues in 8-bit. We see in Table 4 that ResQ consistently outperforms such a strong baseline across various precisions of W/A/KV high- lighting ResQ's s PCA driven theoretically optimal approach of choosing high precision components.
Memory usage We evaluate end to end memory usage on NVIDIA RTX 3090 (24GB) for different sequence lengths in Table 7. ResQ consumes 1.84 -3.08x lower memory than the FP16 baseline. Notably, Qwen2. 5 5-14B model leads to out of memory (OOM) error while ResQ is able to support its inference upto sequence length of 8192 tokens.
Compared with the INT4 bascline QuaRot, the memory used by ResQ is 4-11% higher.
Multi GPU inference We evaluate end-to-end batched infer- ence latency GPU server with 3 NVIDIA A100 (82 GB) GPUs running - -3-1amm---70.. ResQ's weight, ac- tivation, and KV cache quantization enable the 70B model to fit on a single GPU, while FP16 requires model paral- lelism across all three GPUs. This allows ResQ to support data-pare paralparallll inference, unlike FP1 In Table 8, we show time first token (i.e. end to end prefill latency) at different sequence lengths and batch sizes. Compared to FP16 base- line, ResQ achieves upto 4.98x improvement in end to end latency under batched inference setting. This improvement stems from two factors, first is computational complexity re- duction achieved in ResQ due to weight and activation quan-
iso-bitwidth comparison We also perform iso-bitwidth comparison of ResQ with SpinQuant and QuaRot at W/A/KV of 4-bit. To enable 4-bits with ResQ, we keep i/s channels corresponding to highest eigen values in P in 6-bit, 1/s channels corresponding to lowest eigen values in P in 2-b and remaining 4-bit. Within each quan- tization group, we apply random orthogonal rotations to minimize quantization error. As shown in Table 5, even at same bitwidth of 4-bit, ResQ achieves improved perfor- mance on Qwen2.5 family of models. Complete results are provided in Table 14.
Training rotation matrix R To further improve the per-
8

---

ResQ: Mixed-Preeiste Quantization of Large Language Models with Low-Rank Residuals
Table 8: Time to first token (in ms) for Meta-Llama-3-70B
3.5
on GPU server with 3 NVIDIA A100 GPUs.
3.0
IN14
8,03x
Speedup
2.5
2.42x
2.42x
2.76x
2.008
batch size
seq len
FP16
ResQ
over FP16
Improv.
2.0
2:14x
1.055
1.95a
10240
20783
4242
4.90
1.603x
8192
16373
3361
4.87
1.0
3
4096
7871
1609
1.89x
0.5
3
2048
3888
806
4.82
0.0
2048
7733
1560
x.96
L/ana 3.2
Mota--
Over:2.5
Hcta-
0
Liama-J-8B
Liama-3-708
9
2048
11493
2309
4.98x
Figure Speedup of ResQ and INT4 kernel on single decoder block NVIDIA RTX 3090 over 16-bit floating point baseline for batch size of 1.
But removing both of them leads to trivial increase in perplexity (particularly for Meta-Llama-3 -ama---B and L1 Llama-3 3. 2BB which employ grouped query attention).
Table 7: Memory usage (in GB) for different sequence lengths on NVIDIA RTX 3090.
Rank of high-precisin subspace ResQ allows for seam- less trade-off between accuracy and performance by modu- lating the rank r of high precision subspace (Figure 6-1eft). Increasing the rank improves perplexity albeit at the cost of increased computations in high precision.
QuaRot (Compression)
ResQ
Model
seq
FP16
(Compresslon)
8192
21.9
[1.4 92X)
(1.81
Meta-Ii ----amm--------------- 3-88
2048
16.7
(2.45x)
x2.31x)
512
15.4
(2.77))
6.1 (2.54x)
Calibration dataset size We change number of Wikitext calibration samples used to obtain projections and evaluate performance Figure 6-right. For
8192
18.1
(2.91))
6.8 (2.66x)
Lllam-2-7B-hh
2048
13.9
4.2 (3.30x)
(2.6x)
(3.48x)
(3.08x)
8192
0OM
19.5
21.3
MMLU accuracy increases with increasing samples and saturates beyond 128 samples. For Llama -322-32 the trend is unclear with 512 samples achieving best perfor- mance.
Uwen2.5-14z
2048
OOM
14.0
14.9
512
0OM
12.6
13.5
4/4/4, down
pmjj peoj INT8
down 3/333, down
Avg. danc
0.56
pcoj:
Calibration dataset We evaluate the sensitivity of ResQ's projections to the calibration dataset. While the random rotation matrix is iate-endeee,,,nt, the PCA-based pro- jection matrix depends on the data. We obtain P using samples from Alpaca (Taori et al., 2023), PTB (Marcus et al., 1993), and C4 (Raffel et al., 2020), and Table 16 in Appendix I shows minimal performance variation, demon- strating the robustness of ResQ's Q's calibration.
0.54
Wikitast
12
Meta-Ll Llana
3-65
0.52
11ama-3 -322--33
10
0.50
0.48
16-bbf
100
200
300
400
0.2
0.4
0.6
0.8
nsamples
r/channel dim
Figure 6: Ablation study on changing rank of high precision subspace for Llama-33 3.2-- (left) and changing number of calibration samples (right).
6. Conclusion
We introduce ResQ, a novel -exxed-preiison,, accelerator- friendly PTQ technique toward 4-bit quantization of large language models. ResQ projects weight, activation, and KV cache tensors to subspaces spanned by principal compo- nents, quantizing a low-rank (2/s of hidden dimension) high- variance subspace to 8-bit and the rest to 4-bit. ResQ out- performs both uniform- and mixed- mixeed-preciion quantization methods. We demonstrate the effectiveness ResQ across variety of tasks includin language modeling, language understanding, common reasoning, language genera- tion and multi modal understanding using the Llama and Qwen models. Compared to SpinQuant, the strongest base- line, ResQ achieves up to 33% lower perplexity on the WikiText dataset without requiring any additional training and offers up to 5x speedup over the 16-bit baseline.
tization, and, second is the memory compression achieved in ResQ due weight and kv cache quantization which enables serving the 70B parameter model single GPU device.
5.4. Ablation Studies
Projection bases We evaluate the impact of different projec- tions employed in ResQ by removing them and evaluating performance in Table 6. We see that removing UD or UA has catastrophic impact on perplexity highlighting their importance. Up and Uc which aid in quantization of KV cache have less severe impact when removed independently.

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Impact Statement
Bisk, Y., Zellers, R., Gao, J., Choi, Y., al. PIQA: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelli- gence, volume 34, pp.
ResQ a significant step forward towards efficiently serv- ing LLMs in resource-constrained. coneeneeeeeoocononnnieeeee on-device scenarios, po- tentially expanding the application space for these models. Although our approach aims to make LLMs more accessible and widely used, it does not address the potential risks of misuse for malicious purposes, To mitigate these risks, a strong commitment to user data protection, clear ethical guidelines, and transparency mechanisms is essential.
2020.
Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y--.,, Wang, P.-S., Huang, N.-C., Ceze, L.,, Abdelfattah, M. S., and Wu, K.-C. Palu: Compressing kv-cache with low- rank projection. arXiv:2407.211 2024.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. QuIP: 2-bit quantization of large language models with guaran- tees. In 37th Annual Conference on Neural Information Processing Systems, pp. 4396 4429, 2023.
Acknowledgements
The authors would like thank Wanzin Yazar and Tristan Webb for infrastructure and technical assistance and Zifei Xu and Sakshi Choudhary for helpful discussions. The authors also thank Amogh Joshi for providing access to personal NVIDIA RTX 3090. This work was supported by the Center for the Co-Design of Cognitive Systems (CO- COSYS), DARPA sponsored JUMP center of Semicon- ductor Research Corporation (SRC), Intel, SRC AIHW Pro- gram.
Choi, J., Wang. Z., Venkataramani, S., Chuang, I.JJ, Srinivasan, V., and Gopalakrishnan, K. PACT: Parame- terized clipping activation for quantized neural networks. arXiv: / 1805 000.06 2018.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2924-2222222 2936, 2019.
References
Dao-alla//iat--ddaaardrrraasssso https:///ithhb (com/Dao-AILan /ar///astttae wasaaaaaataaaaaaeeea faas--aaaaarddtrraasssstee
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: 103.00/80000555. 2018.
rm, 2023. Accessed: 2025-01-2.
Ashkboos, S., Croci, M. L., do Nascimento, M. G., Hoefler, T., and Hensman, J. SliceGPT: Compress large language models by deleting rows and columns. In 12th Interna- lional Conference on Learning Representations, 2024a.
Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 10. 2021.
Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. QUIK: Towards end-to-end 4-bit inference on generative large language models. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 3355-3371, Association for Computational Linguistics, 2024b.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 1000088-333,,,,---00000003333338 2022.
Dettmers, T., Svirschevski, R. A., Egiazarian, V. Kuznedelev, D.,, Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. SpQR: A sparse-quantizee representation for near-lossless Ilm weight compression. In 12th International Conference on Learning Represen- tations, 2024.
Ashkboos, S., Mohtashami, A., Croci, M., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-free 4-bit inference rotated 1lms. In 38th Annual Conference on Neural Information Process- ing Systems, 2024c.
Dong, S., Cheng, W., Qin. J., and Wang, W. QAQ: Quality adaptive quantization for Ilm kv cache. arXiv:2403.0466 2024.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z.,, Du, Z., Liu, X., Zeng, A., Hou, L.,, al. LongBench: A bilingual, multitask benchmark for long context under- standing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Pupers), pp. 31 3119-3137. Association for Computational Linguistics, 2024.
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In Pro- ceedings of the 41st International Conference on Machine Learning, pp. 12284 12303. PMLR, 2024.
10

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Frantar, E., Ashkboos, S., Hoeffer, T., and Alistarh, D.-A. OPTQ: Accurate post-training quantization for generative pre-trained transformers. In 11th International Confer- ence on Learning Representations, 2023.
ference on Machine Learning, pp. 4466-4475. PMLR, 2021.
Kang, H., Zhang. Q., Kundu, S., Jeong. G., Liu, Z., Krishna, T., and Zhao, T. GEAR; An efficient kv cache compres- sion recipefor near-lossless generative inference of Ilm. 2403::2403.05527, 2024.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C.,, Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K. Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: https://zenodee zenodo. org/records// 2608602.
Kim, S., Hooper, C. R. C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezel cense-and-sparse quantization. In Proceedings of the 41st International Conference on Machine Learning, pp. PMLR, 2024.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ: Outlier-aware weight quantization for efficient fine- tuning and inference of large language models. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 33355-13344 2024.
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for effi- cient neural network inference. In Low-Power Computer Vision, pp. 291-226. Chapman and Hall/CRC, 2022.
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. SAM- Sum Corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNP)), pp. 70-79, 2019.
Li, J., Li, F., and Todorovic, S. Efficient riemannian opti- mization on the stiefel manifold via the cayley transform. In International Conference Learning Representations, 2020. URL https: / http//ppwwweeeeeeeeeeeeeeo netriiiwwweeeeeeeemmm forum ?id- HHJV-AANDD
Li, M., Lin, Y., Zhang, Z., Cai, T., Guo, J., Li, X., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. SVDQuant: Absorb- ing outliers by low-rank component for 4-bit diffusion models. In 13th International Conference on Learning Representations, 2025.
Guan, Z., Huang, H., Su, Y., Huang. H., Wong, N., and Yu, H. APTQ: Attention -wwrr post-training mixed-preeee precision quantization for large language models. In Proceedings of the s1 ACM/IEEE Design Automation Conference, pp. 1-6, 2024.
Lin, B., Zeng, Z.,, Xiao, Z., Kou, S., Hou, T. Gao, X., Zhang, H. and Deng, Z. MatryoshkaKV: Adap- tive kv compression via trainable orthogonal projection. arXiv:2410.1473/ 2024a.
He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. ZipCache: Accurate and efficient kv cache quanti- zation with salient token identification. In 38th Annual Conference on Neural Information Processing Systems, 2024.
Lin, H., Xu, H., Wu, Y., Cui, Zhang, Y., Mou, L.., Song, L., Sun, Z., and Wei, Y. DuQuant: Distributing outliers via dual transformation makes stronger quantized llms. In 38th Annual Conference on Neural Information Pro- cessing Systems, 2024b.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference Learning Representations, 2021.
Lin, J., Tang, J., Tang, H.. Yang, S., Chen, W.- Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware weight quantization for on-device I1m compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024c.
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length I1m inference with kv cache quantization. In 38th Annual Conference on Neural Information Processing Systems, pp. 1270-1303, 2024.
Lin, Y., Tang, H., Yang, S., 7hang, 7., Xiao, G., Gan, C., and Han, S. QServe: W4a8kv4 quantization and system co- design for efficient Ilm serving. Proceedings of Machine Learning and Systems, 2025.
Huang, W., Qin, H., Liu, Y., Li, Y., Liu, X., Benini, L., Magno, M., and Qi, X. SliM-LLM: Salience-driven mixed-precisionn quantization for large language models. irXiv:2405.14917 2024.
Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang. B. QLLM: Accurate and efficient liw-bitwidth quanti- zation for large language models. In /2th International Conference on Learning Representations, 2024a.
Hubara, L. Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small calibra- tion sets. In Proceedings of the 38th International Con-

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Liu, T., Xu, C., and McAuley, J. RepoBench: Benchmarking repository-level code auto-completion systems. In /2th International Conference Learning Representations, 2024b.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning- free asymmetric 2bit quantization for kv cache.
Sakr, and Khailany, B. Espacc: Dimensionality reduction of activations for model compression. In 38th Annual Conference Neural Information Processing Systems, 2024.
Proceedings of the 41st International Conference on Machine Learning, pp- 32332-32344. PMLR, 2024c.
Sap. M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNLP-JJCNLE pp. 2019.
Liu, 7., Zhao, C.,, Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. SpinQuant: Llm quantization with learned rotations. In 13th International Conference on Learning Represen- tations, 2025.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19())313-330, 1993. URL https:: patts::/aallatthloggyyo https///llaaattool.ooooooooo Jrg -2004/.
Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen Attention: Attention in low-rank space for kv cache com- pression. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pp. 15332-15344, 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, 2017.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z. Zhang, K., Gao, P., Qiao, Y., and Luo, P. OmniQuant: Omnidirectionally calibrated quantization for large lan- guage models. In /2th International Conference on Learn- ing Representations, 2024.
Meta. Llama 3.2: Revolutionizing edge Al and vision with open, customizable models, 2024a. URL https: //ai .ee... wtt.ccom//lloc httpa///////oooooa//tae/eaaaae/a 3 connect- 202 4- --viisioo-eeeee-mbbi-e----------
Sharify, S., Saxena, U., Xu, Z., Yazar, W., Soloveychik, I., and Wang, X. Post training quantization of large language models with microscaling formats. NeurlPS Efficient Natural Language and Speech Processing Workshop, pp. 241-258. PMLR, 2024.
Meta. Introducing Meta Llama 3: The most capable openly available LLM to date., 2024b. URL https:://ssee //wt.//iimmmmm eta. www.coooo/oooooeoeeeeeeoooooeeoe meta a- 1lama-3/.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
Sheng. Y., Zheng, L.,, Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang P., Ré, C., Stoica, I., and Zhang, C. Flex- Gen: High-throughput generative inference of large lan- guage models with single gpu. In Proceedings of the 40th International Conference on Machine Learning, pp. 3 31094-31116 PMLR, 2023.
Park, G., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., Lee, D., et al. LUT-GEMM: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In 12th Inter- national Conference on Learning Representations, 2024.
Taori, R., Gulrajani. I., Zhang, T. Dubois. Y., Li, X., Guestrin, C., Liang. P., and Hashimoto, T. B. Stanford al- paca: An instructionnfollloinng Ilama model. https:// gi tithhb..a 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
lab/ aabbstanford_aapaat
Thakkar, V., Ramani, P., Cecka, C.,, Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P.. Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL https: / // tiithhu.coom cOm/NVTD A/ cutlass.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
12

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C.,, Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A.. Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog. I, Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K. Schelten, A., Silva, R., Smith, F. M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnie, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307. //arxiv. otg/ abs 2207. 09288.
Yang, J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S. J., and Lee, D. No Token Left Behind: Reli- able kv cache compression via importance-aware mixed precision quantization. 17XXx:2402.18099 2024b.
Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. ZeroQuant: Efficient and affordable post- training quantization for large-scale transformers. In 36th Annual Conference on Neural Information Processing Systems, pp. 27168-27183 2022.
Yuan, Z., Niu, L., Liu, J., Liu, W. Wang, X., Shang, Y., Sun, G., Wu, Q. Wu, J., and Wu, B. RPTQ: Reorder- based post-training quantization for large language mod- els. arXiv:2304.01089, 2023a.
2023. URL https:
Yuan, Z., Shang. Y., Song. Y., Wu, Q. Yan, Y. and Sun, G. ASVD: Activation-awar singular value de- composition for compressing large language models. arXiv:2312.05821, 2023b.
Tseng, A., Chee, J.. Sun, Q. Kuleshov, V., and De Sa, C. QuIP# even better 11m quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning, pp. 18630-48656 PMLR, 2024.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. MMMU: massive musti-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9556 9866-9567, 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C.,, Zhou, J., and Lin,
J. Qwen2-VL: Enhancing vision-language model's per- ception of the world at any resolution. 1rXiv:2409.21111, 2024.
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4800, 2019.
Wolf, T., Debut, L.., Sanh, V., Chaumond, J., Delangue, C.,, Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer. S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L.,, Gugger, S., Drame, M., Lhoest, Q. and Rush, A. M. Huggingface's ' transformers: State- of-the-art natural language processing, 1rX0v:1910003771, 2020.
Zeng, C., Liu, S., Xie, Y., Liu, H., Wang. X., Wei, M., Yang. S., Chen, F., and Mei, X. ABQ-LLM: Arbitrary-bit quantized inference acceleration for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22299 220997,2277...11 2025.
Xi, H., Li, C., Chen, with 4-bit integers. In 37th Annual Conference on Neural Information Processing Systems, pp. 49146-49168 2023.
J., and Zhu, J. Training transformers
Zhao, Y., Lin, C.-Y., Zhu, K. Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6:196-209, 2024.
Xiao, G., Lin, J., Seznec, M., WU, H., Demouth, J., and Han, S. Smoothhuant Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, pp. 88087-38099, PMLR, 2023.
Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Hassan, A., Celikyilmaz, A., Liu, Y., Qiu, X., et al. QM- Sum: A new benchmark for query-based multi nomti-domain meeting summarization. In Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 5905-5922 2021.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report, arXiv:2412.15115, 2024a.
13

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
A. Proof of Theorem 4.2
begin the proof by introducing the following lemma.
Lemma A.l. For any tensor R quantized following the quantization described equation 1, assuming the values of R follows a normal distribution, we have
π log siize (R)
- ((H]e -
R P
(9)
2n-1
where size( R.) denotes the number of elements R.
Proof of lemma A.1 can be found in (Li et al., 2025). From this lemma we obtain that the quantization error R - QR) bounded by the magnitude of the tensor quantized [R]].. Now for our use case of mixed precision quantization where the low-precision component is quantized to L bits and high precision component is quantized to H bits, we write the quantization error again below,
E X Xalle
= E XU; - Qx(XII
(10)
+ XU,, (XUm)]l
The random rotation matrices R ensure that XU, and XU, are normally distributed by Lemma 4.1. Applying Lemma A.1 to the quantization error in equation 10, we get,
/og(siie( XUI))
X Xalle s
24-1
/og(ssize( XU
XUul|F
2
log(size( X PI)
E P
2
(II)
/og(siie(( X PT)
E||XPP||
21
v Vlogssize{X X P.)
(.P. )
21-1
/log(size(XX))
tr
)
21 -1
We know size( (PP) = d - and size X P,)) since r components are in high precision. With P.P. + P. P. = I, we have
Aog(d-r)z
|X XaI1x
X
E [X
24-1
Mlg(r)T
E E| 2 -1
(12)
Vog(d-r)T E| | 21--1 -1
Mog(d-r)
Mog(r
X
26-1-1
28-1-1
Since / logid-r7
Aogiti
the quantization error is reduced by maximizing | I.Pale
24
B. Distribution of activations
The distribution of activations after projection by U is shown in Figure 7. The formulation of U ensures that the final r channels in the activation map comprise of coefficients along bases with maximum activation variance. Consequently, keep those channels in high precision minimizes quantization error. The remaining channels are more amenable to quantization due to the application of random rotations which suppress outlier values.
14

---

ResQ: Mixed-Precisse Quantization of Large Language Models with Low-Rank Residuals
INT
10
10
Activation Value
Activation Value
Activation Value
INTS
0
de dh le 1S
0
-5
10
U
1:00 Parcentle
-15
-10
25/75 Pasestle
-10
1000 2000 3000 0
4000
1000 2000 3000 4000
1000 2000 3000 4000
1000
2000
3000
4000
Channel
Channel
Channel
Channel
(a) Baseline Attn Input
(b) ResQ Atin Input
(c) Baseline FNN Input
(d) ResQ FFN Input
Figure 7: Input activation distributions of attention and FFN layers, for baseline (a and c) and ResQ and d).
Table 9: Time taken (in NVIDIA A100 GPU hours) to quantize the model. All approaches use GPTQ (Frantar et al., 2023)
for weight quantization. SpinQuant uses 4 GPUs to optimize rotation matrices.
QuaRot
ResQ
SpinQuant
I. ama 3.. 2-18
4m
7m
13m
ama 2-38
8m
16m
38m
Beta-Llama-3--aa
17m
35m
1h41m
Llama-2-7b-ba
15m
33m
1h37m
Llama -11ma-2-1
23m
1h
3h42m
C. Additional implementation details
In this work, obtaining the projection matrices and quantization of weights for all the models is performed on a single NVIDIA A100 80GB GPUs. Time taken by ResQ compared with other approaches is shown in Table 9. Evaluation on various benchmarks for all the models also done on single NVIDIA A100 GPU with the sole exception of Metta-Llmm- -3-70b which requires 4 GPUs for evaluation. We use 1m.eva tuatiin harness version 0.4.5 (Gao al., 2024) and LongBench (Bai et al., 2024) for all the evaluation tasks. For Arc-c/e, Hellaswag, OpenBook QA, PIQA tasks report acc_norm while for BoolQ, SIQA and Winogrande we report acc.
For calibration data, we use 512 randomly choses samples for Wikitext to obtain the projection matrices. While for GPTQ we use 128 randomly choses samples from Wiktiext following the original work Frantar et al. 2023.
The KV cache, as well as the weights and activations of all Linear layers (except mlp- down_proj), are quantized to bit precision, with of channels retained 8-bit precision. While, the weights and activations within down-pro j are uniformly quantized to 4-bit precision. Following Ashkboos al. 2024c and Liu al. 2025, keep query vector in 16-bit.
D. Complete results of main result tables
Detailed results of Table in the main paper, including more models and task-by-task performance, are shown in Tables 10 (Llama families) and 11 (Qwen2.5 family). As expected, ResQ achieves superior performance to baselines across the series of common sense reasoning and MMLU tasks.
E. Additional quantization results: W/A/KV = 4/8/4 bits of precision
This section presents additional comparisons between baselines and ResQ for the Llama family when quantized to W/A/KV= 4/8/4 bits of precision. Across various MMLU tasks and perplexity evaluations on WikiText, ResQ consistently outperforms all baselines. For 0-shot common sense reasoning tasks, except for Meta-Li ama-3-82. ResQ achieves the best average performance. In the case of Met aoeta-Lie aalama-- - 8B, ResQ the second-best method, with QuaRot performing marginally better by less than 0.2%.
15

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 10: Comparison of perplexity on Wikitext, accuracy on eight O-shot common sense reasoning tasks including ARC- challenge, easy BoolQ. HellaSwag, Openbook QA, PIQA, SIQA, and WinoGrande, and O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMLUU-tthr,, for the Llama 2, Llama 3 and Llama 3.2 families. Results of all techniques were obtained using their official codebase. Our work ResQ and QUIK (Ashkboos et al., 2024b) keep l/s channels in 8-bit and remaining in 4-bit for W/A/KV 4.5-bit. Other baselines uniformly quantized to W/A/KV 4-bit. All techniques except RTN use GPTQ (Frantar et al., 2023). (U): lower is better, (t): higher is better.
Llana 2 family
Perplxsit
Dusbot common seuse reasoning tasks
D-shor MMLL lasks
Model
Method
Wiki
ARC-2
ARC-c
RoolQ
HellS
DBQA
PIQA
SIQA
Winafi
Avg.
humannies
Other
SocialS
STEM
Avg.
(7)
(1)
(1)
(1)
(?)
(?)
(1)
(t)
(*)
(1)
(1)
(T)
16-bit
46.3
75.9
46.1
58.0
45.0
46.0
RTK
1768.2
16.3
27'8
51.8
29.2
8
51.6
55.0
"68.7
37.4
22..5
24.7"
2239"
2272"
23.6
GPTQ
9600.0
24.8
31.4
55.4
30.6
25.6
55.8
34.2
53.3
38.9
24.7
24.5
22.7
25.2
23.8
11am-22-7---
SmocthQuant++
QUIK
15.4
7.5
29.3
39,8
63.7
47.1
68.9
56.8
68.3
18.6
31.8
37.8
65.5
72.9
37.2
42.1
62.4
52.4
57.0
16.1
25.0
26.9
29.5
24.5
28.8
24.1
23.4
25.8
27.8
24.2
QuaRat
41.5
71.4
73-2
40.6
43.6
35.1
SpinQuunt
6.0
43.6
71.3
73.8
73.2
40.4
76.0
44.1
65.4
61.0
33.9
38.5
37.5
29.5
34.8
ResQ)
58
44.0
72.6
75.3
74.0
41.0
77.9
43.9
66.9
62.0
35.9
40.9
42.2
32.2
37.7
19.1
77.4
79.4
15.2
47.9
KIN
3543.9
22.8
29.8
40.2
26.5
27.8
51.4
35.6
30.6
33.5
23.7
25.0
23.1
22.5
23.6
GPTQ
$120.0
23.6
$1.1
36.7
27.2
26.8
53.6
35.8
40.8
33.8
25.0
25.4
23.7
25.1
24.8
Mlama-2-13b--
Smoolh-uant+
QUIK
68
34.5
43.7
68.0
71.3
73.3
32.4
40.0
75.7
45.1
64.6
51.0
60.2
54.7
26.1
40.6
19.8
51.8
36.7
QuaRool
5.1
16.9
74.9
76.6
75.8
12.6
79.1
45.5
69.0
63.8
43.8
53.6
54.0
39.4
47.7
SparQuant
49.0
76.3
77.1
42.8
79.3
46.3
69.5
64.8
43.5
53.1
35.4
Rei
5.1
49.1
76.1
79.7
77.9
43.6
79.1
46.6
69.9
65.2
45.3
56.0
58.0
41.0
50.1
Lluma 3 family
Perpexit
D-sbot
reasoning tocks
U-sbox MMLU lasks
Model
Method
Wiki
AKC
ABCC
Bool))
HellaS
UBQA
PIQA
SIQA
Winal:
Avg.
humanries
Uther
SocialS
STEM
Aug-
(U)
(t)
(1)
(7)
(1)
(1)
(7)
(1)
(1)
(1)
(7)
(1)
(1)
(1)
(1)
16-hit
6.1
53.1
77.1
79.2
44.8
47.00
73.4
67.1
55.0
70.6
73.2
53.7
RTN
218.9
25.3
54.9
38.3
27.8
56.5
36.8
50.8
39.3
24.7
25.1
23.3
21.4
GPTQ
166.3
24.7
37.7
44.3
36.8
27.0
57.6
36.4
53.8
39.8
24.7
23.9
22.8
21.8
23.3
Neta-Llama-3--BB
SmoothQuutt
78.2
27.5
42.0
50.7
46,9
28.8
59.0
35.9
50.9
42.5
25.4
25.5
24.5
25.4
24.7
QUIK
14.2
33.6
56.4
60.5
61.5
33.2
68.7
39.9
59.0
$1.6
30.0
34,0
34.8
32.1
32.7
QuaRot
7.8
45.1
70.4
73.8
74.7
42.6
76.6
45.1
68.5
62.1
47.8
59.1
61.4
44,3
53.2
SpinQuuntt
48.0
75.4
75.4
43.8
ResQ
7.1
49.2
75.0
72.5
76.5
43.0
78.3
45.8
71.0
63.9
30.6
64.4
65.8
48.1
57.2
16-bit
2.9
64.2
85.0
85.3
84.9
48.6
64.4
50.8
80.6
73.1
67.6
81.5
86.8
68.4
76.1
RYKK"
-B1.
$2..0
50.3"
542"
'1.3
"1..6
6T.
55.9
553.2
15.5
221.3
25.6
2203"
22.1"
25.2
GPTQ
11055.0
25.9
26.0
37.9
26.2
28.6
50.4
34.3
49.9
34,9
27.1
24.3
24.0
26.5
25.5
SmrethQuunt=
Keta-Llana 2-anaa---7C
QUIK
44.5
36.4
46.6
56.4
QuaRot
5.7
53.7
74.5
81.6
81.1
46.6
81.0
46.8
75.2
67.6
55.7
72.5
75.8
57.3
65.3
SpinQuunt
6.2
52.0
77.3
81.7
75.6
43.8
78.8
43.4
72.8
65.7
50.7
67.0
68.1
51.9
59.1
Resl)
4.1
61.4
$4.3
83.5
46.0
48.6
78.3
71.1
64.9
79.9
84.9
66.1
Tunnly
Perplesity Wiki
D-shvt common senso rossoing tioks
D-shor MMLAT mokk
Model
Methed
ARC-s
ARC-e
BoolQ
Hallas
OBQA
PQQA
SIQA
Winol
humamites (7)
Other
SocialS
STEM
(
(f)
(7)
(1)
(1)
(?)
(f)
(1)
(1)
(1)
(t)
(1)
(1)
bi-bit
9.8
36,3
60.6
63.4
63.6
37.4
74.55
42..8
00.1
34.9
31.8
$1.1
39.9
32.0
36.9
RIN"
1
"2.4
29.9"
S5.4
"11.4"
29..4"
54.8
"54.""
"48.5"
36.1
"24.5"
25.5
22.4"
1227"
25.8
GPTQ
108.9
24.7
52.7
30.7
23.6
54.3
34.4
51.1
38.0
24.7
25.1
25.5
24.5
24.9
Mamm-33.2 3.22-
SmoethQuntt+
228,9
23.3
30.1
52.9
31.3
26.6
$4.2
34,5
$1.2
38.0
23.9
24.1
25.0
23.5
24.1
QUIK
21.8
27.4
46.0
46.0
26.4
62.4
38.6
52.6
44.3
25.6
25.5
24.6
24.5
25.1
QuaRot
14.3
30.0
51.4
59.1
$6.0
34.2
56.7
39.6
$7.1
49.0
25.4
26.9
25.4
24.4
25.5
SpanQuuant
13.6
32.3
51.8
59.3
$5.4
30.4
67.3
38.6
54.7
48.8
25.44
27.6
24.2
25.3
25.6
ResQ
12.4
34.0
54.2
$7.3
31.2
16-hit
788
46.2
71.7
73.1
75.7
43.4
77.4
47.2
69.1
62.7
43.9
62.9
62.3
45.2
54.8
RTN
268.8
23.5
35.4
46.2
35.6
28.2
56.3
33.6
50.6
38.7
25.1
25.6
27.0
24.9
25.7
GPTQ
178.3
27.0
27.0
27.8
59.1
37.1
24.9
24.5
25.7
Llema-3.2-93
SmoothhQuntt
96.1
25.3
33.1
47.8
37.7
25.2
56.2
35.8
50.9
39.0
25.4
26.6
26.4
25.5
25.9
QUIK
158
32.9
50.1
52.6
59.1
33.2
68.7
40.3
$3.0
48.8
20.0
33.2
31.9
30.3
31.1
QuaRat
38.6
59.0
66.5
35.8
43.1
65.2
38.5
47.3
46.7
35.5
SpinQunnt
9.2
38.9
64.8
68.0
69.1
39.4
74.9
45.1
62.9
57.9
37.0
49.4
50.5
39.9
44.2
ResQ
8.8
43,1
65.6
68.8
70.5
38.4
75.1
45.6
64.8
39.0
44.7
57.0
56.5
41,0
49.8
F. Complete results of the MMMU benchmark
This section presents lask-by-task results for the MMMU benchmark across six subjects Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Figgneeriing for the Qwen2 VI family when quantized to W/A/KY = 4/4/4 bits and W/A/KV = 4/8/4 bits of precision. On average, ResQ consistently outperforms all basclines across different models. Notably, the advantage of ResQ becomes more pronounced with larger models. For instance, for
at W/A/KV = 4/8/4 bits of precision, ResQ achieves an average accuracy score of 48.8, significantly outperforming the next-best method, QUIK, which scores 26.4, representing an ~ 85% relative improvement.
16

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 11: Comparison of perplexity score on Wikitext, accuracy on eight O-shot common sense reasoning tasks including ARC-challenge, eac-easy, BoolQ, HellaSwag, Openbook QA, PIQA, SIQA, and WinoGrande, and O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMLJU-other, for the Qwen2.5 family. Results of all techniques were obtained using their official codebase. Our work ResQ and QUIK (Ashkboos et al., 2024b) keep 1/s channels in 8-bit and remaining in 4-bit for W/A/KV 4.5-bit. Other baselines are uniformly quantized W/A/KV 4-bit. All techniques except RTN use GPTQ (Frantar et al., 2023). (4): lower is better, (t): higher is better, In cases where Hadamard matrix does not exist at the MLP dimension, random orthogonal rotation is used instead.
Qwen2.5 family
Peplexity Wiki
O-shol common sense ressoning fasks
misks
Model
Method
ARC-c
ARC-e
BoolQ
HellaS
OBQA
PIQA
SIQA
Winoki
Avg.
bumanities
Other
SocialS
STEM
Avg.
(
(t)
(1)
(7)
(1)
(1)
(7)
(1)
(1)
(1)
(f)
(7)
(t)
(f)
(?)
16-bi
13.1
31.9
58.4
62.1
52.1
35.0
69.7
41.3
57.1
51.3
42.2
53.2
55.5
41.5
48.1
RTN
23204.3
26.2
27.0
26.0
24.0
30.7
51.5
34.9
24.8
22.8
24.3
23.9
GPTQ
16302.3 3
23.7
26.9
39.0
26.5
26.4
50.2
33.4
49.5
34.5
24.1
24.8
23.5
23.0
23.9
Qwen2. 5-0.55
SmootSQuant-
10063.9
25.9
26.3
39.9
27.2
25.4
47.1
35.9
49.5
34.7
24.5
24.7
21.5
22.1
23.2
QUIK
38.6
24.5
38.6
48.0
36.9
28.4
58.1
36.4
51.9
40.4
26.3
25.9
23.6
24-2
25.0
QuaRoce
219.9
25.4
36.6
45.0
28.9
28.6
54.1
32.9
51.7
$7.9
24.4
24.0
23.0
23.5
23.7
ResQ
29.6
28.0
51.3
26.1
27.5
25.3
16-bi
9.3
45.1
72.1
72.9
67.7
40.2
76.3
4K.8
63.7
$3.3
65.55
70.5
$2.8
60366
RTN
14518.9
25.66
52.5
23.8
23.8
GPTQ
25769.7
23.9
26.9
43.9
26.1
27.6
49.7
92.1
51.5
35.2
24.6
24.7
23.7
23.8
24.2
mion2.5-1.a2
SmoothQuant-
31655.9
25.0
26.2
39.9
26.0
26.0
50.8
32.1
49,0
34.4
25.5
24.4
22.7
22.4
23.8
QUIK
6613.5
21.8
31,9
40.9
27.9
27.4
52.8
35.2
48.6
35,8
24.6
24.0
21,9
21.7
23.1
QuaRot
6599,9
23.6
37.3
46.2
28.6
27.0
56.3
35.2
$2.4
38.3
24.5
24.3
23,0
22.4
23.5
ResQ
37.8
60.1
54.9
16:bit
8.0
47.4
73,0
77.5
73.6
42.0
78.7
49.9
08.4
633.8
56.6
71.00
76.3
60.6
66.1
RTN
39033.0
25.6
25.8
41.7
26.3
27.4
51.4
21.5
22.8
21.9
GPTQ
9977.8
26.0
26.7
41.55
26.7
28.2
51.5
31.9
48.3
35.1
24.3
233.8
22.8
21.8
23.2
Qwen2. 3-3B
SmoothQuant-
73306,7
25.4
24.5
41.0
26.4
29.8
48.4
32.4
50.4
34,8
25.6
24.7
23.1
22.4
23.9
QUIK
15.5
36.1
55.4
61.4
$7.2
36.2
67.1
40.8
55.3
51.2
36.4
42.8
42.4
36.1
39.4
QuaRce
68.8
32.4
$3.1
51 6
49.2
33.4
66.7
39.3
56.4
47.7
28.1
$2.0
28.9
26.6
28.9
ResQ
70.5
72:7
53.1
70.5
16 bit
6.8
51.2
77.6
$4.7
76.9
47.2
80.0
$4.8
73.2
68.4
62.6
76.7
82.6
20.1
73.0
"RIN"
2
2113
26.3"
57'3"
26.0
29.0
"50.1"
3459
"2A.5"
2/535
2354"
219
24A4"
GPTQ
13593.7
25.2
25.6
37 8
26.3
28:2
52.4
34.4
48.9
34.8
24.4
24.3
22.8
22.6
23.5
Qwen2. 5-7B
SmoothQuant=
19088.7
26.3
25.2
39.5
26.4
27.6
52.7
33.5
52.0
35,4
25.1
255.4
22,6
24.1
24.3
QUIK
260.3
29.5
42.4
51.7
36.3
28:2 2
59.6
34.5
49.6
41.5
24.3
26.9
23.1
23.8
24.6
QuaRtre
4035.9
25.9
41.0
39.1 1
29.1
27.6
97.99
35.7
50.6
38.4
24.8
24.4
24.4
22.7
24.1
ResQ)
49.0
74.7
$1.4
75.7
68.2
$7.8
79.3
16 bit
5.3
58.8
79.4
$5.4
82.9
45.4
$1.9
55.3
75.8
70.6
69.9
$1.9
$6.2
76.5
78.6
RYNN
2715
21.6
32.7"
51.3
29..8
23.88
552.6
"3.2"
"51.7"
31.3"
225.3
2:5.2
2670"
25.3"
24.9
GPTQ
310033
23.8
29.1
47.7
30.1
27.6
51.3
34.6
$1.2
36.9
25.1
24.7
25.1
24.3
24.8
Owen2.5 5-148
SmootiQuant=
1375.7
27.0
26.3
38.0
26.8
29.2
51.6
32.4
49.3
35.1
25.9
24.5
22.2
22.2
23.7
QUIK
10.5
45.0
67.1
64-7
68.9
37.6
74.8
43.9
59.3
57.6
9
61.1
64.7
51.5
56.6
QuaRce
68
54.8
79.66
79.9
78.7
44.0
79.5
49.9
70.7
67.1
60.9
75.1
80.2
67.3
70.9
ResQ
57.6
$2.1
70.6
75.4
83.4
74.6
16-ba
3.0
55.7
75.0
$7.4
81.1
$1.41
82.3
56.4
75.2
70.4
73.1
$3.6
89.8
812
$1.9
RTN
18474
24.3
35.3
51
31.9
32.8
34.1
151.4
36.5
24.5
25.1
25.3
24.3
24.8
GPTQ
18911
25.4
38.4
48.5
31.8
27.0
53.8
35.8
50.5
38.5
25.9
24.8
23.6
24.0
24.6
Qwen2.5-328 522
SmootiQuant=
QUIK
9.6
41.0
64.6
74.9
72.0
39.6
75.8
44.5
60.2
59.1
54.7
66.8
71.3
53.8
62.9
QuaRoe
6.1
54.5
16..1
85.1 1
81.5
44.2
80.1
$1.3
70.4
67.9
68.5
80.0
86.0
76.0
77.6
ResQ)
78.4
74.0
70.3
$7.9
79.8
16-bit
39
62.6
83.2
89.2
86.0
46.6
83.6
55.4
71.3
73.4
77.2
86.9
98.6
82.4
84.3
RTN
45412.7
25.9
26.3
38.0
48.7
34.3
25.5
23.0
24.0
GPTQ
37967.2
25.4
25.8
38.1
25.6
26.6
51.2
34.2
49.4
34.5
25.1
24.0
21.9
22.2
23.3
Qwen2.5-728 523
SmoothQuant-
QUIK
8.3
45.1
68.1
77.2
77.2
39.0
77.4
45.6
65.6
61.9
60.2
74.3
77.5
65.3
69.3
QuaRot
4.9
55.8
81.1
87.5
84.0
45.2
81.7
52.5
74,5
700.
71.4
84.2
87.7
77.1
80.
ResQ
58.4
80.9
88.4
48.2
77.0
72.8
89.0
79.5
G. Complete results of iso-bitwidth comparison
This section presents detailed results for Table in the main paper for Qwen2.5 family of models. The task-by-task results are provided in Table 14. As expected, ResQ achieves superior performance. For Qwen2.5-72B. SpinQuant baseline is missing because of lack of big enough GPU cluster enable training rotations.
H. Complete results when training rotation matrix R
In Table 15, we provide results on Wikitext perplexity, task-by-task accuracy on 0-shot reasoning benchmarks and task-by- task accuracy on MMLU benchmark when training rotation matrix R in ResQ. The rotation matrix is trained using Cayley SGD (Lii et al., 2020) for 100 training steps at batch size 8 and learning rate of 1.5. The training data involves samples
17

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 12: Accuracy on eight O-shot common sense reasoning tasks including ARC-challenge, ARC-easy, BoolQ, HellaSwag, Openbook QA, PIQA. SIQA, and WinoGrande and O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMI.U-other, for the Ilama 2, Llama 3 and Llama 3.2 families when quantized t/ 4/8/4 bits. Results of all techniques were obtained using their official codebase. Our work ResQ and QUIK (Ashkboos al., 2024b) keep l/s of channels in 8-bit, All techniques except RTN use GPTQ (Frantar et al., 2023). (U): lower is better, (f): higher is better.
Liama 2 family
Peplexity
O-shot common senve reasoning links
Osaa MMLU jasks
Model
Method
Wiki
ARCic
ARC
BoalQ
Hellas
OBQA
PIQA
SIQA
Winol
Avg.
bamanitees
Other
Socials
STEM
Avg.
(t)
(7)
(7)
(t)
(1)
(t)
(t)
16-bit
5.5
46.3
74.6
77.8
75.9
44.2
19.2
46.1
09.1
64.1
38.9
45.9
46.0
33.4
41.1
RTN
7.2
41.5
65.9
71.9
71.8
59.4
76.5
42.7
65.7
59.4
27.6
18.4
31.6
29.0
20.2
GPTQ
11.8
42.5
71.3
73.6
44.9
28.0
32.3
-lama-2-7a-ht
SmoothQuam-
6.8
41.9
69.1
70.7
72.9
40.2
77.1
32.7
66.9
58.9
28.1
30.0
28.6
27.1
28.5
QUIK
5.7
43,9
73.4
77.3
744
44.6
78.2
44.3
68.9
63.1
35.8
39.3
40.1
30.4
56.4
QuaRot
43.6
74.8
45.1
62.6
36.3
SpinQuant
5.7
43.5
73.3
75,4
74.8
42.6
77.5
45.0
68.4
62.6
37,0
42.0
43.4
31.8
38.5
RedQ
5.6
46.3
74.5
77.1
75.0
42.8
78,9
45.6
68.8
63.6
39.1
45.9
47.9
34.9
42.0
16 bit
49,1
77.4
79..
43.2
80.7
47.2
47.9
39.3
52.7
RTN
6.9
41.3%
65.2"
70.38
66.3
"37.3"
76.0
#2.
63.9
5600
37,5
-0338
51.36
36.0
GPTO
46.2
73.2
76.0
73.4
43.2
78.2
44.4
63.0
35,9
42.1
39.0
30.8
36.9
Llana-2-B 11b-hh
SmoothQuant-
QUIK
5.0
45.0
47.5
71.5
76.4
76.8
78.9
73.4
78.4
44.6
42.8
80,4
76.7
46.8
31.9
72.5
65.5
46.1
32.8
42.7
$7.0
58.2
40.0
57.1
50.3
QuaRot
5.0
48.6
77.0
78.9
78.2
44.2
80.3
46.3
72.2
65.7
46.5
56.8
50.4
SpiuQuant
48.3
76.4
78.1
43.8
79.8
46.7
46.7
57.1
58.3
40.1
50.5
ResQ
5.0
49,0
77.1
80.6
78.9
45.4
79.9
47.2
72.3
66.3
47.6
56.2
59.9
41.7
51.9
Lluma 5 family
Perpessity
O-shot
tasks
Dishot MMLU tasks
Model
Method
Wiki
ABC-e
ARC-a
Baoll)
Halls
CBBQA
PIQA
SIQA
Winoo
Avg.
hananitiss (1)
Other
SoctalS
STEM
Avg.
(1)
(1)
(1)
<T)
(7)
(1)
(f)
(1)
(1)
(7)
(t)
(1)
<t>
(1)
16-hit
53.2
79.2
80.9
47.0
67.1
65.1
"RYN""
4738
7223"
12.
78.2
"A.3"
"7..5
766.5
3055
GPTQ
7.5
47.1
71.1
72.2
72.7
42.6
78.2
45,7
72.8
62.8
40.4
60.3
61.6
46.2
52.1
SmeethQuant-
44.8
71.2
73.6
40.0
79.0
43.6
42.1
53.6
54.1
38.5
47.1
Met Maaana BR
QUIK
6.7
50.0
35:7
80.1
77.4
45.8
80.0
45.1
74.8
66.1
52.1
65.7
68.
49.8
58.9
QuaRot
6.7
51.6
78.5
30.0
77.7
45.2
79.8
46.4
73.1
66.5
$1.6
66.8
68.5
4817
58.9
SpinQuant
50.0
77.2
77.9
66.2
59.3
ResQ
6.5
54.3
78.6
77.2
78.4
44.0
79.2
46.3
73.2
66.4
53.6
68.6
70.0
$2.0
61.0
16-bi
3.9
66.2
85.9
85.3
519
45.6
84.4
50.5
80.6
73.
67.6
$1.5
56.3
68.4
76.
KIN"
26.3"
25.7"
51.3
26.1
25.0
3T.1
56.6
53..0"
3535
25,4
1519
2213
E277"
26.1
GPTQ
8586.4
26.5
24.9
38.1
26.4
29.4
51.9
34.9
49.4
35.2
25.7
23.6
22.5
23.4
23.8
SmoothQuant-
Keta Clana 3 TCB
QUIK
60.3
83.5
45.4
79.0
Quakot
3.6
60.0
84.3
84.9
83.9
49.2
83.9
49.4
78.8
71.8
64.3
80.0
85.8
66.7
74.2
SpinQuaat
Reil)
63.0
84.7
84.4
482
84.2
50.1
8600
66.8
Liana family
Perplexity Will
O-shot common sense reasoning tasks
(lthoor MMLU tasks
Modil
Method
ARC--
ARCC-
Bunlq
Holas
OBQA
PIQA
STQA
Winki
hamamitis
Olher
Smetall
STEM
Avp.
(1)
(t)
(1)
(?)
(1)
(f)
(1)
(?)
(7)
(1)
(t)
(1)
(1)
(7)
16-bit
988
36.5
60.6
63.4
63.6
37.4
74.5
42.6
60.1
$4.9
34.8
41.1
30.9
32.0
36.9
RTN
16.6
$0.6
46.7
61.9
55.2
32.4
66.7
58.0
56.7
48.5
25.1
36.7
'261
25.2
25.6
GPTQ
15.3
32.8
50.9
61.6
54.8
31.6
67.4
39.1
55.8
492
26.1
26.2
23.9
24.3
24.6
blema-3,
SmeothQuant-
20.6
30.0
47.7
50.2
50.8
31.2
66.3
37.5
54.1
46.0
26.9
26.6
25.5
23.9
25.2
QUIK
11.6
35.0
57.9
59.4
35.4
71.7
41.9
28.2
31.5
QuaRot
11.1
34.1
58.8
52.3
59.8
36.4
72.3
41.3
51.7
28.3
30.6
29.5
26.4
28.7
SpinQuaatt
11.1
34.2
55.6
61.8
60.0
35.0
72.1
60.8
57.6
52.1
26.1
23.7
27.4
21,0
26.3
Resl)
10.4
34.8
61.1
34.4
72.3
41.8
31.3
36,0
33,6
16-bit
288
46.2
71.7
73.1
73.7
43.4
71.4
47.2
09:1
62.7
48.9
62.9
62.3
45.2
54.3
RYNN"
I73
9676
$1.3"
"6.."
"58.6"
73.5
"42.7
722.4
5236
'38.5
4677"
36.0
21.7
GPTQ
37.0
62.9
41.9
64.4
37.1
47.7
46.2
Llama-3.2-32
SmoothQannt=
QUIK
12.7
8.6
37.0
42.1
54.5
65.9
53.3
71.8
61.9
71.7
34.8
40.0
71.2
76.0
41.6
44.6
63.2
66.7
52.2
59.8
45.2
31.4
57.2
37.6
40.7
57.8
40.7
32.4
35.5
50.2
QuaRot
43.4
69.5
71.2
40.6
76.8
46.0
45.0
56.1
49.3
SpiQQuatt
8.4
45.5
67.8
70.6
71.9
41.6
76.9
44.9
68.5
60.7
46.1
56.7
57.1
39.4
49.8
RexQ
8.1
44.4
69.4
72.4
72.3
41.8
76.3
45.
69.1
61.3
48.2
61.1
59,8
44.5
53.4
of sequence length 2048 taken from Wikitext. For complete training hyperparameters, we guide the interested readers to official implementation of SpinQuant (Liu et al., 2025).
I. Complete results with different calibration datasets
In Table 16, we provide results on wikitext perplexity, task-by-task accuracy on 0-shot reasoning benchmarks and task-by- task accuracy on MMLU benchmarks when obtaining ResQ projection matrix P using different calibration datasets. We find no clear consensus on the optimality of one particular dataset. The performance results for different datasets show no
18

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 13: Accuracy (higher is better) on 0-shot massive multi-discipline multimodal understanding and reasoning tasks
across six subjects: Art & Design, Business, Science, Health Medicine, Humanities Social Science, and Tech Engineering for the Qwen2 VI. Instruct family when quantized to W/A/KV = 4/4/4 bits and W/A/KV 4/8/4 bits. Results of all techniques were obtained using their official codebase. Our work ResQ and QUIK (Ashkboos al., 2024b) keep 1/8 of channels in 8-bit. All techniques except RTN use GPTQ (Frantar al., 2023).
Owen2 VL 2B Instruct
0-shot MMMU tasks
W/A/KV (bit)
Method
Art-Design
Business
Science
Health
Humanities
Tech
Avg.
16/16/16
Baseline
56.7
36.0
37.3
50.8
26.0
31.0
39.6
RTN
28.3
18.7
26.0
26.7
21.3
29.1
25.0
4/4/4
GPTQ
28.3
27.3
27.0
29.0
26.7
27.6
27.7
QuaRot
24.2
23.3
20.7
26.7
26.0
22.9
24.0
QUIK
25.8
26.0
26.7
29.2
26.0
24.3
26.3
4.5/4.5/4.5
ResQ
38,3
21.3
28.7
45.0
21.3
29.7
RTN
27.5
21.3
27.3
24.2
21.3
27.6
24.9
4/8/4
GPTQ
24.2
23.3
24.0
18.3
21.3
29.5
23.4
QuaRot
20.0
24.7
30.0
26.7
26.0
26.5
QUIK
33.3
28.7
32.0
32.5
26.0
18.1
28.4
4.5/8/4.5
ResQ
37.5
32.0
32.7
47.5
26.7
27.6
34.0
Owen2 VL 7B
Instruct
0-shot MMMU tasks
W/A/KV (bit)
Method
Art-Design
Business
Science
Health
Humanities
Tech
Avg.
16/16/16
Baseline
68.3
41.3
54.7
68.3
38.7
38.1
51.6
RTN
24.2
28.0
29.3
22.5
29.3
27.I
26.7
4/4/4
GPTQ
21.7
26.0
25.3
28.3
24.7
23.3
24.9
QuaRot
21.7
21.3
28.7
25.0
20.7
29.5
24.5
QUIK
30.8
30.0
32.0
26.7
28.0
26.2
28.9
4.5/4.5/4.5
ResQ
65.0
39.3
45.3
61.7
47.0
RTN
23.3
28.7
27.3
25.0
22.7
24.3
25.2
4/8/4
GPTQ
20.8
23.3
30.0
19.2
24.0
28.6
24.3
QuaRot
20.8
30.0
19.2
24.7
26.2
24.5
QUIK
25.0
23.3
31.3
26.7
25.3
26.7
26.4
4.5/8/4.5
ResQ
67.5
39.3
51.3
64.2
36.7
33.8
48.8
significant fluctuations.
J. Artifact licenses
According to their licenses, all language models used the paper fall under acceptable use case. The licenses for the models
are linked for perusal: Llama 2-7b 7b-hf, Llama 2 1-33 11b--fh M alama 8B, Meta Meta-Llamd
-Llama ama 1amaa-3-77 -0-7BB Llama-3. 3-2BB, Llama-3, 2-3B, Qwen2. 5-0. 5B, Owen2. 5-1. Owen2. 5-3B, Owen2. 5-7B, Qwen2. 5-14B, Qwen2. 5-32B, Qwen2. 5-72B, Qwen2 VL-7B-Instruce and Qwen2 1ree22-V--2B-Instruett
19

---

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Table 14: Comparison of ResQ, QuaRot, SpinQuant at iso-bitwidth of W/A/KV = 4/4/4. ResQ keeps 1/s channels with high eigen value in 6-bit, 1/8 channels with low eigen value in 2-bit and remaining in 4-bit for average bit-width of 4-bit. Evaluations include perplexity on Wikitext, accuracy on eight O-shot common sense reasoning tasks including ARC-challenge, ARC-easy, BoolQ, HellaSwag, Openbook QA, PIQA, SIQA, and WinoGrande, and on O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMLU-other, for the Qwen2.5 model family. (!): lower is better, (t): higher is better.
Perplexity Wiki
O-shot common sense reasoning tasks
O-shet MMLU usks
Model
Method
ARC-c
ARC-
BoolQ
HellaS
OBQA
PIQA
SIQA
WinoG
Avg-
humanities
Other
SocialS
STEM
Avg.
(1)
(t)
(1)
(t)
(1)
(f)
(t)
(*)
(1)
(1)
(1)
(1)
(1)
(1)
(1)
QuaRot
68.8
5244
53.1
51.6
49.2
33.4
66.7
39.3
36.4
47.7
28.1
32.0
28.9
26.6
28.9
Owen:2 . 5-33
SpinQuant
70.6
33.3
56.4
51.8
49.5
33.0
66.7
41.0
57.1
48.6
30.1
37.2
33,9
30.1
52.8
ResQ
9.8
44.1
70.2
70.2
67.3
41.0
73.7
45.4
61.1
59.1
45.8
58.4
58.9
45.9
52.2
QuaRot
4055.9
25.9
41.0
39.1
29.1
27.6
57.9
35.7
50.6
38.4
24.8
24.4
24.4
22.7
24.1
Oven2 55-72
SpinQuant
3395.4
25.5
44.2
39.5
27.0
58.8
25.0
24.3
24.3
ResQ
34,2
42.6
60.2
75.4
59.4
41.6
63.1
44.6
62.8
56.2
49.3
63.6
68.6
50.4
58.0
QuaRot
6.8
54.8
79.6
79.9
78.7
44.0
79.5
49.9
70.7
67.1
60.9
75.1
80.2
67.3
70.9
Qwen2.5-22
SpinQuant
6.6
54.2
81.2
82.1
77.9
44.6
78.4
50.8
70.0
67.4
59.6
75.3
78.9
66.7
70.1
ResQ
6.5
38.6
81.3
79.2
79.2
43.6
79.9
51.4
70.2
67.5
61.6
75.3
80.1
68.2
71.3
QuaRoi
6.1
54.3
78.6
83.0
43.4
79.8
71.7
67.8
67.4
80.5
65.2
75.1
77.0
Qwen2.5-320
SpinQuant
6.0
$4.5
76.1
85.1
81.5
44.2
80.1
51.3
70.4
67.9
68.5
80.0
86.0
76.0
77.6
ResO
5.9
55,9
79.7
85.8
44.8
80.3
73.4
68.4
86.4
Owen2.5-722
QuaRot
4.9
55,8
81.1
87.5
84.0
45.2
81.7
52.5
74.5
70.3
71.4
84.2
87.7
77.1
80.1
ResQ
4,9
58.8
81.9
87.2
84.1
46.0
82.2
53.2
75.5
71.1
71.3
84.2
87.4
77.4
80.1
Table 15: Comparison of ResQ and ResQutraining rotation R, evaluating perplexity on Wikitext, accuracy on eight O-shot common sense reasoning tasks including ARC-challenge, ARC-easy, BoolQ, HellaSwag, Openbook QA, PIQA, SIQA, and WinoGrande, and on O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMLU-other, for the Meta-Ll ama- 30-----220 8B, models. (1): lower is better, (t): higher is better.
Owen2. 5-7B, and Llama-3. 2-1B
Peplecity Wriki
0-shot common opise masssing lisks
B-sbor MMI U'ak
Model
Method
ABC-
ARC-
Booll
HellaS
OBQA
PIQA
SIQA
WinoG
Avg.
humamiles
Other
SocialS
STEM
(1)
(1)
(1)
(7)
(1)
[7]
(1)
(7)
(*)
(?)
(7)
(?)
(1)
(f)
Msta-Liama- 3-BB
Re+Q ResQ+ training R
7.0
7.1
49.2
48.1
76.1
75.0
76.7
72.5
76.5
76.4
44.2
43.0
77.8
78.3
46.8
45.7
71.0
70.6
63.9
64.5
51.0
50.6
65.9
64.4
67.4
65.8
48.
48.9
58.3
57.2
Llana-2 - 2-Tb-hh
ResQ + training R
ResQ
5.8
44.9
44.0
71.6
75.3
74,9
74.5
74.0
41.0
42.0
77.9
77.6
43.9
44.6
67.4
66.9
62.3
62.0
35.4
42.7
40.9
40.4
42.1
33.4
38,0
37.7
Qwwn2.5 7B
ResQ + training R
Res
8,0
50.0
75.3
80.9
76.1
44,0
78.7
52.3
68.2
69.2
65.1
65.8
59.0
74.0
79.3
79.0
65.0
69.0
1lama 3.2 18
ResQ + training &
Res
12.4
34,0
22.9
54.2
57.0
57.5
58.2
57.3
31.2
70.1
69.4
41,0
39.4
55.6
56.8
50.1
50.1
28.3
30.5
31.3
30.6
28.0
27.6
29.4
Table 16: Performance of ResQ with different calibration datasets, evaluating perplexity on Wikitext, accuracy on eight 0-shot common sense reasoning tasks including ARC-challenge, ARC-easy, BoolQ, HellaSwag, Openbook QA, PIQA, SIQA, and WinoGrande, and on O-shot massive multitask language understanding tasks across four subjects: STEM, Humanities, Social Sciences, and MMLU-other, for the L 11aa--33 3. 2BBB, Meta- -Lla-ammaa--- -3 Owen2. 5-3B, and 2oven.55-777 models. (1): lower is better, (1): higher is better.
Pepplexity Wiki (1)
O-shot common sense reasoming
O-shot MMLU tasks
Model
Method
ARC-e
ARC-c
BanlQ
HellaS
UBQA
PIQA
SIQA
Winoli
Avg.
hamanities
Other
SucialS
STEM
Avg.
(?)
(1)
(7)
(1)
(1)
(*)
(1)
(1)
(t)
(T)
Wikiiest
8.8
43.1
65.6
68.8
70.5
38.4
75.]
45.6
64.8
59.0
44.7
57.0
56.5
41.0
49.8
C4
8.8
42.1
66.7
70.6
41.2
75.4
45.3
61.7
43.9
55.5
55.0
40.2
48.6
21 .1ama-33 3-2-38
FTB
8.8
41.8
65.8
70..2
70.0
39.2
74.9
46.1
65.2
59.1
43.5
54.8
53.1
38.9
47.6
Alpaca
8.8
43.1
65.9
66.6
70.4
38.8
75.5
44.9
66.2
58.9
43.6
54.8
53.9
39.6
48.0
Wikiiext
7.1
49.2
75.0
72.5
16.5
43.0
78C3
45.8
71.0
63.9
30.6
64.4
65.8
48.1
57.2
Meta-il wlm-mmmmmm------
C4
7.1
48.0
76.3
76.6
76.4
42.4
78.5
46.0
68.1
64.0
50.8
64.6
66.7
47.1
57.3
PTB
7.1
18.1
73.3
78.1
76.4
12.6
77.6
45.2
70.3
63.9
19.6
63.5
65.0
47.0
56.3
Alpaca
7.1
47.8
73.1
76.8
76.9
42.8
77.8
45.1
71.0
63.9
50.4
65.5
67.0
47.3
$7.5
Wikitext
9.0
15.3
70.5
72.7
70.2
42.4
76.8
16.7
61.1
61.1
53.1
66.5
70.5
51.8
61.2
@wan2. 5-
C4
9.0
44.2
70.4
71.6
70.6
40.6
76.3
46.9
65.0
60.7
51.8
65.5
68.2
52.7
59.6
9.1
42.4
68.5
70.5
69.7
40.6
75.5
46.3
59.7
51.2
65.6
68.4
52.8
59.5
Alpaco
9.0
46.2
72.5
74.5
70.4
38.6
76.4
46.6
65.4
61.3
52.2
65.1
71.1
55.3
60.9
Wikilesi
8.2
71.7
81.1
15.7
19.1
65.3
37.5
74.4
79.3
64.5
Owen2. 5-7g
C4
8.2
50.4
75.9
82.2
75.9
42.8
78.6
52,3
67.5
65.7
58.4
72.7
79.0
64.1
68.5
PTB
8.0
47.8
74.5
81.8
76.3
45.6
77.2
52.7
65.3
58.1
74.0
78.7
61.4
Alpaca
8.9
50.3
75.0
82.0
75.8
43.2
78.2
52.9
68.1
65.7
58.8
73.3
78.8
62.9
68.4
20


Reviews:
[{"review_id": "kQPC9D10t1", "reviewer": "Reviewer_UnBL", "summary": "This paper proposes ResQ, a post-training quantization (PTQ) framework that targets aggressive 4-bit quantization of large language models (LLMs) for weights, activations, and KV caches. The key idea is to identify and preserve a low-dimensional subspace of “important” activation components in higher bit precision (8-bit) while quantizing the remaining channels to 4-bit. Specifically, the method uses PCA to find the top-r principal directions of the activation distribution (one-eighth of the hidden dimension in many cases) and keeps those in 8-bit; the other channels go to 4-bit. Within each subspace (high-precision and low-precision), ResQ applies a random orthogonal rotation to suppress outliers. This approach aims to minimize overall quantization error, maintain near-baseline performance at 4-bit, and deliver competitive speedups over 16-bit inference. The authors integrate their projection matrices into the model architecture for minimal runtime overhead and benchmark ResQ extensively on multiple LLMs (Llama series, Qwen2.5, and Qwen2-VL) across language modeling, reasoning, and multimodal tasks. Results show perplexity and speedup gains with minimal additional calibration effort compared to other methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer UnBL, for your effort in reviewing our paper and acknowledging the strong empirical results and applicability of ResQ. We answer the listed questions below and incorporate your constructive feedback to further strengthen our work.\n1. **Heuristic theory:** We agree with the reviewer that most quantization error analyses, including ResQ, assume normally distributed activations. In practice, LLM activations deviate from normality. However, in ResQ, the activation distribution becomes approximately Gaussian after projection via the orthogonal matrix $U$ (as shown in Lemma 4.1). To support this, we compute the kurtosis of activations before and after projection, shown in Table 1 below. Post-projection activations $XU_l$ and $XU_h$ exhibit kurtosis near 3, indicating Gaussianity. Thus, our theoretical assumptions hold in practice. \nTable 1: Kurtosis of Activations before and after projection.\n|Model|Layer|$X$|$XU_l$|$XU_h$|\n|---|:---:|:---:|:---:|:---:|\n|Qwen2.5-3B|Attn|91.9$\\pm$38.8|3.0$\\pm$0.005|2.9$\\pm$0.07|\n||MLP|179.8$\\pm$248.3|3.0$\\pm$0.004|2.9$\\pm$0.07|\n|Qwen2.5-7B|Attn|75.6$\\pm$60.5|3.0$\\pm$0.002|3.0$\\pm$0.02|\n||MLP|164.7$\\pm$243.1|3.0$\\pm$0.0|2.9$\\pm$0.04|\n|Meta-Llama-3-8B|Attn|37.9$\\pm$50.3|3.0$\\pm$0.0|3.0$\\pm$0.02|\n||MLP|6.6$\\pm$1.4|3.0$\\pm$0.0|3.0$\\pm$0.02|\n2. **Distributed Inference setting :**  Please refer to Table 7 in comment to Reviewer vpAp.\n3. **Long Context Inference:** We provide per decoder speedup (similar to Fig. 5 in paper) for longer context lengths upto 20000 on Qwen2.5-32B and Meta-Llama-3-70B models in Table 2 below. With increasing sequence lengths, the improvements of ResQ slightly reduces still achieving 2.33x (2.02x) for Qwen2.5-32B (Meta-Llama-3-70B) speedup at sequence length of 20k. \n\nTable 2: Per decoder speedup of long context inference on RTX 3090.\n|Model||||Seq\\_len ||||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Model|8k|10k|12k|14k|16k|18k|20k|\n|Qwen2.5-32B|2.76|2.68|2.58|2.50|2.43|2.38|2.33|\n|Meta-Llama-3-70B|2.36|2.21|2.17|2.12|2.07|2.04|2.02|\n\n4. **Flexible Subspace Dimension:** It is possible to assign different ranks across layers for $U_B$ and $U_C$ projections since they are different for each layer. It is not possible to have different ranks for $U_A$ with the current approach since it is shared across layers. We test one approach of assigning different rank of 8-bit component to $U_B$ and $U_C$ based on the eigen value distribution of key and value. Specifically, for layers with higher eigen values, we keep 15.6\\% channels in 8-bit while for the rest we keep 9.3\\% channels in 8-bit achieving 12.5\\% high precision components in average. Results with such an approach (shown in Table 3 below) show that flexible rank improves the performance on reasoning accuracy but performs worse on MMLU demanding important future exploration. \nTable 3: Comparison of ResQ with a variant of ResQ having\nflexible rank of $U_B, U_C$.\n|Model|Method|Wiki.PPL|Avg.Reasoning acc.|Avg.MMLU acc.|\n|:---:|:---:|:---:|:---:|:---:|\n|Llama-3.2-3B|ResQ|8.8|59.0|**49.8**|\n||ResQ-flex.rank|**8.7**|**59.2**|49.0|\n|Meta-Llama-3-8B|ResQ|7.1|63.9|**57.2**|\n||ResQ-flex.rank|**7.0**|**64.3**|56.8|\n|Qwen2.5-3B|ResQ|**9.0**|61.1|61.2|\n||ResQ-flex.rank|**9.0**|**61.9**|**61.3**|\n|Qwen2.5-7B|ResQ|8.2|65.3|**69.0**|\n||ResQ-flex.rank|**8.0**|**65.4**|68.6|\n\nTask-wise results can be found [here](https://shorturl.at/KTiZR) in Table 1.\n\n5. **Codebook-based quant:** Codebook-based non-linear quantization methods (e.g., AQLM mentioned by the reviewer) focus solely on weight quantization, whereas ResQ takes a more holistic approach by quantizing weights, activations, and KV cache. To the best of our knowledge, no existing work explores non-linear quantization of LLM activations. This is an interesting direction, and we leave it for future work. Thank you for bringing AQLM to our attention; we will cite the work.\n\n6. **Calibration Dataset:** To analyse the impact of calibration datasets we evaluate performance of ResQ on 3 different datasets. These include two out of distribution language modeling datasets C4 and PTB and one instruction tuning dataset Alpaca. The results provided in Table 4 below shows no significant performance fluctuations with different calibration datasets. \n\nTable 4: Performance of ResQ with different calibration datasets.\n|Model|Calib.dataset|Wiki.PPL|Avg.Reasoning acc.|Avg.MMLU acc.|\n|:---:|:---:|:---:|:---:|:---:|\n|Llama-3.2-3B|Wikitext|**8.8**|59.0|**49.8**|\n||C4|**8.8**|**61.7**|48.6|\n||PTB|**8.8**|59.1|47.6|\n||Alpaca|**8.8**|58.9|48.0|\n|Meta-Llama-3-8B|Wikitext|**7.1**|63.9|57.2|\n||C4|**7.1**|**64.0**|57.2|\n||PTB|**7.1**|63.9|56.3|\n||Alpaca|**7.1**|63.9|**57.5**|\n|Qwen2.5-3B|Wikitext|**9.0**|61.1|**61.2**|\n||C4|**9.0**|60.7|59.6|\n||PTB|9.1|59.7|59.5|\n||Alpaca|**9.0**|61.3|60.9|\n|Qwen2.5-7B|Wikitext|8.2|65.3|**69.0**|\n||C4|8.2|65.7|68.5|\n||PTB|**8.0**|65.3|68.8|\n||Alpaca|8.9|65.7|68.4|\n\nTask-wise results can be found [here](https://shorturl.at/KTiZR) in Table 2.", "claims_and_evidence": "ResQ’s key proposition is that it can enable robust 4-bit quantization of weights, activations, and KV caches, largely closing the performance gap to 16-bit baselines. The authors support this by benchmarking on tasks such as Wikitext perplexity and MMLU zero-shot accuracy, showing clear improvements compared to established PTQ methods (e.g., GPTQ, SpinQuant). While the results strongly indicate that ResQ maintains accuracy in low-bit regimes, a gap remains in exploring full multi-GPU or large-batch deployment. Another core claim is that retaining a small low-rank subspace in higher precision is theoretically near-optimal. This is justified by a theoretical bound positing that the directions with largest eigenvalues dominate quantization error, and rotating activations in each subspace helps suppress outliers. However, it relies on assumptions of near-Gaussian distributions, which may not always hold under real-world activation behaviors. A third claim addresses ResQ’s runtime overhead, contending that by fusing projection matrices into existing layer weights, the impact on throughput is minimal. Indeed, single-block timing tests on an RTX 3090 GPU show up to 3× speedup over 16-bit baselines with only minor slowdowns relative to purely INT4 kernels. Yet, the paper does not provide extensive breakdowns of concurrency or other (or multi-) GPU scenarios, leaving some open questions about overhead at scale. Lastly, ResQ’s generalization to large and multimodal models is highlighted by successful application to 70B parameter Llama and Qwen2 VL, although overhead details for extremely large or specialized models are not examined in depth.", "methods_and_evaluation_criteria": "ResQ fits within the broader family of mixed-precision post-training quantization methods. The authors evaluate primarily on:\n\n- Wikitext Perplexity: A common measure of pure language modeling fidelity.\n- 0-shot accuracy on standard reasoning tasks (ARC, BoolQ, etc.) and MMLU to test knowledge retention and general understanding.\n- Generative tasks (GSM8K for math, code completion, summarization) to assess the approach on auto-regressive generation.\n- MMMU for multimodal comprehension using Qwen2-VL.\n\nSuch a diverse evaluation set is a strength: it shows that ResQ is robust across typical PTQ tasks (language modeling, reasoning, generation) as well as specialized tasks (multimodal). However, as with many PTQ papers, the chosen tasks mostly focus on correctness or perplexity rather than fine-grained analysis of speed–accuracy trade-offs under real-world concurrency or large-batch scenarios.", "theoretical_claims": "The authors prove an upper bound on quantization error under Gaussian assumptions, showing that the subspace-based approach is near-optimal in minimizing the Frobenius norm difference. Although typical in quantization research, the proof remains partly heuristic (due to normality assumptions), but this is consistent with contemporary PTQ literature.", "experimental_designs_or_analyses": "The authors primarily employ Wikitext perplexity and multiple 0-shot benchmarks—MMLU, ARC, BoolQ, HellaSwag, and more—to evaluate how well quantized models retain their reasoning or language modeling capabilities. They also include a set of generation tasks such as GSM8K math problems and code completion, offering a more comprehensive view of performance beyond classification or short-answer tasks. ResQ is compared against well-known baselines (e.g., GPTQ, SmoothQuant+, SpinQuant), and the consistent outperformance on perplexity and accuracy indicates that subspace-based 4-bit quantization indeed preserves essential model quality. The authors further examine a small but crucial set of ablations, for instance by removing or altering projection matrices for attention and feedforward blocks, and see noticeable drops in performance when these projections are omitted. Finally, while they do measure kernel-level speedups on a single GPU, there is comparatively limited exploration of multi-GPU scaling or diverse inference conditions. This narrower hardware focus, although still useful, suggests that additional profiling under larger-batch or distributed-serving scenarios would strengthen the paper’s overall claims regarding real-world usability.", "relation_to_broader_scientific_literature": "ResQ extends a line of LLM PTQ research focusing on 4-bit quantization with minimal accuracy loss:\n\n- GPTQ introduced Hessian-based weight-only quantization.\n- SmoothQuant, OmniQuant addressed outlier channels through amplitude scaling.\n- QUIK, QuaRot, SpinQuant used outlier or rotation-based methods to handle activation extremes.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths**\n- Good empirical results: Achieves near-best perplexities, 0-shot accuracies, and generative outcomes at 4-bit.\n- Implementation details: The authors describe how to fuse the projection matrices into the model to reduce overhead. They also measure actual runtime speedups, reinforcing practicality.\n- Applicability: The method is tested on a broad set of LLMs (1B to 70B+ parameters) and even vision-language models, suggesting decent generalizability.\n\n**Weaknesses**\n- Limited multi-GPU or concurrency analysis: Real-world inference typically runs parallel requests or large batches. The paper focuses on single-GPU, small-batch speed tests.\n- Random rotation overhead: While somewhat mitigated by fusing, there remains an unquantified overhead in larger contexts or many decoding steps, especially for the UD projection in the FFN.\n- Minimal exploration of extremely large contexts: Since KV cache quantization is a key selling point, more exhaustive or real-time scenarios with 8K+ tokens might better showcase memory/time improvements.\n- Heuristic theory: The analysis of normal-distribution-based error bounds is fairly standard in modern quantization, but it might under- or overestimate the actual distribution complexities in LLM activations.", "other_comments_or_suggestions": "None", "questions_for_authors": "- Have you tested how ResQ’s overhead scales in multi-GPU or distributed settings where activation projections might incur extra synchronization costs? Are the speedups consistent with single-GPU results?\n- For models that support context lengths beyond 8k tokens, how does ResQ perform in practice? Does the overhead of repeated rotation or subspace decomposition increase, or does it remain relatively stable?\n- You allow a flexible subspace dimension (e.g., d/8). Have you tested intermediate ranks or adaptive ranks per layer? How does the rank selection differ among different layers or for different activation distributions?\n- Could combining PCA-based subspace extraction with advanced codebook-based or cluster-based quantization (similar to AQLM) yield further gains? Or is the advantage of random rotation overshadowed by codebook overhead?\n- Since you observed that small calibration sets (128–512 samples) can suffice, do you see performance fluctuations if the calibration set is domain-specific vs. general Wikitext? Are there domain mismatches that degrade performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znjnF24RIV", "reviewer": "Reviewer_vpAp", "summary": "The paper proposes a novel algorithm to separate high/low values and respectively smooth and quantize them with different precisions. Theoretical analyses suggest that by introducing a designed matrix $P$, the upper bound of the error can be minimized. The experimental results illustrate the effectiveness and efficiency of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer vpAp, for your effort in reviewing our paper. We are delighted that you find our approach impressive. We provide response to your questions below. \n\n1. **Details about compute kernel**: The compute kernel goes beyond simple combination of INT4 and INT8 kernels. More precisely, the mixed precision quantization of activations into 4/8-bit components is handled by a single kernel call as opposed to two calls to the quantization kernel. Further, the hardware implementation involves quantizing and packing the mixed precision kv cache for memory efficiency. We will provide more details in the camera ready version of the paper and will release the code upon acceptance of the paper. Additionally, we compare the compute kernel with simple combination of INT4 and INT8 kernels as mentioned by the reviewer below (Table 5 below). **The inference latency with ResQ is upto 1.3x lesser than simple combination of INT4 and INT8 kernels.** \n\nTable 5: Per decoder latency (in ms) of ResQ over simple compute kernel.\n| Model | Seq\\_len | Simple | ResQ | Improv. |\n|:---:|:---:|:---:|:---:|:---:|\n| Llama-3.2-3B | 512 | 1.8 | 1.3 | 1.33 |\n|  | 8192 | 27.6 | 20.5 | 1.34 |\n| Meta-Llama-3-8B | 512 | 2.6 | 2.0 | 1.3 |\n|  | 8192 | 40.9 | 31.9 | 1.29 |\n| Qwen2.5-72B | 512 | 5.9 | 5.0 | 1.16 |\n|  | 8192 | 95.3 | 85.5 | 1.11 |\n\n2. **Memory reduction with ResQ:** We show memory usage of ResQ on RTX 3090 (24GB), FP16 baseline and Quarot (INT4) baseline at different sequence length in Table 6 below. **ResQ consumes 1.84-3.08x lower memory than FP16 baseline** and requires slightly (4-11\\%) more memory compared to QuaRot. Further, ResQ supports inference of Qwen2.5-14B where FP16 baseline runs into OOM error.\n\nTable 6: Memory of ResQ and baselines on RTX 3090.\n| Model | seq\\_len | FP16 | QuaRot | ResQ | Improv.  over FP16 | Improv. over QuaRot |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Meta-Llama-3-8B | 8192 | 21.9 | 11.4 | 11.9 | 1.84 | 0.96 |\n|  | 2048 | 16.7 | 6.8 | 7.2 | 2.31 | 0.94 |\n|  | 512 | 15.4 | 5.6 | 6.1 | 2.54 | 0.93 |\n| Llama-2-7B-hf | 8192 | 18.1 | 6.2 | 6.8 | 2.66 | 0.91 |\n|  | 2048 | 13.9 | 4.2 | 4.7 | 2.95 | 0.89 |\n|  | 512 | 12.9 | 3.7 | 4.2 | 3.08 | 0.89 |\n| Qwen2.5-14B | 8192 | OOM | 19.5 | 21.3 | -- | 0.92 |\n|  | 2048 | OOM | 14.0 | 14.9 | -- | 0.94 |\n|  | 512 | OOM | 12.6 | 13.5 | -- | 0.93 |\n\n3. **Inference on NVIDIA A100**: Our intention with ResQ was to target inference on consumer devices, which is why we report speedups on an RTX 3090 at batch size 1. While datacenter-scale 4-bit LLM inference is valuable future work, we also provide latency results for Meta-Llama-3-70B on NVIDIA A100 server (Table 7 below). Notably, **ResQ enables the 70B model to fit on a single GPU**, whereas the FP16 baseline requires three GPUs. In this setting, ResQ runs data-parallel inference, while FP16 uses model parallelism.  ResQ achieves up to 4.98× lower latency across various batch sizes and sequence lengths. More sophisticated model parallel inference approaches like pipeline parallelism will only improve throughput of FP16 baseline but will not improve per batch latency.\n\nTable 7: Meta-Llama-3-70B inference latency (in ms) on 3x NVIDIA A100 GPUs.\n| batch\\_size | seq\\_len | FP16 | ResQ | Improv. |\n|:---:|:---:|:---:|:---:|:---:|\n| 3 | 10240 | 4242 | 20783 | 4.90x |\n| 3 | 8192 | 3361 | 16373 | 4.87x |\n| 3 | 4096 | 1609 | 7871 | 4.89x |\n| 3 | 2048 | 806 | 3888 | 4.82x |\n| 6 | 2048 | 1560 | 7733 | 4.96x |\n| 9 | 2048 | 2309 | 11493 | 4.98x |", "claims_and_evidence": "The claims made are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for the problem.", "theoretical_claims": "I have not carefully checked the proofs of the theoretical claims. However, the claims make sense and are likely to hold.", "experimental_designs_or_analyses": "The experiment is well-structured. However, from my perspective, there are two issues that should be addressed:\n\n1. Memory reduction compared to FP16, INT4, and/or other baseline methods should be provided.\n\n2. Is it possible to validate speedup on other NVIDIA GPUs, such as the A100?", "relation_to_broader_scientific_literature": "The key contributions can be summarized as follows: The proposed method employs PCA-like permutation and rotation matrices to effectively mitigate outliers in WA quantization and enhance performance, surpassing SOTA methods such as QuaRot and SpinQuant. The proposed method is also validated through speedup comparisons, further illustrating its efficiency, leading to the development of (mixed-precision) WAKV quantization.", "essential_references_not_discussed": "To the best of my knowledge, there are no missing references.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method is novel and efficient. The use of the PCA method addresses the difficulties in separating high/low values in $X$, contributing to the success of the mixed-precision quantization method, which is quite impressive.\n\n2. The experiment is solid and persuasive in supporting the proposed method. The results are strong, and the overhead is acceptable.\n\n3. The paper is well-structured, making it easy for readers to follow.\n\nBtw, this is quite an impressive paper I have reviewed.\n\nWeakness:\n\nAside from the two issues mentioned in the \"Experimental Designs or Analyses\" section, there is one more weakness: The authors should try to provide more details regarding the ResQ kernel. Is it a simple combination of INT4 and INT8 kernels with an addition? If so, I believe that developing CUDA kernels alone can hardly be considered a main contribution in the introduction section.", "other_comments_or_suggestions": "No additional comments or suggestions.", "questions_for_authors": "See weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G42Y3MWRo0", "reviewer": "Reviewer_x9DL", "summary": "The paper presents ResQ, a mixed-precision post-training quantization (PTQ) method for large language models (LLMs). The core idea of ResQ is to compute the orthogonal transformations using PCA and decompose the orthogonal matrices for high-precision and low-precision based on their corresponding eigenvalues. Moreover, high-precision weights and activations are cast in 8-bit. Experiments demonstrate that ResQ outperforms strong baselines such as SpinQuant, QuaRot, and QUIK.\n\n**Update after rebuttal**: My latest reply reflected my final update.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer x9DL, for your effort in reviewing our paper. We appreciate your recognition of the strength of ResQ's experimental section. Below, we respond to the concerns you raised.\n\n1. **Claim regarding quantization error:** ResQ's approach of choosing coefficients along principal eigen vectors of activations to keep in 8-bit is indeed **optimal to minimize local quantization error**. Projection with PCA basis $P$ minimizes quantization error across high/low precision groups while random orthogonal matrix $R$ improves quantization error within high/low precision group. We will further clarify this point in the main paper. While SpinQuant learns rotation matrices minimizing final loss, it still keeps all the components in 4-bit which does not minimize quantization error enough while ResQ intelligently performs mixed precision quantization. Moreover, SpinQuant's training of rotation matrices can additionally be incorporated in ResQ as well at the cost of high calibration time. The 8-bit components can be chosen via PCA basis ($P$) and orthogonal matrix to minimize quantization error within 4-bit and 8-bit quantization groups ($R$) can be learned minimizing output loss. Such an approach further improves performance of ResQ as shown in Table 8 below surpassing SpinQuant by even higher amount.\n\nTable 8: Performance of ResQ and variant of ResQ which trains rotation $R$. \n| Model | Method | Wiki. PPL | Avg. Reasoning acc. | Avg. MMLU acc. |\n|:---:|:---:|:---:|:---:|:---:|\n| Meta-Llama-3-8B | ResQ | 7.1 | 63.9 | 57.2 |\n|  | ResQ + training $R$ | **7.0** | **64.5** | **58.3** |\n| Llama-2-7B | ResQ | **5.8** | 62.0 | 37.7 |\n|  | ResQ + training $R$ | **5.8** | **62.2** | **38.0** |\n| Qwen2.5-7B | ResQ | 8.2 | 65.3 | **69.0** |\n|  | ResQ + training $R$ | **8.0** | **65.8** | **69.0** |\n| Llama-3.2-1B | ResQ | 12.4 | **50.1** | 29.4 |\n|  | ResQ + training $R$ | **11.7** | **50.1** | **29.6** |\n\nDetailed task-wise results for the above table can be found [here](https://shorturl.at/KTiZR) in Table 3.\n\n2. **Iso-bitwidth comparison:** Both ResQ and previous state of the art mixed precision quantization technique, QUIK (EMNLP2024), keep $\\frac{1}{8}$ channels in 8-bit which brings average bit-width to 4.5 bits. For other baselines, achieving fractional bit-width is impossible with uniform precision quantization across layers and channels. Additionally, as suggested by the reviewer, we perform comparison at equal bit-width of 4-bit. We create a variant of ResQ which keeps first $\\frac{d}{8}$ channels corresponding to lowest eigen value of $P$ in 2-bit, $\\frac{d}{8}$ channels corresponding to highest eigen value of $P$ in 6-bit and remaining in 4-bit to achieve average bit-width of 4-bit. As shown in Table 9 below, **even at iso-bitwidth of 4-bit, ResQ outperforms SpinQuant and QuaRot** highlighting its capabilities. \n\nTable 9: Iso-bitwidth comparison between ResQ, QuaRot and SpinQuant.\n| Model | Method | Wiki. PPL | Avg. Reasoning acc. | Avg. MMLU acc. |\n|:---:|:---:|:---:|:---:|:---:|\n| Qwen2.5-3B | QuaRot | 68.8 | 47.7 | 28.9 |\n|  | SpinQuant | 70.6 | 48.6 | 32.8 |\n|  | ResQ | **9.8** | **59.1** | **52.2** |\n| Qwen2.5-7B | QuaRot | 4e3 | 38.4 | 24.1 |\n|  | SpinQuant | 3e3 | 38.6 | 24.3 |\n|  | ResQ | **34.2** | **56.2** | **58.0** |\n| Qwen2.5-14B | QuaRot | 6.8 | 67.1 | 70.9 |\n|  | SpinQuant | 6.6 | 67.4 | 70.1 |\n|  | ResQ | **6.5** | **67.5** | **71.3** |\n| Qwen2.5-32B | QuaRot | 6.1 | 67.8 | 77.0 |\n|  | SpinQuant | 6.0 | 67.9 | 77.6 |\n|  | ResQ | **5.9** | **69.1** | **77.9** |\n| Qwen2.5-72B | QuaRot | **4.9** | 70.3 | 80.1 |\n|  | ResQ | **4.9** | **71.1** | **80.1** |\n\nDetailed task-wise results for the above table can be found [here](https://shorturl.at/KTiZR) in Table 5.", "claims_and_evidence": "[Correct Claims]\n\n* Custom cuda kernels to speed up inference.\n* Theoretical analysis on PCA-based projections.\n* ResQ outperforms recent uniform and mixed precision PTQ methods. -> partially correct and I have some comments in Weakness in Experimental Designs Or Analyses.\n\n[Problematic Claims]\n\n* In line 29, the paper claims using PCA to compute orthogonal matrices is \"provably optimal mixed precision quantization scheme\". However, in theorem 4.2, the method is optimal in minimizing layer loss, but it may not be optimal on the final output loss, and the latter is done in SpinQuant. Therefore, it would be great to tune down this claim a bit, and I am curious why ResQ performs better than SpinQuant (please see my comment in Weakness in Experimental Designs Or Analyses).", "methods_and_evaluation_criteria": "* The datasets and baselines make sense.", "theoretical_claims": "I didn't check the proof of Theorem 4.2.", "experimental_designs_or_analyses": "[Strength]\n\n* The experiments are conducted on multiple models and benchmarks.\n\n[Weakness]\n\n* For the compared methods, some are uniform and some are mixed-precision quantization, and it is unclear how many additional bits the mixed-precision methods used. There should be a column in the tables about the number of bits for each method for a fair comparison. It is also fairer to use the same bitness for all the approaches, and maybe the authors can adjust the number of bits for weights or the scale and zeros that are introduced in quantization to do it.", "relation_to_broader_scientific_literature": "* PCA for orthogonal transformation -> related to low-rank decomposition, like LoRA and other rotation methods (QuaRot, SpinQuant).\n* The method combines rotation and mixed-precision quantization -> which is an interesting combination from previous methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* As the PCA part is the main novelty of this paper, it would be great to have a standalone section or at least a paragraph to introduce it. I spent some time to finally find it is in Sec 4.2.\n* In line 219, defining how to compute the quantization SNR would be great as it seems to pop out from nowhere here.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qk1iNo240q", "reviewer": "Reviewer_Azbj", "summary": "This paper introduces ResQ, a post-training quantization (PTQ) method for large language models (LLMs) that enables mixed-precision quantization of weights, activations, and KV caches. Experimental results demonstrate that ResQ achieves superior performance compared to existing methods.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer Azbj, for putting effort in reviewing our paper. We provide a response to your concerns below. \n\n1. **Limited Novelty:** While we agree with the reviewer regarding the statement that ResQ is a rotation and mixed precision quantization based approach, we respectfully disagree regarding the novelty aspect of our work. To the best of our knowledge, there is no work which does what ResQ does and we would be interested if the reviewer can point us to the related paper. In Table 4 of main paper, we also show that ResQ outperforms a hypothetical outlier+rotation based approach which is an amalgamation of related baselines QUIK (EMNLP2024) and QuaRot (NeurIPS2024). This highlights optimality of PCA based high precision component extraction which again to the best of knowledge has not been explored before in literature. Additionally, we provide even more comprehensive comparison in the Table 10 below where we compare ResQ and outlier+rotation baseline's Reasoning and MMLU accuracy in addition to Wikitext perplexity for Qwen2.5 models. Emphasizing that **ResQ outperforms the outlier+rotation baseline across the stack**. Based on these insights we humbly request the reviewer to re-evaluate their stance on novelty of our work. \n\nTable 10: Comparison of ResQ with outlier+rotation approach.\n| Model | Method | Wiki. PPL | Avg. Reasoning acc. | Avg. MMLU acc. |\n|:---:|:---:|:---:|:---:|:---:|\n| Qwen2.5-3B | outlier+rot. | 9.4 | 58.4 | 59.2 |\n|  | ResQ | **9.0** | **61.1** | **61.2** |\n| Qwen2.5-7B | outlier+rot. | 10.5 | 64.1 | 64.9 |\n|  | ResQ | **8.2** | **65.3** | **69.0** |\n| Qwen2.5-14B | outlier+rot. | 6.4 | 68.0 | 73.8 |\n|  | ResQ | **6.2** | **69.2** | **74.6** |\n\nDetailed task-wise results for the above table can be found [here](https://shorturl.at/KTiZR) in Table 4.\n\n2. **CUDA Kernel:** Thank you for highlighting this point. We will include additional details about the CUDA kernel in the main paper and plan to release the code upon acceptance. The CUDA Kernel involves, the mixed precision quantization of activations into 4/8-bit components within a single kernel, low precision GEMM of 4-bit and 8-bit operands and a fused dequantization of 4-bit and 8-bit results. Further, the hardware implementation involves quantizing and packing the mixed precision kv cache for memory efficiency. Additionally, we provide several new hardware-related results to address the reviewer's concerns: improved memory usage with ResQ (Table 5, response to Reviewer vpAp), speedup achieved by ResQ on long context lengths (Table 2, response to Reviewer UnBL), inference latency in a distributed setting representative of datacenter workloads (Table 7, response to Reviewer vpAp), and a comparison of ResQ with a baseline quantization kernel (Table 5, response to Reviewer vpAp). We hope these additions comprehensively address the concerns regarding the CUDA kernel.\n\n3. **Figure 4:** ResQ and SpinQuant project weights and activations by projection matrices in similar manner. While SpinQuant trains the projection matrices and utilizes uniform precision quantization across the layers, ResQ uses PCA basis and random orthogonal rotations as projection matrices and performs mixed precision quantization. The key differentiation in Figure 4 is the layers and activations which are quantized to mixed precision. For ResQ all the layers and activations except down\\_proj layer is mixed precision.", "claims_and_evidence": "The paper claims the development of custom CUDA kernels as a main contribution. However, this claim is not adequately supported. There is no detailed description of these kernels in the manuscript, and the anonymous code link does not appear to contain their implementation or sufficient details to verify this claim.\n\nFurthermore, the fast hadamard transformation is well-established in prior work (e.g., https://github.com/Dao-AILab/fast-hadamard-transform). The paper does not clearly propose any novel contribution.", "methods_and_evaluation_criteria": "The choice of mixed-precision quantization and a rotation-based approach is reasonable, given the established effectiveness of both techniques in addressing quantization challenges. However, the novelty of combining these specific approaches is not fully confirmed.", "theoretical_claims": "The proofs for the theoretical claims in this paper are correct.", "experimental_designs_or_analyses": "The experimental design generally follows established practices in the field.", "relation_to_broader_scientific_literature": "The key contributions of the paper do not appear to introduce significant new findings or ideas compared to the existing scientific literature.", "essential_references_not_discussed": "The paper appears to comprehensively cover the relevant related works and does not seem to have omitted any essential references.", "other_strengths_and_weaknesses": "A primary concern is the limited novelty of the proposed approach. While the paper combines mixed-precision quantization and a rotation-based method, this combination lacks significant new insights. The work feels more like a technical report demonstrating the application of existing techniques rather than a substantial research contribution.", "other_comments_or_suggestions": "A significant claim made in the paper is the development of custom CUDA kernels for performance acceleration. However, this claim is not supported by sufficient evidence within the paper. There is no dedicated section or subsection detailing the design, implementation, or optimization strategies employed in these kernels.", "questions_for_authors": "Figure 4 in the submission appears to be highly similar (**only color changed?**) in structure and content to Figure 1 in SpinQuant paper (https://arxiv.org/pdf/2405.16406). Could the authors clarify the relationship between these two figures and explain any differences or novel aspects of Figure 4 in the current work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces ResQ, a post-training quantization (PTQ) which targets weight, activation, and KV-cache tensors for quantization.  Novelly, ResQ applies mixed-precision, targeting a low-rank subspace of critical (high-variance) activation components which are kept in 8-bit precision, while remaining components are aggressively quantized to 4-bit precision.  Invariant random rotation is subsequently applied to further suppress outliers.   The paper theoretically proves that this mixed-precision scheme is theoretical near-optimal, assuming normality (which is standard in LLM quantization, e.g., NF4).  The paper extensively considers the Llama-3 (Llama-3-8B, Llama-3-70B, Llama-3.2-1B, Llama-3.2-3B) and Qwen 2.5 (Qwen2.5-3B, Qwen2.5-72B) models, benchmarks (Wikitext perplexity, average 0-shot common sense reasoning, 0-shot MMLU accuracy, 5-shot GSM8K performance, and LongBench performance), and SOTA PTQ methods ( (e.g., GPTQ, SmoothQuant+, SpinQuant), showing that ResQ significantly outperforms other methods at retaining overall performance.  Finally, the paper demonstrates 1.61 to 3.0 speedups compared to 16-bit inference, with minor slowdowns compared to native int4 inference.

The majority of the reviewers agree that this is a high-quality paper, that both empirical and theoretical results are impressive, and that the novelty of ResQ is an interesting innovation over recent SOTA PTQ methods (e.g., SpinQuant).  Reviewers had several concerns, e.g., Reviewer UnBL questioned the appropriateness of normality for the main theoretical results, Reviewer vpAp questioned the depth of the fused kernel contributions and inclusion of higher-end GPUs (i.e., A100s), and Reviewer UnBL requested a direct 4-bit (non-mixed precision) comparison to other SOTA PTQ methods.  The authors did well to address all criticisms, resulting in raised reviewer ratings (I look forward to the inclusion of these additional experiments in the camera ready).  Reviewer Azbj initially voted in favor of rejection, however, the authors addressed these concerns in the rebuttal and the reviewer did not subsequently engage in further discussion with either the AC-reviewers or authors (for further clarifications).
Decision: Accept (spotlight poster)