SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Authors: Yeonsik Park
Hyeonseong Kim
Seungkyu Choi
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Quantization
PTQ
LoRA
Error Reconstruction
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose SERQ, a saliency-aware error reconstruction method for low-bit (W4A4) LLM inference that employs only a single low-rank compensation matrix by jointly addressing activation- and weight-induced quantization errors.

Abstract: Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity.

SERQ: Salieney-Aware Aware Low-Rank Error Reconstruction for LLM Quantization
SERQ: SALIENCY-AWARE LOW-RANK ERROR RECONSTRUCTION FOR LLM QUANTIZATION
Yeonsik Park
Hyeonseong Kim Kyung Hee University
Seungkyu Choi Yonsei University
Kyung Hee University plsik125e khu. ac. kr
hssoun2222ekhu.ac..k
seungkc@yonsei.ac.kk
ABSTRACT
Post-training quantization (PTQ) has emerged as prevailing technique for de- ploying large language models (LLMs) efficiently terms both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quan- tization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces lightweight auxiliary computation path without requiring heavy optimization additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conven- tional low-rank adaptations rely on two sequential factors, necessitating interme- diate quantization during inference and thereby limiting low-precision efficiency. this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error re- construction, and (3) offline weight permutation. The method incurs additional computation only for low-rank eгror reconstruction via single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically. SERQ outperforms prior eтror reconstruction methods un- der both W4A8 and W4A4 settings, and achieves higher accuracy than state-of- the-art Dotation-based W4A4 approaches, while substantially reducing calibration complexity. Code available at inttps://ght thub.... com/ aca abys SERQ
INTRODUCTION
The demand for efficient deployment of large language models (LLMs)) has been rapidly increasing across both server and edge platforms. Quantization has emerged proaches to reduce the substantial memory and computational costs associated with LLM inference. In particular, post-training quantization (PTQ) techniques (Nagel al. (202I)) enable low-precision representations of weights and activations involved in large-scale computations, thereby avoiding expensive fine-tuning while maintaining competitive performance.
of the most effective ap-
A central challenge in minimizing quantization errors for LLMs lies in addressing outlier activa- tions across channels. To alleviate this issue, several distribution flattening approaches have been proposed, including pre-quantization scaling methods (Xiao et al. (2024); Shao et al. (2024)) and online transformation-bart based techniques that leverage random Hadamard or learned transformations Ashkboos et al. (2024); Liu et al. (2025)). While recent rotation transformation methods have demonstrated effectiveness in enabling 4-bit integer (INT4) quantization, they typically rely on computationally expensive calibration procedures suffer from performance variability induced by random matrices, thereby limiting their practicality in general deployment.
An alternative strategy for mitigating activation outliers is matrix decomposition. Recent advances have introduced low-rank error reconstruction methods that integrate quantization with low-rank
adaptation (LoRA) (Dettmers et al. (2023); Saha et al. (024); Zhang al. (2024); Zhao et al.

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
These approaches leverage low-rank decompositions in matrix multiplication to reduce quantization error by compensating for it through separate low-rank factors. For example, L'QER Zhang et al. (0224) introduces fully quantized path for low-rank error reconstruction, yielding near-zero accuracy loss under the 4-bit weight, 8-hit activation (W4A8) configuration. Despite their superior adaptability, these methods have yet achieved W4A4 quantization without noticeable performance degradation. Furthermore, they remain unsuitable for fully low-precision execution, as decomposed matrices are multiplied sequentially, producing intermediate values, requiring an additional on-the-fly fly quantization process.
2024
In this work, we propose SERQ, a saliency-awart eπror reconstruction method that enables low- precision LLM inference (e.g., W4,44 W4A8) using single low-rank deconposition. Unlike stan- dard low-rank approximations that rely on two low-rank factors, our method unifies error correction into single matrix by jointly addressing activation and weight saliency. This design avoids the overhead of an additional sequential low-rank branch during inference, while effectively mitigat- ing quantization errors arising from both activation outliers and salient weights. As result, SERQ achieves more efficient low-precision inference while maintaining high accuracy, outperforming prior quantization approaches in the challenging W4A4 precision setting.
To the best of our knowledge, this is the first work to realize 4-bit matrix multiplication in linear layers by employing low-rank error reconstruction, method recognized for its adaptability and minimal calibration overhead. Following this principle, SERQ introduces no additional layers for online processing and avoids costly calibration procedures such as hyperparameter search, other compute-intensive training operations. Our contributions are summarized as follows.
We propose SERQ, novel W4A4 quantization scheme for LLMs that employs a single saliency-guidees low-rank matrix for accurate error reconstruction. Our method operates in three steps: static activation flattening, saliency-aware enror reconstruction, and offline weight permutation.
SERQ enables 4-bit matrix multiplication (e.g. INT4, MXFP4) in linear layers, thereby minimizing inference overhead in low-rank computation. Moreover, the proposed flat- tening and permutation schemes are merged into weight parameters and preprocessed in adjacent layers, allowing them to be fully managed offline with no additional latency.
We validate our scheme across various LLMs with comprehensive evaluations. Compared to prior LoRA A-bsed methods, our approach achieves superior performance in both W4A8 and W4A4 configurations while using only a single low-rank matrix. Furthermore, we com- pare against statc-of-the-art rotation- ootation-based W4A4 quantization approaches, demonstrating superior accuracy while significantly reducing calibration complexity.
BACKGROUND AND MOTIVATION
2.1 LLM QUANTIZATION
Quantization maps high-precision values to low-precision representations, improving both memory and compute efficiency. The basic integer quantization with max-scaling can be expressed as:
X, = clip([X/8]]
8 = (max((X))/Ee (1) -1), x 8 Xg
where s is the scale factor, n2 is the bit-width, [-] denotes round to to-nearest, and clip() clamps values to [-2n-1 1, 2n 1]. Reducing bit-width provides near-linear storage compression, allevi- ating the growing memory demands of model parameters and K KV-cache, At runtime, quantization also reduces memory traffic, thereby improving bandwidth efficiency. When activations are quan- tized in addition to weights, the core linear operation can be executed as an integer GEMM, further accelerating inference latency and throughput. For a transformer linear projection y Wæ,
y Sw (W xx)xx
As widely recognized, the accuracy of LLM quantization for both weights and activations critically depends on how activation outliers are handled. Numerous approaches have been proposed to miti- gate the quantization error introduced by such outliers. Early methods, such as SmoothQuant (Xiao)
2

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
et al. (2024) and OmniQuant (Shao et al. (2024)), employ distribution flattening techniques that
balance activations and weights through pre- queen-izaizziinnn scaling. Another line of work decomposes matrix multiplication by assigning separate sigh-precsiion path for outliers, in LLM. int 8
(Dettmers et al. (2022) and QUIK (Ashkboos et al. (023)). ). However, these primitive techniques exhibit significant accuracy degradation when applied to sub-8-bit L.Mss.
More recently, online transformation techniques that rotate tensors to flatten the distribution have demonstrated effectiveness for 4-bit quantization with minimal latency overhead. Quarot (Ashkboos)
et al. 2024)) applies random Hadamard transformations, while SpinQuant (Liu et al. 2025) ) learns rotation matrices to suppress outliers and reduce the performance variance. While effective in the W4A4 setting, existing methods still incur notable accuracy loss and either suffer from high variance due to random Hadamard matrices or require costly training to optimize transformation matrices, limiting deployment practicality. Meanwhile, LoRA-based LLM quantization has emerged as powerful approach for mitigating quantization errors. low-rank factors via matrix decomposition has proven particularly effective, while avoiding heavy optimization or additional online layers. Further details are discussed in section 2,2
similar vein, error reconstruction with
2.2
LOW-RANK ERROR RECONSTRUCTION
Recent advances in LoRA for parameter-efficient fine-tuning of foundation LLMs suggest its po- tential for applicability to LLM quantization well. LoRA has been adapted to compensate for quantization errors by leveraging the auxiliary path of low-rank matrices, a strategy we refer to as low-rank error reconstruction. This approach introduces LoRA-style compensator that restores the principal quantization error without requiring heavy retraining or complex deployment modifi- cations. Concretely, it corrects the quantization error of full cull-precision weight matrix Wdxd, where d denotes the hidden dimension of a layer, by augmenting its low-bit proxy with low-rank factors.
W = Q(W) + L2 L W Q(W)
Where Q(W) is the quantized weight, L1 E Raxr, L2 E Rexd, and < the rank. The parameter and compute overhead of the low-rank factors is only 2rd; with commonly used ranks r E (32, 64) (e.g.. d 4096), this amounts to about 1.6-3.1% %. Moreover, the LoRA-path is simple and lightweight, which makes it highly applicable across diverse model architectures and hardware platforms. Consequently, it has emerged as prominent and widely adopted quantization method.
Gradient-based Methods. Gradient-based methods compensate the quantization errors by lever- aging loss gradients on a small calibration set to learn low-rank factors, typically instantiating a compensator L2 that reduces mismatch between the full full-precision model and its quantized coun- terpart. QLoRA (Dettmers et al. (2023) advanced this line of work by fine-tuning LoRA adapters on top of 4-bit weights, recovering task performance with modest resource overhead. At sub-4-bit precision, methods such as LQ-LORA (Guo et al. (2024). LoftQ (Li et al. (202)) and QA-LoRA (Xu et al. (2023) consistently mitigate degradation and sustain competitive accuracy, highlighting the viability of ultra-low-bit quantization.
SVD-based Methods. Another approach is to reconstruct quantization errors by low-rank matri- ces obtained via singular-value decomposition (SVD). This approach forms the compensator L, L from the quantization error E by exploiting its rapidly decaying singular spectrum, allowing the dominant eиror to be captured with compact low-rank factors. Recent methods such as ZeroQuant- v2 (Yao al. (2023)), CALDERA (Saha et al. (0024)) and I.2QER (Zhang al. (2024)) adopt this SVD--based optimization to extract low-rank factors in a training-free and lightweight calibration process. In particular, L'QER formalizes weight-activation quantization using SVD-based error re- construction, preserving integer matrix multiplications across both the main and low-rank branches.
2.3 DEPLOYMENT OF LOW-RANK ERROR RECONSTRUCTION
In low-rank егтоr reconstruction, linear layer introduces an additional computation path for the low-rank branch, which cannot be parallelized cleanly with the main branch. Consider the case where both weights and activations are quantized. To enable the main branch to leverage low- precision matrix multiplication (c.g.. INT4 or MXFP4 GEMM kernels), the -aggmeente linear layer must be evaluated in its decomposed form, such as (W,,, L, L2) W,, X, X, L2-
3

---

SERQ: Saliency-Awand Low-Rank Error Reconstruction for LLM Quantization
EP16
INTB
Scatter
X
LLM.int8()
LQER
SERQ
Figure 1: Computation flow of linear layer under different matrix decomposition methods. LLM. intt () employs mixed-precision scheme of INT8 and FP16 by assigning separate com- putation paths for outliers and non-outliers. L'RRER applies SVD-based error reconstruction with a mixed-precision path of INT4 and INT8. In contrast, the proposed SERQ leverages a saliency- guided low-rank matrix and provides unified computation path with INT4 or MXFP4 precision.
This requires two sequential matrix multiplications with low-rank matrices. Although each operation relatively inexpensive, the overhead becomes non-negligible when the main branch benefits from highly optimized low-precision GEMM kernels. To address this, L'QER Zhang et al. 22024) quan- tizes the low-rank matrices as well, allowing the entire computation to proceed in a fully quantized path, as expressed X, Wg + Q(X,L1, L2.g. However, this formulation introduces an additional on-the-fly fly quantization step for intermediate results between the sequential multiplications, resulting in inefficiency for ow-prrccssion deployment.
Figure [ shows the overall datapath of prior approaches compared to ours, all based on decomposed matrix multiplication. Prior to eтrоr reconstruction methods, LLM. int () (Dettmers et al./ (2022) handled outliers via separate high-precision path, but this required on-the-fly scattering and FP16 computation, introducing substantial latency. An error reconstruction method I, L'2QER (Zhang et al. 22244) later achieved fully quantized datapath with mixed precision, yet still relied on on-the-fly quantization and two additional narrow matrix multiplications, resulting in latency and degraded accuracy in the W4A4 setting. In contrast, propose SERQ. novel error- reconstrtttttn method that employs single low-rank matrix, eliminating on-the-fly quantization and enabling a fully 4-bit end-to-end computation path. Further details are provided in section
METHOD
We present SERQ, saliency-aware error reconstruction method that jointly accounts for weight and activation saliency within a single low-rank matrix. We first show that saliency plays a central role in compensating weight-side quantization eиror, motivating saliency-guided low-rank design. We then detail how SERQ incorporates activation saliency into the weight matrix, enabling error reconstruction based integrated weight saliency while minimally impacting inference latency.
3.1 SALIENCY-AWARE LOW-RANK ADAPTATION
Prior emoorrecoostrructinn methods typically approximate the full-weight quantization еттоr E W Q(W) using a truncated SVD of the weight matrix, E U,E,V," where U, Rd and V E IR contain the top-r singular vectors and Σ, E IRrr is the diagonal matrix of the corresponding singular values. While effective, this global decomposition overlooks where the error actually concentrates: the fixed rank budget is distributed across all rows and columns, diluting capacity on the most problematic regions (c.g., salient weights). In addition, quantizing the low-rank factors themselves introduces further loss, undermining the efficiency of error reconstruction.
Building on the theoretical insight that quantization errors vary across weight rows in linear opera- tions, AWQ (Lnn et al. (2024b)) demonstrates that protecting only small fraction (~1%) of salient channels (i.c., their corresponding weight rows) using the activation distribution can substantially reduce the Motivated by this, evaluate error reconstruction using SVD with fixed rank
4

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
@a) Calibration
Inference
g
3 tnd
Permute
X E R feexd
WiNT g
SERQ
(b)
Self-Attention
Feed-Forward Network
Rope
W
Layrer-atte mtortt
Quant
RoPE
Ps. Pa.
Permute
Figure 2: (a) Overall SERQ implementation. During calibration, saliency rows are determined via activation scaling, followed by weight row permutation. During inference, error reconstruction is performed through a residual path computed only on the salient components, alongside the main path. (b) Computation flow of decoder layer. The merged row- and column- -wie weight permuta- tion enables offline preprocessing of both current weight rows and subsequent activation channels.
size but restricted to salient rows selected by activation scales (see Appendix A.1). We find that selecting only small number of salient rows for SVD improves perplexity compared to using the entire matrix. In other words, rather than extracting ranks from the full weight matrix via SVD, we directly identify rank candidates based row-wise saliency, allowing to reconstruct quantization error only on the most influential data. This method requires only a narrowed matrix from these salient rows, equal the rank size, ensuring high-fidelity approximation while enabling a single matrix decomposition for the low-rank branch. In section 3.2 we describe how to integrate activa- tion statistics into the weight matrix determine salient rows that disproportionately affect linear operations, and how to jointly reconstruct the error to improve accuracy using full low-precision computing path.
3.2
IMPLEMENTATION
Static Activation Flattening. Activation quantization is notoriously fragile due to channel-wise outliers. Existing methods often rely on online outlier-handling techniques (e.g., rotation transforms auxiliary layers), which effective but introduce additional latency. Instead, since method provides a residual path for error reconstruction at the linear layer, we avoid the online flattening process and revisit SmoothQuant (Xiao et al. ). which flattens activation distributions using static per-channel scaling. Specifically, activations are sealed by a factor s, and the corresponding scale is folded into the weights. The operation in linear layer can therefore be expressed
Y XW (X diag(s (diag(s W) XW
The scaling factors are obtained during calibration and merged into adjacent layers offline, incur- ring no runtime overhead. While this strategy alleviates outliers, shifts the quantization burden onto the weights, increasing the difficulty of weight quantization. However, unlike standalone prior approaches, the combined use of low-rank reconstruction enables effective compensation for the induced weight errors. Concretely, SERQ identifies salient weights after the flattening step and restores the residuals using a single low-rank matrix.
Saliency-Awan Error Reconstruction. The per-channel static flattening process pushes the scale of activation outliers into the corresponding weight rows. Assuming that the original weights fol- low normal distribution, the salient in the folded weights can be identified directly by their scales. These rows subsequently accumulate significant errors when repeatedly multiplied with the activation matrix. To mitigate this, introduce low-rank compensator matrix R E Rexd that
5

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
conrects the quantization errors in the r salient weight rows, denoted W.s Considering the weight rows permuted in descending order of saliency (P), the folded matrix W and the saliency-aware low-rank matrix R can be defined as:
W = P. diag(s) W - P W W W.). R - Q
(5)
Here, W, denotes the remaining weight rows. As shown, the residual errors of the salient rows are captured by the low-rank matrix to be used for reconstruction during matrix multiplication via an additional path. Then the overall linear operation can be described as:
Y (X diag(s P (PP diag(s) W) XW Q(X) Q(W) Q(X, x,I) Q W,I) Q(x.) R
(6)
X,, Wg X..q Q(R)
Importantly, quantize the low-rank matrix well, enabling pure low-precision computation across the entire path. Unlike SVD, directly extracting salient rows requires only single matrix multiplication in the residual path, thereby eliminating intermediate quantization. Furthermore, only
the activation channels corresponding to the salient rows (X, E RR =leexrr)) participate the residual operation, enabling computation-efficies low-rank multiplication of Raxr x Rgxx The overall SERQ process, including both calibration and inference, is illustrated Figure [a).
Offline Weight Permutation. We have aforementioned that the weights and activations must be
properly reordered based on their saliency (e.g. X | X,). W W.; W,)). To address this, we propose a mergeable weight permutation scheme that eliminates latency overhead during infer- ence. Both the rows and columns of the weight matrix are permuted offline according to the saliency order, enabling matrix multiplication to be executed directly on the appropriately reordered weights and activations. Figure ( illustrates the computation flow of single decoder layer, highlighting the offline permutation step. Based on saliency levels determined during calibration, row-wise per- mutations are prearranged for all weight parameters. The corresponding activations must follow the same channel order, which can be achieved by applying column-wise permutations to the preceding layer's weight matrix. For example, the permutation order P4 of the down-projection layer can be propagated to the weight columns of the preceding up- and gate- proteeprojettion layers, ensuring that the activation outputs are produced in the desired order. Consequently, the resulting activations natu- rally align with Pa. allowing the down-projection operate without additional reordering. In this way, all linear layers avoid on-the-fly reordering, and inference proceeds without latency overhead.
4 EXPERIMENTS
4.1 SETTINGS
Models and Tasks. We conduct experiments LLaMA-2 (7B, 13B and 70B: Touvron et al. (2023), ) LLaMA-3 (3.1 8B and 3.2 1B/3B; Grattafiori et al. (2024)), and Qwen-2.5 3B (Owen al. (2055) models. Our evaluation covers eight zero-shot commonsense reasoning tasks PIQA
Bisk et al. (2019)). SIQA (Sap et al. (019)), ARC-Easy/Challenge (Clark et al. (2018)) Hel- laSaag (Zellers et al. (2019). Winogrande (Sakaguchi et al. (202T)) BoolQ (Clark al. (2019).
and OpenBookQA QA (Mihaylov et al. (018)) and also provides perplexity scores on the WikiTexx2
test (Merity al, (2016)) as well the MMLU benchmark (Hendrycks et al. (202T)) We further report results on generation tasks using the GSM8K (Cobbe et al. (2021) and LongBench (Bai et al. 2024 datasets.
Implementation Details. The calibration set is constructed from 128 random samples of the WikiText-2 dataset (Merity et al. (2016)) to identify salient rows. SERQ uses rank size 128, which is equivalent in effective bit width using two low-rank matrices of rank 64. For quantization, the group size set to 128, and weights are quantized using either GPTQ (Frantar et al. (2023)) or round-to-neares (RTN) symmetric integer quantization, while activations asymmetric integer quantization. For the W4A4 configuration, we also provide the standard 4-bit Microscaling (MX) format, MXFP4, alongside the integer format, to demonstrate its implementation
quantized with RTN

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table I: Comparison with matrix decomposition methods. We compare perplexity scores, average zero-shot common sense reasoning accuracy, and average MMLUU accuracy. Results under different precision settings are obtained by modifying their publicly released codebase (See Appendix A.7).
LLaMA-2 7B
LLaMA-2
70B
LLaMA-3 8B
#Bits
Method
NEIT. (or)
PPL(U)
D-shot(1)
MMLU(()
PPL())
MMLU(")
PPE())
0-bbo(())
MMLUX1)
FP16
haseline
16
5.47
64.09
41.83
3.53
70.54
65.44
6.13
67.16
62.13
_1
7.28
59.64
29.01
7.41c43
49.71
43.56
10.82
60.86
46.72
W4A8
Q.R SERQ (RTN)
4.35
5.83
63.35
39.4
3.55
69.78
64.45
7.16
65.68
57.81
4.24
5.64
63.41
40.21
3.44
70.35
64,45
6.71
66.4
58.49
SERQ (GPTQ)
4.24
5.59
63.04
40.29
3,43
70.31
64.62
6.52
66.23
60.25
LLM. imt 4
6.32c+2
34.85
24.41
2.76e+4
39.77
23.67
4,87c+2
36.12
23.61
1. QUR
4.24
7.37
57.67
29.63
4.55
66.92
56.44
11.44
55.44
38.33
W4A4
L'QER-MXEPE
6.3
69.03
7.83
SERQ (RTN)
4.24
6.03
61:77
38.03
3.65
69.68
63.15
8.07
62.49
51.84
SERO (GPTQ)
4.24
5.97
61.87
37.03
3.64
69.38
63.09
7.75
62.41
53.8
SERO-MXHP4
61.64
7.63
LLaMA-3.2
LLaMA-3.2 3B
Qwen-2.5-3B
#Bits
Methoo
(wc)
PPL())
0ssbot(t)
MMLU(()
PPE())
0-shoi(t)
MMLU())
PPL())
0-sbot(?)
MMLU(())
FP16
haseline
16
9.75
54.82
36.76
7.81
62.66
54.05
8.03
63.82
65.12
LIM, 4
32.39
24.01
12.47
56.06
41.48
9.98c45
25.52
W4A8
L'RER
4.35
12.06
50.97
31.23
8.78
60.36
51.24
12.25
$4.71
26
SERQ (RTN)
4.24
11.14
52.76
32.77
8.38
61.27
51.45
8.67
63.78
62.79
SERO (GPTQ)
4.24
10.45
53.93
27.1
8.18
61,54
50.24
8,36
63.7
63.25
LAM, 41
1.7e43
35.19
24.11
4.37c+2
35,77
23,5
8.51e45
36.67
22.97
Q.R
4.24
30.83
42.17
25.67
14.11
50.92
32.25
221.5
38.17
23.81
W444
LXEER-MXPP
4.37
13.78
50.22
27.1
9.59
58.41
46.85
10.46
60.56
55.87
SERQ (RTN)
4.24
13.57
50.15
29.11
9:43
58.4
46.36
9.66
61.24
59.52
SERO (GPTQ)
4.24
12.52
50.44
26.34
58.5
45.7
9.35
60.78
59.55
EXPQ4
4.37
13.71
49.94
27.23
8.74
57.5
41.33
9.79
60.43
56.13
the case of int4 (1. the effectivve bit-width varies dynamically.
and measured speed on an NVIDIA RTX PRO 6000 GPU with Blackwell architecture support for MX format kernels (Rouhani et al. 2023) (See Appendix A.2.2).
Compared Methods. We primarily compare our method against the state-of-the-art error recon- struction approach Q.R under both W4A4 and W4A8 configurations. As baseline for ma- trix decomposition with low-rank error reconstruction, we include LLM. LiM.int/ 4 (), the 4-bit vari- of LLM. int (Dettmers al. (2022). We further compare against state-of-the-art W4A4 quantization methods that employ distribution-flaateennng techniques, including the rotation-based (2024) and SpinQuant Liu et (2025).
approaches Quarot Ashkboos Asskoo/A
4.2 EVALUATION RESULTS
Comparison with Low-Rank Matrix Decomposition Methods. We evaluate the W4A8 and W4A4 precision configurations to demonstrate the effectiveness of SERQ in low-rank error recon- struction. Both L.2QER and SERQ are compared under the same group size, with low-rank ma- trix dimensions matched to ensure the same parameter counts. The difference in effective weight bit-width under W4A8 arises because L'QER employs 8-bit precision for its low-rank matrices, whereas SERQ preserves 4-bit precision. As shown in Table J SERQ consistently outperforms LLM. i i 4 () and L'2RER across most tasks, while achieving the lowest effective bit-width. No- tably, SERQ is compatible with both RTN and GPTQ for weight quantization. Details of applying GPTQ robustly to SERQ are provided Appendix A.3.
The accuracy gap is more pronounced under the W4A4 setting. While prior INT4 quantization methods suffer from severe degradation in most cases, SERQ maintains high accuracy across all tasks. Lo performs especially poorly on LLaMA-3 models, failing to preserve accuracy when both activations and low-rank matrices are quantized to 4-bit. We further provide MXFP4 imple- mentations with the default group size of 32. Although MXFP4 proves adequate for both methods, the sequential reconstruction path in L'QERR, which requires two low-rank matrices, introduces sig- nificant latency overhead, which is examined in detail in the GPU performance analysis.
Comparison with W4A4 Distribution Flattening Methods. Rotation-based methods are recog- nized as oate-offtthe-rt W4A4 quantization approaches, achieving the lowest effective bit-width
7

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table Comparison with W4A4 distribution flattening methods. Latency overhead measured as the additional computation time per linear layer relative to 4-bit GEMM (See Appendix A.7).
LLaMA-2 7B
LLaMA-2 13B
#Bits
Method
(u-bits)
Training- free
Latency overhead
PPL())
0-abot(t)
MMLU())
PPL())
0-abol(
(MLLU(F)
FP16
basclinc
81.4%
5.47
64.09
41.83
4.88
66.53
52.04
1.51e+4
1.32c+4
SmoothQgg
4.13
7.49
57.15
30.4
6.31
61.28
39.83
W4A4
QuaRot
QuinQuant
m
19.8%
6,15
59.53
33.58
5.41
62.55
47.25
SERQ
4.24
18.7%
5.97
61.87
37.03
5.2
64.82
47.17
LLaMA-3 8B
LLaMA-3.2 1B
LLaMA-3.2 3B
#Bits
Method
(PL(()
0-shot(1)
MMLD(11
PPL(()
0-shot(1)
MMLU(()
PPL(()
0-shot(1)
MMLU(()
FP16
basclinc
6.13
67.16
62.13
9.75
54.82
7.81
54.06
4.71e+5
1.53e+5
3.73e+4
SmoothQ(g Q(g228)))))))))))
17:26
48.97
29.3
69.22
40.04
24.43
53.33
44.17
27.31
W4A4
QuaRot
8.41
59.12
47.29
13.17
50.03
26.64
9.73
55.76
44.75
SpinQuant
SERQ
7.75
62.41
53.8
12.52
30.44
26.34
9.15
38.5
45.7
L'GEB-AB
L'Q0lluu4
SERQ ž LuRA Path
PPL
Spieqquent
PPV.
1200
2.0
800
2.0
Latency (us)
Speedup
Porplesity
Speeduu
Perplesity
600
2.0
Latency
400
2.0
300
0.5
200
0.5
(1096,
(4096,
(14336,
(9096.
(4096,
(14336,
4096]
14356)
4096)
40%6]
14336)
40%6)
Figure 3: GPU performance comparison. We report latency overhead analysis across various matrix sizes (batch size is I and token length is 4k). SERQ is particularly effective for larger row-sized matrices. (See Appendix A.6)
with minimal additional parameters. We therefore compare two representative rotation meth- ods, Quarot and SpinQuant, along with SmoothQuant's 's cearibution-flatteiing baseline (Xiao et al. (2024)). The reported accuracies for rotation methods are obtained without key-value quantization for fair comparison, using GPTQ without grouping for compatibility with low-precision GPU ker- nels. shown in Table 2 SpinQuant generally outperforms Quarot due to its learned rotation matrices, but this advantage diminishes on compact LLaMA-332 models, where both methods suf- fer from significant degradation. In contrast, SERQ achieves consistently higher accuracy, with the improvements most pronounced LLaMA-3 models. Although SERQ incurs slightly higher ef- fective bit-width due to the inclusion low-rank matrices and scaling factors, its per-layer latency overhead is lower than that of rotation-bast methods.
GPU Performance Comparison. To examine inference performance with the proposed method, execution time using low precision GEMM kernels implemented in the NVIDIA CUT- LASS library (cut) (2025)). Since the fifth generation Blackwell Tensor Core architecture does not support INT4 GEMM kernels, while being substantially faster than the previous Ampere generation, we evaluate ootation-based methods in the FP4 format on the same latest core. Mixed precision com- putation for L QOR with W4A8 is measured with the MXFP mixed-precision kernel, while the Fast Hadamard transform for online rotation is executed using FP32 kernel (Ashkboos et al. (2044))
We benchmark the latency of linear operations across different dimension sizes in the LLaMA-3 8B decoder layer, and also report the overall speedup relative to an FP16 GEMM baseline. As shown in the left panel of Figure 3 SERQ outperforms both W4A4 and W4A8 settings L.2QER, while maintaining competitive perplexity. Compared directly with I. LºRERR under W4A4, our single matrix error reconstruction path reduces latency overhead by up to 4.5x compared to the LoRA path, which requires two sequential low rank multiplications, thereby delivering the highest speedup. The methods. Since key-value
right panel of Figure 3 compares performance against rotation-bart quantization is excluded, the only online rotation occurs between the up gate projection layer and the down projection layer. However, due to the unbalanced matrix dimensions, rotation introduces a significant overhead, about 1,6x greater than SERQ, which increases the latency of single decoder
8

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table 3: End-to-end GPU inference speed and memory measurements for LLaMA-3 8B. We report Time to First Token (TTFT), Time per Output Token (TPOT), and peak memory consumption. The input sequence length is fixed 2k tokens for all measurements.
Bsz
Method
TTFT(ms)
Speedup
Peak MemiGB)
Saving
(POT(ms)
Speedup
FP16
132.38
x1
17.44
x1
18.44
x1
MXFP4
6,94
x2.51
25.48
SERO MXFP4
62.31
X2.12
7,03
X2.48
39.28
x0.47
FP16
1252.24
x1
27.9
x1
61.93
x1
8
542.58
X2.11
17.4
x1.6
54.49
x1.14
SERQ-MXFP4
587.17
17.48
$7.08
FP16
2515.08
x1
39.85
x1
114.24
X1
16
only MXFP4
1113.06
X2.26
29.35
1.36
100.71
x1.13
SERO MXEP4
29.44
x1:35
103:27
FP16
4987.05
x1
63.76
x1
218.01
X1
32
Only-MXPP4
2257,54
X2.21
53.25
x1.2
193.7
X1.13
SERQ
2453.29
X2.03
53.34
x1.2
202.13
x1.08
Table 4: Effect of rank size on perplexity.
Table 5: Effect of calibration data on perplexity.
#Rank LLaMA-38B 8B
LLaMA-3.2 IB LLaMA-3.2
Datasets #Sample LLaMA-3 8B
LLaMA-3.2 IB LLaMA-3.2 3B
9.8
15.09
10.21
13.57
16
8.28
14.32
9.71
Wiki
128
7.98
13.6
9.43
32
8.24
14.09
9.59
8.08
13.51
9.45
64
8.18
13.97
9.57
13.63
128
8.07
13.57
9.43
Pile
128
7.96
13.54
9.44
256
7.98
13:25
9.32
13:44
layer. As a result, SERQ achieves the best perplexity score while delivering comparable speedups rotation-based methods, with only about one percent additional latency overhead.
End-to-End GPU Performance. To further demonstrate the feasibility of deploying SERQ on Blackwell GPUs, we evaluate the end-to-end latency of both the prefill and decoding stages using the LLaMA-3 8B model. We report Time to First Token (TTFT) for the prefill stage and Time per Output Token (TPOT) for decoding, along with peak memory consumption. As shown in Table SERQ consistently achieves more than 2x speedup across all batch sizes, while incurring less than a 10% latency overhead relative to vanilla MXF4 acceleration, yet delivering substantially improved accuracy. Although single batch decoding exhibits increased latency under quantization, a trend also observed with MXFP4, we observe moderate speedups at larger batch sizes with only marginal latency overhead compared MXFP4. Furthermore, consistent with the primary objective of quantization, peak memory usage during the prefill stage is reduced by up to 2.48 x relative to the FP16 baseline.
4.3 ABLATION STUDIES
Sensitivity on the Rank Size. SERQ constructs a low rank matrix of size r. While we fix the rank size to align with prior methods, we also analyze its impact separately. We evaluate three models, LLaMA-3 8B and LLaMA-3.2 1B and 3B, by varying the rank size, which corresponds to the number of salient rows. As shown in Table 4 perplexity decreases monotonically with larger ranks, indicating improved accuracy. However, the improvement quickly saturates, and even the smallest setting of r 16 (equivalent to rank in LoRA) yields competitive results.
Sensitivity Calibration Samples. Table 5 reports perplexity scores obtained with varying calibration dataset sizes, which are used to determine saliency. We further evaluate our scheme the Pile dataset Gao al. (2020) to assess sensitivity to both sample size and dataset choice. The results indicate that SERQ is robust to the calibration dataset size and dataset characteristics, achieving similar perplexity scores across all settings.
Evaluation on Generation Tasks. While prior work on linear layer quantization primarily focuses on prefill-sensitive tasks (See section 4.11
4.1). Th, additionally evaluate generation tasks demonstrate

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table 6: Generation task evaluation with GSM8K and Long datasets.
GSMSK S-shot(f)
LongBench(())
Model
#Bils
Method
llexible extract
strict match
quisa
samsum
repobench-p
FP16
baseline
48.07
47.69
23.43
46.26
66.56
L.M. 4
7.88
4.62
13.29
34.2
44,91
W4A8
L'QER
LLaMA-3
SERQ
42.38
W444
Q'R
SERQ
23.65
7,96
23.12
7,66
19.08
0.09
40.85
1.98
$4.78
FP16 baseline
26.08
25.93
23.94
42.98
64.42
LIM.
O
6.76
6.44
36.42
35.24
W4A8
18.95
18.42
21.53
40.63
60.18
T.M.A-32..38
SERQ
21.68
21.23
22.25
42.47
60.78
W444
LER SERQ
16.22
3.56
15.77
2.73
19.31
14.92
33.87
42.14
41.99
$3.67
Table 7: Effect of static activation flattening (SAF) perplexity.
#Bits
#Q-config.
Method
LLaMA-2 7B
LLaMA-3.2 1B LLaMA-3.2 3B
Qwen-2.5 3B
FP16
baseline
5.47
9.75
7.81
8.03
Only
6.64
25.49
12.75
12:41
W4A8
INT (RTN)
SERQ wla SAF SERQ
5.64
11.13
8.37
8.56
5,63
11.1
8.36
8.57
only
7.58
INT (RTN)
SERQ w/o SAF SERQ
6.05
13.6
9.36
10.83
W444
6,03
13:49
9.42
9.57
only SAF SERQ SAF
7.03
17.21
11.49
12.88
MXFP4
6.21
13.56
9.59
10.4
SERQ
6.22
13.53
9.69
9.84
accuracy in generation. We conduct experiments on GSM8K (Cobbe et al.
(202 and LongBench
(Bai et al. (024)) under W4A8 and W4A4 settings using LLaMA-3 8B and LLaMA-3.1 3B models. Compared with other matrix decomposition methods, SERQ achieves superior performance with only minor accuracy degradation in the W4A8 setting. Although the W4A4 setting leads to notable accuracy drop overall, SERQ maintains reliable results where other methods fail.
Ablation Study on Static Activation Flattening. In SERQ, static activation flattening (SAF) is combined with saliency aware low rank eπrоr reconstruction, effectively stabilizing the activation distribution during quantization. To isolate the contribution of each component, we conduct an ab- lation study, summarized in Table For the LLaMA-2-77 model, SERQ without SAF maintains accuracy well when only low error reconstruction is applied. However, for more error-sensitive small-scale models such as Qwen-2.5-3BB SAF provides substantial accuracy improvement. This behavior arises because smaller models accumulate fewer partial sums per output element, making quantization errors from a single group disproportionately influential on the final output. Results from the MXFP4 implementation exhibit a similar trend. Although the smaller grouping in MXFP4 partially mitigates outlier effects, SAF still yields additional accuracy gains, demonstrating the im- portance of this technique and the potential of SERQ for edge-oriented LLM deployment.
CONCLUSION
In this work, we introduced SERQ, a saliency-aware erro reconstruction method that enables end- to-end 4-bit quantization of both weights and activations using single low rank matrix. By com- bining static activation flattening with mergeable weight permutation, SERQ identifies salient rows the weight matrix without incurring additional latency overhead and reconstructs them through an auxiliary low rank branch. SERQ consistently outperforms prior matrix decomposition and dis- tribution flattening methods, including state-of-thed LoRA-based and rotation- based approaches, in terms of accuracy. Our implementation with the MXFP4 data format NVIDIA Blackwell architecture further demonstrates that the auxiliary branch introduces minimal latency overhead, achieving significant speedups while preserving accuracy in low precision computation.
10

---

SERQ: Saliency-Awar Low-Rank Error Reconstruction for LLM Quantization
ACKNOWLEDGMENTS
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-003524668), K-CHIPS (Korea Collaborative & High- tech Initiative for Prospective Semiconductor Research) (2410000620, RSS2024-00405944 24052- 15TC) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea), and Institute of In- formation & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (RS-2025-0226366) Development of server-level DRAM-stacked PIM solu- tion for accelerating ultra-large Al models).
REFERENCES
Nvidia cutlass documentation. 2025. Accessed: 2025-09-25.
indwx//idxxx.mo// html
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Quik: Towards end-to-end 4-bit inference on generative large language models, 2023. URL
0 09259
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li. Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, 2024.
www.:/xwwwww..abb/bb/oaaaaa 2245 00456
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, mul- titask benchmark for long context understanding, 2024. URL https: org. abs/
2308. 14508
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language, 2019. URLL/tttp:::/ax....oorr////////
11641
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019, URL
/ttpps//aarxinn wttpx//....//////19...... 10044
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge, 2018. BRLLttps: arxiv. org. 011//11333.0555
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiis
org/abs/2110 14168
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int00: 8-bit matrix multi- plication for transformers scale, 2022. utt.httsps://axxiv..rrg// http://wwww.rrr//////////////// 2220000339
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training
quantization for generative pre-trained transformers, 2023. URL. https://arxiv.orrt
3bbs/2210.133
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL htttts://axxiv... org/ abs/2100 00027
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. The llaa 3 herd of models, 2024. URL htt.:/tts///axv..orr////e//

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Han Guo, Philip Greengard, Eric P. Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning, 2024. URL https:://///...
/ arxiv..rq/./////
12023
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
wwwxxi.orgg/ba//20000 03300
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
7/arxiv.oro abbs//ba//// 221. 09685
Xing Hu, Yuan Cheng, Dawei Yang. Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting, 2025. URL htt.ttts://aatt. org abs/ /2001 13987
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models, 2025. URL
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tunnn-awaal quantization for large language models, 2023. URL
https:/ /tts:///axvv..m wwt.oogg///ooooooooooo
abw/ http:///a///311... 08659
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized Iims, 2024a. URL https:t///// /ttp://aaavvvv//a org/ abs/ abss/444 2406. 01721
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. In Gibbons, G. Pekhi- menko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6, pp. 87-100, 2024b. URL https: /eatps://proceet ngs m sys wwtp://rgppppppeaeeeeeeeee httpr//pppppesslepppppeeeer fi http://lee422245222m 2ttp4///.2.222222222222222266222 htt.a//aeeee.aaaaaa..e.e....eeee .f
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantiza- tion with learned rotations, 2025. URL https: http:///axvvvvve wwwpiii.rrg/bbb//44//00
htttps://axxvv.oto
16406
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
07843
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018. URL httpp:://rxxiv org/ abs///b/// 1809.000000000 02789
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper neural network quantization. CoRR, abs/2106.08295, 2021. URL httttttss///xii..rtttt
2106.0000000 08295
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL htlpstt::: / wxaxxxxxii orgg abs/ 2412. 15115
12

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Comea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Ven- mugil Elango, Maximilian Golub, Alexander Heinecke, Phil nomes-Roxby, Dharmesh Jani, Gau- rav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Ro- driguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Mi- cikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger. and Eric Chung. Mi- croscaling data formats for deep learning, 2023. URL https::/aaaaee org/ 10537
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. Com- pressing large language models using low rank and low precision decomposition, 2024. URL
https://arxiv org/ /ttp://aa/22231188883 18886
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. Commun, ACM, 64(9):99-106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381, URL https: org http:///ooo...//10..1110000 21t.g///111111134444333
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqu: Common- sense reasoning about social interactions, 2019. URL http:pps//aaai..oog////bbas//
09728
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models, 2024.
13137
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. Flatquant: Flatness matters for Ilm quantization, 2025. URL https: /tttps://axxi.... wrw.v..iiivv.//ii//..aaaaaaaavva
org/ abs/ /2100 09426
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https::///::.vv. org/abs 2307. 7 09288
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https:
//arxiv. haa///b/22220 /211 10438
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization aware low-rank adaptation of large language models, 2023. URL
httptts::/:axii..ooott httte:::/arrii..rrr///////33 14711
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization llms from comprehensive study to low rank compensation, 2023.
httph:///:/aat http://axxvv..rroo abs abs/2303... 08302]
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019. URL https: /tte:://axxxxi httpxiivvvo... org. abs////b////// /tt///9950088888
Cheng Zhang, Jianyi Cheng, George A. Constantinides, and Yiren Zhao. Lqer: Low-rank quantiza- tion error reconstruction for llms, 2024.
Cheng Zhang, Jeffrey T. H. Wong. Can Xiao, George A. Constantinides, and Yiren Zhao. Qera: analytical framework for quantization error reconstruction, 2025. URL https://axxiv: lorg/abs 22410.06040 06040
Weibo Zhao, Yubin Shi, Xinyu Lyu, Wanchen Sui, Shen Li, and Yong Li. Aser: Activa- tion smoothing and enror reconstruction for large language model quantization, 2024. URL
https://arxiv wtt.:/o///////11...7700.../ 07762
13

---

SERQ: Saliency-Awar Low-Rank Error Reconstruction for LLM Quantization
A APPENDIX
A. EXPERIMENTS ON LOW-RANK SALIENCY
In this analysis, we vary the number of weight rows included in SVD, selected by saliency in de- scending order of activation channel scales, to examine the trade-off between loss from rank reduc- tion and the coverage of error reconstruction. To isolate the quantization effects of the weight matrix, apply W4A16 16 quantization with group size of 128 and fix the SVD rank size to 64. Figure shows that decomposing only salient weights more effective for reducing quantization error, as the perplexity score decreases slightly on the leftmost side when using row size 64. This result indicates that restricting reconstruction to salient rows not only avoids accuracy degradation but also yields consistent improvement across all tested LLaMA models. We observe about to 4 percent improve- ments compared to using the same limited rank to cover the full weight matrix. Expanding beyond the salient subset offers little or even negative benefit, since the additional low rank factorization loss outweighs the marginal coverage. These findings motivate a single-layer reconstruction, rather than two low-rank factors, and support saliency-aware егrог reconstruction strategy that keeps both the main and auxiliary paths in pure 4-bit, achieving efficiency without sacrificing accuracy.
12-
10-
Perplexity
258
1024
2045
Salient-Weigh
Figure 4: The trade-off between loss from rank reduction and the coverage of error reconstruction. The figure shows that higher accuracy achieved by reconstructing errors for salient rows with smaller ranks, rather than covering larger portion of the weight matrix.
A.2 RELATED WORKS
A.2.1 LOW-RANK ADAPTOR FOR QUANTIZATION ERROR RECONSTRUCTION
Low-Rank Adaptation (H et al. 2021)) originally introduced as a representative Parameter-Efffcient Fine-Tuning (PEFT) technique, has demonstrated remarkable effectiveness in reducing training overhead while facilitating rapid adaptation to down-stream tasks. Its architecture, which injects two trainable low-rank matrices into the pre-trained weights, exhibits high compati- bility with existing transformer models. Leveraging this property, recent LLM quantization utilizes these auxiliary low-rank matrices reconstruct and compensate for quantization errors inherent in compressed weights. In particular, the low-rank structure of LoRA align intrinsically with Singular Value Decomposition (SVD), which yields the optimal solution for low-rank approximation. The approximation of the quantization error, E W - Q(W), can be formally defined using truncated SVD as follows:
E Ex U~ L2
where usually U. and Σ V correspond to the low-rank factors L1 and L2, respectively, and denotes the rank. This paradigm of LoRA-based quantization has been extensively explored both weight-only and weight-activation quantization regimes.
Weight-only Quantization. ZeroQuant- V2 Yao et al. (202)) and LQER [Zhang et al. (2024) demonstrated that integrating optimized low-rank matrices derived via SVD with quantized weights effectively mitigates quantization errors. This methodologies was further advanced by
CALDERA A Saha al. (2024) and QERA(Zhang al. (2025)). (Saa al. (0244)
14

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
initializes low-rank matrices using SVD and employs iterative update scheme for both the quan- tization error and the low-rank adaptors, thereby maintaining competitive performance even under ultra-low quantization(e.gg ~2-bits). QERA(Zhang et al. (2025) addresses the limitations of heuris- tie approach to incorporating activation sensitivity by deriving a closed-form solution that effectively minimizes the quantization error of the linear layer's output.
Weight-Activation Quantization. A critical challenge in weight- quantization is how to manage outliers present in activation channels. L ZZQER et al. (2024)) and ASER Zhao al. ) propose mixed-precision configurations (e.g., W4A8 or W4A6) to address this issue. To specifically mitigate activation outliers, L SFQRe(hhagg al. (2024)) applies per-channel activation scaling, while absorbing the corresponding scaling factors into the SVD-based low-rank matrices. Similarly, ASER(Zhao
al. ) enhances the efficiency of low-rank approximation via whiten- ing SVD and alleviates outlier effects through activation smoothing. Furthermore, SVDQuantt(). al. (2025)) migrates the quantization difficulty of activations to the low-rank adaptor while estab- lishing hardware-efficits pipeline, demonstrating superior quantization framework particularly in diffusion models().
A.2.2 MICROSCALING (MX) FORMAT
The Microscaling (MX) format Rouhani et al. 20233) is block-scaled, low-precision representa- tion that associates each small block with single scale and quantized elements, thereby retaining dynamic range while enabling efficient 4-8 bit computation. Concretely, data are partitioned into blocks; each block stores one scale, estimated from its maximum value, and the elements within the block are stored in low-bit precision-either in boating-point formats (MXFP8, MXFP6, MXFP4) or in integer format (MXINT8). Note that INT4 is not included among MX variants. All elements defined relative to the block's 's shared scale, which is fixed at 8-bit precision.
In contrast, naive single-scale low-bit formats often degrade sharply on LLMs: a few outliers in- flate the global scale, wasting quantization levels and clipping inlier values. By adopting block- level granularity (typically set to 32), MX preserves usable dynamic range and maintains effective quantization. As result, MX-based models achieve low-precision inference with minimal accu- racy loss, reduced memory and bandwidth, and hardware-friendly execution, since computations reduce to standard low-bit arithmetic plus lightweight per-block scaling. Owing its effectiveness for low-precision quantization in deep learning, NVIDIA's Blackwell architecture, equipped with fifth- fftth-generation Tensor Cores, provides CUTLASS kernels supporting MX data formats for LLM workloads.
A.3 GPTQ IMPLEMENTATION
Recently, Frantar al. (203)) has demonstrated strong performance in weight-only quan- tization through a lightweight, Hessian-guided optimization. Specificallyy, GPTQ(Frantar et al. 2023) quantize weights sequentially while utilizing Hessian information to compensate for the induced quantization error by remaining (unquantized) weights. The weights update is defined as follows:
H
wj w Ej-
H-1
H
Where i denotes the index of the weight currently being quantized, and j represents the indices of the remaining unquantized weights (i.e., j > i). Here, e, = w, - Q(w;) corresponds to the quantization error induced at the current step, and H -1 refers to the inverse Hessian matrix, which guides the optimal compensation for the quantization error. However, this update process is not orthogonal to the low-rank error reconstruction targeting E w Q(W. This is because the iterative updates in GPTQ shift the weight distribution, causing it to diverge from the original statis- tics. To address this, we modify the error reconstruction pipeline to achieve GPTQ--fieende de- composition. we prioritize extracting the high-variance components from the salient rows into 4-bit low-rank matrix prior the GPTQ process. Specifically, we define the low-rank matrix R as Q(W.) and replace the original salient rows with the residual error Wa R. This substitution effectively migrates the flattening-driven outlier away from the main weights, thereby stabilizing the weight distribution. Furthermore, it reduces computational complexity by eliminating the need for dequaaniiatioo-rrquanizzz process during enror reconstruction. In this way, (PTQrrrantar al.
15

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table 8: Comparison of Quantization Signal-to-Noise Ratio (QSNR) between SERQ and SVD.
Module
Rank
Method
QSNR(()
Module
Rank
Method
QSNR(1)
19.87
SVD
18.91
128
128
SERQ w/o SAF
25.91
SERQ weo SAF
21.24
SVD
19.41
64
64
SERQ w/o
25.49
SERQ w/o SAF
20.81
LLaMA-3 8B
SVD
19.21
LLaMA-3
SVD
18.58
10 proj
32
20 o.proj
32
SERQ w/o SAF
20.22
SERQ SAF
SVD
19.06
SVD
18.52
16
16
SERQ w/o SAF
20.42
SERQ w/o SAF
18:54
SVD
19.01
18.49
8 SERQ w/o SAF
SERQ w/o SAF
21.29
18.75
2023)
targets a more quantizatiion-mmenbble main weight matrix, while the low-rank path captures
the salient components.
A.4 ANALYSIS OF QUANTIZATION ERROR RECONSTRUCTION.
Quantization Error Reconstruction aims to approximate and compensate for the quantization error of the main weights by leveraging the LoRÁ architecture. A pivotal aspect of this process is the selection of the saliency metric used to minimize the quantization error. Truncated SVD approaches approximate the residual between the original weights and their quantized counterparts, optimizing the following objective function:
min | CIF C
(BB Cij)² 1.
(10)
where E represents the quantization error and C is the low-rank compensate term. To minimize this objective, truncated SVD employs singular values as the saliency metric, providing closed- form solution via truncated SVD. However, in Transformer models, the outputs of linear operations (i.e.. XW) propagate sequentially through the layers. Consequently, the cumulative quanti- zation error that directly impacts model performance is effectively defined by the output error, i.e., AY XAW. Therefore, while minimizing the weight error A AW necessary, is far more crit- ical to mitigate the error contribution from indices i where the activation inputs X possess large magnitudes. To validate this theoretical justification, we present a comparison of the Quantization Signal-to-Noise Ratio (QSNR) for the linear layer outputs in Table We specifically compare the truncated SVD method against our SERQ (w/o SAF) under a weight- results demonstrate that SERQ (w/o SAF) yields superior quantization performance compared to SVD at identical rank budgets Notably, we observe a sharp increase in QSNR for SERQ (w/o SAF) beyond specific rank threshold. This indicates that reducing the rank budget below this critical point fails to capture all salient weights. Consequently, to ensure the stability of model performance, we selected a rank of 128.
quantization setting. The
A.5 ADDITIONAL COMPARISON WITH SOTA ROTATION METHODS.
Although our primary research focus lies in LoRA-based quantization, the proposed SERQ method marks a significant breakthrough by enabling a robust W4A4 quantization scheme. Given that we quantize both activations and weights to 4-bit, we provide a comprehensive comparison against state- of-the-art (SOTA) rotation-based methods, which currently dominate the W4A4 quantization. In Table we present performance benchmarks alongside calibration costs for leading rotation-bass techniques, including SpinQuant((Liu Liu et al. (2025)), FlatQuant(Sun al. (2025). D) DuQuant(Linn et al.) (2024))) OSTQuant(Hu et al. (2025)) and Quskoo(Ashhhboo et al. (2024)). The results indicate that methods employing extensive training or heavy iterative optimization generally yield the high- est accuracy. FlatQuant(Sun al. (2025)). ), in particular, demonstrates superior performance, outper- forming all the compared models. However, it is important to note that this advantage comes at the expense of optimizing large number of trainable parameters and introducing floating-point (FP) affine transformations during runtime, which can incur additional computational overhead. Regard- ing calibration efficiency, the training-free ooproaches-ssecfffcllll SERQ and QuaRot(Ashkbon
16

---

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Table 9: Comparison of SERQ with SOTA rotation-based quantization methods in terms of calibra- tion time and model accuracy (PPL, O-shot, MMLU).
1.aMMA-277 7B
I.aMMA-2 13B
I.aMA-3.22. 3B
Method
Calih, Time
PPL
O-shot
MMLU (t)
Calib. Time
PPL
0-shot
MMLU
Calib. Time
PPL
0-shot (t)
MMLU
(1)
(7)
(3)
(?)
(t)
(
(f)
FP16
5.47
64.09
41.83
66.53
52.04
54.06
SpinQuant
~598m
mm
5.2
64.8
47.8
156m
10.15
42.42
FuanQuant
5.75
225m
64.94
49.59
DuQuant
255m
5.95
61.84
33.46
<.486m
5.23
64.88
48.59
1440m
9.56
58.31
44.79
OatQuant
~72mm
5.91
61.72
37.39
~m92m
5.26
64.92
47,77
~226m
9.04
58.81
46.05
QuaRot
~m1m
6.15
59.53
33,58
444
5.41
62,55
47.25
~-8m
9.73
55.76
44.75
SERQ
25m
3.97
61.87
37.03
Km4
5.2
64.82
47.17
~-15m
9.15
58.5
45.7
et al. )2024) offer drastically lower calibration costs. What is particularly compelling is that, de- spite this negligible overhead, SERQ delivers performance that fully comparable computationally intensive, training-baste based methods. This confirms that SERQ acts highly efficient, lightweight framework, offering an optimal trade-off between calibration cost and model accuracy.
A.6 GPU PERFORMANCE ANALYSIS DETAILS
This section reports the absolute latency obtained with NVIDIA Blackwell CUTLASS kemnlss when operating on linear layers of different sizes used in LLM models. We also provide the inference overhead introduced by low rank factors, where SERQ includes additional multiplication path using R, and L'RER includes the factors L1 and L2.
A.7 OVERALL LLM ACCURACY RESULTS
We report overall accuracy results on the evaluated benchmarks, which are not captured Tables and since they only present average values. Tables 7 and 8 provide detailed accuracy results for low-rank matrix decomposition methods under the W4A8 and W4A4 settings, respectively. Table reports detailed results for different distribution flattening methods.
A.8
LLM USAGE
In this paper, we used large language model to aid and polish the manuscript to improve the overall writing quality.
17

---

SERQ: Saliency-Awe Aware Low-Rank Error Reconstruction for LLM Quantization
Table 10: GPU latency results in linear layers.
Inference overhead (MXFP4)
Sequence Length
Weight Size
FP16(μs))
MXFP4(us)
R(ps)
Li(us)
L2(ps)
L, L2(ss)
2048 2048
71.402
27.457
10.501
16.606
12.009
28.615
2048 x 8192
207.482
72.7
20.934
16.606
24.86
41.466
8192 x 2048
212.87
88.738
10.501
43.235
12.009
55.244
4096 x 4096
211,894
70.255
14.637
26.833
14.66
41,493
2048
4096 x 11008
519.322
181.359
27.0112
26.833
31.034
57.867
11008 4096
512.25
175.964
14.637
55.51
14.66
70.171
4096 x 14336
643.27
235.397
38.342
26.833
40.638
67,472
14336 4096
655.757
226.379
14.637
69.887
14.659
84.547
8192 x 8192
755.494
260.054
20.934
43.235
24.86
68.095
8192 28672
2604.973
946.403
174.537
43.235
174.857
218.09
28672 8192
2444.912
946.787
20.934
132.005
24.86
156.865
2048 2048
122.298
37.664
14.4035
16.669
14.662
31.331
2048 x 8192
403,859
142.159
56.211
16.669
55.991
72.66
8192 x 2048
394.138
132.903
14.4035
43.292
14.662
57.954
4096 x 4096
382.79
136.076
20,936
26.805
25.155
51.96
4096
4096 x 11008
1009.898
355,986
129.704
26.8045
129.876
156.681
11008 4096
986.624
345.739
20,938
55.947
25.155
81.1
4096 x 14336
1262.093
477.976
175.101
26.8045
175.268
202.073
14336 4096
986.97
445.022
20.938
70.016
25.155
95.171
8192 x 8192
1560.854
506.364
56,211
43.292
55,991
99.283
8192 28672
4868.259
1911.05
355.274
43.292
355.969
399.261
28672 8192
4849.363
1853.27
56,211
132.019
55.991
188.01
18

---

SERQ: Saliency-Awe Low-Rank Error Reconstruction for LLM Quantization
Table 11: Accuracy results of low-rank matrix decomposition methods when tested with W4A8 settings.
O-Shot Common Sense Reasoning tanks
MMLU
Model
Methods
PPL.
BoolQ
PIQA
SIQA
ARC-e
ARC-c
HellaS.
WinGG. OBQA
(f)
Arg.
(t)
Human. (1)
Other SocialS. (t)
STEM
Avg.
baseline
5.47
77.74
79.05
46.11
74.49
46.25
76
68.9
44.2
64.09
39.72
47.12
47.45
34.28
41.83
11AMA-3-78
LCM. 1nt Q
7.28
69.3
76.39
68.48
43.43
71.19
39.6
59.64
26.23
21.9
33.54
29.84
5.83
75.99
78.51
44.08
74.75
44.03
75.14
68.98
44.4
63.35
37.99
43.13
43.48
34.76
39.4
SERQ (KUN)
5.64
76,85
78.4
44.93
75.7
45.31
75.12
68.75
44.2
63.41
37.39
45.98
45.47
33.65
40.21
SERQ (GPTQ)
5.59
77
78
44.83
73.15
44.2
75,05
68.27
43
63.04
37.9
45.22
44.85
34.54
40.29
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL (1)
BooQQ (1)
PIQA (7)
SIQA
(1)
ARC-e
(1)
ARC-c
(1)
HellaS
(1)
WinGG
(1)
OBQA
(t)
Avg.
(1)
Human. (1)
Other (1)
SocialS.
(1)
STEM
(1)
Avg.
(1)
hrseline
4.88
80.61
50.52
77.53
49.33
79.38
72.58
45.2
66.33
89
59.29
61.16
42.21
LLaMA-2-13B
L.M. iit
5.35
79.91
79.38
46.52
75.34
49.15
77.46
69.69
42.6
65.01
44.31
53.07
54.76
28
47.12
LOER
5,08
$1.41
79.87
46.37
76.85
50.17
78.52
72.3
45.6
66
46.5 63
58.71
59.02
42.02
SERQ (RTN)
50:18
45.47
76.3
48.38
78.7
71.67
44.8
65
47.08
37.64
42.09
SERO (GPTQ)
4.98
79.34
79.38
46.26
76.18
47.27
78,83
72.45
45.4
65.06
45.59
55.52
58.01
40.91
49.46
Common Scnse Reasoning tacks
MMLU
Model
Methods
PPL
BoolQ)
PIQA
SIQA
ARC-
ARC- HellaS, WhnGG, OBQA
Avg.
Haman. Other SociatS,
STEM
(1)
(1)
(7)
(1)
(1)
(1)
(1)
(1)
(7)
(7)
(1)
(t)
(t)
(t)
(1)
baseline
6.13
$1.07
80.74
47.08
77.69
52.99
79.2
73.48
67.16
54,81
70.55
73.25
53.89
62.13
11aMA-3-88
1. iitt A i)
10.82
75.23
74.97
44.22
67.38
42.24
72.63
70.01
40.2
60.86
42.54
53.91
$2.65
40.41
46.72
7.16
80.67
65 08
66.4
66
SERO
06.4
09.16
48.59
SERO (GPTT)
78.25
72.53
66.33
69.46
70.62
0-Shot Common Sense Reasoning tanks
MMLU
Model
Methods
PPL
BooQQ
PIQA
SIQA
ARC- ARC- HellaS, WinGG, OBQA
Avg.
Human. (t)
Other SocialS, (1)
STEM
Avg.
(t)
(f)
(t)
(f)
hiseline
9.75
63.67
74.48
42.84
60.44
36.11 18
63.75
59.83
37.4
54.80
34.93
41.1
39.78
32.20
36.76
LLaMA-32-1B
LCM, 1nt i)
24.24
23.17
22.96
12.06
60.67
73
41.45
53.45
38.68
59.04
$6.91
32.8
50.97
28. 99
35.92
33.99
27.85
31.23
SERQ (RTN)
11.14
62:75
73.85
42.07
56
34
60.74
56.35
36.2
52.76
32.56
36.53
11.44
28.7
32.77
SERO (GPTQ)
10.45
63.24
72.47
42.02
58.92
36.95
61.7
$8.96
37.2
53.93
27.72
29
23.72
23.69
27.1
0-Shot Common Scnse Reasoning tacks
MMLU
Model
Methods
PPL
BoolQ
PIQA
SIQA
ARC-
ARC-c HellaS,
OBQA
Avg.
Human. (1)
Other SoclatS. STEM
Avg.
(1)
(1)
(7)
(1)
(1)
(?)
(1)
(1)
(t)
(?)
(1)
(1)
(1)
(1)
62.66
LLAMA-32-BB
L.N, Litt
12.47
67.55
73.99
42.94
62.58
37.46
66.57
61.8
35.6
56.6
37.26
46.6
47.71
36.63
41.48
71.8
59.67
SERO
8.38
71.22
72.31
59.86
39.02
42:53
SERO (GPTQ)
8:18
71.19
46.76
58.93
56.65
0-Shot
Reasuning
MMLU
Model
Methods
PPL
BookQ)
PIQA
SIQA
ARC-
ARC-c
HellaS. (1)
WinoG, OBQA
Avg.
Human. Other (1)
SocialS. (f)
STEM
(1)
(1)
(7)
(1)
(1)
(1)
(1)
(?)
(f)
(f)
hoseline
8.03
77.58
78.73
50.1
73.11
47.1
73.62
68.51
41.8
63.82
56.77
71.07
76.15
60.96
65.12
Qwen-225 38
LON, d.
9.98c+5
37.85
51.52
34.19
24.58
26.19
26.51
51.3
32.6
35.59
27.14
24.01
23.79
26 26
25.52
1.2 OIFR
12.25
68.1
73.07
42.09
47.33
37.54
66.57
64.01
38.2
54.71
76 82
76.97
25.32
24
26
SERQ (KTN)
8.67
76.85
77.75
50.41
76.68
45:38
72.13
65.82
43.2
63.78
55.30 39.39
16
73.71
57.6
62.79
SERQ (GPTQ)
8.36
77.68
77.86
69.23
75.46
48.63
72.35
66.61
41.8
63.7
55.07
68.
74.65
58.2
63.25
19

---

SERQ: Saliency-Awan Low-Rank Error Reconstruction for LLM Quantization
Table 12: Accuracy results of low-rank matrix decomposition methods when tested with W4A4 settings.
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL.
(L)
BoolQ
(1)
PIQA
(f)
SIQA
(1)
ARC-e (1)
ARC-c
(1)
HellaS.
(1)
WinoG. OBQA
(1)
(f)
Avg. (1)
Human. (1)
Other (1)
SocialS,
(1)
STEM
(1)
Avg.
(1)
baseline
5.47
77.74
79.05
46.11
74.49
46.25
44.2
39.72
47.12
47.45
34.28
41.83
1aMMA-277
M.. i-b- 1)
6,32e+2
43.3
49.84
34.08
27.44
25.43
26.46
50.18
22
34.85
25.04
23,46
24.6
24.2
24.41
QARR
7.37
66.73
76.28
43.09
65.95
39.51
70.24
61.56
38
57.67
33.41
39.59
38.45
30.45
29.63
PU MXPP4
6.3
73.33
76,71
44.37
70.71
42.41
72.3
66.93
40.8
00.95
33.41
39,59
38.48
30.45
35.22
SERQ (RTN)
6.03
75.96
77:26
43.96
72.26
44.8
73.71
65.08
40.2
61.77
35.78
42.87
41.7
33:78
38.03
SERO (GPTTQ)
5.07
74.56
78.13
44.22
72.9
43.09
73.88
66.61
41.6
61.87
34.67
41.33
40,92
32.54
37.03
SERQ-MXFP4
6:22
73
77.2
44.17
70.2
43.69
72.59
68.19
41
61.26
32.16
40.01
39.1
31.43
35.25
0-Sbot Common Seuse Reasoning tasks
MMLU
Model
Methods
PPL
(
BoolQ
(1)
PIOA
(t)
SIOA
(1)
ARC-e
(f)
ARC-c
(f)
HellaS,
(7)
WlnoG. OBQA
(1)
(f)
Avg.
(1)
Human. (t)
Other (1)
SociaSS,
(1)
STEM
(1)
Avg.
(1)
baseline
4.58
80.6]
80.52
47.39
77.53
49.23
79.38
72.38
45.2
66.53
47.89
59.29
61.16
42.21
$2.04
LLaMM-2-133
LIK. 1.tt:4 ()
2.23ex3
40.24
49.89
35.11
26.64
26,19
26.08
49.8
24
24.1
23.04
22.26
22.2
23.04
LPQER
6.27
74.31
76.99
44.12
71.25
44.88
73.71
63.61
40
61.15
38.55
44.06
45.73
35.68
40.7
L. QER: MXFFP
5:46
77.86
78.13
46.47
74.37
47.1
75.8
70.24
43
64.12
43.34
$2.95
54.6
40.37
47.27
SEEQ (RTN)
5:04
78.35
80.25
46.37
76.39
46
78
70.64
44.6
65.14
45.89
55.29
56.45
41:14
49.22
SERO (GPTQ)
5.2
78.29
29.27
46.06
78.63
46.5
78:01
70.01
46.8
64.82
43.83
53.3
55.48
47.17
SERO MXFPN
5.39
77,86
78.75
45.19
75.59
48.12
76.18
70.09
43
61.35
43.59
52.33
56.6
40,28
47.19
0-Shol Common Sense Recsoning tasks
MMLU
Model
Methods
PPT.
(3)
BoolQ
PIQA
SIQA
(T)
ARC-
(?)
ARC-c
(t)
HellaS.
(7)
WinoG. OBQA
(T)
(f)
Avg.
Human. (t)
Other
SociatS, STEM
(1)
Asg.
(7)
baseline
6.13
81.07
80.74
47.08
77.66
52.99
79.2
73:48
45
67.16
54.81
70.55
73.25
53.89
62.13
LLAMA-3-8B
L.M.
4.87es2
50.89
32.65
23.95
23.29
28.29
48.3
27,4
36.12
25.16
23.56
22.55
22.39
23.61
L'QUER
11.44
61.71
72.63
42,02
62.37
39.39
67.29
50.54
37.4
55.44
35.47
42.68
42.7
34.06
38.33
L QER MXFPP
7.83
76.12
77.97
44.01
74.66
48.29
74.6
59.38
11.6
63.33
48.57
61.15
62.89
45.58
53.82
SERO (KEN)
74.5
44.37
46.67
72.98
46.14
43.83
51.84
SERO (GPTQ!
7.35
76.85
43.55
44.88
73.6
69.38
40.8
62.41
SFRO-MXFP4
45.
75.13
47.72
$3.48
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL
BuolQ
PIQA
SIQA
ARC- (f)
ARC-c HellaS. WinoG. OBQA
Arg.
Human. (t)
Other SocialS, STEM
Avg.
(3)
(1)
(t)
(t)
(t)
(1)
huseline
9.75
63.67
74.48
42.84
60
36.18
63.73
99.83
37.4
54.82
34.92
41.1
39.78
32.29
36.76
LLaMA-32-IBB
LIN, 1 ()
1.7e+3
51.03
33.73
26.5
22.61
26.92
27.8
24.21
24.59
24.87
24.11
Q.RRRR
50.83
$1.19
60.01
36.10 18
40.95
26.62
43.41
$2.57
26.4
42.17
25.02
26.68
25.41
25.88
25.67
13.78
53.91
$6.75
26.67
27.1
SERO (RTN)
55.93
32.94
29.11
SERQ (GPTQ)
12.50
34
SERQ-MXFT4
13.71
27.55
27.23
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL.
BoolQ
PIQA
SIQA
ARC-c
(f)
ARC-e Hella8. WlneG.
(t)
(1)
OBQA
(t)
Arg.
Human. (t)
Other
SocialS. STEM
Avg.
(1)
baseline
7.81
72.97
77.53
46.93
71.39
46.25
73.49
59.53
43
62.66
48.78
63.08
62.59
44.72
54.06
LLaMA-32--3B LIN. 1 ()
4,37e+2
45.41
52.67
33.93
28.07
22.61
27.94
29.49
26
35.77
25.02
24.27
22.07
21.88
23.5
L'QER
14.11
57.37
69.21
40.53
53 $3.28
34.04
60.73
58.17
34
50.92
30.44
34,63
33.99
30.89
32.25
L PQUR MXFPS
63.57
41.47
68.37
SERQ (RTN)
73.72
54.96
46.36
SERO (GPTQ)
9.15
67.52
74.48
73
69.57
66.38
38
$3.97
38.09
45.7
SEKO-MXFF
8.74
68.56
73.83
61
40.02
67.16
57.5
38.88
47.89
41.33
0-Shot Common Reasoning tasks
MMLU
Model
Methods
PPL
(5)
BoolQ
(1)
PIQA
SIQA
(1)
ARC-
(1)
ARC-c HellaS. WinoG. OBQA
(1)
(1)
(1)
(1)
Human. (t)
Other SocialS,
(1)
STEM
Avg.
(1)
baseline
S03
77.58
78.75
50.1
73.11
47.1
73.62
68.51
41.8
63.82
56.77
71.07
76.15
60.96
65.12
Qwen 2.5 3B
TON. 1im 1) P.ER
R.51e+5
51.96
50.49
34.19
24.58
25. 09
26.52
96.96
30.6
36.67
22.12
23.66
21.9
21.63
22.97
221.5
40.00
56,91
35.06
35.45
25.17
33.78
$1.93
28.6
39.37
24.65
24.91
22.56
22.68
23.81
1. "QUR-MNH-H
72.51
75.84
47.13
73.4
45.31
67.21
64.25
28.8
35.87
SERQ (RTN)
72.6
75.63
49.03
72.
44:11
41.8
61:24
69.97
53.5
59.52
SERO (GPTQQ)
9.35
74.74
75.75
45.55
45.31
69.92
39.6
60.78
52.54
65.92
70.45
53.12
59.55
SERQ-MXTTN
9.79
75.17
74.81
46.88
69.57
44.28
68.85
63.85
40
60.43
49.8
61.92
64.9
51.32
$6.13
20

---

SERQ: Saliency-ANe Low-Rank Error Reconstruction for LLM Quantization
Table 13: Accuracy results of distribution flattening methods when tested with W4A4 settings.
0-Shot
Reasuning
MMLU
Model
Methods
PPL.
BoolQ
PIQA (t)
SIQA
(t)
ARC-e
ARC-c
HollaS, WinoG,
OBQA
Avg.
Human.
Other
SociatS.
STEM
(*)
Avg.
(t)
hoseline
5.47
77.34
79.05
46.11
74,89
46.25
76
68.9
44.2
64,00
39.72
47.12
47.45
34.28
41.83
Smsoothi
49.55
47.51
LLaMA-2-7B
SmonthhQgt (8))
7.49
74.48
41.45
39.83
67.99
61.64
57.15
28.63
30.4
QuaRot
72.84
77.3
33.06
72.3
59.53
32.52
SpinQuant
73.8
76
46.1
43.6
71.3
73.2
65.4
40.4
61
33.9
38.5
37.5
29.5
34.8
SERO MXTTN
6.22
73
77.2
16.17
70.2
43,89
72.39
68.19
61.26
32.15
40.01
39.1
31.43
35.25
0-Shot Common Sems Reasoning tasks
MMLU
BuolQ
HollaS. (1)
WinGG,
Avg.
SocialS,
STEM
(t)
(1)
(t)
(t)
(f)
(1)
(1)
(t)
(1)
(?)
(*)
(t)
baseline
81.52
47.39
77.53
72.38
66.53
47.89
61.16
42.21
$2.04
SuscottO)
1.32es4
38.47
49.24
34.7
26.81
27.73
25.72
47.75
25.8
34,83
24.48
26.62
23.4
23.83
23.85
LLaMA-2-138
SmonthQUgt 28)
6.31
75.23
76:03
43.19
70.5
44.45
72.52
68.43
30
61.28
35.9
44.13
45.76
35 68
39.8
QuaRot
3.41
78.47
78.89
13.32
75.7
46.25
76.29
70.48
62.55
43.68
$2.85
55.41
59 11
47.25
SpinQuant
5.2
78.2
79.3
46.3
49
76.1
77.1
69.5
42.8
64.8
43.5
53.1
55.4
39.1
47.8
SERQ-MXFTM MXFIN
5.39
77.86
76.73
15.19
75.99
48.12
76.18
70.09
43
64.35
43.59
52.33
54.6
40.28
47.19
0-Shot Common Sense Reasoning tasks
MMLLU
Model
Methods
(1)
BoolQ
(t)
PIQA
(7)
SIQA
(1)
ARC-e
(1)
ARC-e
(1)
HellaS,
(t)
WinGG,
(f)
ORQ4
(1)
Avg.
(1)
Human. (t)
Other (t)
SociaS. (t)
STEM
(*)
Avg.
(1)
baeline
70.55
62.13
SmoothhQ
4.71e85
50.61
51.69
33.78
24.2
26.02
26.65
49.57
28.2
36.34
26.65
24.36
24.21
24.71
25.17
LLAMA-3-BB
SmoothQgg112
17.26
61.83
64.2
39.92
46.90
30.38
55.61
59.59
33.6
48.97
29.33
30.16
29.61
28.1
29.1
QuaRot
8.41
70.49
77004
12.96
69.57
43.36
72.22
64.64
42.8
59.12
42.76
$3.56
53.56
41.74
47.29
SpinQuant
8.26
73.4
75.2
44.4
72
46.9
71.9
67.7
42.4
61.75
46.8
56.5
57.2
42.5
49.93
SERQ-MXFFV
7.63
76.15
77.
46.11
72.89
45.39
75.13
68.43
62.71
47.72
60.7
63.18
46.84
53.48
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL.
BoolQ
PIQA
SIQA
ARC-
ARC- HellaS,
WinGG, ORQ4
Avg.
Human, Other (1)
SociatS. STEM
Avg.
(T)
(1)
(1)
(f)
(f)
(7)
(?)
(t)
(*)
(t)
beseline
9.75
63.67
74.48
42.84
60.44
36.18
63.73
59.83
54.82
34.92
41.1
39.78
32.29
36.76
Smoothh
1.53e85
51.8
32.7
26.35
28.36
35.59
24.59
24.39
23.44
24.37
1LaMA-322-18
SmoothO(g1220
69.22
51.62
56.09
35.26
37.9
49.35
40.04
24.65
26.1
23.24
23.63
24.43
QuaRot
69.7
39.92
55.25
50.03
27.62
26.
26.64
SpiuQuant
68.8
26.38
SERQ-MXTT4
0-Shot Common Sense Reasoning tasks
MMLU
Model
Methods
PPL
BoolQ PIQA
SIQA ARC- ARC-c HellaS, WinoG, OBQA Avg.
Human. Other SociatS. STEM
Avg.
(7)
(*)
baseline
7.81
72.97
77.53
46.93
71.59
46.25
73.49
69.53
43
62.66
48.78
63.08
62.59
44.73
54.06
Smocthh))
LLaMA-3 2-38
1m10thQ(21255)
QuaRot
75:34
SpinQuant
73
67.6
47.7
SERQ-MXFTM
8.74
68.56
73,83
45.24
67.16
64.33
57.5
38.68
34:44
41.33
21


Reviews:
[{"review_id": "XmdYmXgkzn", "reviewer": "Reviewer_emTw", "summary": "This paper proposes a saliency-aware error reconstruction method named SERQ for LLMs in W4A4 using a single low-rank compensation matrix. The main contributions are threefold: static activation flattening, saliency-aware error reconstruction, and offline weight permutation. Experimental results show accuracy improvements with small computation overhead.", "questions": "* Ablation study: how much does static activation flattening affect your results?\n* What are the design space of choosing saliency metrics? What do the authors mean by \"sufficient\"?\n* Could the authors comment on how their work compares to QERA (ICLR 2025)?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Well-written with good presentation\n* Systematic evaluation and good results", "weaknesses": "* Missing related work\n* Missing theoretical justification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MlqiJ2PPkR", "reviewer": "Reviewer_cLFy", "summary": "This work proposes SERQ, a post-training quantization method that uses a single low-rank term to only compensate the error for salient weight rows in a pretrained linear layer, which further improves the inference throughput over baselines.", "questions": "Apart from the questions in Weakness section, \n\n1. What does the \"Quarot\" block in Figure 2(b) mean? Is this for rotating activations?\n2. SERQ is a heuristic based method. An ablation study of the static activation flattening seems needed here. \n    - SmoothQuant was originally proposed for INT8 quantization, while SERQ uses MXFP.\n    - In line 262, \"However, unlike standalone prior approaches, the combined use of low-rank reconstruction enables effective compensation for the induced weight errors.\". Thus the static activation flattening transfers the challenge of activation quantization to weights using the scaling $s$, but the term to compensate to this compensate weight error, $R$, is in low-precision in SERQ. I wonder whether the activation flattening is really helpful in this case considering MXFP has shared scales to accommodate outliers.\n3. The GPU performance evaluation is very helpful. Could the author offer more details of the setup of GPU performance, like the batch size, token numbers, and tokens per second throughput?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The motivation is clear and the method is straightforward but effective\n- The evaluation includes both model performance and hardware results", "weaknesses": "- The evaluation lacks model larger than 13B: all results were collected on model between 1B and 13B. It will be convincing if the author could offer results on larger models like 70B to verify the scalability of SERQ\n- Details of runtime/hardware performance needs further clarification. Could the author elaborate more how a SERQ layer is accelerated on GPU? are there kernel fusions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AZvLQOvXo", "reviewer": "Reviewer_oK6e", "summary": "The paper proposes SERQ, a method for 4-bit quantization of large language models that uses a single low-rank compensation matrix to correct quantization errors. The approach combines three components: (1) static activation flattening via SmoothQuant-style scaling, (2) saliency-aware error reconstruction using only the most significant weight rows, and (3) offline weight permutation to avoid runtime overhead. The authors claim their method enables true end-to-end 4-bit computation while outperforming prior low-rank error reconstruction methods like L2QER and rotation-based approaches like QuaRot and SpinQuant.", "questions": "Please address my concerns in Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "● Novel Design for Latency Reduction: The paper proposes a scheme that combines a single compensation matrix with an offline permutation, with the goal of eliminating the latency overhead found in conventional two-factor error correction methods.\n● Presents Experimental Results on Multiple Models: The paper reports experimental results across several modern LLMs and includes latency measurements on recent hardware to support its efficiency claims.\n● Addresses a Significant Problem: The work targets the challenging and important problem of W4A4 quantization, which is of high interest to the community for efficient LLM deployment.", "weaknesses": "While the paper presents some interesting ideas, it suffers from significant flaws in its theoretical grounding, experimental validation, and practical considerations, which severely undermine the credibility and value of its contributions.\n1. Critically Flawed and Incomplete Experimental Validation\n● Flawed Experimental Comparisons: The paper's main results are built on unfair comparisons. For instance, SERQ is allocated a higher bit-budget (4.37 bits) than its main competitors (~4 bits), which may in itself account for its accuracy advantage. Furthermore, the performance of the key L2QER baseline is suspiciously low on certain models, casting doubt on the entire set of comparative results.\n● Critically Limited Scope and Unproven Generalization: The paper’s experimental validation is fundamentally disconnected from modern, real-world LLM scenarios. The evaluation is confined to small-scale (<13B), dense models, failing to address three critical dimensions: (1) Scale: It lacks validation on large-scale models (70B+), where quantization challenges are most severe. (2) Architecture: It provides no evidence on diverse, prevalent architectures like Mixture-of-Experts (MoE), where its core saliency heuristic may not hold. (3) Context Length: It fails to evaluate performance in long-context scenarios (e.g., 8K-32K), which are vital for modern applications and where quantization errors are known to accumulate. This critically narrow scope makes it impossible to assess the method's true scalability and practical utility.\n2. Dismissal of Practical Issues and Lack of Transparency\n● Dismissal of Practical Drawbacks: The paper touts its offline permutation as a \"zero-latency\" solution while completely ignoring the immense software engineering challenges and loss of modularity it introduces, especially concerning its interaction with distributed inference techniques like tensor parallelism. This dismissal of practical drawbacks makes the method's utility questionable.\n● Missing Analysis of Quantization Cost: The paper completely omits an analysis of its quantization cost. As a Post-Training Quantization (PTQ) method, the time and compute resources required for calibration are critical efficiency metrics. The absence of this data prevents a full assessment of its overall cost-effectiveness.\n● Poor Explanation of Key Components: The paper fails to clearly explain how its best-performing variant (SERQ-GPTQ) works. The description in the appendix is contradictory and confusing, which is a major reporting weakness for a paper that claims top results with this variant and impacts its reproducibility.\n3. Weak Theoretical Underpinnings and Incremental Contribution\n● Incremental Contribution and Overstated Novelty: The method is essentially a patchwork of existing techniques (SmoothQuant, AWQ's saliency metric, LoRA-style correction). Its primary novel component—the offline permutation—is more of an engineering trick than a fundamental algorithmic breakthrough. The paper conveniently ignores the incremental nature of its contribution and overstates its novelty.\n● Limited Theoretical Justification for Saliency Metric: The method relies entirely on a heuristic for saliency borrowed from AWQ (identifying rows by activation scale). The paper fails to provide any theoretical justification as to why this simple statistical metric is optimal for error reconstruction, nor does it rigorously compare it against other potential channel selection paradigms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5Snz7kJOC", "reviewer": "Reviewer_3ZBf", "summary": "This paper introduces SERQ. While existing low-rank adaptation methods are effective, they often face severe accuracy degradation in low-bit W4A4 (4-bit weight, 4-bit activation) settings and require two sequential factors, which limits inference efficiency. SERQ addresses this by employing a single low-rank compensation matrix to jointly mitigate quantization errors arising from both activation and weight saliency. The method operates in three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. This design preserves efficient 4-bit matrix multiplication, keeps latency overhead minimal by performing most operations offline , and empirically outperforms prior error reconstruction and state-of-the-art rotation-based W4A4 approaches.", "questions": "What quantization format is used for flattening methods in Table 2?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is clear, quantizing both main branch and lora branch to W4A4, as well as reducing the overhead of lora branch through saliency channel detection.\n2. The ablation studies is comprehensive.", "weaknesses": "1. More recent state-of-the-art baselines—such as DuQuant [1], OSTQuant[2] and FlatQuant[3] for W4A4KV4 quantization—should be included for comparison.\n2. SVDQuant [4] also use channel-wise scaling (static activation flattening in this paper) and LoRA error compensate, though it was used in diffusion model. This paper should disscuss more about such similar work.\n3. End-to-end prefilling and decoding speedup should be included to demonstrate the efficiency of proposed method. \n4. The comparison results in Table 2 seems unfair. Flattening methods and proposed SEQR use different quantization formats. Additionally, the SEQR-MXFP4 average bit-width is higher than QuaRot and SpinQuant.\n[1]  DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs, NeurIPS 2024\n[2] OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting, ICLR 2025\n[3] FlatQuant: Flatness Matters for LLM Quantization, ICML 2025\n[4] SVDQUANT: ABSORBING OUTLIERS BY LOW-RANK COMPONENTS FOR 4-BIT DIFFUSION MODELS, ICLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)