No outlier channels but with outlier blocks
Authors: Shanwen Mao
Hao Zhang
Jiasheng Li
Haoyu Qiao
Chenxin Cai
Tingting Wu
Jie Liu
Venue: ICLR 2026 Poster (2026)
Keywords: outliers
Quantization
Primary area: foundation or frontier models, including LLMs
TL;DR: Flexible arbitrary bit-width non-uniform quantization with multi-level outlier compensation for efficient LLM compression.

Abstract: With the rapid scaling of large language models, achieving efficient compression while maintaining model performance has become a critical challenge. To address the limitations of existing non-uniform quantization methods, which typically rely on fixed codebooks and require costly optimization, we propose a novel arbitrary bit-width non-uniform Quantization (NuBitQ). The framework enables flexible, layer-specific quantization strategies, significantly enhancing adaptability and efficiency. Notably, traditional outlier compensation methods used in uniform quantization are ill-suited for the anomalous distribution characteristics encountered in our context. To address this, we design a novel outlier evaluation metric that integrates weight perturbation, activation distribution, and perturbation propagation. Based on this metric, we further develop an Outlier Compensation Plugin (OCP) that implements multi-level, fine-grained outlier compensation strategies, effectively mitigating performance degradation caused by outliers. Our approach avoids direct complex Hessian computation and fine-tuning, offering strong applicability and scalability. Extensive experiments on multiple tasks and across various model series demonstrate the effectiveness of the proposed approach.

Published as a conference paper at ICLR 2026
No OUTLIER CHANNELS BUT WITH OUTLIER BLOCKS
Shanwen Mao!.2 1.2 Hao Zhang .2+ Jiasheng Li
Haoyu Qiao!.2
Chenxin Cai Tingting Wu Jie Liu
Harbin Institute of Technology, Harbin, China
2 State Key Laboratory Smart Farm Technologies and Systems, Harbin, China HHarbin Engineering University, Harbin, China
Harbin Institute of Technology (Shenzhen), Shenzhen, China
{24s103313, 2021113691, 24s103393, chxcai)estu. hit 2hh1000@htt.edu.co
www. edu, ttwu@hrbeu.c wwwu@hrbeu.edu.cm en jieliuehit. edu.. cn
ABSTRACT
With the rapid scaling of large language models, achieving efficient compression while maintaining model performance has become a critical challenge. To ad- dress the limitations of existing non-uniform quantization methods, which typi- cally rely fixed codebooks and require costly optimization, we propose novel arbitrary bit-width non-uniform Quantization (NuBitQ). The framework enables flexible, layer-specific quantization strategies, significantly enhancing adaptability and efficiency. Notably, traditional outlier compensation methods used in uniform quantization are ill-suited for the anomalous distribution characteristics encoun- tered in our context. To address this, we design novel outlier evaluation metric that integrates weight perturbation, activation distribution, and perturbation prop- agation. Based on this metric, we further develop an Outlier Compensation Plugin (OCP) that implements multi-level, fine-grained outlier compensation strategies, effectively mitigating performance degradation caused by outliers. Our approach avoids direct complex Hessian computation and fine-tuning, offering strong ap- plicability and scalability. Extensive experiments multiple tasks and across various model series demonstrate the effectiveness of the proposed approach
6-=0-x-0f
Uniform Quantization
By channel
Non-uniform Quantization
By Sy-channel
Eg 0 X
Figure 1: Comparison of uniform non-uniform quantization effects. Left: original matrix multi- plication; middle: uniform and non-uniform quantization; right: quantization error comparison.
"Corresponding author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable performance across various tasks. How- ever, their deployment is significantly constrained by substantial demands on computation and memory. For instance, running LLaMA3-70B in FP16 precision requires least four A A100-40GB GPUs (Dubey et al. 2024), rendering real-world deployment costly and often impractical.
Quantization has emerged key technique address the challenge by converting floating-point model parameters into low-precision integers, significantly reducing model size and inference la- tency with minimal performance loss. Uniform quantization, predominant approach, divides the value range into equal intervals using per-channel scale and zero scro-point parameters (see the upper part of Figure D.. However, it is particularly vulnerable to outliers (rare but large-magnitude val- ues), which lead to uneven quantization errors. This is especially problematic in channels with high activation variance, where these errors concentrate (illustrated as peaks in the "By channel" sub- plot at the upper right part of Figure (T). ultimately degrading model performance. To mitigate this, techniques such as LLM.int8 (Dettmers et al., 2022), AWQ (Lin al. 2024b), and FlatQuant (Sun) et al. 2025) introduce various outlier compensation mechanisms, including isolating outliers, ad- justing precision for critical channels, and applying affine transformations.
In contrast, non-uniform quantization (lower part of Figure D leverages clustering-based methods to build more flexible codebooks that better match real-world distributions of model parameters. This significantly reduces quantization error (smaller and more uniform errors in the lower-right part of Figure However, introduces new challenges that error patterns become less predictable and more scattered across channels, resulting in novel outlier characteristics different from uni- form quantization. Therefore, existing amplitude-based compensation strategies tailored for uniform quantization do not generalize well. Furthermore, ofate-of-the-ar non- non-uniform quantization meth- ods such as AQLM (Egiazarian et al. (2024) and VPTQ (Liu et al. (024) typically rely on enlarged codebooks or residual fitting to reduce overall error, but often overlook error sensitivity. Although techniques like BCQ (Elangovan et al. (225) and GPTVQ (Van Baalen et al.) (2024) attempt to address this via layer-wise fine-tunin tue-tuning or Hessian- guided clustering, these approaches impose sig- nificant computational and memory overheads, making them less viable for large-scale models.
To overcome these limitations, we propose a quantization framework called NuBitQ, designed to support flexible, layer-wise quantization with high efficiency. Crucially, NuBitQ is complemented by a modular OCP that addresses the unique outlier distribution introduced by non-uniform quanti- zation. We begin by formulating layer-specific outlier impact metric that combines three dimen- sions: weight perturbation, activation distribution, and perturbation propagation. The metric enables precise identification of the layers containing critical outliers.
Building upon the insight, NuBitQ introduces fine-gre tra-graiddd, multi-codebookk, multi-vector quanti- zation strategy that adapts bit-width and codebook design per layer. In parallel, OCP implements multi-granularity compensation scheme across (i) individual linear outliers, (ii) attention discrepan- cies Transformer blocks, and (iii global output distribution, together providing hierarchical and targeted eгror correction. Notably, our approach eliminates the need for layer-wise fine-tuning or Hessian matrix computation, making it scalable and practical for LLM compression.
Our main contributions are summarized as follows:
We proposed NuBitQ, an efficient and flexible non-uniform quantization framework that supports arbitrary bit-widths and enables layer-wise wise differentiated strategies for improved adaptability and compression.
We found the unique distribution outliers in non-uniform quantization. We designed a plug-an houg-and-play OCP module based a novel layer-wise outlier impact metrie that jointly considers weight perturbation, activation statistics, and error propagation.
We verified our method's 's effectiveness and scalability on representative LLMs, including LLaMA Dubey et al. 2024). Qwen (Yang et al., 2025), and Gemma (Team et al.) 2024). Results show near-lossless performance at 4-bit precision and strong competitiveness at lower bit-ws,ths, outperforming existing non-uniform quantization methods.
2

---

Published as a conference paper at ICLR 2026
ourflers +2.35
Layer
Figure 2: Overview of NuBitQ-OCP: The top-right part shows NuBitQ, fine-grained layer-wise quantization method. The bottom-right and left parts together constitute OCP: the bottom-right illustrates compensation methods at different granularities, while the left side displays outlier scores that highlight outliers guide the compensation.
RELATED WORK
2.1 NON-UNIFORM QUANTIZATION
Non-uniform quantization uses variable intervals based on data distribution, reducing global error and enabling lower-bit quantization. For instance, [Teng et al. (2024) applies vector quantization to exploit the spherical sub-Gaussian distribution of incoherent weights, achieving higher preci- sion but relying independent second-order optimization, yet it uses only two codebooks for original values and resid- uals. AQLM (Egiazarian et al., 2024) advances LLM compression through additive quantization and layer-wise fine-tuning while depending on fixed codebook sizes per layer. GPTVQ (Van Baalen al. 2024) improves quantization for L.LMs by increasing dimensions and incorporating per-layer output MSE and Hessian information, though Hessian computations incur significant costs. In contrast, our method avoids Hessian computations and, through adaptive codebook arrangements, achieves flexible bit-width compression, yielding optimal performance at 4- bit results.
fixed codebook. VPTQ (Liu al. 2024) enhances granularity with channel-
2.2 OUTLIERS
Research shows some feature dimensions in LLM have outlier values much larger than other chan- nels (Kovaleva et al. 2021), making quantization challenging. Mitigation methods mainly fall into types: relies on first-order statisties of weight and activation values before and after quan- tization, such as magnitude, to identify outlier channels (Dettmers et al., 2023)or uses smoothing factors reduce quantization errors (Xiao al., 2003). The other employs second-order statistics, utilizing Hessian matrices or their variants, such as Fisher Information Matrices (Kim et al.) (0024), to reduce outlier impact through Hessian weighting (Frantar et al. (003). rotation transformations (Ma et al. 2024) (Lin et al., 2024a), or other affine transformations (Huang et al., (2024) (Saxena et al.) 2025). These methods target outlier channels. Studies (Gong et al. 2024) show that outlier handling heavily influenced by the quantization method, with clipping uniform quantization causing prominent outlier channels, while non-umin non-uniform quantization produces more dispersed and smaller errors that cannot be simply identified by channels. Dedicated compensation methods for outliers in non-uniform quantization remain limited. We proposes a specialized optimization strategy to better handle outlier blocks in non-uniform quantization, enabling effective compression below 4 bits.

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 NUBITQ: FLEXIBLE QUANTIZATION
To address the challenges outlined in the previous section, designed NuBitQ, flexible non- uniform quantization that supports layer-wise customization and ultra-low bit-widths. NuBitQ builds upon the codebook index paradigm, with several key improvements aimed at achieving a better trade-off between compression ratio and model accuracy.
As illustrated in Figure 2 given linear layer weight matrix of size m. first divide it into xxm subvectors of dimension d. These subvectors evenly assigned to groups, each equipped with learnable scaling factor q. For each group, we construct a codebook containing cluster centers, each of which is vector of length d.
d
To improve clustering quality, we apply beam search with width b during the k-means procedure. Furthermore, we introduce residual quantization strategy by using r sequential codebooks: the first codebook encodes the original subvector, while each subsequent codebook encodes the residual enror from the previous quantization step. As result, each subvector is approximated by sequence of r indices, each pointing to a codeword in its corresponding codebook. Hence, the compressed model stores r x d codebook parameters and nxm indices, replacing the original Hoating-poii weights.
In terms of memory usage, the original weight matrix, stored as 32-bit floats, requires n x x 32 bits. In the quantized form, each index requires logg ciiss, resulting in total of rx pxm x xlog, e bits. The codebooks, stored at 16-bit precision, consume weight matrix is large, the codebook memory becomes negligible compared to the index storage. Therefore, the total quantized memory be approximated the of index and codebook storage. We denote the compression ratio R as the ratio of quantized memory to original memory: R logge exexdxax1c
x d 16 bits in total. Notably, when the
When n is sufficiently large, the second term becomes negligible,ani we further reduce memory by re-clustering the codebooks each group. The compression ratio simplifies to
X log2 R ~0
32 x d
By performing a grid search over the number codebooks r, the number cluster centers e, and the subvector dimension d, it allows for flexible adjustment of the prrrssioonperfommance trade-off, supporting diverse quantization configurations from bigc-precision to ultra-low- bit settings.
3.2
OCP: OUTLIER COMPENSATION PLUG-IN
3.2.1 OUTLIER PATTERN
In low-bit uniform quantization, large errors often concentrate in specific channels, forming clear outlier channels that significantly degrade model accuracy. However, in non-uniform quantization, errors tend to be generally smaller and more dispersed, making standard outlier detection methods less effective. Here, we define outliers as quantization errors that substantially impact accuracy. A common heuristic is to use the difference between quantized and original weights to identify poten- tial outliers. Yet, further analysis reveals these errors are not equally harmful- some contribute to performance degradation, exhibiting patterns clustered at the block level, while others are relatively benign.
To investigate this, we analyze the impact of 2-bit quantization on LLaMA3-8B's Transformer blocks 0 through 31. We observe significant variability in quantization sensitivity across blocks: for example, independently quantizing Transformer block 1 causes the largest increase in perplex- ity (PPL), whereas other blocks show minor effects (see Figure Ba). Simultaneouslyy we provide a detailed breakdown of the internal blocks (see Figure 3bb and perform experiments with different input data (see Figure 3c). This suggests that outliers do not appear as isolated channels but rather localized blocks.
4

---

Published as a conference paper at ICLR 2026
Apel oakktear
Tayer
Layer13row
Staple
(a) Quantize Only Layer to bits
(b) Only L to 2 bits
(c) Laaerl. 2bits on Sample
Figure From observing the outlier Transformer blocks of the entire model, to examining the outlier sublinear within outlier Transformer block, and finally to analyzing the outlier sample inputs of the outlier sublinear.
Quantization error of weights
Input
activation
0.9675
0,8790
0.6623
0.8743
0.3218356
0.58:9435
0.0138336
963369979
Inner Product
0.6368802
1.1185847
0.0300466
0.0218548
Output
0.0917659
0,0697323
0.2717550
0.0011277
Figure 4: Simplified Diagram of Output Error Caused by Non-Uniform Quantization
To illustrate how non-uniform quantization errors develop into outliers the block level, consider the following simplified example. Suppose the vector dimension = 4, and after quantization with codebook replacement, the quantization errors for four vectors in block are shown in Figure
Although the overall block error is low due to vector clustering, some individual vectors incur sig- nificant single-point deviations fit their assigned codebook entries, leading amplified output erros. These scattered errors can be further magnified by input activations and propagated through subsequent layers, eventually developing into significant outliers that markedly affect model perfor- mance.
Overall, the impact of errors between quantized and original weights model performance varies. The outliers that cause performance drops are unevenly spread across blocks and depend on the input data.
3.2.2 OUTLIER SCORE
From the above observations, we develop a theoretical metric to quantify the impact of weight outliers within each Transformer block. The metric accounts for hierarchical structure and input- output correlations to better capture the propagation of quantization- induced errors.
Consider the i-th Transformer block containing 7 linear sublayers. We define the original and quan- tized weights for the j-th sublayer W and Q(W.,), respectively. The difference between them is AW,, = W Q(W.,), regarding as the weight perturbation. The perturbation propagates through the layer's nonlinear computations, ultimately influences model output. We de- scribe its propagation via Jacobian matrices (Novak et al. 2018):
7
AY, J. Σ Jr (AW,,). f=1
where Ji, represents the local sensitivity of the j-th sublayer's weight perturbation the layer output, and J, denotes the aggregated sensitivity from layer to the final model output. To simplify computation, we approximate the outlier impact using the expected squared Frobenius norm (Sato & Suzuki 2024) of the output perturbation:
4=
E ((W [[.WW.)...(AWW.....
5

---

Published as a conference paper at ICLR 2026
where M,, -., represents both the sensitivity of the sublayer to perturbations and their amplification through subsequent layers. Specifically, the sublayer's sensitivity is mainly determined by the input activation covariance matrix C2, E2' و which reflects how dif- ferent input directions affect the output locally. The amplification of perturbations as they propagate to the final output is mostly governed by the norms of downstream weights, approximated by the product of squared Frobenius norms I [Will?
Following Hessian trace approximation techniques Dong et al. 2020). we decompose the Eq. (3) into three interpretable components:
The perturbation magnitude: |
The trace of the input activation covariance: tr(C,,)), representing sensitivity to weight perturbations.
The product of squared Frobenius norms of subsequent layers' weights: Ih W
describing hierarchical amplification.
To better integrate values of different magnitudes and mitigate the impact of numerical disparities, we adopt the logarithmic form and define the logarithmic outler score for Transformer block
3 log |l + log (r,g)))) + Σ log W
(4)
Ami+1
The metric integrates perturbation magnitude, input statistics, and inter-layer propagation effects, effectively reflecting the potential influence of outliers in each Transformer block on overall model performance. More detailed analysis of our proposed outlier score is in Appendix D
3.2.3 OUTLIER COMPENSATION
Based on the guidance of outlier scoring, we maintain an outlier codebook pool and employ a slid- ing window mechanism select codebook entries that align with our compensation strategies. We design three hierarchical compensation strategies to mitigate performance degradation caused by quantization. These methods target different structural levels based on the severity of outlier impact and aim to restore model performance with minimal overhead. Each strategy optimizes a compensa- tion term, either a codebook or weight delta, so that the compensated quantized weight ((WW)+WW better approximates the original weight w*.
MSE Minimization (Linear-sublayer level). For the i-th Transformer layer's j-th linear sublayer, given input activation 24,j E Ra (batch size feature dimension h), we optimize the com- pensation perturbation AW,, by minimizing the expected output mean squared error induced by quantization perturbations, where we let Q* Q(W.,)) + AW denote the combined quantized weight and compensation perturbation. Then the optimization problem
AW arg min Ez,, AW.
- Q"
(5)
The strategy leverages input activation statistics finely calibrate weight compensation, suitable for sublayers with prominent outlier scores.
Attention Score Deviation Mininization (Transformer-blll block level). For the i-th Transformer blocks with atle-atten,ion, we minimize the difference in attention score matrices before and after compen- sation:
0 arg min 0.
A (., 13
(6)
where A,,),) denotes the attention score matrix after compensation. The strategy preserves the model's self-attention capability and works well for structurally complex layers with concentrated, stable outlier perturbations.
KL Divergence Minimization (Whole-model level). For coarse-grained global compensation, minimize the KL divergence between the original and quantized model output distributions:
8" arg min Ezg. DKL. (p" ((

---

Published as a conference paper at ICLR 2026
Table 1: Perplexity results of various bit quantization methods applied multiple language models. Quantization method used (A AQLM, V VPTQ, G GPTVQ. N ours, O=OCP).
#Bits
Llama3-8B
Qwen3-8B
Gemma2-9B
Qwen3 14B
Gemmaa-277
Llama3-70B
Wiki Ptb
Wiki L Ptb J Wiki L. Ptb J
Wiki L Ptb J.
Wiki 4 Ptb 1
Wiki L Ptb J
16
5.57
8.92
8.58
13.71
10.69
37.03
7.58
12.09
5.39
17.49
2.53
7.06
6.04
9.50
8,91
14.59
10.91
39.41
8.06
13.68
6.02
19.63
2.85
7.68
6.10
9.60
22.67
35.11
66.02
274.77
16.20
27.99
54.73
133.90
3.12
7.85
G
5.81
9.18
8,86
14.21
10.70
37.35
8.01
14.48
5.91
18.28
2.63
7.61
N
5.79
9.14
8.81
14.17
10.68
37.39
7.86
12.90
5.69
18.06
2.59
7.54
7.05
11.11
9.49
16.63
11.43
42.14
8.91
14.14
6.73
21.40
3.45
8.11
A+0
6.93
10.99
9.33
16.33
11.32
42.07
8.79
14.02
6.68
21.37
3.42
8.09
3
7.01
10.46
30.43
49.92
185.33
939.84 839.84
23.55
37.10
113.40
441.72
3.62
9.28
V+0
6.86
10.23
12.34
25.23
14.44
53.06
19.49
19.32
8.91
46.98
3.59
9.27
G
6.03
9.98
9.42
15.14
11.13
39.66
8.99
14.46
6.65
22.17
3.07
7.87
G+O
6.12
9.73
9.33
14.99
11.04
38.73
8.85
14.35
6.58
21.88
3.05
7.83
N+O
5.66
8.98
8.87
13.79
10,80
38.09
8.63
13.27
6.22
20.54
2.98
7.60
A
7.28
11.61
10.15
17.54
12.27
49.43
9.75
15.59
7.97
27.95
5.52
8.59
A+0
7.07
11.40
9.85
17.03
12.04
49.28
9.52
15.44
7.85
26.76
5.44
8.51
9.19
12.77
1.6566
8.08e6
3.27e6
4.93e6
1.08e6
2.22e6
1.28e6
3.79e6
6.19
10.75
V40
8.92
12.02
5.34e4
3.64e4
1.06e5
8.83e5
8.98e5
9.55e5
7.65e5
6.93e5
6.13
10.67
G
10.34
21.89
13.19
28.07
13.83
56.30
9.83
22.52
7.47
27.24
5.43
8.48
G+O
9.14
20.69
12.91
27.69
13.41
54.85
9.61
22.29
7.13
27.02
5.32
8.39
N+O
6.42
9.74
9.35
15.32
11.45
47.98
9.33
14.99
6.94
26.74
4.99
where p" represents the output distribution of the original model, p denotes the output distribution of the quantized model, and DKL is the Kullback-Leiblee divergence, used to measure the difference between the two probability distributions. The strategy directly maximizes the probability of gener- ating correct tokens, suitable for overall performance restoration and preserving high-level semantic consistency.
These strategies adjust compensation from fine to coarse granularity based on outlier scores and re- sources, mapping the granularity differences shown in the Outlier Pattern. Moreover, the selected data samples are taken from the few outlier samples identified above. Our main goal is to reduce overall errors while increasing the chance of generating correct tokens. The improvement comes from the optimization goals, not the specific compensation methods. Follow-up experiments show all compensation methods perform better than traditional quantization.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We follow the evaluation protocol of LLMCBench (Yang et al. 2024) to compare the perfor- mance of our method against AQLM (Egiazarian et al. 2024). vPTQ (Liu et al.] 20024) and GPTVQ (Van Baalen et al. 2024) in terms of knowledge ability, reasoning ability. and reliability. The evaluation tasks include perplexity measurement on WikiText2 (Merity et al. (017) and Penn Treebank (PTB) (Prasad et al.] 2014), well multiple benchmarks such MMLU (Hendrycks)
et al.
2021), QNLI (Rajpurkar et al. 2016). MNLI (Williams et al. 2018). AdvGLUE (Wang et al. and TruthfulQA (Lin al. (022)) Detailed experimental settings Appendix E.
202
4.2
MAIN RESULTS
4.2.1
4-BIT QUANTIZATION COMPARISON
Table 4-ittt shows that, at 4-bit without using the OCP, our method achieves the lowest perplex- ity among all quantization methods. This indicates that NuBitQ effectively preserves the model's generative and comprehension capabilities while compressing it, highlighting its superiority. As the model size increases, the perplexity of our method approaches that of FP16. This suggests that approach adapts better to complex language patterns in larger models and retains important fea- ture information effectively during quantization, thereby reducing performance loss. It should
7

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy of different bit quantization methods on Llama3-8B across various tasks.
Knowledge ( (%)
Inference (%) t
Trustworthiness ( (%) f
Method
#Bits
Hums.
STEM
Social Other
Avg.
QNLI
MNLI
advglu
T.mel
T.me2
16
55.05
53.38
73.90
69.59
62.18
50.90
40.95
44.17
28.52
46.75
AQLM
4
49.34
51.90
67.60
65.05
57.83
49.81
36.68
42.95
25.46
44.37
VPTQ
52.77
50.83
70.69
66.04
59.34
50.84
36.18
43.50
27.30
45.87
GPTVQ
$1.80
48.87
69.39
65.55
58.20
51.05
39.09
44.58
25,46
42.93
NuBitQ
53.43
52.82
71.99
68.66
60.88
50.73
42.05
44.04
25.95
44.83
AQLM
29.59
29.82
35.53
31.83
31.41
49.32
34.56
43.36
25.21
43.29
AQLM+OCP
29.54
30.15
35.72
31,92
31.58
49.27
34.67
43.22
25.34
43.32
VPTQ
50.75
47.78
67.01
63,82
56.69
50.58
38.45
43.26
25.83
43.15
VPTO+OCP
50.83
49.11
66.92
63.93
57.70
50.12
38.97
43.76
26.33
44.18
GPTVQ
43.68
42.35
58.01
55.06
50.71
50.71
35.06
44.58
28.03
44.96
GPTVQ+OCP
43.49
41.68
57.59
54.84
48.81
50.57
33.74
43.50
27.78
44.97
NuBE
54.75
53.02
74.16
69.65
62.07
50.71
40.79
44.72
27.17
46.15
AQLM
42.23
39.63
50.02
47.84
44.67
49.75
36.41
43.36
25.58
42.40
AQLM-OCP
40.09
41.83
51.51
50.74
45.63
49.62
38.90
43.36
25.34
42.85
VPTQ
2
42.12
38.84
50.99
42.92
43.69
49.21
34.54
43.22
25.21
46.35
VPTO+OCP
2
43.18
40.92
52.17
45.86
45.53
49.62
36.78
42.94
26.89
46.90
GPTVQ
28.86
24.12
30.94
24.89
26.81
48.49
33.67
43.36
26.56
47.79
GPTVO+OCP
24.10
28.96
30.78
24.86
26.78
48.12
33.46
43.36
26.19
47.80
NuBiQQ+CCP
2
49.48
48.61
67.47
64.81
56.77
49.66
49.60
43.77
29.63
47.58
be clarified that, since the VPTQ method does not provide the Hessian matrix data required by its own approach for Qwen3 and Gemma 2, the results obtained without Hessian tuning lead to VPTQ quantization performance being lower than the theoretical values.
Table 2 shows that NuBitQ achieves the highest accuracy 4-bit. This indicates that not only does our method excel in perplexity, but it also demonstrates competitive accuracy in task performance, enabling effective quantization. Notably, our method's ' accuracy the underlined positions exceeds that of FP16, which is pleasant surprise post-quantization. The result further supports the effective- ness of our approach and showcases the potential of our quantization strategy in specific scenarios.
4.2.2 ULTRA-LOW-BIT QUANTIZATION COMPARISON
We integrate our OCP with various methods for quantization below bits across different mod- The perplexity results presented in Table The results indicate that method achieves the lowest perplexity. Additionally, when other methods integrate our OCP, the perplexity can be fur- ther Cecduced.Howeeee since AQLM and GPTVQ themselves employ fine-tuning techniques for op- timization, the scope for improvement with OCP is limited. For VPTQ on Qwen3-8B and Gemma2- 9B, as its own optimization was not enabled, our compensation method significantly reduced the perplexity. Table 2 reports the accuracy results after quantizing the Llama3-8B model. Our method combined with OCP achieves the highest accuracy, while other methods using our OCP occasionally show a decrease in accuracy on certain tasks; however, the overall accuracy remains improved.
From these experiments, we observe that the outierr-compensated non-uniform quantization meth- ods consistently outperform their original counterparts, particularly when the model size is smaller (such as 7B and 13B) or when the bit-width is lower (such as 2-bit). The phenomenon indicates that OCP serves powerful enhancement for non-uniform quantization.
4.3
ABLATION STUDIES
4.3.1
IMPACT OF QUANTIZATION PARAMETERS
In the 4-bit quantization setting of the LLaMA3-8B model, we quantized only the 7th linear layer of the model's s O-th layer and observed the changes in perplexity Figure 5a Intuitively, larger values of parameters r, c, and are preferable, while a smaller value of parameter d is advantageous. However, under the constraint of maintaining 4-bit quantization com- pression ratio, we can see that: it clearly shows a significant impact of the r parameter on perplexity, with optimal range identified. It indicates that smaller values of d yield better performance, but
the WikiTex12 dataset, as shown in
8

---

Published as a conference paper at ICLR 2026
Level
Trmndemtr Level ModelLeval
(wacctot-2)
20
129
160
224
64
139
15e
512
104
200
Threshold Selection
11
(a) Effects of Hyperparameter Selection quantiza- tion.
(b) Different combinations of thresholds for the 2- bit NuBitQ quantized LLaMA3-8B are presented.
Figure 5: The red dashed line on the left indicates the baseline perplexity before quantization, and on the right indicates the perplexity before compensation after quantization. The bars indicate the proportions of three compensation strategies utilized across total of 32 layers.
Table 3: Overall ablation study on compensation methods and clustering configurations.
(a) Compensation strategies on the 31st layer of (b) Computation time (T) and memory usage (M) for clus- LLaMA3-SB
tering vector lengths in the 7th linear layer
Strategy
Time (8)
Mem (%)
APPL
FP16
d=4
d=8
d=16
d=32
d=64
Random
7.43
1.00%
1.00x
Ada
T(ms)
134.3
63.6
63.7
91.3
93.5
98.8
Linear
51.65
1.00%
5.27x
M(MB)
378
144
144
145
148
176
Transformer
15.53
0.29%
2.26x
T(ms)
84.0
50.1
53.0
54.9
56.5
63.0
Ampere
Model
8.33
0.14%
2.12x
M(MB)
408
144
144
144
148
176
both excessively small or large values can lead to increased perplexity. The effects of parameters, and с on model perplexity are relatively minor, resulting in flatter curves. This suggests that the and parameters have smaller and more stable impact quantization performance, while the and d parameters play a dominant role in overall performance.
It is worth noting that points marked with pentagons in the figure indicate that the post-quantization perplexity is lower than the baseline before quantization, which is somewhat counterintuitive. Sub- sequently, attempted increase the value in the parameter combination and found that the quantization performance actually declined (in Appendix [E..3), indicating that such results are somewhat coincidental. However, this also highlights the significant potential of non-uniform quan- tization parameter combinations in performance optimization.
4.3.2 IMPACT OF COMPENSATION STRATEGY
Table Ba illustrates the impact of different compensation strategies. Using the random compensation strategy as baseline, which compensates for 1% of the original memory. The results indicate that the linear-level compensation consumes the most memory and time, yet provides a compensation effect that is 5.27 times greater than that of the random strategy. In contrast, the model-level compen- sation uses the least memory, occupying only 1,4, while achieving performance improvement of 12 times over the random strategy.
Specifically, linear compensation requires more resources but provides finer correction; model com- pensation uses less memory but is coarser; Transformer compensation balances both and suits gen- cral layers. Choose compensation based on needs to improve performance and save resources.
4.3.3 IMPACT OF COMPENSATION THRESHOLD
Different combinations of thresholds lead to varying proportions of compensation strategies. A lower 0, indicates a greater use of model-level compensation, while a lower 82 signifies a greater of livear-level compensation.

---

Published as a conference paper at ICLR 2026
Figure 5b presents our results: when the threshold combinations is set to O, achieve signifi- cant reduction in perplexity. As the threshold combinations increase from 1-4, incorporating more Transformer-leeel compensation, the perplexity shows overall decreasing trend, although at slower pace. When the threshold combinations range from 6-14, compensating for more linear-level layers, the perplexity decreases more rapidly compared the previous range.
These results highlight the importance of selecting appropriate thresholds for optimizing model performance and reducing perplexity. If memory constraint, can opt for lower threshold 0, while if accuracy is paramount, a lower threshold 02 is more appropriate.
4.3.4 IMPACT OF VECTOR LENGTH SELECTION
Using FP16 precision as a baseline, we compared computation times and peak memory usage for various vector length settings (d 4, 16, 32, 64) across two hardware architectures: Ada Lovelace and Ampere. The results are shown in Table 3h For hyperparameter selection, to maintain a 2-bit compression rate, we fix r and let change with the vector length This design ensures that the compression effect is preserved while adapting to different vector length settings.
The findings indicate that under the Ada Lovelace architecture, using vector lengths of results in computation speeds approximately 2.1 times faster than FP16, while memory consumption de- creases by about one-third. As the vector length increases, the speed decreases and memory usage increases, illustrating a trade-off between performance and resource utilization. A similar trend is observed in the Ampere architecture. Overall, shorter vector lengths (4 8) contribute to enhanced inference efficiency and reduced memory consumption, suitable for efficient deployment.
5 CONCLUSION
In this paper, we introduced novel non-uniform quantization framework NuBitQ. The framework enables flexible layer-specific quantization strategies, outperforming existing methods at 4-bit preci- sion. To enhance the effectiveness of ultra-low-bit quantization, we developed modular OCP that leverages layer-specific metrics to maintain near-lossless performance. Experiments demonstrate that the NuBirQ-OCP framework excels in various natural language tasks. Currently, research focused on LLMs. In future we could explore multimodal models to improve scalability and effectiveness in more complex scenarios.
ACKNOWLEDGEMENTS
This work is supported in part by the Project of Laboratory of Advanced Agricultural Sciences of Heilongjiang Province under Grant No. ZY04JD05-010, the National Natural Science Foundation of China under Grant No. 62350710797.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
T. Dettmers, M. Lewis, Y. Belkada, et al. Gpt3.int80: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-30333 2022.
T. Dettmers, R. Svirschevski, V. Egiazarian, et al. Spqr: A sparse-quantized representation for near-lossless IIm weight compression. arXiv preprint, 2023.
Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang. Zhiwei Xiong, Baoqun Yin, and Yunhe Wang. CBQ: cross-block quantization for large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL /tLthhts: /tELDp:://opnreeiee... net forum? did= 9
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ-V2: hessian aware trace-weighted quantization of neural networks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL
newrips...p... CC wwtprrisscccppppppeeee htterrriis.ccpppper////////eee/ http/pppper/222/aathtt
d7777035367 809588813330000 (e033300331t2e5ffeee Abst html
A. Dubey, A. Jauhri, A. Pandey, al. The llama herd of models. CoRR, abs/2407.21783, 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. In Forty-first International Conference Machine Learning, 2024.
R. Elangovan, C. Sakr, A. Raghunathan, et al. Beq: Block clustered quantization for 4-bit (w4a4) Ilm inference, arXiv preprint, 2025.
Christopher Fifty, Ronald Guenther Junkins, Dennis Duan, Aniketh Iyengar, Jerry Weihong Liu, Ehsan Amid, Sebastian Thrun, and Christopher Ré. Restructuring vector quantization with the rotation trick. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL /RLhhtsss///ppereeeeee net/forumt wow.roommnididdieeeeeee
E. Frantar, S. Ashkboos, T. Hoefler, et al. Optq: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. What makes quantization for large language model hard? an empirical study from the lens of per- turbation. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on In- novative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 18082 18089. AAAI Press, 2024. doi: 10.6609/AAALV381 297665. URL https: //doi. 10. 1609//69ia...... v38i16........ 29765
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- net www.mfrrmmmiooo wwdoormmmidddddii. Jm 3GmQ
view.net, 2021. BRRLhttts:///peeeeeee
Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, Jiangyong Yu, Chen Xu. Zhi- hang Yuan, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:/:::// /pwpr/eieee/eeeeiiee....e..eeeee net wttts//foommiideeeeee http://rmmm..dd.pad........
Xijie Huang, Zechun Liu. Shih- Yang Liu, and Kwang Ting Cheng. Rolora: Fine-tuning rotated outlier-free free 1mms for effective weight-activation quantization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7563-7576, Miami, Florida, USA, 2024.

---

Published as a conference paper at ICLR 2026
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Ma- honey, and Kurt Keutzer. Squeezellm: Dense-and-spatee quantization. In Forty-firs International Conference Machine Learning, 2024.
O. Kovaleva, S. Kulshreshtha, A. Rogers, et al. Bert busters: Outlier dimensions that disrupt trans- formers. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp- 3392-3405, 2021.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized LLMs, In The eiirty- Annual Conference on Neural Information Processing Systems, 2024a. URL ptttps://openrview..ro net/ Eorum? deieiiidmppptpi emaz
J. Lin, J. Tang. H. Tang, et al. Awq: Activation-aware weight quantization for on-device Ilm com- pression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Asso- ciation for Computational Linguistics, 2022. doi: ong 229
URL
https: httt::://doi.orgg
Y. Liu, Wen, Y. Wang, al. Vptq: Extreme low-bit vector language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, 2024.
quantization for large
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang. Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, Vienna, Austria, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. hRRpttps: / //operreview net wwwp::nttteetttteeemiem ByaimndddByy22aaae
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha
Sensitivity and generalization neural networks: an empirical study. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL
Forum? Pi HJC22ZZZC
Rashmi Prasad, Bonnie L. Webber, and Aravind K. Joshi. Reflections on the penn discourse tree- bank, comparable corpora, and complementary annotation. Comput. Linguistics, 40(4):921- 950, 2014. doi: 10.116//COL1 URL https:://oll.org//...1111/ C0204
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2383-2392. The Association for Computational Linguistics. 2016. doi: 10.18653/V1/DI1- 1264. URL https: /ttps:///oi... wtw.:...rr/....1...111..11 d16-1263
Kazuhiro Sato and Masato Suzuki. The nearest graph laplacian in frobenius norm. Numerical Linear Algebra with Applications, 31 2578, 2024. doi: https:://oi. wwtp://///....100222....... URL
https://onl wnte ibrary.brr...r wi ey. com/ doi /ttp://////ee 1002 a
12

---

Published as a conference paper at ICLR 2026
Utkarsh Saxena, Saych Sharify, Kaushik Roy, and Xin Wang. Resq: Mixed- seeed-precision quantization of large language models with low-rank residuals. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2025. URL ptttps://cperrevieee
forum? d=sRROm="TTT HyW.
Y. Sun, R. Liu, H. Bai, et al. Flatquant: Flatness matters for Ilm quantization. In Forty-second International Conference on Machine Learning, 2025.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, tean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Harleen
Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black. Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang. Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona COmanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Me Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ron- strom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dra- gan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi. and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL http::/p........oo/////ooo/
wrxiv. org/ abs//bs//22 2400s/2248... 00118
A. Tseng, J. Chee, Q. Sun, et al. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint, 2024.
M. Van Baalen, A. Kuzmin, M. Nagel, et al. Gptvq: The blessing of dimensionality for IIm quanti- zation. In Workshop on Efficient Systems for Foundation Models II at ICML. 2024, 2024.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Has- san Awadallah, and Bo Li. Adversarial GLUB: multi-task benchmark for robustness evaluation of language models. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: httass///dattssttt-eennnmmmaa wwwp:erdiisoooooeoeeeeeeee www.iinnnnnn.uuurnee wttpiiiss.cccpppeee pcpppppp///22///////////////// haaa//3555522288000/329022222222 wownstrrattoound.httte html
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda
13

---

Published as a conference paper at ICLR 2026
Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018. Volume 1 (Long Papers), pp. 1112-1122. As- doi: 10. 10.11855//1/1188-111
sociation for Computational Linguistics, 2018.
URL https:
7/doi. orgei.org/10 000
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099, 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: / http//axvvv../// org/ abs/ 2505. 09388
Ge Yang, Changyi He, Jinyang Guo, Jianyu Wu, Yifu Ding, Aishan Liu, Haotong Qin, Pengliang Ji, and Xianglong Liu. Llmebench: Benchmarking large language model compression for efficient deployment. In Amir Globersons, Lester Mackey, Danielle Bel- grave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference
Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: httt:://papers nips. cc wacp/pappr_fiicce files hasss/app//222/ Abweeeel-dd--bbttreeee Datasets and
2aah/9944c6220662216611 Benchmarks Track html
Yuxuan Yue, Zukang Xu, Zhihang Yuan, Dawei Yang, Jianlong Wu, and Liqiang Nie. PCDVQ: enhancing vector quantization for large language models via polar coordinate decoupling. CoRR, Abs/2506.05433. 2025a. doi: 10.45550AAXXX225560552222. URL https:: /oi. org/10.
18550/arXiv 2506. 05432
Yuxuan Yue, Zukang Xu, Zhihang Yuan, Dawei Yang, Jianlong Wu, and Liqiang Nie. PCDVQ: enhancing vector quantization for large language models via polar coordinate decoupling. CoRR, abs/2506.05432, 2025b. doi: 10.48550/ARXIV.25000553333 URL hhttt::: //w.///.o//o org/10.
48550/arXiv 2506. 05432
A DIFFERENCES IN METHODS
In this section, we examine the differences among several quantization methods. For brevity, we re- fer to these methods in Table 4 using their respective abbreviations: QUIP (Tseng et al. 2024).
VPTQ (Liu et al. 2024), AQLM (Egiazarian et al. 2024), GPTVQ ( (Van Baalen et al.
2024
PCDVQ Yue al. 2025a). Rotation (Fifty et al.] 2025). SSVQ Yuue et al. 2025b), SpinQuant
(Liu et al. 2025). ), OSTQuant (Hu et al. 2025). and CBQ Ding et al. 2025).
Since we focusing non-uniform quantization for large language models, our experimental comparisons select several baselines from the latest non-uniform quantization methods. Among them, PSSVQ and Rotation intended for visual models, and PCDVQ has publicly released its code.
Notes:
[1] Uses two codebooks; channel-independent 2nd-order optimization.
[2] Additive quantization; layer-wise fine-tuning.
14

---

Published as a conference paper at ICLR 2026
Table Comparison of Quantization Methods Key Features. Non-uniform Q: whether the quan- tization is non-uniform; Layer Q: layer-wise quantization; Channel Q: channel- quantization; Outliers: handling of outlier values; Adaptive CB: use of adaptive codebook; Any BW: support for any bit-width; 2nd-order Info: use of second-order information in optimization.
Method
Non-uniform Q Layer Q Channel Q Outliers Adaptive CB Any BW 2nd-order Info Notes
VPTQ
x
Partial
x
X
[1]
AQLM
X
X
X
[2]
GPTVQ
/
X
X
X
/
[3
PCDVQ
X
X
X
X
[4]
Rotation
X
Partial
X
X
[5]
SSVQ
X
X
J
X
[6]
QUIP
X
Partial
X
X
X
[7]
SpinQuant
X
X
[8]
OSTQuant
X
A
X
[9]
XX
CBQ
X
[10]
X
DuQuant
[11]
ResQ
/
X
[12]
Ours
[13]
[3] Dimension expansion; Hessian-barn based quantization.
[4] Converts weights to standard Gaussian distribution, similar to QUIP.
[5] Uses rotation and scaling linear transformation smooth gradient propagation.
[6] Separates sign and absolute value of weights; uses iterative freezing for optimization.
[7] Fixed codebook; exploits sub-Gaussian weight distribution.
88] Adds Hadamard rotation for low-bit activation and KV cache quantization.
[9] Introduces learnable transformations to enhance quantization space utilization.
[10] Proposes cross-block dependency mechanism to maintain inter-layer relationships.
[11] layer-wise differentiated strategies and novel outlier impact metric
[12] uses rotation and permutation to smooth outlier
[13] combines PCA and rotation to reduce activation outliers
B ALGORITHM SUPPLEMENT
Our proposed quantization method supports flexible bit-width allocation for each linear layer within Transformer model. We mention arbitrary bit quantization, more specifically, the combination of bit-widths assigned to different linear layers collectively reflects the overall arbitrary bit quantization effect for the entire model. We allow independent selection of quantization parameters for each layer to realize the desired bit-width configuration. Within the search space formed by these parameter combinations,
seek a suitable solution that balances compression and accuracy. It is important to note that in the current work, we have not optimized the parameter search procedure itself. Instead, use common grid search techniques determine the optimal parameter allocation. Our main focus remains on improving the quantization algorithm itself, particularly in handling outliers after quantization.
The quantization workflow proceeds as follows: for each Transformer layer, we sequentially com- press seven linear sublayers. Different quantization parameters can be chosen for each sublayer, resulting in varied compression effects. Within each layer, we select the hest-performing configu- ration for given compression ratio. For the cross-layer parameter selection, we formulate grid search problem with the objective of maximizing overall model performance while satisfying constraint on the total model size. This procedure is summarized in Algorithm [ which takes as input the original weights, candidate configurations (each with associated compression ratio and performance metric), and a maximum size budget. The algorithm outputs the selected parameters, codebooks, and quantization indices.
15

---

Published as a conference paper at ICLR 2026
Algorithm NuBitQ Compression Parameter Selection for Transformer blocks
Input: Original weights W*, candidate configurations {(a Pi,s.) size limit SizeMax Output: Selection ai, E {0, 1}, codebooks Ci,s, indices idxi,.
I: Prepare (Ri,s.h P.,..) for all 8,8,
2: Solve:
max
Σ 1,6
Pree Fi,e
Σ Di,s = 1,
s.t.
Σ Risj LayerSize,, Fias S SizeMax.
6,80
3: For each i, 8, find
5 arg max mi,si,
4: Quantize W using NuBitQ with parameters 0,- = (d. g. r,
Split W into subvectors of dimension d, partition subvectors into ! groups; apply scaling, for each group k I.....g: build codebooks ((h), ...... C(k) where c(A) Rexd, 1= 1=1,..... using beam search with width b,
assign indices idx;,e to closest centers in codebooks.
5: Return {Krech {C,,a), {idxi,e}.
This Segorithmmmmppemeettt the core procedure for selecting quantization parameters across Trans- former layers and sublayers. For each linear sublayer, multiple candidate configurations-ddifeee in quantization granularity, codebook size, compression ratio, and other hyperparameters- are pre- pared. The goal is to select exactly one configuration per sublayer to maximize the overall per- formance metric (e.g., accuracy) while ensuring the total compressed model size does not exceed specified budget. The optimization is formulated lem, which we solve via grid search given the manageable parameter space. After selection, the weights quantized using NuBitQ method, featuring vector splitting, group-wise scaling, multi-codehook construction via beam search, and index assignment.
constrained combinatorial selection prob-
To address the accuracy degradation caused by extreme weight values (outliers) that poorly represented by standard quantization, we introduce an outlier compensation mechanism as a com- plement to NuBitQ in Algorithm [ The procedure begins by quantizing the original weights with NuBitQ to obtaín the normal codebook and quantization indices. Then, a sliding window approach scans the weight matrix to identify outliers ciiiierr-eeemettss whose absolute deviation from local statis- tics (mean and standard deviation) exceeds a predefined threshold. Extracted outliers are clustered separately form an seeliff- codebook and corresponding indices. This two-tier approach enables better representation of extreme values without excessively increasing the overall codebook size. Finally, layer-wise compensation applied based on per-layer outlier statistics. Depending on the outlier ratio of each Transformer block, different compensation strategies are used: from direct mean squared error compensation to parameter tuning via KL-divergence or a global model-based update. The reconstructed weight matrix combines the compensated normal quantized weights with the outlier reconstructions, effectively retaining important information carried by outliers and thus improving the accuracy of the quantized model.
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 NuBitQ-OCP with Outlier Compensation
Input: Original weights W*, outlier threshold a, sliding window size
Output: Normal codebook (E), normal indices Q()), outlier codebook E., outlier indices 4.
1: Quantize W with NuBitQ to get Q(E),Q(1
2: Initialize outlier mask <- with shape of W*
3: for each sliding window segment S of size in W do
4: Compute mean μss and std Øs
5: for each element Whe in S do
6: if [ly usl >
then
7:
Mij
8: end if
9: end for
10: end for
Extract outliers O {wn Mij = =}
12: if [0] > 0 then
13:
Cluster to obtain outlier codebook E, and indices Lo
14: else
15: Eg + 5, + 0
16: end if
17: for each Transformer block i do
18: if (3 > 02 then
19: Compute AW; by Eq. (5)
Update W Wcomp.t t QW,,) AW;
21:
else if 01 < 3/ S 02 then
22:
Compute 0 by Eq. (6)
23:
Update layer parameters accordingly
24:
Compute 0" by Eq. else
25:
26: Update global compensation parameters
27: end if
28: end for
29: for each position (i,j) do
30: if Mij then
31:
W Whomp E. (i.jj])
32:
else
33:
W Woomp Q(E)Q())))))))
34:
end if
35:
end for
return Q(), Lo
C ERROR DISTRIBUTION COMPARISON
In this section, we supplement the comparison of error distributions between non-uniform quanti- zation and uniform quantization across various models, illustrated in Figure This comparison aims to demonstrate that the difference in еттоr distribution due to different quantization methods is not only theoretically supported by previous research (Gong al. 2024), but also empirically validated through experiments on a diverse set of models.
D OUTLIER SCORES ADDENDUM
In this section, we will provide a detailed derivation our Outlier Scores, compare the compu- tational complexity between the Hessian matrix and our proposed metric, and finally explain the applicability of our metric. It is important to note that our metric is not purely an empirical ob- servation, nor is it merely theoretical speculation. Instead, it integrates both empirical findings and theoretical derivation. Experimental results further demonstrate that our metric aligns well with practical outcomes.
17

---

Published as a conference paper at ICLR 2026
unifom
LLama3-ae 88
Gemmaz
Qwen
Figure 6: Error distribution between uniform quantization and non non-uniform quantization across dif- ferent models (the analysis is based the weights of the second layer, quantized using 4-bit preci- sion.).
D.1 OUTLIER SCORES' THEORETICAL DERIVATION
D.1.1 NOTATION AND PRELIMINARIES
Consider the i-th Transformer layer, which contains / 7 linear sublayers. Denote the original and quantized weight matrices the j-th sublayer as
W E R X Q(W,,) E R
and define the weight perturbation
AW == Q(W W
Let the final model output be Yz, and the output perturbation caused by quantization be AYz.
D.1.2 PERTURBATION PROPAGATION AND JACOBIAN MATRICES
The perturbation Á AW., leads to an output perturbation AY; from the i-th Transformer layer, which in turn affects the final output Yr. By first-order Taylor expansion, we have the approximation:
AY, R J. J.nn Σ (AW,,))
(8)
where:
Jij = aw., denotes the local sensitivity (Jacobian matrix) of the j-th sublayer's weight perturbation on the layer output perturbation.
aY.
J++L = aY;
aYr denotes the sensitivity from the i-th layer output to the final output.
D.1.3 EXPECTED SQUARED NORM OF OUTPUT PERTURBATION
We are interested in the average magnitude of output perturbation caused by quantization, measured by the expected squared Frobenius norm:
:: E ||YY
Substituting Eq. 8 and assuming the perturbations of different sublayers are mutually independent and zero-mean, obtain
18

---

Published as a conference paper at ICLR 2026
1: E Ji Σ (aW())
(9)
E (AW () M (AW
(10)
where the matrix
Myj = (i
Cross terms vanish due independence.
On the Assumptions of Independence and Zero Mean These assumptions are based the following considerations:
The 7 sublayers within Transformer layer not strictly sequentially connected. Since quantization errors mainly arise independently from each sublayer's weights, it is reason- able approximate their perturbations statistically independent.
The zero-mean assumption corresponds to unbiased quantization noise model, meaning that on average the quantization does not introduce systematic bias, which simplifies the analysis of second-order effects.
D.1.4 INTERPRETATION OF Mijj
The matrix M.,j jointly captures the sensitivity of sublayer to weight perturbations and the ampli- fication effect of perturbations propagated through subsequent layers. It thus encodes the complex coupling of local input activations and downstream propagation.
Decomposition Based on Hessian Trace Technique turing the sensitivity of the loss weight perturbations in sublayer (i.j) can be approximately decomposed into two interpretable components:
The complex second-order matrix M,, cap-
The statistical properties of the input activations, represented by the covariance matrix
C., := E [
where Fij denotes the input activation the j-th sublayer in layer i. This matrix char- acterizes the dominant directions and energy distribution of the input space relevant to perturbations.
The amplification effect of subsequent layers, approximated by the product of the squared Frobenius norms of their weights,
II I
A=++1
serving as a coarse estimation of how perturbations propagate and potentially grow or tenuate through the network.
Consequently, according to Hessian trace approximation techniques (Dong et al. 2020). the quadratic form involving weight perturbations AW,,; can be approximated as
(AW..) |AW | tr((.,))
II k=it1
I
where |WW,, l quantifies the magnitude of the weight perturbation itself, reflecting the intensity of quantization error or deviation in the sublayer.
19

---

Published as a conference paper at ICLR 2026
Mathematical Approximation of the Decomposition By exploiting these independence and low- rank approximations, the quadratic form can be approximated by scalar product:
M.,W,) || tr(C,))
II &
Here:
(r((C,)) approximates the overall energy or sensitivity input activations;
- || || approximates the perturbation amplification factor through subsequent layers;
the original complex high-dimensional matrix Maj is simplified into product of scalar fac- tors.
This decomposition on the core idea of Hessian trace techniques: by leveraging the linearity of the trace and expectation, the complicated second-order matrix impact is transformed into a product of input activation statistics and weight amplification effects, enabling computational simplification and clearer physical interpretation.
D.1.5 DEFINITION OF OUTLIER SCORE
Combining the above, the expected perturbation magnitude of the i-th layer can be approximated as
JAw. L (r(C,,))
II
To avoid numerical instability and convert multiplicative relations into additive ones, define the outlier score via logarithms:
:= log IAA log (r(
Σ log W
(11)
This score collectively reflects the perturbation magnitude, input statistical properties, and inter- layer amplification effect, and can be used to identify the modules most vulnerable to outlier pertur- bations.
D.2 OUTLIER SCORES' COMPUTATIONAL COMPLEXITY
Consider the weight matrix of a linear sublayer W E Ruxx, where m is the output dimension and is the input dimension.
Hessian Matrix The Hessian matrix is the second-order derivative matrix of the loss function with respect the weights, having size (mn) (mn).
Storage complexity: O(m"n²) which grows quadratically with the number of parameters, resulting in very high memory requirements.
Computational complexity: Direct computation of each second derivative also costs about Omm?n?). Although Hessian-vector product approximations exist, they still require multi- ple forward and backward passes and remain computationally expensive.
Therefore, computing and storing the Hessian matrix is typically infeasible for large-scale models.
Outlier Score From the derivation, the key computation in the outlier score involves:
(AW) AWW MAW = ||WI tr(C)) II
W I2,
20

---

Published as a conference paper at ICLR 2026
where:
|AW = >. Σラ
() can be computed in (mm) time.
The input activation covariance matrix C' = E[ I E R is estimated by sampling S input activation vectors (I(11,
..... . where S denotes the number of sampled activations used to compute the empirical covariance. Computing the full covariance matrix
requires (Sn²² operations and Onn²) storage.
However, since only the trace tr(C) = Ei= E[z.] is needed for the outlier score, we only compute and store the diagonal elements (i.e., the average squared activations per input dimension). This reduces the storage complexity to Onn), and the computation to ((Sn).
The product of Frobenius norms of subsequent layers' weights, II ||Wi involves oper- ations on much smaller matrices and therefore has negligible computational cost compared to Hessian computations.
Hence, the overall computational complexity of the outlier score is approximately
O + Sn + Σ mgmk);
with storage complexity Onn), which is significantly lower than that of explicitly computing and storing the full Hessian matrix.
Complexity Comparison Computing the Hessian matrix is computationally and storage-wise prohibitive in large-scale models due to its quadratic complexity, summarized in Table 5 In contrast, the outlier score avoids explicit second-order computations by employing approximations, reducing the complexity to simpler norm and covariance calculations. Therefore, the outlier score provides a theoretically meaningful and practically feasible metric for quantifying model sensitivity to perturbations.
Computational Complexity
Storage Complexity
Hessian Matrix
O(m²n²)
O(m
Outlier Score
Omn + Sn Σx m.ns)
((n)
Table 5: Complexity comparison
D.3 OUTLIER SCORES' APPLICABILITY
Previous studies Lin et al. (2024b) have demonstrated that during model quantization, the impor- tance the earlier layers significantly greater than that of the subsequent layers, while the final layer directly impacts the ultimate output performance.As shown in Figure 7 when computing the outlier scores for the Llama3-8B model, our metric shows clear positive correlation with these findings, assigning higher outlier scores the carly layers and the last layer. This behavior stems from the design of our outlier score: it incorporates not only the quantization errors of the original weights but also the amplification effect of activations cus solely on the influence activations, our score additionally accounts for the impact of weight perturbations propagating through subsequent layers, which theoretically leads a stronger bias ward the sensitivity of the earlier layers. In summary, this result reflects a well-founded combination of theoretical derivation and empirical performance of our outlier score, validating its effectiveness and applicability as a metric for estimating layer-wise wise quantization sensitivity.
the error. Unlike other methods that fo-
E EXPERIMENT ADDENDUM
E. EXPERIMENTAL SETUP
Hardware Environment All comparative experiments conducted cluster equipped with four NVIDIA A100 GPUs In the ablation studies, the Ampere architecture corresponds to the A100 GPU, while the Ada Lovelace architecture corresponds the RTX 4070 GPU.
21

---

Published as a conference paper at ICLR 2026
Figure 7: Outlier scores of Llama3-8B with 2-bit quantization across 32 layers.
Models The base models include LLaMA3 (Dubey et al.) 2024) with 8B and 70B parameters, Qwen3 Yangg et al. 2025) with 8B and 14B parameters, well as Gemma 2 (Team et al. 2024) with 9B and 27B parameters. These models represent different types and sizes, enabling a comprehensive evaluation of the proposed framework's performance.
Baselines We use FP16 6 precision as the baseline. For 4-bit quantization, we compare our method with AQLM (Egiazarian et al.) 2024), VPTQ (Liu et al. 2024), and GPTVQ (Van Baalen et al, 2024]. and conduct similar comparisons for ultra-low-bit quantization. We also evaluate the inte- gration of our OCP into existing methods. The compensation thresholds set so that 10% of parameters receive linear-level compensation, 10% model-level compensation, and the remaining transformer-level compensation.
Evaluation Metrics The evaluation metrics follow LLMCBench (Yang et al. 2024). Perplexity measured WikiText2 (Merity et al. 2017) and Penn Treebank (Prasad al. 2014). We
comprehensively evaluate three aspects: knowledge ability, reasoning ability, and reliability. Knowl- edge ability, tested by MMLU (Hendrycks et al. (2021), measures world understanding: reasoning ability, evaluated via QNLI (Rajpurkar et al.) (016) and MNLI (Williams et al. 2018), reflects in- ference capacity; reliability, assessed with AdyGLUE (Wang al. (021) and TruthfulQA, gauges robustness to noise.
E.2 ADDITIONAL COMPARATIVE EXPERIMENTAL RESULTS
To further verify the effectiveness of our method in ultra-low-bit quantization, we evaluated Qwen3-8B and Gemma2-9B models. We compared different quantization schemes-2-bii and 3- bit-against 16-bit floating-point baseline. The methods tested include AQLM, VPTQ, GPTVQ, and our approach combined with Optimization Correction Procedure (ours+OCP).
All models were tested on the same datasets covering four knowledge domains: Social Sciences, and Others. We also used downstream tasks (QNLI, MNLI) and trustworthiness metries (advglu, T.me1, T.me2) to assess overall model performance and reliability. Tables and 7 show that 16-bit baselines achieve the best performance. Traditional ultra-low-bit methods like AQLM and VPTQ suffer clear drops in accuracy and trustworthiness, especially below bits.
Humanities. STEM,
Our method with OCP consistently outperforms other ultra-low-bie quantization techniques at both 2-bit and 3-bit levels. It achieves accuracy and inference results close to or better than some 4-bit methods, with more stable trustworthiness. For example, at 3-bit quantization, Qwen3-8B reaches 70.26% average knowledge accuracy, higher than others 71.36% nearly matching the 16-bit baseline (71.86%). These results confirm our approach effec- tively compresses large models while maintaining performance, making it suitable for resource- limited deployments.
and Gemma2-9B hits
E.3 ADDITIONAL ABLATION EXPERIMENTS
E.3.1 COMPARISON BEFORE AND AFTER USING OCP
This subsection experimentally demonstrates the effectiveness and adaptability of the OCP method.
22

---

Published as a conference paper at ICLR 2026
Knowledge (%)
Inference (%) t
Trustworthiness (%)
Method
#Bits
Hums.
STEM
Social
Other
Avg.
QNLI
MNLI
advglu
T.mel
T.me2
16
63.55
72.33
83.23
76.93
72.84
89.88
71.62
58.81
35.00
55.18
AQLM
59.94
73.10
81.98
72.31
71.83
68.46
44.30
54.07
32.80
52.78
VPTQ
4
24.78
25.84
23.33
25.91
24.95
49.51
33.69
43.09
22.64
48.08
GPTVQ
61.74
72.30
82.71
75.72
71.83
89.22
68.92
52.30
34.88
54.71
NuBitQ
62.47
73.00
82.55
77.27
72.55
87.53
69.75
55.69
35.13
55.74
AQLM
54,54
63.35
74,46
69.34
64.21
56.31
63.50
48.37
33.17
52.93
AQLM+OCP
54.73
63.55
74.81
69.62
64.46
56.01
63.75
48.97
34.03
$2.96
VPTQ
25.06
29.03
31.13
25.51
27.35
49,40
34.62
43.36
22.77
48.28
VPTO+OCP
32.33
37.82
37.39
32.91
35.11
50.37
41.73
46.58
31.83
48.39
GPTVQ
56.85
67.00
78.91
72.67
67.52
79.25
50.71
50.68
34.15
55.44
GPTVO+OCP
57.00
67.20
79.04
72.49
67.60
75.63
52.53
51.08
34.27
55.31
NuBitQ+OCP
58.17
68.59
81.02
73.26
70.26
82.49
64.43
53.17
34.55
55.75
AQLM
53.75
59.21
73.32
66.59
62.18
79.60
52.79
52.30
30.48
50.76
AQLM-OCP
52.50
58.05
72.9
66.04
61.30
79.54
56.74
51.76
30.36
50.41
VPTQ
26.65
25.65
24.05
24.21
25.30
50.95
32.40
45.26
22.77
49.95
VPTO+OCP
30.37
33.23
31.78
31.09
31.62
49.23
39.01
47.82
30.03
48,26
GPTVQ
41.89
44.27
53.04
50.89
46.92
54.73
47.80
44.44
30.72
49.15
GPTVO+OCP
42.64
45.00
53.30
50.89
47.39
58.20
47.27
44.17
31.21
49.51
NuBitQ+OCP
54.07
60.64
74.52
67.78
64.25
80.13
57.08
52.83
30.22
51.81
Table 6: Accuracy of different ultra-low-bit quantization methods on Qwen3-8B across various tasks.
Knowledge (%) f
Inference (%) f
Trustworthiness (%)
Method
#Bits
Hums.
STEM
Social
Other
Avg.
QNLI
MNLI
advglu
T.me1
T.me2
16
64.99
65.44
83.49
76.74
71.86
87.36
68.61
60.16
43.33
60.69
AQLM
65.14
64.55
82.71
75.72
71.31
85.12
69.56
59.21
42.72
60.84
VPTQ
24.14
28.83
31.13
24.98
26.88
50.60
31.29
43.90
23.26
49.13
GPTVQ
65.19
64.21
83.33
76.43
71.55
87.07
69.90
58.67
43.05
60.42
NuBitQ
64.74
65.34
83.49
76.74
71.75
87.38
67.93
60.57
43.21
60.61
AQLM
3
62.42
62.69
80.40
74.06
69.14
87.18
70.27
56.78
40.64
58.55
AQLM+OCP
62.47
62.72
80.47
74.09
69.17
87.33
70.29
57.05
41.13
58.67
VPTQ
24.17
28.86
31.07
25.05
26.89
50.60
31.29
43.09
23.01
48,66
VPTO+OCP
30.99
31.49
37.68
33.04
33.30
48.16
38.50
47.81
39.94
$3.90
GPTVQ
61.91
61.99
80.89
75.29
69.18
88.03
61.04
59.49
42.11
59.83
GPTVQ+OCP
61.90
62.15
81.05
75.48
69.31
88.01
61.00
59.08
41.99
59.92
NuBitQ+OCP
64.19
63.29
82.77
75.17
71.36
87.34
71.98
59.73
42.38
59.91
AQLM
$7.96
55.24
74.13
69.12
63.50
86.75
69.72
48.51
41.49
$8.03
AQLM+OCP
58.04
55.47
74.33
69.15
63.62
86.81
70.28
49.05
41.98
58.30
VPTQ
24.17
28.86
31.07
25.05
26.89
50.60
31.29
43.09
23.01
48.66
VPTQ+OCP
2
29.76
27.09
31.81
31.75
30.10
46.62
35.99
47.28
38.23
$4.37
GPTVQ
51.92
50.60
68.90
63.88
58.12
56.27
45.57
44.99
38.56
55.97
GPTVQ+OCP
51.05
50.20
68.61
63.94
57.69
53.72
44.22
45.53
39.17
55.86
NuBitQ+OCP
59.35
56.94
74.63
70.20
65.28
85.23
70.49
53.22
41.25
58.95
Table 7: Accuracy of different ultra-low-bit quantization methods on Gemma2-9B across various tasks.
In the experiment, we apply 2-bit quantization only to the first layer's Linear sublayer weights of the Transformer model, keeping other layers at 16-bit precision. After quantization, outlier scores for weights and activations are computed, and OCP compensation is applied based on the outlier codebook pool. Evaluation on the validation set uses perplexity as the metric, analyzing changes before and after quantization both sublayer and sample levels.
Figures Sa and compare sublayer perplexity before and after OCP, showing that OCP stabilizes perplexity and prevents abnormal spikes caused by quantization errors. Figures & and 3d show that OCP reduces perplexity variance across individual samples, further confirming its effectiveness in improving model stability and performance.
Our method identifies outliers via outlier scores, which are key sources quantization error. We build an outlier codebook pool with multiple compensation codehooks learned from calibration data
23

---

Published as a conference paper at ICLR 2026
to correct various outlier types. The compensation sliding greedy selection parameter activation tensors, selecting compensation vectors that minimize local quantization error iteratively, achieving fine-graines correction while balancing accuracy and computation.
The compensation ratio at different granularities can be flexibly adjusted by thresholds. By default, 10% fine-grained and 10% coarse-grained compensation, which works well in practice. If needed, one can start with full fine-grained compensation and relax thresholds to find better settings, with minimal extra computational cost.
19p Sinvaaerr
LayorLllacer
Before OCP
(b) Before OCP
4gp (idrrahh)
Laywr
Sanglo
(c) After OCP
(d) After OCP
Figure Perplexity comparison at different stages: before/after OCP compensation.
E.3.2 COMPARISON BETWEEN FINE-TUNING AND OCP
After applying 3-bit quantization to the Transformer layer the Llama3-8B model, we evaluated two categories of recovery methods: fine-tuning and compensation based on OCP. The fine-tuning approaches included layer-wise linear fine-tuning and layer-wise Transformer fine-tuning, while the OCP methods involved compensation the linear layer level, Transformer layer level, and the entire model level. It is important to note that full model fine-tuning was not considered in this experiment, it falls under quantization-aware training, whereas focus here is lightweight strategies to recover accuracy qost-quantization.
Regarding the experimental setup, the OCP compensation utilized activation data from 50 WikiText- 2 samples as reference, following the methodology described in the main text. For fine-tuning, the initial learning rate was set to 10-5 and 500 WikiText-2 samples were used to ensure adequate yet efficient fine-tuning, The final evaluation was conducted on the WikiText-22 validation set using perplexity as the performance metric to assess the impact of quantization and the effectiveness of recovery methods.
Table presents the perplexity results the different recovery strategies applied to the quantized Transformer layer 0. It is evident from the results that OCP compensation consistently achieves lower perplexity than fine-tuning, whether applied at the linear layer level or the Transformer layer level, indicating a more effective and efficient recovery of quantization-induce accuracy degrada- tion. Notably, the OCP compensation applied at the whole-model level also outperforms fine-tuning, demonstrating strong global adjustment capabilities. Furthermore, fine-tuning requires significantly more training data and computational resources compared to OCP compensation, which only needs small set of activation samples, highlighting its superior cost-efficiency.
24

---

Published as a conference paper at ICLR 2026
before
linear
transformer
model
Fine-tuning
5.6176
5.6217
5.6269
OCP
5.5918
5.6014
5.61371
Table 8: Perplexity comparison of fine-tuning and OCP compensation after 3-bit quantization (lower is better).
8. 9.910
X.9136
8.900
8.9076
8.9103
X.912
Perplexity
ptb
wiki
14-
65-
56666
5.5666
5,5674
5,5666
Queelattin Bassles
Layer Snapping Valiiaiio
Figure 9: Analysis of unexpected results.
E.3.3 RANDOMNESS ASSESSMENT PENTAGRAM ANOMALIES
To verify the "unexpectedly good" result observed in the main text's hyperparameter abla- tion-where the perplexity after quantization was surprisingly lower than before quantization we conducted thorough investigation from three perspectives:
First, we examined whether this phenomenon occurs consistently across different datasets under the same quantization setting. Second, we increased the number of groups (both theoretically and empirically, increasing g should lead to an increase in perplexity after quantization). perplexity continues to decrease g increases, it would suggest that the phenomenon is not accidental. Fi- nally, we applied the same hyperparameter configuration to other layers with the same shape. If the "unexpectedly good" result also appears in other layers, it could further rule out randomness.
The experimental results are shown in Figure 9 Here, "Quantization Baseline" represents the per- plexity results of the original hyperparameter setting that produced the anomaly: "Add g" corre- sponds to the results after increasing the number of groups g only; "Layer Swapping Validation" refers to applying the hyperparameters different layers. Each of the six line segments has its left endpoint representing perplexity before quantization and the right endpoint representing perplexity after quantization.
We observe that the "unexpectedly good" result does not reproduce across different datasets, with increased 9, or in different layers. In all other cases, perplexity increases after quantization as expected. This indicates that the anomalous improvement is due to specific combination of dataset, layer, and quantization parameters, and is thus a coincidental outcome rather than generalizable effect.
F REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our results. All datasets used our exper- iments are publicly available. The source code to reproduce experiments will be released anony- mously as supplementary material, including detailed instructions for data preprocessing, model training. and evaluation. Hyperparameters, model architectures, and training settings provided in the Appendix and the readme.txt file. Additional implementation details are also described the supplementary materials.
25

---

Published as a conference paper at ICLR 2026
G LLM USAGE STATEMENT
We used large language models (LLMs) to aid and polish the writing of this paper. All substantive scientific contributions and ideas are our own.
26


Reviews:
[{"review_id": "4sRT7nZsWy", "reviewer": "Reviewer_pUW6", "summary": "This paper proposes NuBitQ, a flexible non-uniform quantization framework for large language models (LLMs), along with a plug-and-play Outlier Compensation Plug-in (OCP) to address outlier-induced performance degradation in low-bit settings. Extensive experiments showing near-lossless 4-bit performance and strong results at 2–3 bits across multiple LLMs.", "questions": "- Can you provide the end-to-end latency or throughput comparison against FP16 or other quantization methods.\n\n- Could the authors present quantitative or visual evidence (e.g., density plots or clustering metrics) supporting this claim?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Introduces a novel outlier score that integrates both local sensitivity and global propagation effects, which is more tailored to non-uniform quantization than prior Hessian-based methods.\n\n- The OCP module is a lightweight, tuning-free compensation mechanism that is also applicable to other quantization methods.\n\n- Supports arbitrary bit-widths and layer-wise customization, offering more flexibility than fixed-bit approaches.\n\n- Comprehensive evaluations on multiple models (LLaMA, Qwen, Gemma) and tasks (perplexity, MMLU, QNLI, MNLI, etc.) demonstrate consistent improvements over SOTA methods.", "weaknesses": "- Lack of Hardware-Aware Latency Evaluation : Although Table 3(b) compares Ada vs. Ampere architectures, there is no end-to-end latency or throughput comparison against FP16 or other quantization methods.\n\n- Lack of Direct Validation:The experimental results do not directly validate these specific theoretical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m9z7AVDYBy", "reviewer": "Reviewer_7kTW", "summary": "The paper proposes NuBitQ (a vector-quantization-style, layer-wise, residual codebook design) and OCP (a three-level compensation guided by a novel block-level outlier score). It achieves near-lossless 4-bit and strong 2–3-bit results on several LLMs, without Hessian or fine-tuning.", "questions": "1. How stable is the outlier score ranking under domain shift and smaller calibration sets?\n\n2. Can OCP selection be cast as a budgeted optimization to auto-derive Pareto frontiers?\n\n3. What is the end-to-end latency/throughput impact on 70B-scale serving?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Identifies a different outlier pattern for non-uniform quantization (block-level), and formalizes a usable score.\n\n2. Practical PTQ: no Hessian, no fine-tuning; clear knobs (𝑟, 𝑐, 𝑑, 𝑔) with an explicit compression formula.\n\n3. Consistent empirical gains at 4/3/2 bits across model families; ablations on hyper-params and compensation granularity are informative.", "weaknesses": "1. While the method avoids fine-tuning/Hessian, it still requires a nontrivial search over (r,c,d,g) and threshold(s) to balance accuracy vs. memory/latency. The paper shows trends, but lacks a budget-constrained, step-by-step selection procedure (e.g., how to pick (r,d) first, then threshold, under a fixed memory/latency cap). This limits immediate reproducibility in large-scale deployments.\n\n2. The paper does not discuss systems constraints: current vendor fast paths primarily target uniform INT/FP formats; non-uniform (codebook/LUT) typically needs custom kernels or on-the-fly dequant, making throughput sensitive to group size, batch size, and cache behavior. A brief analysis of runtime/throughput vs. baseline INT8/FP8 would clarify practicality.\n\n3. Prior work shows that rotation/affine transforms can mitigate outliers before quantization. The paper compares methods side-by-side, but does not systematically evaluate combinations (e.g., rotation → non-uniform weight quant → OCP). A combined study could reveal whether gains are additive or saturating, and where each component contributes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EFFZIg9TMe", "reviewer": "Reviewer_343g", "summary": "This paper proposes a **fine-grained, multi-codebook, multi-vector quantization** strategy that adapts bit-width and codebook design on a per-layer basis. In addition, an **outlier-based compensation** mechanism is introduced to further improve the overall quantization performance.", "questions": "1. Could you include additional baselines such as **DuQuant** and **ResQ** in **Table 4** for fairer comparison?\n2. What is the additional computational or inference cost introduced by **NuBitQ**?\n3. Could you provide end-to-end measurements of speedup and memory usage to validate the claimed efficiency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed outlier compensation plugin effectively enhances the performance of the quantized model.\n2. The evaluation covers multiple types of LLMs and diverse downstream tasks, demonstrating general applicability.", "weaknesses": "1. The discussion in the appendix omits some important state-of-the-art baselines, such as **DuQuant** and **ResQ**.\n2. Certain results are not state-of-the-art but are marked as “best” in Table 2, which may cause confusion.\n3. The proposed non-uniform quantization may introduce additional inference overhead, which is not clearly discussed.\n4. The abbreviation **OCP** is introduced in the Introduction without prior definition or explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYGY2PVs56", "reviewer": "Reviewer_R323", "summary": "This paper addresses the challenges of quantizing LLMs, particularly focusing on the limitations of non-uniform quantization methods. The authors argue that while non-uniform quantization reduces overall error compared to uniform methods, it creates novel and less-understood outlier patterns that are not concentrated in specific channels but rather distributed across what they term \"outlier blocks.\" To tackle this, they propose a two-part framework. The first part is NuBitQ, a flexible non-uniform quantization scheme supporting arbitrary bit-widths and layer-specific configurations without relying on costly Hessian computations. The second contribution is the OCP, a module designed to mitigate performance degradation from these outlier blocks. The OCP is guided by a novel, efficiently computed outlier score that approximates the impact of quantization error by considering weight perturbation, activation statistics, and error propagation through subsequent layers. Based on this score, OCP applies a hierarchy of compensation strategies at the sub-layer, block, and model levels. The authors conduct extensive experiments on LLaMA3, Qwen, and Gemma models, demonstrating that their framework achieves state-of-the-art performance, especially in ultra-low-bit settings.", "questions": "1.The results in Table 2 indicate that applying OCP to AQLM and GPTVQ does not consistently improve, and sometimes slightly degrades, performance on downstream tasks like MMLU. Why does OCP fail to provide a clear benefit in these cases, unlike the significant perplexity improvements shown in Table 1? Does this imply that OCP is specifically tuned for the error patterns produced by NuBitQ or that it might conflict with the optimization strategies already present in methods like AQLM and GPTVQ?\n\n2.The compensation strategy relies on two thresholds, $\\theta_1$ and $\\theta_2$, to switch between model-level, Transformer-level, and linear-level compensation. How were these thresholds determined for the experiments reported in the paper? How sensitive is the final performance to their values, and would they need to be re-tuned for different model architectures or bit-widths, potentially limiting the \"plug-and-play\" nature of the OCP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper provides a valuable analysis of how outlier patterns differ between uniform and non-uniform quantization. The identification of \"outlier blocks\" rather than just \"outlier channels\" as the primary challenge in non-uniform quantization is a key insight that effectively motivates the need for new compensation strategies.\n\n2.The Outlier Compensation Plugin is designed as a modular, plug-and-play component. This highlights its general utility.\n\n3.The paper presents comprehensive experimental results across multiple model families and sizes. The framework demonstrates near-lossless performance at 4-bit and shows particularly strong gains in the challenging ultra-low-bit regimes, outperforming state-of-the-art baselines in both perplexity and downstream task accuracy.", "weaknesses": "1.The paper claims that OCP is a generalizable, \"plug-and-play\" module. While perplexity results in Table 1 support this, the downstream task accuracy results in Table 2 present a more mixed picture. When OCP is added to AQLM and GPTVQ, the average MMLU score shows negligible improvement or even a slight decrease. This suggests that the OCP's effectiveness may be dependent on the underlying quantization scheme, potentially conflicting with the claim of broad applicability.\n\n2.The complete NuBitQ-OCP framework involves a considerable number of hyperparameters, including the quantization parameters, the number of compensation levels, and the thresholds that govern the selection of compensation strategies. Although the paper includes some ablation studies, it lacks a broader discussion on the sensitivity and practical guidance for setting these parameters, especially the compensation thresholds, which appear crucial for balancing performance and overhead. This complexity may hinder the method's ease of adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYW0OxflBn", "reviewer": "Reviewer_jNs7", "summary": "This work studies the (1) Non-uniform quantization, (2) outlier impact, and (3) outlier compensation by minimization. Overall, the ultimate goal of this work is to push the performance boundary in non-uniform quantization. However, I found the organization of this work to be unclear and missing.", "questions": "I am curious to see if this non-uniform weight-only quantization can be accelerated on the modern LLM inference engine such as vLLM and SGLang.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "+ Thorough evaluation of non-uniform quantization across various models and benchmarks.", "weaknesses": "- The figures in this paper are not informative. I cannot understand them if not reading the papers. \n\n- I have several concerns regarding OCP. First, the final OCP methodology seems like the standard calibration with different levels of granularity. This resembles the AQLM's multi-phase optimization that first tackles the layer-wise quantization and then tackles the inter-layer error. Conceptually, there is nothing new about OCP, given that it has been extensively applied in dozens of PTQ works. \n\n- Moreover, the OCP method seems to be compatible with any quantization algorithms, not just non-uniform quantization. If this is the case, then the authors need to compare a lot of uniform baselines. \n\n- Third, I did not understand the necessity of Section 3.2.2. I have checked Appendix D for the detailed derivation of the outlier score. However, I should emphasize that the derivation is definitely not rigorous. The logarithm makes no sense to add. To me, this is just to prevent the exploding effect of the subsequent layers' weight norm. The Jacobian should contain the gradients of na on-linear function, while this work directly ignores them. Furthermore, this outlier score has nothing to do with outliers; it is just a metric to analyze importance. Again, this score seems to be applicable to all kinds of parameters and activations, not just outliers. I could not see how this is *outlier-centric*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)