ICQuant: Index Coding enables Low-bit LLM Quantization
Authors: Xinlin Li
Osama Hanna
Christina Fragouli
Suhas Diggavi
Venue: COLM 2025 (2025)
Keywords: LLM Quantization
LLM Compression
Post Training Quantization
TL;DR: This paper presents ICQuant, a novel framework that leverages outlier statistics to design an efficient index coding scheme for low-bit weight-only quantization.

Abstract: The rapid deployment of Large Language Models (LLMs) highlights the need for efficient low-bit post-training quantization (PTQ) due to their high memory costs. A key challenge in weight quantization is the presence of outliers, which inflate quantization ranges and lead to large errors. While a number of outlier suppression techniques have been proposed, they either: fail to effectively shrink the quantization range, or incur (relatively) high bit overhead. In this paper, we present ICQuant, a novel framework that leverages outlier statistics to design an efficient index coding scheme for outlier-aware weight-only quantization. Compared to existing outlier suppression techniques requiring $\approx 1$ bit overhead to halve the quantization range, ICQuant requires only $\approx 0.25$ bits; a significant saving in low bit regimes (e.g., 2-3 bits). ICQuant can be used on top of any existing quantizers to eliminate outliers, improving the quantization quality. Using just 2.3 bits per weight and simple scalar quantizers, \ours improves the zero-shot accuracy of the 2-bit Llama3-70B model by up to 130\% and 150\% relative to QTIP (Tseng et al. (2024b)) and QuIP\# (Tseng et al. (2024a) respectively; and it achieves comparable performance to the best-known fine-tuned quantizer (Malinovskii et al. (2024)) without any fine-tuning.

Published as a conference paper at COLM 2025
ICQuant: Index Coding enables Low-bit LLM Quantization
Xinlin Lit, Osama Hanna", Christina Fragouli", and Suhas Diggavi*
*University of California, Los Angeles Meta, GenAl Amail:(xinlinli: ohanna, christina.fragoull suaadigaavjj@@@uu..ccuu
Abstract
The rapid deployment of Large Language Models (LLMs) highlights the need for efficient low-bit post-training quantization (PTQ), due to their high memory costs. A key challenge in weight quantization is the presence of outliers, which inflate quantization ranges and lead to large errors. While a number of outlier suppression techniques have been proposed, they either: fail to effectively shrink the quantization range, or incur (relatively) high bit overhead. In this paper, we present ICQuant, novel framework that leverages outlier statistics to design an efficient index coding scheme for outlier-aware weight-only quantization. Compared to existing outlier sup- pression techniques requiring R 1 bit overhead to halve the quantization range, ICQuant requires only R 0.3 bits; a significant saving in extreme compression regimes (e.g,, 2-3 bits per weight). ICQuant can be used on top any existing quantizers to eliminate outliers, improving the quantization quality. Using just 2.3 bits per weight and simple scalar quantizers, ICQuant improves the zero-shot accuracy of the 2-bit Llama3-70B model by up to 130% and 150% relative to QTIP (Tseng et al., 2024b) and QulP# (Tseng et al., 2024a); and it achieves comparable performance to the best-known fine-tuned quantizer (Malinovskii et al., 2024) without fine-tuning.
1 Introduction
Despite their success, the accessibility of Large Language Models (LLMs) is curbed by their significant memory and computational power requirements. Weight quantization, technique that represents model weights using lower precision, is promising to alleviate these demands while preserving the model's performance. Quantization not only enables reduced memory footprint, but also accelerates inference time memory fetch latency is reduced. As a result, quantized models benefit from reduced energy consumption and lower running costs. Quantization opens the road for the deployment of LLMs over resource- constrained devices (for instance, smartphones that have memory and battery constraints), better positions LLMs to serve real-time applications (such speech recognition and real- time translation that require fast inference times), and overall lowers the deployment cost, leading to more sustainable computing and broader usage.
Weight-only post-training quantization (PTQ) has emerged as a popular compression method (Frantar al., 2022; Lin et al., 2024; Zhu al., 2024). It avoids the heavy compu- tational cost of retraining associated with quantization-aware training (QAT) and enables the on-demand quantization at a required precision. Yet, a major obstacle in achieving low-bit (S 4 bits) weight quantization without significant performance deterioration is the presence of weights with exceptionally high magnitudes that reside in the tails of the weight distribution, known as outliers. Outliers significantly expand the required quantization range, making it hard to cover with a small number of bits without detrimental quantization errors.
Given its importance, a number of techniques have been proposed to mitigate the challenge of outliers. For instance, common technique is to clip the outliers to fixed thresholds.
Code available at: https://gtit thtp///gtttuut...eeee//eteee.e wtt.hh..co//vvrr//// http:/veryxxxIIIuuue

---

Published as a conference paper at COLM 2025
Recent work, such BitDistiller (Du et al., 2024) and OmniQuant (Shao et al., 2023), ad- vanced this concept by optimizing the clipping range dynamically, However, the substantial quantization errois associated with outliers limit its practical application. Another prevalent approach is weight grouping (Frantar et al., 2022; Shao et al., 2023; Lin et al., 2024). Weight quantization typically occurs at per-(oulput) channel, and grouping further divides weights into small continuous blocks (e.g., with size 128, 64) to leverage the reduced local ranges, Our statistical analysis (in Section 2) reveals that the effectiveness of this method is limited, as outliers frequently persist within individual groups. Furthermore, the storage require- ments for group-specific quantization parameters, such as scales, zero-points, and look-up tables, make this approach impractical for non-uniform and vector quantization schemes.
More recent approaches include SqueezeL.LM (Kim al., 2023) and SpQR (Dettmers al., 2024), which maintain outliers (and /or important weights) in full precision (FP16). While effective, this method also incurs significant storage overhead due to storing the full preci- sion outliers and their indices. Qull' (Chee et al., 2023) proposed an innovative incoherence processing technique, which employs random rotation matrices on both sides of weight matrices to suppress outliers. However, additional matrix operations introduce notable computational overhead during inference. Moreover, when many layers's weights exhibit independently and identically distributed Gaussian behavior (as observed in Dettmers et al. (2023)), such rotations often yield small improvement, as discussed in Appendix H.2.
In this paper, we introduce ICQuant, novel quantization framework that enables us to deal separately with outliers with minimal storage overhead (approximately 0.3 bits per weight). In particular, ICQuant explicitly partitions the weights into outliers and inliers, keeps track of the positions of outliers using an optimized indexing scheme, and uses different quantization codebook with reduced range for the outlier and inlier weights. A key insight on why our approach works well, is the empirical observation that, within each row of the weight matrices, the positions (indices) of outliers tend to follow a uniform distribution, and thus can be efficiently encoded. We verify this property across multiple models (including the Llama2, 3, 4, and Qwen2.5 families) that have a wide range of scales. We also note that, even if this property were not to hold in other other/future future models, one-time random permutation can enforce uniformity without affecting model output and inference speed, and thus ICQuant would still work well.
Our contributions include:
We introduce ICQuant, novel outlier-aware LLM quantization framework that leverages statistical properties of outliers for effective weight quantization. An attractive feature of ICQuant is that it can be universally applied on top of any quantization scheme.
We conduct extensive experiments showing that ICQuant can significantly im- prove the quantization quality, in 2-4 bits regimes, even of simple scalar quantizers, achieving comparable results to state-of-the-ar schemes that rely on computation- ally intensive vector quantization and expensive fine-tuning procedures.
We analytically derive an upper bound (see Lemma 1) on the bits required by our index coding scheme, which closely aligns with our experimental results, and amounts to R 0.3 bits per weight.
Our work reveals that the outlier weight spatial distribution is well approximated by a uniform distribution, which may be of independent interest.
The rest of the paper is organized as follows. Section 2 presents the statistical properties of outliers that motivate our scheme; Section 3 introduces our design of ICQuant; Section provides our experimental evaluation; Section 5 discusses additional related work, and Section concludes the paper.
2 Statistics of Outliers
Although various techniques exist (as described in Section 1) to reduce the weight quantiza- tion range, the statistical properties of weight outliers, such their distribution and impact
2

---

Published as a conference paper at COLM 2025
down_ proj
10% outliens
up proj
2% ourliess
gale proj
1% outliess
Inlenn Cutliers
p_pro
Frequency
220
V_proj
128-
k_rr
proj
0.0
0.1
0,3
Normalized Weight Range
0.5
044
0.5
0.5
0.7
0.81
0.9
1.0
-8.6
-2.44
6.0 -8.00
Weight Value
adz 004 3.6
(b)
Figure 1: [Llama2-7B] (a) the normalized range taken by different amounts of outliers (starting from 1%), where the values of each type of layer are averaged over the whole model; (b) histogram of row of weights.
mip ap. proj
self sttn pmj
4)-
Freuuncy motilen
Ispoctod
Expecre
90-
Group index
Group undex
Figure 2: [Llama2-7B] The frequency of outliers in each group of 256 consecutive weights.
on quantization efficiency, remain poorly understood. This section examines these statistical characteristics in detail.
In this paper, we define outliers as the small percentage of weights with high magnitude (i.e., those in the tails of the weight distribution). Specifically, for a weight matrix W E R xda, outliers are identified as the top Y weights (e.g., Y 5%) with the highest absolute value in each output channel (i.e., each row w E Rd.).
1). 5% outliers take approximately the 50% range of all weights
Previous studies have focused primarily on a narrow subset of outliers (< 1%), and we extend this analysis to examine the relationship between the proportion of outliers and the quantization range. Figure (a) plots the normalized weight range taken by outliers across layers in Llama2-7B model (more detailed analysis provided in Appendix B), which demonstrates that the top 5% of weight outliers account for approximately 50% of the total value range r (where r max(w) (w) mi)) Figure (b) provides a closer view, showing the distribution of (a row of) weights in Llama2-7B, where 5% of outliers are highlighted.
This observation reveals the significant inefficiency in low-bit quantization. For example, considering the uniform scalar quantization, using n bits yields resolution of 2nr If the
range halved, then n bits would suffice reach an equivalent resolution --ي/ $ Our finding, therefore, indicates that one bit of quantization capacity effectively allocated to represent just 5% of the weights. This creates notable inefficiency, particularly when n is small (e-g., or bits are allowed in total).
q.proj
k. k.proj
v-proj
o_proj
up-proj
gate_proj
down_pre proj
Ilama2 - 7B
3.38%
3.39%
3.14%
62.15%
3.10%
3.12%
2.37%
Llama3 8B
3.01%
3.13%
2.95%
95.25%
3.10%
3.08%
2.11%
Table 1: The rejection rate (i.e., the percentage of weight channels where outlier indices do not follow uniform distribution) according to the Chi-Square test with 0.05 significance.
3

---

Published as a conference paper at COLM 2025
2). Outliers spread out uniformly.
More importantly, we found that the outlier indices are uniformly distributed across each output channel (i.e., each row of the weight matrix). In Figure we show the frequency of outliers within each group of 256 consecutive weight elements. We validated this dis- tribution using the Chi-Square test (Pearson, 1900), with results for Llama2-7B shown in Table 1 (see more results in Appendix C.1). The analysis reveals that around 97% of the weight outliers' positions follow a uniform distribution, except for out projection layers in self-attention blocks. This uniform distribution pattern explains why simply using small group sizes for quantization (explained in Section 1) provides limited benefit in addressing the outlier issue. In the next section, we present an efficient coding scheme for storing out- liers by leveraging this structural characteristic. Notably, we empirically observed that the deviations in out_prooection layers have minimal impact on the overhead of our proposed coding scheme.
Observation.
The uniform distribution of outlier positions likely stems from the trans- former's Gaussian-like initialization (Glorot & Bengio, 2010) and over-parameterization (Kaplan et al., 2020) of large models. Yet we note that, even when this uniform structure does not naturally emerge, we can enforce it by randomly permuting the input channels of each linear layer in advance process that preserves both model output and architecture, as we discuss in Appendix C.2.
3 Methodology
In this section, we present novel quantization framework ICQuant for low-bit (S 4 bits) weight-only quantization of LLMs. Our approach preserves the effective quantization range through strategic separation of outliers and efficient index storage. The proposed method is compatible with various quantization schemes, including vector quantizations.
scalar and
We use Q--) to denote quantization scheme (or quantizer), which is associated with set of quantization parameters (such as scales, zeros, and lookup tables). For simplicity, we refer to all these quantization parameters codebook. We define the quantization size as the average number of bits used to store each quantized weight.
3.1 Outlier-Grouping Strategy
Building upon our first observation from Section 2, we propose separately quantizing the top 5% outliers and inlier values, with each utilizing approximately half of the quantization range for the models we quantize in Section 4. Given an output channel (i.e., row) of weights E Rda, we partition them into two groups {wo, wi), where w. collects outliers and wi collects inlier values. These are quantized into (Qi (Wo,) ), 1,22(w)) using two independent quantizers Q1 and Q2(-) and the same number of bits. Our analysis (in Section 2) shows that 5% of outliers take approximately 50% of the weight range, namely
range(w.) ~ (ange(w:) A = rrange(w).
Figure (a) demonstrates an example using rounding rounding-to-nearest te-reet (RTN) uniform quantiza- tion, where we compare vanilla-RIN against our proposed approach. Notably, since the ranges of both w, and w; are halved, we employ 2-bit quantization in our method versus 3-bit quantization in vanilla-RTN. Figure (c) visualizes the experimental results w from Llama2-7B model, which shows that our INT2 quantization achieves comparable resolution to INT3 vanilla-RTN.
is worth noting that traditional grouping requires di codebooks, where g is the size of g each group. The storage overhead for one additional codebook in our case is negligible, which makes it suitable for non-uniform quantization or vector quantization schemes that typically use large codebooks.
4

---

Published as a conference paper at COLM 2025
Outlier Grouping
Cosdbess
single codebook
Doppemiie Wiighte
800
( vanilaRTTN (INT3: nagel (-4, 3D
2.0
08
-12
4.1
33
07
41
Calaboat
Coolbook
codehooks
-054
-000
0.0
Unqqantized Weights
W (FP16)
(il) ours (INT2: range [-2, I])
(b) Outlier Index Coding
Ouifee
Dequiieed Wicihhe
800
absolute positions:
45
66
47
45
40
50
51
52
3.8
outiier index gaps:
40
-200
inliers
b- bit representitions:
-055
-0.0
-020
0.00
000
1.0
1.0
outliers
Unsquantied Weight
(code magg: (l, 31]. value: 32)
(G)
Figure 3: (a) Comparison of vanilla-RTN and ICQuant in 2-bit uniform quantization. (b) An example of our outlier index coding scheme. (c) Quantization results on a row of weights from the Llama2-7B model.
3.2 Efficient Outlier Index Coding
critical challenge is how efficiently store the outlier position information. Traditional methods such as binary flags direct index storage are prohibitively expensive. Specifically, attaching binary flag to signal outliers requires extra bit per weight. Although storing outlier indexes needs much fewer entries, each entry requires at least 16 bits due to the large dimensionality of LLMs (e.g., din reaching 50K in Llama3-405B model). This approach also results in roughly 0.8 extra bits per weight for 5% outliers for the Llama3-4058 model.
Leveraging the uniform distribution pattern of outlier positions in w, as discussed in Section 2, we propose storing the index gaps between adjacent outliers rather than their absolute indices. This significantly reduces storage requirements. Let y denote the outlier ratio, b denote the number of bits used to represent each index (gap), and B represent the effective average index storage cost per weight (total number of bits to store outlier indices divided by the row size). Given weights E Rda,, let {i) be the absolute indices of the outliers, where p [ydin] Now taking simple example, if y 5% and the distance between each pair of consecutive outliers is up to 32, i.e, i++ - is E f1,...,32) Vk = 1,...,p- then b 5 bits (supporting the range [1, 32)) is sufficient to store each index gap, reducing the storage overhead to merely B = yb = 0.25 extra bits per weight.
Although outliers are uniformly spread out, need to address the practical randomness that the inter-outlier gaps vary. Consider the supporting range [1,2*] of b-bit representations, gaps not larger than 2 can be well covered. The challenge arises when the inter-outlier gaps exceed the 2 unin threshold. To address this issue, we designate the value 2 as flag to indicate large gaps requiring index count accumulation, as illustrated in Figure 3 (b). As result, we have 2 1] left to represent the value of gaps. While this accommodation could potentially introduce additional storage overhead, we provide tight upper bound on this cost in Lemma 1, with proof given in Appendix A.I.
Lemma 1 For a weight w E Rdd with ydin outliers, if the weight values are randomly permuted (with uniform outlier positions) and b bits are used to encode each index gap, the total storage overhead B (defined as the mumber of bit per weight to store the outlier positions) satisfies
E Euniform (B) S yb(11
e7 1)

---

Published as a conference paper at COLM 2025
Outhier
Outlier Ratio 5%
0.500
Thi Uniform Distribution
0.55
Wint Case
Upper Bound (Lemma 1)
1ama22788
(oxtra per weight)
0,473
1lsma2-78
8 (extra bit weight)
0.50
Worst (afeer pemmutaton)
0.425
0.45
0,400
0,575
0.350
0.35
b (bit per index)
b (bit per index)
(b)
Figure 4: (a) ICQuant requires B ze 0.31 bits/ /weghht with b = 6 and = 5%; (b) ICQuant adapts to any outlier distributions with graceful change in storage overhead.
Lemma 1 not only provides a theoretical guarantee on the efficiency of our index coding
scheme, but also guides the optimal choice of with a given outlier ratio y. In Figure (a),
we compare different choices of b and the resulting storage overhead B in three cases: (i)
the upper bound calculated using Lemma 1, (ii) simulation using uniformly distributed
outlier positions (synthetic) and (iii) empirical estimation using Llama2-7B. We notice that the three plots almost coincide, which confirms the uniform distribution observation and the tightness of our upper bound. Note that the convex behavior is due to the trade-off between two factors: (i) the base storage cost associated with each index; and (ii) the extra overhead caused by the index count accumulation of large gaps. Indeed, ICQuant supports a flexible range of outlier ratios that exhibit different trade-offs, with details in Appendix D.
In addition, although our index coding scheme inspired by the uniform distribution property, it remains compatible with any outlier position distribution - the distribution only affects the storage efficiency. The worst case occurs, for example, when outliers cluster at the beginning and end of a weight channel, requiring additional flag values to encode the large gaps in between. Lemma provides a universal upper bound on the total storage overhead to encode outlier positions, regardless of the underlying distribution. The proof, including details on the worst-case scenario, provided in Appendix A.2.
For weight E RR with ydin outliers, and for any distribution of outlier positions, if bits are used to encode each index gap, the total storage overhead B (defined as the number of bit per weight to store the outlier positions) satisfies
Lemma 2
(1
7
den + +7) b.
B <
26
In Figure 4 (b), we compare the worst-case storage cost against the empirical estimation using Llama2-7B. Although the worst-case distribution incurs more storage overhead, the difference is less than 0.1 bit per weight. Furthermore, when outliers are not uniformly distributed, we can perform one-time random permutation as described in Appendix C.2, noting that such permutation does not incur additional computational overhead. As shown in Figure (b), this random permutation nearly eliminates the gap.
4 Experiments
Models and Datasets.
We evaluate our method on the Llama2 and Llama3 family of models (Touvron et al., 2023; Grattafiori et al., 2024), across a wide range of scales from 1B to 70B parameters. Following recent PTQ work (Frantar et al., 2022; Chee et al., 2023; Tseng et al., 2024a; Malinovskii et al., 2024), we report perplexity on WikiText-2 (Merity et al., 2016) and C4 (Raffel et al., 2020) validation sets, well as zero-shot accuracy on WinoGrande (Sakaguchi et al., 2021),

---

Published as a conference paper at COLM 2025
RTN Quantization Brror
Groiping
5-
Perplxiity on WikeTex12
622
1-35 FPI6
ICQuant (ours)
4-hit RTN
Gropping 14
MSB
FP15
5.8
ICU (ours)
Incoherne Processin bit RTN
5.6
3.0
3.2
3.4
3,6
3.8
4,0
15 Transformrr Block Index
Average Bits
(a)
Figure 5: (a) Wikitext2 perplexity (L) and (b) the quantization error (MSE) of 3-bit rounding- to-nearest (RTN) uniform quantization using different outlier suppression techniques, where the MSE is averaged across all linear layers in each transformer block.
PiQA (Bisk et al., 2020), ARC-easy, and ARC-challenge (Clark et al., 2018). The zero-shot accuracies are evaluated using LM Eval Harness v0.3.0 (Gao et al., 2024).
Baseline Methods.
We compare against the state-of-the-art weight-only algorithms, including scalar quantization algorithms, OmniQuant (Shao et al., 2023), SqueezeLLM (Kim et al., 2023) and Qull' (Chee et al., 2023), as well as vecior quantization algorithms QulP# (Tseng al., 2024a), AQLM (Egiazarian et al., 2024), and QTIP (Tseng et al., 2024b). Another recent work, PV-Tuning (Malinovskii et al., 2024), builds on AQLM and focuses on optimizing the fine-tuning strategy. Since we do not perform any fine-tuning, we do not use it comparison in this paper. Note that fine-tume see-tunnng can be done on top of our method and is left for future work.
Choice of Quantizers.
As mentioned before, our proposed quantization framework ICQuant can be applied with any quantization scheme. We here implemented two simple scalar quantization schemes that typically show significant performance degradation in extreme quantization regimes (2-3 bits per weight): (i) RTN: rounding-to-nennt te-arestt uniform quantization and (ii) SK: sensitivity- aware K-means clustering (Kim et al., 2023). We refer to the overall quantization methods as ICQuant and SCQuantSK
4.1 Compare Outlier Suppression Techniques.
To verify the effectiveness of ICQuant (described in Section 3) in outlier suppression, we compare it with the following techniques:
Grouping: Divide the weights into small continuous blocks and quantize each group separately.
Mixed-Precision: Keep outliers in full precision (FP16).
Incoherence Processing (Chee et al., 2023): Apply random orthogonal matrices to both sides of the weight matrices before quantization.
We first evaluate these outlier suppression techniques on 3-bit rounding-to-neent quantization. Figure 5 (a) shows the Wikitext2 perplexity a function of the average number of bits per weight, where we adjust the nyperparameters (e.g group size, outlier ratio) to achieve the different bitrates. Although grouping, mixed-precision, and ICQuant all improve quantization quality the cost of extra storage, ICQuant demonstrates the best trade-off between performance and storage efficiency. Notably, ICQuant surpasses 4-bit RIN performance with less than 3.2 bits per weight. In addition, found that incoherence processing on weights alone offers minimal benefits in extreme quantization scenarios, with the resulting perplexity falling outside the range reported in Figure (a).
(RTN)
7

---

Published as a conference paper at COLM 2025
(ctx. 2048)
(ctx. 4096)
Model
Method
bits
Wiki2|
C41
Wiki2|
C41
FP16
16
5.47
6.97
5.12
6.63
SqueezeLLMM
0.45%
2.2
10.79
Llama2
OmniQuant- g64
2.3
9.62
12.72
QulP
2
ICQuant 5%
2.3
7:21
8.97
6.75
8.84
FP16
16
4.88
6.46
4.57
6.05
SqueezeLLM 0.45%
2.2
7.91
Llama2 13B
OmniQuant g64
2.3
7.56
10.05
QulP
2
13.5
16.2
ICQuant SK 5%
2.3
6.10
7.80
5.74
7.57
FP16
16
3.31
5.52
3.12
4.97
Squeeze 0.45%
2.2
4.99
Llama2 70B
OmniQuant g64
2.3
6.11
7,88
QuIP
2
5.90
8.17
ICQuant SK
2.3
4.26
6.22
4.01
Table 2:
Wikitext2 and C4 perplexity (L) of all Llama2 models quantized to 2-bit regime using scalar quantization algorithms. We use different context lengths (ctx.) match the settings used in their original papers; blank entries indicate results that were not reported by the original authors.
In Figure 5 (b), further compare the quantization error (i.c., [[(()) - w113) of x 3.3-bit RTN quantization using different outlier suppression techniques. We adjust the hyperpa- rameters for all methods (apart from incoherence processing) to share similar extra storage overhead. The results show that ICQuant leads consistently lower error across the whole model. Specifically, ICQuant reduces the quantization error to approximately 1/4 of the baseline, which verifies our intention of halving the quantization range by separating 5% outliers. Similarly, we found that incoherence processing yields little improvement. This is because when the weights already mostly follow normal distribution, the random rotation does not effectively reduce the quantization range, as discussed in Appendix H.2.
Next, we compare (in Table 2) the performance of ICQuantSK to the state-of-the-art scalar quantization algorithms supported by the aforementioned outlier suppression techniques in the 2-bit regime. In particular, ICQuant and SqueezeLLM use the same quantizer (i.e., the sensitivity-aware K-means clustering), but SqueezeLLMM (Kim et al., 2023) keeps outliers in FP16. OmniQuant (Shao et al., 2023) combines grouping with learnable clipping ranges. QulP (Chee et al., 2023) applies incoherence processing and adaptive rounding. Table 2 shows that ICQuant SK significantly outperforms other algorithms in all models, with similar storage overhead.
4.2 Compare with SoTA weight-only PTQ
Finally, conduct comprehensive evaluation of ICQuant on different models and bitrates, comparing our results with the best-known baselines (AQLM, QuIPH, and QTIP) in weight-only PTQ. It is worth noting that all these baseline methods complex vector quantization schemes and fine-tuning during after quantization. Vector quantization typically involves significantly higher complexity in both quantization and inference, while fine-tuning requires substantial time and computational resources. contrast, ICQuant aims to achieve their performance levels using a simple scalar quantizer with much less calibration data and no fine-tuning. The ablation study on the choice of outlier ratio and the improvements on the vanilla SK quantizer can be found in Appendix F. More implementation details are given Appendix E.
Table 3 shows the perplexity and zero-shot accuracy of quantized Llama3-70B models (results of other Llama3 models are provided in Appendix G.). Since baseline models only report results with fine-tuning, known to be more challenging to quantize than Llama2 models (Huang et al., 2024), with state-of-the-art methods (QulP# and QTIP) showing significant performance degradation
compare against their fine-tuned versions. Llama3-70B is
8

---

Published as a conference paper at COLM 2025
in downstream tasks, even after fine-tuning. Remarkably, ICQuant outperforms these methods by large margin. For example, with 0.31 bits extra overhead, ICQuant improves QTI''s 2-bit accuracy on ARC-challenge from 28% to 50.7% and on ARC-easy from 35.3% to 81.7%, even surpassing QTIP's 3-bit performance. Interestingly, while Qull* and QTIP achieve low perplexity, they perform poorly on zero-shot reasoning tasks, likely due to overfitting during fine-tuning- Moreover, ICQuant even achieves comparable zero-shot accuracies to PV-tuning, which fine-tunes vector quantization with an improved but more expensive fine-tuning strategy with complexity comparable to QAT.
Table 4 compares the perplexity of quantized Llama2 families. Overall, ICQuant achieves the best performance without fine-tuning. Compared to fine-tuned baselines, ICQuant consistently outperforms AQLM, reaching results comparable to Qull# and QTIP. In ad- dition, we observe that separating more outliers (e.g., 8.25%) improves the performance by further shrinking the range of inliers while quantizing more coarsely the outliers. This improvement occurs not only because of the larger proportion of inliers, but also because the inliers tend to be more important (see Appendix H.1). We report the evaluation on zero-shot reasoning tasks in Appendix G.
Llama3 77BB
Method
bits
Wiki2,
C4
ArcC1
ArcEf
PIQA
Wino
BP16
16
2.59
5.78
60.2
86.9
82.3
80.6
QulP#
4.0
[2.99]
[5.96]
[35.0]
[67.3]
[71.9]
[76.7]
QTIP
4.0
[2.75]
[5.83]
[56.1]
[83.9]
[81.3]
[80.6]
ICQuant 55%%%
4.3
2.72
5.83
59.6
87.2
82.6
80.4
QulP#
3.0
[3.59]
[6.18]
[31.1]
[36.6]
[58.8
[76.4]
QTIP
3.0
[3.18]
[5.98]
[48.6]
[77
[77.8]
[79.7]
ICQuantSKK -5%
3.3
3.24
6.03
56.6
84.9
81.9
80.0
QulP#
2.0
[5.77]
[7.46]
[18.3]
[32.2]
[54.7]
[68.9]
QTTP
2.0
[4.97]
[6.80]
[28.0]
[35.2]
[57.1]
[72.6]
ICQuant5K.g.222
8.225%
5.83
8,22
52.0
82.3
79.7
74.0
ICQuanttK.5%%
2.3
5.65
7.64
50.7
81.7
79.4
75.5
PV-tuning
2.1
[4.55]
[6.54]
[50.8]
[80.2]
[79.2]
[78.1]
Table 3: Perplexity (L) and zero-shot accuracy (1) of Llama3-70B models (context length
8192), quantized to 2-4 bit regime, using vector quantization algorithms and ICQuant (scalar quantization), where the values after fine-tuning are wrapped in [-]. We highlight the
cases where ICQuantSK outperforms all fine-tuned baselines.
Llama2
7B
Llama2
13B
Llama2
70B
Method
bits
Wiki2
C41
Wiki2.
C41
Wiki2
C41
FP16
16
5.12
6.63
4.57
6.05
3.12
4.97
AQLM
4.0
-
[5.21]
[6.75]
[4.65]
- [6.14]
- [3.19]
[5.03]
QulP#
4.0
5.22 [5.19]
6.79 [6.75]
4.65 [4.63]
6.15 [6.13]
3.18 [3.18]
5.02 [5.02]
QTIP
4.0
5.17 [5.17]
6.71 [6.69]
4.62 (4.61]
6.10 [6.09]
3.16 [3.16]
5.00 (5.00)
ICQuant
4.3
5.17
6.70
4.61
6.09
3.16
5.00
AQLM
3.0
- [5.46]
[7.08]
[4.82]
[6.37]
[3.36]
[5.17]
QulP#
3.0
5.60 [5.41]
7.34 [7.04]
4.90 [4.78]
6.50 [6.35]
3.41 [3.35]
5.20 [5.15]
QTIP
3.0
5.38 [5.28]
6.99 [6.87]
4.74 [4.69]
6.28 [6.22]
3.27 [3.26]
5.09 (5.08]
ICQuant SK_Ksss5p -5%
3.3
5.35
6.95
4.75
6.26
3.28
5.10
AQLM
2.0
[6.59]
[8.54]]
[5.60]
-[7.49]
[3.94]
[5.72]
QulP#
2.0
8.22 [6.19]
11.0 [8.16]
6.60 [5.35]
8.07 [7.20]
4.16 [3.91]
6.01 [5.71]
QTIP
2.0
6.82 [5.86]
8.96
[7.73]
5.52 [5.11]
7.39 [6.85]
3.87 [3.70]
5.70 [5.48]
ICQuan(SK-8.255
-8.25%
2.4
6.35
8.25
5.54
7.25
3.86
5.61
ICQuant 5K_5%
2.3
6.75
8.84
5.74
7.57
4.01
5.78
Table Wikitext2 and C4 perplexity (4) of all Llama2 models (context length = 4096)
quantized to 2-4 bit regime, using vector quantization algorithms and ICQuantSK (scalar quantization), where the perplexity values after fine-tuning are wrapped in [-]. We highlight in each case the best result without fine-tuning.

---

Published as a conference paper at COLM 2025
4.3 Inference Efficiency
Our outlier index decoding and weight dequantization procedure can be efficiently fused with matrix multiplication in a single kernel, thereby minimizing costly memory transac- tions. Alternatively, to fully accelerate the inference, outlier indices can be pre-decoded into a compact bitmask before the layer is executed. This offers flexibility depending on deployment constraints and latency requirements. To demonstrate the practical efficiency of ICQuant, we implemented an example 2-bit matrix-vector multiplication kernel, building on the open-sourced SqueezeLLM kernel. Our implementation supports on-the-fly decoding and de-quantization. In Table 5, we benchmark the matrix-vector multiplication subroutine on Llama2-7B linear layer, using Nvidia RTX 4090 GPU. The result shows that ICQuant can deliver faster inference than all baseline methods.
Although ICQuant requires additional decoding steps during inference, the overall latency remains lower than the baseline methods at comparable accuracy lev-
Method
CUDA Latency
FP16
96.32 us
els. Note state-of-thee overhead: structured vector quantization schemes like AQLM require loading from a codebook that is too large to fit in cache, while QTIP and QulP# require additional matrix multiplications and codebook gen- eration. In addition, our example kernel serves as proof-of-concept to demonstrate feasibility rather than peak performance. For instance, our current imple- mentation uses FP32 arithmetic, rather than FP16 (as used in the baselines), which does not fully leverage the tensor core acceleration available on modern GPUs.
baselines have non-trivial
AQLM
86.14
QuIP#
31.28 us
QTIP
32.92 us
SCQuantSK-5%
23.29 us
Table 5: Latency of gate.proj layer in 2-bit Llama2-7B model.
5 Other Related Work
Model quantization has two main categories: post-training quantization (PTQ) and quantization-awan training (QAT) (Ma et al., 2024), depending on whether retraining is needed. Quantization can target weight-only, weight-activation quantization (Xiao et al., 2023; Liu et al., 2024a), and KV-cache (Liu et al., 2024b; Hooper et al., 2024). In this paper, we focus on weight-only PTQ. Initial research efforts focused on improving uniform quanti- zation through innovative approaches. Notable contributions include error compensation introduced in GPTQ (Frantar et al., 2022) and equivalent scale transformations proposed in AWQ (Lin et al., 2024). Subsequently, Dettmers et al. (2023) and Kim et al. (2023) ad- vanced the field using non-uniform quantization techniques, taking into account the weight sensitivity and statistical properties. Contemporary state-of-the-art methods Quip# (Tseng et al., 2024a), AQLM (Egiazarian al., 2024), QTIP (Tseng et al., 2024b) have achieved re- markable results with 2-bit models using vector quantization and fine-tuning, Furthermore, PV-Tuning (Malinovskii et al., 2024) explored new quantization-awar fine-tuning strategy for LLMSs. Nevertheless, it is important to note that applying fine-tuning risks overfitting the calibration set and diminishes the efficiency advantage of PTQ over QAT.
6 Conclusions
In this paper, we present ICQuant, a novel method to decrease quantization range by separately quantizing outliers and efficiently encoding their indices a method appli- cable to any quantization scheme. We demonstrate its effectiveness in improving weight quantization quality in extreme compression regimes (S 4 bits), achieving state-of-the- art performance using a simple scalar quantizer and fixed outlier ratio. Looking ahead, promising directions include jointly optimizing the outlier ratio and storage overhead by leveraging layer-specific statistics, combining ICQuant with advanced quantization schemes and fine-tuning procedures, and extending the technique to quantize other components such as activations and KV-caches. Overall, ICQuant provides flexible and highly effective mechanism for improving low-bit weight quantization, paving the way for more efficient deployment of LLLSs under tight resource constraints.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This work was supported in part by NSF grants #2221871, #2139304, #2146838, and the Army Research Laboratory grant under Cooperative Agreement W911NF-17-2-0196.
References
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning 2018
challenge. Xiviv preprint ar arX:::
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systemss, 35:30318-303333 30332,, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088- 10115, 2023.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse- quantized representation for near-lossless 11m weight compression. In ICLR, 2024. URL
https://openreviiet net//Foruumlddquuzsshhuuu
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit ilms via self-distillation. ar preprint 1rXi::2402.10631, 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. Xiv preprint ar xrXiv:2401.06118 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate
post-training quantization for generative pre-trained transformers. ar arXiv preprint arXiv:2210.17323, 2022.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. arg/reccords/12ee5 2688602..
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. Xrxiv preprint ar Xix:::2407 Xix::2407.22783, 21783, 2024.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances Neural Information Processing Systems, 37:1270-1309 2024.
11

---

Published as a conference paper at COLM 2025
Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study, Xiv e-prints, pp. arXiv-2404, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural 2020.
language models. Xiv preprint ar arXiv:2001.0888
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arviv preprint ar Xixx:2306.07629, 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. arXiv preprini ar arXiv:2405.16400 2024a.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-fii frene asymmetric 2bit quantization for kv cache. Xrv preprint ar X0X2:22402.02750, 2024b.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are 1.58 bits. arXiv preprint ar XfXi:2402.17764, 1, 2024.
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme IIm compression. Advances in Neural Information Processing Systems, 37:5074- 5121, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
AI Meta. The Ilama 4 herd: The beginning of new era of natively multimodal ai innovation. https://ai. meta.
-ammmtiiodaa--iillllieennn checked on, 4(7):2025, 2025.
Karl Pearson. X. on the criterion that a given system of deviations from the probable in the e supposed have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(502) 1900.
case of correlated system of variables is such that it can be reasonably
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 211140):1-67,2000 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXio:2308.12 131337, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. Xiu preprint 40Xi::2402.17762 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xrv preprint arXi::230 23t.02330000888,,
09288, 2023.
12

---

Published as a conference paper at COLM 2025
Albert Iseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint ar 2et.:242220449666 2024a.
Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37: 59597-59620, 2024b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint axXiv:2412.15115, 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024.
13

---

Published as a conference paper at COLM 2025
Appendix
In this appendix, we provide further details as follows:
A. Proofs of Lemmas
B. Range of Outliers
C. Uniform Distribution of Outlier Positions
D. Index Storage Cost Analysis
Implementation Details
F. Ablation Study
G. Other Experiment Results
H. Other Observations
A Proofs of Lemmas
A.1 Proof of Lemma
Lemma 1 For weight w E Rd with ydin outliers, if the weight values are randomiy permuted (with uniform outlier positions) and b bits are used to encode each index gap, the total storage overhead B (defined as the number of bit per weight to store the outlier positions) satisfies
Euniform (B) S 7b(1+ +
Proof: Recall that (1) are the absolute indices of outliers, where in E [1, and = [ydm]] We define the index gaps Xg, x1, Xp as follows L,.
Xg = i Xp = diz - ip 1, and x = i
Following the index coding scheme described in Section 3.2, we need to store No. Xp using b-bit representation and there are two possible cases:
(i) x; < 2h 1, we directly store X;.
(ii) Otherwise, we store {2 22,,2,,.. (x; - 1) mod 11) + 1}, where 2 is a flag value that means we accumulate (2b 1) index count during decoding, and the number of such flags 2 equals 1
Let & be the total number of empty intervals (i.e., the number of times the flag value 2 is stored). Then we can write the expected total storage overhead as
bp + E(E).
BEB)
(1)
din din
It remains to bound E(E). Note that Xo, X1,..., Xp are positive and satisfy Σk= Xx din + 1. Moreover, by symmetry, any permutation of x (xg, xI, defines valid distribution of the outlier positions with the same probability as &. Hence, ~ xj, Vi # (i.e., all x;
share the same marginal distribution). Let m (2* 1). The expected number of empty intervals & can therefore be written as
X1 1 BB
E(E)=
m
Xx
(2)
<
Σ
k=0
m
X0
pE[
m
14

---

Published as a conference paper at COLM 2025
where [E( 11
P(
2 i).
We observe that
x0
x0 S jm.
(3)
Therefore, we can write
[[
X0
1)
Σ 121
Px0
jm)
din jm
Σ
121
p
Claim S
jm
11
din
(4)
(i))
Σ
jmmy
11
e
where Claim is proved after this lemma; (i) is due to the inequality that - x S e any I; and (ii) follows p [ydnn]]
for
Finally, combining equation 1, equation 2 and equation 4, we have
S bp
bp 1
1
IE (B)B)
e S 7b + 7bb 2
(5)
din
dm
1
where the last inequality follows p [dd and 2 11
1.
Claim 1 For a, p,
Z with < N and p N, we have
p
N p
(6)
Proof:
(N a)!(N p)!
(N a - P!!N!
p-1 N a-i
N
(7)
N-i
N
1-0
(1
P
15

---

Published as a conference paper at COLM 2025
A.2 Proof of Lemma 2
For a weight w E RR with ydin oufliers, and for any distribution of outlier positions, if b bits used to encode each index gap, the total storage overhead B (defined the number of bit per weight to store the outlier positions) satisfies
Lemma 2
7
din + 37) b.
B s
2
Proof: Let {1) denote the absolute indices of the outliers, with i E [1, dmn and = [ydm] We define in = 0. When applying the index coding scheme proposed in Section 3.2, we store each index gap ix i-1 as & (& 2 0) number of flag values (i.e., 2 and an overflow part Vt (1 < yt S 2 1), such that is k-1 (2 - 1) + +-y We need b bits to store each yx or flag value, and the total storage overhead can be written as
b din
<tpp)
(8)
Now, it remains to upper bound the total number of flag values required Σ Ex. Since ip S din and io = 0, we have
(2 1) yk)) = - i0 S dins
(9)
which implies
din - E.
1K
dip p 2 -
& <
(10)
2 -
where the second inequality is due to y/ 2 1. Combining (8) and (10), we can then upper bound the total storage overhead as follows:
b
d
B
1 + +p)
din
2
b
d
|ydin]
+ Lyddin]
din
2
(11)
b
dj
Ydji +1
+ydin
din
(1
T + d
+7) b.
b
-
Worst Case. The upper bound (10) is obtained when i, = dis and yı 1, 1,..., p. One example of such worst cases is when outliers are concentrated at the end (and the beginning) of a weight channel (row). Specifically, for a weight vector w, the first (0 < < p) elements and the last p elements are outliers. In this case, the absolute outlier indices are
(0)_1 {1,2....,j,d,, (p (1, + Ai },
with index gaps of between consecutive indices, except for single large gap in the middle.
16

---

Published as a conference paper at COLM 2025
B Range of Outliers
Similar to Figure 1 (a), we here provide plots of the normalized weight range taken by the outliers for other models in Llama2 and Llama3 Families in Figure
Llama-2-138
down_proj
10% outliers
down_pro
10% outliers
up_proj
5% outliers
up_prd)
outliers
outliers 19 outtlers
1% outiers
o_pro
0_proj
Y_proj
v.pro
Krproj
Prproj
prp
04
Weight Range
Weight Range
Llame-3-88
Llama-3-708
down pro)
10% outliers
down pro)
10% outliers outtlers
up_proj
outtiers
up_proj
outliers
outliers
gate_proj
1% outliers
gate_proj
176 outliers
prproj
prpro
pror
pror
pror
pror
6. 02 0.3
0.7 0.8 Normalized Weight Range
0.9 0.2 9. 04
0. 1.0
Weight Range
Llama-3. 2-38-Instruct
Llama-3.2-1B-mm --1-strrtt
down_pro
10% outliers
down_pro
10% outtiess
up_pro]
5%
2% outfiers
2%, outllers
gate proj
outliers
pro)
1% outliers
0_pro
0_pro
v.pro
kuproj
pro
Quproj
pro
60 0.1 0.2 0'3 04 05 3.7 0.8 0.9 1.0 Normalized Weight Range
05 53 02 03 24 05 06 DJ 0.0 0.9
Normalized Weight Range
Figure 6: The normalized range taken by different amounts of outliers (starting from 1%), where the values of each type of layer are averaged over the whole model.
C Uniform Distribution of Outlier Positions
C.1 Chi-square Test
We Chi-square goodness-of-fit test (Pearson, 1900) at significance level of 0.05 to examine whether weight outliers are uniformly spread out across each output channel. Specifically, for each output channel (i.e., each row of the weight matrix), divide the weight vector into non-overlapping groups of 256 elements. Assuming an outlier ratio of 6.25%, each group expected contain 16 outliers under the null hypothesis of uniform distribution. The Chi-square test is applied to each channel to test this hypothesis, and we report the rejection rate-the proportion of channels for which the null hypothesis is rejected-in Table 6. We observed consistent results across wide range of popular LLMs, including Llama2 (Touvron et al., 2023), Llama3, Llama3.2- (Yang et al., 2024). This selection spans models, as well as dense and MoE architectures. Our results
(Grattafiori al., 2024),
Llama4 Scout (Meta, 2025), and Qwen2.5-Inntree both base and instruction-tumm show that the positions of most weight outliers follow uniform distribution, except for the output projection layers in self-attention blocks. In addition, we observe that within each model family, larger models show lower rejection rates.
17

---

Published as a conference paper at COLM 2025
Model
9.proj
k.proj
v.proj
0.proj
up-proj
gate_proj
down_pro
7B
3.38%
3.39%
3.14%
62%15%
3.10%
3.12%
2.37%
Llama
13B
2.95%
2.98%
2.91%
59.37%
2.96%
2.96%
2.16%
70B
2.73%
2.80%
2.51%
94.56%
2.60%
2.60%
1.51%
3.01%
3.13%
2.95%
95. 35.25%
3.10%
3.08%
2.11%
Ilama 3
70B
2.54%
2.57%
2.57%
70.64%
2.39%
2.61%
1.64%
1B
3.49%
3.94%
3.66%
82.33%
3.45%
3.45%
2.65%
Llama 3.2
3.20%
3.18%
3.27%
85.02%
3.26%
3.27%
2.55%
3.19%
3.13%
3.24%
94.70%
3.12%
1.82%
Qwen 2.5
32B
2.77%
2.73%
2.98%
90.37%
2.78%
2.80%
1.56%
Llama 4 Scout
17Bx16E
3.09%
3.06%
3.13%
97.37%
2.99%
2.99%
2.60%
Table 6: Chi-Square test rejection rates for different LLMs.
C.2 Example of Random Permutation
As mentioned in Section 2, when outlier positions do not naturally follow uniform distribu- tion, we can enforce uniformity through a one-time random permutation before quantization. Given weight matrix W, we aim to distribute outliers across each output channel (i.e., row of the matrix). To achieve this, we randomly shuffle the columns plication with random permutation matrix P, producing WP. We then reorder the input (activation) X by left multiplication with pT, yielding p X. Since the permutation matrix orthogonal, the linear transformation output remains unchanged: WPP X WX. Note that if W is not the first linear transformation, the reorder of the input X can be combined with the previous linear layer by permuting its output channels (rows of the weight matrix).
by right multi-
Figure 7 illustrates this multilayer permutation process using an MLP block from Llama's architecture, where P1 and P3 are random permutation matrices. Although the permutation for each layer can potentially be different, we use the same P1 throughout to propagate the input order, accommodating the residual structure of the transformer. Note that these be
permutations preserve the model architecture because the permutation matrices absorbed into W we only need to store P WP2 (or P3 WP)). For the last linear layer in the model, we maintain the original order of its output channels, ensuring the model output remains unchanged.
Before Random Permutation
W
Wap
Wane
W hood
X
Swish
W. guiz
After Random Permutation
P. W.PPi
Pa W W
Warnn
W
PT
Wond
Y
x
Swish
P W
permotation masiies
Figure 7: Example of random permutations for an MLP block in Llama-like transformers,
18

---

Published as a conference paper at COLM 2025
D
Index Storage Cost Analysis
Figure 8 plots the storage overhead of ICQuant under different outlier ratios.
Outlier Ratio 3%
Outlier Ratio 8.25%
True Uniform Distribution
0.65
True Uniform Distribution
0.45
Upper Bound (Lemma 1)
Upper Bound (Lemmr 1)
B (extra bit per weiglt)
Liara2-7B
B (extra per weight)
Llamu2-78
0.40
0.60
0.35
0.55
0.30
0.25
0.45
0.20
3
b (bit per index)
(bit per index)
Figure& 8: Index storage cost estimation of ICQuant.
E Implementation Details
E.1 Quantizers
In the following we describe the implementation details of the two quantization schemes used in Section 4.
ICQuant We apply standard rounding toahiigg-t--earest (RTN) uniform quantization on the inlier weights that are centered around zero. Since the positive and negative outliers separated on the two tails of the distribution, use 1 bit to denote the sign and quantize them separately using и 1 bits RTN.
ICQuantSK, Following SqueezeLLM (Kim et al., 2023), we apply weighted K-means clustering to minimize the proxy objective,
Q(w)* = arg www.iinww-... Wo) ((W-Wo)
where the Hessian matrix H is approximated using the Fisher information matrix that is generated with 128 randomly selected sequences from the C4 dataset (Raffel et al., 2020). Each sequence has length of 2048 for Llama2 and 8192 for Llama3. We apply such k-means clustering twice- once on the inlier weights and once on the outliers. For non-uniform quantization, we can quantize the negative and positive outliers together using и bits.
E.2 Evaluation
We follow the evaluation setup in recent PTQ literature (Frantar et al., 2022; Lin al., 2024; Tseng et al., 2024b), and compare our results directly with those reported in the baseline methods' papers.
In particular, we use the same test split as GPTQ (Frantar et al., 2022) to calculate the perplexity on Wikitext2 and C4. For zero-shot evaluations, we use LM Eval v0.3.0 for all Llama2 models, matching the baseline methods. Note that the latest version of the library (v0.4.x) produces different results (typically > 2% difference), while even within the same version there is minor variance (< 0.5%).
19

---

Published as a conference paper at COLM 2025
E.3 Inference Kernel
At runtime, we first decode the outlier indices and dequantize them to full precision before multiplying them with the unquantized activation vectors. To minimize global memory access and kernel launch overhead, we implement decoding and dequantization on the fly and fuse them with matrix multiplication. However, since the decoding step is inherently sequential and difficult to parallelize on GPUs, we encode outlier gaps within smaller blocks (e.g., 256 columns). This approach enables parallel decoding of each block by single thread. we can store this information as a small While the number of outliers per block may m of outlier positions, the variation of this auxiliary value. Due to the approximate number remains minimal, resulting in negligible additional storage overhead.
To isolate the impact of quantization algorithms from unrelated system-level factors, we benchmark GPU latency using a standard linear layer (mlp gate-proj) from the LLaMA2-7B model. All latency measurements are conducted using the Py PyTorch CUDA profiler.
F
Ablation Study
F1
Outlier Ratio
In our experiments, we initiate with a 5% outlier ratio based on our empirical observation (Figures 1 and 6) that approximately 5% of outliers account for half of the total range. This separation effectively splits the weight range in half, ensuring consistent quantization resolution across all weights (both inliers and outliers).
To validate the impact of different outlier ratios, we evaluate ICQuant with 3-bit RIN uniform quantization Llama2-7B, with outlier ratios ranging from 1% to 10%. Table 7 shows that the perplexity score decreases (indicating improved performance) as the outlier ratio increases, which aligns with our expectation since smaller quantization range for inliers benefits the substantial majority (95%) of weights. However, we observed that the performance gain from separating additional outliers saturates beyond 5%. This is consistent with the Gaussian-like distribution of model weights, where range variations become negligible beyond this point. We therefore maintain 5% outlier ratio throughout most experiments to balance between storage overhead and quantization quality.
Outlier Ratio
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
Wiki2 Perplexity
6.27
5.77
5.67
5.64
5.59
5.57
5.57
5.57
5.56
5.55
Table 7: Wikitext2 perplexity (1) of quantized Llama2-7B using ICQuant RIN with different outlier ratios.
F2
Compare with Vanilla SK Quantizer
In main experiments, we apply ICQuant to the sensitivity-aware K-means (SK) quantizer proposed in the Squeezel.LM paper (Kim et al., 2023). By separating outliers and inliers during quantization, ICQuant effectively gains an additional bit of quantization capacity. Using only R2 bits, we expect ICQuant to achieve performance comparable to vanilla SK's 3-bit result, which outperforms all 2-bit state-of-the-art baselines. To demonstrate this, we present WikiText2 perplexity scores for Llama2-7B quantized at 2-3 bits in Table 8. ICQuant substantially improves upon 2-bit vanilla SK, approaching the 3-bit performance level. The remaining gap between 2-bit ICQuant and 3-bit vanilla SK exists because the non-uniform quantization biases toward the more concentrated inlier region. This finding also confirms that increasing the outlier proportion (e.g., to 8.25%) can further enhance model performance.
20

---

Published as a conference paper at COLM 2025
Llama2 7B
Method
bits
Wiki24
FP16
16
5.12
Vanilla SK
3.0
5.71
QulP#
2.0
8.22
QTIP
2.0
6.82
ICQuant -8.25%
2.4
6.35
(CQuantSK-5%
2.3
6.75
Vanilla SK
2.0
38.05
Table 8: Wikitext2 perplexity (4) of quantized Llama2-7B using ICQuant and Vanilla SK.
G
Other Experiment Results
Across wide range of LLMs, from 1B to 70B, ICQuantSK (scalar quantization) matches and sometimes exceeds the state-of-the-art (vector quantization) performance.
Method
bits
ArcC1
ArcEt
Llama2 7B
PIQA
Wine
ArcCr
ArcF
Llama2 13B
PIQA1
Winot
ArrCt
ArrEt
Llama2 70B
PIQAT
Wino?
69.3
AQLM
4.0
[41.0]
[70.2]
[78.2]
[67.3]
[44.8]
[73.3]
[78.4]
[69.9]
[90.]]
[77.3]
[81.]]
[76.5]
QuilP
4.0
[40.4]
[68.]]
[78.5]
[67.4]
[43.6]
[71.3]
[78.7]
[69.6]
[50.]
[77.7]
[81.4]
[77.3]
QTIP
[68.9]
[78.4]
[44.8]
[73.6]
[78.9]
[69.9]
[50.0]
[77.8]
[81.3]
[76.9]
ICQuant
5%%
4.3
40.5
69.0
78.1
67.4
45.6
72.5
78.8
69.1
50.6
77.9
81.1
76.6
AQLM
[38.4]
[68.1]
[76.9]
[66.9]
[42.6]
[70.9]
[77.3]
[68.4]
[5.00
[77.6]
[81.3]
[77.2]
QulPa
[68.4]
[22.3]
[66.5]
[44.0]
[22.5]
[78.]]
[509]
[81.4]
[76.1]
QTIP
3.0
[38.9]
[68.1]
[78.1]
[66.9]
[44.0]
[72.8]
[78.0]
[69.5]
[50.0]
[78.2]
[80.6]
[77.0]
ICQuanfSKggg
39.0
77.7
49.7
AQLM
[32.8]
[63.7]
[74.8]
[65.7]
[38.8]
[69.3]
175.9]
[68.8]
[4.99
[77.7]
[80.4]
[75.9]
QulPa
2.0
[35.2]
[65.3]
[25.4]
[64.9]
[39.6]
[69.0]
77.33
[67.4]
[47.6]
[77.1
[79.5]
[74.6]
QTIP
[35.7]
[65.6]
[75.9]
[64.7]
[41.4]
[70.8]
3]
[67.6]
[8.00
[76.3]
[80.2]
[75.1]
ICQuant www.kK..5 8.23%
2,4
35.9
63.6
76.1
65.6
39.8
68.4
77.0
64.2
48.6
75.2
81.3
74.7
ICQu
Table 9: Zero-shot accuracy (f) of all Llama2 models using vector quantization algorithms and ICQuantSK (scalar quantization), where the accuracy values after fine-tuning are wrapped in [-]. We highlight the cases where ICQuant outperforms all fine-tuned baselines.
Llama3 -
8B ArcEf
Method
bits
Wiki2,
C41
ArcC1
PiQAt
Wino
BP16
16
5.54
7.10
50.4
80.1
79.7
72.5
QulP#
4.0
[5.81]
[7.32]
[50.2]
[79.7]
[79.7]
[73.1]
QTIP
4.0
[5.67]
[7.20]
[50.2]
[79.6]
[79.4]
[73.4]
ICQuant
5%
4.3
5.69
7.23
49.7
79.8
79.7
73.0
QulP#
3.0
[6.27]
[7.71]
[46.4]
[77.4]
[77.9]
[72.9]
QTIP
3.0
[6.01]
[7.48]
[49.2]
[79.3]
[79.2]
[74.5]
ICQuantSK.ge
3.3
6.22
78.1
78.6
72.9
QulP#
2.0
[7.84]
[9.04]
[39.2]
[72.9]
[75.6]
[68.2]
QTIP
2.0
[7.33]
[8.62]
[44.2]
[75.2]
[77.6]
[70.7]
ICQuant SK 8.25%
2.4
9.67
10.97
38.0
71.6
76.6
67.9
ICQuant 555%
2.3
11.53
12.78
35.8
70.7
74.9
66.6
Table 10: Perplexity (L) and zero-shot accuracy (t) of Llama3-8B models (context length 8192), quantized to 2-4 bit regime, using vector quantization algorithms and ICQuant (scalar quantization), where the values after fine-tuning are wrapped in [.]. We highlight the cases where ICQuant outperforms all fine-tuned baselines.
21

---

Published as a conference paper at COLM 2025
Model
Method
bits
Wiki2]
ArcC1
ArcEf
HellaSwagt
PIQAT
WinoGrandef
BPl6
16
11.57
35.9
68.5
45.2
74.3
59.7
Llama3.2- 1B
QTIP
4.0
[11.93]
[34.8]
[68.4]
[44.5]
[73.3]
ICQuanf 55%
4.3
11.96
35.7
68.1
44.8
73.9
59.7
BP16
16
9.58
43.3
74.3
52.2
75.7
67.3
-lama3.2-3B
QTIP
4.0
[9.77]
[43.5]
[74.3]
[51.9]
[75.1]
ICQuant SK 5%
42.1
74.1
52.0
75.1
66.8
Table 11: Perplexity (1) and zero-shot accuracy (t) of Llama3.2 instruction-tuned models (context length 8192), quantized to 4-bit regime, using QTII' (vector quantization) and ICQuant (scalar quantization), where the values after fine-tuning are wrapped in [-]. For these smaller models, 4-bit ICQuant still achieves performance nearly identical to FP16, comparable to the state-of-the-art
H Other Observations
H.1 Outliers are less important
Although large-magnitude outliers in activations (i.e., input features) consistently sensi- tive to quantization (Dettmers et al., 2022; Lin et al., 2024; Sun et al., 2024), weight outliers appear to behave differently. Interestingly, previous research has shown improvements through both clipping weight outliers (Shao et al., 2023) and by preserving them in full precision (Kim et al., 2023). To examine the importance of weight outliers, we measure their sensitivity scores using Fisher information the sensitivity scores against the weight values of two representative weight channels in Llama2-7B. The result shows that weights in the distribution's 's tail regions have significantly lower sensitivity than those the central region. This observation highlights the inefficiency of mix-precision approaches that preserve outliers in full precision. More importantly, it explains our experimental observations that separating more outliers which enables more precise quantization of the critical inlier values improves the quantization performance.
proposed in Kim et al. (2023). Figure 9 plots
Inliers
Inliers 5% Ourliers
Sonsitivity Score
5% Outliers
Sansidvity Score
2.5
2.5
2.0
2.0
1.0
1.0
0.5
0.5
-0.06
-004
-002
0.00
0.02
0.06
0.08
-0005
-0.56
-002
0.00
0.92
604
0.06
Weight Value
Weight Value
Figure 9: [Llama2-7B] Examples of weight values v.s. sensitivity scores.
H.2 Examples of Incoherence Processing
To investigate the effect of incoherence processing (Chee et al., 2023) on reducing the quantization range, we visualized the weight distributions before and after applying random rotation, with examples provided in Figure 10 and Figure 11. We observed that incoherence processing can significantly reduce the weight range when there exist extremely large outliers, resulting in Gaussian- becomes negligible when the weight distribution already exhibits Gaussian-like behavior. The first case appears mostly in the initial layers of the model, which explains the results in Figure 5 (b) that the incoherence processing yields a substantial reduction in quantization MSE of the first transformer block, but shows small returns in subsequent blocks.
distributions. However, the benefit of such rotation
22

---

Published as a conference paper at COLM 2025
Before (W): mwxlwwl 0.1089
After IP [mWinmaxi 0.0520
Frequency
200
-100 skm casa sizewone dds niso sits Weight Value
Weight Value
(a) layers.O self, attona__rooj
Before IP (W) maxlms 0.0723
After (UWV)): 0.0903
Frequency
Frequency
-508
Weight Value
Weight
(b) mrrss.....effaaaee 1.0.seefaatttn_______ __ttn.a_ttaroo
Figure 10: Examples of weight distribution in query projection layers before/aftt affte incoher- ence processing.
Before maxiww 0.0633
After IP
0.0605
Frequeecy
556
-204 -224 -20 cae 623
4.00
-204
2200
con
Weight Value
Weight Value
(a) layers.0 self, attn. www.ddwwoooooooooooopo
Before IP max|m] = 0.0830
After (UWVT): - 0.0794
Frequency
Frequency
0.0
321
ods
Weight
Weight
(b) layers. 10.se selef
Figure 11: Examples of weight distribution in down layers before/ after incoher- ence processing.
23


Reviews:
[{"review_id": "aMpGk5jyWf", "reviewer": "Reviewer_BVJU", "summary": "The paper identifies that only 5% of outliers account for 50% of the magnitude and proposes a more efficient parameter storage scheme, achieving 1-bit information storage at a cost of only 0.3 bits.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The paper presents an effective method to reduce the cost of model storage and loading, and it can be orthogonally applied to various other quantization techniques.\n\n2.  Experiments demonstrate the effectiveness of the proposed method compared to approaches with similar cost.", "reasons_to_reject": "1.  The paper suffers from poor writing, characterized by undefined symbols and unclear expressions. For instance, line 94 mentions \"weights with high magnitude (i.e., those in the tails of the weight distribution).\" It is unclear whether this refers to weights with the largest absolute values, as Chapter 3 suggests. Besides, line 101 states, \"the top 5% of weight outliers account for approximately 50% of the total value range r (where r = max(w) − min(w)).\" Here, 'w' is not defined. It should likely represent all parameters within a specific parameter matrix or the entire model's weights. If the weight distribution follows Figure 1(b), the range 'r' of outliers should be equal to the range of all weights. It is questionable whether \"outliers\" should be \"inliers\". Furthermore, the \"range\" in the last line of 141 is also undefined, making it difficult to assess the correctness of the formula. A significant portion of the paper's novelty lies in this finding, but the ambiguous descriptions obscure its clarity.\n\n2.  The innovation of the proposed method appears limited. It's uncertain if my understanding is correct, but could the paper's finding be summarized as the observation that a specific bit after quantization has a significantly higher probability of being 1 (or 0), and grouping parameters base on that bit? Additionally, the paper does not discuss whether a 3-bit representation is insufficient for some parameters and requiring higher precision. It also fails to address whether this storage method effectively reduces **GPU memory consumption** and does not impact **inference efficiency**.\n\n3.  Are there other related papers proposing alternative outlier detection schemes? The paper does not compare its method with them. Moreover, the paper lacks a comparison with 3-bit results, both in terms of performance and inference efficiency.", "questions_to_authors": "Refer to the above", "extra_scores": {}}, {"review_id": "oiRd8Q1Now", "reviewer": "Reviewer_bufP", "summary": "This paper introduces ICQuant, a novel framework for efficient low-bit Large Language Model (LLM) quantization that leverages statistical properties of weight outliers. Major contributions include:\n* Separating weights into outliers and inliers for separate quantization\n* Designing an efficient index coding scheme based on the uniform distribution of outliers\n* Providing theoretical upper bounds on the storage overhead required by their index coding approach", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The method is quantizer-agnostic and compatible with various existing quantization schemes\n* Using only 2.3 bits per weight, ICQuant significantly improves quantization performance, substantially enhancing Llama model series accuracy compared to state-of-the-art methods like QTIP and QuIP#\n* The index coding scheme is highly storage-efficient, requiring merely ~0.3 bits overhead per weight\n* Comprehensive evaluation across multiple model families", "reasons_to_reject": "1. **Limited novelty**: The approach of separating outliers and inliers for separate processing is not fundamentally novel, as similar concepts appear in existing works like SqueezeLLM. Dividing weights into two groups for separate quantization is essentially a specific case of existing group quantization methods.\n2. **Distribution assumption issues**: The claim regarding uniform per-channel distribution of outliers needs more validation. Table 1 shows rejection rates as high as 95.25% for certain layers (e.g., o_proj), significantly contradicting the uniform distribution hypothesis. Authors only briefly mention these anomalies have \"minimal impact on the overhead,\" but provide insufficient evidence.\n3. **Limited innovation in coding scheme**: The proposed index coding scheme is essentially a basic data compression technique, limiting its innovativeness.\n4. **Insufficient analysis of permutation impact**: Authors suggest using random permutations to enforce uniformity in non-uniform cases but fail to analyze potential impacts on model performance, particularly on forward propagation speed. Figure 7 indicates that activations require an additional matrix multiplication, potentially increasing inference latency.\n5. **Lack of theoretical justification for parameter selection**: The assertion that 5% of outliers occupy 50% of the value range appears somewhat arbitrary, lacking theoretical support or systematic investigation of different proportions.\n6. **Puzzling experimental results**: The demonstration that results are better than quantization methods using fine-tuning is difficult to comprehend. Additional analysis should be provided, and code should be open-sourced promptly.", "questions_to_authors": "1. **Adaptive outlier ratio selection**: Consider implementing mechanisms for varying outlier ratios by layer characteristics, fully leveraging weight distribution statistics.\n2. **Integration with advanced quantization schemes**: Explore how ICQuant could complement more sophisticated quantization approaches and targeted fine-tuning methods to further enhance performance.\n3. **Deeper theoretical analysis**: Strengthen the theoretical analysis of outlier distribution properties, particularly examining robustness when the uniform distribution assumption doesn't hold.", "extra_scores": {}}, {"review_id": "0UjtTxT9SI", "reviewer": "Reviewer_ykoc", "summary": "This paper presents ICQuant, a framework that leverages outlier statistics to implement an index coding scheme for weight-only quantization. Compared to some existing methods, ICQuant reduces the overhead to about 0.3 bits, which may offer advantages in low-bit (2–3 bits) compression scenarios. It is designed to be compatible with existing quantizers to potentially enhance quantization quality. Experimental results show that ICQuant improves the zero-shot accuracy of a 2-bit Llama3-70B model and achieves performance comparable to fine-tuned quantizers, despite not requiring fine-tuning.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide an empirical observation of a uniform distribution of outliers and propose methods that require fewer bits of overhead to handle long-tail distributions.\n2. The included proofs enhance the transparency and interpretability of the proposed algorithms.\n3. The quantized models achieve perplexity comparable to prior works on WikiText2 and other benchmarks.\n4. Overall, the paper is well-written and easy to follow.", "reasons_to_reject": "1. Why are methods like GPTQ and AWQ not included in the comparisons? These are popular baselines in LLM quantization and their absence weakens the evaluation.\n2. A paper titled \"Scaling Laws for Quantized LLMs\" (https://arxiv.org/pdf/2411.17691) finds that low-bit quantization methods (e.g., GPTQ, AWQ) tend to fail on fully trained LLMs due to increasing quantization loss rather than decreasing it. Could you run similar experiments on the Pythia suite (across more than one model size) along their training trajectories to examine whether ICQuant exhibits the same weakness?\n3. Table 2 lacks performance data for certain configurations, such as the entire row of QuIP results for Llama2-7B. This makes it difficult to assess how well ICQuant performs in comparison.\n4. While Table 3 shows substantial improvements—up to ~50% on some LLM benchmarks using Llama3-70B. However, the same trend is not clearly reflected in Tables 6, 7, and 8 for smaller models (1B–13B). Additionally, the performance improvement on Llama2-70B (Table 6) appears significantly smaller than that on Llama3-70B. More experimental results are needed to determine whether ICQuant is agnostic to model architecture, size, and dataset. These factors may influence the statistical distribution of weights and thus affect quantization performance.", "questions_to_authors": "See above", "extra_scores": {}}, {"review_id": "GG6tOatI0g", "reviewer": "Reviewer_PnYh", "summary": "This paper introduces ICQuant, a quantization framework aimed at low-bit compression of LLMs. ICQuant specifically addresses the challenge of weight outliers with large magnitudes that inflate quantization ranges. Leveraging the observation that outliers distribute uniformly across weight matrices, ICQuant reduces the index storage cost to approximately 0.3 bits per weight. Experimental evaluations demonstrate that ICQuant improves quantization performance at extremely low bits, achieving SOTA performance without fine-tuning. This makes ICQuant particularly promising for deploying resource-constrained devices.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, well-structured, and easy to follow.\n2. The paper introduces a novel approach to addressing the challenge posed by weight outliers, utilizing separate quantization ranges combined with an optimized indexing scheme.\n3. The authors provide rigorous statistical analyses and theoretical bounds on their indexing scheme, ensuring methodological soundness and offering insights that may benefit future quantization techniques.\n4. Experimental evaluations show that ICQuant substantially improves quantization performance and zero-shot accuracy over SOTA methods.", "reasons_to_reject": "1. Limited generalizability of assumptions: The efficiency of ICQuant strongly depends on the assumption of a uniform distribution of outliers. While empirically validated in popular models, it remains unclear if this assumption is universally held, such as whether it is still suitable for the MoE structure models.\n2. Complexity of practical implementation: Although the indexing scheme significantly reduces overhead theoretically, practical implementation complexities, such as hardware support and runtime performance implications, were not thoroughly analyzed. It is unclear if the theoretical advantages could translate into a practical, deployable kernel.\n3. Insufficient analysis of computational cost: The proposed indexing scheme, while storage-efficient, introduces additional decoding complexity. The paper does not comprehensively evaluate the computational cost during inference.\n4. Lack of ablation on hyperparameters: The paper chooses a fixed outlier ratio (e.g., 5%) for many experiments without extensive exploration or justification. More rigorous analysis or ablation studies on the sensitivity to hyperparameters would enhance confidence in their choice.", "questions_to_authors": "1. Weight clipping has been shown to effectively manage weight outliers, as demonstrated in Sec. 3.3.2 of [LLMC](https://aclanthology.org/2024.emnlp-industry.12/). Could you provide a detailed comparison between your proposed method and this clipping approach?\n\n2. The proposed method is promising, but practical implementation details related to hardware deployment are missing. Could you provide an example of an ICQuant GEMM kernel or relevant benchmarks to illustrate the practical efficiency of your method?\n\n3. Could you report and discuss the performance of your quantized models on more challenging benchmarks, such as MMLU?", "extra_scores": {}}]

Author rebuttal:

Meta-review: While I, as an area chair, usually will side with reviews in their overall assessment and stand back and moderate the process, I find myself compelled to take over this process and override the reviewer sentiment.

All reviewers started with a score of 5/5/5/5 and the authors provided strong evidence against any negative points and alleviated all concerns of the reviewers, yet the reviewers did not increase their scores.

From the reviews I can see that some reviewers do not see the full context of this work. Some approaches like SqueezeLLM cannot work under the low-bit regime that the authors work in. The authors work has the best CUDA performance which is a big feat. Methods like fast QuIP inference has been a large effort and being more performant than that is a big achievement. Outliers beyond 5% are strongly established in the literature and the precision scaling laws work is irrelevant in this context.

There are absolutely no concerns for this paper. I personally would recommend it for a best paper award since it is a significant step up of quantization state of the art in software, theory, and methodology with exceptional results across the board. 

While I will strongly support acceptance, I will discuss with the PCs to see if a recommendation for oral or best paper would be fair given that the reviewers have different opinions.
Decision: Accept