SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
Authors: Runsheng Bai
Bo Liu
qiang liu
Venue: ICML 2025 poster (2025)
Keywords: model quantization
large language model
efficient inference
Primary area: general_machine_learning->scalable_algorithms
TL;DR: SKIM introduces an effective post-training quantization technique for large language models.

Abstract: Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller, less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently experience significant performance drops at lower precision levels. Additionally, they typically provide only a limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose a new method called \textbf{SKIM}: Scaled K-means clustering wIth Mixed precision. Our approach introduces two novel techniques: 1. A \textit{greedy algorithm} to solve approximately optimal bit allocation across weight channels, and 2. A \textit{trainable scaling vector} for non-differentiable K-means clustering. These techniques substantially improve the model performance and can be adapted to any given bit. Notably, in terms of perplexity, our method narrows the gap between quantized LLaMA models and their full precision counterparts by around \textbf{14\%} on average.

SKIM: Any-bit Quantization Pushing the Limits of Post- post-Training Quantization
Runsheng Bai
Bo Liu 2
Qiang Liu
Abstract
7.50
Ours
Large Language Models (LLMs) exhibit impres- sive performance across various tasks, but deploy- ing them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller. less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently expe- rience significant performance drops at lower pre- cision levels. Additionally, they typically provide only limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose new method called SKIM: Scaled K-means clustering with Mixed precision. Our approach introduces novel techniques: 1, greedy algorithm solve approximately optimal bit allocation across weight channels, and 2. A trainable scaling vector for non-differentiable K-means clustering. These techniques substantially improve the model per- formance and can be adapted to any given bit. Notably, in terms of perplexity, our method nar- rows the gap between quantized LLaMA models and their full precision counterparts by around 14% average.
7.25
FP16
7.00
OmniQuant
SqueezeLLM
perplexity
6.75
6.50
6.25
6.00
5.75
2.50
2.75
3.00
3.25 bit
3.50
3.75
4.00
Figure I. Our SKIM method adaptively quantizes model to any specified bit and achieves superior performance. The perplexity reported of LLaMA-7B on the Wiki dataset.
telligence (Bubeck et al., 2023). However, the computa- tional and memory demands of LLMs pose significant chal- lenges. For instance, when loading parameters in FP16, GPT requires 350GB of memory, while LLaMA-65B needs at least 130GB, both of which far exceed the capabilities of A 100-800 GPUs. Even when conducting inference with the smallest LLaMA model (7 billion parameters), Out-of- Memory exception can occur on widely used 24GB GPU, These challenges significantly complicate the storage and practical deployment of such models.
1. Introduction
Large Language Models (LLMs) including GPT (Radford et al., 2019) and LLaMA (Touvron et al., 2023a), have achieved remarkable performance across a diverse range of tasks. These models not only excel in language process- ing (Brown et al., 2020; Dubey et al., 2024; Chowdhery et al., 2023; Zhang al., 2022) but also adapt effectively to multimodal applications (Wang et al., 2024; Driess et al., 2023), marking a crucial step toward artificial general in-
One promising technique address these issues quantiza- tion, which involves transforming high-precision data into lower-precision formats, such converting FP16 parame- ters to INT4. This method directly reduces memory require- ments for deploying and loading LLMs, while improving inference latency by alleviating the memory bandwidth bot- tleneck, also known as the memory wall (Gholami et al., 2024). In addition, quantization has shown promising perfor- mance benefits. For example, previous studies have shown that both LLM weights and activations bits (Xiao al., 2023), only LLM weights can be stored in 4 bits (Kim et al., 2023), with little performance degrada- tion. This encourages researchers to explore lower- pwwer-precision solutions while maintaining reasonable performance levels.
'EECS Department, Massachusetts Institute of Technology 'Department of Computer Science, University of Texas Austin. Correspondence to: Qiang Liu <Iqiang @ et.u Run- sheng Bai <runsheng@@ne mit..ddu..
be stored in 8
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
However, standard quantization techniques resent meth-

---

SKIM: Any-bit Quantization
ods often suffer a significant drop in performance when us- ing low bit widths. To mitigate this decline, these methods often introduce additional techniques that incur extra mem- ory costs. For example, SqueezeLLM (Kim et al., 2023) retains certain sensitive elements and outliers with full pre- cision using a sparse lensor, while AWQ (Lin et al.. 2024) divides the quantization group into smaller ones, requiring the storage of more quantization factors. Additionally, the extra memory needed to achieve reasonable trade-off be- tween memory usage and performance often requires man- ual tuning and selection, making the process cumbersome.
Non-uniform quantization Non-uniform quantization (Jeon et al., 2022; Liu et al., 2022h) uses varying quantiza- tion intervals to better align with data distribution, leading to improved performance. Among various techniques, such as space transformation (Y et al., 2023) and trainable quantization factors (Jeon et al., 2022), K-means clustering (Krishma & Murty. 1999; Kanungo et al., 2002) is widely adopted (Xu et al., 2018; Zadeh al., 2020; Kim al.. 2023). It generates cluster labels and centroids, enabling us to store the labels directly as low-bit data and the centroids as a codebook for recovery.
Contribution In this paper, we address the above issues with our proposed method, Scaled K-means clustering wIth Mixed Precision (SKIM), which optimizes the bit allocation using a greedy algorithm and regularizes the column-wise difference with scaling vector. Our method can easily adapt to any specified bit, including even non-integer val- ues, and achieve better performance. Figure illustrates how our method breaks the fixed bit grid and delivers bet- ter results. Our main contributions can be summarized as follows: (1) We conduct a mathematical analysis of two optimization targets: layer-wise and sensitivity-based quan- tization, identifying a unified framework that highlights their core differences and allows us to evaluate their ef- fectiveness. (2) We observe significant disparity in data distribution across channels and propose a greedy algorithm for approximately optimal bit allocation in response to this disparity. Our mrxxed-precision method supports any bit level and demonstrates significant performance gains. (3) For the non-differentiable K-means clustering operator, we incorporate trainable scaling vector based on our novel iterative optimization strategy. This vector effectively reg- ularizes the data across columns and serves as valuable complement to the mixed precision method.
Quantization Techniques Outliers have been a signifi- cant obstacle for LLM quantization in achieving lossless solution (Dettmers et al., 2022; Xiao et al., 2023). Previous works have proposed various methods to address this issue, some of which share similarities (Dettmers et al., 2023; Xiao et al., 2023) with our mixed precision technique and scaling vector. However, our approach differs notably. Existing mixed precision techniques typically combine a specific bit level with the original precision, like INT3 and FP16, using element-wise mixing to preserve crucial elements. In con- trast, our method adaptively blends all available bit levels and uses a channel-wise mixture for better resource allo- cation. Regarding scaling factors, most existing methods apply them in uniform quantization, considering only differ- ential operators. Our work shifts the focus to a non-uniform context, optimizing scaling on non-differential grouping operators through a novel strategy.
3. Review The Quantization Objectives
Previous works have proposed different quantization ob- jectives, significantly broadening the scope of this field. However, these objectives are often considered in isolation, making the evaluation and selection of objectives unneces- sarily complicated, especially for our multi-process SKIM method. In this section, we conduct a comparative analy- sis on two widely adopted approaches: the layer-wise and sensitivity- based objectives. This analysis highlights their similarities and key distinctions, providing foundations for informed selection, which will be discussed in Section 4.1.
2. Related Work
Quantization of LLMs Quantization can be viewed from different perspectives. It can be categorized into two types based on whether the entire model is trained with quantiza- tion in mind: Quantizatinn-Aare Training (QAT) (Jacob et al., 2018; Xi et al., 2023) and Post- bast-Trainin Quantization (PTQ) (Cai et al., 2020; Shomron et al., 2021). While QAT methods typically perform better, their high resource re- quirements for retraining make them less practical for LLMs, making PTQ the preferred choice for language models and the focus of our work. Quantization methods can also be clas- sified into Weight-Activation Quantization (Yao et al., 2022; Xiao et al., 2023) and Weight-Only Quantization (Frantar et al., 2022; Chee et al., 2023; Lin et al., 2024), depending on whether both weights and activations are quantized. This paper focuses on the Weight-Only method, which directly addresses demands for deploying the model.
3.1. Notations
We define a general linear layer using following notations:
W E R= and E Rm denote the weight and input matrix, respectively. And the corresponding output matrix is Y W e guxk
The quantized weight, denoted as W E Rax= is the full-precision matrix reconstructed from its low-bit rep- resentation. It satisfies the constraint that its values are restricted to 2bir discrete centroids for each group.
2

---

SKIM: Any-bit Quantization
The i-th row of W is represented wy RIxm The same definition applies to wq 22 and yi-
3.4. Reformulation
Although layer-wise and sensitivity- based objectives have different assumptions and initial forms, the following re- formulation aims to organize them into unified structures, either full or diag, which will be discussed later.
The gradient and Hessian matrix are computed with respect to the final loss L. Taking wi as example, 9v. = V w. L E Rlxm represents the gradient, and Hw;- 2 L E Rmxm represents the Hessian matrix.
mn
Layer-wise quantization Since the 1.2-norm of a matrix can be expressed as the sum of that of its rows, Equation (1) can be reformulated in terms of the i-th row of W:
3.2. Layer-wise Quantization
The Layer-wise Quantization Framework has been widely adopted (Frantar et al., 2022; Hubara et al., 2021) to make the task more targeted. This framework aims to quantize each layer individually and addresses corresponding recon- struction problems. Concretely, let W be the full-precision weight matrix, and X the input data. The goal is to find the quantized weight W that minimizes the layer-wine squared error between the outputs of the original and quantized weights, which can be formally expressed as:
arg min [1w, - xxI
(5)
w?
= arg min (w; - x (n0: w w
(6)
Here XX happens to be the Hessian approximation de-
rived from the Fisher information matrix with respect to the layer-wise target, sharing the same structure as Equation (2). In the following discussion, we refer to this objective as L- full. Similarly, we can also take the diagonal approximation here, by assuming that X has an expectation close to zero. We use L-diag to refer to this approximated objective:
arg min W - 2
(1)
W4
3.3. Sensitivity-based Quantization
arg min (wa- w!)) diag((X)) (wr w!! T
Instead of minimizing the layer-wise squared error, Squeezel.LM (Kim et al., 2023) proposes minimizing the overall perturbation with respect to the final loss. They use the second-order Taylor expansion to analyze how changes in specific layer weight W influence the final loss, and fur- ther assume the first-order term is approximately zero since the model to be quantized should have already converged. For simplicity in understanding, here we use Wt, the i-th row of W, to explain the target. With its Hessian matrix Hw,, the objective can be written as:
= arg min 109
|,,51 (w:z - =, j=1
(8)
Sensitivity-based quantization For the i-th row of W and Y. the relationship y; w,X holds. Consequently, the gradient relationship can be expressed as 9w. = X Using this result, Equation (2) with Fisher approximation can be rewritten as:
arg min (w, wi)) Hw (w, - (w::-w)) T w/
(2)
arg min (w: w 9 T Iw. (w: w()T
(9)
wf
Furthermore, two additional approximations are incorpo- rated: 1. Take Fisher information matrix computed on a calibration dataset D as a Hessian approximation to avoid heavy computation. It can be formally expressed as H ~ F B[g g) z [o] EdeD(g(()) g(d),, where F is the Fisher information matrix and g(d)) represents the gradient computed on sample d: 2. The Fisher informa- tion matrix is further approximated as diagonal matrix by assuming that cross-weight interactions are negligible, reducing the complexity from quadratic to linear. Conse- quently, with diag(-) representing the diagonal function, the final objective can be written as:
arg min (lw, - )))
(10)
This Equation shares the same form Equation (5) and the only difference between them is whether include 9g. is included. It be referred to as S-full in the following text. Similarly, Equation (4) can also be expressed in a form that includes the output gradient and input:
arg min U
Σ (zz 9
10--wu -wwij -wg w²))²
(11)
This form is structurally identical to Equation (8) and will be denoted as S-diag.
arg min (wi- - w diag(Fm..) (wi - -w:) mn?
(3)
To conclude, the above transformation highlights the similar- ities and core differences between layer-wise and sensitivity- (4) based quantization, allowing us to intuitively evaluate their effectiveness and make selections, as detailed in Section 4.1 under the context of our SKIM method.
m
arg min ar?
(g, 2. -wis w
3

---

SKIM: Any-bit Quantization
Quantization Error
Level
Dequantized Weight
4.0
1000
3.5
0.6
1000
1000
1000
0.4
Recover
800
3.0
2000
2000
2.5
2000
0.0
400
2.0
-0.2
Greedy Algorithm
3000
3000
-0.4
200
13
4000
4000
1.0
Scaling Vector 0.994, 1.388]
Pre-Record
Mixed Precision
[1.013, 0.577,
Weight Matrix
Grouping Labels
Grouping Centrolds
Scaled Grouping Centrolds 2bit (-000058, 0.0462, 0.0121, 00.0433]
0
2bit [-0.0058, 0.0453, 0.0120, -0.0222]
K-Means Clustering
0.
3bit [.0046, -0.1596, uonm0nr 0.0053]
3000
-0.4
4bit [ 0011827, -0.1283, 0.0170J
Scale 4bit (0.1873, 00.135, 0.017X]
-0.6
4000
Figure Overall procedure of our proposed SKIM algorithm. The method consists of three main part: greedy algorithm for bit allocation, weighted K-Mcans Clustering based on allocation, and the trainable scaling vector. More details are available in Section 4.
4. Methodology
synchronization towards the final goal and facilitates cross- objective optimization in our work. Related experimental results are provided in Section 5.4.2.
The entire process of our SKIM algorithm is detailed in Figure 2. It begins with greedy algorithm that allocates different bits to each channel based on pre-recorded quanti- zation errors. If channel i is allocated by bits, 2 centroids will be generated for it. Next, weighted K-means clustering is applied to compute the centroids and labels for each chan- nel. Finally, we incorporate the scaling vector and train it through an iterative optimization strategy. The dequantized weights can be recovered from the final labels, centroids, and scaling vector. Full algorithm of our SKIM method is illustrated in Appendix A. In the following subsections, we will detail our mixed precision and scaling vector techniques, as well as the principles behind our objective selection.
Therefore, we can intuitively assess the effectiveness of each objective in the following order, from best worst: S-full, L-full, S-diag, and L-diag. This ranking reflects the fact that the S form incorporates gradient information as guide, while the diag form takes aggressive diagonal approximation. This analysis aligns with our experimental results, which will be detailed in Section 5.4.1.
However, due to the interdependence among the elements of the gradient and the input, we cannot compute the four objectives using directly recorded expectations of g or X. This limitation prevents us from adopting the most effective S-full objective, as recording the corresponding B(gg) g) for each row requires quadratic memory complexity which is impractical for LLMss.
4.1. Objective Selection
As discussed, our SKIM framework consists of three main steps: the greedy algorithm, weighted K-Means clustering, and the scaling vector. This multi-step process raises some questions: Should the same objective be used consistently across all steps, or is cross-objective optimization feasible? Moreover, under different computational scenarios, which objective is most effective, and how should it be selected?
Consequently, in any scenario requiring a complete error calculation, such as the proposed mixed precision and scal- ing vector methods, we adopt the L-full form. In contrast, for scenarios involving element-wise sums, such as the weighted K-means clustering, we adopt the S-diag form. This will be the main principle for our objective selection.
These questions can be addressed through our previous re- formulations, from which we can conclude that, although the layer-wise and sensitivity-based objectives differ, they can ultimately be transformed into either the full or diag forms. In these forms, both objectives share a similar struc- ture, differing only in the introduction of 9y, guide the subsequent model architecture. This alignment ensures their
4.2. Mixed Precision
As previously mentioned, the quantization error of weight matrix W is the sum of that of its individual rows. However, each row, representing each channel, often exhibits distinct data distributions and quantization errors, as illustrated in
4

---

SKIM: Any-bit Quantization
35
800
25
Frequency
600
3-bit Error
X
20
x
X
400
15
X
10
x
200
5
0
100
200
300
400
500
600
Log(Error)
2-bit Error
Figure 3. Histogram of the channel-wise quantization error for the self atttn.q-pooj second layer of Llama-7B. Errors vary significantly and exhibits long-tail mail distribution on the larger side.
Figure 4. Error variation of the sclf _alm..q.pro in the first layer of Llama-7B. We randomly sampled 10% of the total rows for clearer visualization, with each point representing one. The hori- zontal axis indicates the quantization error when using 2 bits, while vertical axis shows the error after increasing to 3. It is important to note that after the increase, same previous quantization error does not imply a similar post-increase error, larger error does not lead to larger result well.
level from
Figure 3. Additionally, Figure 4 demonstrates that the quan- tization error after one-bit increase is also unpredictable, even when the error at the current bit is known. These observations indicate that applying the same bit-width for quantizing rows results disproportionate allocation of resources. Motivated by these insights, we propose the adaptive channel-wise Mixed Precision technique, which aims solve:
LLMs. Therefore, opt for faster greedy algorithm to approximate the optimal solution. The greedy algorithm yield a solution sufficiently tight to the optimal one while only operates with a time complexity of O(n logn)". Simi- lar greedy algorithms have been widely adopted in various topics related to language models (Chen al., 2021; Liu et al., 2022a), demonstrating its effectiveness. Algorithm 1 describes the specific steps detail.
arg min by b2.........
Σ Crr(w,,bu bi), s.l. Σ 1=1
b < b.
(12)
where b; represents the bit allocated for channel i, is the total bit constraint, and Err(-) function refers to the L- full form of error between original un and by bit quantized weight, as analyzed earlier. This formulation identifies the minimiza-
4.3. Scaling Vector
Inspired by previous works (Xiao al., 2023), we recog- nized that scaling vector could effectively regularize differ- ences across channels, complementing our method. This operation can be formally expressed as:
mixed precision issue bit-constrained tion problem, which can be viewed variation of the knapsack problem (Martello & Toth, 1987; Cacchiani et al., 2022) and precisely solved using a dynamic programming algorithm. However, the quadratic computational complex- ity of this algorithm renders it infeasible for scalability in
W Quant(W * *a a-) a,
where W is the original weight, represents the element- wise multiplication with broadcasting, and a indicates the element-wise reciprocal of the scaling vector.
Algorithm Algorithm for Bit Allocation
input Error Matrix E, Minimum Available Bit bmins Maxi- mum Available Bit b Row Number Bit bit
However, unlike previous works, a significant challenge in our scenario lies determining the values the scal- ing vector a. Outliers play less dominant role in weight distributions compared to activations, and under channel- wise quantization, the most intuitive approaches based on similarity measures cannot be directly applied. Empirical values suggested by SmoothQuant (Xiao et al., 2023) and similar choices fail in our setting, leading to counterintuitive increases perplexity. This highlights the need for robust,
output Bit Allocation b
Initialize {bmin) h 12. E R.
repeat
arg maNggien (Erb,+1 Er,b, l by < Umnx)
6; b; +
until Σm= b, 2 bit
return b
5

---

SKIM: Any-bit Quantization
automated approach derive scaling values, rather than relying on fixed heuristics.
set the maximum available bit to in order maintain high memory efficiency. Consequently mixed precision is disabled under the INT4 setting. And to optimize the scaling vector, we utilize the Adam (Kingma, 2014) optimizer with a learning rate of 0.01, a decrease rate 0.5 every 40 steps and a maximum number of iterations of 120.
An intuitive solution is to make the scaling vector learn- able and optimize it toward a reasonable target, similar to OmniQuant (Shao et al., 2023) which employs gradient de- scent on block-wise error to train equivalent transformations. However, the presence of non dindiffeerntiable grouping oper- ator in K-means clustering precludes direct gradient descent on a. To tackle this challenge, we propose novel strat- egy called iterative optimization, which optimizes grouping and scaling separately. By keeping the grouping results fixed during training, this approach enables effective gradi- ent computation for optimizing a. Algorithm 2 provides a detailed explanation of the strategy.
Baselines We primarily compare our method against three baselines: Squeezel.LM (Kim al., 2023), Omni- Quant (Shao et al., 2023) and QulP# (Tseng et al., 2024). SqueezeLLM provides state-of-the-an performance under INT4 and INT3 settings, while OmniQuant offers greater flexibility and performs better in the INT2 setting. QulP# achieves comparable SOTA performance as well; however, given the increased resource demands associated with its finetuned version, we opt to use QulP# without finetuning one of our baselines. And since most existing works offer limited available bit levels, we ensure fairness by align- ing our chosen bit levels with those that are more widely adopted and only comparing under similar conditions. Com- parison with other baselines, including DecoupleQ (Guo et al., 2024) and ABQ-LLM (Zeng et al., 2024), can be found in Appendix B.2.
Iterative Optimization input Weight W, Iteration 1=1
Algorithm/ 2
output Labels L, Centroids c, Scaling Vector
n. m = W.shape Initialize a = 11xm E R11m
for i = to do W W a-
Welement-wise multiplication
L 0nxm E Raxm
for = 1 to n do
// opttmizze on grouping
L. + Kmm
Models and Datasets We evaluate our method across ious models, including LLaMA (Touvron et al., 2023a), LLaMA2 (Touvron et al., 2023b), and OPT (Zhang et al., 2022), to assess its generalizability. Results for the LLaMA models are emphasized in the main text due to their widespread adoption, while comprehensive results for other models can be found in Appendix B.1. Regarding datasets, we primarily utilize WikiText2 (Merity et al., 2016) and C4 (Raffel al., 2020) for evaluation, along with 100/128 samples from the C4 dataset for calibration.
end for
repeat
c= cale centroidst (W * a L)
W replace(c, L) a
loss= = Err(W, W*)
loss. backward()
Il optimize scaling
until Converged end for return L, c, a
Note that Algorithm is a simplified version of our SKIM method, clarifying the functionality of iterative optimiza- tion; the full SKIM method is illustrated in Appendix A. The loss calculation adopts the L-full form of error, as discussed in our previous analysis, while the optimization of grouping corresponds to the previously mentioned K-means Clus- tering. When mixed precision is enabled, the K Kmeans(.) function uses the allocation results from Algorithm 1, quan- tizing each channel with different precisions.
Evalutation We use the perplexity of language generation experiments as a primary metric, reporting results on the WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020) datasets. Since calibration dataset is derived from C4, perplexity on WikiTex12 represents a zero-shot scenario, while perplexity on C4 corresponds to a few-shot scenario. We omit specific average bits under integer settings due to negligible differences between methods, opting instead for reporting unified bit levels for clarity. Additionally, we evaluate accuracy on all PiQA(Bisk et al., 2020), ARC- Challenge/Eass(Clate et al., 2018) and MMLU (Hendrycks et al., 2020) benchmarks under both INT3/INT4 settings to assess the problem-solving capability of our quantized model. Since none of the calibration data directly relates to these benchmarks, the results may represent a complete zero-shot scenario and provide a high-level representation of our method's knowledge maintenance capability.
5. Experiments
5.1. Setups
Quantization Details We evaluate our method within the context of post-training and weight-only quantization. The default setting includes INT4 and INT3, as well as INT3 and INT2 with extra memory usage. Note that we have
6

---

SKIM: Any-bit Quantization
Table Quantization Result LLaMA models. We report perplexity of quantized LLaMA-7B and LLaMA2-7B in this table. Note that since SqueezeLLM did not provide an kernel implementation for bit setting. merge their official code with our functions and report the reproduced results. Perplexity of other models and comparison with other baselines can be found Appendix B.
bit
3.x bit
bit
2.x bit
LLaMA-7B
PPL
PPL
PPL
PPL
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FPI6
5.68
7.08
5.68
7.08
5.68
7.08
5.68
7.08
Sq
5.79
7.21
3.24
6.13
7.56
3
6.32
7.75
2.23
11.32
15.69
OmniQuant
5.86
7.34
3.24
6.15
7.75
6.48
8.19
2.25
9.72
12.79
QulP#
5.83
7.25
6.29
7.82
9.95
11.70
SKIM
5.79
7.20
3.2
6.07
7.52
6.21
7.68
2.25
8.99
11.00
4 bit
3.x bit
bit
2.x bit
LLaMA2-7B
PPL
PPL
PPL
PPL
Bit
Bit
Bit
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
5.47
6.97
5.47
6.97
5.47
6.97
5.47
6.97
SqueezeLLM
5.62
7.12
3.24
5.96
7.51
3
6.18
7.72
2.23
OmniQuant
5.74
7.35
3.25
6.03
7.75
3
6.58
8.65
2.25
11.06
15.02
QulP#
5.66
7.17
6.19
7.85
2
12.30
14.80
SKIM
5.60
7.11
3.2
5.91
7.48
6.09
7.66
2.25
10.10
12.42
5.2. Perplexity Results
Since SqueezeLLM does not provide benchmarking results for LLaMA models, we benchmark the quantized model generated by their official code to ensure consistency and reliability in our evaluation. As highlighted in Table 3, our SKIM method demonstrates a notable enhancement in the performance of the quantized LLaMA-7B across all tested benchmarks and both INT3 and INT4 precision lev- cls. Specifically, in the 3-bit setting, SKIM outperforms Squeezel.LMM in all benchmarks, showcasing its effective- in preserving knowledge retention during quantization. Additionally, in the 4-bit configuration, SKIM achieves higher average accuracy level as well, reinforcing the method's robustness across various settings. As result, this evaluation, covering diverse domains, clearly evidences the consistent performance improvements of our method.
The results our SKIM method applied to the LLaMA- 7B/13B models are presented in Table 1. Our method consistently outperforms alternatives across various con- figurations, demonstrating its versatility and effectiveness. Notably, with 3-bit quantization, our method achieves sig- nificant reduction in perplexity, narrowing the performance gap between full precision and 3-bit quantized models by 18.5% on LLaMA-7B and 15.7% on LLaMA-133 Further- more, our 3.2-bit model even surpasses others that operates slightly higher bit level, further highlighting the effec- tiveness of our method. These substantial reductions also evident in other models, detailed in Appendix B.
5.3. Benchmark Result
We evaluate our proposed method on various benchmarks, including PIQA, ARC-Challenge, ARC-Easy, and MMLU.
5.4. Ablation Study
5.4.1. EFFECTIVENESS OF OPTIMIZATION OBJECTIVES
As mentioned earlier, different forms and contents of op- timization objectives can lead to varying levels of effec- tiveness. Here, effectiveness refers to how much the final loss or perplexity positively influenced by optimizing specific target. We selected two scenarios evaluate the objectives: the sampled weights for K-means clustering and the loss calculation for optimizing the scaling vector. In the K-means clustering scenario, both mixed precision and the scaling vector are disabled, while the scaling vector scenario, we use S-diag as weights for clustering and disable
Objective
Perplexity
Objective
Perplexity
L-full
L-full
6.24
S-diag
6.33
S-diag
6.27
L-diag
6.36
L-diag
6.29
Table 2. Comparison of effectiveness between different objectives. On left, we present the perplexity results weighted K-means clustering, excluding the results of L-full due to its non-conformity to the element- wise sum. On the right, we display the perplexity results for the scaling vector scenario.

---

SKIM: Any-bit Quantization
Table Comparison of averaged benchmark accuracy a.aM22-7B. Tested benchmarks include PIQA, ARC-C, ARC-E and MMLU.
Method
Bit
PIQA
ARC-C
ARC-E
MMLU
Avg
LLaMA2-7B
16
77.0%
42.1%
64.5%
45.9%
57.4%
Lq
75.6%
38.8%
61.1%
41.3%
54.2%
SKIM
76.1%
40.6%
61.3%
42.3%
55.1%
Lq
77.0%
41.8%
63.8%
45.1%
56.9%
SKIM
76.9%
42.5%
64.6%
45.4%
57.3%
mixed precision. We employ 3-bit quantization and final perplexity illustrate effectiveness. Detailed results can be found in Table 2, aligning well with theoretical analysis. Evaluation of the S-full form is excluded due to its quadratic complexity, which renders it impractical.
5.5. Memory Efficiency
Following the approach outlined in SqueezeLLM (Kim al., 2023), we evaluated the Peak Memory Usage of SKIM method when generating 64 tokens, detailed in Table 4. Our method achieved lower peak memory usage even at the same bit level due to the utilization of lower precision for cluster centroids. Additionally, the 3.2-bit quantized model offers comparable memory savings to the 3.24-bit model from SqueezeL.LM, while also outperforming memory efficiency, our approach stands for its ability to break the fixed bit grid. Given specific GPU capacity constraint, users can select the maximum bit level to fully exploit the machine. For more detailed information, please refer to Appendix C.
5.4.2. ITERATIVE OPTIMIZATION
Separating the optimization and assigning different objec- tives for different processes does lead fluctuations in perplexity, illustrated in Figure 5. Disabling scaling gives us the grouping-onkl curve, while enabling it produces the grouping-scaling curve. The consistent drop in perplexity after enabling the vector validates the synchronization be- tween the targets. Regarding the number of iterations, we empirically find that the perplexity reduction brought by extra iterations is much more modest compared to the sig- nificant decrease observed after the first iteration between grouping and scaling. Additionally, increasing the number of iterations can sometimes lead to overfitting, which causes slight fluctuations in perplexity. Therefore, we are confident to set the maximum number of iterations to one.
Beyond
LLaMA-7B
3bit
3.xbit
FP16
12,72GB
Squeeze SKIM
3.01GB 2.98GB
3.26GB
3.13GB
LLaMA-13B
3bit
3.xbit
FP16
24.63GB
Squeeze SKIM
5,45GB
5,88GB
grouping-only gccaping
5.41GB
5.69GB
7.0
Table 4. Memory efficiency with I.L.MA-7B and aLaMA-33..
perplexity
6.8
6. Conclusion
6.6
We propose Scaled K-means clustering wIth Mixed Preci- sion (SKIM), an effective posting-training and weight-only quantization method. Building on previous non-uniform quantization methods. SKIM further incorporate two novel techniques: Adaptive Mixed Precision and Trainable Scal- ing Vector. Our method is evaluated across wide range of models, tasks, and bit levels, consistently outperforming pre- vious state-of oftee--f-the-ate the-hartt art methods. Its memory efficiency and ability to break the fixed bit grid facilitate the deployment of large language models.
6.4
6.2
2.6
2.7
2.8 bit
2.9
3.0
Figure Perplexity variation after enabling scaling vector. Per- plexity consistently decreases when additional optimization on the scaling vector applied.
8

---

SKIM: Any-bit Quantization
Impact Statement
Chen, J., Zheng, L.,, Yao, Z.., Wang. D., Stoica, I., Mahoney, M., and Gonzalez, J. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pp. 1803- 1813. PMLR, 2021.
SKIM advances efficient LLM deployment through adap- tive quantization, enabling shource-constraingg applica- tions like on-device Al and reducing cloud inference costs. This could democratize access to powerful language models while lowering resources consumption. However, wider accessibility may also lower barriers for malicious actors deploying LLMs at scale. Though SKIM maintains high ac- curacy, subtle quantization artifacts could propagate biases or errors from base models in safety-critical domains like healthcare or finance. To mitigate risks, we encourage: (1) robustness testing for quantized models in sensitive appli- cations, (2) developing auditing standards for compressed models, and (3) continued research into high and safe ma- chine learning efficiency. While SKIM itself introduces no novel harms, its compression features and efficiency gains warrant responsible deployment protocols. Overall, the ben- efits of SKIM in improving model accessibility outweigh its risks, but careful consideration of these factors is essential for responsible deployment.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-13,, 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: / 00188803.05555 2018.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318 1000188-0332,, 2022.
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: sparse-quantized representation for near-lossless 1lm weight compression. arXiv preprint 2023.
References
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1432,7439, 2020.
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowd- hery. A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duck- worth, D., Levine, S., Vanhoucke, V., Hausman, K. Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. PaLM-e: An embodied multimodal lan- guage model. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 8469-8488 PMLR, 23-29 Jul 2023. URL UtLhttps: http::///ccccceeeeeeeeoee mlr. mlr.press/ v202 (e22/drisss233 html.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.127/11 2023.
Cacchiani, V., Iori, M., Locatelli, A., and Martello, S. Knap- sack problems - an overview of recent advances. part Single knapsack problems. Computers & Operations Research, 143: 2022. ISSN 0305-0548. doi:
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
http::/ddb..rr/11..006jj.j. .0..0//6....022211005569
Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13169- 13178, 2020.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. Ai and memory wall. IEEE Micro, 2024.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36: 4396 4429, 2023.
Guo, Y., Kong. F., Li, X., Li, H., Chen, W., Tian, X., Cai, J., Zhang Y., and Liu, S. decoupleq: Towards 2-bit
9

---

SKIM: Any-bit Quantization
post-training uniform quantization via decoupling pa- rameters into integer and floating points. arXiv preprint arXiv:2404.122 12759, 2024.
Activation compressed training for generic network ar- chitectures. In International Conference Machine Learning, pp. 14139-14152. PMLR, 2022a.
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint Stt1:20009033000 2020.
Liu, Z., Cheng, K.TT., Huang, D., Xing, E. P., and Shen, Z. Nonuniform-to-umie
quantization: Towards accurate quantization via generalized straight-througe estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4942-4952, 2022b.
Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small cal- ibration sets. In International Conference on Machine Learning, pp.
Martello, S. and Toth, P. Algorithms for knapsack problems. North-Holland Mathematics Studies, 132:213-257, 1987.
PMLR, 2021.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint 1609::1609.000 (609.07843, 2016.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2002-2222222 2713 17,12.222 2018.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99, 2019.
Jeon, Y., Lee, C., Cho, E., and Ro, Y. Mr. biq: Post-training non-uniform quantization based on minimizing the recon- struction error. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 12329-12338, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (1 (140):1-67, 2020.
Kanungo, T., Mount, D. M., Netanyahu, N. S.,, Piatko, C. D., Silverman, R., and Wu, A. Y. An efficient k-means clus- tering algorithm: Analysis and implementation. IEEE transactions on pattern analysis and machine intelligence, (M(7)881-892. 2002.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L.,, Li, Z., Zhang. K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Ommidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308. 13137, 2023.
Shomron, G., Gabbay, F., Kurzum, S., and Weiser, U. Post- training sparsity-aware quantization. Advances in Neural Information Processing Systems, 34:17737- 2021.
Kim, S., Hooper, C., Gholami, A. Dong, Z., Li, X. Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-spane quantization. arXiv preprint ar Xrhv:23066.7669, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.cm 1412.6980, 2014.
Krishna, K. and Murty, M. N. Genetic k-means algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29(3):433-433 1999.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 23X.:2200..0928, 2023b.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Tseng, A., Chee. J., Sun, Q., Kuleshov, V., and De Sa, C. Quipit: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXhv:2402.04396, 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, Awq: Activation-awan weight quantization for on-device I1m compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Wang. W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision- centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
Liu, X., Zheng, L., Wang, D., Cen, Y., Chen, W., Han, X., Chen, J., Liu, Z., Tang, J., Gonzalez, J., et al. Gact:
10

---

SKIM: Any-bit Quantization
Xi, H., Li, C., Chen, J., and Zhu, Training transform- with 4-bit integers. Advances in Neural Information Processing Systems, 36:49145 14-4414,,49,8,, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099. PMLR, 2023.
Xu, Y., Wang, Y., Zhou, A., Lin, W., and Xiong, H. Deep neural network compression with single and multiple level quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post- training quantization for large-scale transformers. Ad- vances in Neural Information Processing Systems, 35: 27168-27183, 2022.
Yvinec, E., Dapogny, A., and Bailly, K. Nupes: Non- uniform post-training quantization via power exponent search. arXiv preprinl 23Xiv:2308.00555
2023.
Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nlp models for low la- teney and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitec- ture (MICRO), pp. 811 1-844. IEEE, 2020.
Zeng. C., Liu, S., Xie, Y.. Liu, H.. Wang. X., Wei, M., Yang, S., Chen, F,, and Mei, X. Abq-llm: Arbitrary- bit quantized inference acceleration for large language models. arXiv preprint arXiv:2408.0855. 2024.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint 2rXiv:2205.0/068, 2022.

---

SKIM: Any-bit Quantization
A. Full Algorithm
The full algorithm for our SKIM method is illustrated Algorithm Note that we provide the simplified version with number of iterations equal to one. Our method includes three main steps: 1. compute the bit allocation (line 2-11): apply channel-wise K-Means Clustering (line 12-15); train the scaling vector (line 16-22). Additionally, error pre-recording (line 2-10) only needs to be executed once for each model, and all KMeans functions accelerated using multi proleiipoeessing and shared memory.
Algorithm 3 Overall Algorithm for SKIM
input Weight W, Gradient Square G, Bit bit, Minimum Available Bit minn Maximum Available Bit bmux output Labels L, Centroids e, Scaling Vector
1: n. W.shope
2: E = [0=[]]axe ( ban -m)+1) E Ru bmiltt))
II Error Matrix
3: for = to n do
4: for b = mmin to max do
5: result Kmeans(Wa, weights = Gt n.centriods 22) Il apply Kmeans(-) with actual configuration
6: 1 result.abbels, resull.cenlroids
7: w reploce(c,
I/ replace labels with corresponding centroids
8: I., Err(W.... w.)
|| pre-record quantization error with Equation 5
9: end for
10: end for
11: (it bmins bmax: n, bit) E R"
compute allocation with Algorithm
L 0nxm Ruxm
13: for = to n do
14: Li.: Kmeans(Ws.. weights Gi n.centriods 22))) labelss
// optimize grouping towards Equation 4
15: end for
16: Initialize = 11xm R Rxxm
17: repeat
18:
c c= cawc_ Sell.ccetroods(WW * + L)
W replace(c, L) * a
replace labels with centroids and unscale to reconstruct the weight // calculate quantization error with Equation 5 // optimize scaling
loss Err(W, W*)
21:
bassbbakward()
22: until Converged
23: return L,
B. Additional Experiment Result
B.1. Perplexity Evaluation on Other Models
Table presents results for other LLaMA models, including LLaMA-30B and LLaMA2-7B. Our method continues to outperform other with larger model size and different model series. Additionally, table contains all results on OPT models, where our method also demonstrates superior performance, highlighting its generalizability. However, on OPT models, mixed precision would greatly reduces the quantization error yet slightly increase the final perplexity. For example, on the up-proj layer mixed precision reduces the quantization error by more than 50% but increases the perplexity. We attribute this phenomenon to overfitting the calibration data, and address it by disabling mixed precision under integer bit levels while initializing the bit allocation with flooring to the specified bits under non-integer levels. Consequently, even with the slight increase in perplexity, our method still provides better or comparable results.
B.2. Comparation with other baselines
DecoupleQ (Guo et al., 2024) decouples model parameters into integer and floating point parts, achieving statc-of-the-art performance in certain low-bit configurations. Table compares our SKIM method with DecoupleQ, and C4 results in excluded as DecoupleQ does not provide corresponding results. SKIM consistently outperforms DecoupleQ across most settings by a significant margin, highlighting its effectiveness. When operating at INT2 precision with additional memory,
12

---

SKIM: Any-bit Quantization
Table Quantization Result of other LLaMA models
bit
3.x bit
bit
2.x bit
LLaMA-13B
PPL
PPL
PPL
PPL
Bit
Bit
Bit
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
5.09
6.61
5.09
6.61
5.09
6.61
5.09
6.61
Lq
5.18
6.71
3.25
5.45
6.92
5.60
7.08
2.23
8.74
12.57
OmniQuant
5.21
6.76
3.25
5.44
7.05
5.68
7.32
2.24
7.93
10.76
QulP#
5.20
6.70
5.53
6,98
SKIM
5.17
6.70
3.2
5.42
6.92
5.52
7.04
2.25
7.40
9.22
4 bit
3.x bit
bit
2.x bit
LLaMA-30B
PPL
PPL
PPL
PPL
Bit
Bit
Bit
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
4.10
5.98
4.10
5.98
4.10
5.98
4.10
5.98
SqueezeLLM
4.22
6.06
3.25
4.44
6.23
3
4.66
6.37
2.22
OmniQuant
4.25
6.11
3.25
4.56
6.37
4.74
6.57
2.24
6.59
9.36
QuIP#
4.23
6.06
4.54
6.29
5.80
7.55
SKIM
4.20
6.05
4.46
6.22
4.53
6.31
2.25
5.80
7.49
Table 6. Quantization Result of OPT models
4 bit
3.x bit
bit
2.x bit
OPT-2.7B
PPL
PPL
PPL
PPL
Bit
Bit
Bit
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
12.47
13.17
12.47
13.17
12.47
13.17
12.47
13.17
SqueezeLLLM
4.07
12.80
13.38
3.25
13.43
13.88
3
13.85
14.45
OmniQuant
12.76
13.58
3.24
13.18
14.15
3
13.80
14.93
2.25
18.13
21.11
SKIM
12.72
13.35
3.2
13.34
13.92
13.66
14.21
2.25
19.79
19.96
bit
3.x bit
bit
2.x bit
OPT-6.7B
PPL
PPL
PPL
PPL
Bit
Bit
Bit
Bit
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
10.12
11.20
10.12
11.20
10.12
11.20
10.12
11.20
SqueezeLLM
11.03
11.85
3.26
11.31
12.18
11.70
12.44
OmniQuant
11.03
11.97
3.25
11.27
12.31
11.65
12.78
2.25
14.43
16.67
SKIM
11.02
11.84
3.2
11.27
12.20
11.46
12.39
2.25
14.79
16.01
SKIM provides comparable performance to DecoupleQ at the same bit level. However, with just a slight increase in memory- specifically, 0.05 bits-our method surpasses DecoupleQ by considerable margin. We present results with this minimal extra memory to illustrate the flexibility of our approach.
ABQ-LLM (Zeng et al., 2024) proposes block-wise distribution correlation and compensation schema for Post-Trainin Quantization, demonstrating strong performance at lower bit-widths as well. We report perplexity results on LLaMA-7B and LLaMA-13B models for INT4 and INT3 settings, as these overlap with our experiment data. As shown in Table 8, SKIM outperforms ABQ-LLM in both the INT4 and INT3 configurations.
13

---

SKIM: Any-bit Quantization
Table Comparation with DecoupleQ. Perplexity on WikiText2 is reported this table.
bit
3 bit
2.x bit
Method
Bits
PPL(()
Bits
PPL(()
Bits
PPL({)
LLaMA-7B
5.68
5.68
5.68
DecoupleQ
4
5.85
3
6.38
2.25
8.65
SKIM
4
5.79
6.70
2.30
8.64
LLaMA-13B
5.09
5.09
5,09
Decouple
4
5.21
3
5.60
2.25
7.25
SKIM
5.17 4
3
5.52
2.30
7.11
LLaMA-30B
4.10
4.10
4.10
DecoupleQ
4
4.24
3
4.67
2.25
6.04
SKIM
4
4.20
3
4.57
2.30
5.66
Table 8. Comparation with ABQ-LLM. Perplexity on both WikiText2 and C4 dataset is reported.
bit
3 bit
4 bit
bit
LLaMA-7B
LLaMA-13B
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
FP16
5.68
7.08
5.68
7.08
FP16
5,09
6.61
5.09
6.61
ABQ-LLM
5.83
7.29
6.29
8.01
ABQ-LLM
5.19
6.75
5.56
7.24
SKIM
5.79
7.20
6.21
7.68
SKIM
5.17
6.70
5.52
7.04
C. Memory Usage With Varying Bit Levels
Figure 6 illustrates the correlation between memory usage and bit levels. The peak memory usage increases linearly with the bit level, which facilitates the selection of an appropriate level to meet specific memory capacity requirements and to maximize the model's performance under specific machine.
3800
3600
Peak Memory(MB)
3400
3200
3000
2800
2600
2400
2.5
3.0
3.5
4.0
Bit
Figure 6. The actual peak memory usage of LLaMA-7B when generating 64 tokens.
14

---

SKIM: Any-bit Quantization
D. Quantization Cost
D.I. Memory Demands
As post-training quantization method, SKIM inherits the positive characteristics of memory savings. Although it involves training, this process is layer-wise, and the scaling vector is the only trainable parameter. As a result, the memory requirements during the quantization phase are significantly lower than those during inference. For instance, quantizing LLaMA-7B requires peak memory usage of less than 8GB, which is well within the memory capacity of most modern GPUs.
D.2. Quantization Time
In terms of K-means clustering, leverage parallel execution and shared memory enhance efficiency. Speed for both error matrix pre-recording and K-means quantization benefits from these improvements. To record the errors of LLaMA-7B using our framework, the process takes less than half an hour on dual AMD EPYC processors, which feature a total of 128 cores and 256 threads. And once the errors are recorded, they can be utilized for quantization at any specified bit level without needing to repeat the process. For K-means quantization, we can process approximately 8000 rows per second under the same machine conditions, while a transformer block LLaMA-7B only contains 42496 rows.
Regarding the packing and unpacking phases, by avoiding sparse matrices and consolidating all rows with same bit level, the packing phase is significantly faster than that of Squee7el.LM. It takes about minute to pack our quantized LLaMA-7B, whereas SqueezeLLM typically exceeds five minutes.
Overall, the entire process for quantizing LLaMA-7B takes around one hour with dual AMD EPYC processors and an RTX 3090 GPU. Compared to OmniQuant, our method is more efficient as we break the block-wise training to layer-wise one. And compared to SqueezeLLM, although the time consumption slightly higher due to the computations involved in bit allocation and scaling vector training, still remains comparable, ensuring high efficiency.
15


Reviews:
[{"review_id": "Usj0wtsmCD", "reviewer": "Reviewer_CLBX", "summary": "The work describes a post-training quantization technique that allows for non-integer size quantization of model parameters.\n\n## update after rebuttal\n\nThe rebuttal process helped to address my concerns and I thank the authors for their supporting answer(s). As a result I increased my score to 3, in hope the disscussed additions will be added to the final submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Comparison with ShiftAddLLM:\nThank you for the suggestion. We have now included a comparison with ShiftAddLLM. Overall, our method is competitive compared to ShiftAddLLM. In fact, ShiftAddLLM uses a smaller group size (128) rather than channel-wise quantization when quantizing LLaMA, so its precision should be classified as non-integer level. ShiftAddLLM does not provide detailed explanations about the precision of the scaling factors or additional overhead, so we compare its performance under the common practice of group size 128. We represented the perplexity of quantized LLaMA-7B on Wikitext2 in the table below. our method performs better at 3-bit, while being comparable at 2-bit. Moreover, by slightly increasing the bit level using the any-bit feature, significant improvements can be achieved. Additionally, beyond perplexity (PPL), our method achieves state-of-the-art performance on some widely-adopted benchmarks (see the table under answer 2), with some benchmarks even approaching FP16 performance. Those empirical results demonstrate that our method achieves comparable or superior performance to ShiftAddLLM. Furthermore, we want to emphasize the superior quantization efficiency of our method. While ShiftAddLLM requires approximately 20 GPU-hours to quantize the LLaMA2-7B, our approach completes the same process in under one hour. This dramatic difference in computational overhead becomes even more significant when scaling to larger models, making our method more practical for real-world applications.\n| Method         | Bit | Group Size | PPL (Wikitext2) |\n|----------------|-----------|------------|-----------------|\n| FP16 (LLaMA)  | 16        | -          | 5.68            |\n| ShiftAddLLM    | 3.0       | 128        | 6.04            | \n| SKIM (Ours)   | 3.25      | Channel-wise| **6.02**        | \n| ShiftAddLLM    | 2.0       | 128        | 7.98            | \n| SKIM (Ours)    | 2.25      | Channel-wise| 8.44            |\n| SKIM (Ours)    | 2.4       | Channel-wise| **7.76**        | \n\n\n\n2. Improvements for >3 Bits:\nIn fact, the improvements shown in perplexity are nonlinear—smaller improvements at lower PPL can still lead to notable performance gains. For example, when quantizing LLaMA2-7B, our compressed model demonstrates significant improvements on PiQA, ARC, and MMLU, as detailed in the table below. Meanwhile, the smaller PPL improvement at 3.x-bit is because, for consistency, we compressed the model to 3.2 bits rather than higher bit levels like 3.25 or above used by some other methods. For 4-bit, we set the maximum available bit to 4 to ensure quantization efficiency, effectively disabling mixed precision. If we were to use higher bit levels for 3.x-bit (e.g., 3.25 or 3.3 bits), PPL could be reduced by an additional 50%-100% (from 6.13 to 6.07 to 6.02), making the improvement more pronounced. Similarly, for 4-bit, if we relaxed the restriction, the gap to FP16 could be reduced from 0.13 to 0.11. This 0.02 reduce is relatively considerable for the INT4 level which is quite close to the original FP16 result.\n| Method                | Precision |   PIQA    |   ARC-C   |   ARC-E   |   MMLU    |    Avg    |\n| --------------------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n| **LLaMA2-7B**         |   FP16    |   77.0%   |   42.1%   |   64.5%   |   45.9%   |   57.4%   |\n| **INT3 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT3    | **76.1%** | **40.6%** | **61.3%** | **42.3%** | **55.1%** |\n| SqueezeLLM            |   INT3    |   75.6%   |   38.8%   |   61.1%   |   41.3%   |   54.2%   |\n| **INT4 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT4    |   76.9%   | **42.5%** | **64.6%** | **45.4%** | **57.3%** |\n| SqueezeLLM            |   INT4    | **77.0%** |   41.8%   |   63.8%   |   45.1%   |   56.9%   |\n\n\n\n3. Clarification on Any-Bit:\nPrevious methods do include some works that support any-bit due to mixed precision, but most fail to demonstrate \"continuity\" in bit usage. For example, SqueezeLLM allows increasing the precision of certain elements, but if an extra available bit is added to INT3, its performance would be worse than INT4 rather than matching it. In contrast, our any-bit method, based on allocation, can fully utilize the given bit budget, making it unique and practical.", "claims_and_evidence": "The claim that this beats baselines is wrong as it only beats the presented baselines. A recently presented technique (NeurIPS2024) named ShiftAddLLM has shown to give better perplexity / quantization ratios for 2 and 3 bits (for reference: Table 3 lists results using LLaMa models). The work of ShiftAddLLM may not official list as PTQ technique, although it clear is. But I view missing reference and exclusion of a better working approach published earlier to be a major defect of a submission.", "methods_and_evaluation_criteria": "An important, better baseline was omitted from the comparison.", "theoretical_claims": "The theoretical claims were all sound to me.", "experimental_designs_or_analyses": "The improvements to presented baselines for >3 bits are minor.", "relation_to_broader_scientific_literature": "Non-integer size quantization of large sets of integers/floats may help other research areas apart from ML as well. I didn't check, but the idea may have been presented elsewhere already.", "essential_references_not_discussed": "ShiftAddLLM: https://arxiv.org/abs/2406.05981 with code at https://github.com/GATECH-EIC/ShiftAddLLM", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9bVzxsytLr", "reviewer": "Reviewer_eA4i", "summary": "The paper introduces SKIM (Scaled K-means clustering wIth Mixed precision), a novel post-training quantization method for Large Language Models (LLMs) that supports any-bit quantization. The key contributions include a greedy algorithm for optimal bit allocation across weight channels, addressing the significant disparity in data distribution across channels, and a trainable scaling vector for non-differentiable K-means clustering, which regularizes column-wise differences and complements the mixed-precision method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. More Information about the Greedy Algorithm: \nThrough empirical validation, we have demonstrated the effectiveness and efficiency of the greedy algorithm:\n\n - Time Efficiency: The cpp/jit implementation of the dynamic programming algorithm remains slow. For a single block in LLaMA-7B, it requires several minutes of computation. Given that LLaMA-7B has 32 layers with 7 blocks per layer, this algorithm struggles to scale to large models. In contrast, the greedy algorithm, even when implemented in Python, only requires approximately 3 seconds per block, ensuring computational efficiency.\n\n - Practical Effectiveness: To guarantee a near-optimal solution, we employ two different initialization points for the greedy algorithm and select the superior solution. Empirical validation on the first layer of LLaMA-7B shows that the average error between the suboptimal solution from the greedy algorithm and the true optimal solution is below 2%, confirming its capability to deliver sufficiently high-quality results.\n\n\n2. Additional experiments:\nWe further validated our method by comparing it with more recent works (e.g., adding QuIP# as a new baseline) and extended benchmarking on tasks like PiQA and ARC. These results further validates the effectiveness of our method and detailed results are shown below:\n\n| Method                | Precision |   PIQA    |   ARC-C   |   ARC-E   |   MMLU    |    Avg    |\n| --------------------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n| **LLaMA2-7B**         |   FP16    |   77.0%   |   42.1%   |   64.5%   |   45.9%   |   57.4%   |\n| **INT3 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT3    | **76.1%** | **40.6%** | **61.3%** | **42.3%** | **55.1%** |\n| SqueezeLLM            |   INT3    |   75.6%   |   38.8%   |   61.1%   |   41.3%   |   54.2%   |\n| **INT4 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT4    |   76.9%   | **42.5%** | **64.6%** | **45.4%** | **57.3%** |\n| SqueezeLLM            |   INT4    | **77.0%** |   41.8%   |   63.8%   |   45.1%   |   56.9%   |\n\n| LLaMA2-7B      | 4-bit |          |          | 3.x-bit |          |          | 3-bit |          |          | 2.x-bit |           |           |\n| ---------- | :-----: | -------- | -------- | :-------: | -------- | -------- | :----: | -------- | -------- | :-------: | --------- | --------- |\n|            | Bit   | Wiki     | C4       | Bit     | Wiki     | C4       | Bit   | Wiki     | C4       | Bit     | Wiki      | C4        |\n| FP16       | -     | 5.47     | 6.97     | -       | 5.47     | 6.97     | -     | 5.47     | 6.97     | -       | 5.47      | 6.97      |\n| SqueezeLLM | 4     | 5.62     | 7.12     | 3.24    | 5.96     | 7.51     | 3     | 6.18     | 7.72     | -    | -         | -         |\n| OmniQuant  | 4     | 5.74     | 7.35     | 3.25    | 6.03     | 7.75     | 3     | 6.58     | 8.65     | 2.25    | 11.06     | 15.02     |\n| QuIP#      | 4     | 5.66     | 7.17     | -       | -        | -        | 3     | 6.19     | 7.85     | 2       | 12.30     | 14.80     |\n| **SKIM**   | 4     | **5.60** | **7.11** | 3.2     | **5.91** | **7.48** | 3     | **6.09** | **7.66** | 2.25    | **10.10** | **12.42** |", "claims_and_evidence": "The mixed-precision technique with greedy bit allocation improves quantization performance by adaptively allocating bits per channel. This is demonstrated through experiments showing reduced quantization errors and improved perplexity across different bit levels. The trainable scaling vector effectively regularizes data across columns and improves quantization accuracy. Experimental results confirm that incorporating the scaling vector leads to consistent drops in perplexity.", "methods_and_evaluation_criteria": "The proposed methods are appropriate for the problem of post-training quantization of LLMs. The greedy algorithm for bit allocation addresses the observed disparity in quantization errors across channels, making sense for resource optimization. And the trainable scaling vector provides a solution for regularizing non-differentiable K-means clustering, which is a common challenge in quantization methods.", "theoretical_claims": "The authors provide a detailed analysis of layer-wise and sensitivity-based quantization objectives, showing how they can be transformed into unified structures. The derivation of the mixed-precision problem as a bit-constrained sum minimization is logically sound, though a more in-depth examination of the dynamic programming algorithm's limitations would strengthen this section.", "experimental_designs_or_analyses": "The authors evaluate on multiple LLMs (LLaMA, LLaMA2, OPT) to demonstrate generalizability, and they compare against relevant baselines (SqueezeLLM, OmniQuant) under similar conditions.", "relation_to_broader_scientific_literature": "Builds on previous work in non-uniform quantization methods, particularly K-means clustering approaches. Extends the research on post-training quantization for LLMs, addressing limitations of existing methods that often experience significant performance drops at lower precision levels. Contributes to the growing body of work on efficient inference methods for LLMs, focusing on memory and computational efficiency.", "essential_references_not_discussed": "The paper could benefit from discussing more recent works on adaptive quantization and hardware-aware optimizations for LLM inference.", "other_strengths_and_weaknesses": "The article is clearly structured and written. The methods are well presented. If possible, more illustrations or ablation studies would be helpful.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pJNcxK4yI7", "reviewer": "Reviewer_ABnm", "summary": "This paper introduces a mixed-precision weight-only quantization method for large language models (LLMs). The authors propose a greedy algorithm to allocate bitwidths across weight channels, followed by K-means clustering based on the assigned bitwidths. To enhance performance, the authors further incorporate a trainable scaling vector to refine the non-differentiable clustering process. Experiments on LLaMA models demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Clarification on Objective Rankings:\nThe ranking of objectives (S-full > L-full > S-diag > L-diag) is elaborated in Section 4.1, where we demonstrate that L-full outperforms S-diag. This is because S-diag simplifies computation via a diagonal assumption, introducing a certain degree of bias compared to the full form. Practically, Section 5.4.1 demonstrates that L-full outperforms S-diag in perplexity, validating our analysis. \n\n2. Dequantization Efficiency in Mixed-Precision Settings:\nIn mixed-precision implementations, we group channels with identical bitwidths and track their original indices to leverage GPU parallelism. Although dequantization introduces minor overhead compared to uniform-precision methods, this overhead is smaller than the computational cost of sparse tensor operations (e.g., in SqueezeLLM) and is justified by significant accuracy improvements. Our method performs similarly to SqueezeLLM latency-wise and faster than it throughput-wise.\n\n3. Differentiation from Prior Mixed-Precision Work:\nExisting methods like SqueezeLLM, SliM-LLM, and CLAQ mix only a limited set of predefined precisions. For instance, SqueezeLLM combines INT3/4 with FP16, while SliM-LLM and CLAQ mix INT X with INT X+k and also include INT X-k for compensation. In contrast, our framework adaptively assigns arbitrary bit widths (e.g., mixing 1~4bit) across channels without relying on fixed thresholds or compensation rules. This flexibility broadens the scope of discussion and enables finer-grained error minimization.\n\n4. Expanded Baseline Comparisons:\nWe have added comparisons with QuIP#, a state-of-the-art baseline, which outperforms AWQ and QuIP. We choose the pure-PTQ version of QuIP# (QuIP# with fine-tuning consumes greater computational resources; for example, it requires 50 GPU hours to quantize LLaMA-70B on an 8 GPU node) and the the result is included in the following table:\n\n| LLaMA2-7B      | 4-bit |          |          | 3.x-bit |          |          | 3-bit |          |          | 2.x-bit |           |           |\n| ---------- | :----: | -------- | -------- | :----: | -------- | -------- | :----: | -------- | -------- | :----: | --------- | --------- |\n|            | Bit   | Wiki     | C4       | Bit     | Wiki     | C4       | Bit   | Wiki     | C4       | Bit     | Wiki      | C4        |\n| FP16       | -     | 5.47     | 6.97     | -       | 5.47     | 6.97     | -     | 5.47     | 6.97     | -       | 5.47      | 6.97      |\n| SqueezeLLM | 4     | 5.62     | 7.12     | 3.24    | 5.96     | 7.51     | 3     | 6.18     | 7.72     | 2.23    | -         | -         |\n| OmniQuant  | 4     | 5.74     | 7.35     | 3.25    | 6.03     | 7.75     | 3     | 6.58     | 8.65     | 2.25    | 11.06     | 15.02     |\n| QuIP#      | 4     | 5.66     | 7.17     | -       | -        | -        | 3     | 6.19     | 7.85     | 2       | 12.30     | 14.80     |\n| **SKIM**   | 4     | **5.60** | **7.11** | 3.2     | **5.91** | **7.48** | 3     | **6.09** | **7.66** | 2.25    | **10.10** | **12.42** |\n\nNote that QuIP# officially supports only integer-bit quantization (e.g., INT2); thus, we compare its INT2 results with our 2.x-bit performance.\n\n5. OmniQuant Implementation Details:\nUnder non-integer bitwidths(eg: 3.x-bits), OmniQuant is evaluated under its group-wise quantization setting (eg: 128 elements per group), as specified in their official implementation. For integer bits (e.g., 3/4-bit), per-channel quantization is used, same as our method. Previous works commonly adopt this setting.\n\n6. LLaMA3 Benchmarking:\nWe focus on LLaMA-1/2 and OPT models to align with established baselines (SqueezeLLM, OmniQuant, and baselines you mentioned), as LLaMA3 results are not yet widely reported in quantization literature. This ensures apples-to-apples comparisons. We will be happy to include LLaMA3 in future work once baselines adopt it.\n\n7. Expanded Evaluation Metrics:\nFollowing suggestions, we have added results on ARC-E, ARC-C, and PIQA. Below is the detailed result and our method maintains its consistent superiority.\n| Method                | Precision |   PIQA    |   ARC-C   |   ARC-E   |   MMLU    |    Avg    |\n| --------------------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n| **LLaMA2-7B**         |   FP16    |   77.0%   |   42.1%   |   64.5%   |   45.9%   |   57.4%   |\n| **INT3 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT3    | **76.1%** | **40.6%** | **61.3%** | **42.3%** | **55.1%** |\n| SqueezeLLM            |   INT3    |   75.6%   |   38.8%   |   61.1%   |   41.3%   |   54.2%   |\n| **INT4 Quantization** |           |           |           |           |           |           |\n| SKIM                  |   INT4    |   76.9%   | **42.5%** | **64.6%** | **45.4%** | **57.3%** |\n| SqueezeLLM            |   INT4    | **77.0%** |   41.8%   |   63.8%   |   45.1%   |   56.9%   |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper proposes a mixed-precision quantization method for LLMs. However, it lacks a thorough discussion and experimental comparisons with existing methods, which would help contextualize the contributions within the broader literature.", "essential_references_not_discussed": "Please see my detailed comments in the following.", "other_strengths_and_weaknesses": "Strengthens:\n\nThe proposed method is simple yet intuitive. The authors introduce a preliminary unified perspective on layer-wise and sensitivity-based quantization, offering valuable insights that enhance the reader’s understanding.\n\nWeaknesses:\n\nThe paper lacks a thorough discussion and experimental comparisons with several existing weight quantization methods, which would help situate the proposed approach within the broader context of related work. Please see my detailed comments.", "other_comments_or_suggestions": "Please see my detailed comments.", "questions_for_authors": "1.\tIn Section 4.1, the authors state that the effectiveness of different objectives ranks from best to worst as S-full, L-full, S-diag, and L-diag. Could the authors clarify whether S-diag performs better than L-full? Additionally, it would be helpful to explain how this ranking is determined.\n\n2.\tIn Section 4.2, the authors propose assigning different bitwidths to different weight channels, which effectively reduces quantization errors. However, during inference, the quantized weights must be dequantized to FP16 for computation. Could the authors clarify whether dequantization in mixed-precision settings is slower than in uniform-precision settings? If so, to what extent?\n\n3.\tThe proposed method uses mixed-precision quantization to improve the accuracy of the quantized models. Several works [1][2] have also explored mixed-precision quantization. It would be beneficial for the authors to include a more detailed discussion of these related approaches, highlighting the differences.\n\n4.\tThe experimental comparisons are insufficient, as several state-of-the-art weight-only quantization methods are not included, such as [3][4][5], among others.\n\n5.\tThe settings of the compared quantization methods are not clearly specified. For example, do the results reported for OmniQuant correspond to group-wise or channel-wise quantization?\n\n6.\tThe experimental results on LLaMA, LLaMA2, and OPT are outdated. It would be beneficial for the authors to include additional results on LLaMA3 [6].\n\n7.\tThe experimental results based solely on MMLU are insufficient. The authors should include additional commonly used datasets, such as LAMBADA [7], ARCEasy (ArcE) [8], and PIQA [9], which are frequently used in the literature to evaluate the performance of quantized models.\n\nReference:\n\n[1] SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models. arXiv 2024.\n\n[2] CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs. arXiv 2024.\n\n[3] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024.\n\n[4] QuIP: 2-Bit Quantization of Large Language Models With Guarantees. NeurIPS 2023.\n\n[5] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024.\n\n[6] The Llama 3 Herd of Models. arXiv 2024.\n\n[7] The LAMBADA dataset: Word prediction requiring a broad discourse context. ACL 2016.\n\n[8] A systematic classification of knowledge, reasoning, and context within the ARC dataset. ACL 2018. \n\n[9] Piqa: An algebra for querying protein data sets. International Conference on Scientific and Statistical Database Management 2003.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is a strong consensus among the reviewers that this paper should be accepted, so I'm going along with that. But I think it's missing comparisons to some more recent related work that should be included: if you're going to compare to QuIP# you should also (or instead) compare to QTIP ("QTIP: Quantization with Trellises and Incoherence Processing," NeurIPS 2024), which is a more recent and better method from the same family (a more SOTA baseline than QuIP#). SpinQuant ("SpinQuant: LLM quantization with learned rotations") also seems relevant.
Decision: Accept (poster)