SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
Authors: Wei Huang
Haotong Qin
Yangdong Liu
Yawei Li
Qinshuo Liu
Xianglong Liu
Luca Benini
Michele Magno
Shiming Zhang
XIAOJUAN QI
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Low-bit Quantization
Machine Learning
Mixed-Precision
Primary area: deep_learning->large_language_models
TL;DR: A structured mixed-precision quantization method for large language, achieving both hardware efficiency and improved language performance.

Abstract: Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs). However, while uniform-precision quantization is computationally efficient, it often compromises model performance. To address this, we propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise with high accuracy. Our approach leverages the observation that important weights follow a structured distribution and introduces two key components: 1) Salience-Determined Bit Allocation adaptively assigns bit-widths to groups within each layer based on their salience; and 2) Salience-Weighted Quantizer Calibration optimizes quantizer parameters by incorporating element-level salience, retain essential information. With its structured group-wise partitioning, SliM-LLM provides a hardware-friendly solution that matches the efficiency of uniform quantization methods while significantly improving accuracy. Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths. For example, a 2-bit quantized LLaMA-7B model reduces memory usage by nearly 6x compared to the floating-point baseline, decreases perplexity by 48% compared to state-of-the-art gradient-free PTQ methods, and maintains GPU inference speed. Additionally, the extended version, SliM-LLM+, which incorporates gradient-based quantization, further reduces perplexity by 35.1%. Our code is
available at https://github.com/Aaronhuang-778/SliM-LLM.

SIiM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
Wei Huang' Haotong Qin Yangdong Liu Yawei Li
Qinshuo Liu'
Xianglong Liu Luca Benini Michele Magno Shiming Zhang
Xiaojuan Q!
Abstract
1. Introduction
LLMs have demonstrated remarkable performance var- ious natural language benchmarks (Brown et al., 2020; Hendrycks et al., 2020). Models like LLaMA (Touvron et al., 2023a) and GPT (Brown et al., 2020) have driven progress toward universal language intelligence. Their ca- pabilities have also extended to multi-modal domains (Li et al., 2024b; Achiam et al., 2023; Team et al., 2023; Zhang et al., 2023; Huang et al., 2024b), advancing efforts toward artificial general intelligence (AGI) (Bubeck al., 2023). However, their high computational and memory demands remain significant challenge for practical deployment.
Post-training quantization (PTQ) is an effective technique for compressing large language mod- els (LLMs). However, while mniform-precisti quantization computationally efficient, it of- ten compromises model performance. To ad- dress this, we propose SliM-LLM, salience- driven mixed-precision quantization framework that allocates bit-widths at the group-wise. Our approach leverages the observation that impor- tant weights follow a structured distribution and introduces two key components: 1) Salience- Determined Bit Allocation adaptively assigns bit- widths groups within each layer based on their salience; and 2) Salience- Salience-Weighted Quan- tizer Calibration optimizes quantizer parameters by incorporating element-level salience. With its structured partitioning, SliM-LLM provides hardware-friendl! solution that matches the ef- ficiency of uniform quantization methods while improving accuracy. Experiments show that SliM- LLM achieves superior performance across vari- ous LLMs at low bit-widths. For example, a 2-bit quantized LLaMA-7B model reduces memory us- age by nearly 6x compared to the floating-pon point baseline, decreases perplexity by 48% compared to state-of-the-an gradient-free PTQ methods, and maintains GPU inference speed. Additionally, the extended version, SliM-LLM*, which incorpo- rates gradient-based quantization, further reduces perplexity by 35.1%. %. Our code is available at http:ggiiiubcommmaaaaooiieee 778/S/MM-LM////
To address resource constraints of LLMs, PTQ has emerged as an efficient yet effective compression tech- nique (Dettmers et al., 2022), showing success in quantizing the weights of pre-trained LLMs (Frantar et al., 2022; Lin et al., 2023; Shao al., 2023; Lee al., 2023; Chee et al., 2024). As LLMs continue to scale, the demand for more ag- gressive low-bit bit compression becomes critical due to limited computational and storage resources in application (Huang et al., 2024a; Tseng et al., 2024). However, significant performance degradation remains a challenge in low-bit scenarios (& 3-bit). To mitigate this, unstructured mixed- precision quantization (Shang et al., 2023; Huang al., 2024a; Dettmers et al., 2023) and vector quantization (Chee et 2024; Tseng et al., 2024; Egiazarian et al., 2024) meth- ods have been developed to preserve performance. While these approaches have advanced the field, they are often hardware- crrdwarr-unfriendly.. introducing extra storage requirements such as storing bitmaps or code indices. This creates bot- tleneck, limiting further reductions in memory and compu- tational demands during deployments.
This paper presents Salience-Driven Mixed- Group LLM (SIiM-LLM) framework, an accurate and inference-efficient PTQ method for LLMs ( 3-bit). Our approach is grounded in the key observation that salient or important weights, which are critical to model performance, exhibit struc- tured distribution, often clustering within certain channels (see Sec. 3.2.1 and Fig. 3). This insight, largely overlooked by prior research (Frantar et al., 2022), forms the basis for designing SliM-LLM as a structured, hardware-friendll
'The University of Hong Kong *ETH Zürich Beihhan Uni- versity. Correspondence to: Haotong Qin, Shiming Zhang, Xiaojuan Qi <haotong. wwwpiooonggi@ppl......mm @ttee....e..tt..cch beszahng hkk.hkk, xjqi @ eec hk..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SliM-LLM
266 7
207
6.5
Method
No Fine- Tuning
Miaed Preckton
Stractured Quamtiattonn
Quantiser
160
AWQ
Omri
AQUM QuiP*
X
X
X
C/C/S
GPTQ
DB4LLM
Wikitex22 Perplexity
80
PB4LLM
AfinnQuantt
QuiP
X
X
QuIP
SIIM-LLM
AWO
/
QPTQ
40
SHIM-LLM
BILLM PBILLM LLM MQ
X
S/S/S
20
Onnr Cuant AftineQuaant
4
x
GIG
10
SIM-LLM
5
3.5
SIM-LLM
J
7b
13b
30b
650
Tb
13b
30b
65b
Model Size (Parama)
Model Size (Params)
(a) LLaMA 2-bit Results
(b) LLaMA 3-bit Results with Gradient Quantizer
(c) PTQ Method Classification
Figure 1. (a) The perplexity (L) existing low-bit PTQ methods of LLaMA at 2-bit. Solid-line indicates methods with structured
quantization group. (b) Compare PTQ methods with gradient quantizer 3-bit. (c) Features of current low-bit quantization methods, C denotes codebook- -assed, S is statistic-based, and G represents gradient- based quantizers.
mixed-precision low-bit method. It preserves performance through two key designs that retain important weights at both the global group and local element levels. First, we develop a novel Siffence-DDtermineee Bit Allocation (SBA) method, which adaptively assigns bit-widths to each quan- tization group based on their group-level salience ranking. The allocation strategy is optimized to reduce activation reconstruction errors. By applying higher precision more important groups and reducing the bit-widths for less critical ones, SBA achieves a low average bit-widths while enhanc- ing the overall performance of LLMs. Next, we introduce the Salience- Weeghtted Quantizer Calibration (SQC), which enhances sensitivity to locally salient weights, ensuring that critical information within groups is preserved. SQC works collaboratively with SBA, exploiting the local and global salience of weights to preserve the performance of LLMs after quantization. Unlike element-wise mixed-precision methods (Shang et al.,, 2023; Dettmers et al., 2023; Huang et al., 2024a). SliM-LLM is inherently structured, eliminat- ing additional bit or computational overhead while preserv- ing high performance. This is further demonstrated through our deployment of SliM-LLM in an application-leva infer- ence tool for LLMs, enabling efficient mixed precciion inference on GPUs with consistently strong performance.
its superior low-bit accuracy and efficiency. We also inte- grate SliM-LLM into OmniQuant (Shao et al., 2023) and obtain SliM-LLM through gradient optimization to further improve quantization quality. Moreover, the group-wise mixed-precision strategy can smoothly be adapted to exist- ing quantization-awar training (QAT) (Liu et al., 2023), fine-tuning based (Guo al., 2023; Liao Monz, 2024;
Dettmers et al., 2024), codebook- based (Chee et al., 2024; Egiazarian et al., 2024; Tseng et al., 2024) LLMs com- pression methodologies. This structure of weight salience introduces a new practical view of compression for LLMs.
2. Related Work
Large Language Models have been significantly developed in diverse natural language processing domains, establish- ing a prominent paradigm in these fields (Bubeck et al., 2023; Chang et al., 2024; Zhao et al., 2023; Brown et al., 2020; Touvron et al., 2023a). Nevertheless, the exceptional success of LLMs depends on massive parameters and com- putations, posing significant challenges for deployment in resource-constrained environments. Consequently, research into the compression of LLMSs has emerged as a promising field. Existing compression techniques for LLMs primar- ily include quantization, pruning and distillation (Xu et al., 2023; Ganesh et al., 2021; Frantar et al., 2022; Xiao et al., 2023a; Shao et al., 2023; Chee et al., 2024; Zhu et al., 2023; Frantar Alistarh, 2023; Huang et al., 2024a; Qin et al., 2024). Low-bit quantization has received notable attention for efficiently reducing the size of the model without chang- ing the structure of the network(Zhu) et al., 2023; Zhao et al., 2023; Chang et al., 2024).
Experiments show that for various LLM families, SliM- LLM surpasses existing training-free PTQ methods on di- verse benchmarks, particularly in low-bit scenarios. Using GPTQ as the backbone, SliM-LLM improves the perplexity scores of 2-bit I.LMMA-13B and ILaMA2-133R
Wiki- Text2 (Merity et al., 2016) from 20.44 and 28. 18.14 14th to 8.87 and 9.41, denoting performance improvements of over 56%, re- spectively. SliM-LLM even outperforms other element-wise mixed-precision PTQ methods, such as PB-LLMM (Shang et al., 2023), APTQ (Guan et al., 2024) and LLM-MQ (Li et al., 2024a), in deployment-friendy: manner, showcasing
Quantization of LLMs can generally be categorized into Quantization Aware Training (QAT) and PTQ. PTQ has emerged as more practical alternative. Techniques such as Ln et al., 2023) and ZeroQuant(Yao et al.,
2

---

SIiM-LLM
Original Wyp
Gallence-Deteminee Bit Allocation
Selience-Weighted Quantizer Calibration
"Slim"
16-bit 16-bit bi 15-bit 16-bit16-bit b-bit
W, Mask,
DAi (xw" [|xw]]
Z
2ible 1-bit
3-blt 2-bi
2
Group Sailencs Sorting
L
W Mask
16-bit
91
5
3-bit
2-bit 1-bit
Figure 2. Illustration of our proposed SliM-LLM. The Salience-Determiing Bit Allocation (SBA) optimizes activation-awart structured precision, optimizing the global information distribution in quantization. Salience- Weighted Quantizer Calibration (SQC) detects discretely distributed salient weights, enhancing the local important information in L.L.s.
2022) introduce block-wise quantization, cost-effective grouping method that reduces hardware overhcad. Fur- ther advancements, including AWQ (Lin et al., 2023) and OWQ (Lee et al., 2023), apply scaling transformations to outlier weight channels, preserving their representational capacity. GPTQ (Frantar et al., 2022) minimizes group quantization errors using Hessian-baas based error compensa- tion (Frantar & Alistarh, 2022), achieving notable perfor- mance at 3-bit quantization. OmniQuant (Shao et al., 2023) introduces learnalle scaling quantizer to mitigate quanti- zation errors in an output- aware manner. For ultra-low bit- widths quantization, approaches such as QulP (Chee et al., 2024), QuIPH(Tseng al., 2024), and AQLM(Egiazarian al., 2024) improve 2-bit quantization performance through matrix decomposition with learnable codebooks and fine- tuning. Recent works (Qin al., 2024; Liao & Monz, 2024; Dettmers et al., 2024; Guo et al., 2023) have further refined quantization techniques by leveraging parameter- efficient fine-tuning (PEFT), enabling enhanced perfor- mance through additional parameter learning.
grouping with element-wise mixed-precision for more rate weight quantization. Despite their improvements, these low-bit approaches rely heavily on fine-graied partition- ing, which imposes significant challenges for real hardware deployment and inference speed.
3. SliM-LLM
This section introduces structured mixed-precision quanti- zation method, SliM-LLM, to overcome the accuracy and efficiency bottlenecks of miixdd-preciste LLMs. We de- vise two novel strategies, including the use of Salience- Determined Bit Allocation (SBA) based on global salience distribution determine group bit-widths, and Salience- Weighted Quantizer Calibration (SQC) enhance the per- ception of locally important weight information. We intro- duce SBA and SQC in Sec. 3.2 and Sec. 3.3, respectively.
3.1. Preliminaries
Quantization Framework. We first present the general uni- form quantization process of LLMs according to common practice (Liu et al., 2023; Shao et al., 2023; Achiam et al., 2023). The quantization process requires mapping float- point weights distributed within the interval [Wmin; Wmax] to an integer range of 2 where N is the target bit-widths. The quantization function for weight wy R=xm follows:
Mixed-Precision Quantization leverages the varying im- portance and redundancy of model parameters by assigning different bit-widths to each component. In traditional vi- sual networks, HAWQ V2 (Dong et al., 2020) and HAWQ V3 (Yao et al., 2021) optimize bit-widths allocation on a layer-wise basis using Hessian analysis and Integer Linear Programming (ILP). Similarly, OMPQ (Ma et al., 2023) employs network orthogonality instead of Hessian for bit- widths optimization. For large language models (LLMs), APTQ (Guan et al., 2024) extends HAWQ's approach by allocating mixed bit-widths to transformer blocks based on Hessian trace, achieving improved accuracy for 3-bit quantization. However, block-wise or layer-wise mixed- precision strategies at 2-bit fail to maintain performance after compression. To address this, recent methods such as SpQR (Dettmers al., 2023), PB-LLM (Shang et al., 2023), and LLM-MQ (Li et al., 2023) adopt imer-grained
w. clamp(L
Wi
+ 2, 2, 2 1),
8
(I)
W Wmax
Wmin
Wmin
2.V
where w. indicates quantized weight which is integer, 1-1 is round operation and clamp(-) constrains the value within integer range (e.g. [0, 1,2, 3), N = 2). A is scale factor and quantization zero point, respectively. When converted
3

---

SliM-LLM
to I-bit quantization, the calculation follows:
< - sign( (y), a = 5lle if 2 0,
(2)
(w)
others.
where W6 binary result. a denots binarization scales and the number of elements in weight (Qin al., 2023), used for dequantization through awx, We can formalize the per- layer loss in PTQ. following the common practice (Nagel et al., 2020; Frantar et al., 2022):
(a) Salience of layer-2-out
(b) Salience of layen-2-Down
((w,) = [m - xw ze t ww)
(3)
where æ E Rtxm denotes the input vectors from cal- ibration dataset, w E Raxm is dequantized weight from quantization result in Eq. (I) or Eq. (2), and H
(c) Salience of layer-10-Oa Out (d) Salience of layen- 10. 10-Down
Figure 3. Salience weight distribution in layer-2 and layer- of LLaMA-7B.
7 EE xM 2. is proxy Hessian matrix by Levenberg- Marquardt approximation (Marquardt, 1963; Frantar & Al- istarh, 2022) from set of input activations.
Determined Bit Allocation (SBA) technique to allocate the optimal precision to each group.
3.2.1. SPATIAL DISTRIBUTION OF GLOBAL SALIENCE
Parameter Salience. In LLMs, the importance of each element in the weight matrix is various (Dettmers et al., 2023; Frantar & Alistarh, 2023). According to Eq. (3), quantizing different elements causes different impacts on the model's 's output loss. Elements that significantly influence the loss termed salient weights. Consequently, follow the SparseGPT (Frantar & Alistarh, 2023) to define the salience of each element as:
We first conduct an empirical investigation into the weight salience distribution. The results reveal that certain channels exhibit higher salience and show tendencies to spatial clus- tering. As illustrated in Fig. 3, salient clustering is identified around the 2100*h, 3218th and 3853rd channels within the 2nd layer's 's attention projection of the LLaMA-7B model. A similar structured pattern is observed in other layers. Also, clustered salience is detected in other layers (as shown in Fig. 3). More examples are provided in Appendix G.
Definition 3.1. In the quadratic approximation of the loss as expressed in Eq. (3), we give the Hessian matrix H E mm generated by p ER=n [x) 2/ for a weight matrix, the removal of the element at ji induces an error or,j = wf
Then, we analyze the underlying reasons for this phe- nomenon. According to Definition 3. salience of weights is proportional to the magnitude of the weights and the trace of the Hessian matrix, which can be approxi- mated by the product of input activations x æ. LLMs, activations exhibit extreme outlier channels, while the nu- merical differences in weights are relatively slight (Xiao et al., 2023a; Nrusimha et al., 2024). Therefore, we pro- pose an analysis of how the outlier channels in activations influence the distribution of weight salience:
to the output matrix for linear projection in LLMs. "..
where [H 'ن3 denotes the jth diagonal entry for the inverse Hessian, and H-1 can be efficiently calcu- lated through Cholesky decomposition (Krishnamoorthy & Menon, 2013). According to Definition. 3.1, we map the elimination error dij to the salience measure of each weight element in LLMs, representing the impact of differ- ent weights on the output loss and the language capabilities, which also leads the generation of mixed- mixxedpreccision quanti- zation strategies (Dettmers et al., 2023; Shang et al., 2023; Huang et al., 2024a; Li et al., 2024a) for LLMs. However, existing mixed -recisspe solutions require the discrete allo- cation of bit-widths across the entire weight matrix, which imposes a significant burden on hardware computations, thereby affecting the inference efficiency.
Theorem 1. Given the input calibration activation x R xm with an outlier channel "t,,, >> æ..j. Vj [0, m].j # p at the position of channel-p. The trace elements of H x: x. will show great outlier value at (p.p), where H p.P. > H Hj E [0,m],j # p. as H p.p is produced by " * " -= Et xx2 which further leads to the pa- rameter salience larger at the tt channel of weight, where w² S.p > ,, T
Vk E [0, 1
3.2. Determinndd Bit Allocation
Theorem I elucidates the influence of outlier activation on the distribution of channel salient weights (detailed proof in the Appendix G. Furthermore, recent research indicates that outlier channels in LLMSs activations consistently appear
We reveal the spatial clustering of weight salience, which in- spires our proposed concept of group-wise mixed precisson quantization for L.Mss, and then introduce the Salience-
4

---

SliM-LLM
in fixed yet clustered patterns (Nrusimha al., 2024). Ac- cording Theorem 1, these consistently occurring anoma- lous activations result in the distribution of salient weights, as depicted in Fig. 3. Then, during group-wise quantization, the average salience of each group shows different features.
Group
Group
Local Salience
Soleet Migg
Meanwhile, previous unstructured mixed-preee precciioe,ssnon in- curred additional storage requirements and computational overheads, affecting the real-time inference. However, the strong spatial structured characteristics observed in the salient of weights in this section strongly inspire to first develop a group-wise mixed-preee precision strategy within the weight matrix while maintaining inference efficiency. There- fore, we aim to allocate bit-widths based on intra-group salient disparities, which not only enhances quantization accuracy but also ensures the inference efficiency of LLMs with structured bitts saving and dequantization.
Figure 4. Local salience distribution of the 10th MHA output layer.
size is 128, k = m
128 the search region for weight is limited
to [0, $). which is highly efficient with limited searching space, e.g., only 16 iterations are needed in LLaMA-7B. We also provide detailed searching error examples in Ap- pendix C. Notably, SBA diverges from traditional quantiza- tion with mean squared error (MSE) in Eq. (3) by instead utilizing the KL divergence as its measure of loss. Com- pared to using mean squared error (MSE) for weights, SBA utilizes the KL divergence of block outputs as a precision allocation metric, aiming to align the distribution of the LLM's output activation matrix with that of the quantized activation. This approach improves the model's information representation under low-bit quantization, enabling more effective bit-widths allocation. While HAWQ v2 (Dong et al., 2019) employs Integer Linear Programming (ILP) to allocate bit-widths for layers, its method can be adapted to group-wise targets. Detailed comparisons between SBA and ILP are provided in Section 4.2.
3.2.2. GROUP-WISE BIT ALLOCATION
To allocate optimal bit-widths to each group, we introduce a Salience-Determmined Bit Allocation (SBA) technique for mixed-precision LLMs, as depicted in Fig. 2. This tech- nique. predicated on the differences in group salience, deter- mines the optimal bit-widths allocation for different groups by minimizing the distance of information entropy with the original weight output. Specifically, we first utilize the av- erage salience as the importance indicator for each weight group and rank them accordingly. Then, the proposed SBA optimizes the bit allocation following:
Objective argmin D&I ( || x ( (ba) ),
where Vzba = |200,bo 10,,, ... -1.b-1b-11 wk.be
Constrain =| = 1 where GN-1 = [bilb, -N -11,,
3.3. Salience- Salience-Weighted Quantizer Calibration
In addition to the global group-wise distribution of salience, we notice that salience within the group still shows local differences in discrete distribution. Common existing quan- tizers apply uniform consideration across all weights to minimize the effect (error) of quantization, lacking the ca- pability to perceive differences in local salience. Therefore, in this section, we introduce a Salience- Salience-Weighted Quantizer Calibration (SQC) to enhance the information of signifi- cant weights within the group by amplifying the quantizer awareness of salient weight.
9N+1= = (bblb, = N +1}
(4)
where Dat(-Il)) denotes the Kullback-eebler (KL) divergence between two outputs, ú zba generally represents the de-quantization results of weight, em- ploying group-wise mixed-precision designated (u -......00s-.., L, where b, represents the bit-widths for the 7th group and G is set of groups with the same bit-width, N is the targeted average bit-width. We apply a compensation constraints strategy to maintain a consistent average bits for our SBA. For example, in 2-bit quantization, the more salient groups are quantized to 3-bit. To offset the additional bits, we quantize equal number of groups with the lower salience to 1-bit (1 = |9N+1|) while the remaining groups are set to 2-bit.
1. DISCRETE DISTRIBUTION OF LOCAL SALIENCE
In the aforementioned section, we group- wisely allocate the bit-widths for each group based on the global salience. To maintain the efficiency of quantized inference, we employ commonly used sequential structured grouping (Frantar et al., 2022; Lin et al., 2023; Shao et al., 2023). However, this group-wise mixed-precision also leads to differences in salience among the various elements within the same group. Specifically, as the salience distribution in Fig. 4, within
We utilize an effective double-pointer search (more de- tailed examples in Appendix C) to optimize our objective in Eq. (4). When the weight output channel size m and group
5

---

SHiM-LLM
the 10th attention output layer of LLaMA-7B, subset of sparse weights within the comparatively less salient Group-2 (Fig. 4) still maintains a high level of importance. In LLMs, a small number of weight elements with outliers affect the local distribution of salience. These discrete weights typi- cally account for only approximately 1% within the group but play a crucial role the modeling capability of LLMs.
over, the SQC process allows t and whe to share a set
of parameters r" and r"2, eliminating the need to differ-
entiate intra-group weights during storage and inference. This facilitates straightforward group-wise dequantization calculations, thereby avoiding the hardware overhead asso- ciated with element-wise bitmap and unstructured grouping. SQC and SBA capture both local salient weight informa- tion within groups and global weight combinations across groups, effectively preserving critical information and main- taining LLM performance at extremely low bit-widths.
Vanilla quantizers struggle to represent local salient infor- mation they only consider the mean error of all elements within group. During group-wise quantization (Eq. 1), non-salient weights often dominate, degrading critical infor- mation and negatively impacting LLM performance.
3.4. Implementation Pipeline of SliM-LLM
We integrate our mixed-precision framework into advanced PTQ methods, such as GPTQ (Frantar et al., 2022) and OmniQuant (Shao et al., 2023), all of which are inference- friendly with group-wise quantization. We primarily in- tegrate SBA and SQC into GPTQ to get SliM-LLM. For SliM-LLM+, the SBA is plugged into OmniQuan with a learnable quantizer. The plugging pipeline of SIiM-LLM is provided in Algorithm 1 (line 4 and line 9), detailed func- tions are shown in Algorithm 2.
3.3.2. SALIENCE- WEIGHTED QUANTIZER
To prevent the degradation of local salient information in each group, we propose the Salience- Salience-Weighted Quantizer Calibration (SQC), which enhances the expression of salient weights through local salience awareness, thereby reducing the quantization eгror of these significant elements and im- proving the compressed performance of L.MSs.
Based on a common observation (Dettmers et al., 2023; Huang et al., 2024a), the proportion of relatively salient em-
4. Experiments
weights in each group is only 1-5%. Therefore, play the 3-o rule for a mask to select the salience part (w < (u - 30r U w >> (u+ + )) in each group (Fig. 2), which accounts for about 1% elements. After the selection, we get w; = w U w where w is the salient part and w0t represents the non-salient elements within group i. To effectively keep the information of local salient weights, SQC first introduces the calibration parameter 7 to the SQC quantizer, liberating the perception interval during quanti- zation. Then we define the local salience awareness loss of the SQC quantizer through calibration:
We evaluated SliM-LLM and SliM-LLM* under weight- only conditions, focusing on 2/3-bit precisions. Per- channel group quantization is utilized in our framework with groupsize = 128 in experiments. Since no back- propagation in SliM-LLM, the quantization is carried out on single NVIDIA A800 GPU. For SliM-LLM+, we em- ploy the Adam optimizer, following OmniQuant (Shao et al., 2023), which is also feasible on a single A800. We randomly select 128 samples from WikiText2 (Merity et al., 2016) as calibration data, each with 2048 tokens.
argmin
| - T (Qw T S, 2) - T +
Models and Evaluation. To comprehensively demonstrate the low-bit performance advantages of SliM-LLM and SliM- LLM we conduct experiments across OPT (Zhang et al., 2022), LLaMA (Touvron et al., 2023a), II ..M (Touvron et al., 2023b) and LLaMA-3. We employ the perplexity as our evaluation metric, which is widely recognized as a stable measure of language generation capabilities (Frantar et al., 2022; Lin et al., 2023; Huang et al., 2024a; Shang et al.,
(5)
||w N T T - (Q@(w)
S, 2) - 2
where Q(-) denotes the quantization process in Eq. (1)
represents the &g loss, aligned with Eq. (3). w and uo de- notes the salient and less salient part of group i, respectively, generated from mask operation. In Eq. (5), 7 expands the solution space of s and z, flexibly adjusts s and 2 to search the optimal loss under r", without bringing additional pa- rameters, as w and w. share the same quantizer. The search space for t by linearly dividing the interval [1->, 1+A] into 2n candidates. We empirically set À at 0.1 and n at 50 to achieve balance between efficiency and accuracy.
2023; Shao et al., 2023; Chee et al., 2024; Egiazarian et al., 2024; Huang et al., 2024b), particularly in compression sce- narios. Experiments are carried out on the WikiTex12 (Mer- ity et al., 2016) and C4 (Raffel et al., dataeessssets. Fur- thermore, to assess the practical application capabilities of quantized LLMs, we also evaluate on zero-shot benchmarks.
Compared to traditional quantizer calibration methods, SQC effectively mitigates the degradation of intra-groud local salient weights caused by general average loss by enhancing the loss sensitivity to salient elements during the calibra- tion (more experiments are detailed in Appendix E). More-
4.1. Main Results
We show experiments within the LLaMA family in this section and results for the OPT models are available in
6

---

SliM-LLM
Table Quantization results of LLaMA family with statistic quantizer. We report the WikiText2 perplexity in this table, C4 results are shown in Appendix H. denotes that the selected works did not give the results listed models the codes.
PPI.
Method
1-7B
1-138
1-30B
1-65B
2-7B
2-13B
2-70B
3-8B
3-70B
16-bit
5.68
5.09
4.10
3.53
5.47
4.88
3.31
5.75
2.9
APTQ
6.76
I.M-MQQ
8.54
RTN
4.87
4.24
6.66
5.51
3.97
27.91
11.84
3-bit
AWQ
6.46
5.51
4.63
3.99
6.24
5.32
8.22
4.81
GPTQ
6.55
5.62
4.80
4.17
6.29
5.42
3.85
8,19
5,22
SIiM-LLM
6.40
5.48
4.61
3.99
6.24
5.26
3.67
7.16
4.08
LLM-MQ
12.17
RTN
1.9e3
781.20
68.04
15.08
4.2e3
122.08
27.27
1.9e3
4.6e5
AWQ
2.6e5
2.8e5
2.4e5
7.4e4
2.2e5
1.2e5
1.7e6
1.7e6
2-bit
GPTQ
152.31
20.44
13.01
9.51
60.45
28.14
8,78
210.00
11.90
QulP
29.74
12.48
11.57
7.83
39.73
13,48
6,64
84.97
13.03
PB LLM
24.61
17.73
12.65
7.85
25.37
49.81
NAN
44.12
11.68
SliM-LLM 14.58
8.87
7.33
5.90
16.01
9.41
6.28
39.66
9.46
Table Quantization results of LLaMA-1 and LLaMA-2 models with learnable quantizer. We report the WikiTex12 perplexity in this
Table, C4 results are shown in Appendix
#W PPL
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
16-bit
5.68
5.09
4.10
3.53
5.47
4.88
3.31
OmniQuant
6.15
5.44
4.56
3.94
6.03
5.28
3.78
3-bit
AffineQuant
6.14
5.45
4.59
5.28
SliM-LM
6.07
5.37
4.34
3.72
5.94
5.11
3.35
OmniQuant
9.72
7.93
7.12
5.95
11.06
8.26
6.55
2-bit
AffineQuant
13.51
7.22
6.49
10.87
7.64
SliM-LLM
9.68
7.17
6.41
5.74
10.87
7.59
6.44
Appendix H. For language generation tasks, depicted in Tab. 1, SliM-LLM markedly outperforms its backbone GPTQ. particularly under the b-bit. Specifically, on LLaMA- 7B, SliM-LLMM achieves 90% decrease in perplexity, while on LLaMA-3-8B, it improves by 81%. In comparison with the element-wise mixed mixed-precision PB-LLM and the codebook-band based QulP method, SliM-I.LM further reduces the perplexity by 41 1%-5111 %. As shown Tab. the per- formance of S is still ahead compared to Omni- Quant and AffineQuant. We also provide dialogue examples of 2-bit instruction fine-tuning Vicuna-13B (Chiang et al., 2023) and LLaMA-133 in Appeandix Our method demon- strates zero-shot advantages at 2-bit, as shown in Tab. 3, where SliM-LLM and + outperform other meth- ods. For example, compared to GPTQ and OmniQuant, approach improves LLaMA-7B performance by 4. 19% and 1.91% average. On LLaMA-65B, 2-bit SliM-LLM and SliM-LLM+ achieve accuracy within 6% of FP16. To further showcase the general performance of SL,M-LMM, we also compare the low-bit results on multi-modal models (Tab. 15), where SLiM-LLM presents leading accuracy on 4 benchmarks.
Rastion Haad-Tal (SBA+500)
500
(SBA/80C)
5.7h Model (Parato)
Mocel Sioe (Pararr)) Compenent
Figure 5. Ahlation results OPT models. Random refers domly selecting lower- and higher- bit groups, while head-tail as- signs lower- bit precision the head groups and higher bit preci- sion equal number of tail groups in the original sequence.
the impact of each component. Fig. 5(a) compares three strategies for allocating bit-widths across groups, including random allocation, head-tail allocation by spatial order, and our proposed SBA. When the average bit-widths remains constant, random and head-tail mixed-precisiog allocation prove ineffective and even result in performance degrada- tion, as shown in Fig. 5(a). In contrast, SBA consistently delivers significant improvements post-quantization per- formance, validating the efficacy of our mixed-precision precision approach. Fig. 5(b) presents the ablation effects of SBA and SQC, demonstrating that both methods, based on the per- ception of global and local salience, enhance quantization performance.
4.2. Ablation Results
Abliation of and SQC. We conduct a detailed ablation study illustrate the benefits of bit- widths allocation and
Compare of SBA and ILP. We compare the performance

---

SIiM-LLM
Table 3. Performance comparisons of different quantization methods for zero-shol tasks.
Model Acct
#W
Method
PIQA
ARC-
ARC-c
BoolQ
HellaSwag
Winogrande
Avg.
16-bit
77.47
52.48
41.46
73.08
73.00
67.07
64.09
2-bit
GPTQ
55.49
31.02
22.17
53.49
33.84
41.91
39.65
2-bit
AWQ
47.78
28.77
21.31
31.19
24.47
40.03
32.26
LLMMA-7B
2-bit
SliM-LLM
57.83
33.46
25.09
56.05
36.70
52.64
43.84
2-bit
OmnnQuaant
63.63
43.91
27.32
58.02
48.78
52.97
49.1
2-bit
SliM-LLM
64.96
45.66
28.67
64.59
48.86
53.35
51.02
16-bit
79.10
59.89
44.45
68.01
76.21
70.31
66.33
2-bit
GPTQ
70.37
47.74
35.88
51,37
61.39
60.84
54.63
2-bit
AWQ
49.23
30.01
29.49
30.88
26.72
46.30
35.44
LLaMA-13B
2-bit
SliM-LLM
73.19
47.95
36.27
55.92
63.04
61.79
56.36
2-bit
OmmiQuunnt
73.14
49.38
36.93
63.34
62.79
61.77
57.64
2-bit
SliM-LLM"
74.15
50.26
37.04
64.31
63.57
63.11
58.74
16-bit
80.08
58.92
45.47
68.44
79.21
72.53
67.44
2-bit
GPTO
7132
48.27
36,20
61.27
65.76
63.7
57.76
2-bit
AWQ
49.17
28.56
25.97
34.73
24.97
46.99
35.07
LLaMA-30B
2-bit
SliM-LLM
75.52
51.29
39.29
62.01
66.10
64.07
59.71
2-bit
OmmiQuant
76.23
53.23
39.52
63.34
65.57
64.82
60.22
2-bit
SliM-LLM
76.31
54.07
39.79
63.35
67.14
64.93
60.91
16-bit
80.79
58.71
46.24
82,29
80.72
77.50
71.04
2-hit
GPTT
76.16
52.48
30.14
77.23
71.96
70.22
64.70
2-bit
SliM-LLM
77.09
53.72
40.25
77.51
72.05
70.91
65.26
LLaMA-65B
2-bit
OmniQuant
77.78
53.71
40.90
78.04
74.55
68.85
65.64
2-bit
SliM-LLM
78.06
53.90
41.18
78.33
75.59
69.99
66.18
Table WikiTex12,, performance SBA on LLaMA.
we evaluate the deployment performance of LLaMA-7/13B and LLaMA-2-7B under 2/3-bit settings. The results indi- cate that our mixed- sexeedprreciion approach maintains a good compression rate GPUs and significantly enhances model aceuracy, only with slight decrease in inference speed on the A800. We also deploy the LLaMA-2-7000 in Tab. 14. Since current 1 1-bit operations lack well hardware support, additional consumption of storage and computation is re- quired on device. There remains considerable scope for optimization in mixed-precision computing, and we aim to further improve this future work.
Method
#W
7B
13B
30B
65B
ILP
2-bit
17.55
9.51
9.27
7.46
SBA
2-bit
14.58
8.87
7.33
5.90
Table Deployment results of GPTQ and Slim-LLM on GPU. Group size set 128.
AW
LLaMA
1-78
1-13B
WM
RM
PPL.
Tokeals
WM
RM
PPL.)
Tokeak
FP16
12.60
14.4G
5.68
69.2
24.3G
27.1G
5.00
52.5
3-bii
SiM--LLM
GPTO
3.2G
3.2G
5.1G
5.2G
6.55
6.40
83.4
79.1
3.80
5.8G
8.8G
8.7G
5,48
5.62
51.6
48.5
2-bit
GPTQ
220
4.40
152.31
83.9
4.00
7.60
20.44
92.6
5. Conclusion
SIM--1I
233
444
14,58
61.
4.1G
788
8,87
73.7
between the ILP model in HAWQ v2 (Dong et al., 2019) and SBA on the LLaMA model. Tab. 4 shows that SBA achieves comprehensive performance superiority on LLaMA. served that under 2-bit scenario, ILP ensures an equal num- ber of 1-bit and 3-bit groups within the search space { (l-bit, 2-bit, 3-bit]. The advantage of ILP lies in broader selec- tion range for target bit-ws but under commonly used fixed integer bit-widths (e.g. 2-bit, 3-bit), SBA's double- pointer search strategy based on output feature KL proposed by SBA can achieve a more optimal matching strategy.
In this work, we propose SliM-LLM, a group-wise mixed- precision PTQ framework for L.LMs, designed to optimize performance with low-bit weights while maintaining de- ployment efficiency. The core of SliM-LLM is the Salience- Determined Bit Allocation, which dynamically assigns bit widths to preserve global salience information. Additionally, the Salience- Weighted Quantizer Calibration enhances local information retention, reducing the impact of quantization on locally salient weights. Experimental results demonstrate that SliM-LLM significantly improves accuracy across var- ious LLMs while ensuring inference efficiency. Overall, SliM-LLM is versatile solution that integrates seamlessly with existing quantization frameworks, enabling practical deployment in resource-constraines environments.
4.3. Efficient Inference on Device
We utilize the open-source AutoGPTQ to extend CUDA kernel supporting experimental preccisnn inference, with detailed process Appendix B.2. As shown in Tab. 5,
8

---

SUiM-LLM
Acknowledgments
evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024.
The authors acknowledge the Innovation and Technol-
ogy Fund from Innovation and Technology Commis- sion of Hong Kong SAR Government (Mainland-Hong Kong Joint Funding Scheme MHP/053/21), Shenzhen- Hong Kong-Macau Technology Research Programme SGGDX20210822103557703 from Shenzhen Science and Technology Innovation Commission, and Seed Funding for Strategic Interdiseiplinary Research Scheme from the Uni- versity of Hong Kong (HKU) for supporting this work. This work has also been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422, 17212923), the Swiss National Science Foundation (SNSF) project 200021E_21111 21992229933 Neuromorphic Attention Models for Event Data (NAMED)
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Chiang, W.-L.., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing GPT-4 with 90%* * chatgpt quality. See https://vccua. lmsys. org (accessed 14 April 2023), 2023.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, LLM.int8 (): 8-bit matrix multiplication for transformers scale. arXiv preprint arXiv:2208.075:: 2022.
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. SpQR: A sparse-quantized representa- tion for near-lossless LLM weight compression. arXiv preprint arXiv:2306.0300 2023.
Impact Statement
This paper introduces mixed-precision technique to achieve accurate and efficient low-bit weight quantization for large language models (LLMs). This approach makes LLMS more efficient and accessible, potentially extending their pervasive impact. From a positive perspective, quanti- zation makes the use of LLMs easier, benefiting broader audience, particularly those in lower-income groups It re- duces the cost and hardware barriers deploying LLMS and promotes edge inference of these models (mitigating the risk of privacy data breaches), contributing to societal productivity. On the downside, LLMs could be exploited by malicious users to generate and spread false information. Quantization does not prevent the inherent negative impacts of LLMs, nor does exacerbate them.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Dong. Z., Yao, Z., Gholami, A., Mahoney, M. W., and
Keutzer, K. Hawq: Hessian aware quantization neural networks with mixed sexed-precision In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 293-302. 2019.
Dong, Z., Yao, Z., Arfeen, D., Gholami, A., Mahoney, M. W., and Keutzer, K. Hawq-v2: Hessian aware trace- weighted quantization of neural networks. Advances in neural information processing systems, 33:18518-18522 2020.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.066 18,006111111011111 2024.
Anadkat, S., et al. GPT-4 technical report. arXiv preprint wwXi::2303.08774, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are sho- learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruming. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Frantar, E. and Alistarh, D. SparseGPT: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Bubeck. S.. Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, F., Kamar, F., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with GPT-4. arXiv preprint 1221v:2303.12712.222 2023.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 13X1v:2200.13333, 2022.
Chang, Y., Wang, X., Wang, J., WU, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. survey on

---

SHi-LLMM
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., and Winslett, M. Compressing large-scale transformer-based models: A case study bert. Transactions of the Association for Computational Linguistics, 9: 10101-1080, 2021.
Liao, B. and Monz, C. Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.05147, 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. AWQ: Activation weight quantization for LLM compression and acceleration. arXiv preprint arXiv:2306. 00978, 2023.
Guan, Z.,, Huang, H., Su, Y., Huang, H., Wong, N., and Yu, H. APTQ: Attention-awart post-training mixed-precision quantization for large language models. arXiv preprint arXiv:2402.14866, 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. Llava-next: Improved reasoning, oer, and world knowledge, January 2024. URL https://1Lav--vL.
Guo, H., Greengard, P., Xing, E. P., and Kim, Y. Lq- lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.1202. 2023.
io/blog/
Liu, Z., Oguz, B., Zhao, C., Chang, E.,, Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LLM- QAT: Data-Free Quantization Aware Training for Large Language Models. arXiv preprint arXiv:2305. 17888, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 0rXiv:2009.03300 2020.
Huang. W., Liu, Y., Qin, H., Li, Y., Zhang. S., Liu, X., Magno, M., and Qi, X. BiLLM: Pushing the limit of post-training quantization for 1lms. arXiv preprint arXiv:2402.0429), 2024a.
Ma, Y., Jin, T., Zheng, X., Wang, Y., Li, H., Wu, Y., Jiang, G., Zhang, W., and Ji, R. Ompq: Orthogonal mixed pre- cision quantization. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 37, pp. 9029-90299 2023.
Huang, W., Ma, X., Qin. H., Zheng, X., Lv, C., Chen, H., Luo, J., Qi, X., Liu, X., and Magno, M. How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study. arXiv preprint arXiv:2404. 14047, 2024b.
Marquardt, D. W. An algorithm for least-square, estimation of nonlinear parameters. Journal of the society for Indus- trial and Applied Mathematics, 11(2):431-44 1963.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0884/ 2016.
Krishnamoorthy, A. and Menon, D. Matrix inversion using cholesky decomposition. In 2013 signal processing: Al- gorithums, architectures, arrangements, and applications (SPA), pp. 70-72. IEEE, 2013.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post-training quantization. In International Conference
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02277 2023.
Machine Learning, pp. 7197-7206. PMLR, 2020.
Nrusimha, A., Mishra, M., Wang, N., Alistarh, D., Panda, R., and Kim, Mitigating the impact of outlier channels for language model quantization with activation regular- ization. arXiv preprint ar arXivv:2044 24360550 2024.
Li, S., Ning, X., Hong, K., Liu, T., Wang, L., Li, X., Zhong, K., Dai, G., Yang. H., and Wang. Y. LIm-mq: Mixed- precision quantization for efficient 11m deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurlPS, volume 9, 2023.
Qin, H., Zhang, M., Ding, Y., Li, A., Cai, Z., Liu, Z., Yu, F., and Liu, X. Bibench: Benchmarking and analyzing network binarization. arXiv preprint arXiv:2301.14 11233, 2023.
Li, S., Ning, X., Hong, K., Liu, T. Wang, L., Li, X., Zhong, K., Dai, G., Yang, H., and Wang, Y. LLM-MQ: Mixed-precision quantization for efficient LLM deploy- ment. In Advances in Neural Information Processing Systems (NeurIPS) ENLSP Workshop, 2024a.
Qin, H., Ma, X., Zheng, X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., and Magno, M. Accurate A-FFnnttunnnn Quantization of LLMs via Information Retention. arXiv preprint arXiv:2402.05445, 2024.
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi molii-modality vision language models. arXiv preprint arXiv:2403. 18814 2024b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text
10

---

SHi-LLLM
transformer. The Journal of Machine Learning Research, 21()):5485-5511, 2020.
Yao, Z., Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Z.. Efficient and affordable post-training quanti- zation for large-scale transformers. URL https://arxiv. org/abs/2206.01 01661 2022.
Shang, Y., Yuan, Z., Wu, Q., and Dong, Z. PB-LLM: Par- tially binarized large language models. arXiv preprint 10.00033 2023.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M.. Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint mrXiv:2205.01066 2022.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. arXiv preprint 11XXi::2308.1133377 2023.
Zhang, Y., Gong, K., Zhang, K., Li, H., Qiao, Y., Ouyang, W., and Yue, X. Meta-transformer: A uni- fied framework for multimodal learning, arXiv preprint arXiv:2307, 10802, 2023.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. //000.///99,, 2023.
Zhao, W. X., Zhou, K., Li, Tang, T., Wang, X., Hou, Y.. Min, Y. Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18221 2023.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.118000 2023.
Zhu, X., Li J.. Liu, Y., Ma. C., and Wang, W. A survey on model compression for large language models. arXiv preprint arXiv:2308.076.. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- 2023a. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
guage models. arXiv preprint arXiv:2302.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quip#: Even better LLM quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint ar0iv:2309. 1753,, 2023b.
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z.., Zhang, X., and Tian, Q Qa-lora: Quantization- aware low-rank adaptation of large language models. arXiv preprint ar arX:v:2309.1477 14717, 2023.
Yao, Z., Dong, 7., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., Mahoney, M., et al. H3wq-v3: Dyadic neural network quantization. In Inter- national Conference on Machine Learning, pp. 11875- 11886. PMLR, 2021.

---

SliM-LLM
A. Limitations
In the mixed-precision deployment of AutoGPTQ. the weight memory layout is organized by group, with each group sharing the same precision, which is shown in Fig. 6. Within each group, elements with the same precision are packed as integers, eliminating the need for additional padding, which saves space. Given that the bit-widths of integers is power of 2, this is compatible with group size that is also a power of 2. For instance, even with the odd- bit such 3-bit storage, integers can store these numbers without padding, as the commonly used group size is 128, a multiple of almost all definition of integer type. This ensures that elements within a group fully utilize the space provided by integers, without storing numbers of different precision within the same integer. Sint. follow the original logic of AutoGPTQ but are packed with a uniform precision along the channel direction for ease of use. Other tensors, like scales, remain in the same sooating-poin format to ensure the correctness of dequantization calculations.
Though the mixed-precision framework significantly im- proves the quantization performance of LLMs, the current out-of-the-box deployment tools still cannot well support efficient mixxed-precisionn computing. Meanwhile, the sup- port for 1/2/3-bit inference on GPUs remains limited, which affects the inferencing advantages of low-bit models. We believe there is significant room for improvement in the hardware efficiency of mixed preecisineiiioon LLMs in the future.
B. SliM-LLM Implementation
B.1. Detailed Implementation
In this section, present the specific implementation de- tails of SliM-LLM, which utilizes GPTQ (Frantar et al., 2022) as its backbone for -ixeed-reecsion quantization and incorporates both SBA and SQC. SliM-LLM* is consistent with SliM-LLM in SBA computations but does not include the SQC component, instead retaining learnable weight clip- ping (LWC) approach in OmniQuant (Shao et al., 2023) for gradient optimization.
To indicate the precision of each group, we also introduce an additional array to store bit-widths of each group, where each number is represented as a 2-bit value aggregated into integers, marking the quantization precision of each group for accurate reconstruction. We use cumulative calculations to determine the starting index of each group, ensuring cor- rectness despite changes in Wint height and starting indices caused by varying precision. Using the above methods to store the quantized weights, zeros, and additional bit arrays effectively reduces memory usage during model storage and loading, thereby lowering the resource overhead required for model deployment.
Algorithm 2 primarily encompasses the core details of both SBA and SQC. In SBA, the importance of each group is determined by sorting the average salience of groups, fol- lowed by a bi-pointer search that increases the number of (N 1)-bit and (N + 1)-bit groups to maintain their quan- tity equilibrium. The optimization function then utilizes the KL divergence from Eq. (4) to determine the optimal mixed-precision ratio. SQC, on the other hand, enhances its information by amplifying the quantization error of unstruc- tured weight groups. When the last two parameters, scale and zero point, in the fakequant()) function are omitted, the default values from Eq. (1) are used.
Once the weights are packed, we follow the modified Au- toGPTQ logic for GPU inference. The GPU processes and dequantizes the weights group by group for computation. During GPU computation, a thread dequantizes a segment of continuous memory data in one column of Wint and performs vector dot product calculations with the input ac- tivation shared within the block, accumulating the results in the corresponding result matrix. When threads form a logical block, the block handles the computation and reduc- tion of a continuous channel region. We complete the linear layer computation by iterating through all logical blocks. Leveraging AutoGPTQ's initial logic and CUDA Warp's s 22-thread units, we ensure similar code structure and data access logic for threads within each warp when group size is 128. This method was primarily conducted to validate feasi- bility os SliM-ILLM, demonstrating that the mixed precision quantization with integer packing does not cause additional computational overhead, indicating the efficiency and ac- curacy advantage of SliM-LLM. In summary, by dividing weight into several structured groups with mixed precision and employing reasonable GPU utilization strategy, Slim- LLM balances performance and efficiency.
B.2. Mixed Bit Storage and Computing
We developed a framework for storage and inference de- ployment supporting sixeed-precsion quantization based on AutoGPTQ. The deployment process is as follows. After completing mixed-precision quantization with SliM-LLM, it outputs scales, zeros, and group- wise bit-widths generated during the quantization process to identify the quantization parameters and precision of each group in the Linear Projec- tion weights. AutoGPTQ then packs the weights and zeros into integer-compressed representations (denoted by wint and int respectively) based on the precision of different groups, significantly reducing storage and operational bit- width. After the quantized weights are packed, AutoGPTQ loads the model onto the GPU, where the mixed precision quantization kernel on the GPU performs dequantization on the weights and zeros of different groups and calculation with input activation, ultimately producing the final output.
12

---

SIiM-LLM
Algorithm 1 Main Framework of SliM-LLM.
func SlMM-LLM( FF, B, A, N)
9(-) == SwA F. Hin ,B,N) N)
Input: E -axm -P16 weight
for b 0, do
2p Rtxm_ calibration data
w = w.bbb
3 group size
7: Jb = Gl
A hessian regularizer
8:
w w == sal mask(ur)
N average bit-width
9: i = SQC( w 96)
Output: Wg- quantized weight
10: GPrQ-error compensation:
11: E = (wbbb w H 98-8
[+] 14T hessian matrix 1: H = 7 k= F F
12: W..+BB: = w.b+8. E. H bbb+9
2: H in = Cholesky (H++M
13: end for
3: W 0m
14: return w.
Algorithm 2 Detailed functions in SliM-LLM. func SBA(w, ær, H ,,,,N,
func SQC(w,, wi. 9h)
1: - {0} // initialize group bit-width
1: Wmax = max(w, U w
2: e = inf // bit-widths searching error
2: Wmin min(w U un
3: p" == /// number of (NV-1)-bit and (N++))-bit
3: À == 0.1
4: == N - -11/ // lower bit-width
4: n = 50
5: h == N + // higher bit-width
5: e = inf // scale searching error
6: S(.) average( 2
6: A E <Rax1 // per-channel scale
GLAB
7: =* E RTx1 per-channel zero point
7: for p- 1,
do
for T E [1 - X, + X] with 2n slices do
8:
w fakequant Setop k min(p): 4)
A = 7 (Wmax Wmin (2 1) = [[TWmin) A
9:
W = fakequant b max(p) h,
bb
10:
10:
b fakequant wwwaantttwweeerr beothers: N,)
11:
W b = fakequant 9b, A, 2)
C
11:
in = . U U U wh
12:
w us = fakequant w us 9b, A 2)
13:
L. = w i
12:
if 'DD (xw" T || ww
y < then
14:
Lus = w us w us
e Dai (xw xú,
15:
if C, + Lus < e then
14:
p==
16:
= C, + Lus
15: end if
17:
2"
16: end for
18:
A = A
17: G(I) = S(top_k_mmnff = 13
19: end if
18: G(h} = Sttop__ k max(p") = h}
20: end for
19: G[N]= = (midddl k([m] - 2p') = N}
21: w. = fakequant 9b. 2²,2"
20: return g(-)
22: return "
C. Searching Details of Group-Wise Salience-Determined Bit Allocation
tively balancing the global average bit-widths compensation. We then calculate the relative entropy under the correspond- ing precision ratio and search for the optimal ratio. Fig 7 displays the search error curves related to the 2nd th and 15th Transformer layers in the OPT1.3B model, showcasing the search curves for certain self-attention layers (Query, Key, Value, FC2).
We optimize the mixed-precision precision configuration based the output information entropy (KL-divergence), searching for the optimal compensation bit-widths ratio as shown in Eq. (4).
Initially, we rank each group by their average salience, met- ric for quantization, and employ double-pointer that moves simultaneously from both the beginning (lowest salience) and end (highest salience) of the sorted list. This ensures equal number of groups at low and high bit-widths, effec-
Due the limited range of the search, extreme scenarios involve either half (N 1)-bit and half (N 1)- with- out Nibit or all groups being N-bit (uniform precision). Fig 7 demonstrates that lower quantization errors can be
13

---

SIiM-LLM
FP16 to 4bit
group1
Int-4bit
group1
quantize
FP16 to 2bit
group2
Int-2bit
group2
pack
FP16 to 3bit
group3
Int-3bit
group3
W
Wint
0.2
0.22
32bit
rows
0..3
FP16-2bit
quantize
0
Int-2bit
0.6
FP16-3bit
pack
0
0
0
Int-3bit
0.4
rows
0.5
0.6
Figure 6. The memory layout shown in the figure is madified based on AutoGPTQ. The transposed original weights w E Rm are still divided into multiple groups along row direction after quantization. The elements within each group are vertically packed into integers and then reassembled into Wint The figure employs corresponding colors to indicate how each original number is mapped to specific position within the packed integers after quantization, which finally generates Wint E R. x where m compressed from m by packing several low-bit number. Similarly, Zin is also packed into integers to save memory.
achieved under mixxedprecision compared to quantization at the uniform bit-width. We also find that multiple low- error precision combinations are possible within group of weights, allowing SBA to flexibly select the optimal ratio through its versatile search.
0.0055, while with SQC it was reduced 0.0039. This further demonstrates that the search parameter as applied in Eq. (5), effectively optimizes the quantizer parameters, thereby reducing quantization errors.
More importantly, SQC effectively perceives the informa- tion of locally salient weights, indicated by the red re- gions in Fig. 8. Compared to the vanilla quantizer, SQC significantly reduces the error of salient weights. Specif- ically, the prominent weights at indices 375 in Fig. 8(a) show higher quantization errors, while in Fig. 8(b), this er- ror is effectively reduced. This confirms SQC's ability to perceive locally salient weights, effectively preventing the degradation of critical information.
D. Evluatiion Function of SBA
In Tab. 6, employ various objective functions and com- pare their performance in SBA across different models. Compared to the commonly used Mean Squared Error (MSE) loss, Kullback-Leibler (KI.) divergence ensures the distribution of critical activation positions within the model from an information entropy perspective, making a supe- rior choice for the bit-widths allocation strategy SBA for the OPT and LLaMA models. When computing KL diver- gence in this context, we first transform the layer outputs into probability distributions using softmax.
F. Extension Ablation on Quantization Group-Size
investigate the impact of different group sizes on quantization effectiveness of SliM-LLM, we evaluated per- formance with 256 and 512 columns at 3-bit level, observ- ing that larger group sizes enhance GPU efficiency during inference. The findings suggest that increased group granu- larity does not substantially elevate perplexity across four models, indicating that SliM-LLM is robust and conducive to more efficient deployment methods. In contrast, at 2-bit, we assessed group sizes of 64 and 32 columns. With finer group granularity, the models displayed reduced perplex-
E. Extension Ablation on SQC
In this section, we visualize the effectiveness of SQC in mit- igating the degradation of information in locally salient weights. We observed the absolute error of weights in randomly selected channel of the quantized OPT-1.3B model. As shown in Fig. 8, the overall absolute error of the weights post-quantization with a standard quantizer was
14

---

SIMM-LLM
Query
Layer-5
Query
Out
FC2
(b) Layer-10
Query
Key
Out
FC2
(c) Layer-15
Figure 7. Error of SBA for select weights in the 5th 10th, and 15th layers of OPT-1.3B.
Table 6. Comparison of MSE and KI. Divergence in SBA.
Method
#
OPT-1.3B
OPT-2.7B
OPT-6.7B
ОРT-13B
LLaMA-7B
LLaMA2-7B
MSE
2-bit 2-bit
32.50
27.58
15.14
13.28 10.12
21.94 14.58
16.86
KL Divergence
30.71
13.26
11.27
16.01
Table Ablation results OPT-6.7B, LLaMA-7B, LLaMA-2-7B. LLaMA-3-8B with SliM-LLM under different group size (#ig denotes the group size).
Precision / PPL.
#g
OPT-6.7B
LLaMA-7B LLaMA-2-7B
LLaMA-3-8B
512
11.65
6.96
6.69
8.87
3-bit
256
11.33
6.92
6.94
8.14
128
11.27
6.40
6.24
7.62
128
14.41
14.58
16.01
39.66
2-bit
64
13.95
13.41
15.02
29.84
32
12.47
11.91
11.95
16.93
ity. This is attributed to smaller groups providing more detailed data representation and utilizing additional quanti- zation parameters, although they also raise computational and storage demands. A group size of 128 strikes a better balance between efficiency and quantization performance.
rameter salience larger at the t channel of weight, where w
8.
VK
Proof. Given E Re with outlier channel 2;,p- P [0, ml, and other elements with small magnitude Zi.j. where Fy >> Tigj and i, * 4. We can get the Hessian matrix with Lovenberg-Marquaae (Marquardt, 1963) approxima- tion in Eq. (3):
G. Extension on Salience Channel Clustering
G.1. Discussion of Theorem 1
Theorem 1. Given the input calibration activation æ E p with an outlier channel x;" >> Z. Vjj E [0, m].j + p at the position of channel-p. The trace elements of И x x will show great outlier value at (p, p). where H p. > Hig. Vj E [0, m].j # p. as Hp.p produced by [x x = Ei 2 which further leads to the pa-
1 +..
H =
(6)
P.P
15

---

SLiM-LLM
0.03
Average absolute error: 0.0055
0.02
0.01
Error
0.00
0.01
530
1000
1500
2000
(a) Quantization Error of Vanilla Quantizer
0.03
Average absolute error: 0.0039
0.02
Error
0.01
-0002
-003
500
1000
2500
2000
How Index
(b) Quantization Error SQC
Figure 8. Absolute channel error of the weight of the OPT-1.3B model. The red line represents the quantization error for the locally salient weights, and the lightmauve represents other weights. (a) Vanilla quantizer error on the 794 channel of OPT-1.3B. (b) SQC error on the t channel of OPT-1.3B
where xp x will appears at position H.pp- And fol- iip. lowing SparseGPT (Frantar & Alistarh, 2023), the inverse matrix of H can be formulated as:
we can further approximate salience as:
81,
[diag (æ x + Al) 11]1
(8)
81. diag (x
w
1))-
(diag( T xx))) -1) (wis. ||x,)11))
where (x To + A is the new representation of Hessian matrix H for the layer-wise reconstruction problem, and the dampening factor for the Hessian to prevent the collapse of the inverse computation. Additionally, in accor- dance with the configuration in LLMs (Frantar & Alistarh, 2023; Frantar et al., 2022; Sun et al., 2023), the value of X set is extremely small (A < e at the diagonal of Hessian are large. Therefore, only consid- ering the influence of diagonal elements (Sun al., 2023),
Here the diagonal of x T x is diag( (axag(/|=1)) [|==, and ||æ ll2 eval- uates the & norm of 7th channel across different tokens. Consequently, it can be summarized that when there is an outlier channel-p. the value of || is primarily influenced by [xt æ;, Additionally, since the activation values are relatively large and the differences in weight values are com- paratively small, the nh channel of weights will also exhibit salience.
I)), )) while the values located
16

---

SliM-LLM
G.2. Distribution of salience, activation and weight magnitude
H. More Comparisons
In this section, we provide supplementary experiments for SliM-LLM. Tab. displays the comparative results of SliM- LLM and SliM-LLM* with other methods on the OPT series models. Tab. 9 shows the performance of SliM- LLM when quantizing the LLaMA family models on the C4 dataset, while Tab. 10 also compares the results of SliM- LLM + on the C4 dataset. In Tab, 11, we compared the quantization results of GPTQ, AWQ, and SliM-LLM at 2- bit on the Gemma2 and Mixtral models, demonstrating the greater stability of SIiM-LLM across wider range of model structures. Additionally, in Tab. 12, supplemented the 4-bit results of different quantization methods in the LLaMA series models, showing that SliM-LLM and SliM-LLM+ exhibit the smallest quantization errors at practical 4-bit lev- cls. To provide a comprehensive evaluation across a broader set of benchmarks, we further compared the quantization results on MMLU and MathQA in Tab. 13.
Fig. 9 illustrates the distribution of salience among certain weights in LLMs. This section provides additional examples to demonstrate how the distribution of weights and input ac- tivation characteristics influence the salience of parameters in LLMs. The figure captures seven lincar projections in the multi-head self-attention (MHA) and feed-forward block (FFB) layers of the 2nd and 10th Transformer modules in the LLaMA-7B model.
In line with previous findings (Nrusimha et al., 2024; Xiao et al., 2023a), activations demonstrate particularly marked outlier phenomena on anomalous tokens and channels, with extremes differing by more than two orders of magnitude. Notably, distinct anomalous channels are present the MHA's Query, Key, and Value layers, where outliers vary significantly across different tokens. This pattern is con- sistent in the FFB layers. We observe that disparities in weight magnitudes are less pronounced than those in ac- tivation, thus exerting reduced impact on outlier chan- nels. Moreover, weights distribute structurally along rows or columns (Dettmers et al., 2023; Huang et al., 2024a), affecting the overall distribution of salience from row- wise perspective (Fig. 9). However, the most prominent salience is predominantly driven by activation across chan- nels (column-wise).
I. Real Dialog Examples
In this section, we show some dialogue examples of LLaMA- 2-13B and Vicuna- 13B with SliM-LL-M-bbit and GPTQ- 2bit in Fig. 11.
J. Model Experiments on Efficiency
Tab. 14 presents further efficiency comparisons between AutoGPTQ and SliM-LLM. The results shows that even under the larger LLMs with 70B parameters, SliM-LLM can also show the better accuracy with comparable inference efficiency.
G.3. Hessian Diagonal Clustering
Sec. 3.2.1 demonstrates that outlier tokens in input activa- tions result in significant values at the corresponding po- sitions along the diagonal of the weight Hessian matrix. Additionally, due to the token sink phenomenon (Xiao et al., 2023b; Nrusimha et al., 2024), areas around significantly activated key tokens exhibit increased salience, creating clusters of salient regions along the Hessian matrix diagonal. To further elucidate this phenomenon, Fig. 10 shows the values along the diagonal of the Hessian matrix for selected weights in the 2nd and 1hth layers of the LLaMA-7B model. Within this diagonal, certain positions display pronounced values (indicated in red), whereas others are relatively mod- erate. the attention aggregation layer of the 10th layer, the token sink phenomenon results in pronounced convergence of significant values along the Hessian matrix diagonal, with deep red areas indicating regional clustering. These findings reinforce the influence of input activations on the diagonal of the Hessian matrix, subsequently leading to a clustering phenomenon in the salience distribution of weights across channels.
K. Quantization Performance on Vision Language Models
To further showcase the potential application capabilities of SliM-LLM, in Tab. 15, we deploy SliM-LLM on LLaVA- Next-8B (Liu et al., 2024) evaluated 4 benchmarks. The results show that GPTQ, AWQ and SliM-LLM show compa- rable performance under the 3-bit context. However, when the bit-width is setting as 2, GPTQ and AWQ failed to gen- erate the reasonable answer in each benchmark and get "N" results. SliM-LLM successfully generate the reasonable out- put and the accuracy is closed to 3-bit model, which presents the superior usability of MliM-LLM to wider environments.
17

---

SIiM-LLM
Query
Key
Value
Out
Up
Down
Salence
Weight
(a) layer-2
SaVence
Activation
Weight
(b) layer-10
Figure 9. Salience, activation and weight distribution in the nnd and 10 layers of LLLMA-7B
Query
Key
Down
Out
(a) layer-2
(b) layer-10
Figure 10. Hessian diagonal magnitude in attention layers of 2nd and 10th layers of LLaMA-7B
18

---

SIM--LLM
Table 8. Quantization results of OPT Models WikiText2 (group size is 128).
#W PPL
Method
1.3B
2.7B
6.7B
13B
30B
66B
16-bit
14.63
12.47
10.86
10.12
9.56
9.34
RTN
1.2e2
3.0e2
23.54
46.03
18.80
1.466
GPTQ
16.47
13.69
11.65
10.35
10.96
AWQ
16.32
13.58
11.41
10.68
9.85
9.60
QulP
16.21
13.79
11,51
10.50
9.75
9.59
3-bit
SliM-LLM
15.91
13.26
11.27
10.26
9.70
9.48
OmniQuant
13.72
13.18
11.27
10.47
9.79
9.53
AffineQuant
15.61
12.98
10.51
SliM-LLM
15.58
12.84
11.18
10.44
9.67
9.51
RTN
1.3e4
5.7e4
7.8e3
7.6e4
1,3e4
3.6e5
GPTQ
1.1e2
61.59
20.18
21.36
12.71
82.10
AWQ
47.97
28.50
16.20
14.32
12.31
14.54
QulP
41.64
28.98
18.57
16.02
11.48
10.76
2-bit
PB-LLM
45.92
39.71
20.37
19.11
17.01
16.36
SliM-LLM
30.71
24.08
14,41
13.68
11.34
10.94
OmniQuant
23.95
18.13
T4.43
12.94
1T.39
30.84
SliM-LLM
24.57
17.98
14.22
12.16
11.27
14.98
Table 9. Quantization results of LLaMA Family with statistic quantizer on C4 (group size is 128).
#W PPL.
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
3-8B
3-70B
16-hit
7.08
6.61
5.98
5.62
6.97
6.46
5.52
9.22
6.85
APTQ
6.24
RTN
8.62
7.49
6.58
6.10
8.40
7.18
6.02
1.1c2
22.39
3-bit
AWQ
7.92
7.07
6.37
5.94
7.84
6,94
11.62
8.03
GPTQ
7.85
7.10
6.47
6.00
7.89
7.00
5.85
13.67
10.52
SliM-LLM
6.14
6.05
6.33
5.94
7.74
5,26
5.09
13.10
8.64
RTN
1.0e3
4.5e2
99.45
17.15
4.9e3
1.4e2
42.13
2,5e4
4.6e5
AWQ
1.9c5
2.3e5
2.4c5
7.5e4
1.7c5
9.4e4
2.1c6
1.4e6
GPTQ
34.63
15.29
11.93
11.99
33.70
20.97
NAN
4.1c4
21.82
2-bit
QuIP
33.74
21.94
10.95
13.99
31.94
16.16
8.17
1.3c2
22.24
PB-LLM
49.73
26.93
11.85
29.84
19.82
8.95
79.21
33.91
SIM-LLMM
32.91
13.85
11.27
10.95
16.00
9.41
7.01
1.1e2
15.92
Table 10. Quantization results of LLaMA-1 and LLaMA-2 models with learnable quantizer C4.
#W PPL.
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
16-bit
7.08
6.61
5.98
5.62
6.97
6.46
5.52
OmniQuant
7.75
7.05
6.37
5.93
7.75
6.98
5.85
3-bit
AffineQuant
7.75
7.04
6.40
7.83
6,99
SlIM-LLM
7.75
6.91
6.36
5.96
7.71
6.90
5.85
OmniQuant
12.97
10.36
9,36
8.00
15.02
11.05
8.52
2-bit
Affineuuant
14.92
12.64
9.66
16.02
10.98
SlIM-LLM
14.99
10.22
9.33
7.52
18.18
10.24
8.40
Table 11. PPI. Compariso on Gemma2 and Mixtral,
Model/Evaluation
Method
PPL (wikitext2)
GPTQ 2-bit
186.77
Gemma2-9B
AWQ 2-bit
217.83
SliM-LLM 2bit
26.30
GPTQ 2-bit
16.38
Mixtral 8x7B
AWQ 2-bit
3.2e5
SliM-LLM 2bit
7.44
19

---

SIMM-LLM
Table The PPL results our proposed method and other methods under quantization.
Method
LLaMA-7B
LLaMA-13B
LLaMA2-7B
LLaMA2-13B
LLaMA3-8B
FP16
5.09
5.47
4.88
5.75
AWQ
5.81
5.30
5,62
4.97
6,63
GPTQ
5.85
5.20
5.61
4.98
6.50
SliM-LLM
5.83
5.16
5.59
4.95
6.42
Omniquant SliM-LLM
5.77
5.58
5.75
5.57
Table 13. The results(%) MMLU and MathQA for multiple quantized LLaMA models.
Model
Method
Humanitics
Social Sciences
STEM
Other
MMLU
MathQA
GPTQ 2-bit
24.87
21.84
21.79
24.01
23.32
21.11
LLaMA-7B
AWQ 2-bit
24.21
21.71
21.25
23.98
22.95
22.21
SliM-LLM 2bit
24.94
23.60
23.40
25.50
25.10
23.74
GPTQ 2-bit
24.23
23.20
22.99
24.78
23.85
21.68
LLaMA-13B
AWQ 2-bit
24.17
31.07
28.61
25.14
26.89
21.98
SliM-LLM 2bit
25.12
31.74
29.19
26.17
27.05
23.17
GPTQ 2-bit
25.02
22.13
22.61
23.17
23.44
21.07
LLaMA2-7B
AWQ 2-bit
25.12
22.79
24.26
24.01
24.51
19.06
SliM-LLM 2bit
26.60
23.23
25.70
25.70
25.81
22.55
GPTQ 2-bit
23.91
27.17
26.10
25.78
25.53
20.87
LLaMA2-13B
AWQ 2-bit
24.17
31.07
28.61
25.14
19.53
SliM-LLM 2bit
26.27
32.20
29.98
26.46
27.34
23.48
Table 14. LLaMA-2-70B results of GPTQ and Slim-LLM GPU. Group size is set 128.
#W
LLaMA-2-70B
WM
RM
PPL.
Token/s
GPTQ
28.0G
34.9G
3.85
6.5
3-bit
SliM-LLM
28.0G
35.2G
3.67
6.2
GPTQ
16.4G
23.3G
8.78
9.7
2-bit
SliM-LLM
23.5G
4.4G
6.28
8.4
Table 15. Quantization results on Vision Language Models (VLMs).
#W
#G
AI2D
ChartQ
DocVQA
MMBench
3
128
66.2
65.1
75,6
67.4
GPTQ
2
128
N
N
N
N
3
128
67.7
65.4
74.4
68.0
AWQ
2
128
N
N
N
N
68.2
67.5
74.8
68.9
SliM-LLM
2
128
57.2
49.3
60.6
60.9
20

---

SIiM-LLM
LLaMA-2-13B
Prompt: The eight planets Solar System are Mercury,
GPTQ-2bit: 10.2. The eight pill o free games of 62.2. The way we do this.
Slim-LLM-2bit: Venus, Earth, Mars, Jupiter, Satum, Uranus and Neptune.
Prompt: The tallest and most famous tower in France is
GPTQ-2bit: is 350 and most recently.
Slim-LLM-2bit: 320 meters high.
Prompt: The 2012 Summer Olympic was held in
GPTQ-2bit: The 2012 Summer Olympic was held in 1016 (a.185) and the 775,000 km from each.
Slim-LLM-2bit: 2012 in London, the United Kingdom.
Vicuna-13B
Prompt: What are the capitals of Brazil, Mexico and France?
GPTQ-2bit: What is the RER score?
Slim-LLM-2bit: The capital Brazil Brasília, the capital Mexico Mexico City, and the capital of France is Paris.
Prompt: What are the ancient civilizations Europe?
GPTQ-2bit:
am trying to find book nook over the life of to 300,000 civilization the one which history books be pop and have be, oh, do you
The ancient
know the one the Mayan, the one of the total for Dake.
Slim-LLM-2bit: The ancient civilizations in Europe include the Roman Empire, the Byzantine Empire,
and the Ottoman Empire.
Prompt: What are the logical circuits?
GPTQ-2bit:
What the order the logical cumulated?
What is the best strategy to do in a single?
What is the most popular to be eaten? What are the most sought?
Slim-LLM-2bit: The logical circuits are the basic components for building the logical functions, which are required digital computing. These circuits use logic structures and devices that allow the use of Boolean and multiple logic to implement the functional requirements defined by problem solving method of computer. In digital computing, circuits perform the logical tasks required solving problems.
Figure 11. Some examples of conversations. I.MMA-2-133 and Vicuna- 13B are chosen to show the case of language supplementary and Q&A ability. And GPTQ-2be 2bit is selected as the comparison. We color the text to show the reasonable or inappropriate responses.
21


Reviews:
[{"review_id": "WJsZlmuQ9L", "reviewer": "Reviewer_BegC", "summary": "This paper proposes SliM-LLM, a post-training quantization (PTQ) framework for large language models (LLMs). Its core idea is to allocate bit-widths to weight groups adaptively and locally preserve important (salient) weights. The approach combines two techniques: \n1. Salience-Determined Bit Allocation (SBA): Groups of weights are each assigned a suitable bit-width based on the group’s global importance (salience). \n2. Salience-Weighted Quantizer Calibration (SQC): Within each group, a small subset of highly salient weights is given extra quantizer “attention” to reduce local discretization errors.\n\nBy focusing resources on channels or weight elements deemed most critical, SliM-LLM aims to achieve lower perplexity in ultra-low bit regimes (1–3 bits) without incurring major hardware overhead. Experimental results emphasize LLaMA (1, 2, 3) and OPT model families, showing that SliM-LLM outperforms existing PTQ baselines in perplexity reduction, and can be integrated into popular PTQ toolkits (e.g., GPTQ, OmniQuant). The paper also includes limited results on less-traditional architectures such as Gemma2 or Mixtral, in an effort to demonstrate some level of generalizability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BegC,\n\nThank you for your valuable feedback. We have summarized your questions and concerns. Due to the character limit for reply, if there are any question that are not detailed, we will further reply in next stage. Thanks you so much.\n\n> Q1:(1) There is...diverse deployment conditions. (2)Discussion of trade-offs...conditions.\n\nAs shown in Table 5, we analyzed deployment speed under different memory compression levels using a batch size of 1 (align with other works), focusing on extreme compression for single-GPU. We added inference experiments for the compressed LLaMA-7B model on different GPUs, testing various batch sizes.\n|GPU|Method|Bit|Batch|Token/s|\n|-------------|------------|-------|---------|-----------|\n|RTX4090|SliM-LLM|2|4|55.1|\n||||8|34.4|\n|A100|SliM-LLM|2|4|59.2|\n||||8|40.2|\n\nSlight speed decreases with larger batch, consistent across frameworks like AutoGPTQ and AutoAWQ. \n\n> Q2: (1)Most of the core...to be adapted. (2)The usage of...is short. (3) If future LLM...approach?\n\nThe LLaMA and OPT series are among the most widely downloaded and applied LLM architectures in the community. They are also commonly used as base models in other works. As you noted, we include more experiments in Table 11 to evaluate models with extra normalization layers. We claim that salience is a relative concept, as LLM training inherently produces such salience[1][2], even in multimodal or fine-tuned models. While architectures like Gemma2 use extra normalization layers, relative salience of features still emerges.\n\nSliM-LLM focuses on compressing LLMs. Follow your suggestion, we expanded multimodal task on the LLaVA-Next 8B model(N:collapse accuracy).\n||#W|#G|AI2D|ChartQA|DocVQA|MMBench|\n|--------------------------|------|-------|--------|------------|-----------|------------|\n|GPTQ|3|128|66.2|65.1|75.6|67.4|\n||2|128|N|N|N|N|\n|AWQ|3|128|67.7|65.4|74.4|68.0|\n||2|128|N|N|N|N|\n|SliM-LLM|3|128|68.2|67.5|74.8|68.9|\n||2|128|57.2|49.3|60.6|60.9|\n\n[1]From Attention to Activation: Unraveling the Enigmas of Large Language Models.\n\n> Q3: (1)The experiments a...exhibit high salience. (2) The theoretical discussion...distributions.\n\nThe proportion of local outlier weights within groups is not an assumption but an observation supported by prior studies. Section 3.2.2 and previous work[1] consistently show that salient weights within groups remain a small proportion. Theorem 1 and detailed proofs in Appendix G explain this phenomenon.\n\n[1] SpQR: A sparse-quantized representation for near-lossless LLM weight compression.\n\n> Q4: (1)There is no detailed breakdown ...vary. (2)The paper...large models (e.g., 70B+)?\n\nIn Table 7, we provided the quantization performance of four LLMs under different group sizes. SliM-LLM introduces negligible storage overhead, which decreases further as model size grows. For instance, with LLaMA2-70B (group sizes = 64, 128, 256), a single transformer layer (size 8192 × 8192) requires a group matrix of size 8192 × 64/128/256. Using three precision levels, only a 2-bit flag (e.g., 01 = 1-bit, 10 = 2-bit, 11 = 3-bit) is needed per group. The additional storage overhead is $\\frac{2}{8192 \\times 64/128/256}$—virtually negligible at scale. Other quantization parameters are identical to frameworks like GPTQ.\n\nBased on your suggestion, we will add 70B models results.\n|LLaMA-2-70B #W|Method|WM|RM|PPL|Token/s|\n|------------------|------------|---------|----------|-----------------|-----------|\n|3-bit|GPTQ|28.0G|34.9G|3.85|6.5|\n||SliM-LLM|28.0G|35.2G|3.67|6.2|\n|2-bit|GPTQ|16.4G|23.3G|8.78|9.7|\n||SliM-LLM|16.5G|23.5G|6.28|8.4|\n\n> Q5: The authors rely on standard...domain-specific conditions.\n\nOur paper evaluates quantized model performance on 13 benchmarks, including Wikitext2, C4, and 8 zero-shot tasks, aligning with other LLM quantization works. Appendix Table 13 includes benchmarks from Humanities, Social Sciences, and STEM, with reasoning tasks like MathQA highlighting SliM-LLM's strengths in math reasoning. We will add results for LLaVA-Next-8B on 4 multimodal tasks in Q2.\n\n>Questions For Authors:\n\n(1)As shown in Definition 3.1 and Theorem 1, salience is influenced by weight magnitudes and the Hessian. If all weights exhibit similar salience, the distinction between salient and non-salient weights (Equation 5) becomes negligible. In such cases, τ converges to 1, simplifying quantization and allowing parameters to be derived from basic statistics.\n\n(2)Codebook-based methods can improve accuracy but significantly reduce inference speed—e.g., 2-bit code-book can be 3× slower. SliM-LLM’s structured mixed-precision quantization achieves competitive accuracy while maintaining practical inference speeds.\n\n(3)Frameworks like recent HQQ show promising advancements in GPU kernels for 2-bit quantization. These developments could further improve SliM-LLM's inference speed.\n\n(4)For calibration, SliM-LLM used only 128 random samples from WikiText2, avoiding additional data or adaptive re-quantization.", "claims_and_evidence": "The paper makes strong claims that its proposed SliM-LLM method yields significant performance gains at very low bit-widths (1–3 bits) and generalizes effectively across large language models. While the results on LLaMA (and to a lesser extent OPT) support improved perplexity and moderate memory overhead, **some claims remain only partly substantiated**:\n\n1. **Claim: “SliM-LLM balances compression rate and inference speed with limited overhead.”**  \n   - **Evidence**: The authors do show memory usage and token-throughput comparisons (e.g., Table 5), illustrating that at 2 bits, perplexity drops dramatically yet throughput also declines. The data partly confirms a trade-off.  \n   - **Gaps**: There is **no systematic exploration** of how severe the latency penalty can become in different scenarios (e.g., multi-GPU, different batch sizes). This means the paper has not comprehensively demonstrated that SliM-LLM offers a robust speed–accuracy trade-off across diverse deployment conditions.\n\n2. **Claim: “SliM-LLM generalizes well beyond LLaMA/OPT.”**  \n   - **Evidence**: A few additional experiments (e.g., Gemma2-9B, Mixtral-8×7B) in the appendix are cited to show performance gains relative to GPTQ.  \n   - **Gaps**: Most of the core experiments still focus heavily on LLaMA/OPT families, leaving limited discussion or validation on how the method behaves for architectures with different normalization schemes, or for radically different designs (e.g., multimodal LLMs). The authors do not deeply explore how outlier channels are altered by extra normalization layers in Gemma2, nor whether the salience-based approach needs to be adapted.\n\n3. **Claim: “SliM-LLM’s salience-based bit allocation and quantizer calibration handle local outliers effectively.”**  \n   - **Evidence**: The paper describes Salience-Weighted Quantizer Calibration (SQC) and provides ablation results indicating that it helps preserve a small fraction of crucial, high-magnitude weights.  \n   - **Gaps**: The experiments assume that outliers make up only around 1–5% of weights. For more extreme cases—where outlier or salient weights are more widespread—the paper does not offer specific evidence. Thus, it is unclear how robust the approach is if the model’s salience distribution does not cluster or if many channels simultaneously exhibit high salience.\n\n4. **Claim: “SliM-LLM introduces only minor hidden overhead.”**  \n   - **Evidence**: The authors note a small increase in memory usage (e.g., ~0.1GB more than GPTQ at 2 bits on 7B-scale models) and mention that storing group-level bit-width maps is not a huge burden.  \n   - **Gaps**: There is no detailed breakdown of overhead for very large models (70B+ parameters) or a thorough demonstration that the additional groupwise metadata stays modest across scales. The paper would be stronger if it rigorously measured overhead as model size and group granularity vary.\n\nOverall, while the **performance claims** at low-bit quantization (especially on LLaMA and OPT) are convincingly supported by perplexity gains and ablations, the **generalizability and overhead** claims are not explored in as much depth. Addressing the trade-offs with speed, broader model architectures, and more diverse salience patterns would make the evidence more conclusive.", "methods_and_evaluation_criteria": "The paper’s methodology—mixed-precision post-training quantization evaluated via perplexity on language modeling benchmarks—fits well with the core objective of aggressively compressing large transformer-based LLMs. Perplexity and zero-shot accuracy are \breasonable metrics for confirming whether the quantized model still preserves core language understanding. \n\nHowever, **some limitations remain**:\n\n1. **Narrow Focus on LLaMA/OPT Families**  \nMost tests rely on LLaMA and OPT, which both use Pre-LN Transformer designs widely known for pronounced outlier channels. This raises the question of whether SliM-LLM’s promising results might partly hinge on the fact that these specific architectures emphasize salience clustering. If newer Transformers (or non-Transformer models) suppress or redistribute outliers—an effect sometimes seen with extra normalization layers —the gains from SliM-LLM might be less pronounced or require modifications. More diverse experiments on architectures without those strong outlier characteristics would clarify whether these gains arise purely from leveraging Pre-LN behaviors or can generalize more broadly.\n\n*Sun, Mingjie, et al. \"Massive activations in large language models.\" COLM2024.  \n*Kedia, Akhil, et al. \"Transformers get stable: an end-to-end signal propagation theory for language models.\" ICML2024.  \n*Oh, Jaehoon, Seungjun Shin, and Dokwan Oh. \"House of cards: Massive weights in llms.\" arXiv 2024.  \n*Sun, Wenfang, et al. \"The Curse of Depth in Large Language Models.\" arXiv 2025.  \n\n2. **Sparse Analysis of Inference Trade-offs**  \nWhile the paper includes perplexity vs. token/s results, there is no deeper experiment varying batch sizes, GPU setups, or real-time conditions. It remains unclear whether SliM-LLM consistently meets real-world speed demands, especially under heavy concurrency or stricter latency targets. The discussion on throughput is largely single-GPU, and additional multi-GPU or distributed benchmarks could confirm if memory/performance scales similarly.\n\n3. **Evaluation Datasets**  \nThe authors rely on standard data like Wikitext2, C4, and limited zero-shot tasks. Although these are reasonable starting points, evaluating reasoning or multimodal tasks could uncover further performance nuances and better reflect practical deployments. While perplexity is a solid metric, real-world usage often involves more specialized tasks. The paper does not yet assess how SliM-LLM behaves under such domain-specific conditions.\n\nAltogether, the presentation (PTQ for LLMs assessed by perplexity and memory/speed) is well-aligned with the basic goal of compressing large language models. Yet the limited architectural variety tested, the lack of detailed real-time inference experiments, and the narrow range of evaluation tasks leave open questions as to how broadly and reliably SliM-LLM can be applied in production-scale environments.", "theoretical_claims": "The authors briefly justify salience clustering by referencing Hessian approximations and outlier activation channels. These are not extremely formal proofs but rather high-level derivations consistent with prior work on Hessian-based importance. No obvious flaws stand out, though they remain somewhat heuristic in nature.", "experimental_designs_or_analyses": "The experimental protocol mostly follows conventional PTQ setups (calibrate on a small set of input tokens, measure perplexity on standard test sets, compare with established baselines). The additional tables for Gemma2 and Mixtral do help, but only in a small-sample manner. Some analyses of memory overhead and speed trade-offs are provided (Table 5), but a deeper exploration of “how big is the hidden overhead for group-bit metadata?” or “which scenarios lose the most throughput from mixed-precision?” could be more thorough.", "relation_to_broader_scientific_literature": "- The paper aligns with ongoing efforts to aggressively compress LLMs under 3-bit (e.g., QuIP, OmniQuant, GPTQ, etc.).\n- Extends the salience-based approach of GPTQ but adds a structured mixed-precision angle, reminiscent of older “HAWQ” or “APTQ” but at a finer group level.\n- Prior works that address outlier channels or element-wise mixed precision are referenced (e.g., AWQ).", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths**\n\n- Empirical gains at low bit quantization, often outperforming prior PTQ baselines.\n- Fairly robust experiments on standard language modeling tasks.\n- Engineering details (Appendix B) on how to pack bits group-wise, which is helpful for reproducibility.\n\n**Weaknesses**\n\n- Discussion of trade-offs in real-world inference scenarios remains somewhat limited. While the paper acknowledges that speed can drop at 2 bits, it does not systematically measure it across different GPU kernels or larger-batch conditions.\n- The usage of only a few beyond-LLaMA architectures (Gemma2, Mixtral) is a first step, but the explanation for why it should work in every scenario is short.\n- Overall paper formatting (some references, layout) does not strictly follow typical ICML style, giving an unfinished impression.\n- The theoretical discussion is primarily heuristic—though typical in quantization research, it might have benefited from deeper analysis or direct ablation on truly scattered salience distributions.", "other_comments_or_suggestions": "None", "questions_for_authors": "- Could the authors elaborate how SQC would respond if *every* channel had moderate outliers rather than a few large outlier channels? Would the search for τ-parameters become unstable or scale poorly?  \n- Would additional codebook-based or vector-quantization steps (e.g., SpQR) further improve results?\n\n- The paper includes a small table (Table 5) comparing memory usage and token throughput. Could the authors detail how much extra overhead is stored in the group-wise bit metadata at 2 bits vs. 3 bits, especially for large models (e.g., 70B+)?  \n- Do you foresee more advanced GPU kernels that specifically accelerate group-wise mixed-precision to mitigate the 2-bit throughput drop?\n\n- If future LLM variants (e.g., multi-modal or heavily fine-tuned) produce qualitatively different activation distributions, how stable is the salience-based approach?  \n- Are additional calibration samples or adaptive re-quantization steps needed to maintain performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmIAthLeY2", "reviewer": "Reviewer_Ut3n", "summary": "This paper introduces a group-wise mixed-precision quantization method for LLMs, addressing challenges in accuracy and efficiency. The key contributions are two strategies: SBA, which optimally allocates bit-widths by minimizing entropy divergence through Hessian and weight salience analysis, and SQC, which enhances salient weight representation by adjusting quantizer sensitivity using a calibration parameter. To handle group outliers, the method balances scale and zero-point adjustments with a focus on scale sharing. Both strategies are compatible with various quantizers. Experiments show that Slim-LLM outperforms existing methods on LLAMA and OPT models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ut3n,\n\nWe sincerely appreciate your insightful feedback and suggestions. Below, we will respond to your questions and recommendations individually.\n\n> Q1: The authors are advised to consider testing on more challenging LLM benchmarks, such as GSM8K in the mathematics domain. Compared to MathQA, GSM8K may be more sensitive to the loss caused by quantization.\n\nA: Thank you for your suggestion. We have conducted additional tests to evaluate the performance of our method on more challenging datasets, such as GSM8K. The results are shown in Table below. \n\n|Model/Evaluation|Method|GSM8K|\n|-|-|-|\n|**LLaMA-7B**|GPTQ 3-bit|11.5|\n||AWQ 3-bit|11.5|\n||SliM-LLM 3-bit|11.5|\n|**LLaMA-7B**|GPTQ 2-bit|0.0|\n||AWQ 2-bit|0.0|\n||SliM-LLM 2-bit|9.2|\n|**LLaMA2-7B**|GPTQ 3-bit|13.0|\n||AWQ 3-bit|13.4|\n||SliM-LLM 3-bit|13.6|\n|**LLaMA2-7B**|GPTQ 2-bit|0.0|\n||AWQ 2-bit|0.0|\n||SliM-LLM 2-bit|10.3|\n\nThe findings demonstrate that, even on GSM8K, our method exhibits less accuracy loss compared to other approaches, highlighting its robustness across different test sets.\n\n> Q2: In group-wise mixed-precision inference, the allocation of quantization scales across different channels and the process of dequantization could provide deeper insights into the inference details of this structure. The authors are encouraged to provide further explanations on these aspects.\n\nA: We have explained some details on how to allocate, store, and perform inference with quantization scales in Appendix B.2. This means that, while storing the quantized weights at extremely low bit widths, we also store the corresponding quantization scales for each row, and are able to perform dequantization operations on both scales and quantized integers on the CUDA Kernel, restoring them to the floating-point values required for inference. We will clarify and explicitly refer to this section in the main text.\n\n> Q3: There are minor typos, and the abbreviations WM and RM in Table 5 lack annotations.\n\nA: Thank you for your careful and thorough review! We will further check for any writing errors in the paper to ensure better readability. Regarding the issue you raised about the lack of explanations for WM and RM in Table 5, we would like to clarify that WM stands for Weight Memory and RM stands for Running Memory. We will also add the definitions of WM and RM in the relevant section to ensure they are accurately explained. Thank you for your helpful reminder.\n\n> Q4: Although SLiM-LLM is a leading PTQ method in the 2-bit and 3-bit settings, which is a popular topic in the research field, it appears that the commonly used 4-bit quantization for industrial applications has not been explored in the paper. Could the authors provide a comparison of 4-bit PPL with RTN, AWQ, and GPTQ?\n\nA: We sincerely appreciate your suggestion regarding comparative experiments with 4-bit quantization. We would like to clarify that although SliM-LLM is primarily optimized for 2-bit and 3-bit quantization, our method is also compatible with 4-bit mixed-precision quantization. In response to your suggestion, we have included additional experiments in the revised versions of Table 1 and Table 2. Furthermore, we have tested our method on several 4-bit models and are pleased to share the results with you below:\n\n|Method|LLaMA-7B|LLaMA-13B|LLaMA2-7B|LLaMA2-13B|LLaMA3-8B|\n|-|-|-|-|-|-|\n|FP16|5.68|5.09|5.47|4.88|5.75|\n|AWQ|5.81|5.30|5.62|4.97|6.63|\n|GPTQ|5.85|5.20|5.61|4.98|6.50|\n|SliM-LLM|5.83|5.16|5.59|4.95|6.42|\n\n|Method|LLaMA-7B|LLaMA2-7B|\n|-|-|-|\n|Omniquant|5.77|5.58|\n|SliM-LLM+|5.75|5.57|\n\n\\*The results of RTN are too worse than GPTQ and AWQ to be listed here.", "claims_and_evidence": "The submission's claimed phenomena and results are supported by detailed evidence.", "methods_and_evaluation_criteria": "The authors propose an innovative group-wise structured mixed-precision quantization strategy for LLMs, balancing accuracy and efficiency. It offers new insights into extreme compression under 2-bit and 3-bit settings and can be easily integrated into existing quantization tools. The observation and proof of the structured distribution of significant weights introduce a new paradigm for future LLM compression strategies.", "theoretical_claims": "reviewed the correctness of the theoretical claims in Section 3.2.1. Figure 3 provides detailed evidence visualizing the weight clustering characteristics, and Section 3.2.1 theoretically establishes the relationship between the metric in Definition 3.1 and weight clustering. The discrete weight characteristics discussed in Section 3.2.2 are also supported by visualizations and additional explanations in the appendix. These proofs validate the proposed structured mixed-precision quantization framework.", "experimental_designs_or_analyses": "The authors provide detailed experiments demonstrating the performance advantages of SliM-LLM, particularly achieving better results than existing methods on the ppl evaluation metric and commonsense benchmarks. The appendix includes additional results on challenging tasks like math after quantization. Figure 5 presents thorough ablation studies showing the contribution of each component in the quantization method. The paper also provides real hardware deployment results for memory usage and inference speed, making the experiments comprehensive.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThis work contributes a lot to the area of low bit quantization and compression. Make it efficient and accurate for different kinds of post training LLMs, especially under 2-bit and 3-bit. The structured mixed-precision proposed in SliM-LLM is a straightforward and effective method for the committee.\n\n2.\tEvaluation of the method’s efficiency is rigorous and supported by an extensive set of experiments. The results are well-documented and demonstrate the practical applicability and effectiveness of the proposed approach. The performance comparisons with existing methods highlight the strengths of the paper’s contributions, offering promising insights into its potential impact on the field. The inclusion of various evaluation metrics further strengthens the reliability and generalizability of the findings.\n\nWeaknesses\n\n1.\tThe authors are advised to consider testing on more challenging LLM benchmarks, such as GSM8K in the mathematics domain. Compared to MathQA, GSM8K may be more sensitive to the loss caused by quantization.\n\n2.\tIn group-wise mixed-precision inference, the allocation of quantization scales across different channels and the process of dequantization could provide deeper insights into the inference details of this structure. The authors are encouraged to provide further explanations on these aspects.", "other_comments_or_suggestions": "There are minor typos, and the abbreviations WM and RM in Figure 5 lack annotations.", "questions_for_authors": "Although SLiM-LLM is a leading PTQ method in the 2-bit and 3-bit settings, which is a popular topic in the research field, it appears that the commonly used 4-bit quantization for industrial applications has not been explored in the paper. Could the authors provide a comparison of 4-bit PPL with RTN, AWQ, and GPTQ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pu6FazNr2L", "reviewer": "Reviewer_iFeq", "summary": "The paper introduces SliM-LLM, a novel PTQ framework for LLMs. The proposed method leverages the authors’ observation that important weights follow a structured distribution to preserve the model performance at extremely low-bit precision. Their two key contributions are:\n\n- **Salience-Determined Bit Allocation** analyzes the structured distribution of weight salience to assign different precisions to groups of weights.\n\n- **Salience-Weighted Quantizer Calibration** adjusts the quantizer parameters with a focus on the few highly salient weight elements within each weight group.\n\nThe experimental results show that SliM-LLM reduces perplexity compared to SOTA gradient-free PTQ methods while reducing memory usage by nearly 6x. The extended version with graidient-based optimization, SliM-LLM+, further improves the model performance.\n\n## update after rebuttal\nSliM-LLM achieved state-of-the-art performance in 2-bit quantization of large language models (LLMs) through two main contributions: Salience-Determined Bit Allocation and Salience-Weighted Quantization Calibration. While this approach effectively reduces the memory requirements of LLMs, it introduces a trade-off in terms of speedup — another critical benefit typically expected from quantization — in exchange for higher accuracy.\nOverall, while the paper's claims are well-supported empirically and demonstrate strong experimental results and my other questions are well-addressed, further exploration is still needed regarding inference efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer iFeq,\n\nThank you for your valuable feedback and suggestions. We will address your questions and recommendations one by one.\n\n> Q1: (1)The analysis of local salience is relatively underdeveloped, and it has also been explored in recent previous works. (2)The authors may add references for works that mentions occurrence of unstructured outliers (salient values) such as [1], [2] of part 2.\n\nA: The variation in weight importance distribution is crucial for LLM quantization, particularly regarding local salience. As noted in prior studies, our key contribution is using the element sensitivity criterion (Definition 1) to analyze significant weight clustering and provide a theoretical explanation for salience clustering.  \n\nWe did not explore the underlying causes of local salience in detail, as they have been thoroughly discussed in works like [1] [2] and Section 3.3.2. Instead, we built on these insights to develop the Salience-Weighted Quantizer Calibration (SQC) strategy to reduce quantization errors. We appreciate your reference suggestions and will expand the discussion in the revised version.\n\n> Q2: Inference efficiency is another important aspect ... efficient inference, appears to discuss a different topic.\n\nThank you for identifying the typo. This was a layout error, and we will correct it by including the full content of the \"Efficient Inference on Device\" section in the final version.\n\nSpecifically, we extend the CUDA kernel in AutoGPTQ to support experimental mixed-precision inference (details in Appendix B.2). We evaluate LLaMA-7/13B and LLaMA-2-7B under 2/3-bit settings, showing that our approach maintains a high compression rate on GPUs while significantly improving model accuracy, with only a slight inference speed reduction on the A800 due to bit-width alignment. As 1-bit operations currently lack hardware support, additional storage and computation are required. We recognize the potential for further optimization in mixed-precision computing and aim to improve this in future work.\n\n> Comments Or Suggestions.\n\nA: (1)Thank you for your valuable suggestions! We will follow your advice and add the complete inference settings in Section 4.3.\n\n(2)The deployment time of SliM-LLM consists of two main parts: SBA bit-width search and SQC quantization parameter determination. As detailed in Section 3.2.2, the SBA dual-pointer search is highly efficient, and SQC computation time is comparable to standard calibration-based quantizers. SliM-LLM integrates seamlessly with existing PTQ strategies and, being training-free in a plug-and-play manner, completes 7B LLM quantization in about 25 minutes.\n\n(3)For the ablation study on group size, we have discussed the differences in detail in Appendix F and provided experimental results in Table 7. We will further highlight this content in the main text to improve the readability of the paper.\n\n> Questions For Authors.\n\nA: (1)Thank you for your insightful observations. You are correct about the challenges with models like LLaMA-3. Despite this, SliM-LLM maintains leading quantization performance and remains highly practical.  \n\nAs noted in prior works [1–4], quantization methods degrade more severely in knowledge-dense models like LLaMA-3, especially at ultra-low bit widths. Our experiments confirm this: in Table 1, LLaMA-3 8B with AWQ and GPTQ yields PPLs of 8.22 and 8.19 under 3-bit quantization, rising to 210 and 1.7e6 at 2-bit. This suggests that as models grow in knowledge density, conventional methods suffer greater quantization losses. In contrast, SliM-LLM achieves PPLs of 7.16 (3-bit) and 39.66 (2-bit), demonstrating its effectiveness.  \n\nAdditionally, we have included 4-bit quantization experiments, which are more practical, to further highlight SliM-LLM’s advantages on LLaMA-3.\n\n|Method|LLaMA-7B|LLaMA-13B|LLaMA2-7B|LLaMA2-13B|LLaMA3-8B|\n|-|-|-|-|-|-|\n|FP16|5.68|5.09|5.47|4.88|5.75|\n|AWQ|5.81|5.30|5.62|4.97|6.63|\n|GPTQ|5.85|5.20|5.61|4.98|6.50|\n|SliM-LLM|5.83|5.16|5.59|4.95|6.42|\n\n|Method|LLaMA-7B|LLaMA2-7B|\n|-|-|-|\n|Omniquant|5.77|5.58|\n|SliM-LLM+|5.75|5.57|\n\n(2)SliM-LLM+ incorporates our structured mixed-precision strategy into OmniQuant’s gradient quantizer. However, as OmniQuant and its comparator, AffineQuant, currently lack support for models like Gemma and Mixtral, we did not report results on these. We have contacted the OmniQuant authors to request updates and are actively working to extend SliM-LLM+ to more models.\n\n[1] Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference. arXiv:2409.20361.\n\n[2] Efficientqat: Efficient quantization-aware training for large language models[J]. arXiv:2407.11062.\n\n[3] Compressing large language models using low rank and low precision decomposition[J]. NeurIPS, 2024.\n\n[4] An empirical study of llama3 quantization: From llms to mllms[J]. Visual Intelligence, 2024.", "claims_and_evidence": "The analysis of the structuredness of global salience of weights is well supported by results across various layers and models. However, the presence of structural outliers in the output activations of LLMs—and the corresponding salient weight channels—has been extensively addressed in previous works.\n\nThe analysis of local salience is relatively underdeveloped, and it has also been explored in recent previous works [1][2].\n\n[1] Yi, Ke, et al. \"Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference.\" arXiv preprint arXiv:2409.20361 (2024).\n\n[2] Yu, Mengxia, et al. \"The super weight in large language models.\" arXiv preprint arXiv:2411.07191 (2024).", "methods_and_evaluation_criteria": "SBA and SQC are logically designed methods that directly address the aforementioned problems. Evaluating using perplexity and zero-shot tasks is appropriate, as these metrics are standard in LLM compression work.", "theoretical_claims": "The structured distribution of salience weights due to outlier activations has been pointed out and addressed in various works, and the existence of locally important weights also is not a novel issue introduced by this paper. \n\nRather than providing a mathematical proof, the paper seeks to validate these issues through experimental evidences.", "experimental_designs_or_analyses": "Inference efficiency is another important aspect just as quantized model performance. Although Table 5 presents comparison results with GPTQ, the main text lacks an explanation and analysis of the comparison scenario. The title of subsection 4.3, which claims to address efficient inference, appears to discuss a different topic.", "relation_to_broader_scientific_literature": "This research is influenced by existing work on post-training quantization of LLMs. The goal of preserving accuracy by effectively handling outlier or salient values was already established by previous studies, and this work extends that by demonstrating a more hardware-friendly implementation. \n\nAdditionally, by addressing both local and global salience, the paper represents a significant extension of the field.", "essential_references_not_discussed": "The authors may add references for works that mentions occurrence of unstructured outliers (salient values) such as [1], [2] of part 2.", "other_strengths_and_weaknesses": "**Strength**\n\nStrong performance at 2-bit compared to baselines\n\n**Weakness**\n\nLack of analysis on latency and throughput, Relatively bad performance on new models", "other_comments_or_suggestions": "I suggest to provide more details on\n\n1) experiment settings on evaluating inference efficiency.\n\n2) required time to apply proposed method\n\n3) ablation study on group size", "questions_for_authors": "- In recent models like LLaMA-3, where there is a significant performance drop, can this method be considered practical?\n- Why doesn't the paper provide results on the latest models for SliM-LLM+?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VPNIcBEQcr", "reviewer": "Reviewer_H7Hr", "summary": "This work proposes a mixed-precision quantization approach with coarse-level and fine-level partitioning via proposed Salience-Determined Bit Allocation and Salience-Weighted Quantizer Calibration, respectively. The former leverages the double-pointer search algorithm, optimizing KL-divergence between the original model and with the given weight quantized and the line search to determine local-level outliers. The proposed approach is compatible with different quantizers. Slim-LLM is evaluated on top of GPTQ and learnable quantizer Slim-LLM+ on Llama-1,2,3 model family.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer H7Hr,\n\nThank you for your feedback. We will address your questions and recommendations one by one.\n\n> Q1: The key aspect of the proposed method ... for uneven weight sensitivity. The proposed saliency measure ... However, no reference is provided.\n\nA: We would like to clarify that in lines 45–46 of the paper, we emphasize that weights \"exhibit a structured distribution\" a phenomenon that has not been overlooked in previous works. Prior studies mainly focused on compressing weights based on the sparse distribution of salience but did not propose specific strategies to handle structured distribution. This insight motivated us to develop an inference-friendly mixed-precision quantization strategy. Thank you for your suggestion—we will highlight the characteristics of structured distribution more clearly.\n\nRegarding SPQR, we explicitly reference SPQR (Line 199) in Definition 3.2 and also cite other works that adopt the same salience definition[1] [2] [3]. We would like to reiterate that the purpose of our work is not to emphasize the formulation of salience itself, but rather to identify its clustering characteristics within LLM weight matrices. SliM-LLM aims to illustrate, through Definition 3.2, the numerical impact of weight magnitude and activation distribution on the salience of LLM weight matrices and provides in-depth theoretical analysis in Theorem 1 and Appendix G, which further confirms the cause of the observed structured clustering of salience in SliM-LLM. \n\n[1]SparseGPT: Massive Language Models Can Be Accurately Pruned In One-shot. ICML, 2023.\n\n[2]PB-LLM: Partially Binarized Large Language Models. ACL 2024.\n\n[3]LLM-MQ: Mixed-Precision Quantization for Efficient LLM deployment. NIPS, 2024.\n\n> Q2: Overall, experimental protocol and choice of baselines ... be added for completeness of evaluation.\n\nA: As summarized in Figure 1, DB-LLM is a QAT-based method, which requires a significant amount of data and extended distillation tuning to complete the quantization process. In other words, it relies on standard backpropagation techniques to adjust the weights. In contrast, SliM-LLM is a totally PTQ method. More importantly, the key advantage of SliM-LLM is its ability to deploy group-wise quantization kernels directly on the GPU, and its seamless integration with AutoGPTQ. DB-LLM is an effective weight-compression method, focusing on memory compression, and is not yet capable of achieving efficient inference and real acceleration, which is the gap we aim to address.\n\n> Q3: While the approach yields quite good performance ...... [1, 2], which achieve same or better speed-ups.\n\nA: QuIP# and AQLM are both highly effective 2-bit LLM codebooks-based or vector-based quantization methods that can be improved by additional fine-tuning to restore performance (referenced in our paper). They rely on backpropagation to adjust pre-trained weights to fit the quantization scenario. In contrast, as we emphasize, SliM-LLM is a fully PTQ-based quantization method, with no training on the original weights. For a fair comparison of quantization strategies, we only compare with state-of-the-art PTQ methods, and we have achieved strong low-bit quantization performance within the PTQ framework. \n\n|#W PPL↓|Method|1-7B|1-13B|2-7B|2-13B|\n|-------|---------|----|-----|-----|-----|\n|2-bit|QuIP#|9.95|7.18|12.30|7.60|\n|2-bit|SliM-LLM+|9.68|7.18|10.87|7.59|\n\nFollowing your suggestion, we compared the non-trained QuIP# with our SliM-LLM+ at 2-bit. The results above show that even without additional weight tuning, SliM-LLM+ still achieves superior performance.\n\nRegarding the speed-ups you mentioned, we conducted a thorough investigation and reproduction of results. We found that codebook-based and vector-based quantization methods, while effective in compressing weight memory, require complex lookup and decoding operations during real deployment, resulting in inference times that are approximately three times longer than that of fp16 LLMs (https://github.com/Cornell-RelaxML/quip-sharp/issues/63). In contrast, the structured mixed-precision quantization strategy employed by SliM-LLM allows for efficient deployment in AutoGPTQ through a group-wise approach, achieving significant speed-ups. \n\n> Q4: Which dataset is used for the double-pointer search - is it the same calibration set used for SliM-LLM+ (i.e 128 samples from Wikitext-2)?\n\nA: Yes, as demonstrated in Section 4, we used the same calibration data setting (128 samples from WikiText-2), and during quantization, we selected samples randomly. The calibration selection method for SliM-LLM+ is identical to that of OmniQuant, both using random sampling of 128 data points from WikiText-2.\n\n> Q5: How much does it take to produce a quantized model with SliM-LLM?\n\nA: When applying SBA and SQC for PTQ mixed-precision quantization under single-GPU (RTX 4090) edge conditions, the quantization process for a 7B model takes approximately 25 minutes.", "claims_and_evidence": "The key aspect of the proposed method is the uneven bit width allocation according to weight saliency. While the introduced idea in the presented form is novel, the claim that it is largely overlooked in prior literature is inaccurate, as there are numerous methods that account for outliers in prior work via storing them in sparse format [1, 2] or orthogonal transformation [3, 4]. I would suggest restating the claim that this work proposes a new solution that accounts for uneven weight sensitivity.\n\nThe proposed saliency measure is identical to the one adopted in [1] to determine outliers and for sensitivity analysis (see Equation 2 and Figure 2). However, no reference is provided. \n\nThe presented empirical results show pretty reasonable performance for 2-bit quantization as compared to other uniform quantization methods. \n\n---\n\n**References**\n\n[1] Dettmers, Tim, et al. \"Spqr: A sparse-quantized representation for near-lossless llm weight compression.\" arXiv preprint arXiv:2306.03078 (2023).\n\n[2] Kim, Sehoon, et al. \"SqueezeLLM: Dense-and-Sparse Quantization.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Chee, Jerry, et al. \"Quip: 2-bit quantization of large language models with guarantees.\" Advances in Neural Information Processing Systems 36 (2023): 4396-4429.\n\n[4] Liu, Zechun, et al. \"Spinquant: Llm quantization with learned rotations.\" arXiv preprint arXiv:2405.16406 (2024).", "methods_and_evaluation_criteria": "The evaluation protocol adopted is standard for research on LLM compression.", "theoretical_claims": "This work provides primarily a practical contribution. The theoretical motivation of the proposed approach is sound.", "experimental_designs_or_analyses": "Overall, experimental protocol and choice of baselines is sensible.\nHowever, the comparison with DB-LLM [1], leveraging mixed precision, a natural competitor to the proposed approach, is absent. I believe it should be added for completeness of evaluation. \n\n---\n\n**References**\n\n[1] Chen, Hong, et al. \"DB-LLM: Accurate Dual-Binarization for Efficient LLMs.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.", "relation_to_broader_scientific_literature": "This work proposes a new way to account for uneven importance of weights via mixed precision with global and local criteria.", "essential_references_not_discussed": "DB-LLM is mentioned in the related work, but not compared with.", "other_strengths_and_weaknesses": "While the approach yields quite good performance at 2 bit compression, the performance still lags behind vector quantization methods [1, 2], which achieve same or better speed-ups.\n\n---\n\n**References**\n\n[1] Tseng, Albert, et al. \"QuIP $\\# $: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks.\" Forty-first International Conference on Machine Learning.\n\n[2] Egiazarian, Vage, et al. \"Extreme Compression of Large Language Models via Additive Quantization.\" Forty-first International Conference on Machine Learning.", "other_comments_or_suggestions": "-", "questions_for_authors": "* Which dataset is used for the double-pointer search - is it the same calibration set used for SilM-LLM+ (i.e 128 samples from Wikitext-2)? \n\n* How much does it take to produce a quantized model with SilM-LLM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper makes a solid contribution to LLM quantization, presenting a novel and effective PTQ method (SliM-LLM) that achieves state-of-the-art results, particularly in the challenging ultra-low-bit regime (2/3-bit). The core ideas are well-motivated, and the empirical validation, strengthened significantly by the rebuttal, is convincing. While the analysis of inference speed trade-offs could be more exhaustive, the demonstrated accuracy gains at high compression rates are significant. 

The authors' engagement during the rebuttal phase substantially improved the paper and addressed many key reviewer concerns. Given the strong results, methodological contribution, and positive reception (3 Weak Accepts and 1 Accept, with two scores raised post-rebuttal), the paper warrants acceptance. The authors are encouraged to incorporate the additional results and discussions from the rebuttal, particularly regarding inference efficiency trade-offs and careful positioning relative to prior work on salience, into the final version.
Decision: Accept (poster)