SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Authors: Mohammad Mozaffari
Amir Yazdanbakhsh
Maryam Mehri Dehnavi
Venue: ICML 2025 poster (2025)
Keywords: sparsity
2:4 sparsity
quantization
low-rank
lora
Primary area: deep_learning
TL;DR: We introduce SLiM, a one-shot quantized sparse plus low-rank method for compressing LLMs without retraining, reducing memory and inference costs while improving the accuracy, offering efficient deployment in memory-constrained environments.

Abstract: Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot compression methods eliminate retraining cost, but struggle to achieve accuracy comparable to dense models. This paper presents SLIM, a new one-shot compression framework that holistically integrates hardware-friendly quantization, sparsity, and low-rank approximation into a unified process. First, we formulate the quantization process using a probabilistic approach (SLIM-Quant) that enables us to apply uniform quantization. Then, we use an existing one-shot pruning method to apply semi-structured sparsity on top of the quantized weights. Finally, to compensate for the introduced aggregated quantization and sparsity error, we use a novel saliency function with unique invertible and additive features that enables us to
mathematically compute the value of low-rank adapters. SLIM improves model accuracy by up to 5.66% (LLaMA-2-7B) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods. Models compressed with SLIM achieve up to 4.3× and 3.8× on Nvidia RTX3060 and A100 GPUs, respectively. Additionally, they achieve up to 0.23× end-to-end memory reduction in comparison to their dense counterparts. We also propose an optional PEFT recipe that further improves accuracy
by up to 1.66% (LLaMA-2-13B) compared to SLIM without fine-tuning.

SLIM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Mohammad Mozaffari Amir Yazdanbakhsh 2 Maryam Mehri Dehnavi 13
Abstract
1. Introduction
Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot compression meth- ods eliminate retraining costs, but struggle to achieve accuracy comparable to dense models. This paper presents SLIM, a new one-shot com- pression framework that holistically integrates hardware-friendly quantization, sparsity, and low- rank approximation into a unified process. First, formulate the quantization process using probabilistic approach (SLIM-Quant) that en- ables us to apply uniform quantization. Then, we use an existing one-shot pruning method to apply cemi-structured sparsity on top of the quan- tized weights. Finally, to compensate for the intro- duced aggregated quantization and sparsity error, we use novel saliency function with unique in- vertible and additive features that enables to mathematically compute the value of low-rank adapters. SLIM improves model accuracy by up to 5.66% (LLaMA-227BB) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods. Models compressed with SLIM achieve up 4.3x and 3.8x layer-wise speedup on Nvidia RTX3060 and A100 GPUs, respectively. Additionally, they achieve up to 0.23x end-to-end memory reduction in comparison to their dense counterparts. We also propose an optional PEFT recipe that further improves accuracy by up to 1.66% (LLaMA-2-133 --3B) compared to SLIM with- out fine-tuning."
LLMs (Team et al., 2024: Dubey et al., 2024) have signifi- cantly advanced natural language understanding and genera- tion (Suzgun et al., 2022: Zhou et al., 2023). However, their extensive parameter counts lead to significant memory over- head and high inference costs (Frantar & Alistarh, 2023; Ma et al., 2024; Guo et al., 2024). To mitigate these challenges, recent methods (Shao et al., 2023; Sun et al., 2023; Lin et al., 2024) leverage compression reduce inference costs while aiming to retain accuracy much possible, albeit often with some trade-offs compared to dense models.
Pruning and quantization methods effectively reduce the computational and memory overhead of LLMs, but they of- ten require costly retraining on large-scale datasets to restore accuracy (Sanh et al., 2020; Park et al., 2018). The com- putational overhead of retraining, coupled with the numer- ical and optimization challenges of fine-tuning quantized models (Gholami et al., 2022), makes these approaches im- practical for many real-world applications (Frantar et al., 2022).
To climinate the need for retraining, one-shot compression methods have gained traction, achieving high accuracy using only relatively small of calibration data. State-of-the- art methods such as SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023) have demonstrated strong one-shot pruning performance. However, these methods perform well with unstructured sparsity but struggle with semi-structured patterns, such as NVIDIA's 2:4 sparsity pattern (Mishra et al., 2021), which crucial for efficient Hardware-acceleratte inference. To solve this challenge, unlike SparseGPT and Wanda that focus layer-wise error minimization, MaskLLM (Fang al., 2024) uses learnable mask that minimizes the end-to-end model error, resulting in a significant boost to the accuracy of the model at the cost of an expensive mask training phase. One-shot quanti- zation methods use various methods including row/column reordering (Frantar & Alistarh, 2023), scaling (Xiao et al., 2023; Lin al., 2024), and other transformations (Shao et al., 2023; Ma al., 2024) to mitigate model accuracy loss. However, their dependence on complex GPU kemne implementations can limit the practical benefits of quantiza- tion and add challenges to their development.
'Department Computer Science, University Toronto Google DeepMind 'NVIDIA Research. Correspondence to: Mo- hammad Mozaffari Emmozaffaai@css.torooo .ootttooto.......>......
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Cooee and data for SLIM is available at: https::/g/t //gtt/bb//// com/ com/Moh /t///Mohhammmmm ammad- Moza /ari/simm

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Although pruning and quantization are effective individually in reducing model size and inference costs, combining them provides even greater compression potential (Frantar & Al- istarh, 2023). However, jointly applying these techniques often compounds the accuracy degradation introduced by each method, leading to a significant performance gap be- tween compressed and original models. Recent work (Guo et al., 2024) attempts to mitigate this issue by jointly pruning and quantizing weights. While effective at 8-bit precision, this work struggles to recover the accuracy of dense models under lower bit- bit-width quantization. This persistent accu- racy gap highlights the need for alternative compression techniques that can maintain efficiency while minimizing quality degradation, especially at lower bit widths.
racy, making uniform quantization less attractive than more complex per-group quantization methods. Optimizing uni- form quantization is inherently non-convex problem, and recent approaches (Nagel et al., 2021) rely on grid search to find an optimal scaling factor. However, grid search can be both suboptimal and computationally expensive, often leading to poor model quality. To address this, we introduce "LIM-Quant, a probabilistic formulation of the quantiza- tion process. Our approach reformulates the inherently non-convex quantization problem into convex optimiza- tion problem, allowing for an efficient and tractable solution. This transformation significantly reduces uniform quanti- zation error, achieving accuracy levels comparable to more complex group quantization methods while retaining the computational efficiency of uniform quantization (up 6% speedup). After quantization, we apply Wanda (Sun et al., 2023), a state-of-the pruning method, to introduce dif- ferent forms of unstructured and structured sparsity on the quantized weights.
Low-rank adapters have emerged as a promising approach to mitigate the accuracy loss introduced by model compression techniques. Recent studies (Guo et al., 2023; Li et al., 2023) have explored learnable low-rank adapters to reduce weight reconstruction errors caused by pruning or quantization. However, these methods typically require an expensive re- training process on hundreds of millions of tokens to recover performance (Dettmers et al., 2023; Nikdan et al., 2024). This retraining is necessary because low-rank adapters are often initialized based on weight norms rather than their direct impact on model outputs, resulting in suboptimal starting points (Guo et al., 2023). To address this limitation, L'QER (Zhang et al., 2024a) introduces& a one-shot low-rank adapter approach that compensates for quantization loss by directly minimizing its impact on model outputs. QERA (Zhang et al., 2024b) and CALDERA (Saha et al., 2024) find closed form solutions to the problem that L'QER tries to solve. Although effective for quantization, methods such as LER struggle to maintain accuracy when combined with sparsity, underscoring the need for compression tech- niques that seamlessly integrate low-rank approximations with both sparsity and quantization"
While quantization and pruning significantly reduces the model's computational and memory footprint, the resulting compression error is unavoidable, underscoring the need for effective mitigation strategies. To address this, we propose SLIM-LoRA, a one-shot low-rank adaptation method de- signed to compensate for the aggregated error introduced by quantization and sparsity. However, determining the optimal values for adapter matrices typically requires an expensive retraining process, making it impractical for large-scale models. To eliminate this retraining overhead, we develop a novel saliency function that is both invertible and additive, enabling to mathematically derive the low-rank adapter values without iterative optimization. These properties allow SLIM-LoRA to effectively correct compression-indiee er- rors while maintaining computational efficiency.
Compared to state-of-the methods, SLIM achieves an average accuracy improvement of 5.66% on LLaMA-2-77 3.89% on LLaMA-2-13BB and 0.60% on OPT-13B under 2:4 sparsity and 4-bit weight quantization. More importantly, SL.M shifts the Pareto frontier, delivering higher model accuracy at the same total bit budget compared to existing compression techniques (up to 0.5%) and even outperform- ing dense models (up to 0.6%). These results highlight the effectiveness of SLIM in maximizing model quality under stringent resource constraints, making an appeal- ing solution for efficient large-scale deployment. Beyond accuracy improvements, SL.M achieves up to 4.3x and 3.8x layer-wise speedup on NVIDIA RTX3060 and A100 GPUs, respectively, demonstrating its efficiency on modern hardware. Finally, to further narrow the gap between com- pressed and dense models, we use an optional lightweight PEFT method, which provides up to an 1.66% additional accuracy improvement for 1-.MA-2-13BB
To address these limitations, propose SLIM, one-shot compression framework that seamlessly integrates hardware- friendly sparsity, quantization, and low-rank approxima- tion to minimize accuracy degradation while maintaining computational efficiency. We decompose the primary ob- jective of SLiM-one-hhun hurdware--fiendly sparsity and quantization with minimal accuracy loss-into three key sub- tasks. For quantization, we prioritize uniform quantization (c.g., one scale per tensor) because of its computational efficiency on commodity hardware and its simplified encod- ing/decoding process. However, standard uniform quanti- zation often introduces significant quantization errors, par- ticularly in tensors with a wide dynamic range or outliers (Gholami et al., 2022). These errors degrade model accu-
Foor a more detailed discussion of the related work, see Ap- pendix S.
2

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
2. Preliminaries
Model Compression. Model compression reduces the com- pute and memory demands of large models while maintain- ing predictive accuracy by minimizing output differences between compressed and original models. However, directly optimizing these differences across the entire model is com- putationally infeasible due to the high dimensionality of neural networks. Optimal Brain Surgeon (OBS) (Hassibi et al., 1993) simplifies this challenge by focusing on mini- mizing output discrepancies layer by layer, using calibration datasets.
SLM M-MQuu
Saliency-Besed Praning
W
W4
W
Ei
Saliency- Based
Saliency-Based
R
Low-Rank Adapter
Figure I. The SLIM weight compression pipeline consists of three main steps: (1) Quantizing weights using the symmetric SLIM-Quant algorithm, producing quantized weights W and quantization error Eq; (2) Sparsifying quantized weights w through pruning method, resulting in compressed weights we and sparsity error Es; (3) Mitigating compression errors through SLIM saliency- based low-ran low-rank approximation, generating left and right low-rank adapters L and R. Optionally, these adapters can be fine-tuned with sparse quantized weights frozen to further enhance model accuracy.
OBS applies layer- approach to compress feed- forward layers efficiently. Denoting compressed matrices with a superscript C, for a layer with input X' E bxxd,., weight W E Rdoat, and output y E bxduui, it min- imizes output differences by optimizing Equation 1. This method ensures compression fidelity and has become foun- dational for many modern compression techniques.
3. Quantized Sparse Plus Low-rank Approximation of LLMs
min 120000000 = we =min |X(VV- 2WI? WC
(L)
To achieve effective compression of LLMs while preserv- ing accuracy, SLIM combines quantization, pruning, and saliency- low-rank adapters into an integrated pipeline. First, SI.IM applies SI.IM-Quant, , a novel scheme designed to minimize quantization error, laying the foundation for subsequent pruning using methods such Wanda (Sun et al., 2023). Finally, low-rank adapters are introduced to reduce the impact of compression errors from both quanti- zation and pruning, ensuring minimal accuracy loss. The overall process is illustrated in Figure I, providing a visual summary of how these components interact to achieve ef- fective model compression. In the following subsections, we dive into the details of each step, highlighting the in- novations and contributions of SLIM-Quan the pruning strategy, and the saliency-based low-rank adapters.
Symmetric Quantization. Symmetric quantization is core technique for reducing model size and boosting com- putational efficiency. It computes the quantized matrix MM & rownd( M) ), where a is a scaling factor based on the range or norm of the matrix. This scaling ensures Mº values stay within the representable range, enabling efficient matrix multiplications with minimal overhead. However, its effectiveness depends on selecting carefully, as this choice significantly impacts precision.
AbsMax, the most common symmetric quantization method, selects a as the matrix's 's maximum absolute value, ensuring all values remain within the target range. Unfortunately, it highly sensitive to outliers; a single large value can inflate o, reducing the precision of most quantized weights. For zero-centered, bell-curved distributions typical in LLMs, AbsMax maps many weights to zero, leading to significant quantization errors.
3.1. SLIM-Quant Quantization Method
SLIM adopts symmetric weight quantization due to its low dequantization and memory overhead and ease of implemen- tation. Denoting the quantized matrices by Q superscript, Equation 2 shows the symmetric quantization formula for q- bit quantization, where a is the quantization scaling param- eter and clip(.) operator clips the input to values between [-1,
Group quantization (Alistarh et al., 2017: Gunho et al., 2022) tackles AbsMax's outlier sensitivity by assigning separate scaling factors to subgroups of the weight matrix. This approach captures local variations in weight magnitudes, reducing quantization error for non-uniform distributions. However, storing multiple scaling factors increases mem- ory usage, and subgroup-specific dequantization increase computational complexity, potentially slowing inference. The challenges of using group quantization are discussed in Appendix U.
W
e relnd(clipl 1)20-1
The objective of quantization is to reduce the weight recon- struction error shown in Equation 3, where the superscript
3

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
shows the optimal value. But the objective function in Equa-
Algorithm 1 SLIM-Quant Algorithm
tion 3 is not convex, and to our best knowledge, does not have a closed form solution.
Imput:
Weight Magnitudo PDF: Jaba. High Resolution Step Size: Thh gh. Low
Resolution Step Size: NRow Weight Matrix: W. Quantization Bitwidth: q
Output: Wysant
function EstimatEEnorcca
4: (quane(a) 1 fasa(z)]a round( +) x 2'-9 - =12dd
5: Easp(()) S0²0 faba(x))aa =//ddr
a' = arg min |WW - W||2
return Equant + Eetrp
7: function
W arg min |roundcclip( - 1)20-1- (3)
8: E = EmppyDictionary)
for a in range(0, M. Mowl de
10: Ela) EstimaleEimo((a)
11: end for
12: cloww arg mina E(a)
To solve the mean squared error (MSE) problem in Equa-
13: for a in range(Aiow Voows Klow + Nows Phigh) do
tion 3, we propose a probabilisticc reformulation as shown in Equation 4, where Q(-) and Q-) are the quantiza- tion and dequantization functions respectively, and f() is the probability distribution function (PDF) of the weight elements.
14: E(a) = EEtimateEror(a)
15: for
16: a" mig F(a)
17: Wonens = round(clsp(())) x 2
of these matched the observed weight distributions accu- rately. This discrepancy underscores the need for a more adaptable method, motivating the data-driven approach we adopt in SLIM-Quant.
a' = arg min Fq- = min |W - 2112
arg min
f(=))Q- xx| =12 (4)
To address the absence of a closed-form weight PDF, we employ numerical integration on the weight histogram to solve Equation 7. To enhance efficiency, we adopt a multi- grid strategy: starting with 10 uniform samples in the range (0, max(W)). the grid is iteratively refined around the re- gion of minimum error. This iterative process converges to the optimal with minimal computational overhead. The full procedure is detailed in Algorithm 1.
By incorporating the quantization formula from Equation 2 into Equation 4, we can simplify the integration into the sum of two terms based on the absolute value of the data: the quantization error for absolute values less than a (Equation 5) and the clipping error for absolute values larger than (Equation 6). Here, [-hx(-) represents the probability density function (PDF) of the absolute value of the weights. Equation 7 presents the simplified version of Equation 4.
Aetivation-aware SLIM-Quant, Recent work has shown that the quantization of a subset of the weight channels has higher impact on output error of the model (Xiao et al., 2023; Lin et al., 2024). We extend SLIM-Quant by incor- porating an output error minimization approach inspired by AWQ (Lin et al., 2024). Similar to AWQ, our revised algo- rithm applies scaling strategy to activations, reducing the quantization error of salient weight channels. Specifically, we scale up the weights associated with the most significant (6) channels and correspondingly scale down the related input activations. This approach maintains computational equiva- lence while effectively lowering the quantization-indueel output error. In particular, scaling approximately 1% of the channels does not alter the overall quantization parameters but significantly reduces errors in the critical channels.
Equane(a) (a)
= 1o¹ faba(z)la x round( x 21-99 - -x|dx (5)
/ 00 fats(z)la- - -xfdz
Edtspaa)= /
a' arg min Eq((a) ((((a)
= arg min Feguant (a) + (ecss(a)) (7)
However, our approach diverges from AWQ by introducing novel saliency metric that jointly considers both activa- tions and weights. We define the saliency of each channel as the product of the normalized average magnitudes of inputs and weights, expressed as Idiag(x) x W/. where x and W denote the average magnitude of activations and weights, respectively, and |.] denotes the element- wise absolute value operator. Channels with the highest saliency are scaled by a factor of s > 1, while their corresponding activations
Equation 7 can be solved theoretically by differentiating the objective function with respect to a, provided the prob- ability density function (PDF) of the weight distribution is known. However, the weight distribution of neural net- works rarely conform to standard PDFs. To verify this, we tested various candidate distributions, including Gaussian, Laplace, Pareto, q- Gaussian, and Weibull, as they are com- monly used in modeling natural data. Unfortunately, none
4

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
are scaled by 1. Although this method introduces mod- est computational overhead, that is attributed to on-the-fly adjustments of roughly 1% of activations and resulting irreg- ular memory access patterns, it yields measurable accuracy improvements. These results underscore a clear trade-off between computational complexity and model performance, highlighting the relative strength of SLIM-Quant (SLIM- Quant with output error minimization) over SLIM-Quant" (SLIM-Quan) with weight error minimization). Please note that SLiM-Quant without the superscript W denotes the weight error minimization version of SL.M--uant..
x*
C. arg max L
|I + -R)]1
arg
min [(W -wv + +R))]l (8)
L.R
C, = arr min || ERR
- -WO) - F(CR))?
= arg min [IF(((EQ + Es)) - F(CR)]I (9)
C.R.
Now, we can find F(CR) by computing the SVD of (E + Es)), and using the invertibility property of F, we can obtain the exact value of L and R.
3.2. SLIM-LORA Low-rank Adapters
After quantizing the model using SLIM-Quant, we sparsify it using an off-the-shelf one-shot pruning method such as Wanda. The combined effects of quantization and pruning of a weight matrix can be modeled as additive noise, such that y W + Eq + Es, where Eq = W - We and Es = WC - We are the quantization and sparsity errors respectively. To mitigate these errors, we introduce low- rank adapters that adjust the compressed weights such that W R WC + LR, where L E RR xr and R. E rxdut are the low-rank adapters and r is the adapter rank.
The saliency function used in SL.M must satisfy three essen- tial criteria--iveeribiiity additivity, and the effective utiliza- tion of input and weight statistics optimize weight im- portance during compression. Recent works such as Wanda, AWQ, LLM.int80, and L'ºQER suggest that the product of the magnitude of the weights and activations is a useful metric for identifying important weights during pruning and quantization. Motivated by this observation, we propose a saliency function formulation for F that meets these criteria and leverages weight-activation interactions for effective compression.
A straightforward approach minimizes the total error norm between W and WC, focusing solely on reducing the er- ror magnitude while ignoring the saliency of individual elements in the weight matrix. We call this method Naive- LoRA as it overlooks the importance of individual elements in the weight matrix. However, this method is suboptimal and can be substantially improved.
To incorporate input statistics into the saliency function, we define F(W) - diag(x)W, where x E TRd represents the average absolute value of inputs from a calibration set. This formulation ensures that the saliency function effectively weights the matrix elements based on their significance during compression, facilitating more accurate approxima- tion. By replacing F(W) in Equation 9, the optimization problem transforms into a computationally efficient solution using singular value decomposition, followed by an inverse saliency transformation to derive the left low-rank adapter (Equation 11).
To address the limitations of Naive-LoRA, we propose a novel low-rank approximation formulation that integrates weight saliency and uses a carefully designed saliency func- tion to determine optimal adapters. The saliency function (F) in our formulation needs satisfy two key proper- ties. First, it needs to be invertible, enabling the retrieval of low-rank adapters from their saliency. Second, it must be additive, meaning VA, B [(A + B) = F(A) + F(B). The additive property is crucial for isolating the saliency of low-rank adapters from the compressed matrix and distin- guishing the saliency of the error from that of the original weights. These properties ensure that the saliency func- tion can effectively isolate and optimize the contribution of low-rank adapters, forming the foundation of our proposed formulation.
R,R arg min (xag(x)(Eg))))))) + Es) - Ci 2,R
(10)
- ((dogxxi((()) + Es))
(11)
We refer to this method of computing saliency-based low- rank adapters as SLIM-LORA, a practical and efficient approach tailored for addressing compression errors in large language models. To ensure numerical stability and guar- antee the invertibility of the saliency function, an identity matrix with small values can be added to diag(x). This ad- justment is equivalent to uniformly shifting all elements of x and ensures that the saliency function remains robust even when contains near-zero elements. Algorithm 2 provides
Assuming that there exists an additive invertible saliency function F: : Igd.,.xd... - Id xdout, we need to solve Equation 8 to find the optimal adapters. Using the additive property of the saliency function F()), we can simplify Equation& 8 to Equation 9.
5

---

SLIM: : Onn-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Algorithm 2 SLIM-LORA Saliency-based Low-rank
parameter-efficient alternatives.
Adapter Computation
To overcome the challenges of fine-tuning compressed mod- els, SLIM employs seaumeter-effcceem low-rank adapters as the only tunable components during the fine-tuning phase. During this optional phase, SLIM freezes the sparse and quantized weights, enabling focused fine-tuning solely on the adapters. If the adapters are quantized, SLIM uses straight-through estimator (STE) for quantization-an aware fine-tuning and reduces its overheads with custom quanti- zation and dequantization kernels implemented in Triton. This parameter- efficceet fine-tuning method allows rapid accuracy improvements for the compressed model, requir- ing only a short phase over thousands of tokens. By limit- ing the fine-tuning process to a small subset of parameters, SLIM significantly reduces computational requirements while ensuring the model can adapt effectively to new data or tasks. This approach maintains the benefits of compres- sion while enabling efficient adaptation, as demonstrated by the significant improvements achieved during fine-tuning.
1: Input: Original Weight: W. Compressed Weight: W Calibration Input:
2: Output: L.R Saliency-based Low-rank Adapters
3: Ee Eqg + Ey = W W
// Compute Evror II Average over all the samples
4: & mean(X))
5: x x+ (1))
# Shift values avoid zeros in
6: Sc Ex)g(x)Ec
# Compute error saliency
7: L. R = SVD(Se)
# Law-rank approsimation
8: L d
I/ I/ Converting saliency to weight
9: R= R
a comprehensive overview of the steps involved in comput- ing saliency-based low-rank adapters using SLIM-LoRA, ensuring reproducibility and clarity.
3.3. Low-rank Adapter Quantization
While pruning and quantizing the weights significantly re- duce the model's computation and memory requirements (~8x memory footprint reduction), incorporating full- precision low-rank adapters reintroduces overhead, partially offsetting these gains. To address this, we applied 4-bit quantization to compress the adapters. This step ensures that the compression efficiency achieved through weight pruning and quantization is preserved, while maintaining the performance benefits of the low-rank adapters.
4. Experimental Results
Models, Datasets, and Evaluation. 3 We evaluate SLIM on the OPT (Zhang al., 2022) and LLaMA-2 (Touvron et al., 2023) model families, both of which serve as standard base- lines in model compression studies (Ma et al., 2024; Frantar & Alistarh, 2023; Sun et al., 2023). Model accuracy is as- sessed on a range of zero-shot downstream tasks, including MMLU (Hendrycks et al., 2020), Piqa (Bisk al., 2020), Arc-Easy, Are-Challenge (Clark et al., 2018), WinoGranne (Sakaguchi et al., 2021), and OpenBookQA (Mihaylov et al., 2018) 8). For zero-shot evaluations, we utilize the Language Model Evaluation Harness (Gao et al., 2024) framework. In line with prior work (Sun et al., 2023; Frantar & Alis- tarh, 2023; Ma et al., 2024), we also report the perplexity of the models on a language modeling task on the WikiText2 (Merity et al., 2016) dataset, provided in Appendix G.
Quantizing low-rank adapters poses unique challenges due to the long-tailed distribution of their elements, which limits the effectiveness of advanced non-group quantization meth- ods, such SLIM-Quant. To address this, we adopt AbsMax group quantization scheme for the adapters, where groups of 128 elements share the same quantization parame- ter. By grouping elements, this method effectively captures the distribution's 's variability while minimizing quantization error, striking a balance between accuracy and compression. This approach not only reduces the adapter overhead by 4x but ensures that their contribution to overall model compres- sion and performance is retained; as demonstrated in our experimental evaluation.
Baselines. We compare SLIM against state-of-ther oftee----hherrr art one- shot pruning methods, including Wanda (Sun et al., 2023), SparseGPT (Frantar & Alistarh, 2023), and Magnitude Prun- ing (Han et al., 2015), as well as one-shot quantization techniques like OPTQ (Frantar et al., 2022), OmniQuant (Shao et al., 2023), AffineQuant (Ma et al., 2024), I. L'QERR (Zhang et al., 2024a), and AbsMax. Additionally, we ex- tend Joint Sparsification and Quantization (JSQ) (Guo et al., 2024) to support 4-bit weight quantization and include it in our experiments. To ensure fairness, we use the op- timal hyperparameters reported for each method, or the default byperparameters if not explicitly reported. For a thorough description of the notations used to show the dif-
3.4. Optional Post-compression Fine-tuning
Fine-tuning large language models post-compression has many challenges because the high parameter count and memory demands of traditional methods make them com- putationally prohibitive. For example, using a simple op- timizer such as ADAMW leads to 4x additional memory overhead store gradient and optimizer states, rendering these approaches impractical for compressed models. Thus, parameter-efficient fine-tuning is essential for preserving the benefits of compression while avoiding excessive com- putational and memory costs. This necessity is further highlighted by the results in Section 4, which illustrate the overheads of traditional fine-tuning and the advantages of
All the experiments in the paper were run the University of Toronto
6

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
ferent variants of SLIM, please see Table 4 in Appendix A. For more details about the hyperparameters used in different experiments, please see Appendix T.
its saliency-aware design. Further details on the fine-tuning process and its overhead are provided in Appendix K, il- lustrating its practicality for enhancing compressed model performance.
L' (Zhang et al., 2024a) and OATS (Zhang & Papyan, 2024) are the two independent and concurrent compression methods utilizing zero-shot low-rank adapters to enhance model accuracy. Our approach, SLIM, significantly di- verges from them in several key aspects. First, we employ -basendc---sed low-rank adapters to mitigate compression loss in quantized and sparse models, whereas L.R is tailored exclusively for quantization, resulting in reduced accuracy when combined with sparsity, as demonstrated in the subsequent subsections, and OATS is designed for unstructured sparsity only without quantization, which does not have acceleration support on NVIDIA GPUs. Second, we introduce SLiM-Quant, which lowers the overhead and complexity of group quantization compared to methods like Q.R Finally, SLIM compresses and fine-tunes low- rank adapters efficiently to minimize overhead. In contrast, L'QER and OATS rely on full-precision low-rank adapters, which incur additional overhead and do not benefit from the parameter-efficient fine-tuning proposed in our work.
SLIM with MaskLLM. MaskLLM is the state-of-the-art pruning method designed for 2:4 sparsity. It keeps the origi- nal weights in the model intact, while finding the optimal 2:4 masks for the model through a mask training phase. As a result, it can be combined with SLIM to boost the accuracy of the models even further. Table 3 summarizes the average accuracy results of MaskLLM on six zero-shot downstream tasks and its perplexity on WikiText2 dataset.
Large Compressed vs. Small Dense Models. This section compares large compressed models with dense models of equivalent parameter size, offering guidelines for configu- ration selection under hardware constraints. We focus on 2:4 sparsity due to its hardware acceleration support and evaluate the OPT model family, which spans a wide range of sizes for comprehensive analysis.
We analyze model performance by plotting average accuracy against parameter size, calculated as detailed in Appendix L. This visualization enables direct performance comparison between models with equal number of bits.
Accuracy Results. We evaluate the accuracy of SLIM and other state-of-the-art pruning and quantization methods across 2:4 and unstructured sparsity benchmarks, high- lighting SLIM's superiority in Table 1. SparseGPT and Group OPTQ, designed to work together, achieve compet- itive performance. For other advanced quantization meth- ods, we pruned models using Wanda and quantized the sparse checkpoints with Group Abs.ax AWQ, OmniQuant, and AffineQuant, reporting the best results (detailed in Ap- pendix H). In particular, methods such as OmniQuant and AffineQuant struggle to quantize OPT-350M, often resulting in NaN values. Moreover, AWQ, OmniQuant, AffineQuant, and L. L.EQER encounter out-of-memory (OOM) errors when compressing models on a single A100-40GB GPU. While ISQ performs well for the LLaMA-2 family, its difficulty compressing the OPT family limits its broader applicability.
Figure 2 presents the accuracy results of the OPT model family across different compression methods. The x-axis represents the model parameter size in gigabytes, while the y-axis denotes accuracy (higher better). The results demonstrate that SLIM-LORA 2, both with and without fine- tuning, consistently outperforms dense models and other compression techniques at the same parameter size. Notably, compressed models achieve higher accuracy than dense models of equivalent size, highlighting the effectiveness of the proposed method. This trend underscores the advantage of SLIM-LoRA @ in maximizing model efficiency under strict hardware constraints.
Speedup. Leveraging sparsity and quantization enhances GPU resource utilization, enabling faster model inference. Following Wanda's 's experimental setup. we evaluate the speedup achieved across different model layers and sizes. Similar to Wanda, AWQ, and QuaRot (Ashkboos et al., 2024), we focus on consumer-grade GPUs and conduct our experiments on NVIDIA RTX 3060 GPUs. Speedup results for NVIDIA A A100 GPUs are provided in Appendix J.
The progression from Nai to SLIM-LORA and SLIM-LORA demonstrates the benefits of incorporat- ing weight saliency into low-rank adapters and applying quantization for reducing overhead. While Noi improves model accuracy across different sizes, SLIM- LoRA achieves additional gains by effectively leveraging the saliency of the weights in the adapter design. Extending this, SLIM-LORA @ applies quantization to the low-rank adapters, further minimizing overhead with minimal impact on accuracy, adding negligible improvements or degradation to the accuracy of the model.
SL.MM achieves notable speedups through optimized sparse and quantized matrix multiplication, utilizing Sparse Marlin (Frantar et al., 2024) integrated with vLLM (Kwon et al., 2023). For inference, we adopt small batch sizes during decoding, as recommended by prior works (Xia et al., 2023; Zheng al., 2022). Dense Quantized Marlin or Py kernels handle the low-rank adapters based on their quan- tization status. Figure 3 highlights the speedup achieved
Table 2 demonstrates how lightweight fine-tuning (FT)
improves the accuracy of both SLIM-LORA and Naive-
LoRA, with SLIM-LoRA exhibiting greater gains due to
7

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table Average zero-shot accuracy LLaMA-2 OPT models with 50% % sparsity and 4-bit weight quantization. Best Method"
indicates the best quantization method out of Group AbsMax, AWQ, OmniQuant, and AffineQuant. indicates better performance.
Pruning/LoRA
Weight
OPT
LLaMA-2
Method
Quantization
125M
350M
1.38
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43.4
45,5
48.3
48.7
56.6
60.8
2:4 Sparsity
Magnitude
Group AbsMax
32.19
31.94
33.82
33.43
34.81
34.68
44.64
44.18
SparseGPT
Group OPTQ
33.70
33.38
38.75
40.15
44.32
45.64
45.49
51.05
Wanda
Best Method
33.39
32.79
38.43
40.00
43.41
44.07
44.86
48.94
JSQ
JSQ
32.30
31.84
35.23
32.89
38.06
37.24
44.80
50.20
L'QER
Group AbsMax
33.34
31.68
36.68
38.11
41.37
OOM
43.77
OOM
Naive-LoRA
SLIM-Quam
34.28
33.38
38.36
41.21
44.91
45.25
48.45
51.94
SI.IM-LRRA
MMMM-uumm"
34.62
34.36
40.61
42.73
45.99
46.09
51.15
54.94
SLIM-LORA
SLIM Quant"
34.43
34.30
40.11
42.37
46.33
46.24
51.02
53.55
50% Unstructured
Magnitude
Group Aas
33.34
33.51
32.12
39.90
36.44
32.33
47.03
51.04
SparseGPT
OPTQ
35.10
35.13
38.72
43.43
46.97
47.38
51.09
55.94
Wanda
Best Method
35.11
33.89
41.02
42.89
46.52
46.84
53.62
56.76
JSQ
JSQ
32.14
30.34
38.86
35.48
42.75
30.73
52.25
57.00
L'RER
Group AbsMax
34.45
34.45
38.38
41.28
45.08
OOM
50.60
OOM
Naive-LoRA
SLIM-Quande
34.77
34.23
40.40
43.37
46.64
47.30
51.52
55.33
SLIM-LoRA
SLIM- #LiM-Quamt"
35.20
35.32
41.85
43.48
47.08
47.96
54.26
57.85
SI.M--LRRA
S1.1M- MMM-Quant"
35.35
35.13
41.74
43.63
47.16
47.86
54.18
57.33
Table 2. Effects of fine-tuning on the average zero-shot accuracy LLaMA-2 models with. 1 indicates better performance.
Table 3. Accuracy (Acc) and perplexity (PPL) of MaskLLM com- bined with SLIM on LLaMA-2-7B.
Pruning/LoRA Method
Weight Quantization
LLaMA-2
Pruning/LoRA
Weight Method Quantization
-LaMA-2-7B
13B
Acc
PPL
Dense
56.6
60.8
Dense
56.6
5.47
50% 2:4
MaskLLM
49.7
7.3
Naive-LoRA FT SLIM-LORA FT
SLIM-Quant W
50.89
55.70
Naive-LoRA
52.3
6.6
SLIM-Quants MaMM-Quntt* W
52.12
56.60
SLIM-LoRA
52.2
7.0
SLIM-LORA 9 + FT
48.31
56.50
Naive-LoRA + FT
52.6
6.6
SLIM-LORA FT
52.9
6.6
50% Unstructured
Naive-LoRA SLIM-LORA + FT
ELIM-Quant's
52.90
57.08
MaskLLM
Group AbsMax
49.2
7.6
SLIM-Quant
54.69
57.96
Naive-LoRA
SLIM--uant
50.5
7.3
SLIM-LORA + FT
SLIM- "LIM-Quant" W
53.57
57.78
SLIM-LoRA
SLIM-Quant
51.4
7.5
Naive-LoRA FT
MLIM-Quant
51.2
6.9
SLIM-LORA + FT
SLIM SLaM-Quant
52.1
6.8
across different LLaMA-2 layers compared dense, un- quantized models. The breakdown of the speedup, showing the contribution of the quantization and sparsity, is demon- strated using brighter and darker colors respectively. Larger matrices, such as those in feed-forward modules, consis- tently yield greater speedups, aligning with trends detailed in Appendix J.
Additional Experiments. Due to the page limit, we provide additional experiments for a comprehensive evaluation in the appendix.
An evaluation of SLIM with input quantization using FP8 is provided in Input Quantizatoin (Appendix B).. The results show that input quantization has a minimal impact on the accuracy of the models using SLIM.
Model Memory Reduction. We evaluate SLIM's memory reduction on A100 GPUs and our experiments show that S achieves 0.23x and 0.23x memory reduction on LLaMA-2-78 and LLaMa-2-11 13B respectively. The reduc- tions for SLIM are 0.33x and 0.34x x respectively.
A comparison between weight error minimization and ac- tivation error minimization in SLIM -uqunt provided in SLiM-Quant" vs. SLIM-Quant" (Appendix C).. The exper-
8

---

SLIM: : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Accuracy vs. Model Parameter Size (2:4 Sparsity)
SLiM Speedup on RTX 3060
Batch Size 16
Batch Size 32
Batch Size 64
48.8
Accuracy
3.93
40
2
Model
35
SLiM-LoRA
SLIM-LORA SLIM-Quant (PT)
WANDA Quantizacion
SparscGPT OPTQ
10
LLaMAs-2-70B LLaMA-2-131
Parameter Size (GB)
3.4
Figure Accuracy results the OPT family across different com- pression methods (f indicates better performance). At equal pa- rameter size, SI.MM outperforms both dense models and other compression techniques, demonstrating that model compression with SLIM yields superior performance under the same budget.
M 1.1.919.
iments show that the gap between output error minimization and weight error minimization not significant.
Up-Projattion Aftention
Down. Projection Down-Projection
Up- Up-Projection Projection
Self-Atteote Atteation
Down- Projection
Down-Praiection
Up- Up-Projectind Projection Self- Aftantiio Dooj-Projoction
Down- Projection
We evaluate SL.M on sparse-only and quantized- only models isolate their effectiveness. sults in Additional Sparse-only Results (Appendix D) and Additional Quantization-oni only Results (Appendix E) demon- strate that SLIM and LLIM-uuant consistently outperform state-of-the-art compression methods.
Re-
Figure 3. LLaMA-2 family of models speedup (x) using SLIM compared original dense unquantized model NVIDIA RTX-3060. t shows higher speedup. The brighter color shows the contribution of quantization to the total speedup.
The Language Modeling Experiments (Appendix G) evalu- ates SLIM across sparse and quantized, sparse-only, and quantized-only models WikiText-2. The results align with the accuracy trends reported in the main paper, further validating the effectiveness of SLIM.
Compression Costs (Appendix N) details the time re- quired to compress models of various sizes across dif- ferent methods. [Rank Analysis (Appendix O) explores how rank choices in low-rank adapters impact com- putational and memory costs, well as model accu- racy. SSparsity Analysis (Appendix D) analyzes the ef- fects of different sparsity ratios on model compression. Lastly, Effects of Calibration Sample Count (Appendix P) evaluates the influence of calibration sample counts on the accuracy calibration-bart methods.
The Fine-tuning Costs (Appendix K) shows that SLIM re- duces fine-tuning overhead from over 36 days for 13B pa- rameter models to just 14 hours on a single GPU, demon- strating its practicality and efficiency.
We provide comparison between Sparsity vs. Quantization (Appendix I) to show that combining 50% sparsity and 4-bit quantization helps achieve better compression results comparison to solely using 2-bit quantization, while maintaining a similar compression ratio (~-8).
Finally, more details on per-task accuracy results on down- stream tasks reported in different tables, please refer to Weights & Biases report at https:://brr http///tt.../ooowhttt
5. Conclusion
Additional speedup results for SLIM on NVIDIA A100-40GB GPUs provided in the Additional Speedup Results (Appendix J). A theoretical analysis of computation and memory reductions can be found in the Computation Reduction Analysis (Appendix M) and Memory Reduction Analysis (Appendix L)) highlighting the efficiency of SLIM.
We introduced SLIM, a one-shot quantized sparse plus low-rank approximation method for large language mod- els, optimizing both efficiency and accuracy. By com- bining quantization, sparsity, and saliency-based low-rank adapters, SL.M achieves substantial reductions in memory and computation while preserving competitive performance. SLIM outperforms state-of-the-ar methods in accuracy.

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Impact Statement
Schoenick, C., and Tafjord, O.
Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
The SLIM framework advances model compression by enabling efficient, one-shot quantization and sparsity for large language models (LLMs) while maintaining accuracy through low-rank approximation. This has the potential to make LLMs more accessible and sustainable by reducing their computational and energy requirements, thereby en- abling deployment on wider range of devices, such as smartphones and edge computing platforms, and contribut- ing to environmental sustainability. However, the increased accessibility of compressed models raises important consid- erations regarding potential accuracy trade-offss in critical applications, such as healthcare or legal systems, and the eth- ical implications of broader Al deployment, including risks of bias propagation and misuse. To address these challenges, is crucial to ensure that efficiency gains do not compro- mise model reliability and that appropriate safeguards, such as transparency and rigorous evaluation, are in place for responsible Al development.
arXiv preprint
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.in80):: 8-bit matrix multiplication for transformers at scale. NeurIPS, 2022.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv preprint arXiv:2305.14314 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 2773, 2024.
Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17433 2024.
Acknowledgments
Frantar, E. and Alistarh, D. Optimal brain compression: quantization and
A framework for accurate pruning. NeurlPS, 2022.
This work was also supported in part by NSERC Discov- ery Grants (RGPIN-06516, DGECR00303), the Canada Research Chairs program, Ontario Early Researcher award, the Canada Research Chairs program, the Ontario Early Researcher Award, and the Digital Research Alliance of Canada (www, alllaanceccal aineccannn..... ca). We extend our gratitude towards Ray Hung, Behrooz Zarebavani, Joan Puigcerver, James Laudon, Suvinay Subramanian, and Cliff Young for reviewing the paper and providing insightful feed- back. We also thank the extended team at Google DeepMind who enabled and supported this research direction.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICML. 2023.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Optq: Accurate quantization for generative pre-trained trans- formers. In ICLR, 2022.
Frantar, E., Castro, R. L.., Chen, J., Hoefler, T., and Alis- tarh, D. Marlin: Mixed-precisioe auto-regressive paral- lel inference on large language models. arXiv preprint arXiv: 11.448..1770 11733, 2024.
References
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. Qsgd: Randomized quantization for communication- efficient stochastic gradient descent. In NeurlPS, 2017.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L.., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L.,, Schoelkopf, H., Skowron, A., Sutawika,
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free b-b inference ro- tated Ilms. arXiv preprint Mrxxiv2444.004555 2024.
L., Tang, E., Thite, A., Wang, B.. Wang, K., and Zou, A. A framework for few-shot language model evaluation,
07 2024. URL https: /tap::://eeeeeoee zenodo. org/ heggrrecord//// 12608602.
Bambhaniya, A. R., Yazdanbakhsh, A., Subramanian, S., Kao, S.-C., Agrawal, S., Evei, U., and Krishna, T. Pro- gressive gradient flow for robust n: M sparsity training in transformers. arXiv preprint arXiv:2402.04474 0474444444 2024.
Gholami, A., Kim, S., Dong, Z.,, Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for effi- cient neural network inference. In Low-Power Computer Vision. Chapman and Hall/CRC, 2022.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
Gunho, P., Baeseong, P., Se Jung, K., Byeongwook, K., Youngjoo, L., and Dongsoo, L. nuqmm: Quantized mat- mul for efficient inference of large-scale generative lan- guage models. arXiv preprint arXiv:2206.0m 095557, 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A.,
10

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Guo, H., Greengard, P., Xing. E. P., and Kim, Y. LQ- LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning, arXiv preprint arXiv:2311. 2023.
Lu, Y., Agrawal, S., Subramanian, S., Rybakov, O., De Sa, C., and Yazdanbakhsh, A. Step: learning n: M structured sparsity masks from scratch with precondition. In Inter- national Conference on Machine Learning, pp. 22812- 22824. PMLR, 2023.
Guo, J., Wu, J., Wang, Z., Liu, J., Yang, G., Ding, Y., Gong, R., Qin, H., and Liu, X. Compressing large language models by joint sparsification and quantization. In ICML, 2024.
Ma, Y., Li, H., Zheng, X., Ling, F., Xiao, X., Wang. R., Wen, S., Chao, F., and Ji, R. Affinequant: Affine transformation quantization for large language models. arXiv preprint 1.XXv:2403.12544 2024.
Han, S., Pool, J., Tran, J. and Dally, W. Learning both weights and connections for efficient neural network. NeurlPS, 2015.
Merity, S., Xiong, C., Bradbury, J., and Socher, Pointer sentinel mixture models, 2016.
Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv 2022.
Harma, S. B., Chakraborty, A., Kostenok, E., Mishin, D., Ha, D., Falsafi, B., Jaggi, M., Liu, M., Oh, Y., Subrama- nian, S., al. Effective interplay between sparsity and quantization: From theory practice. arXiv preprint arXiv:2405.20933, 2024.
preprint arXiv:2209.0544:
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 1809 000999 2018.
Hassibi, B., Stork, D., and Wolff, G. Optimal brain surgeon: Extensions and performance comparisons. NeurIPS, 1993.
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Ac- celerating sparse deep neural networks. arXiv preprint 1rXiv:2104.08375 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint wxXiv:2009003300 2020.
Mozaffari, M., Li, S., Zhang, Z., and Dehnavi, M. M. MKOR: Momentum Enablldd Fronecker-Facterr Basod Optimizer Using Rank-1 Updates. In NeurIPS, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank Adaptation Large Language Models. arXiv preprint (rXiiv2206.09660 2021.
Mozaffari, M., Yazdanbakhsh, A., Zhang, Z., and Dehnavi, M. M. Slope: Double-pruned sparse plus lazy low- rank adapter pretraining of 1lms. arXiv preprint 2rX 2024.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In CVPR, 2018.
Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., Van Baalen, M., and Blankevoort, T. A white pa- per on neural network quantization. arXiv preprint 1FXh::200600825, 2021.
Kwon, W. Li, Z., Zhuang, S., Sheng. Y.. Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023.
Nikdan, M.. Tabesh, S., and Alistarh, D. Rosa: Accu- rate parameter-efficien fine-tuning via robust adaptation. arXiv preprint arXiv:2401.0460 2024.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. NeurIPS, 1989.
NVIDIA Corporation. NVIDIA CUTLASS. https:// gi titthub.. /wwo///VIII///// wwtp//sttllaaaaasaaaa 2025.
Li, Y., Yu. Y., Zhang, Q., Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large language models based on low-rank and sparse approxi- mation. In ICML, 2023.
Park, E., Yoo, S., and Vajda, Value-aware quantization for training and inference neural networks. ECCV, 2018.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awarr weight quantization for on-device Ilm compression and acceleration. MI.ys, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.

---

SLIM: : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Saha, R., Sagan, N., Srivastava, V., Goldsmith, A., and Pilanci, M. Compressing large language models using low rank and low precision decomposition. NeurIPS, 2024.
Wang, S. and Kanwar, P. BF The secret to high performance Cloud TPUs. http://bit. 3WEtCGm, 2019.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint urXiv:191.00377 1910093770,,1.. 2019.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Xia, H., Zheng. Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, I., Flash-llm: En- abling cost-effective and highly-efficient large genera- tive model inference with unstructured sparsity. arXiv preprint 22t::::300.00288. 2023.
Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adap- tive sparsity by fine-tuning. NeurlPS, 2020.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308. 13137, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023.
Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In ICMI., 2018
Zhang, C., Cheng, J., Constantinides, G. A., and Zhao, Y. Lqer: Low-rank quantization error reconstruction for Ilms. arXiv preprint M.XXx::242202245 2024a.
Singh, S. P. and Alistarh, D. Woodfisher: Efficient second- order approximation for neural network compression. NeurlPS, 2020.
Zhang, C., Wong, J. T., Xiao, C., Constantinides, G. A., and Zhao, Y. Qera: an analytical framework for quantization error reconstruction. arXiv preprint arXiv:2410.0mm 060400 2024b.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama. https://bit..yy/ss.an/ jamas, 2023. URL https::/hugginge: Gerebras/SSimPajaannn 627B.
Zhang, S. and Papyan, V. Oats: Outlier-aware pruning through sparse and low rank decomposition. arXiv preprint (xXxv:2409.13652 2024.
co/ datasets/
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01063 2022.
Sun, M., Liu, Z.., Bair, A., and Kolter, J. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 16995, 2023.
Suzgun, M.. Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D.,, and Wei, J. Challenging big-bench tasks and whether ch can solve them. arXiv preprint arXiv:2210.09261, 2022.
Zheng, N., Lin, B., Zhang, Q. Ma, L., Yang. Y., Yang, F. Wang, Y., Yang, M., and Zhou, L. (SparTA]: (Deep- Learning} model sparsity via [Tensor-with-Spaasity Attribute). In OSDI, 2022.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint 05530000 2024.
Zhou, J.. Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L.. Instruction-followingg evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Tillet, P., Kung, H. T., and Cox, Triton: an intermediate language and compiler for tiled neural network compu- tations. In MAPL 2019: Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019.
Touvron, H., Martin, I.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
12

---

SLIM: Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Appendix
A. Notations
Table 4 details the key notations, particularly for Section 4.
Table 4, Group quantization slow-down on different LLaMA-2 and LLaMA 3.1 models.
Term
Description
Naive-LoRA
A one-shot low-rank adapter that minimizes the norm of the difference between the original and the compressed weights.
SLIM-LoRA
A saliency-based one-shot low-rank adapter that minimizes the saliency the differ- ence between the original and the compressed weights.
Q (Superscript)
4 indicates that the compression method quantizes the low-rank adapters as well.
FT
FT shows short fine-tuning phase 300,000 tokens from the C4 dataset.
B. Input Quantization
We evaluate SLIM with 8-bit input quantization to assess its impact accuracy. We use AbsMax uniform quantization with single parameter per input tensor and apply FP8 format (Micikevicius et al., 2022) for weight quantization. The choice between E4M3 and E5M2 depends on the tensor's maximum value; if it exceeds E4M3's range, we switch to E5M2 for greater expressivity. Next, we examine how input quantization affects model accuracy.
Table 5 presents accuracy results for different SLIM variants with input quantization. A comparison with Table 9, which reports accuracy without input quantization, reveals minimal accuracy loss, demonstrating SLIM's robustness. For further validation, extend these experiments to language modeling tasks (Appendix G).
Table Average zero-shot accuracy LLaMA-2 and OPT models with 4-bit weight quantization and 8-bit input quantization with 50% weight sparsity. 1 indicates better performance.
Pruning/LoRA Method
Weight Quantization
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43.4
45.5
48.3
48.7
56.6
60.8
50% 2:4
SL.M--LRRA SLIM-LORA +TT
SLIM- SLIM-Quant" W
34.85
34.27
40.29
42.58
45.78
46.21
50.99
54.66
SLIM-Quant 35.28
34.33
41.14
43.29
46.44
47.33
51.77
56.28
SLIM-LORA
SLIM-Quant' 34.30
33.85
39.92
41.99
46.08
45.94
50.70
53.56
SLIM-LORA FT
SLIM-Quant 34.92
34.80
41.66
43.69
46.03
46.87
50.26
56.28
50% Unstructured
SLIM-LORA
SLIM- MLMM-uunt
35.12
34.86
41.94
43.53
47.27
47.70
54.28
57.82
SI.IM-LORA + FT
SL.M-Quan W 35.18
35.30
42.37
44.02
47.01
48.52
54.43
57.70
SLIM-LORA
ULIM-Quant" W
35.26
34.67
41.48
43.46
47.25
47.76
53.91
57.16
SLIM-LORA + +FT
SLIM-Quant
35.52
35.31
42.66
44.50
47.08
48.53
53.23
57.55
C. SLIM-Quant vs. SLIM-Quant
Table 6 compares the average accuracy different models when using SLIM-Quant with weight error minimization (SLIM- Quant W) and activation-awerr aware output error minimization (SLIM- ) SLIM-Quant outperforms SLIM-Quant" by a small gap, while adding computational overhead with irregular memory access patterns at inference time.
13

---

SLIM: Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table 6. Average zero-shot accuracy of LLaMA-2 and OPT models with 50% sparsity and 4-bit weight quantization for SLIM-Quant W SLIM-Quant".
Pruning/LoRA Method
Weight
OPT
LLaMA-2
Quantization
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43.4
45.5
48.3
48.7
56.6
60.8
2:4 Sparsity
SLIM-LORA
SLIM-Quant W
34.62 34.63
34.36
40.61
42.73
45.99
46.09
51.15
54.94
SLIM-LORA
SLuM-Quant
34.36
40.29
42,45
45.71
46.24
51.22
55.05
50% Unstructured
SLIM-LoRA
LLLM-Quant
35.20
35.32
41.85
43.48
47.08
47.96
54.26
57.85
SLIM-LORA
SL.IM-Quant
35.20
34.78
41.29
43.31
47.09
47.86
54.46
57.97
D. A Additional Sparse-only Results
To evaluate the isolated impact of sparsity on model accuracy, we disable quantization and benchmark Magnitude Pruning, SparseGPT, and Wanda, alongside low-rank approximations like Wanda-SVD and SLIM Our experiments assess both 50% unstructured sparsity and 2:4 structured sparsity patterns.
Table 7 shows the accuracy results for sparse models. Magnitude Pruning performs the worst, while Wanda and SparseGPT achieve comparable results, with larger accuracy gaps for semi-structured sparsity. Low-rank adapters improve accuracy, with SLIM leveraging saliency-based approximation for superior performance. A brief fine-tuning phase further boosts the accuracy of low-rank approximations.
Table 7. Average zemo-shot accuracy of ILaMA-2 and OPT models with pruning. The quantization is disabled in this experiment. indicates hetter performance.
Pruning/LoRA Method
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43.4
45.5
48.3
48.7
56.6
60.8
2:4 Sparsity
Magnitude
32.6
31.8
35.4
33.9
36.4
30.7
31.2
32.0
SparseGPT
33.8
33.2
37.7
41.3
45.2
45.6
47.3
52.3
Wanda
34.0
32.5
38.3
40.5
43.2
44.1
46.1
49.7
SLIM-Naive
34.1
34.1
40.4
42.8
46.0
45.9
51.6
55.8
SLIM-Naive
34.8
34.5
41.3
43.4
46.5
47.2
52.4
56.9
SLIM-LORA
34.5
32.9
40.7
43.1
46.4
46.3
51.4
56.1
SLIM-LORA FT
35.1
34.9
41.5
43.8
46.5
47.3
51.6
56.4
50% Unstructured
Magnitude
33.3
33.7
34.0
40.6
35.8
30.9
32.6
31.9
SparseGPT
35.5
35.1
39.6
43.5
47.4
47.8
53.3
57.3
Wanda
35.0
34.5
41.1
42.9
46.5
46.8
52.7
57.2
SLIM-Naive
35.3
35.2
41.9
44.1
47.5
47.8
54.9
58.5
SLIM-Naive FT
35.74
35.7
42.7
44.6
47.8
48.4
54.9
58.7
SLIM -LoRA
35.2
35.1
42.0
44.1
47.7
48.2
55.0
58.8
SLIM -LoRA +
35.9
35.7
42.5
44.7
47.7
48.4
55.0
58.8
14

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
E. Additional Quantization-only Results
To evaluate the impact of SLIM-Quant and low-rank compensation in SLIM, we conduct experiments without sparsity, testing quantization schemes like Group AbsMax, OPTQ, AWQ, OmniQuant, AffineQuant, L'QER, and SLiM-Quant To enhance accuracy, we add low-rank adapters SLiM-Quant and Group AbsMax, optimizing either error saliency (SLIM-LoRA) reconstruction error norm (Naive-LoRA). Other quantization methods cannot incorporate low-rank adapters due to conflicting weight/activation update rules.
Table 8 presents the quantization results. Adding low-rank adapters to Group AbsMax significantly boosts model accuracy, outperforming most advanced methods. While SLiM-Quant alone is not designed for high accuracy. its integration with SLIM variants achieves results comparable to or better than Group AbsMax with low-rank adapters, highlighting the value of co-design in compression methods. Furthermore, lightweight fine-tuning phase with SLIM-Quan delivers state-of-the-art accuracy.
Table Average zero-shot accuracy of LLaMA-2 and OPT models with quantization. The sparsity is disabled this experiment. indicates better performance.
Quantization
Low-rank Adapter
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43,4
45.5
48.3
48.7
56.6
60.8
OPTQ
35.64
36.46
42.83
44.20
47.46
48.24
53.53
59.80
AWQ
36.16
31.83
42.98
45.28
48.45
48.76
53.97
OOM
OmniQuant
35.46
NaN
42.15
44.71
46.65
OOM
54.33
OOM
Affineuuant
35.73
NaN
42.62
44.92
47.91
OOM
54.52
OOM
Group AbsMax
35.45
36.67
42.57
44.79
48.30
48.49
55.56
60.12
Group AbsMax
L. L'QRER
34.75
35.63
40.60
44.22
46.90
OOM
55.95
OOM
Group AbsMax SLIM-Naive
36.30
36.58
43.07
45.13
48.26
48.72
56.23
60.53
Group AbsMax
SLIM-LORA
36.18
36.72
42.89
45.65
48.45
48.89
55,99
60.16
SLIM-Quant
31.98
36.46
36.19
40.08
45.61
38.27
31.11
30.51
SLIM- MEMMt-untt" W
SLiM-Naive
35.29
36.02
42.48
45.01
47.75
48.38
55.96
60.85
A1.IM-Quant
SL.M-LLRA
35.69
36.42
42.59
45.26
48.18
48.52
56.26
60.59
SLIM-Quant
SLIM-LORA FT
35.91
36.61
43.29
45.58
48.29
49.04
56.51
60.65
F. Addiiiona Fine-tuning Results
To complement the results in Section 4, we provide accuracy measurements for PEFT- based fine-tuning of low-rank adapters on the OPT and I.LaMA-2 model families in Table while showing the accuracy results without fine-tuning for comparison. The results confirm the previously observed trend: lightweight fine-tuning enhances the accuracy of all baselines, with SLIM-LoRA achieving the most significant improvements due its saliency-based design.
G. Language Modeling Experiments
We evaluate all benchmarks from Section 4 and Appendix B ,D, and E on the WikiText2 language modeling task. Tables 10 and 1l show perplexity results for 4-bit quantized models with 2:4 and unstructured sparsity, respectively. Table 12 summarizes the results for 8-bit input quantization. To examine sparsity and quantization independently. Tables 13 and 14 report results for pruning-only and quantization-only models. Consistent with Section 4, SLIM achieves superior performance across all settings.
15

---

SLIM Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table Effects fine-tumi fine-tuning on the average zero-shot accuracy of LLaMA-2 OPT models with. t indicates better performance.
Pruning/LoRA Method
Weight
OPT
LLaMA-2
Quantization
125M
350M
1.3B
2.7B
6.7B
13B
13B
Dense
35.9
37.1
43.4
45.5
48.3
48.7
56.6
60.8
50% 2:4
Noive-LoRA Naive-LoRA + FT
SLIM-Quant W
34.28
33.38
38.36
41.21
44.91
45.25
48.45
51.94
SLiM-Quant W
34.41
34.70
39.72
42.88
46.16
46.76
50.89
55.70
SLIM-LoRA
SLIM- SLiM-Quant" 34.62
34.36
40.61
42.73
45.99
46.09
51.15
54.94
SI.1M-LRRA
MLIM-Quante 35.03
34.58
41.11
43.35
46.71
47.25
52.12
56.60
SLIM-LORA
SLIM-Quamt
34.43
34.30
40.11
42.37
46.33
46.24
51.02
53.55
SLIM-LORA FTT
SLIM-Quant
34.92
34.85
41.84
43.87
46.31
46.91
48.31
56.50
50% Unstructured
Naive-LoRA Naive-LoRA FT
SLIM-Quant 34.77
34.23
40.40
43.37
46.64
47.30
51.52
55.33
SLIM-Quan W
35.70
35.47
41.89
44.16
47.08
47.78
52.90
57.08
SLIM-LORA
SLIM- SLIM-Quant 35.20
35.32
41.85
43.48
47.08
47.96
54.26
57.85
SI.IM-LORA + FT
SL.M- MuMM-QuantW
35.59
35.71
42.37
44.58
47.69
48.26
54.69
57.96
SLIM-LORA
SLIM-Quam
35.35
35.13
41.74
43.63
47.16
47.86
54.18
57.33
S + FT
SLIM-QU W
35.65
35.67
42.74
44.54
47.48
48.40
53.57
57.78
Table 10. Perplexity of LLaMA-2 and OPT models with 2:4 sparsity and 4-bit weight quantization WikiText-2 dataset language modeling task. 4 indicates better performance.
Pruning/LoRAA Method
Weight Quantization
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
27.66
22.00
14.62
12.47
10.86
10.13
5.47
4.89
Magnitude
Group AbsMax
5.1E2
4.4E2
1.2E3
1.3E3
3.6E2
4.9E2
86.34
8.98
SparseGPT
Group OPTQ
78.18
59.86
27.36
18.62
15.31
13.25
12.07
9.46
Wanda
Group AbsMax
1.8E2
1.3E2
32.76
24.48
17.29
16.86
14.36
9.38
Wanda
AWQ
9.3E1
8.1E5
29.56
22.91
16.28
16.72
12.79
OOM
Wanda
OmniQuant
9,7E1
NaN
33.61
25.89
19.09
OOM
12.77
OOM
Wanda
AffineQuant
9.7E1
NaN
30.32
1.6E3
16.85
OOM
12.21
OOM
JSQ
JSQ
3.5E3
1.7E4
1.1E2
6.6E2
36.94
2.3E2
12.68
8.70
Naive-LoRA
Group AbsMax SLIM-Quant" 83.08
69.23
50.02
20.52
16.05
12.83
13.12
8.04
6.38
Naive-LoRA
58.69
27.06
20.92
14.29
13.20
8.19
7.09
Naive-LoRA FT
SLIM-Quant" 51.82
38.84
20.59
16.19
13.13
12.55
6.96
6.01
SLIM-LORA
SLIM-Quant" W
57.91
50.09
19.64
15.65
12.71
12.13
7.77
6.80
SLIM-LORA + FT
SLIM-Quant
44.03
37.32
18.25
14.89
12.68
12.06
6.70
6.60
SLIM-LORAQ
SI.IM-Quant
53.09
46.96
19.62
16.01
12.48
12.15
7.75
6.96
SLIM-LORA + +TT
SLIM-Quant"
42.80
37.39
18.38
15.40
12.65
12.35
7.08
6.36
16

---

SLIM Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table Perplexity 2 OPT models with unstructured sparsity 4-bit weight quantization on WikiText-2 dataset
language modeling task. indicates better performance.
Pruning/LoRA
Weight Quantization
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
27.66
22,00
14.62
12,47
10.86
10.13
5.47
4.89
Magnitude
Group AbsMax
3.2F2
1.1E2
3.2F3
3.6E2
7.2F2
5,4E3
17.18
6.77
SparseGPT
Group OPTQ
42.60
34.19
21.41
14.30
12.15
11.26
8.28
5.92
Wanda
Group AbsMax
62.64
39.60
19.93
15.01
12.31
12.46
6.80
5.75
Wanda
AWQ
42.49
3.8E5
18.80
14.67
12.17
12.34
7.28
OOM
Wanda
OmniQuant
43.55
NaN
20.58
15,82
13.29
OOM
7.40
OOM
Wanda
AffineQuant
43.66
NaN
19.40
14.94
12.39
OOM
7.21
OOM
JSQ
JSQ
4.2E3
3.3E4
31.78
1.7E2
19.97
8.9E5
7.17
6.19
Naive-LoRA
Group AbsMax
40.37
30.99
17.02
13.91
11.68
11.38
6.12
5.28
Naive-LoRA Naive-LoRA FT
SLiM-Quant"
46.66
33.90
19.46
15.36
12.16
11.41
6.56
5.58
SLiM-Quant
38.05
29.27
17.52
14.39
12.28
11.84
6.10
5.28
SLIM-LORA
SLIM-Quant 39.62
31.51
16.52
13.65
11.42
10.82
6.16
5.36
SLIM-LORA + FT
SL.M-
34.92
28.67
16.16
13.66
11.83
11.47
5.36
5.19
S
SLIM-Quant
38.79
30.16
16.64
13.82
11.43
10.80
6.26
5.58
SLIM-LORA FT
SLiM-Quant*
35.17
28.31
16.46
13.96
11.42
10.80
5.94
5.46
Table 12. Perplexity of LLaMA-2 and OPT models with 4-bit weight quantization and 8-bit input quantization. J. Liniicates better performance.
Pruning/LoRA Method
Weight Quantization
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
27.66
22.00
14.62
12.47
10.86
10.13
5.47
4.89
50% 2:4
SLIM-LORA
Ml.M--uant
48.4
49.6
16.6
16.2
12.9
12.3
7.2
6.5
SLIM-LORA FT
SLIM-Quant W
39.8
37.5
18.3
15.5
12.8
12.1
6.6
5.8
SLIM-Quant
54.2
50.8
20.8
16.8
13.0
12.4
7.8
7.0
SLIM-LORA FT
SLIM-Quant W
43.4
39.1
19.3
16.0
13.1
12.6
7.1
5.8
50% Unstructured
SLIM-LORA
SLIM-Quame Quant W SLIM-Quant' W
36.8
31.1
16.8
14.0
11.7
10.9
6.1
5.4
SLIM-LORA FT
33.8
28.6
16.5
14.0
12.0
11.5
5.9
5.2
SLIM-LORA Q
SL.M-Quant
39.5
31.3
17.3
14.2
11.8
10.9
6.3
5.6
SLIM-LORA FT
SLIM-Quant
35.6
29.1
17.0
14.3
12.2
11.7
6.2
5.5
17

---

SLIM Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table 13. Perplexity of LLaMA-2 and OPT models with pruning on WikiText-2 dataset language modeling task. The quantization disabled in this experiment. J indicates better performance.
Pruning/LoRA
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
27.66
22.00
14.62
12.47
10.86
10.13
5.47
4.89
2:4 Sparsity
Magnitude
341.5
417.1
427.2
1.2E3
264.1
4.0B4
9.1E4
2.0E5
SparseGPT
60.7
50.7
23.8
17.2
14.1
12.9
10.2
8.3
Wanda
81.6
116.0
27.8
21.4
16.0
16.4
12.0
8.5
Naive-LoRA
46.9
45.0
18.8
15.2
12.5
12.9
8.1
6.5
Naive-LoRA + FT
39.6
35.1
15.0
16.3
12.7
12.3
6.5
5.7
SLIM-LoRA
45.2
43.6
18.6
15.0
12.4
12.6
7.3
6.2
SLIM-LoRA + FT
37.1
33.7
17.0
14.2
12,4
12.1
6.4
5.8
50% % Unstructured
Magnitude
193.4
97.8
1.7E3
265.2
968.7
2.4E4
9.9E4
1.1E5
SparseGPT
36.7
31.8
17.6
13.4
11.5
11.1
6.5
5.6
Wanda
39.3
36.4
18.3
14.3
12.0
12.3
6.4
5.4
Naive-LoRA
33.3
29.1
16.3
13.5
11.5
11.2
6.2
5.4
Naive-LoRA FT
31.9
27.5
16.3
13.8
12.0
11.6
5.8
5.1
SLIM -
32.7
29.0
15.9
13.2
11.2
10.8
5.9
5.2
SLIM LLRA FT
31.0
26.8
15.5
13.1
11.6
11.0
5.8
4.7
Table 14. Perplexity of LLaMA-2 and OPT models with quantization WikiText-2 dataset language modeling task. The sparsity disabled in this experiment. 1 indicates better performance.
Quantization
Low-rank Adapter
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
27.66
22.00
14.62
12.47
10.86
10.13
5.47
4.89
OPTQ AWQ
33,0
24.4
16.0
13.0
11.3
10.3
4.9
29.1
2.7E5
14.9
12.7
11.0
10.2
6.0
OOM
OmniQuant
30.2
NaN
15.8
13.3
11.6
OOM
5.7
0OM
AffineQuant
28.7
NaN
14.9
12.6
11.0
OOM
5.7
00M
Group AbsMax
35.1
23.3
15.5
12.9
11.1
10.3
5.4
4.7
Group AbsMax
Naive-LoRA SLIM-LORA
30.4
22.9
15.1
12.7
11.0
10.2
5.3
4.7
Group AbsMax
29.3
22.8
15.0
12.7
10.9
10.2
5.2
4.7
SLIM-Quant
1.4E3
26.0
1.7E3
33.1
31.0
6.7E2
1.3E5
7,8E4
SLIM- M
Naive-LoRA
32.1
24.1
15.6
13.4
11.2
10.5
5.4
4.8
SLIM--uant
SLIM-LORA
30.8
23.1
15.2
12.9
11.1
10.3
5.4
4.8
SLIM--uant
SLIM-LORA + FT
30.7
23.5
15.3
13.3
11.6
10.0
5.3
4.7
H. Additional Sparse and Quantized Results
In Section 4, we provided the accuracy results for different pruning and quantization methods. When using Wanda for pruming, we only reported the best guantization method out Group AbsMax, AWQ, OmniQuant, and AffineQuant. For completeness, we have provided the accuracy achieved by each of these quantization methods separately in Table 15.
18

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Methods like OmniQuant and AffineQuant encounter difficulties in quantizing OPT-350M, resulting in NaN values. Additionally, approaches such as AWQ, OmniQuant, and AffineQuant cause memory issues (OOM) when attempting compress the models on a single A100-40GB GPU.
Table 15. Average zero-shot accuracy of L.aMA-2 and OPT models with 2:4 spansity and 4-bit weight quantization. indicates hetter performance.
Pruning/LoRA Method
Weight
OPT
LLaMA-2
Quantization
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Dense
35.9
37.1
43.4
45.5
48.3
48.7
56.6
60.8
2:4 Sparsity
Wanda
Group AbsMax
33.27
32.79
37.47
39.45
42.95
43.64
43.89
48.94
Wanda
AWQ
33.33
31.50
38.43
40.00
43.41
44.07
44.86
OOM
Wanda
OmniQuant
33.37
NaN
37.35
39.39
41.50
OOM
43.95
OOM
Wanda
AffineQuant
33.39
NaN
37.48
33.51
42.88
OOM
44.62
OOM
50% Unstructured
Wanda
Group AbsMax
34.67
33.89
40.38
42.77
45.88
46.60
51.76
56.76
Wanda
AWQ
35.11
31.57
41.02
42.89
46.52
46.84
50.68
OOM
Wanda
OmniQuant
34.85
NaN
39.84
42.16
44.67
OOM
50.51
OOM
Wanda
Affineuuant
34.64
NaN
41.23
42.68
46.05
OOM
53.62
OOM
I. Sparsity vs. Quantization
A natural question that arises compressing models is whether it is more efficient to reduce the model size through pruning or quantization. To answer this question, we conduct a set of experiments, which evaluate the perplexity of different models under three different conditions, all with around 8x model size reduction factor: (1) 2-bit weight quantization with no sparsity. (2) 4-bit weight quantization with 50% unstructured sparsity, and (3) 4-bit weight quantization with 50% 2:4 sparsity. We have used S with SLIM--uant in all the experiments. The accuracy and perplexity results of these experiments are summarized in Tables 16 and 17, showing that combining sparsity and quantization yields better results in comparison to quantization-oni -nnl settings with lower bitwidth.
Table 16. Average accuracy of different models WikiText-2 dataset using different pruning and quantization schemes. 1 indicates hetter performance. Combining sparsity and quantization provides better accuracy results comparison to solely using quantization.
OPT
LLaMA-2
Quantization
Sparsity
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
2-bit
33.5
32.5
38.5
39.2
43.8
44.4
42.4
44.9
4-bit
2:4
34.6
34.4
40.6
42.7
46.0
46.1
51.2
54.9
4-bit
50% Unstructured
35.2
35.3
41.9
43.5
47.1
48.0
54.3
57.9
Table Perplexity of different models on WikiTexx-2 dataset using different pruning and quantization schemes. indicates better performance. Combining sparsity and quantization provides better accuracy results in comparison to solcly using quantization.
OPT
LLaMA-2
Quantization
Sparsity
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
2-
116.2
169.7
35.1
27.1
16.2
15.0
12.5
11.7
4-bit
2:4
47.5
45.6
18.8
15.7
12.4
12.1
7.2
6.5
4-bit
50% Unstructured
36.3
29.9
16.3
13.7
11.4
10.8
6.0
5.4
19

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
J. Additional Speedup Results
Section presents speedup of SLIM on consumer-grade GPUs, while this section provides results on NVIDIA A100-40GB GPUs. Figure 4 summarizes the speedup for the LLaMA-2 and LLaMA-3.1 model families, including LLaMA- 3.1-405B, highlighting SLIM scalability large models. As with consumer-grade devices, larger models achieve higher speedups.
Additionally, the breakdown of the speedup, showing the contribution of quantization and sparsity, is demonstrated using brighter and darker colors, respectively.
SLiM Speedup on A100-40GB
FP16 LoRA
INT4 LoRA
Batch Size 16
Batch Size 32
Batch Size 64
2.2
LLaMA-2-7B
1.7
1.2
1.1
1.0
1.0
0.8
0.7
0.6
0.6
0.5
0.5
0.5
0.6
0.3
0.3
0.2
LLaMA 2- 13B
2.00
2.0
1.9
1.8
1.70
1.6
1.6
1.5
1.50
1.3
1.3
1.25
1.1
0:30
100
0.8
0.9
0.7
0.6
0.8
0.5
0.50
0.4
0.3
0.35
0.00
2.7
2.7
2.5
LLaMA-2-TT 1AMA-2-70B
9.3
2.1
2.2
2.0
2.0
1.7
1.6
1.6
1.8
1.5
1.3
1.1
1.0
0.9
0.6
6.5
LLaMA-3.0 3..1-00BB
3.8
3.6
a.s
3.2
3.2
2.9
2.9
2.8
2.6
2.7
2.22
2.1
2.4
2.2
2.4
2.2
2.3
2.0
1.9
1.9
1.5
1.0
6.5
6.0
Up- Hp-Prrjection
Self- Attention
Down-P Projection
Up-Projection Self- Attention
Down- Projeccion
Up-Projee Ap-PPrjeection Self- Atfenttnnno Down- Projeccio
Figure SLIM speedup for LLaMA-2 family of models on NVIDIA A100-40GB GPUs. The brighter color shows the contribution of quantization to the total specdup.
20

---

SLIM : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
K. Fine-tuning Costs
Fine-tuning compressed models can recover lost accuracy, but the high parameter count leads to substantial time and memory costs. In our experiments, we fine-tuned models with low-rank adapters, where the quantized weights are frozen and only the adapters are fine-tuned. This results in a more parameter-efficieet approach, reducing both memory and computational costs. When no low-rank adapter is used, the ttright-througg estimator (STE) fine-tunes the quantized weights.
Table 18 presents the fine-tuning results for 300,000 tokens from the C4 dataset, using batch size of 64 and sequence length of 1024 on a single H100 GPU. Fine-tuning models without low-rank adapters took 12 hours for 125M parameter models and over 36 days for 13B parameter models. Given these high costs, completing fine- tuning was challenging with our limited resources. contrast, using low-rank adapters and freezing the sparse quantized weights made fine-tuning more efficient, enabling us report accuracy results Table 1.
Table 18. The required time for fine-tuning the models with single H100 GPU on 300,000 tokens from the C4 dataset with batch size 64 and a sequence length 1024.
Pruning Method
Weight Quantization
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Magnitude
Group AbsMax OPTQ
SparseGPT
12h
43h
164h
36th
866h
867h
842h
844h
Wanda
Group AbsMax
SL.IM-Qaan SL.M--Naiv
W
1.5h
3h
6h
8h
16h
18h
14h
14h
SLIM-LORA
SLIM-Quant* W
L. Memory Reduction Analysis
SLIM prunes and quantizes the models and adds additional low-rank adapters to them. Additionally, it supports quantization methods for the low-rank adapters to reduce their overhead. In the following, we propose an analysis of the reduced memory when using SI.IM and other pruning and quantization methods.
Assuming the hidden dimension of model is d and the low-rank adapter ratio used in the model is of rank 1. Furthermore, by denoting the number of transformer blocks with and the vocabulary size of the model by V and by denoting the ratio of the un-orojection and down-projection layers in the model by we can get the memory reduction as the ratio of Compressed Model Size from equation 12.
Dense Model Size
rA4d + 2dP + dV
Memory Reduction n2442/22+ 4 20 2d + 2d(dr + dra)) + dV
(12)
Table 19 summarizes the memory reduction of different pruning and quantization methods. Please note that when using low-rank adapters (in Naive-LoRA and SLIM-LoRA), assume a rank of 0.1.
Table 19. Theoretical memory reduction (x) of different compression methods across various OPT and LLaMA models. In Quantized SLIM the Jow-rank adapters are also quantized.() indicates better performance.)
Compression
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
SparseGPT + OPTQ
0.40
0.30
0.25
0.17
0.15
0.14
0.15
0.14
Wanda + AbsMax
0.40
0.30
0.25
0.17
0.15
0.14
0.15
0.14
Naive-LoRA + AbsMax
0.50
0.42
0.38
0.31
0.30
0.29
0.31
0.30
SLIM-LORA SLIM-Quant
0.50
0.42
0.38
0.31
0.30
0.29
0.31
0.30
SLIM-LORA @ + SLIM-Quant
0.42
0.33
0.28
0.20
0.19
0.18
0.19
0.18
21

---

SLIM: Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
M. Computation Reduction Analysis
SLIM and other compression methods reduce the number of floating point operations (FLOPs) at the inference of models. Additionally, the low-rank adapters used in SLIM and Wanda SVD can add additional computational overheads the inference of the models. Following JSQ (Guo al., 2024), in this section, we provide an analysis on the FLOP reduction in the inference of different methods. It is noteworthy that even though quantization reduce the memory overhead of models, since all the computations are done in floating point format, it does not lead to reduction in the computation of the inference.
Assuming the hidden dimension of model is d and the low-rank adapter ratio used in the model is of rank r < 1. Furthermore, by denoting the number of transformer blocks with n and the vocabulary size of the model by V and by denoting the ratio of the up-projection and down-pun cown-projeccion layers in the model by a, we can get the memory reduction as from equation 13, where is the batch size, and is canceled in the numerator and the Dense Inference FLOOP Count
the ratio of Compressed Inference FLOP Count denominator of the equation.
n 2bd2a) bdV
FLOP Reduction n(4bd2/2 + 2bd + 2bdaa/2 + 2b(d2r + dPra)) + bdV
(13)
Table 20 summarizes the FLOP reduction of different compression methods. As it can be seen, the overhead of adding the low-rank adapters (r 0.11) in SLIM-LoRA and Naive-LoRA is not significant.
Table 20. Compute (FLOP) reduction ratios (x) of different compression methods across various OPT and LLaMA models. In Quantized SLIM, the low-rank adapters are also quantized. (f indicates better performance.)
Compression
OPT
LLaMA-2
Method
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
SparseGPT + OPTQ
1.52
1.66
1.75
1.91
1.94
1.96
1.95
1.97
Wanda + AbsMax
1.52
1.66
1.75
1.91
1.94
1.96
1.95
1.97
Naive-LoRA + AbsMax
1.32
1.39
1.43
1.50
1.51
1.52
1.49
1.49
SLIM-LORA SLIM-Quant
1.32
1.39
1.43
1.50
1.51
1.52
1.49
1.49
SLIM-LORA ? + SLIM-Quant
1.32
1.39
1.43
1.50
1.51
1.52
1.49
1.49
N. Compression Costs
The computational cost of compression methods varies depending on their complexity. While all approaches can compress a single layer at a time, the memory usage is similar across methods, as each stores only one layer the GPU's global memory. Techniques like Wanda, which rely on matrix multiplication, are faster than more complex methods like SparseGPT, which computes the inverse Hessian matrix for each layer. Adding low-rank adapters to Wanda-SVD and SLIM increases computational complexity due to the need for singular value decomposition (SVD), making them comparable to SparseGPT in terms of computation.
Table 21 summarizes the time required to compress various models using the discussed methods. Methods incorporating low-rank adapters (SLIM and Wanda-SVD) generally take longer to compress due to their higher complexity. Interestingly, SparseGPT's compression time is comparable to methods with low-rank adapters, despite only performing pruning and quantization. The saliency-baas based approach in SLIM does not add significant overhead compared to Wanda-SVV SVD. maintaining efficiency despite its added complexity.
22

---

SLIM: : Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Table 21.
The required compresion time for different models and compression methods using single H100 GPU.
Pruning Method
Weight
OPT
LLaMA-2
Quantization
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
Magnitude
AbsMax
1s
1s
1s
1s
2s
4s
28
4s
SparseGPT
OPTQ
Im
2m
5m
11m
22m
41m
25m
46m
Wanda
SLIM- -uMtu
0.5m
1m
3m
5m
8m
13m
8m
14m
Wanda-SVD
SLIM-Quant
Im
2m
7m
13m
33m
60m
38m
67m
SLIM
SLIM-Quad
1m
2m
7m
13m
34m
63m
39m
68m
O. Rank Analysis
The key hyperparameter in low-rank approximation is the rank of the adapters. While increasing the rank reduces approximation error, also leads to higher computational and memory overhead. Therefore, is crucial to analyze the trade-off between the accuracy improvements and the overhead introduced by the chosen approximation rank.
Assuming the rank of the low-rank adapter is rd, where square feed-forward layer, the low-rank adapters are represented L. R² RRdxrd, resulting in memory overhead of ((2rd²) for storing them. To compute XLR, where X' E 6 is the input with batch size of b, the computational complexity is (2brd²). Given that the original memory and computational complexity of the layer are (dd²) and Obd?), respectively, the overhead introduced by the low-rank adapters becomes negligible when r < 1.
is a fixed factor and is the dimension of the weights in
Figure O-a shows the average zero-shot accuracy of the OPT-6.7B 7B and LLaMA-2-7B models for various ranks. As expected, increasing the rank leads to improved model accuracy. Based on these results, rank of 0.1 provides a substantial boost in accuracy without introducing significant overhead to inference.
Adapter Rank Sensitivity Analysis
Calibration Data Sensitivity Analysis
20.0
20.0
ILaMA-2-7B Perplexity
LoiV
LLaMA 2-BB- Perplexity
17.5
SLAM-LoRA
17.5
Denee
15.0
15.0
Naive-LoRA
12.5
12.5
Group SparseGIT
LL.i
10.0
10.0
7.5
7.5
0.05
0.10
0.15
0.20
0.25
20
40
60
80
100
120
Adapter Rank
Number of Calibration Samples
(a)
(b)
Figure 5. Sensitivity amalysis for the rank of the adapter (a) and the number of calibration samples (b) for different one-shot compression methods. For Naive-LoRA and SLIM-LoRA, we have used the SLIM-Quant quantization method, and for the SparseGPT, we have used the Group quantization version of OPTQ.
P. Effects of Calibration Sample Count
Similar to previous work (SparseGPT, Wanda, AWQ, OmniQuant, and AffineQuant), SLIM leverages a set of calibration data from the C4 dataset to assess weight saliency for pruming and low-rank approximations. Figure O-b illustrates the perplexity of LLaMA-2-7B using varying numbers of calibration samples. As shown, SLIM demonstrates low sensitivity to the number of calibration samples, making effective even in scenarios with limited data.
23

---

SLIM: Onc-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Q. Sensitivity to Calibration Dataset
Similar to other pruning and quantization methods such as Wanda, SparseGPT, OPTQ, and AWQ, SLIM relies on a calibration dataset evaluate weight saliency. The C4 (Raffel al., 2019) and SlimPajama (Soboleva al., 2023) datasets are among the most commonly used calibration for LLM compression. Table 22 presents the perplexity results for SLIM-LORA and SLIM-Quam across different calibration datasets. The results indicate that SLIM is largely insensitive to the choice of dataset, achieving comparable accuracy regardless of the calibration dataset used.
Table 22. Perplexity of different models on WikiText-2 dataset using SLIM-LoRA with 4-bit quantization using SLIM-Quant with different calibration datasets. indicates better performance.
Calibration Dataset
OPT
LLaMA-2
125M
350M
1.3B
2.7B
6.7B
13B
7B
13B
50% 2:4
C4
57.91
50.09
19.64
15.65
12.71
12.13
7.56
6.50
SlimPajama
46.27
44.77
19.35
16.04
12.56
12.32
7.15
6.49
50% Unstructured
C4
39.62
31.51
16.52
13.65
11.42
10.82
6.16
5.36
SlimPajama
36.49
29.94
16.64
14.08
11.61
11.02
5.99
5.34
R. Sparsity Analysis
To analyze the impact of sparsity on model accuracy, we conduct experiments on LLaMA-2-13B with 4-bit quantization, pruming it to varying sparsity ratios. Figure 6 presents the perplexity results for SLIM-LoRA with SLIM-Quant, SparseGPT with OPTQ, and Wanda with Group AbsMax. As expected, increasing the sparsity ratio leads to higher perplexity, indicating a trade-off between compression accuracy. Notably, SLIM-LoRA combined with SLIM-Quan maintains competitive accuracy up to 60% sparsity, whereas other methods experience noticeable degradation lower sparsity levels.
Sparsity Sensitivity Analysis
LLaMA-2-137 Perplexity
20.0
SLiM-LoRA + SLiM-Quant
17.5
SparseGPT + OPTQ
Wanda + Group AbsMax
15.0
12.5
10.0
7.5
0
10
20
30
40
50
60
70
80
Sparsity Ratio (%)
Figure 6. Sparsity analysis on LLaMA-2-13R model using perplexity on WikiText-2 datasct. | indicates better performance.
S. Related Work
SLIM combines model pruning and quantization for compression, complemented by zero-shot low-rank adapters to recover lost accuracy. This section reviews related work on these topics.
24

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
S.1. Pruning
Eliminating redundant weights reduces computation and memory costs during inference. Optimal Brain Damage (OBD) (LeCun et al., 1989) leverages second-order information of the loss function to identify the least important weights but is computationally prohibitive for large language models (LLMs) (Mozaffari et al., 2023). WoodFisher (Singh & Alistarh, 2020) approximates the Hessian matrix using Kronecker Factorization to mitigate this overhead but struggles to scale to LLMs.
Optimal Brain Surgeon (OBS) (Hassibi et al., 1993) evaluates weight matrices layer-wise using the layer-wise Hessian matrix to preserve layer outputs. However, the cubic growth in the cost of inverting the layer-wise Hessian with model size renders this approach impractical for LLMs. Optimal Brain Compression (OBC) (Frantar & Alistarh, 2022) addresses the OBS-defined compression problem using greedy algorithm, while SparseGPT reformulates it as sparse regression problem. Wanda introduces lightweight method based on weight and activation magnitudes to identify unimportant weights without updating their values.
In addition to post-training sparsity, a recent line of work targets sparsity during training (Lu et al., 2023; Mozaffari et al., 2024; Bambhaniya et al., 2024); however, their applicability is limited because of the expensive costs of training.
S.2. Quantization
Quantizing all elements in a matrix is challenging due to the significant impact of outliers on the model (Dettmers et al., 2022). Group quantization (Alistarh et al., 2017; Gunho et al., 2022) addresses this by quantizing small groups of a weight matrix with a shared quantization parameter, but introduces challenges discussed in Appendix U.
AbsMax (Jacob al., 2018) with round-to-nearest (RTN) is the simplest quantization scheme for matrix elements. OPTQ (Frantar et al., 2022) minimizes layer-wise error using an approach akin to OBS. AWQ (Lin et al., 2024) shifts the challenge of quantizing salient weights to activations, while SmoothQuant (Xiao et al., 2023) balances quantization error between weights and activations, enabling input quantization. OmniQuant (Shao et al., 2023) improves accuracy with learnable clipping and channel scaling. AffineQuant leverages equivalent affine transformations to reduce quantization error, and QuaRot (Ashkboos et al., 2024) uses rotations to eliminate outliers during quantization.
Advanced methods like JSQ (Guo et al., 2024) jointly prune and quantize weights to 8 bits but struggle to recover accuracy in low bit-width quantization, limiting their utility. An analysis of the interplay between sparsity and quantization can be found in (Harma et al., 2024).
S.3. Low-rank Adapters
Low-rank adapters were first introduced to LLMMs to reduce the overhead of fine-tuning (Hu et al., 2021; Mozaffari et al., 2024). Q-LoRA (Dettmers et al., 2023) extended this approach by quantizing weights before fine-tuning, allowing the process to recover accuracy lost during quantization. LQ-LoRA (Guo et al., 2023) further improved Q-LoRA by initializing the adapters using the SVD of the quantization error. LoSparse (I. et al., 2023) has a similar approach as I LQ-LORA, but for sparsity, initializing the low-rank adapters to the norm of the pruning error. RoSA (Nikdan et al., 2024) expands the learning capability of the model by adding both low-rank and sparse adapters to the model. This approach adds an extra sparse matrix multiplication to the inference, increasing the adapter overhead even further. However, all these methods require hundreds of millions of tokens for fine-tuning, making them costly and not comparable to one-shot pruning and quantization methods, or methods that use much shorter fine-tuning phases.
L'QER (Zhang et al., 2024a) avoids fine-tuning by using one-shot low-rank adapters to mitigate quantization error. However, it performs poorly when combined with sparsity, resulting in a significant accuracy gap between the compressed and dense models. More recent methods, QERA (Zhang et al., 2024b) and CALDERA (Saha et al., 2024) find closed-fore solutions to the problem discussed in I. I.2QER, but they still do not support sparsity.
T. Settings and Hyperparameters
To ensure a fair comparison and robust performance, SLIM utilizes calibration data and fine-tuning datasets under the same conditions as leading one-shot pruning and quantization methods. Similar to Wanda, SparseGPT, and OPTQ, SLIM leverages calibration data to extract statistics and assess weight saliency. Specifically, we use 128 sequences sampled from the widely-
25

---

SLIM: : One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
used C4 (Raffel al., 2019) dataset for calibration. Additionally, for all fine-tuning experiments, we employ 300,000 tokens from the C4 dataset improve model accuracy post-compression. This standardized approach to data usage ensures that SLIM operates under the same conditions as its peers, enabling fair evaluation of its compression and fine-tuning performance.
SLLM-Quant uses the histogram of the weight elements to find the optimal scaling factor. The use of the histogram reduces the overhead of finding the optimal parameter by sharing the error computations between the elements that fall into the same histogram bin. The number of histogram bins provides a trade-off between the computational overhead and the accuracy of the integration. We set the number of bins in the histogram max(512, min( day Kdaut 20, 000) to achieve an accurate approximation of the PDF of the data.
1000
We standardize our experimental setup by detailing the quantization scheme, group quantization parameters, and low- rank adapter configurations to ensure reproducibility and comparability across methods. All quantization methods in the experiments follow 4-bit weight-only quantization scheme. Consistent with prior work (OPTQ, OmniQuant, AffineQuant, etc.), group quantization uses group size of 128. For experiments involving Naive-LoRA and SLIM-LoRA, we set the adapter rank to 10% of the model's hidden dimension unless stated otherwise. These standardized configurations ensure consistency with prior work and enable fair comparison of SLIM against baseline methods.
For fine-tuning the models, we utilized the Hugging Face Trainer (Wolf, 2019). The AdaFactor (Shazeer & Stern, 2018) optimizer was employed during the fine-tuning process, accompanied by linear learning rate scheduling. The optimization and learning rate scheduling parameters were set to their default values in the Hugging Face Trainer. To prevent numerical overflow and divergence, we used BFloat- 16 data types (Wang& & Kanwar, 2019) available on NVIDIA A100 GPUs during fine-tuning. The training was conducted with a local batch size of and a gradient accumulation factor of 64 to reduce memory overhead. Weight updates for the sparse and/or quantized weights, as well as the corresponding biases, were disabled. Due to our limited resources, did not tune any of the hyperparameters aimed at improving fine-tuning speed or accuracy; tuning these parameters is planned for future work.
U. Group Quantization Challenges
Group quantization allows sharing the same quantization parameters for a small group of the elements in the quantized matrix, leading to smaller errors. But, using group quantization adds additional challenges to the training and inference of the model, e.g. more complicated implementation and additional memory and compute overheads.
The state-of-the-art group quantization GPU kernel, dense and sparse Marlin (Frantar et al., 2024), consists of thousands of lines of CUDA code optimized for only limited number of GPU architectures, showcasing the amount of effort needed to implement a version of group quantization. Furthermore, other libraries and frameworks, such as Triton (Tillet et al., 2019) and CUTLASS (NVIDIA Corporation, 2025) do not provide support for 4-bit group quantization, limiting its flexibility and possibility of modification.
Furthermore, using group quantization can lead to an additional overhead during matrix multiplication, since more parameters need to be loaded for dequantizing each group. As example, Table 23 shows the slow-down of using group quantization on the down-projection matrices in different LLaMA-2 and LLaMA-3.1 models on NVIDIA A100-40GB GPU, with a batch size of 16.
Table 23, Group quantization slow-down (x) different LLaMA-2 and LLaMA 3.1 models. J indicates worse. Model
LLaMA-2-7B
LLaMA-2-13B
LLaMA-2-70B 1LaMA-3.1-405.
Slow-Down (x)
0.94
0.95
0.95
0.94
26


Reviews:
[{"review_id": "iwSW2oF33j", "reviewer": "Reviewer_SvGK", "summary": "The paper introduces SLIM, a one-shot post-training compression framework for large language models. It integrates three components: (1) quantization, (2) pruning for hardware‑friendly sparsity, and (3) a low‑rank adapter to compensate quantization errors. Experimental results show that SLIM improves model accuracy by up to approximately 5.66% and delivers significant GPU inference speedups, making it an effective solution for deploying large models in resource‑constrained environments.  \n\n\n**update after rebuttal**  \n\nI appreciate the authors' clarifications and the additional experimental results.\n\nFrom my initial review through to the current rebuttal phase, my primary concern has consistently been the authors’ decision to minimize quantization error rather than directly targeting output error. In the first-round rebuttal, the authors did not provide a convincing justification for this design choice. In my reply-to-rebuttal comment, I reiterated this concern and requested a more thorough comparison between the two approaches.\n\nIn their latest response, the authors presented new experimental results aimed at addressing this issue. However, these results indicate that minimizing output error actually yields better accuracy than minimizing weight error. This directly contradicts the rationale for their initially chosen approach and undermines the justification for the proposed methodology. Based on this evidence, I remain unconvinced that the current approach is rationable.\n\nI believe the methodology presented in this paper requires substantial revision and clearer justification of key design choices. While the paper has certain merits, I do not think it is ready for publication in its current form. I insist on a weak reject rating for this paper.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Minimizing Quantization Error\n\nWe agree that minimizing the final output error, i.e., $|XW - XW^C|$, is the ideal objective for any compression method. However, directly optimizing this quantity is computationally intractable in general. It is known to be NP-Hard and difficult to scale across layers.\n\nIn SLiM, we address this challenge by decomposing the problem into two tractable subgoals: (1) minimizing weight quantization error in a close-form, and (2) applying a lightweight, saliency-guided LoRA module to recover the residual output error post quantization. This design choice allows SLiM to scale to large models, while achieving state-of-the-art results across multiple tasks. Our ablation studies (`Table-1,6`) support the effectiveness of this decomposition: using LoRA on top of weight quantization consistently improves output accuracy, validating that our approach approximates the harder output-error objective effectively in practice.\n\n# Comparison with QUIP# and AQLM\n\nWe compare SLiM’s quantization-only (no pruning) zero-shot accuracy on LLaMA-2 models against QUIP# and AQLM. QUIP# does not support sparsity due to its Hadamard transform densifying sparse weights, and AQLM quantization of sparse models takes days (as reported on their code base). We are processing these with AQLM and will share results promptly, given rebuttal constraints.  SLiM outperforms other methods in 2 out of 4 tasks..\n\n## LLaMA-2 7B (4-bit Quantization)\n\n\\\\begin{array}{|c|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} \\\\textbf{Method} & \\\\textbf{Arc Challenge} & \\\\textbf{Arc Easy} & \\\\textbf{PiQA} & \\\\textbf{Winogrande} \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} QUIP-Sharp & 40.5 & \\\\textbf{69.1} & \\\\textbf{78.4} & 67.6 \\\\\\\\\n\\\\rowcolor[gray]{1.0} AQLM & 40.3 & 68.9 & 77.7 & 67.3 \\\\\\\\\n\\\\rowcolor[gray]{0.95} SLiM-Quant + SLiM-LoRA & \\\\textbf{43.8} & 68.4 & 78.1 & \\\\textbf{68.4} \\\\\\\\\n\\\\hline\n\\rowcolor[gray]{0.9} \\\\Delta \\text{with best alternative method} & +3.3 & -0.7 & -0.3 & +0.8 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n## LLaMA-2 13B (4-bit Quantization)\n\n\\\\begin{array}{|c|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} \\\\textbf{Method} & \\\\textbf{Arc Challenge} & \\\\textbf{Arc Easy} & \\\\textbf{PiQA} & \\\\textbf{Winogrande} \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} QUIP-Sharp & 45.5 & \\\\textbf{73.9} & \\\\textbf{78.9} & 69.9 \\\\\\\\\n\\\\rowcolor[gray]{1.0} AQLM & 43.9 & 72.2 & 78.6 & 70.4 \\\\\\\\\n\\\\rowcolor[gray]{0.95} SLiM-Quant + SLiM-LoRA & \\\\textbf{47.1} & 72.5 & 78.5 & \\\\textbf{72.5} \\\\\\\\\n\\\\hline\n\\rowcolor[gray]{0.9} \\\\Delta \\text{with best alternative method} & +1.6 & -1.4 & -0.4 & +2.1 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n# Order of Pruning and Quantization\n\nWe evaluated two SLiM variants: (1) Prune-First, where pruning is applied before quantization, and (2) Quantize-First, where quantization precedes pruning. The table below reports average accuracy across six zero-shot tasks and shows that the compression order has a negligible effect on performance. In both cases, SLiM-LoRA effectively mitigates any induced errors.\n\n\n\\\\begin{array}{|c|c|c|c|c|c|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9}Method&Structure&OPT125M&OPT350M&OPT1.3B&OPT2.7B&OPT6.7B& OPT13B & LLaMA2-7B & LLaMA2-13B  \\\\\\\\\n\\\\rowcolor[gray]{0.95} Quantize First & 2:4 & 34.62 & \\\\textbf{34.36} & 40.61 & \\\\textbf{42.73} & 45.99 & \\\\textbf{46.09} & 51.15 & \\\\textbf{54.94} \\\\\\\\\n\\\\rowcolor[gray]{1.0} Prune First & 2:4 & \\\\textbf{34.81} & 33.80 & \\\\textbf{40.66} & 42.10 & \\\\textbf{46.02} & 45.15 & \\\\textbf{51.50} & 54.77 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} Quantize First & Unstructured & 35.20 & \\\\textbf{35.32} & 41.85 & \\\\textbf{43.48} & 47.08 & \\\\textbf{47.96} & 54.26 & \\\\textbf{57.85}  \\\\\\\\\n\\\\rowcolor[gray]{1.0} Prune First & Unstructured & \\\\textbf{35.46} & 35.06 & \\\\textbf{41.49} & 43.16 & \\\\textbf{47.09} & 46.87 & \\\\textbf{53.61} & 57.94 \\\\\\\\\n\\\\hline\n\\\\end{array}\n# Joint Quantization and Pruning\nThank you for the suggestion. With the current formulation of SLiM, joint optimization of pruning and quantization is not feasible. However, because of the SLiM’s unique decomposition of tasks, SLiM-LoRA  is compatible with various compression methods and one can readily use it regardless of the compression order. \n# Assumptions in Theoretical Derivations\n- **SLiM-Quant:** For SLiM-Quant, we avoid assumptions about the weight matrices’ distribution during optimization, instead using an empirically derived histogram of the weights to guide the process.\n- **SLiM-LoRA:** For SLiM-LoRA’s saliency function, we assume it is additive and invertible. \n   - **Additivity** holds as we define $F(M)=diag(x)M$, where $x$ is the layer’s average input. For matrices $A$ and $B$, $F(A+B)=diag(x)(A+B)=diag(x)A+diag(x)B=F(A)+F(B)$. \n   - **Invertibility** is ensured by guaranteeing that $diag(x)$ is non-singular, achieved by enforcing positive values in $x$ (see line 5, Algorithm 2, page 5). This allows the inverse to be computed as $F^{-1}(M) = diag(1/x)M$. These properties enable SLiM-LoRA to effectively map recovered saliency to low-rank adapters.", "claims_and_evidence": "The paper provides extensive experimental results that support its claims of improved accuracy and significant inference speedups on GPUs.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-aligned with the goal of compressing large language models. However, certain design choices are less convincing. For instance, the decision to minimize quantization error rather than directly targeting activation or output error is not thoroughly justified, and the cascaded application of quantization followed by off-the-shelf pruning raises concerns about potential error amplification. Without sufficient ablation studies or evidence that this ordering is optimal, these aspects leave room for skepticism regarding the overall effectiveness of the approach.", "theoretical_claims": "I reviewed the derivations for the probabilistic formulation of quantization error minimization (Equations 3–7) as well as the formulation of the saliency-based low-rank adapter (Equations 8–11). While the derivations are logically coherent, they depend on assumptions that are not rigorously justified. For example, assumptions about the underlying weight distribution for numerical integration and the properties (invertibility and additivity) of the proposed saliency function.", "experimental_designs_or_analyses": "The experimental designs and analyses are generally sound, with extensive evaluations on standard benchmarks and comparisons against SOTA methods on popular model families (LLaMA‑2, OPT). However, there are some minor concerns. For example, while the experiments validate overall accuracy improvements and speedup claims, the lack of detailed ablation studies—particularly regarding the ordering of quantization and pruning—limits our understanding of error propagation in the cascaded approach. Moreover, the trade-off between minimizing quantization error versus feature or activation error is not thoroughly explored, which could affect the validity of the conclusions drawn from the reported experiments.", "relation_to_broader_scientific_literature": "The paper’s contributions build directly on established work in model quantization, and pruning. It extends prior methods (post-training quantization approaches and pruning techniques like SparseGPT or Wanda), by proposing SLIM‑Quant, a probabilistic formulation for uniform quantization that reframes error minimization as a convex problem via numerical integration. Additionally, it integrates ideas from low‑rank adaptation research ( like L2QER and LoRA) by introducing a saliency‑based low‑rank adapter (SLIM‑LoRA) that leverages invertible and additive saliency functions to compensate for compression errors . By combining these approaches into a unified, cascaded pipeline, the work aims to deliver efficient compression and hardware‑friendly inference for LLM.", "essential_references_not_discussed": "Some recently published LLM quantization methods have been omitted from the discussion, such as:  \n[1] Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., & Alistarh, D. (2024). Extreme compression of large language models via additive quantization. in ICML 2024.  \n[2] Tseng, A., Chee, J., Sun, Q., Kuleshov, V., & De Sa, C. (2024). Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. in ICML 2024.", "other_strengths_and_weaknesses": "**Strengths:**\n- The paper creatively combines probabilistic quantization, off‑the‑shelf pruning, and a saliency‑based low‑rank adapter into a unified one‑shot compression framework.\n- By targeting hardware‑friendly sparsity (e.g., 2:4 patterns) and demonstrating significant inference speedups on GPUs, the approach is well-suited for real-world deployment of large language models.\n\n**Weaknesses:**\n- The decision to minimize quantization error instead of directly targeting activation or output error is not thoroughly justified, potentially limiting overall performance.\n- The cascaded application of quantization followed by off‑the‑shelf pruning risks compounding errors, with no sufficient justification provided for this specific ordering.\n- There is a lack of joint optimization between quantization and pruning, which may exacerbate error accumulation.\n- The theoretical derivations rely on assumptions about weight distributions and saliency properties that are not fully validated, potentially affecting the robustness of the method.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Can you elaborate on why you chose to minimize quantization error rather than targeting activation or output error? Have you performed any experiments to compare the two approaches? Notably, recent methods such as QUIP# and AQLM focus on minimizing activation error for each layer or group.\n\n2. What is the rationale for applying quantization before pruning? Did you consider or experiment with reversing the order, and if so, what were the outcomes?\n\n3. Is it possible to integrate quantization and pruning into a joint optimization framework rather than treating them as separate cascaded steps? If so, what are the trade-offs, and how do you justify not pursuing this integrated approach?\n\nThe justification for minimizing quantization error instead of directly targeting activation or output error, as well as the possibility of integrating quantization and pruning into a joint optimization framework, are my two major concerns. I may reconsider my rating based on the authors' responses to these questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkoOkL4iUk", "reviewer": "Reviewer_q6Mp", "summary": "This paper proposes to use low-rank approximation to reduce the compression error for quantization and pruning on LLM. SLIM-Quant minimizes the quantization error by selecting the optimal scaling parameter. Low-rank adapters are applied to compensate the quantization and pruning error. Quantization and fine-tuning on the adapter further improve the SLIM. The results show the model performance improvement for 4 bits+50% sparsity compressed model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback. Below, we provide answers to all the points raised.\n\n# Speedup Comparison without LoRA\n\nAs requested, we have added two tables showing the layer-wise speedups of compressed models, with and without low-rank adapters, on an RTX-3060 GPU. While low-rank adapters slightly reduce speedup, the impact is minimal due to their low overhead. Importantly, SLiM-LoRA enables a flexible trade-off between speedup and model accuracy.\n\n\n\\\\begin{array}{|c|c|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} Model Size & Batch Size & LoRA Type & Self-Attention & Up-Projection & Down Projection \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{1.0} 13B & 16 & No LoRA & 3.89 & 3.86 & 3.86 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 13B & 16 & FP16 & 2.18 & 2.53 & 2.60 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 13B & 16 & INT4 & 1.28 & 3.24 & 3.17 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} 13B & 32 & No LoRA & 2.78 & 3.14 & 3.50 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 13B & 32 & FP16 & 2.23 & 2.68 & 2.91 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 13B & 32 & INT4 & 1.43 & 2.96 & 3.20 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{1.0} 13B & 64 & No LoRA & 1.46 & 1.88 & 1.98 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 13B & 64 & FP16 & 1.38 & 1.78 & 1.67 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 13B & 64 & INT4 & 1.21 & 1.69 & 1.65 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n\\\\begin{array}{|c|c|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} Model Size & Batch Size & LoRA Type & Self-Attention & Up-Projection & Down Projection \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} 70B & 16 & No LoRA & 3.63 & 4.22 & 4.03 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 70B & 16 & FP16 & 2.18 & 2.86 & 2.75 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 70B & 16 & INT4 & 3.11 & 3.99 & 3.79 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{1.0} 70B & 32 & No LoRA & 2.91 & 3.52 & 3.55 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 70B & 32 & FP16 & 2.00 & 2.63 & 2.67 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 70B & 32 & INT4 & 2.75 & 3.19 & 3.39 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} 70B & 64 & No LoRA & 1.89 & 1.98 & 2.12 \\\\\\\\\n\\\\rowcolor[gray]{1.0} 70B & 64 & FP16 & 1.38 & 1.70 & 1.86 \\\\\\\\\n\\\\rowcolor[gray]{0.95} 70B & 64 & INT4 & 1.51 & 1.77 & 1.94 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n\n# Accuracy Comparison with Dense Models\n\nWe believe that **comparing a compressed model with a dense model of the same original parameter count can be misleading**. For gaining a better insight on the accuracy of the models, one needs to compare quality at iso-model size (e.g., effective number of parameters).\n\n`Figure-2-Page-8` presents the accuracy of different models vs. their parameter size in GB, allowing for a more fair comparison of the different models. Based on this figure, the **compressed models provide higher accuracies in comparison to dense models of the same parameter size (in GB)**. Additionally, SLiM improves the accuracy of the compressed models by adding negligible additional parameters to them. More discussions about this topic can be found in `Results-Section-Page-7` under the “Comparison of large compressed and small dense models” subsection.\n\n# Speedup Breakdown \n\nPlease refer to our response to [Reviewer UKqS11](ADD-LINK-HERE) regarding an ablation study on the breakdown of the speedups.\n\n# Order of Pruning and Quantization\n\nPlease see [our response to Reviewer SvGK14](https://openreview.net/forum?id=4UfRP8MopP&noteId=5SPU1BHLiL) for a detailed answer regarding this important question.\n\n# Effects of Fine-tuning\n\nSLiM is designed as a one-shot compression method (similar to SparseGPT) that delivers strong performance without any additional fine-tuning (`Table-1-Page-7`). \n\nWhile an **optional** PEFT step can further enhance accuracy, it is not central to SLiM’s approach. Improvements with optional PEFT are in `Table-2-Page-7` and `Table-7-Page-14`.\n\n# Improved Accuracy of Quantized Models\n\nWith 4-bit quantization-only methods, performance is generally on par with dense models of the same effective size. As shown in `Table-6-Page-13`, some quantized models even slightly outperform their dense counterparts—a trend also observed in LQER [1] and QUIP# [2]. In such close cases, perplexity (`Table-9-Page-17`) can provide a more sensitive comparison metric.\n\n\n# Unstructured vs. 2:4 Sparsity\n\nUnstructured sparsity offers greater flexibility and often better accuracy, but is difficult to accelerate on modern GPUs [3, 4]. In contrast, 2:4 semi-structured sparsity is supported by recent GPUs (starting with Ampere architecture), enabling real speedups at the cost of some accuracy degradation.\n\n[1] Zhang, et al. “LQER: Low-Rank Quantization Error Reconstruction for LLMs”, ICML 2024\n\n[2] Tseng, et al. “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks”, ICML 2024\n\n[3] Xia, et al. “Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity”, VLDB 2023\n\n[4] Zheng, et al. “SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute”, OSDI 2022", "claims_and_evidence": "Yes. It claims improvement on model accuracy and the speedup on RTX3060 and A100 GPUs. The results show the improvement on different models and tasks, which verify the claim.", "methods_and_evaluation_criteria": "Yes. The proposed methods can effectively improve the quantized and pruned model performance by using low-rank adapters. But the evaluation for speedup performance should be compared with other quantization and pruning tasks besides FP16 model.", "theoretical_claims": "Yes. The SILM-Quant and SILM-LoRA algorithms were checked.", "experimental_designs_or_analyses": "I keep up with the literature in this area.", "relation_to_broader_scientific_literature": "LLM compression and combination of different compression works is efficient way for memory problem. Related works are referenced in the paper.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n- The proposed methods have good performance on pruned and quantized model.\n- The Additional Experiments are comprehensive.\nWeakness:\n- The speedup for LLM mainly benefit from the quantization, pruning architecture and the hardware. So, the speedup performance comparison is not fair for the proposed methods. It should be compared with structure pruned model and quantized model. The SLIM-LoRA may reduce the inference performance because of additional Low-Rank operation.\n- Although the performance is good than other quantized and pruned method, it still has much loss compared with dense model. For Llama2-13B, the SLIM-LoRA compressed model cause 5.86% accuracy loss for zero-shot tasks.\n- The speedup for quantization and pruning should be evaluated. The performance improvement breakdown can help better understanding on those different compression methods for LLM.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Why do quantization first in the paper? How about pruning + quantization model?\n- Is fine-tuning the key process for enhancing model performance?\n- Why the quantized model with low rank can perform better than dense model in Table 6? Do the quantized models always show better accuracy across different tasks?\n- It seems that unstructured pruning always performs better than 2:4 pruning in Table 7, but 2:4 pruned model has faster inference speed. Can you provide some insights on accuracy vs. inference speed of these different pruning methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqOnKRgZzI", "reviewer": "Reviewer_UKqS", "summary": "Large language models contain extensive parameter counts leading to significant memory overhead and high inference costs. Pruning and quantization methods solve this, but typically both need retraining on large-scale datasets. One-shot methods can reduce the cost, but jointly pruning and quantizing weights under low-bit scenarios is a challenge. This study proposes SLIM, which combines one-shot quantization, low-rank adapters and sparsity to compress language models for efficient inference. SLIM is validated on OPT and LLaMA-2 families, achieving significant improvements in model efficiency and accuracy across various benchmarks.\n\nThis study highlights the following:\n\n- **SLIM-Quant**: SLIM adopts symmetric weight quantization, and clips the weights before rounding to the nearest integer. The clipping threshold is a hyperparameter, and SLIM-Quant searches this parameter by transferring the problem into a probabilistic formulation of the quantization process. The authors finally use grid search to tackle the optimal $\\alpha$.\n- **SLIM-LoRA**: Naive-LoRA is a straightforward approach that minimizes the total error norm between the original weight matrix and the compressed weight matrix, but it overlooks the importance of each weight value. SLIM-LoRA takes $\\operatorname{diag}(\\mathbf{x})$ as a saliency function F, and minimizes $F(E_Q)$. The solution to this is to compute the SVD of $F (−(E_Q + E_S))$.\n- **Low-rank adapter quantization**: The authors further quantize the low-rank adapters to save memory and computation. They use AbsMax group quantization scheme for the adapters and group size is 128.\n- **Post-compression fine-tuning**: The authors only fine-tune the adapters for efficiency. Straight-through estimator (STE) method is used in back propagation.\n\nSLIM is evaluated on the OPT and LLaMA-2 model families. Downstream tasks include MMLU, Piqa, Arc-Easy, Arc-Challenge, WinoGrande and OpenBookQA. The authors also report PPL on WikiText2. Baseline methods are one-shot pruning methods like Wanda and SparseGPT, L$^2$QER, JSQ. Compared with those methods, SLIM show advantage both on structured and unstructured sparsity. The authors also report acceleration results, on RTX 3060 and A100 GPUs. Acceleration ratio is about 2~3x compared with dense full precision models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We have provided a detailed reply to address all of your points.\n\n# Significance of Accuracy Improvements and MATH Benchmark\nSLiM achieves up to 5.66% higher average accuracy than leading compression methods across six zero-shot tasks.\nPer your request, we evaluated LLaMA-2-7B/13B on the MATH benchmark using default LM-Evaluation Harness settings. \nNote that our vanilla models do not employ CoT[1], which can boost math problem performance. Evaluations on GSM8K are ongoing but slow (over 20 hours for a 13B model). We are also testing quantization on sparse checkpoints and will share results promptly, despite rebuttal constraints.\n\n## LLaMA-2 7B\n\n\\\\begin{array}{|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} Method & Sparsity Pattern & MATH \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} Dense & N/A & 0.26 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{1.0} SparseGPT + OPTQ & 2:4 & 0.04 \\\\\\\\\n\\\\rowcolor[gray]{0.95} Wanda + Group AbsMax & 2:4 & 0.00 \\\\\\\\\n\\\\rowcolor[gray]{1.0} SLiM-LoRA + SLiM-Quant & 2:4 & 0.64 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} SparseGPT + OPTQ & Unstructured & 0.18 \\\\\\\\\n\\\\rowcolor[gray]{1.0} Wanda + Group AbsMax & Unstructured & 0.04 \\\\\\\\\n\\\\rowcolor[gray]{0.95} SLiM-LoRA + SLiM-Quant & Unstructured & 0.34 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n## LLaMA-2 13B\n\n\\\\begin{array}{|c|c|c|c|}\n\\\\hline\n\\\\rowcolor[gray]{0.9} Method & Sparsity Pattern & MATH \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} Dense & N/A & 0.60 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{1.0} SparseGPT + OPTQ & 2:4 & 0.34 \\\\\\\\\n\\\\rowcolor[gray]{0.95} Wanda + Group AbsMax & 2:4 & 0.12 \\\\\\\\\n\\\\rowcolor[gray]{1.0} SLiM-LoRA + SLiM-Quant & 2:4 & 0.62 \\\\\\\\\n\\\\hline\n\\\\rowcolor[gray]{0.95} SparseGPT + OPTQ & Unstructured & 0.46 \\\\\\\\\n\\\\rowcolor[gray]{1.0} Wanda + Group AbsMax & Unstructured & 0.36 \\\\\\\\\n\\\\rowcolor[gray]{0.95} SLiM-LoRA + SLiM-Quant & Unstructured & 0.69 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n# Speedup Breakdown \n\nWe’ve included [this graph](bit.ly/3QXcit2), showing the contributions of quantization and sparsity to SLiM’s speedup, evaluated in Quantized-only and Sparse+Quantized settings using Sparse Marling kernels in vLLM. The results indicate that quantization drives most of the speedup, with sparsity contributing less.\n\n# Optionality of Post-Compression Fine-Tuning\n\nSLiM is a one-shot compression method that improves model accuracy without requiring fine-tuning. The results in `Table-1-Page-7` demonstrate that SLiM surpasses state-of-the-art compression methods without any fine-tuning.\n\nWhile an **optional** PEFT step can further enhance accuracy, it is not integral to SLiM’s core approach. Additional improvements from PEFT are detailed in the ablation studies in `Table-2-Page-7` and `Table-7-Page-14`.\n\n[1] Wei, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”,  NeurIPS 2022", "claims_and_evidence": "Most of the claims are clear and convincing.", "methods_and_evaluation_criteria": "Most of the methods and evaluation criteria make sense.", "theoretical_claims": "As I check, this article is based on experiments and does not require strict theoretical proof.", "experimental_designs_or_analyses": "Most of the experimental designs are valid.", "relation_to_broader_scientific_literature": "This study can be seen as an extension or improvement for the N:M post-training pruning. This study is closely related to previous N:M pruning works, such as SparseGPT and Wanda.", "essential_references_not_discussed": "No missing related works.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well organized and well written. The equations are very clear and easy to understand. The technical content is explained in sufficient detail. Additionally, the use of figures, tables, and examples enhances the clarity of the presentation, ensuring that the key contributions and findings are easy to follow and understand.\n2. Combining sparsity, quantization and low-rank adapters is challenging. However, the authors do this successfully.\n\nWeaknesses:\n\n1. The improvement does not seem significant. I wonder in practical instruction-following cases, such as MATH or GSM8K, would the advantage still exist.", "other_comments_or_suggestions": "Typos:\n\n1. Extra space on line 218.", "questions_for_authors": "Questions:\n\n1. Regarding the acceleration in Table 3, is the acceleration mainly from quantization, or is it from sparsity? It would be better to provide an ablation study.\n2. Since 'one-shot' is mentioned in the title, what are the results if the post-compression fine-tuning stage is omitted? An ablation study on this would be beneficial.\n\nOverall, the paper demonstrates strong clarity and structure. I appreciate the authors' efforts in translating these techniques into practical applications. I recommend a weak accept and suggest that the authors address W1, Q1 and Q2 to provide a more comprehensive perspective. The scores may be revised following the author-reviewer discussion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SLIM, a new one-shot LLM compression framework.  SLIM consists of quantization (via SLIM-Quant, a new probabilistic formulation of quantization error minimization), pruning (via off-the-shelf methods, e.g., Wanda), and low-rank adaptation (via SLIM-LoRA a novel saliency-based method, using calibration data).  Notably, to further reduce memory overhead of the adapters introduced in SLIM-LoRA, the paper also further quantizes the saliency-based adapters, leading to SLIM-LoRA^Q.  The efficacy of SLIM is compared to other SOTA one-shot post-training quantization (1shot-PTQ) schemes, i.e., SparseGPT, Wanda, L^{2}QER, JSQ, as well as a naive version of SLIM-LoRA (called Naive-LoRA).  SLIM outperforms previous 1shot-PTQ methods across a wide range of OPT/Llama-2 model sizes, average common sense reasoning benchmarks, and sparsity patterns (i.e., 2:4 and unstructured).  Impressive speedups are shown post-SLIM for both 16-bit and 4-bit precisions, as well as on commodity and high-end GPUs (the former are found in the Appendix).    The appendix contains a expansive range of experiments; due to the borderline nature of the review, I extensively consulted the main text/appendix as well as the source code to confirm lingering reviewer concerns were taken of.

The paper is well written and the 3 stages of SLIM are intuitive.  Many reviewers had follow up questions about the various stages, which the authors did well in answering by cross-referencing the results from the main paper.  However, to proactively address some of the possible confusing stages of the pipeline--e.g., SLIM-LoRA naturally leads readers to think of fine-tuning, but this is actually saliency-based one-shot calibration, which may be followed up by actual fine-tuning--the authors should add a summary of contributions at the end of the intro, including a delineation between the one-shot calibration stage in SLIM-LoRA versus actual fine-tuning (e.g., anticipating the second question from Reviewer q6Mp as a regular source of confusion when reading the paper and the authors reply as an early statement to clear this up).  

Reviewer SvGK raised major concerns by the authors’ decision to minimize quantization error rather than directly targeting output error.  The authors addressed this concern in a variety of ways, resulting in further experiments comparing the two approaches: i) minimizing output error (MOE), and (ii) minimizing weight error (MWE).  Ultimately, MOE outperformed MWE, which Reviewer SvGK noted could not address his concerns.  However, the performance discrepancy is extremely small, which means that MWE achievers near MOE without MOE's computational difficulties (an inclusion of this discussion in the appendix will similarly address future readers' concerns).  Moreover, both Reviewer UKqS  and q6Mp note that the frameworks of MOE and MWE are ultimately heuristic design choices, and the version of MWE serving as the backbone for SLIM outperform existing non-SLIM SOTA methods.  Thus, SLIM beats SOTA (which ultimately matters for this paper), and the methodology of further improving SLIM by incorporating MOE may be addressed in future work.

Additional reviewer concerns include potential generalization issues stemming from the calibration set.  However, these were raised late in the discussion period and are extensively addressed in Figure 4 and Table 18 of the appendix.  Reviewing all the submitted materials (main text, appendices, entire codebase), I appreciate the lengths the authors took to recreate the results of the previous SparseGPT and Wanda papers (e.g., using the same C4 sampling and calibration set curation from SparseGPT).  This is a point that should be stated clearly (and proudly) in the main paper (as it was in the Wanda paper, Section 4 third paragraph), which can quickly allay reader questions about comparison-fairness in an information-dense paper such as this.  I also would have liked to see more Llama-3 results in the main text (and it is worth pointing out those in the appendix), although it is understandable that the authors reproduced previous SOTA methods using their supplied code built for Llama-2 and OPT.
Decision: Accept (poster)