SLiM: One-shot Quantized Sparse Plus Low-rank Approximation of LLMs

Authors: Mohammad Mozaffari, Maryam Mehri Dehnavi
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: sparsity, 2:4 sparsity, quantization, low-rank, lora

Abstract:
Large Language Models (LLMs) have revolutionized natural language understanding and generation tasks but suffer from high memory consumption and slow inference times due to their large parameter sizes. Traditional model compression techniques, such as quantization and pruning, mitigate these issues but often require retraining to maintain accuracy, which is computationally expensive. This paper introduces SLiM, a novel approach for compressing LLMs using a one-shot Quantized Sparse Plus Low-rank Approximation. SLiM eliminates the need for costly retraining by combining a symmetric quantization method (SLiM-Quant) with a saliency-based low-rank approximation. Our method reduces quantization error while leveraging sparse representations compatible with accelerated hardware architectures. Additionally, we propose a parameter-efficient fine-tuning recipe that significantly reduces overhead compared to conventional quantization-aware training. SLiM achieves up to a 5.4\% improvement in model accuracy for sparsity patterns like 2:4, and the fine-tuning step further enhances accuracy by up to 5.6\%, demonstrating state-of-the-art performance. This work provides a pathway for efficiently deploying large models in memory-constrained environments without compromising accuracy.

Decision: reject
Reviewer scores: 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730200457006

thread_type:
official_review

<thread_object>
creator: reviewer_i42n

<content>
summary:
The paper introduces SLIM, a method for compressing Large Language Models (LLMs) using a one-shot approach that combines quantization, sparsity, and low-rank approximation. It reduces memory usage and inference time without retraining, while maintaining high accuracy. SLIM also includes an efficient fine-tuning process, achieving up to 5.8% accuracy improvement over existing compression techniques, making it ideal for deploying large models on resource-constrained hardware.

strengths:
1. The proposed symmetric quantization minimizes quantization error without altering the weights, resulting in reduced computational and memory overhead.
2. SLIM introduces a saliency-based method to reconstruct weights based on their importance, targeting weights with the highest impact on model output, which helps retain model accuracy. 
3. SLIM improves model accuracy by up to 5.4% over state-of-the-art pruning and quantization methods for 2:4 sparsity patterns. With fine-tuning, the accuracy gain can increase to 5.8%.
4. The fine-tuning recipe for sparse, quantized models significantly reduces the time and resources needed for fine-tuning. For example, a 13B parameter model that typically takes 36 days to fine-tune is reduced to just 14 hours on an H100 GPU.

weaknesses:
1. The specific method for determining the quantization scale in L199-L202 should be described in more detail to clarify the contribution. Some background information (e.g., symmetric quantization, description of quantization errors, etc.) could be shortened or moved to the supplementary material to reduce the length.
2. Some more advanced quantization methods [1-5] significantly outperform GPTQ. Would combining these methods further enhance compression performance?
3. The combination of these different compression methods (sparsity, quantization) should result in a higher compression rate. However, related experiments on the memory footprint of the compressed model and the analysis of inference efficiency are not included in the paper.


[1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.
[2] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024.
[3] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. ICLR 2024.
[2] AffineQuant: Affine Transformation Quantization for Large Language Models. ICLR 2024.
[3] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. Arxiv 2024.

questions:
1. The issues mentioned in the weaknesses should be addressed.
2. In L205-206, why can the impact of weight matrix quantization and pruning be modeled as additive noise? Is there any experimental or theoretical support for this claim? Why wouldn’t the combination of the two errors introduce larger errors? Intuitively, if $W^Q= W+ E_Q$ and $W^S =W+ E_S $, why would $W^Q+W^S=W_C$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641371966

thread_type:
official_review

<thread_object>
creator: reviewer_7uws

<content>
summary:
In this paper, the authors propose to combine quantization and pruning to compress LLMs. A new quantization method called SLiM-Quant is proposed to reduce the quantization error. To develop the new method, the mean squared error used in the literature was converted to a probabilistic formulation, which is solved by numerical integration over the weight histogram. Then a pruning method (structured or unstructured pruning) is applied. To compensate for the accuracy loss due to pruning and quantization, a new low-rank adapter is proposed, which is based on the assumption of an additive invertible saliency function. Finally, the low-rank adapters are quantized in a manner of 16x16 tiles, which is finally merged into the quantized sparse weight.

strengths:
1. Experimental results show that the proposed method leads to improved model accuracy under the same pruning and quantization scheme.

2. A novel low-rank adapter is proposed in this paper.

weaknesses:
1. But why is the proposed low-rank adapter not compared with the previous low-rank adapter? Without such a comparison, it's hard to tell the advantage of the proposed method.

2. The combination of quantization and pruning seems to be a straightforward method. In this paper, the two methods seems to be loosely coupled. As mentioned in Questions, the joint optimization has some unanswered questions.

3. Actually acceleration performance is not reported in this paper.

questions:
1. When merging the quantized sparse weight and low-rank adapters, there might be overflow due to addition. The non-zero values from the low-rank adapters might also be added to originally the pruned weights, which disturbs the pruning mechanism (in particular the 2:4 structured pruning). How should those potential problems during weight merging be solved?

2. What is the evaluation metric used in Table 1-3?

3. Does the quantization method proposed in this paper lead to zero values?

4. The paper did not propose a new pruning method. But in Table, SLiM works better than other methods under a pruning-only scheme.

5. The additive invertible saliency function comes from nowhere. The property is not explained in the supplementary either.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731304769

thread_type:
official_review

<thread_object>
creator: reviewer_6f8m

<content>
summary:
The manuscript proposes an LLM compression method that combines quantization, sparsity, and low-rank approximation. This approach applies low-rank approximation guided by a significance-driven optimization function, followed by fine-tuning of LoRA blocks, ultimately achieving advanced improvements in compression accuracy.

strengths:
1. Previous algorithms aimed at compression during inference generally do not retain LoRA. This manuscript innovatively retains it, and demonstrates its feasibility in terms of time efficiency through inference efficiency analysis.

2. The idea of using saliency to guide the optimization of LoRA is highly novel.

weaknesses:
1. The motivation for introducing Equation 8 is unclear. For instance, the authors claim that deriving LR from E_Q and E_S is intended to “cancel the compression errors”, yet introduce a significance function F that does not seem directly related to these compression errors.

2. Prior to Equation 10, the significance calculation method from Wanda is introduced without clear motivation. To my knowledge, there are alternative methods for measuring significance, such as AWQ[1] and SparseGPT[2], yet the authors appear not to consider them.

3. In Section 3.2, the manuscript claims that previous error minimization methods did not consider "the importance of different weights", though methods like OBS[3] clearly address the importance of weights (see its explanation of L_q).

4. The manuscript does not explain the pruning approach used and lacks relevant formula descriptions for E_S.

5. There is a lack of comparison with advanced joint strategies for sparse quantization, such as JSQ[4].

6. The manuscript lacks statistics on model size, which may lead to unfair comparisons. Since SLIM introduces additional LoRA layers with a substantial rank (r=0.1), describing the extra parameter overhead is necessary. Similarly, theoretical computations for the number of OPs are also not provided.

[1] Lin, et al. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration[J].

[2] Frantar, et al. Sparsegpt: Massive language models can be accurately pruned in one-shot.

[3] Hassibi, et al. Optimal brain surgeon and general network pruning.

[4] Guo, et al. Compressing large language models by joint sparsification and quantization.

questions:
1. Significance is usually used to guide pruning decisions, so why is the significance calculation method proposed in Wanda used as the optimization target for quantization and sparsification? Why does optimizing the significance metric result in better low-rank adapters?

2. Why does not pruning weights in LQ-LORA lead to a significantly higher error rate?

3. Why does the discussion on Inference Speedup not consider the impact of quantization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732219011163

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you!

comment:
Dear ACs and Reviewers,

Thank you for taking time to review our manuscript and for your valuable feedback. We appreciate the positive feedback of the reviewers, commending SLiM for its (1) highly novel idea and innovative approach for retaining LoRA during inference ([Reviewer 6F8M](https://openreview.net/forum?id=Usa4pF1e5I&noteId=qHg1W7bgX1)), (2) improved accuracy and novel low-rank adapter ([Reviewer 7uwS](https://openreview.net/forum?id=Usa4pF1e5I&noteId=QyIhjnLAxc)), and reduced memory usage and inference time without retraining while maintaining high accuracy in addition to significant reduction in the time and resources needed for an optional fine-tuning ([Reviewer i42N](https://openreview.net/forum?id=Usa4pF1e5I&noteId=kp4dq7YC3O)). We believe that your comments were helpful in further improving our work. As a result, we have incorporated additional clarifications and results as summarized below:

- [Rebuttal-A1](https://openreview.net/forum?id=Usa4pF1e5I&noteId=qHg1W7bgX1), [Rebuttal-A3](https://openreview.net/forum?id=Usa4pF1e5I&noteId=kp4dq7YC3O): Comparing SLiM against multiple one-shot pruning and quantization papers, including JSQ [1], AWQ [2], OmniQuan t[3], and AffineQuant [4].
- [Rebuttal-A1](https://openreview.net/forum?id=Usa4pF1e5I&noteId=qHg1W7bgX1), [Rebuttal-A2](https://openreview.net/forum?id=Usa4pF1e5I&noteId=QyIhjnLAxc), [Rebuttal-A3](https://openreview.net/forum?id=Usa4pF1e5I&noteId=kp4dq7YC3O): Memory and compute reduction analysis for SLiM and other pruning and quantization methods.

We believe that we have addressed all the reviewers’ comments individually. We look forward to a productive discussion during the author response period.

Best regards,

Authors

---

[1] Guo, et al. “Compressing large language models by joint sparsification and quantization.” ICML 2024

[2] Lin, et al. “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration.” MLSys 2024

[3] Shao, et al. “OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models.” ICLR 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732671777980

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Speedup Experiments for SLiM

comment:
Dear Reviewers, 

Thank you for your insightful feedback regarding the speedup analysis of SLiM. We have conducted additional experiments to address concerns about the practical acceleration of our method, and we would like to share these findings with you.

As highlighted in our rebuttal, there is currently no open-source repository supporting the combination of 2:4 sparsity and the specific 4-bit quantization approach employed in our work. This limitation restricts us to analyzing scenarios involving either sparsity or quantization individually, where efficient implementations exist. While implementing optimized kernels for simultaneous sparsity and quantization is beyond the scope of this paper, we believe it is a valuable direction for future work. 

To provide a clearer picture of SLiM's potential speedup, we performed an additional experiment to establish a **lower bound** on the achievable acceleration. In this study, we applied 8-bit weight quantization and incorporated low-rank adapters with a rank ratio of 0.1 into each feedforward layer of the model. These experiments were conducted on NVIDIA RTX-3060 GPUs using a batch size of 8, reflecting typical inference-time configurations for large language models. 

The table below summarizes the layer-wise speedups observed for various weight sizes in LLaMA and OPT models, with dense models using float-16 weights as the baseline:

### 8-Bit Weight Quantization Speedup ($\times$)

AR denotes the aspect ratio (output dimension / input dimension) of the layer. $\uparrow$ indicates better performance.

| Hidden Dimension       | 768   | 1024  | 2048  | 2560  | 4096  | 5120  | 12288 | 16384 |
|-------------------------|-------|-------|-------|-------|-------|-------|-------|-------|
| Down-Projection (AR=1/4) | 1.39  | 1.6   | 2.43  | 3.35  | 3.25  | 3.45  | 3.1   | 3.27  |
| Up-Projection (AR=4)     | 0.46  | 0.65  | 1.01  | 2.17  | 1.55  | 2.69  | 1.82  | 1.88  |
| Attention Layers (AR=1)  | 0.42  | 0.62  | 1.24  | 2.12  | 1.76  | 2.92  | 2.47  | 2.69  |

We also present a detailed visualization of these results in Figure 2 (Page 18) of the revised manuscript. These findings demonstrate that the inclusion of low-rank adapters does not significantly impede the speedup benefits of quantization, except in specific cases involving small hidden dimensions (e.g., <1024), which correspond to model sizes under 125M parameters. In more practical scenarios involving larger models with billions of parameters, the combination of quantization and low-rank adapters consistently yields substantial speedups.

It is important to note that these results represent a **lower bound** on the speedup achievable with SLiM. Introducing 2:4 sparsity and reducing the quantization bitwidth to 4 bits would further enhance these speedups. 

We hope these additional experiments address any remaining questions about the performance implications of our method. Thank you for your valuable input, which has greatly contributed to strengthening this work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a post-training method for compressing large language models (LLM). It merges quantization and pruning to reduce the computational overhead, optionally followed by a fine-tuning stage to improve the model's performance. The reviewers are generally negative on this paper, with final scores of 3/3/5, making this case a clear rejection. The reviewers are unanimously concerned about the novelty, technical motivation, and running performance of the proposed method, and the authors did not fully address these concerns during the rebuttal. The AC finds no reason to overturn the reviewers' recommendation.