SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Authors: Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model; Post-training Model Compression

Abstract:
The advancements in Large Language Models (LLMs) have been hindered by
their substantial sizes, which necessitates LLM compression methods for practical
deployment. Singular Value Decomposition (SVD) offers a promising solution for
LLM compression. However, state-of-the-art SVD-based LLM compression meth-
ods have two key limitations: truncating smaller singular values may lead to higher
compression loss, and the lack of update on the compressed weights after SVD
truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM
compression method that addresses the limitations of existing methods. SVD-LLM
incorporates a truncation-aware data whitening technique to ensure a direct map-
ping between singular values and compression loss. Moreover, SVD-LLM adopts
a parameter update with sequential low-rank approximation to compensate for
the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10
datasets and seven models from three different LLM families at three different
scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts,
especially at high model compression ratios.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729670506922

thread_type:
official_review

<thread_object>
creator: reviewer_kqw6

<content>
summary:
This paper proposes SVD-LLM, a novel SVD-based LLM compression method that addresses efficiency issues in previous methods. It establishes a direct mapping between singular values and compression loss with theoretic proof. Extensive experiments demonstrate the effectiveness of the proposed method and support the author’s claims.

strengths:
1. SVD-LLM can achieve state-of-the-art in terms of compression efficiency and performance on downstream tasks.
2. The inference speedup that SVD-LLM can achieve is inspiring given SVD-LLM’s good performance on PPL and downstream tasks.
3. The research is well presented and it is easy to follow.

weaknesses:
(a) “SVD for LLM Compression” part in “Related works” is insufficient, and it can be improved by discussing more papers related to SVD-based LLM compression and papers that relate to the Atomic Feature Mimicking (AFM) technique, where the latter is also dedicated to model decomposition for LLM compression. Papers that authors should be aware of are listed as follows.

1. Compressing Transformers: Features Are Low-Rank, but Weights Are Not!, AAAI’23

2. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML’23

3. Adaptive Rank Selections for Low-Rank Approximation of Language Models, NAACL’24

(b) The experiments on the latest model are missing. Authors should test their proposed method, i.e., SVD-LLM, and compare it with baselines on the latest model such as LLaMA-3-8B.

questions:
The concept of “SVD-LLM (U)” in S4.3 - "Modular Sensitivity Study" needs to be clarified. From what I understand, the core part of SVD-LLM is the truncation-aware data whitening, the concept of “SVD-LLM (U)” thus needs to be clarified since SVD-LLM will be degraded to normal truncated-SVD when "truncation-aware data whitening" is disabled.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730123925115

thread_type:
official_review

<thread_object>
creator: reviewer_8cqq

<content>
summary:
This paper introduces a LLM compression approach based on SVD called SVD-LLM. SVD-LLM incorporates a whitening process to maintain the orthogonality of activations and align the number of retained singular values and the compression loss. After compressed via SVD, SVD-LLM further finetune the model for better performance.

strengths:
1. The activation whitening is supported by theoretical analysis, which is convincing.
2. The experiments are extensive, demonstrating improvements comparing to selected baselines.
3. SVD-LLM delivers practical benefits in terms of inference speedup and memory reduction on hardware.

weaknesses:
1. The comparisons in the main experiments might be \textbf{unfair}, as SVD-LLM adds extra finetuning additional finetuning that is not applied to other SVD-based baselines, which could drastically improve performance under high compression ratios. Extra finetuning is not a novel technique and can be seamlessly integrated into the baselines.
2. Theorem 3.2 and Corollary 3.3 only hold when $S$ is calculated on the fly, but in practice SVD-LLM precomputes $S$ using a calibration set, and it is unclear if the orthogonality of activations  is preserved with this approach. Further experiments are needed here to address this gap, for example, the orthogonality of whitened activations using the precomputed $S$.
3. The paper lacks validation of the method on larger models, such as those at the 70B scale, which is crucial for evaluating the feasibility of model compression approaches.

questions:
1. The paper claims to use LoRA finetuning in Section 3.2 but it is not clear how it is done. Does this introduces additional parameters, or fused afterward? Or the finetuning is directly applied on $W_u$ and $W_v$?
2. Section 4.2, the authors discuss memory reduction for the KV cache at the expense of recomputation overhead. How does this impact throughput, and by how much?
3. In Section 4.4 and Table 8, SVD-LLM is combined with QuIP#. More details are needed regarding how this combination is executed.
4. In Algorithm 3,  there is a layer-wise closed-form update procedure that was not mentioned in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730346583599

thread_type:
official_review

<thread_object>
creator: reviewer_p8yj

<content>
summary:
With the remarkable capability of large language models (LLMs) in the field of natural language processing, their large model size becomes a major obstacle for practical deployment. To this end, the paper proposes a Singular Value Decomposition (SVD)-based LLM compression method, which aims to address two key limitations faced by existing SVD methods in compression: the truncation of small singular values that may lead to higher compression loss and the lack of model parameter updating after SVD truncation.

strengths:
- The paper introduces SVD-LLM, a novel SVD-based compression method that incorporates a truncation-aware data whitening strategy to directly map singular values to compression loss, allowing for more informed truncation decisions.
- The paper provides a rigorous mathematical foundation for the proposed truncation-aware data whitening technique, supported by theoretical proofs and lemmas.
- It proposes a parameter update mechanism with sequential low-rank approximation to compensate for accuracy degradation after compression.
- The authors evaluate SVD-LLM across various datasets and models from three different LLM families at various scales, demonstrating its superiority over existing SVD-based methods, especially at high compression ratios.

weaknesses:
- The authors should evaluate SVD-LLM on the latest models in LLaMA family, such LLaMA-3-8B.
- Authors are recommended to analyze the computation costs of SVD-LLM and provide the evaluation of its time expense.
- No comparison with the latest SOTA methods, such as FLAP: Fluctuation-based Adaptive Structured Pruning for Large Language Models. Yongqi An, et al. AAAI 2024, available on arXiv since Dec. 2023.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731373099684

thread_type:
official_review

<thread_object>
creator: reviewer_3pxg

<content>
summary:
The authors investigate the compression of large language models through low rank approximation. They point out that the existing works in this direction have the following two problems. First, existing works approximate the weight matrix without considering the input activation matrix. Second, existing works do not finetune the model after the truncation. Consequently, they propose SVD-LLM, which addresses both issues by data whitening and LoRA finetuning. Their experiments show that their method not only outperforms other SVD approximation methods, but also outperforms other compression methods like structured pruning or model quantization. Furthermore, real inference speedup and memory saving can be observed without specialized hardware.

strengths:
1. The writing is clear and easy to follow.
2. The empirical comparison is extensive. The authors not only compared with the SVD-based baselines, but also compared with other compression methods like structured pruning and quantization to demonstrate the superiority of the proposed method.
3. Real inference speedup and memory saving can be observed without specialized hardware. Unlike some unstructured pruning and/or quantization methods, which require specialized hardware for speedup, this makes the proposed method more practical and applicable.

weaknesses:
1. As mentioned by SAC, the concept of data aware compression is not new and already exists in [1]. Additionally, in [1] they claim optimality for the problem considered, while this paper seems to offer weaker guarantees. (From my understanding, this paper shows that given the basis/decomposition of their choice, pruning the smallest entries of the middle diagonal matrix is optimal. On the other hand, [1] claim that both their choice of the basis and the truncation is optimal)
2. The LoRA training part of this paper also seems not to be novel as existing in LLM-pruner. In addition, the claim for the necessity of sequential finetuning for W_u, W_v seems odd. For lora finetuning of transformers, similar structure exists for key and query attention matrix W_k, W_q, but existing literature does not report any issues for fine tuning.


[1] Chen, P., Yu, H. F., Dhillon, I., & Hsieh, C. J. (2021). Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34, 29321-29334.

questions:
1. Can you clarify the difference/advantage compared to Drone? Specifically, are the methods equivalent (It seems not so as I do not see a simple connection between Cholesky and eigendecomposition)? If not, can you show that your method is also optimal?
I acknowledge that the experiment is done at a different scale (Bert vs LLM), so verifying this direction works at a larger scale itself is a contribution, but seems not enough for a full paper.  
2. Can you show the performance or LoRA fine tuning without the sequential finetuning process proposed by the paper? This can be one of the novelties of the paper but requires careful verification. The failure of original finetuning might be because of bad choice of hyperparameters for training.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436393887

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary. Hope for further discussion.

comment:
Dear Reviewers,

We sincerely thank you for valuable feedback, which has significantly improved the quality of our paper. Based on your suggestions, we have revised and updated the paper, highlighting changes and additions in **blue** within the revised PDF. Below, we summarize the key revisions made:
- **1. Experiments with one more baseline:** Following suggestion from the reviewer `P8yJ`, we have added the comparison with FLAP[1] in Appendix A.11.
- **2. Discussion on more related works:** Following suggestion from the reviewers `3PXg`, `P8yJ`, and `kQW6`, we have revised the "SVD for LLM Compression” part in “Related works" and included the more related works[1,2,3,4,5]. Following suggestion from the reviewer `3PXg`, we have metioned Drone[5] in Section 3.1 and include the detailed description, including how it works and the comparison with SVD-LLM in Appendix A.10.
- **3. Experiments on more recent and larger-scale LLMs:** Following the suggestions from the reviewers `kQW6` and `P8yJ`, we have conducted the experiments on the latest model LLaMA-3-8B in Appendix A.9. Following the suggestions from the reviewer `8cQQ`, we have added the evaluation on LLaMA-2 70B in Appendix A.9.
- **4. Experiments on Sequential LoRA fine-tuning:** Following the suggestions from the reviewers `3PXg`,  we have added the comparison between the standard LoRA fine-tuning and sequential LoRA fine-tuning in Appendix A.12 and mentioned it in Section 3.2. Following the suggestions from the reviewers `8cQQ` and `P8yJ`,  we have added the comparison of SVD-LLM and previous SoTA method ASVD with sequential LoRA fine-tuning in Appendix A.12 and added the row of SVD-LLM*, which indicates the SVD-LLM without sequential fine-tuning for comparison in Table 1,2,3.
- **5. Experiment on the orthogonality of the whitening matrix $S$:** Following the suggestions from the reviewers `8cQQ`, we have added the orthogonality analysis in Appendix A.12.

After the deadline of revision submission, reviewers also arised two additianal suggestions towards the revision. Although we cannot submit the revised paper on Openreview now, we mark them here and promise to include them in the paper upon paper acceptance.

- **Acknowledge more about Drone in Section 3.1.**

- **Compare pruning-based methods with SVD-LLM without the designed sequential LoRA fine-tuning.**


Once again, we deeply appreciate the time and effort you have dedicated to improving this paper. Your insights have made the paper stronger and more comprehensive. We look forward to any **further discussion or clarification** to better refine our work. Wishing you a Merry Christmas and a Happy New Year!

Bests,

Your friends,

Team of 2301

---
[1] FLAP: Fluctuation-based Adaptive Structured Pruning for Large Language Models. Yongqi An, et al. AAAI 2024

[2] Compressing Transformers: Features Are Low-Rank, but Weights Are Not!, AAAI 2023

[3] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML 2023

[4] Adaptive Rank Selections for Low-Rank Approximation of Language Models, NAACL 2024

[5] Drone: Data-aware low-rank compression for large nlp models, NeurIPS, 2021
</content>
</thread_object>

</thread>
<thread>
creation date:
1732509000943

thread_type:
official_comment

<thread_object>
creator: program_chairs

<content>
title:
Comment from Senior Area Chair

comment:
The senior area chair posted the following comment, but the visibility was not set to allow the authors to read it. We are posting it here for transparency.

> I noticed a closely related work (https://proceedings.neurips.cc/paper_files/paper/2021/hash/f56de5ef149cf0aedcc8f4797031e229-Abstract.html) which also conducts low-rank compression by minimizing ||WX - W'X||_F and derives an optimal low-rank solution. Could the authors comment on the relationship and key differences between their approach and this work?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a low-rank compression algorithm for compressing a weight matrix in a neural network. The algorithm is truncation aware and is better than naive SVD. Experimental results show promising performance gain over existing low-rank compression algorithms on LLMs. 

The main strength of this paper is the strong empirical results, while the main concern from the reviewers is that the optimal low-rank compression derived from this paper is equivalent to the DRONE paper published a few years ago. After extensive discussions between authors and reviewers, the authors agree that the factorization formed in this paper is mathematically identical to DRONE, so the contributions of this paper are (1) Conducting a relative comprehensive experiments on recent LLMs, where DRONE was developed before ChatGPT/Llama so the experiments was mainly conducted on BERT. (2) The authors explicitly show that they can maintain the XX^T matrix in a streaming fashion to make it work for large-scale datasets/models. This was not explicitly mentioned in the DRONE paper. However, after the discussion closed, Reviewer 3PXg found that DRONE mentioned that the XX^T matrix can be maintained in the streaming manner, so this part of novelty becomes unclear. 

The paper received 8, 6, 6, 5, and reviewer 3PXg mentioned during discussion to reduce the rating from 6 to 5 due to the point (2) mentioned above, so the actual final rating is 8, 6, 5, 5 which is a borderline paper. Although the novelty may be limited compared to DRONE, given the strong empirical results the AC still votes for weak accept for this paper. 

If the paper is accepted, the AC and reviewers would like to see that the authors address point (2) mentioned above in the final version to better compare their work with DRONE.