SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
Authors: Xin Wang
Yu Zheng
Zhongwei Wan
Mi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model; Post-training Model Compression
Primary area: foundation or frontier models, including LLMs
TL;DR: We design a SVD-based post-training method to compress large language models

Abstract: The advancements in Large Language Models (LLMs) have been hindered by
their substantial sizes, which necessitates LLM compression methods for practical
deployment. Singular Value Decomposition (SVD) offers a promising solution for
LLM compression. However, state-of-the-art SVD-based LLM compression meth-
ods have two key limitations: truncating smaller singular values may lead to higher
compression loss, and the lack of update on the compressed weights after SVD
truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM
compression method that addresses the limitations of existing methods. SVD-LLM
incorporates a truncation-aware data whitening technique to ensure a direct map-
ping between singular values and compression loss. Moreover, SVD-LLM adopts
a parameter update with sequential low-rank approximation to compensate for
the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10
datasets and seven models from three different LLM families at three different
scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts,
especially at high model compression ratios.

Published as a conference paper at ICLR 2025
SVD-LLM: TRUNCATION-AWARE SINGULAR
SVO
VALUE DECOMPOSITION FOR LARGE LANGUAGE
MODEL COMPRESSION
Xin Wang' Yu Zheng² Zhongwei Wan'
Mi Zhang'
The Ohio State University
Michigan State University
ABSTRACT
The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD- LLM compression meth- ods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LIM, a SVD- post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-awer ping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD- LID on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority SVD-LLM over state-of-th te-tthe-rtsss especially at high model compression ratios.
data whitening technique to ensure a direct map-
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in wide range of tasks such as natural language understanding and generation (Zhao et al. 2023 Gozalo-Brizuela
and Stariid-Merrhán 2023). Despite such capabilities, the democratization of LLMs Is is primarily
restricted by their substantial resource demands, which motivates the design of LLM compression
methods (Wan et al. 2024a Wang et al., 2024; Zhuu et al. 2024 Zhou et al.) 20244. These methods are often performed in a post-training manner without requiring retraining from scratch. Post-training LLM compression methods based on quantization (Yuan et al. 2024 Huang al., (024), unstructured pruning (Frantar and Alistarh 2023), and structured pruning (Ma et al. [023] Ashkboos et al. 2024]] Zhong et al.] (0024) have been intensively studied. Despite their success, these methods have certain limitations, such as dependence on specific hardware and low inference speedup. In contrast, compression methods based on low-rank approximation, such as Singular Value Decomposition (SVD) are not limited by those constraints. Moreover, the KV cache of LLMs compressed via SVD at runtime can also be reduced.
Despite these advantages, the potential of SVD for LLM compression has not been fully explored. Several SVD-based LLM compression methods, such as FWSVD (Hsu et al.. 2022) and ASVD (Yuan) et al. 2023) have been proposed. However, these methods exhibit severe performance degradation when model compression ratid increases. Such limitation can be attributed to two fundamental issues involved in their approaches. Misalignment between SVD truncation and compression loss: both FWSVD and ASVD fail to establish direct relationship between singular values and model compression loss. As a consequence, truncating smaller singular values in SVD could lead to higher compression loss. Lack of model parameter update after SVD truncation: as model compression ratio increases. the number of singular values that need be truncated in SVD increases as well. To compensate for the accuracy degradation caused by truncating a larger number of
Model compression ratio refers the percentage of parameter reduction achieved through compression.

---

Published as a conference paper at ICLR 2025
singular values, it becomes necessary to update the remaining parameters of the compressed model. Unfortunately, existing SVD-based LLM compression methods do not incorporate such update in their design, and thus fail to compensate for the accuracy degradation especially under high model compression ratios.
In this paper, we propose SVD-based post-training LLM compression method, SVD-LLM, which effectively addresses the two fundamental issues of existing SVD-based LLM compression methods. SVD-LIM differs from them in two key aspects. 1 Truncation-Aware Data Whitening: supported by theoretical proof, SVD SVD-LLM incorporates a truncation-aware data whitening technique that ensures direct mapping between singular values and model compression loss. In doing so, the proposed iruncation-awer data whitening technique is able to identify which singular values should be trun- cated to incur minimal model compression loss. Parameter Update with Sequential Low-rank Approximation: to compensate for accuracy degradation after compression, SVD LLM sequentially fine-tunes the decomposed low-ranking matrices for a global accuracy recovery.
We compare BVD-LLM with both state-of-the-an SVD- LLM compression methods well as pruning and quantization-baxe LLM compression methods. To demonstrate the generability of SVD-LLM, we conduct our evaluation on total of 10 datasets and seven models from three different LLM families (LLaMA, OPT, and Mistral) at three different scales (7B, 13B, 30B), and evaluate the performance of SVD-LIA LLM on both GPU and CPU. We highlight three of our findings:
SVD-LLM outperforms state-of-the-art SVD-based LLM compression methods FWSVD and ASVD across all 10 datasets, three LLM families at three scales by a large margin.
SVE-LLM also outperforms state-of-the-arr structured pruning-based LLM compression methods, including L.LM--runer, SliceGPT, BlockPruner as well as state-of-the-art 1- bit post-training quantization-baan LLM compression methods, including PB-LLM and BiLLM. More importantly, when combined with 2-bit post-training quantization, SVD-ILM outperforms state-of-the 1-bit training-requee required quantization- based LLM compression method OneBit, presenting a new way to achieve state-of-the-an compression performance without incurring expensive retraining.
LLMs compressed by SVD-LLM are able achieve inference speedup and memory reduc- tion when deployed on real hardware, including both GPU and CPU. At the same time, SVD-LLM able to reduce runtime KV cache memory without additional accuracy drop.
RELATED WORK
Large Language Model Compression: LLMs in general contain billion-scale parameters. Applying conventional model compression methods for LLMs is unfeasible as they necessitate resource- intensive retraining. Given that, post-training methods that avoid retraining in the compression process have been proposed. In general, these methods can be grouped into four categories: unstructured pruning, structured pruning, quantization, and low-rank approximation. Specifically, unstructured pruning methods (Frantar and Alistarh, 2023) set the individual weights of an LLM to zero without changing its shape. However, irregular sparsification of unstructured pruning is difficult to achieve the desired speedup memory saving. Unlike unstructured pruning, structured pruning methods (Ma) et al. 2023) Ashkboos et al., 2024) Zhong et al. 2024) remove entire channels or other structured components from LLMs, making them easier to implement on hardware. One notable contribution is LLM-Pruner (Ma et al. 2023), which groups weight matrices based on their dependency and assigns the pruning ratio to each group based on the estimated importance. Quantization methods (Lin et al. 2024) compress models by reducing the precision of weight matrices of the LLM. However, similar to unstructured pruming, quantization is also difficult achieve the desired inference speedup due the lack of hardware support and efficient kernels for low-precision computation (L et al. (2024). Recent studies including PB-LLM (Yuan et al. 2024) and BiLLMM (Huang et al. 2024) push the frontier to 1-bit quantization. Nevertheless, these approaches often lead to severe accuracy degradation.
SVD for Language Model Compression: low-rank approximation technique to reduce matrix size by approximating matrix with two smaller low-ranking matrices (Golub et al.) 1987). Given that, SVD is commonly used for model compression. For instance, DRONE (Chen al. 2021) achieves optimal SVD compression for small language models such as BERT. However, during SVD compression, DRONE caches all the input activations, making it challenging for L.M compression due to excessive memory usage. For LLMss, directly
Singular Value Decomposition (SVD) a widely used
2

---

Published as a conference paper at ICLR 2025
Original LLM
Compressed LLM
Calibration Data
Wiu
w
Layer
Input Activation
Update
Wu
Layer
Layer 2
Cholesky
decomposition
SVD
Layer
Trunc.
Update
Layer M
Whitening Matrix
Layer M
Original Weights W
Truncation-Aware Data
Parameter Update with Sequential Low-rank Approximation
Compressed Weights W
Whitening
Figure 1: Overview of SVD-LLM.
applying SVD on the weight matrix without considering the importance of the weights leads to large compression loss. To address this issue, Hsu et al. (022) propose FWSVD, which introduces Fisher information to weigh the importance of parameters. However, FWSVD requires a complex gradient calculation that demands substantial computing and memory resources for LLM compression. Another issue of directly applying SVD is that the distribution of activation can affect the compression loss. To address it, Yuan al. 2023) propose ASVD, which scales the weight matrix by diagonal matrix that normalizes the impact of input channels on the weights. However, both FWSVD and ASVD do establish direct relationship between singular values and compression loss. As consequence, truncating the smaller singular values may lead to higher compression loss. Moreover, as compression ratio increases, it becomes necessary to update the compressed weights for accuracy recovery. However, both FWSVD and ASVD do not take it into consideration, and thus incur severe accuracy degradation under high compression ratios.
issue,Hsu
3 SVD-LLM
Figure provides overview of SVD-LLM. At high level, SVD-LLM SVD-bbsed post-training LLM compression method. Specifically, following the standard procedure of post-training LLM compression methods (Frantar and Alistarh 2023 Yuan al. 2023 Xiao et al. (2023), SVD-ILM uses a random set of sentences as calibration data to generate activation for truncation-aware data whitening. Given the generated activation, SVD-LLM derives the whitening matrix S through Cholesky decomposition, and then performs SVD to truncate the multiplication of weight matrices W and whitening matrix S to compress the LLM. After truncation, SVD-LLM updates the remaining model parameters with sequential low-rank approximation to recover accuracy. In the following, we describe both truncation-ave aware data whitening and parameter update with sequential low-rank approximation in detail. The pseudocode of SVD-LLM is provided in Appendix A.]
3.1 TRUNCATION-AWARE DATA WHITENING
Motivation: Due to high variance of input activation, simply applying vanilla SVD for LLM compression leads to severe accuracy degradation (Yuan al., 2023). To address this issue, existing methods (Yuan et al., 2023 Hsu et al. 2022h formulate LLM compression an optimization problem with the following objective:
0 min[[ W W'Xlle)
where W is the weight matrix of the original LLM. is the activation of W. W the compressed weight matrix, and [[ W'X||F
|| is the compression loss in the form of Frobenius loss.
Although existing methods attempt to reduce this compression loss during their SVD truncation, they all fail to establish a direct relationship between singular values and compression loss. As consequence, truncating smaller singular values SVD could lead to significant compression loss. Taking ASVD (Yuan et al. 2023)
example, ASVD extracts diagonal matrix So from X where each element in the diagonal is the absolute mean value of each channel. It then uses So to normalize X and converts W X into (W Sa) (S, X). Subsequently, SVD is performed on W So obtain the decomposed matrices Uo. Σg. and vg Lastly, ASVD truncates the smallest singular values in En to obtain the compressed weight matrix W/ Uo x Trune. (Fn) V, x So1.

---

Published as a conference paper at ICLR 2025
Σg
2.4
1.8
0.9
0.1
Σ
2.4
1.8
0.9
0.1
Trunc.
Trunc.
One Singular Value
Muttipie Singular Values
Singular Value
Muttiple Singuler Valuee
0.1
0.9
0.9
0.1
0.1
0.1
0.1
Loss:
1.1
0.7
1.9
1.7
Loss:
0.1
C.1
(a) Data Normalization (ASVD)
(b) Truncation-Awan Data Whitening (SVD-LIM) LLLD
Figure 2: Compression loss (L- = | X - W of different data preprocessing methods.
Although normalizing the activation improves performance, ASVD does not establish a direct rela- tionship between singular values and compression loss (a detailed proof is included in Appendix (A.2). To better illustrate this point, we show two concrete examples in Figure 2(a) In example where only one singular value is truncated, truncating the smallest singular value 0.1 results in higher compression loss (loss 1.1) compared to truncating the second smallest singular value 0.9 (loss 0.7). In example @ where multiple singular values are truncated, truncating the smallest two singular values 0.9 and also leads a higher loss (loss = 1.9) than truncating 2.4 and 0.1 (loss = 1.7). As such, truncating the smallest singular values does not lead to minimal loss.
The key idea of SVD-LLM is to incorporate a truncation-aware data whitening tech- nique that ensures direct mapping between singular values and compression loss. To achieve this, SVD-LLM enforces the whitened activation S-1 X to be orthonormal such that each channel I, where S is derived through Cholesky decomposition (Meyer 2000). SVD is then performed on WS to obtain the de- composed matrices U, E, V, where U 11, 02, U3; ..., Un]. Σ diag 02 Ø3, ,,,)), and V [v1, U22 V3 .... UyJ. Lastly, the smallest singular values in Σ are truncated (denoted by Trunc.(E) to obtain two low-ranking matrices W", = 17x x Trunc.(E)] W [Trunc. (E)) VT x S-1 and the compressed weight matrix W' = W x W = U x Trunc. VT x S-1
Key Design:
is independent of each other, i.e., (S-1 (S- xx) T S- XXX"
Figure 2(b) illustrates the effect of the proposed truncation-aware data whitening method. In example where only one singular value truncated, the compression loss equals to the truncated singular value. In example @, the compression loss of truncating multiple singular values equals to the square root of the sum of their squares. As such, under the proposed truncation-awerr awwae data whitening technique, truncating the smallest singular values leads minimal compression loss.
In the following, we provide a theoretical proof on why the proposed truncation-awen -wwar data whitening technique ensures direct mapping between singular values and compression loss in the case of one singular value (Theorem 3.2p and multiple singular values (Corollary 3.3), respectively.
Lemma 3.1. The Frobenius norm of matrix A with dimension m n can be deduced into the square root of the trace of its gram matrix, which is:
1
a
m
|||| A
lany 1 = Trace A)]
Let SVD(WSS denote SVD compression matrix W.S. The compressed weight matrix W" can be expressed as W" SVD S) Using Lem we obtain the compression loss L; when truncating the hth singular value of W.S to reduce its rank for compression:
L, = I - X I = Wss-ix SVD(WS)) X = ||(WS - SVVS--SD
= -1 5- x|| Trace (uNTs- (S-1) ( vuu.))
(3)
Since both U = [U1, U2 U3 up1 and V = [81. U2 U3 Un1 are orthonormal matrices, we have:
of 0/ = uif' u: 1; o: = uf uj  vi + Traco(v,uf)) Traco(u,u))) =
Theorem 3.2. If is the Cholesky decomposition of X XTT, the compression loss I, equals as
4

---

Published as a conference paper at ICLR 2025
Proof. Since the whitening matrix S is the Cholesky decomposition of XX",, have SS -XXx. We can further infer Equation to obtain:
L, = Trace(u,v) ).u Trace (u: (v, vo]" u²?) o, Trace (u.u,) 03 (5)
Therefore, L; of truncating equals to the singular value oj itself.
Corollary 3.3. If S the Cholesky decomposition of XX",, truncating the smallest singular values leads to the lowest loss L compared to truncating others.
Proof. If we truncate mm+l, 0 m ...
.. Tr in E for compression, the square of the loss L is:
L =
a.uoo S X
Σ
02o xxx
j=m+1
j=++1 i=++1
Traco xx
E (L))=
Σ (a,)²
i=m+1
i=m+1
(=++1
The squared loss 2 equals to the sum of the squared singular values (More detailed derivation is in Appendix A.3). Truncating the smallest singular values achieves the lowest compression loss.
Given that our proposed truncation aware data whitening technique is built upon SVD, whose applicability depends on certain singular value distribution, we further conduct a spectrum analysis of the singular values obtained by our technique. We refer the readers to Appendix A.4 for details.
should also be noted that our proposed truncation-aas aware data whitening technique not only ensures direct mapping between singular values and compression loss, but is also able to achieve the same theoretical optimal compression loss as DRONE (Chen al. (02T). However, during SVD compression, DRONE stores all input activations in memory, which poses challenge for LLM compression due to high memory consumption. In contrast, input x; which is considerably smaller than the full input activation. In Appendix A.5 we provide our proof on SVD-LLM achieving the same theoretical optimal compression loss DRONE and discuss the advantages of SVD-LLM DRONE in memory saving, compression speed, and numerical stability in details.
incrementally updates its
XX" matrix by adding the xx." T of each
3.2 PARAMETER UPDATE WITH SEQUENTIAL LOW-RANK APPROXIMATION
Although the proposed truncation-aware data whitening technique helps preserve the accuracy during compression, as the compression ratio increases, the accuracy of the compressed LLM may degrade given that more larger singular values will be truncated by SVD compression. Therefore, is necessary to update the remaining parameters in the compressed LLM.
Motivation:
Key Design: SVD-LLM proposes a variant of LoRA fine-tuning to update the remaining weight parameters of the compressed LLM for accuracy recovery. Specifically, given the two low-ranking matrices W W/ generated by truncation- aware data whitening, instead of directly applying LoRA fine-tuning to the compressed weight matrix W' W', x W', as standard LoRA does, we propose to apply LoRA on top of W', and W separately to preserve their low-rank structures follows:
W + W + A, W W + B.A,
where A,, B,, A,, and B, are matrices used for LoRA fine-tuning.
Simultaneously fine-tuning W and W will not guarantee decrease fine-tuning loss. This because the derivatives of W and W are interdependent during the fine-tuning process, where optimization of one matrix may interfere with the optimization of the other. Therefore, as shown in Figure [ we propose sequential low-rank approximation strategy to fine-tune W' and W'. in sequential manner. Specifically, we first freeze matrix W and fine-tune W with LoRA. We then perform the second-round LoRA fine-tuning matrix W while freezing the updated weight matrix W' u. Finally, we add B,, x A,, and B, x A, matrices to W and W' to obtain the final compressed weight matrices.
5

---

Published as a conference paper at ICLR 2025
Table 1: Performance of LLaMA-7B compressed by SVD-LIM, SVD-LLM (W), and baselines under different compression ratio (corresponding weight memory is listed inside bracket) on two language modeling datasets (measured by perplexity (L)), eight common sense reasoning datasets (six measured by both individual and average accuracy (1), TruthfulQA measured by BLEU score (1), and GSM8K measured by Exact Match Accuracy (t)). The best performance is marked in bold. The relative performance gain compared to the best-performing baseline is marked in green inside bracket.
RATIO (MEM))
METHOO
WilToat-21
C4.
Opanbs
ARC_=
WanG.
Hells.
PIQA
MuliQA
Asarage?
TruhfogAA
GSMBK!
0% (13.5 (8)
Original
5.68
7:34
0.34
0.75
0.20
0.52
0.79
0.27
0.57
0.30
0.09
SVI
2005
1880)
0.06
0.01
0.03
0.00
0.83
001
0.00
20% (10.2 GB)
PWSVD
ASVD
11.34
1727
15.90
0.09
0.29
0.64
0.05
0.41
0.00
0.58
0.10
0.05
0.17
0.8
7.96 299)
0.31
0.6
0.45
0.71
0.21
0.49 (1916)
0.26 (40,85)
0.25 (40.01)
SVT-111
7,78 3193
12.28
(2215)
0,33
0.7
0.49
0.58
0.79
0.23
0,58
(1226)
0.28
(4000)
0.88 (+0.04)
SVD
X243M
47774
0.06
0.06
0.35
0,01
0.05
0.82
01133
0L0N)
QU01
40% (7.76 (IR)
PWSVD
ASVD
18156
1407
12847
1309
0.06
0.08
0.11
0.08
0077
0.09
0.00
0.08
0.13
0.05
088
0.08
1104
0.10
0,01
0LE0)
0.001
0,00
SVIC LLM (3)
18.73 (.999)
75.22 (9339)
0.25
0.40
0.63
0.12
0.38
(13809))
0.17 (40L17)
0.02 (+0.02)
SVE-1LEM
(.W9))
(1999))
0.65
0.69
0.50
(1400063
(+0.07)
SVD
108674
0,01
0.01
0,00
0.01
0.02
00001
(5.35 (IR)
IWSVD
ASVD
37057
52194
29101
43036
0.35
0.06
0.04
0.01
0.05
0.01
0.09
0.06
0.00
0,05
0.05
1001
0.06
0.00
0.00
SVIE L.N (3)
66 (1905))
471 83 9999
0.10
0.05
0.17
0.10
0.21
0.04
0.11 (1R5K8)
0.01 c40,013
0.00 (-0.00)
BYE-LLM
15,00
(19915)
26.36
(1999)
0.18
0.42
0.44
0.51
0.35
0.12
0.39 (140066)
0.34 134.+0.110 140
0.04 (+0,044
0.01
MOSS
PWSVD
ASVD
0.04
SVE 1.M 00) SVE-LLM
1849 (L909))
6034 (1919))
0.07
0.05
0.04
0,00
0.07
0.01
0UH4 (1199)
(400
0.00 (+0.00)
(1991))
(11999))
0,11
0.25
0.21
0,14
0.37
0,08
0.16 (143399)
(.044)
0.82 (+0,02)
4 EXPERIMENTS AND ANALYSIS
Baselines. We compare SVD-LLM against three groups of methods: (1) Vanilla SVD and state-of- the-art SVD-based LLM compression methods: FWSVD (Hsu et al.) 2022). ASVD (Yuan et al. 2023) (Section 4.T) and FLAAP (Appendix (A.6). (2) Other types of L.IM compression methods. These include three state-of-the-er pruning-bas based LLM compression methods: LLM-Pruner (Ma et al.) 2023). SliceGPT (Ashkboos et al.] 2024), and BlockPruner (Zhong et al. (024)), and three state-of- the-art quantization-basel LLM compression methods; PB-LLM ( Yuaan et al. 2024), BiLLM (Huang eet al. 2024). and OneBit (Xu et al. 2024) (Section 4.4. (3) Smaller LLM StableLM-3B (Tow et al.)
tra-trained from scratch (Appendix A.7).
Models and Datasets. To demonstrate the generability of SVD-LLM, we evaluate the performance of SVD-LLM seven models from three different L.MM families three different scales (IL.MM--7B.
13B, 30B, LLaMA2-77 (Touvron et al., (023), OPT-6.7B (Zhang et al., 2022), Vicuna-7B (Chiang ct al. (202)) and Mistral-7B (Jiang al. (023)) and 10 datasets including two language modeling datasets (Witi (Merity et al. (017)) 7) and C4 (Raffel et al., 2020). six classification datasets (OpenbookQA (Mihaylov al. 2018). WinoGrande (Sakaguchi et al., 2020), HellaSwag (Zellers) et al. 2019), Are_ e (Clark et al. 2018). PIQA (Bisk et al. 2020). MathQA (Amini et al. 2019)), and two generation datasets (TruthfulQA (Lin et al. (022) and GSM8K (Cobbe et al. 202 p) with the
L.M-Evaluation-Harnes framework (Gao et al. 2023).
Implementation Details. To ensure a fair comparison, we followed ASVD (Yuan et al.) 2023) to randomly select 256 samples from Wiki'eext-2 as the calibration data. We followed the same configuration used in LLM- --rMrrr (Ma et al. (023) to use Alpaca (Taori et al. 2023) dataset with 50K samples for parameter update in SVD- LLM. The inference efficiency experiment is conducted on both NVIDIA A100 GPU and AMD EPYC 7643 CPU while the other experiments are conducted on NVIDIA A100 GPUs
4.1 COMPARISON WITH SFATE-OF-THE-AE SVD-BASED LLM COMPRESSION METHODS
First, we compare the performance of SVD-LLM with state-of-the-art SVD-based L.LM compression methods from four aspects: (1) performance under different compression ratios, (2) performance on different LLMs, (3) performance on LLMs with larger scales, and (4) compression speed (Ap-
pendix A.8. Given that all the SVD-based baselines do not incorporate LoRa fine-tuning, to ensure fair comparison, we also compare to SVD-LLM LLM with truncation -warr data whitening only (denoted as SVD-LLM (W)). ).
Performance under Different Compression Ratios. We first evaluate the performance of LLaMA- 7B compressed by SVD-LIM, SVD-LIM (W), , and the SVD-based baselines under compression

---

Published as a conference paper at ICLR 2025
Table Perplexity (4) of SVD-LLM, SVD-LIM (W), and basclines on WikiText-2 and the average accuracy (1) of the six common sense reasoning datasets of four different LLMs - OPT-6.78, LLaMA 2-7B, Mistral-7B, and Vicuna- 7B under 20% compression ratio. The relative performance gain compared to the best-performing baseline is marked in green color inside bracket.
OPT-6.78
LLAMA 2-78
MISTRAL-7B
VICUNA-7B
METROD
Perplexity]
Accurncy
Perplexity.
Accurncy
Perplexity]
Accuracy"
Perplexity|
Accuracy!
Original
10.86
5.47
5:25
0.56
SVD)
66275
0.03
18192
0.09
159627
0.03
18644
0.05
TWSVD
2360
6357
0.08
0.09
ASVD
82.00
0.32
10.10
0.36
13.72
0.32
16,23
0.33
SVD- LIK
(180%)
(128%)
(116%)
0.53 (147%)
10.21 (12668)
(124%)
8:41
(148%)
("55%)
SVD-LIM
14.47 (1829))
0.49 (1535%)
7.73 (1233%)
0.54 (1509)
7.47 ((459))
0.55 (172%)
7.43 (1548)
0.54 ('64%)
ratios ranging from 20% to 80% on all 10 datasets. The results are summarized inTable I Both SVD-LIM and SVD-LIM (W) consistently outperform vanilla SVD, FWSVD and ASVD across all the compression ratios. In particular, when the compression ratio 40% and above, SVD-IIMM reduces the perplexity by more than 99% on two language modeling datasets and achieves over 400% higher average accuracy on six classification datasets. More importantly the results on two generation datasets (TruthfulQA, GSM8K) of all three baselines when compression ratios are 60% and above are zero, meaning that the compressed LLMs totally lose their generation ability. In contrast, SVD-LLM still outputs good generation even under 80% compression ratio. Example contents generated by the compressed LLMs are included in Appendix AA.9
Performance on Different LLMs To examine the generability of SVD-LLM and SVD-LIM (W) across different LLMs, we compare the perfor- mance between SVD-LLM and the baselines on four different models from three different LLM families - OPT-6.7B (OPT family), LLaMA 2-7B (LLaMA family), Mistral-7B (Mistral family), and Vicuna-7B (LLaMA family) under 20% com- pression ratio on WikiText-2 and six classification datasets. As shown in Table [ both SVD-LLM and SVD-LLM (W) consistently outperform base- lines on all four LLMs, and exhibits more stable performance across different LLMs, especially compared to vanilla SVD and FWSVD.
Table 3: Perplexity (L) of SVD-LIM, SVD-LLM (W) , and baselines on WikiText-2 and the aver- age accuracy (1) of the six classification datasets of LLaMA-13B and LLaMA-30B under 20% compression ratio. The relative performance gain compared to the best-performing baseline is marked in green color inside bracket.
-
LLAMA-13B
LLAMA-308
Micrioe
Porpexity]
Accuracy!
Perplexit]
Accuncy!
Original
5.09
0.59
4.10
0.61
SVD
946.31
0.1
$4.11
0.33
FWSVD
15.98
0:43
20.54
0.42
ASVD
6,74
0.54
22.71
0.44
SVD-LLM
6.61 (.25)
(100)
5.63 (1739)
0.57 (13%)
0.55
(12%)
(1755)
0.59 (134%)
Performance on LLMs with Larger Scales. To examine the generability of SVD-LLM and
SVD-LLM (W) on LLMs with larger scales, we compare the performance between SVD-I.MM and the baselines on LLLMA-133 1-B and LLaMA-30B under 20% compression ratio. As shown in Ta- 3 both SVD-LLM and SVD-LLM (W) consistently outperform vanilla SVD, FWSVD, and ASVD on both model sizes.
4.2 INFERENCE EFFICIENCY OF SVD-LIM
Theoretical Analysis of Inference Efficiency. Assume SVD-LLM compresses the weight matrix W& E |dd into two low-ranking matrices W E _dxr, W/ E gr The compression ratio is then calculated as R. (d+m))r
(1) Compute Complexity Analysis: Given input X E Baadd instead of recalculating the full weight matrix W" W W and then computing the output W' x X, SVD-LLM calculates the intermediate state M- = W x and then computes the output Y W x M. In this way, the computation complexity is reduced from O (d to O (a2r + rad). Taking compression ratio R. 50% as an example, since Ph = (d4n)r
have ाल Then the computation complexity is
O (df rnd) = Ord((d n)) = (f 1 n)), which reduces 50%. In general, given any compression ratio Rw. the computation complexity is reduced to 1 1 RR,, times of the original.
(2) Inference Memory Analysis: Since SVD-LLM does not recalculate the full weight W W x W the weight memory is reduced to 1 Rn, times of the original during inference. As another
7

---

Published as conference paper at ICLR 2025
Ratio=80%
Ratio=60%
Ratio=40%
Ratio=20%
Original
Tokens / sec
4000
Tokens sec
2400
3200
2000
2400
1600
1600
1200
800
64
128
256
512
800
32
64
128
256
Batch Size
Sequence Length
(a) Varying Batch Size on GPU
(b) Varying Sequence Length on GPU
Tokens /sec
Tokens sec
120
350
250
80
150
40
64
128
256
512
32
64
128
256
Batch Size
Sequence Length
(c) Varying Batch Size on CPU
(d) Varying Sequence Length on CPU
Figure 3: Throughput (tokens/sec) of LLaMA-7B and its compressed version by SVD -LLM under different compression ratios on single A100 GPU under different batch sizes (a) and different sequence lengths (b) and on single AMD EPYC 7643 CPU under different batch sizes (c) and different sequence lengths (d). For (a) and (c), sequence length 32. For (b) and (d), batch size 64.
advantage, SVD-LLM is able to reduce the runtime KV cache memory as well (Wan et al. 2024b 2025, Shen et al., 2024). Specifically, instead of keeping W W in the KV cache, SVD-I LLMM provides the option to store the intermediate result M W. X in the KV cache and recomputes original key and value states with the decomposed weight matrix W required. As such, the runtime KV cache is reduced to _ = R...) n+2 times of the original. Moreover, since W is already stored as the weight matrix in the decomposed LLM, the original intermediate state matrix still be recovered by WK. M without accuracy drop. Therefore, SVD-LLM provides unified solution that enables simultaneous model compression and KV cache compression.
Inference Speedup Hardware. To quantify inference speedup achieved by SVD-LLM
real hardware, we measure the numbers of tokens that LLaMA-7B and its compressed version by SVD-LIM generate per second (i.e., throughput) under different batch sizes and sequence lengths on a single NVIDIA A 100 GPU and a single AMD EPYC 7643 CPU, respectively. The results are (1) Under specific batch size or sequence length, the speedup achieved by SVD-LLM related to the original model increases as the compression ratio increases. (2) Under specific compression ratio, the speedup achieved by SVD-LLM related to the original model becomes more significant as the batch size increases as the sequence length decreases. (3) The above two observations are valid for both GPU and CPU.
shown in Figure 3 We have three observations.
Inference Memory Reduction Hardware. Lastly, we evaluate the inference memory saving, including both weight memory and runtime KV cache memory saving on a single A100 GPU. Specifically, measure the peak memory footprint during inference when generating 128 tokens with batch size 32 using LLaMA-7B compressed by WDD-LMM under different compression ratios w/ and w/o considering KV cache reduction. The results are illustrated in Figure 4 where the memory reduction from the dotted line to the blue bars comes mainly from model weight compression and the memory reduction from the blue bars the yellow bars comes mainly from KV cache compression. As shown, both weight memory saving and runtime KV cache memory saving brought by SVD- LLMM near linear to the compression ratio.
4.3 ABLATION STUDY
Modular Sensitivity Study. We conduct ablation studies to evaluate the separate contributions of the two key components (i.e.. truncation-an data whitening and parameter update with sequential low-rank approximation) of SVD-ILM. Let -VD-LLM (W) denote the version of SVD- SVD-LLM -LLM with
truncation-aware data whitening only; SVD-LLM (U) denote the version of SVD-L.IM with normal
8

---

Published as a conference paper at ICLR 2025
SVD LLM wio KV cocne reduction
SVD-LLM w/ KV cache reduction
Table 4: Perplexity (Q) of compressed LLaMA- 7B WikiText-2 under different compression ratios. SVD-LIM (W) denotes the version of SVD-LLM with truncation-aware data whiten- ing only; SVD -LLM (U) denotes the version of SVD-LLM with parameter update with sequential low-rank approximation only: The relative per- formance gain compared to ASVD marked in green color inside bracket.
Peak Mem. (GB)
17.7
14.4%
12.0
2440
2350
9.0
1208
20%
40%
60%
80%
Compression Ratio
Figure Peak memory to generate 128 tokens with batch size of 32 using LLaMA-7B com- pressed by SVD-LIM w/ and w/o KV-cache reduc- tion. The dotted line indicates the peak memory of the original LLaMA-7B. The memory reduction from the dotted line to the blue bars mainly comes from the model compression. The memory reduc- tion from the blue to the yellow bars mainly comes from the reduced footprint of the KV cache.
METHOD
20%
40%
60%
ASVD
11.14
14077
57057
SVD-L11 (8)
7.94 (129%)
13.11 (199%)
42.30 (199%)
SVD LIM (U)
(1996)
19.28 (1999)
49.88 (1999)
SVD LIM
7.73 (13156)
9.27 (199%)
15.00 (1995)
SVD truncation and parameter update with sequential low-rank approximation. As shown in Ta- ble we have three observations. (1) SVD- -LMM (W), SVD- LLM (U) and SVD LLL consistently outperform ASVD across all the compression ratios. Notably, when the compression ratio is and above 40%, all of them reduce the perplexity by more than 99% compared to ASVD, (2) SVD-LLM consistently outperforms both SVD-LLM (U) and SVD-LLM (W) across all compression ratios. This result demonstrates the unique contribution from each of the two key components and the importance of combining both components achieve the best performance. (3) Comparing between SVD-LLM (W) and SVD-LLM (U), SVD-LLM (W) achieves a lower perplexity compared to SVD-LLM (U) across all compression ratios. This result indicates that truncation aawae data whitening plays a more significant role than parameter update with sequential low-rank approximation.
Impact of Calibration Data. Next, we examine the impact of calibration data on SVD-LLM. Figure 5 and Table 6 summarize the performance of compressed LLaMA-7B when changing three key characteristics of the calibration data: (1) number of calibration data, (2) the seed used to randomly sample the calibration data, and (3) dataset from which calibration data is sampled. As shown, the changes of the three key characteristics on calibration data incur no more than 3% to the final performance, indicating that the sensitivity of SVD-LLM calibration data is limited.
Impact of Updating Order. Lastly, we examine the impact of updating order in the parameter update with sequential low-rank approximation component the final performance of the compressed LLM. Table shows the performance of compressed LLaMA-7B under 20% to 80% compression ratios on WikiText-2 with different updating orders. As shown, there is only small difference of the final performance between updating matrix W', first and updating matrix W first. This result indicates SVD-LIM is not sensitive to the updating order.
More ablation studies are included in Appendix A.10
4.4 COMPARISON WITH OTHER TYPES OF LLM COMPRESSION METHODS
SVD-LLM is orthogonal to other post-training LLM compression methods including pruning and quantization /Wan al., 2024a Shen al. (025). Lastly, we compare the performance of SVD-LLM with state-of-the-art structured munningbbssd and quantization- based L.L.M compression methods. As discussed in Section 2 since unstructured pruning methods are difficult to achieve its efficiency on hardware, we do not make a comparison with them in this experiment.
Comparison with Structured Pruning. First, we compare SVD SVD-LLM with three state-of- the-art structured pruning-based LLM compression methods: LLM-Pruner (Ma et al., 2023). SliceGPT (Ashkboos et al., 2024) and BlockPruner (Zhong et al. 2024) under the same mem- ory budget, ranging from 10 GB to 7 GB LLaMA-7B using Wixt dataset. As shown in Table 7 SVD-LLM outperforms all three structured pruning- based LLM compression methods. In particular, SVD-I.A achieves up 56% reduction in perplexity under 7G memory budget.

---

Published as a conference paper at ICLR 2025
6.78
Table 6: Performance of LLaMA-7B compressed by SVD-IIIM under 20% compression ratio us- ing calibration data sampled from WikiText-2 (by default in our paper) and C4 datasets. The performance on WikiText-2 and C4 are reported by perplexity (4), while the performance on six downstream datasets reported by average ac- curacy (t). The performance on TruthfulQA and GSM8K are reported by BLEU score(f) and Ex- act Match Accuracy (f) respectively. The rela- tive performance gain for data sampled from one dataset compared to another is marked in green color inside bracket.
Perplexity
Perglesits
6.78
6.73
32 128
512
6.70
42
Namber of daca 0X2045 takens)
Seed for Random Sampling
(a) Change of Number
(b) Change of Seed
Figure 5: Perplexity of LLaMA-7B under 20% compression ratio using calibration data sampled with different number or seeds from WikiText-2.
Table 5: Perplexity of LLaMA-7B compressed by SVD-LLM under 20% to 80% compression ratio on 2 with different updating orders.
WikiText-21
Averaget
TruhfulQA?
GSMSK
Calibration data sampled from WikiTex-22
UPOATING CREDER
(178)
12.23
0.55
(125)
|
0.28
0.88
W. finst then W.;
7.85
9.32
13.20 (L1%)
31.7(1190)
Calibration data sampled from C4
WC Best, then W.
(128))
9.27 (1198)
15.00
$1.79
7.79
11.97 (119)
0.54
|
0.28
0.08
Table 7: Perplexity (L) of LLaMA-7B compressed by structured pruning methods and various memory budget WikiText-2. The relative performance gain compared to the best best-performing bascline marked in green.
Table 8: Perplexity (1) of LLaMA-7B com- pressed by 1-bit quantization methods and SVD-LLM on WikiTeext2. The relative perfor- mance gain compared to the best-performing baseline marked in green.
under
PERPLEXITY UNDER VARIOUS MEMORY BUDGET
Tyr::
MEMORY
PRRPLEEXIT
METHOD
10 GB
9B
8 GB
7 GB
PB-LLM BiLLM
Post traintainnng Posi-triining
19
104,83
GB
47.67
LLM-Prunce
9.88
SliceGPT
8.78
12.73
16.39
27.41
GVD-LLM
Post-rrining
GB
47.21 (119)
BlockPruner
9.40
12.76
19.78
43.05
Triiinn-roquine
G.
SVD-LIM
7,92 (L10%) 8.18 (13368)
8,33 (1/193)
9.63 (L.5%)
8VD- (2-bir)
Post-rmining
13
(L455)
Comparison with Quantization. Finally, we compare SVD-LLM with three state-of-the-an quantization-base based LLM compression methods: BiLLM (Huang et al. 2024), PB-LLM (Yuan) et al. 2024), and OneBit (Xu et al. (2024) which push the frontier to 1-bit quantization. Among them, both BiLLM and PB-LLM are post-training methods, and OneBit is training-required The results
LLaMA-7B using WikiText-2 dataset are shown in Table 8 We have three observations. (1) Com- pared post-training methods PB-LLM and BiLLM, SVD LLM achieves the best performance. (2) Training-required method OneBit outperforms SVD- LLM. This result is expected because OneBit in- volves resource-ingg interssvee retraining using large-scatl scale datasets to boost the accuracy after compression. (3) Lastly, we combine SVD-LLM with a 2-bit quantization-bast
post-training LLM compression method QulP# (Tseng et al. 2024). This is achieved by first applying SVD-L.LM to the I.I.M under 40% compression ratio, and then applying QuiP# for 2-bit quantization on the compressed model (W' and W generated from SVD-LLM As shown in Table g SVD-LLM outperforms state-of-thr of-the-art 1-bit training-requires method OneBit without involving resource-intensive retraining. This result demonstrates the potential of a hybrid approach integrating -Dseb and quantization based compression techniques to push the boundaries of post-training LLM compression.
CONCLUSION
In this work, we present SVD-LLV SVD-based post-training LLM compression method. SVD-LLM proposes truncation-awai aware data whitening technique to guide which singular values to be truncated with minimal compression loss. It also introduces a sequential low-rank approximation strategy to compensate for accuracy degradation caused by singular value truncation. We evaluate SVD-LLM on 10 datasets and seven models from three L.LM families at three scales. Our results demonstrate the superiority of SVD-LLM over of -fate-of-the-arts. especially at high model compression ratios.
6 ACKNOWLEDGEMENT
This work is supported in part by NSF Award N675-2312265.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation based formalisms. In NAACL-IILT (1), pages 2357-2367. Association for Computational Linguistics, 2019.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation adaptive structured pruning for large language models. In AAAI, pages 10865-10873. AAAI Press, 2024.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In ICLR. OpenReview.net, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAAI, pages 7432-7433 -439. AAAI Press, 2020.
Patrick H. Chen, Hsiang-Fu Yu, Inderjit S. Dhillon, and Cho-Jui Hsich. DRONE: data-aware low-rank 2021.
compression for large NLP models. In NeurlPS, pages
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. (ttpss
///mmyy..c brg.y...rr//r http:/lll///// 20200//222222222 03 30-vviunuea
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1 Mes/100035557 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2 10. 14168, 2021.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one- shot. In ICML, volume 202 Proceedings of Machine Learning Research, pages 10323-10337. PMLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 2023. URL/attps://zenoe wtwpedoo.orrrreeeeeeeeeeeeeeeeee wttp://o.oo.eeeeeer/e025ee5555..
G.H. Golub, Alan Hoffmann, and G.W E.W. Stewart. A generalization of the eckart- matrix ap- proximation theorem. Linear Algebra and Applications, 88-89:317-3221 1987. ISSN 0024-3795 doi: http::///ooooooo/////00000002110 com/ http:///iiooo//rrree/////////2/3
URL https: scctaar/www.sciinneeiirree.
Roberto Gozalo-Brizuela and Eduardo C. Garrido-Mecchnn A survey of generative AI applications. CoRR, abs/2306.0278 2023.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. ICLR. OpenReview.net, 2022.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for 1lms. In ICML. OpenReview.net, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, 1bs/2310.06855, 2023.
11

---

Published as a conference paper at ICLR 2025
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACL (1), pages 3214-3252. Association for Computational Linguistics, 2022.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4A8KV4 quantization and system co-design for efficient LLM serving. CoRR, fos/2405.045332 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In NeurlPS, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR (Poster). OpenReview.net, 2017.
Carl Dean Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, 2000.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP, pages 2381-2391.
Association for Computational Linguistics, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21:1 10:1440:1144:67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-87444 8740. AAAI Press, 2020.
Hui Shen, Zhongwei Wan, Xin Wang, and Mi Zhang. Famba-v: Fast vision mamba with cross-layer token fusion. CoRR, 0bs/2240.098800 2024.
Hui Shen, Jingxuan Zhang, Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, et al. Efficient diffusion models: A survey. arXiv preprint arXiv:2502.06803 066005 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An eesruuttoo-foolovinng llama model.
https:://iithie com wtw/ttttoutteeeeee ttt//ttttuuuuuae lab/ sttanford_alpaeeee 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023.
Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme. Stablelm 3b 3mmm33b-1111]
4elt. URL htt.://www..........aaaaaaaaaaaa
(https:
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better LLM quantization with hadamard incoherence and lattice codebooks. In ICML. OpenReview.net, 2024.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Trans. Mach. Learn. Res., 2024, 2024a.
12

---

Published as a conference paper at ICLR 2025
Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang. Siqi Luo, Jing Xiong, and Mi Zhang. D2O: dynamic discriminative operations for efficient generative inference of large language models. CoRR, abs/2406. 13035, 2024b.
Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, and Mi Zhang. Meda: Dynamic kv cache allocation for efficient multimodal long-context inference. arXiv preprint arXiv:2502. 2500::9902..7599, 2025.
Xin Wang, Zhongwei Wan, Arvin Hekmati, Mingyu Zong, Samiul Alam, Mi Zhang, and Bhaskar Krishnamachari. Iot the era of generative Al: vision and challenges. CoRR, abs/2401.01 001923, 2024.
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 38087-38099 PMLR, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang. Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In NeurIPS, 2024.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: activation-awarr singular value decomposition for compressing large language models. CoRR, abs/2312.058211 2023.
Zhihang Yuan, Yuzhang Shang, and Zhen Dong, PB-LLM: partially binarized large language models. In ICLR. OpenReview.net, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL (I), pages 4791-4800. Association for Computational Linguisties, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.01068, 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. survey of large language models. CoRR, abs/2303. 18223, 2023.
Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fine- grained pruning for large language models. CoRR, Abs/2400.10094 2024.
Zixuan Zhou, Xuefei Ning, Ke Hong. Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey on efficient inference for large language models. CoRR, 0bs/2440..14244 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. Trans, Assoc. Comput. Linguistics, 12:1556-1577 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX.
A.1 PSEUDOCODE OF SVD-LLN
Algorithm shows the pseudocode of SVD-VVDDE LLM. Before compression, SVD LIM randomly collects small amount of sentences as calibration data C, then runs the truncation-aware data whitening Algorithm 2 to obtain the set whitening matrix Sety for the weight to
process as shown compress. After that, it runs the SVD and truncation with Sets each weight matrix in the LLM. Instead of directly finishing the whole compression, it stores the decomposed matrices and further utilizes these matrices in Algorithm
the parameter update with sequential low-rank approximation as shown
Algorithm Pseudocode SVD-LLM
1: Input: M: Original LLM
2: Output: M": Compressed LLM by SVD-LLM
3: procedure SVD-LIM(M)
4: Randomly collect several sentences the calibration data C'
5: Sets < TRUNCATION-AWAL
DATA WHITENING(M, C)
6: Setw <- M
Obtain the of weights in M to compress
7: for W in Setw do
8:
S Sets(W))
Extract the whitening matrix of current weight W
9:
U. Σ, V < SVD(WS)
b Apply singular value decomposition on W Truncate the smallest singular values in E
Σ1 + (runc.(E)
W <- (E,) 1/2 W (ET)
Obtain two low-rank matrices
12:
(WW + Wy. W
> Replace W with W and Wy in L,
end for
14: M" +- PARAMETER UPDATE WITH SEQUENTIAL LOW-RANK APPROXIMATION (M')
return M"
16: end procedure
Algorithm 2 Pseudocode Truncation-Awan Data Whitening
1: Input: M: Original LLM
2: Input: C: Calibration Data
3: Output: Sets: Set of whitening matrices for the weight to compress in M
4: procedure TRUNCATION-ANE AWARE DATA WHITENING(M, C))
5: Set, <-
b Initialize the set of whitening matrices Obtain the set of weights in M to compress
6: Setw <- M
7: for W in Setw do
8:
X < M (W, C)
Obtain the input activation of the weight matrix W
9:
5 Cholesky Decoosly_Dccommpssttionn///
& Apply cholesky decomposition on XXT Store the whitening weight matrix in the set
10: Sets < SU Sets
11: end for
return Sets
13:
end procedure
Algorithm 3 Pseudocode of Parameter Update with Sequential Low-rank Approximation
1: Input: M': Compressed LLM by Truncation-aware Data Whitening
2: Output: M": Compressed LLM with Parameter Update with Sequential Low-rank Approxima-
tion
3: procedure PARAMETER UPDATE WITH SEQUENTIAL LOW-RANK APPROXIMATION(AM)) Fix all W fine-tune all W > Fix all W fine-tune all W
4: M'. <- LoRA,(M")
5: M + LORA,
6: return M
7: end procedure
14

---

Published as a conference paper at ICLR 2025
A.2 COMPRESSION LOSS OF ASVD
ASVD introduces a diagonal scaling matrix So that modifies the weight matrix to reflect the varying significance of different input channels. The linear layer is formulated Y S X. The compression is made by keeping the largest m singular value of W So:
WS. ~ Σ ou
1=1
The resulting activation is expressed as:
m
o
1 So S-1X. ouu
The compression error = 1IWSS - - ouf("" )5T 1-1x X|IF is demonstrated below:
L [[( So - Σ oufu' ) x11
dulu i So x
F
do xx
j=mtl 1im++1
Σ
Σ
xx= oo Trace(u) uol xxx=" (
j=m+1 i=m+1
Σ
e Traco T Xxx (x (S,1))
i=m+1
Σ i=m+1
of 11 x
which is still a complex function that involves the activation X, the diagonal matrix Sp, the singular vector of and the singular value df. As result, compression error is not directly related to the singular value, and the conventional SVD compression by truncating the smallest singular values may lead to suboptimal compression error.
A.3 COMPRESSION Loss OF SVD-LLM
In SVD-LLM, also formulate the linear layer as Y (WS)S- "x, where S X (S-1)² = I. The compression is made by keeping the largest m out of total r singular values of WS. The compression loss I. is demonstrated as:
15

---

Published as a conference paper at ICLR 2025
L7 [WX W'XI' || - S
(IWS S SVD(WS))5
(WS- - Σ l s-1x
0iui0 T s-! X
1=
E
010 Trace (uuuf' "ouf"s-IxX" ( "
j=m+1 i=m+1
E
Σ
0g Trace (.n? (s-1xx² ugus)
j= i=m+1
Σ
Σ
010j Trace (nuNTnn?))
j=m+1 i=++1
u 02 = u u,
1; = uf u, = 0, Trace (u,of) Trace (u,u,) = 1, j
L2
Σ j=m++1 imm++1
Σ
0, Trace (unuf =
Σ of Trace (nnvy v;u
Σ
d
i=m+1
Therefore, the squared loss 2 is equal to the sum of the squared singular values. Therefore, truncating the smallest singular values achieves the lowest compression loss.
A.4 SPECTRUM ANALYSIS OF SINGULAR VALUES DECOMPOSED BY SVD-LLM
In general, SVD is useful for compression when the matrix to be compressed shows sharp decay of the singular values. Since SVD-LLM decomposes the multiplication of the weight matrix W and its corresponding whitening matrix S instead of the original weight matrix W, which is different from the weight decomposition the previous work (Yuan al. 2023 Hsu al. 2022). to study whether SVD compression is also applicable in SVD-LLM, we select the Query (W Q) and Key (Wx) weight matrices and show the spectrum of singular values of their multiplication with corresponding
whitening matrices SQ and Sk. As shown in Figure 6) most of the single values are less than or around 100 with only few extremely large values, indicating that SVD is applicable in SVD-LLM.
A.5 COMPARISON WITH DRONE
Previous work DRONE (Chen et al. 2021 also proposes their data-aware method for SVD compres- sion. They even provide a theoretical analysis to prove the optimal solution that their method achieves. Specifically, DRONE represents the low-rank compressed weight matrix W by W. It performs SVD on both weight matrix W = S V and the transpose of input activation XT U.S. and then split these decomposed matrices as follows:
Uw = [wer Uwr 1,S
Sw. 0
0
Vw = Viir Vwr ]
Sx.t
0
Ux = Ux Ux, 1,Sx
Vx = Vx.e
Vx, ].
where r and k are the rank the W and X. Uw.r, Vw.r Ux.1, Vx, denote corresponding row spaces and column spaces and Uw.r Vw.r. Ux,,, Vx, null spaces. Through theoretical deduction,
16

---

Published as a conference paper at ICLR 2025
Spectrum of Singular Values
10a
800
30
10
40
10a
10
O
0
ORD
10-2
80
00
00
10-4
1
3
5
6
8
10 11 12 13 14 15 16 17 18 19 20 22 23
25 27 28
30 31
Layer Index
(a) Wq X SQ
Spectrum of Singular Values
10
8
10
10a
090
8
8
09
0
10-2
00 SORM
-4-
8
10 12 13 17 2021 24
Layer Index
(b) WK Sx
Figure 6: The singular value spectrum of the decomposed matrices across layers in LLaMA-7B.
DRONE converts the minimization of compression loss IJWX -X'X|====I''' WMX F the minimization of [w,, VTT W., Vx,tVxx Sw., VT M Vx, whose optimal value Liain is the rank-k truncated SVD of = Sw.r Vi. Vx, by the fundamental property of SVD decom- position. To achieve the optimal value, DRONE formulates solution M Sw Z VT where Z is the rank-k SVD truncation of Z.
In short, compared with DRONE, SVD-LIMM is also optimal with the same theoretical compression loss DRONE. Moreover, SVD -LM has three key advantages. Below is detailed explanation.
SVD-LLM is also optimal with the same theoretical compression loss as DRONE. The theoretical minimum compression loss Lmin is the F-norm loss of rank-k SVD truncation of W X, which has also been achieved by DRONE in their paper. Unlike DRONE, SVD-LLM constructs the whitening matrix 5 so that S X is orthonormal. Therefore, decompose with SVD to U,, S, V,, we can have S, = Sr. U, = Uz. Us Uz, V, = QV.. where Q is orthogonal matrix. The matrix WS to which SVD-L.IM applies SVD could be represented
have ||AS X
||A]] F.. Suppose that
17

---

Published as a conference paper at ICLR 2025
Table Compression loss of the randomly generated weight and activation matrices with different shapes under 50% compression ratio using SVD LVDD-LLM, Drone, and the theoretical minimum.
Loss
[128 x 128] [128 x 128] 22048 x 2048) x [2048 x 2048]
(4096 x 4096) x [4096 x 4096]
MINIMUM
276.1130
17784.2637
50321.9141
DRONE
276.1130
17785.6992
50337.2148
SVD-LLM
276.1130
17784.2676
50321.9777
Table 10: Compression Time of the randomly generated weight and activation matrices with different shapes using SVD-LLM and Drone. The compression time is measured for 5 times' compression.
TIME
[28 128] x [128 128] [2048 2048] x [2048 x 2048] [4096 4096 [4096 x 4096]
DRONE
0.07 seconds
5.81 seconds
30.35 seconds
SVD-LLM
0.02 seconds
1.98 seconds
10.37 seconds
by S. UU, UVV.V" Suppose that we use Trunc.(C) to represent the rank-k truncation of the matrix C during SVD compression, the compression loss L is derived as follows:
L =|WX-- W' X|Is= |I SVD(WS)SS X)][= IIWWS SVD(WS))
[Trune (Ws) [ru)c.(WS))]
||Trunc. U.S. =
- [[Trunc(WWX"")
Lmin
Therefore, SVD-LLM shares the same theoretical compression loss DRONE.
Advantage #1: DRONE incurs out-of-memory when compressing LLMs due to its requirement of storing the full large-scale activations, whereas SVD-LLM is feasible. To achieve awtrenwwssssss during compression, DRONE caches all input activations and spans them to calculate the corre- sponding singular vectors and singular values. In the DRONE paper, the authors apply DRONE to small LMs such as BERT. However, the activations generated by LLMs are often extremely large and are much larger than BERT. For example, compress LLaMA-7B with 5,000 calibration data by DRONE, the total memory to cache the activation for a single weight matrix at time is 5000 (data number) 256 (seq_len) len) 11008 (dim) x 32 (fp32) 1024 + 1024 1024 419GB, which is more than 5 times larger than the memory provided by the NVIDIA A100 GPU, which has 80GB memory. Therefore, applying DRONE for LLM compression is infeasbile.
In contrast, SVD-L.IM incrementally updates its X matrix by adding the xx. of each new input J. As such, LID eliminates the need to store the full activations, which requires only the storage of the matrix X which is considerably smaller than the full input activation. To compress LLaMA-7B with 5,000 calibration data, SVD LLM requires only 11008 X 1 11008 x 32 1024 + 1024 + 1024- 3.6GB. Compared to DRONE, SVD LLM achieves 116.38 times less memory reduction than DRONE. In our paper, we use WikiText-2 as a dataset. If we use DRONE, the total memory to cache the activation of single weight matrix is larger than 24.600GB In contrast, SVD-LIM still requires 3.6GB of memory, which is more than 6,000 times less than DRONE. Due to this advantage, SVD-LLM is much more practical compress LLMs of size 7B larger compared to DRONE.
Advantage #2: SVD-LLM incurs much shorter compression time compared to DRONE. DRONE involves more complex matrix operations, leading to longer compression time compared to SVD- -LLM. To illustrate this, we measured the time required by DRONE and SVD -VDM to compress randomly generated weight and activation matrices of varying shapes under 50% compression ratio. The results show that SVD-LLM is approximately three times faster than DRONE.
Advantage #3: SVD-LLM has better numerical stability, which leads to superior empirical compression loss. While SVD-LLM shares the same theoretical compression loss as DRONE, DRONE's higher complexiry-stemming stemming from additional SVD operations and inverse calculations on large-scale matrices-makes it less numerically stable compared to in higher empirical compression losses. To illustrate this, we compare SVD -LLM and DRONE in terms of empirical compression losses for randomly generated matrices of various shapes. We also include the theoretical minimum value, represented by the rank- Frobenius norm loss of WX. The results are summarized in the following table. As shown, we observe that empirical compression losses than DRONE, underscoring its superior numerical stability.
This often results
achieves lower
18

---

Published as a conference paper at ICLR 2025
Table 11: Perplexity (L) of SVD-LIM and FLAP WikiText-2 compress LLaMA-7B under different compression ratios. The better performance is marked in bold. The relative performance gain of SVD-LIM compared FLAP is marked in green inside bracket.
RATIO (MEM.)
20% (10.2GB)
40% (7.76GB)
60% (5.35GR)
80% (2.58GB)
FLAP
7.99
14.43
106.87
15023
SVD-LLM
7.73 (13%%
9.27 (1.3%%
15.00
(1869%)
31.79 (1999))
Table 12: Comparison of LLaMA-3B (compressed from LLaMA-7B by SVD-LLM) and original StableLM-3B (Tow al.) trained from scratch. Both the throughput and the peak memory footprint during the inference are measured with batch size=32, sequence length = 128 on single A100 GPU.
Moom
Throughput
Peak Mem.
Openb.
Are
WinoG.
HellaS.
PRQA
MathQA
Average
TrathalQA:
GSM8K1
Stabiel
8463 Tokenekec
0.51
LLaMA-3B
Tokenokee
0.46 (15%) (+ 0.01) 0-04
A.6 COMPARISON WITH FLAP
Recent work FLAP An et al. 2024) is also a post-training gructured-pruning method. Below
compare the perplexity of SVD SVD-LLM and FLAP on WikiText-2 under different compression ratios when compressing LLaMA-7B, As shown in Table IT SVD-LLM consistently outperforms FLAP, especially under high compression ratios.
A.7 COMPARISON WITH SMALLER LLMS PRE-TRAINED FROM SCRATCH
To compare the performance between SVD- SVD-LLM and scratch training, following the previous ex- perimental design (Ma et al. 2023), we compress LLaMA-7B to the size of the 3B parameter with SVD-LLM and select StableL MM3B (Tow et al.) as the baseline for comparison. As shown in [2ble[[2] LLaMA-3B compressed from LLaMA-7B by SVD-LLM achieves better accuracy in all datasets, indicating that SVD-LIM L.M could even achieve better accuracy than some scratch training methods. Fur- thermore, SVD-LLM ensures higher throughput and lower memory consumption than StableI.M331 as shown in the table, which also meets our efficiency analysis and discussion in Section 4.2
Compression time of SVD-LLM and ASVD LLaMA-7B under 20% compression ratio. The relative speedup is marked green color inside bracket.
Table 13:
SVD-LLM
ASVD
Truncation-Awee
Parameter Update with Sequential
Total
Normalize
Search
Total
Data Whitening
Low-rank Approximation
10min
3.5h
3.5h (136%)
Smin
5.5h
5,5h
A.8 COMPRESSION SPEED EVALUATION
In addition to compression performance, we also evaluate the compression speed SVD-LLM and the baselines. Specifically, we measured the GPU hours used for SVD-LLM and ASVD when compressing LLaMA-7B under 20% compression ratio on an A100 GPU. The results are shown in Table 13 As shown, ASVD takes about 5.5 hours, while SVD-LLX completes the compression process in 3.5 hours, which is 36% times faster. When breaking down the time, most the time consumed by ASVD is dedicated to searching for the specific compression ratio for each weight matrix based on its calculated importance score. In contrast, SVD-LLM maintains a consistent compression ratio in all weight matrices and thus eliminates the time-consuming search process.
19

---

Published as a conference paper at ICLR 2025
A.9 CONTENTS GENERATED FROM LLMS COMPRESSED BY SVD-LLM AND ASVD
Some examples of sentences generated by LLaMA-7B compressed with SVD-LLM and ASVD are shown in Table [4 The sentences generated by the model compressed by SVD-LLM exhibit better fluency, relevance, and informativeness compared to those compressed by ASVD. More importantly, when the compression ratio is increased to 40%, the previous state-of-the-an method ASVD completely loses its generation ability.. In contrast, even when the compression ratio is up to 80%, SVD-LLM is still capable of generating complete sentences.
Table 14: An example of contents generated by the fine-tuned LLaMA-7B with LoRA after being compressed by SVD-LLM and ASVD under different compression ratios. The input is marked in bold and the normal texts are the generated sentences.
RATIO
SVD-LLIX
ASVD
Original
What the responsibility of the Al assistant? The responsibility of Al assistant complete the job wilhout any hesitation. Also, the Al assistant has no problem with taking a decision and executing it. An Al assistant is responsible to needs to learn from his past experience take the best possible decision without any hesitation...
complete the task
20%
What is the responsibiltty of the Al assistant? The Al assistant's role and responsihilities are often ambigueus and vary depending on the context and the type of assistant heing used. Some Al assistants may have presel responses common queries, but their behavior can also depend on the user's 's interaction and expectations...
What is the responsibility of the Al assistant? 1. To act the same way as programmer intended. To make value assessment of the user's intentions and offer appeopriate response. 3. To help the user carry out specific tasks or follow strict script. 4. To make system independent of the user as possible...
40%
responsibility of the assistant? 1. The Al assistant is responsible for performing tasks that only he done by Al technology such as scheduling uppointments, setting reminders and making suggestions, while also pro- viding quick and accurate responses. 2. The Al assistant also responsible for learning and updating its knowledge base through comtnnouu improvement. thus making it more efficient and effective. 3. The Al assistant is responsible for handling and managing user generated and customer- oriented input...
What is the responsihility of the AT assistant? Define the definition of the jeb specification 2. Deline the role of the Al assistant for the joh. 3, Define the skills necessary erder to carryout the tasks assigned work of the same cupacity in the market. 4. Determine the evaluation of the job competence of the Al assistant ensure their professional integrity. S. What type of Al assistant johs are available? 6. What industries require technology assistance work complete the tasks that it own personnel cannot perform...
job do
60%
AT assistant? nut- shell, the responsibilit the Al assistant could vary depend- ing on the task. hut generally, the focus would be on auto- matic tusks, without need human intervention. Some common tasks could include setting reminders, scheduling appointments, and making routine household or productivity tasks. The Al assistant also serves backup relief system, taking on responsibilites when the user is not avail- ...
What responsibility
assistant? 2.3. ?? Brush for discussion wonder is il worth doing is important 2. isn't useful just reducing lahor costs; it helps employees leel sense connected your alten- tion which lends to hetter workplace values aroong staffers and leads to long relationships among org...
What the responsibility
80%
What is the responsibility of the Al assistant? Our Design based on our understanding of world, and we are actively learning. adapting and adapting, so we'ne 're always evolving new ideas, which we see to be most unique and relevant our community...
What is the responsibility of the Al ussistant? ygua Alchtemperaturen2. (64mb2/.3/ 3/ orb these puthebout les bnvolss merginels...
A. MORE ABLATION STUDIES
SVD-LLM + Normal LoRA Fine-tuning v.s. SVD-LLM + Sequential LoRA fine-tuning, To illustrate the superiority of the designed parameter update with the sequential low-rank approximation in SVD-LLM, which is kind of sequential LoRA fine-tuning strategy over the normal LoRA fine- tuning strategy, we compare the compression performance of SVD-LLM by applying either of these two fine-tuning strategies. Let's denote SVD-LLM (SFT) as SVD-LIM by applying sequential LoRA fine-tuning and SVD- LIM (NFT) as SVD-LLM by applying normal LoRA fine-tuning. As shown in Table [5| SVD- LLM (SFT) consistently outperforms SVD-LLA LLM (NFT), which also reaffirms our analysis in Section 3.2 that optimizing both low-rank matrices W., W, at the same time is not stable and may lead to poor fine-tuning performance.
ASVD + Sequential LoRA Fine-tuning v.s. SVD-LLM + Sequential LoRA Fine-tuning, Although the designed sequential LoRA fine-tuning strategy could also be applied in other SVD-based LLM compression methods, other methods' performance is still poorer than SVD-LLM even when inte- grated with this strategy for enhancement. To illustrate this, we compare the performance of the
20

---

Published as a conference paper at ICLR 2025
Table 15: Perplexity ()) of SVD-LLM with original LoRA fine-tuning (denoted as SVD- LLM (SFT)), ASVD with sequential LoRA fine-tuning (denoted as ASVD (SFT)), and SVD-LLM with sequential LoRA fine-tuning (denoted as SVD-LLM (SFT)) on WikiText-2 to compress LLaMA-78 under different compression ratios.
RATIO (MEM.)
20%
(10.2GB)
40%
(7.76GB)
60% (5.35GB)
80% (2.58GB)
SVD-LIM (NFT)
7.87
11.98
16.30
80.23
ASVD (SFT)
8.37
14.86
44.81
271
SVD-LLM (SFT)
7.73
9.27
15.00
31.79
previous state-of-the-art method ASVD when applied with the sequential LoRA finetuning with SVD-LIM. Let's denote SVD-LLA (SFT) as SVD SVD-LLM by applying sequential LoRA fine-tuning and ASVD (SFT) ASVD by applying sequential LoRA fine-tuning, As shown in Table 15 SVD-ILM (SFT) consistently outperforms ASVD (SFT) under various compression ratios.
A.11 LIMITATIONS
There are three limitations of WVD-LM, which are left for future work.
(1) The compression accuracy still needs to be improved under the high compression ratio. Although SVD-LLV -LMM has achieved the state-of-the-eee performance compared to previous works such FWSVD and ASVD, its compression accuracy still suffers from degradation, especially under the high compression ratio. To enhance the practicability of SVD-LIM in real-world scenario, its accuracy should be least comparable to that of the quantization method, including both low-bit and high-bit quantization, rather than being combined with the quantization methods for usage.
(2) The latency of SVD-LIM needs to be optimized while being used to compress the KV cache. As discussed in Section 4.2] SVD-Lu SVD-LLM can also be applied to compress the KV cache for memory saving during inference. However, this benefit does come free. In fact, due to the additional calculation caused by recovering the original key and value states, the inference speed will be impacted by compressing the KV cache with SVD-LLM, which should be optimized in the future.
(3) SVD-LLM should be better guided for high-quality generation. Although SVD-LLM achieves low perplexity, as demonstrated in Table is still possible to generate low-quality content, such as repeated words even under low compression ratios. This phenomenon of compressed LLM has also been mentioned in previous work (Ma et al.) 2023) and should be eliminated the future by guiding the compressed LLM for outputting high-quality generation.
21


Reviews:
[{"review_id": "zwBXA26YpY", "reviewer": "Reviewer_3PXg", "summary": "The authors investigate the compression of large language models through low rank approximation. They point out that the existing works in this direction have the following two problems. First, existing works approximate the weight matrix without considering the input activation matrix. Second, existing works do not finetune the model after the truncation. Consequently, they propose SVD-LLM, which addresses both issues by data whitening and LoRA finetuning. Their experiments show that their method not only outperforms other SVD approximation methods, but also outperforms other compression methods like structured pruning or model quantization. Furthermore, real inference speedup and memory saving can be observed without specialized hardware.", "questions": "1. Can you clarify the difference/advantage compared to Drone? Specifically, are the methods equivalent (It seems not so as I do not see a simple connection between Cholesky and eigendecomposition)? If not, can you show that your method is also optimal?\nI acknowledge that the experiment is done at a different scale (Bert vs LLM), so verifying this direction works at a larger scale itself is a contribution, but seems not enough for a full paper.  \n2. Can you show the performance or LoRA fine tuning without the sequential finetuning process proposed by the paper? This can be one of the novelties of the paper but requires careful verification. The failure of original finetuning might be because of bad choice of hyperparameters for training.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and easy to follow.\n2. The empirical comparison is extensive. The authors not only compared with the SVD-based baselines, but also compared with other compression methods like structured pruning and quantization to demonstrate the superiority of the proposed method.\n3. Real inference speedup and memory saving can be observed without specialized hardware. Unlike some unstructured pruning and/or quantization methods, which require specialized hardware for speedup, this makes the proposed method more practical and applicable.", "weaknesses": "1. As mentioned by SAC, the concept of data aware compression is not new and already exists in [1]. Additionally, in [1] they claim optimality for the problem considered, while this paper seems to offer weaker guarantees. (From my understanding, this paper shows that given the basis/decomposition of their choice, pruning the smallest entries of the middle diagonal matrix is optimal. On the other hand, [1] claim that both their choice of the basis and the truncation is optimal)\n2. The LoRA training part of this paper also seems not to be novel as existing in LLM-pruner. In addition, the claim for the necessity of sequential finetuning for W_u, W_v seems odd. For lora finetuning of transformers, similar structure exists for key and query attention matrix W_k, W_q, but existing literature does not report any issues for fine tuning.\n\n\n[1] Chen, P., Yu, H. F., Dhillon, I., & Hsieh, C. J. (2021). Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34, 29321-29334.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gkMfvMDkA1", "reviewer": "Reviewer_P8yJ", "summary": "With the remarkable capability of large language models (LLMs) in the field of natural language processing, their large model size becomes a major obstacle for practical deployment. To this end, the paper proposes a Singular Value Decomposition (SVD)-based LLM compression method, which aims to address two key limitations faced by existing SVD methods in compression: the truncation of small singular values that may lead to higher compression loss and the lack of model parameter updating after SVD truncation.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces SVD-LLM, a novel SVD-based compression method that incorporates a truncation-aware data whitening strategy to directly map singular values to compression loss, allowing for more informed truncation decisions.\n- The paper provides a rigorous mathematical foundation for the proposed truncation-aware data whitening technique, supported by theoretical proofs and lemmas.\n- It proposes a parameter update mechanism with sequential low-rank approximation to compensate for accuracy degradation after compression.\n- The authors evaluate SVD-LLM across various datasets and models from three different LLM families at various scales, demonstrating its superiority over existing SVD-based methods, especially at high compression ratios.", "weaknesses": "- The authors should evaluate SVD-LLM on the latest models in LLaMA family, such LLaMA-3-8B.\n- Authors are recommended to analyze the computation costs of SVD-LLM and provide the evaluation of its time expense.\n- No comparison with the latest SOTA methods, such as FLAP: Fluctuation-based Adaptive Structured Pruning for Large Language Models. Yongqi An, et al. AAAI 2024, available on arXiv since Dec. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHf0zwxSlh", "reviewer": "Reviewer_8cQQ", "summary": "This paper introduces a LLM compression approach based on SVD called SVD-LLM. SVD-LLM incorporates a whitening process to maintain the orthogonality of activations and align the number of retained singular values and the compression loss. After compressed via SVD, SVD-LLM further finetune the model for better performance.", "questions": "1. The paper claims to use LoRA finetuning in Section 3.2 but it is not clear how it is done. Does this introduces additional parameters, or fused afterward? Or the finetuning is directly applied on $W_u$ and $W_v$?\n2. Section 4.2, the authors discuss memory reduction for the KV cache at the expense of recomputation overhead. How does this impact throughput, and by how much?\n3. In Section 4.4 and Table 8, SVD-LLM is combined with QuIP#. More details are needed regarding how this combination is executed.\n4. In Algorithm 3,  there is a layer-wise closed-form update procedure that was not mentioned in the paper.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The activation whitening is supported by theoretical analysis, which is convincing.\n2. The experiments are extensive, demonstrating improvements comparing to selected baselines.\n3. SVD-LLM delivers practical benefits in terms of inference speedup and memory reduction on hardware.", "weaknesses": "1. The comparisons in the main experiments might be \\textbf{unfair}, as SVD-LLM adds extra finetuning additional finetuning that is not applied to other SVD-based baselines, which could drastically improve performance under high compression ratios. Extra finetuning is not a novel technique and can be seamlessly integrated into the baselines.\n2. Theorem 3.2 and Corollary 3.3 only hold when $S$ is calculated on the fly, but in practice SVD-LLM precomputes $S$ using a calibration set, and it is unclear if the orthogonality of activations  is preserved with this approach. Further experiments are needed here to address this gap, for example, the orthogonality of whitened activations using the precomputed $S$.\n3. The paper lacks validation of the method on larger models, such as those at the 70B scale, which is crucial for evaluating the feasibility of model compression approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nkrdb4IiHo", "reviewer": "Reviewer_kQW6", "summary": "This paper proposes SVD-LLM, a novel SVD-based LLM compression method that addresses efficiency issues in previous methods. It establishes a direct mapping between singular values and compression loss with theoretic proof. Extensive experiments demonstrate the effectiveness of the proposed method and support the author’s claims.", "questions": "The concept of “SVD-LLM (U)” in S4.3 - \"Modular Sensitivity Study\" needs to be clarified. From what I understand, the core part of SVD-LLM is the truncation-aware data whitening, the concept of “SVD-LLM (U)” thus needs to be clarified since SVD-LLM will be degraded to normal truncated-SVD when \"truncation-aware data whitening\" is disabled.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. SVD-LLM can achieve state-of-the-art in terms of compression efficiency and performance on downstream tasks.\n2. The inference speedup that SVD-LLM can achieve is inspiring given SVD-LLM’s good performance on PPL and downstream tasks.\n3. The research is well presented and it is easy to follow.", "weaknesses": "(a) “SVD for LLM Compression” part in “Related works” is insufficient, and it can be improved by discussing more papers related to SVD-based LLM compression and papers that relate to the Atomic Feature Mimicking (AFM) technique, where the latter is also dedicated to model decomposition for LLM compression. Papers that authors should be aware of are listed as follows.\n\n1. Compressing Transformers: Features Are Low-Rank, but Weights Are Not!, AAAI’23\n\n2. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML’23\n\n3. Adaptive Rank Selections for Low-Rank Approximation of Language Models, NAACL’24\n\n(b) The experiments on the latest model are missing. Authors should test their proposed method, i.e., SVD-LLM, and compare it with baselines on the latest model such as LLaMA-3-8B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)