SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

## Abstract

Abstract The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitate LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression.
However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weight after SVD truncation.
In this work, we propose SVD-LLM , a new SVD-based LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening strategy to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a layer-wise closed-form model parameter update strategy to compensate for accuracy degradation under high compression ratios.
We evaluate SVD-LLM on a total of 10 10 10 datasets and eight models from three different LLM families at four different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios.

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide range of tasks such as natural language understanding and language generation [, ].
Despite such capabilities, the democratization of LLMs is primarily restricted by their substantial resource demands [, ].
One of the most effective techniques to reduce the resource demands of LLMs is model compression [].
Compression techniques based on quantization [, , ], parameter pruning [, ], and knowledge distillation [, ] specifically designed for LLMs have been intensively studied.
Regardless of their success, these techniques have their own constraints, such as hardware dependency and the need for expensive retraining.
Compared to those techniques, compression techniques based on low-rank approximation, such as Singular Value Decomposition (SVD) are not limited by those constraints.

Despite these advantages, the potential of SVD for LLM compression has not been thoroughly explored. A few SVD-based LLM compression methods such as ASVD [] and FWSVD [] have recently been proposed. However, these methods exhibit severe performance degradation when the model compression ratio(^1^11The compression ratio refers to the percentage of parameter reduction achieved through compression.) is high.
Such limitation can be attributed to two fundamental issues involved in their approaches:
❶ Imprecise Data Preprocessing: although the data preprocessing strategy proposed by ASVD reduces the negative impact of activation outliers, it does not establish a direct relationship between singular values and the model compression loss. As a consequence, truncating smaller singular values in SVD could lead to significant compression loss.
❷ Lack of Model Parameter Update after SVD Truncation:
as the model compression ratio increases, the number of singular values that need to be truncated in SVD increases as well. To compensate for the accuracy degradation caused by truncating a large number of singular values, it is required to update the remaining parameters in the compressed model.
Unfortunately, existing SVD-based LLM compression methods do not take such update into account, and thus fail to compensate for the accuracy degradation under high model compression ratios.

In this paper, we propose a new SVD-based LLM compression method named SVD-LLM that effectively addresses the two fundamental issues of the existing methods.
SVD-LLM differs from existing SVD-based LLM compression methods in two key aspects:
❶ Truncation-Aware Data Whitening:
Supported by the theoretical proof, SVD-LLM incorporates a truncation-aware data whitening technique that ensures a direct mapping between singular values and model compression loss. In doing so, the proposed truncation-aware data whitening technique is able to identify which singular values should be truncated to incur minimal model compression loss.
❷ Layer-Wise Closed-Form Model Parameter Update:
to compensate for accuracy degradation under high compression ratios, SVD-LLM incorporates a layer-wise closed-form model parameter update strategy
to progressively update the compressed weights layer by layer.

We compare SVD-LLM with three SVD-based methods for LLM compression, including vanilla SVD as well as state-of-the-art methods FWSVD and ASVD. To demonstrate the generability of SVD-LLM, we conduct our evaluation on a total of $10$ datasets and eight models from three different LLM families (LLaMA, OPT, and Mistral) at four different scales (7B, 13B, 30B, and 65B).
We highlight six of our findings:
(1) SVD-LLM consistently outperforms vanilla SVD, FWSVD, and ASVD across all $10$ datasets, three different LLM families, and four different scales and even exhibits significant advantages under high compression
ratios from 30% to 60%.
(2) SVD-LLM also exhibits superiority over the state-of-the-art in terms of compression speed. Specifically, when compressing LLaMA-7B under 20% compression ratio on an A100 GPU, ASVD takes about $5.5$ hours whereas SVD-LLM completes the compression process in $15$ minutes.
(3) The independent performance of either of the two key components of SVD-LLM still consistently surpasses the performance of the current state-of-the-art SVD compression method under different compression ratios.
(4) SVD-LLM can benefit other LLM compression methods. Our evaluation results show that SVD-LLM is able to further enhance the compression performance of well-recognized quantization (GPTQ []) and parameter pruning-based (LLM-Pruner []) LLM compression methods.
(5) SVD-LLM can ensure inference speedup on both GPU and CPU. It is able to achieve at most 1.7x speedup on GPU and 1.5x speedup on CPU under the 40% compression ratio.
(6) Lastly, SVD-LLM brings additional benefit beyond compressing the sizes of LLMs, and is also able to reduce the footprint of KV cache during inference at runtime.

## 2 Related Work

Large Language Model Compression:
LLMs in general contain billion-scale parameters. Applying conventional model compression methods for LLMs is not feasible as they necessitate retraining. To avoid retraining, post-training methods that do not involve retraining LLMs in the compression process have been developed.
In general, these methods can be grouped into four categories: unstructured pruning, structured pruning, quantization, and low-rank approximation.
Specifically, unstructured pruning methods set the individual weights’ elements to zero without changing its shape. A notable contribution is SparseGPT [] which prunes the least important weight elements with the inversion of the Hessian matrix. However, the irregular sparsification of unstructured pruning is difficult to achieve the desired speedup or memory saving and can only demonstrate its best efficiency on certain hardware architecture such as NVIDIA Ampere GPU.
Unlike unstructured pruning, structured pruning methods directly remove entire channels or other structured components from LLMs, making them easier to implement on hardware. For example, LLM-Pruner [] utilizes a small amount of data to obtain the weight, parameter, and group importance of the coupled structure for pruning with LoRA to recover precision. However, due to the great modification of the weight matrix in LLM, it suffers from a great accuracy degradation, especially under high compression ratios.
Quantization methods, on the other hand, achieve model compression by reducing the precision of weight matrices of an LLM. For example, GPTQ [] uses layer-wise quantization and updates the weights with inverse Hessian information. However, quantization has the drawback of only providing a limited range of compression options, typically ranging from 3 to 8 bits. This limited range could prevent full utilization of the available memory budget.

Figure: Figure 1: Overview of SVD-LLM.
Refer to caption: /html/2403.07378/assets/x1.png

SVD for LLM Compression:
Singular Value Decomposition (SVD) is a widely used technique to reduce matrix size by approximating a matrix with two smaller low-ranking matrices [].
In the context of LLM compression, only a few SVD-based LLM compression methods have been proposed.
Specifically, vanilla SVD only focuses on the compression of the original weight matrix without considering the importance of the parameters, potentially giving a larger compression error.
To address this problem, [] propose FWSVD, which introduces Fisher information to weigh the importance of parameters. However, FWSVD requires a complex gradient calculation that demands substantial resources for LLM compression.
Another problem of vanilla SVD is the distribution of activation can affect the compression error. To address this issue, [] propose ASVD, which scales the weight matrix by a diagonal matrix that represents the impact of input channels on the weights.
However, both FWSVD and ASVD do not establish a direct relationship between singular values and compression loss. As a result, truncating the smaller singular values may lead to higher compression loss. Moreover, as the compression ratio increases, it is necessary to update the compressed weight due to truncating a great number of singular values. However, existing methods have no design for this update and thus incur severe accuracy degradation under high compression ratios.

## 3 SVD-LLM

[Figure 1](#S2.F1) provides an overview of SVD-LLM.
At a high level, SVD-LLM is a SVD-based post-training LLM compression method.
Specifically, following the standard procedure of post-training LLM compression methods [, , ],
SVD-LLM uses a random set of sentences as calibration data to generate activation for truncation-aware data whitening and layer-wise closed-form update for model compression.
SVD-LLM whitens the activation through Cholesky decomposition, and performs SVD to truncate the weight matrices to compress the LLM.
Under high model compression ratios, SVD-LLM performs a layer-wise closed-form update to progressively update the remaining weights layer by layer after compression.
In the following, we describe both truncation-aware data whitening and layer-wise closed-form update in detail. The pseudocode is provided in [Section A.2](#A1.SS2).

### 3.1 Truncation-Aware Data Whitening

Motivation:
Due to high variance of the input activation, simply applying vanilla SVD for LLM compression leads to severe accuracy degradation []. To address this issue, ASVD [] formulates LLM compression as an optimization problem with the following optimization objective:

$$ $O=\min(||WX-W^{\prime}X||_{F})$ (1) $$

where $W$ is the weight matrix of the original LLM, $X$ is the activation of $W$ given an input, $W^{\prime}$ is the compressed weight matrix, and $L=||WX-W^{\prime}X||_{F}$ is the compression loss in the form of Frobenius loss.

Specifically, ASVD extracts a diagonal matrix $S_{0}$ from $X$ where each element in the diagonal is the absolute mean value of each channel.
It then uses $S_{0}$ to normalize $X$ and converts $WX$ into $(WS_{0})(S_{0}^{-1}X)$.
Subsequently, SVD is performed on $WS_{0}$ to obtain the decomposed matrices $U_{0}$, $\Sigma_{0}$, and $V_{0}$. Lastly, ASVD truncates the smallest singular values in $\Sigma_{0}$ to obtain the compressed weight matrix $W^{\prime}_{0}=U_{0}\times\text{Trunc.}(\Sigma_{0})\times V_{0}\times S_{0}^{-1}$.

Although normalizing the activation improves the performance, ASVD does not establish a direct relationship between singular values and compression loss (a detailed proof is included in [Section A.1](#A1.SS1)).
To better illustrate this point, we show two concrete examples in LABEL:fig:asvd_whitening.
In the first example ❶ where only one singular value is truncated, truncating the smallest singular value 0.1 results in a higher compression loss (loss $=$ 1.1) than truncating the second smallest singular value 0.9 (loss $=$ 0.7).
In the second example ❷ where multiple singular values are truncated, truncating the smallest two singular values 0.9 and 0.1 also leads to a higher loss (loss $=$ 1.9) than truncating 2.4 and 0.1 (loss $=$ 1.7).
Hence, truncating the smallest singular values does not lead to minimal loss.

Key Design:
The key idea of SVD-LLM is to incorporate a truncation-aware data whitening technique that ensures a direct mapping between singular values and compression loss.
To achieve this, SVD-LLM enforces the whitened activation $S^{-1}X$ to be orthogonal such that each channel is independent of each other, i.e., $(S^{-1}X)(S^{-1}X)^{T}=S^{-1}XX^{T}(S^{-1})^{T}=I$, where $S$ is derived through Cholesky decomposition [].
Then we perform SVD on $WS$ to obtain the decomposed matrices $U,\Sigma,V$, where $U=[u_{1},u_{2},u_{3},...,u_{r}]$, $\Sigma=\text{diag}(\sigma_{1},\sigma_{2},\sigma_{3},\cdots,\sigma_{r})$, and $V=[v_{1},v_{2},v_{3},...,v_{r}]$. Lastly, the smallest singular values in $\Sigma$ are truncated to obtain the compressed weight matrix $W^{\prime}=U\times\text{Trunc.}(\Sigma)\times V^{T}\times S^{-1}$.

LABEL:fig:svdllm_whitening illustrates the effect of the proposed truncation-aware data whitening technique. In the first example ❶ where only one singular value is truncated, the compression loss is equal to the truncated singular value. In the second example ❷, the compression loss of truncating multiple singular values is equal to the square root of the sum of their squares.
As such, under the proposed truncation-aware data whitening technique, truncating the smallest singular values leads to minimal compression loss.

Below, we provide a theoretical proof on why the proposed truncation-aware data whitening technique ensures a direct mapping between singular values and compression loss in the case of one singular value ([Theorem 3.2](#S3.Thmtheorem2)) and multiple singular values ([Corollary 3.3](#S3.Thmtheorem3)).

###### Lemma 3.1 .

The Frobenius norm of matrix $A$ with dimension $m\times n$ can be deduced into the square root of the trace of its gram matrix, which is:

$$ $\|A\|_{F}\triangleq\left(\sum_{j=1}^{n}\sum_{i=1}^{m}\left|a_{ij}\right|^{2}\right)^{\frac{1}{2}}=\left[\operatorname{trace}\left(A^{T}A\right)\right]^{\frac{1}{2}}$ (2) $$

Using [Lemma 3.1](#S3.Thmtheorem1), we obtain the compression loss $L_{i}$ when truncating the $i^{th}$ singular value of $S^{-1}X$ to reduce its rank for compression:

$$ $L_{i}=||(W-W^{\prime})X||_{F}=||\sigma_{i}u_{i}v_{i}^{T}S^{-1}X||_{F}=\sigma_{i}~{}trace(u_{i}v_{i}^{T}S^{-1}XX^{T}(S^{-1})^{T}v_{i}u_{i}^{T})^{\frac{1}{2}}$ (3) $$

Since both $U=[u_{1},u_{2},u_{3},...,u_{r}]$ and $V=[v_{1},v_{2},v_{3},...,v_{r}]$ are orthogonal matrices, we have:

$$ $v_{i}^{T}v_{i}=u_{i}^{T}u_{i}=I;v_{i}^{T}v_{j}=u_{i}^{T}u_{j}=0,\forall i\neq j;trace(v_{i}v_{i}^{T})=trace(u_{i}u_{i}^{T})=1$ (4) $$

###### Theorem 3.2 .

If $S$ is the Cholesky decomposition of $XX^{T}$, the compression loss $L_{i}$ equals to $\sigma_{i}$.

###### Proof.

Since the whitening matrix $S$ is the Cholesky decomposition of $XX^{T}$, we have $SS^{T}=XX^{T}$. We can further infer [Equation 3](#S3.E3) to obtain:

$$ $L_{i}=||\sigma_{i}u_{i}v_{i}^{T}S^{-1}X||_{F}=\sigma_{i}~{}trace(u_{i}v_{i}^{T}S^{-1}XX^{T}(S^{-1})^{T}v_{i}u_{i}^{T})^{\frac{1}{2}}=\sigma_{i}~{}trace(u_{i}v_{i}^{T}v_{i}u_{i}^{T})^{\frac{1}{2}}=\sigma_{i}\\$ (5) $$

Therefore, $L_{i}$ of truncating $\sigma_{i}$ equals to the singular value $\sigma_{i}$ itself.
∎

###### Corollary 3.3 .

If $S$ is the Cholesky decomposition of $XX^{T}$, truncating the smallest singular values leads to the lowest loss $L$ compared to truncating others.

###### Proof.

If we truncate $\sigma_{m+1},\sigma_{m+2},\sigma_{m+3},...,\sigma_{r}$ in $\Sigma$ for compression, the square of the loss $L$ is:

$$ $\begin{split}L^{2}=&\left|\left|\sum_{i=m+1}^{r}\sigma_{i}u_{i}v_{i}^{T}S^{-1}X\right|\right|^{2}_{F}=\sum_{j=m+1}^{r}\sum_{i=m+1}^{r}\sigma_{i}\sigma_{j}~{}trace(u_{i}v_{i}^{T}S^{-1}XX^{T}(S^{-1})^{T}v_{j}u_{j}^{T})\\ =&\sum_{i=m+1}^{r}\sigma_{i}^{2}~{}trace(u_{i}v_{i}^{T}S^{-1}XX^{T}(S^{-1})^{T}v_{i}u_{i}^{T})=\sum_{i=m+1}^{r}(L_{i})^{2}=\sum_{i=1}^{k}(\sigma_{i})^{2}\end{split}$ (6) $$

The squared loss $L^{2}$ is equal to the sum of the squared singular values. Therefore, truncating the smallest singular values $\sigma_{i}$ achieves the lowest compression loss.
∎

###### Lemma 3.1 .

###### Theorem 3.2 .

###### Proof.

###### Corollary 3.3 .

###### Proof.

### 3.2 Layer-Wise Closed-Form Update

Motivation:
Given the same calibration data as input, the compressed weight matrix $W^{\prime}$ generates a new activation $X^{\prime}$ that is different from $X$ generated by the original weight matrix $W$.
As the compression ratio increases, $W$ needs to truncate a larger number of singular values to obtain $W^{\prime}$, thus $X^{\prime}$ deviates further from $X$.
Therefore, it becomes necessary to design a strategy for updating $W^{\prime}$ to minimize $||WX^{\prime}-W^{\prime}X^{\prime}||_{F}$.
However, existing SVD-based LLM compression methods
have no design of parameter update after compression, leading to less competitive performance at high compression ratios.

Figure: Figure 3: Layer-Wise Closed-Form Update.
Refer to caption: /html/2403.07378/assets/x2.png

Key Design:
The key idea of SVD-LLM is to incorporate a layer-wise closed-form strategy to update $W^{\prime}$ to minimize $||WX^{\prime}-W^{\prime}X^{\prime}||_{F}$.
[Figure 3](#S3.F3) illustrates the overall process.
To perform the update in layer $i$, SVD-LLM uses the new activation $X^{\prime}_{i-1}$ from the previous layer $i-1$ that has been updated.
To perform the update without destroying the low-rank structure of $W^{\prime}_{i}$, SVD-LLM only updates the matrix $U_{i}$ to its closed-form solution $U^{\prime}_{i}$ while keeping $\text{Trunc.}(\Sigma)_{i}$ and $V_{i}$ fixed as:

$$ $\begin{split}U_{i}^{\prime}&=\arg\min_{U_{i}^{\prime}}||W_{i}X^{\prime}_{i-1}-W^{\prime}_{i}X^{\prime}_{i-1}||_{F}\\ &=W_{i}X^{\prime}_{i-1}D^{T}(DD^{T})^{-1},D=\text{Trunc.}(\Sigma)_{i}V_{i}^{T}S_{i}^{-1}X^{\prime}_{i-1}\end{split}$ (7) $$

## 4 Experiments

Baselines.
We compare SVD-LLM against three baselines including vanilla SVD as well as state-of-the-art SVD-based LLM compression methods FWSVD [] and ASVD [].

Models and Datasets.
To demonstrate the generability of our method, we evaluate the performance of SVD-LLM and the baselines on eight models from three different LLM families (LLaMA-7B, 13B, 30B, 65B [], LLaMA2-7B [], OPT-6.7B [], Vicuna-7B [] and Mistral-7B []) and $10$ datasets including three language modeling datasets (WikiText-2 [], PTB [] and C4 []) and seven classification datasets (OpenbookQA [], WinoGrande [], HellaSwag [], PIQA [], MathQA [], ARC-e, and ARC-c []) in zero-shot setting with LM-Evaluation-Harness framework [].

Implementation Details.
To ensure a fair comparison, we followed ASVD [] to randomly select 256 samples from WikiText-2 as the calibration data.
Since layer-wise closed-form update is intended to mitigate the accuracy drop under higher compression ratios, we only apply it when the compression ratios are at 40% and above.
All of our experiments are conducted on Nvidia A100 GPUs.

### 4.1 Overall Performance

We evaluate the overall performance of SVD-LLM from four aspects: (1) performance under different compression ratios, (2) performance on different LLMs, (3) performance on LLMs with larger scales, and (4) performance with LoRA fine-tuning (See [Section A.3](#A1.SS3)). Some generated contents by the compressed LLM are listed in [Section A.4](#A1.SS4) to provide a more straightforward comparison.

Performance under Different Compression Ratios.

**Table 1: Zero-shot performance of LLaMA-7B compressed by SVD-LLM and baselines under 20% to 60% compression ratio on three language modeling datasets (measured by perplexity ($\downarrow$)) and seven common sense reasoning datasets (measured by both individual and average accuracy ($\uparrow$)). The best performance is marked in bold. The relative performance gain compared to the best-performing baseline is marked in green color inside bracket.**
| Ratio | Method | WikiText-2$\downarrow$ | PTB$\downarrow$ | C4$\downarrow$ | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA | MathQA | Average$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0% | Original | 5.68 | 8.35 | 7.34 | 0.28 | 0.67 | 0.67 | 0.56 | 0.38 | 0.78 | 0.27 | 0.52 |
| 20% | SVD | 20061 | 20306 | 18800 | 0.14 | 0.27 | 0.51 | 0.26 | 0.21 | 0.53 | 0.21 | 0.31 |
| FWSVD | 1727 | 2152 | 1511 | 0.15 | 0.31 | 0.50 | 0.26 | 0.23 | 0.56 | 0.21 | 0.32 |  |
| ASVD | 11.14 | 16.55 | 15.93 | 0.25 | 0.53 | 0.64 | 0.41 | 0.27 | 0.68 | 0.24 | 0.43 |  |
| SVD-LLM | 7.94 ($\downarrow$29%) | 16.22 ($\downarrow$2%) | 15.84 ($\downarrow$1%) | 0.22 | 0.58 | 0.63 | 0.43 | 0.29 | 0.69 | 0.24 | 0.44 ($\uparrow$2%) |  |
| 30% | SVD | 13103 | 17210 | 20871 | 0.13 | 0.26 | 0.51 | 0.26 | 0.21 | 0.54 | 0.22 | 0.30 |
| FWSVD | 20127 | 11058 | 7240 | 0.17 | 0.26 | 0.49 | 0.26 | 0.22 | 0.51 | 0.19 | 0.30 |  |
| ASVD | 51 | 70 | 41 | 0.18 | 0.43 | 0.53 | 0.37 | 0.25 | 0.65 | 0.21 | 0.38 |  |
| SVD-LLM | 9.56 ($\downarrow$81%) | 26.39 ($\downarrow$62%) | 25.11 ($\downarrow$39%) | 0.20 | 0.48 | 0.59 | 0.37 | 0.26 | 0.65 | 0.22 | 0.40 ($\uparrow$5%) |  |
| 40% | SVD | 52489 | 59977 | 47774 | 0.15 | 0.26 | 0.52 | 0.26 | 0.22 | 0.53 | 0.20 | 0.30 |
| FWSVD | 18156 | 20990 | 12847 | 0.16 | 0.26 | 0.51 | 0.26 | 0.22 | 0.53 | 0.21 | 0.30 |  |
| ASVD | 1407 | 3292 | 1109 | 0.13 | 0.28 | 0.48 | 0.26 | 0.22 | 0.55 | 0.19 | 0.30 |  |
| SVD-LLM | 13.11 ($\downarrow$99%) | 63.75 ($\downarrow$98%) | 49.83 ($\downarrow$96%) | 0.19 | 0.42 | 0.58 | 0.33 | 0.25 | 0.60 | 0.21 | 0.37 ($\uparrow$23%) |  |
| 50% | SVD | 131715 | 87227 | 79815 | 0.16 | 0.26 | 0.50 | 0.26 | 0.23 | 0.52 | 0.19 | 0.30 |
| FWSVD | 24391 | 28321 | 23104 | 0.12 | 0.26 | 0.50 | 0.26 | 0.23 | 0.53 | 0.20 | 0.30 |  |
| ASVD | 15358 | 47690 | 27925 | 0.12 | 0.26 | 0.51 | 0.26 | 0.22 | 0.52 | 0.19 | 0.30 |  |
| SVD-LLM | 23.97 ($\downarrow$99%) | 150.58 ($\downarrow$99%) | 118.57 ($\downarrow$99%) | 0.16 | 0.33 | 0.54 | 0.29 | 0.23 | 0.56 | 0.21 | 0.33 ($\uparrow$10%) |  |
| 60% | SVD | 105474 | 79905 | 106976 | 0.16 | 0.26 | 0.50 | 0.26 | 0.22 | 0.52 | 0.21 | 0.30 |
| FWSVD | 32194 | 43931 | 29292 | 0.15 | 0.26 | 0.49 | 0.26 | 0.22 | 0.53 | 0.18 | 0.30 |  |
| ASVD | 57057 | 45218 | 43036 | 0.12 | 0.26 | 0.49 | 0.26 | 0.21 | 0.51 | 0.18 | 0.29 |  |
| SVD-LLM | 53.74 ($\downarrow$99%) | 438.58 ($\downarrow$99%) | 345.49 ($\downarrow$99%) | 0.14 | 0.28 | 0.50 | 0.27 | 0.22 | 0.55 | 0.21 | 0.31 ($\uparrow$7%) |  |

First, we evaluate the performance of LLaMA-7B compressed by SVD-LLM and the baselines under 20% to 60% compression ratios.
[Section 4.1](#S4.SS1) summarizes the results on all $10$ datasets.
As shown, SVD-LLM consistently outperforms vanilla SVD, FWSVD and ASVD across all of the compression ratios.
More importantly, compared to the low compression ratio scenario in [Section 4.1](#S4.SS1), SVD-LLM exhibits significant advantages over vanilla SVD, FWSVD, and ASVD under high compression ratios.
Specifically, under 30% compression ratio, compared to the best-performing baseline (ASVD), SVD-LLM reduces the perplexity on WikiText-2, PTB, and C4 by 81%, 62%, and 39%, respectively; When the compression ratio reaches 40% and above, SVD-LLM reduces the perplexity by more than 96%.
These results indicate that SVD-LLM is more effective in compressing LLMs for more resource-constrained devices such as smartphones and IoT devices.
On the seven classification datasets,
SVD-LLM performs better than the best-performing baseline on most of the datasets and consistently achieves at least 2% higher average accuracy across all the compression ratios.

Performance on Different LLMs.
To examine the generability of SVD-LLM across different LLMs, we compare the performance between SVD-LLM and the baselines on four different models, including OPT-6.7B, LLaMA 2-7B, Mistral-7B, and Vicuna-7B under 20% compression ratio on WikiText-2 and the common sense reasoning datasets (See [Section A.5](#A1.SS5)).
The result on WikiText-2 is shown in [Section 4.1](#S4.SS1),
SVD-LLM consistently outperforms vanilla SVD, FWSVD, and ASVD across all four LLMs.
In addition, SVD-LLM exhibits more stable performance on different LLM families, especially compared to vanilla SVD and FWSVD.

Figure: Table 2: Perplexity ($\downarrow$) of four different LLMs including OPT-6.7B, LLaMA 2-7B, Mistral-7B, and Vicuna-7B under 20% compression ratio on WikiText-2. The relative performance gain compared to the best-performing baseline is marked in green color inside bracket.

Performance on LLMs with Larger Scales.
To examine the generability of SVD-LLM on LLMs across different scales, we compare the performance between SVD-LLM and the baselines on LLaMA series at four different scales – 7B, 13B, 30B, and 65B – under 20% compression ratio on WikiText-2.
As shown in [Section 4.1](#S4.SS1), SVD-LLM consistently outperforms vanilla SVD, FWSVD, and ASVD across all four model sizes.
Moreover, both FWSVD and ASVD demand excessive memory resources, causing out of memory (OOM) when compressing LLMs at larger scales even on an A100 GPU due to memory-intensive operations for estimating the importance of weight matrices.
In contrast, SVD-LLM does not involve such estimation operations and thus avoids OOM.

### 4.2 Compression Speed Evaluation

Besides compression performance, we also evaluate the compression speed of SVD-LLM and the baselines.
Specifically, we measured the GPU hours used for SVD-LLM and ASVD when compressing LLaMA-7B under the 20% compression ratio on an A100 GPU. The results are shown in [Section 4.2](#S4.SS2).
As shown,
ASVD takes about $5.5$ hours whereas SVD-LLM completes the compression process in $15$ minutes, which is $32$ times faster.
When breaking down the time, most of the time consumed by ASVD is dedicated to calculating the compression ratio of each weight matrix based on its estimated importance through a search process.
In contrast, SVD-LLM maintains a consistent compression ratio among all weight matrices and thus gets rid of the time-consuming search process.

Figure: Table 4: Compression time of SVD-LLM and ASVD on LLaMA-7B under 20% compression ratio. The relative speedup is marked in green color inside bracket.

### 4.3 Ablation Study

**Table 6: Perplexity ($\downarrow$) of compressed LLaMA-7B on WikiText-2. SVD-LLM (W) denotes the version of SVD-LLM with truncation-aware data whitening only; SVD-LLM (U) denotes the version of SVD-LLM with layer-wise closed-form update only; SVD-LLM (W+U) denotes the version of SVD-LLM with both truncation-aware data whitening and layer-wise closed-form update. The relative performance gain compared to ASVD is marked in green color.**
| Method | 20% | 30% | 40% | 50% | 60% |
| --- | --- | --- | --- | --- | --- |
| ASVD | 11.14 | 51 | 1407 | 15358 | 57057 |
| SVD-LLM (W) | 7.94 ($\downarrow$29%) | 9.56 ($\downarrow$81%) | 13.73 ($\downarrow$99%) | 26.11 ($\downarrow$99%) | 66.62 ($\downarrow$99%) |
| SVD-LLM (U) | 9.54 ($\downarrow$14%) | 12.98 ($\downarrow$75%) | 24.16 ($\downarrow$99%) | 72.13 ($\downarrow$99%) | 204 ($\downarrow$99%) |
| SVD-LLM (W+U) | 8.25 ($\downarrow$26%) | 9.95 ($\downarrow$80 %) | 13.11 ($\downarrow$99%) | 23.97 ($\downarrow$99%) | 53.74 ($\downarrow$99%) |
| SVD-LLM | 7.94 ($\downarrow$29%) | 9.56 ($\downarrow$81%) | 13.11 ($\downarrow$99%) | 23.97 ($\downarrow$99%) | 53.74 ($\downarrow$99%) |

Modular Sensitivity Study:
We conduct ablation studies to evaluate the separate contributions of the two key components (truncation-aware data whitening and layer-wise closed-form update) of SVD-LLM.
Let SVD-LLM (W) denote the version of SVD-LLM with truncation-aware data whitening only; SVD-LLM (U) denote the version of SVD-LLM with layer-wise closed-form update only; and SVD-LLM (W+U) denote the version of SVD-LLM with both truncation-aware data whitening and layer-wise closed-form update.
The results are shown in [Section 4.3](#S4.SS3). We have three observations.
(1) Both SVD-LLM (W) and SVD-LLM (U) consistently outperform ASVD across all the compression ratios. Notably, when the compression ratio is at and above 40%, both variants reduce the perplexity by more than 99% compared to ASVD.
(2) Under 20% and 30% compression ratios, SVD-LLM (W) achieves the lowest perplexity compared to SVD-LLM (U) and SVD-LLM (W+U).
(3) Under 40%, 50% and 60% compression ratios, SVD-LLM (W+U) achieves the lowest perplexity compared to SVD-LLM (W) and SVD-LLM (U), highlighting the importance of combining both truncation-aware data whitening and layer-wise closed-form update when compression ratio goes high.

Calibration Data Analysis:
We next analyze the impact of calibration data used for both truncation-aware data whitening and layer-wise closed-form update on the compression performance.
LABEL:fig:cali and [Section 4.2](#S4.SS2) summarize the performance of compressed LLaMA-7B when changing three key characteristics of the calibration data, the number of the calibration data, the seed used to randomly sample the calibration data, and the data set from which the calibration data is sampled.
As shown, changing any of the three characteristics only causes a tiny disturbance of less than 15% to the final performance, demonstrating that SVD-LLM is less sensitive to the design of the calibration data to compress LLM.

### 4.4 Benefits to other LLM Compression Methods

SVD-LLM is orthogonal to other LLM compression methods including quantization and parameter pruning.
In this experiment, we combine SVD-LLM with quantization and parameter pruning-based LLM compression methods that are widely recognized by the community to examine how SVD-LLM could further enhance their performance.

Integrate SVD-LLM with Quantization.
We select GPTQ [] as the quantization method.
Specifically, we compress LLaMA-7B by GPTQ-4bit combined with SVD-LLM, and compare the compressed model against LLaMA-7B compressed by GPTQ-3bit.
As shown in [Section 4.4](#S4.SS4), combining GPTQ-4bit with SVD-LLM achieves a perplexity that is $18\%$ lower than GPTQ-3bit even with a smaller memory footprint (2.1 GB vs. 2.8 GB). This result demonstrates that compared to directly quantizing using smaller number of bits, GPTQ achieves better compression performance with the help of SVD-LLM.

Integrate SVD-LLM with Parameter Pruning.
We select LLM-Pruner [] as the parameter pruning method.
Specifically, we compress LLaMA-7B by LLM-Pruner under 30% compression ratio combined with SVD-LLM, and compare the compressed model against LLaMA-7B compressed by LLM-Pruner under 40% compression ratio.
As shown in Table [4.4](#S4.SS4), LLM-Pruner achieves better compression performance when used in conjunction with SVD-LLM. In particular, with the same memory footprint of 8.8 GB, combining LLM-Pruner under 30% compression ratio with SVD-LLM achieves a perplexity that is $13\%$ lower than LLM-Pruner under 40% compression ratio.

Figure: Table 7: Perplexity ($\downarrow$) of LLaMA-7B compressed by GPTQ w/ and w/o SVD-LLM on WikiText-2. The relative performance gain of combined compression compared to GPTQ-3bit is marked in green color inside bracket.

### 4.5 Benefits of Inference Speedup

SVD-LLM is capable to achieve inference speedup.
To demonstrate this advantage, we measure the number of tokens that the original LLaMA-7B and its compressed version by SVD-LLM can generate on average per second with different batch size and sequence length.  LABEL:fig:gpu_throughput show the results on the GPU and CPU.
As shown, SVD-LLM consistently ensures an acceleration in the generation speed across all the compression ratios illustrated in the figure. More importantly, this enhancement becomes more significant as the batch size increases and the sequence length decreases, resulting in a maximum speedup of 1.7x on GPU and 1.5x on CPU under the 40% compression ratio, where the model performance remains acceptable according to [Section 4.1](#S4.SS1). These results highlight the effectiveness of SVD-LLM in improving the efficiency of LLM for real-world usage.

[Uncaptioned image]: /html/2403.07378/assets/x3.png

Figure: Figure 10: Peak memory to generate 128 tokens with batch size of 32 using LLaMA-7B compressed by SVD-LLM under different compression ratios w/ and w/o KV-cache compression. The difference between the blue and yellow bars marked in red indicates the reduced footprint of the KV cache.
Refer to caption: /html/2403.07378/assets/x4.png

### 4.6 Benefits of KV Cache Compression

SVD-LLM is able to not only compress LLMs but also compress the runtime KV cache at the same time.
Specifically, instead of keeping the original intermediate state matrix $m=WX$ with shape $M\times L$ inside the KV cache, after decomposing and compressing $W$ into $W_{u}W_{v}^{T}$, SVD-LLM only needs to store $m^{\prime}=W_{v}^{T}X$ with shape $r\times L$. Therefore, the size of the KV cache can be compressed to $\frac{r}{M}$ of the original.
Moreover, since $W_{u}$ is already stored as the weight matrix in the decomposed LLM, the original intermediate state matrix can still be recovered by $m=W_{u}m^{\prime}$ without accuracy drop.
Therefore, SVD-LLM provides a unified solution that combines model compression and KV cache compression into a single process. This is different from existing quantization or parameter pruning-based LLM compression methods that need to be combined with other techniques for compressing both weights and KV cache..

In our last experiment, we evaluate this benefit on KV cache compression brought by SVD-LLM.
This is a new avenue since KV cache compression has not been evaluated in previous LLM compression studies.
Specifically, we measure the peak memory footprint during inference when generating 128 tokens with batch size of 32 using LLaMA-7B compressed by SVD-LLM under different compression ratios w/ and w/o considering KV cache compression.
The results are illustrated in [Figure 10](#S4.F10) where the difference between the blue and yellow bars marked in red represents the reduced footprint of the KV cache.
As shown, SVD-LLM is able to effectively reduce the footprint of KV cache. Therefore, the peak memory during inference at runtime across all the compression ratios.

## 5 Conclusion

In this paper, we presented SVD-LLM, a SVD-based
LLM compression method.
SVD-LLM proposes a novel truncation-aware data whitening strategy to guide which singular values to be truncated with minimal compression loss. It also introduces a layer-wise closed-form model parameter update scheme to compensate for accuracy degradation under high compression ratios.
We have demonstrated the effectiveness of SVD-LLM on $10$ datasets and seven models from three LLM families at four scales and have shown its superiority over state-of-the-arts. We also show its effectiveness in further enhancing the performance of other LLM compression methods.

## Appendix A Appendix.

### A.1 The compression error of ASVD

The previous state-of-the-art method ASVD introduced a diagonal scaling matrix $S_{0}$ that modifies the weight matrix to reflect the varying significance of different input channels. The linear layer is formulated as $Y=(WS_{0})S_{0}^{-1}X$. The compression is made by keeping the largest $m$ singular value of $WS_{0}$:

$$ $\displaystyle WS_{0}\approx$ $\displaystyle\sum_{i=1}^{m}\sigma^{\prime}_{i}u^{\prime}_{i}{v^{\prime}}_{i}^{T}$ $$

The resulting activation is expressed as:

$$ $\displaystyle Y\approx$ $\displaystyle\sum_{i=1}^{m}\sigma^{\prime}_{i}u^{\prime}_{i}{v^{\prime}}_{i}^{T}S_{0}^{-1}X~{}.$ $$

The compression error $L=||(WS_{0}-\sum_{i=1}^{m}\sigma^{\prime}_{i}u^{\prime}_{i}{v^{\prime}}_{i}^{T})S_{0}^{-1}X||_{F}$ is demonstrated below:

$$ $\displaystyle L^{2}=$ $\displaystyle||(WS_{0}-\sum_{i=1}^{m}\sigma^{\prime}_{i}u^{\prime}_{i}{v^{\prime}}_{i}^{T})S_{0}^{-1}X||_{F}^{2}$ $\displaystyle=$ $\displaystyle\left|\left|\sum_{i=m+1}^{r}\sigma^{\prime}_{i}u^{\prime}_{i}{v^{\prime}}_{i}^{T}S_{0}^{-1}X\right|\right|_{F}^{2}$ $\displaystyle=$ $\displaystyle\sum_{j=m+1}^{r}\sum_{i=m+1}^{r}\sigma^{\prime}_{i}\sigma^{\prime}_{j}~{}trace(u^{\prime}_{i}{v^{\prime}}_{i}^{T}XX^{T}v^{\prime}_{j}{u^{\prime}}_{j}^{T})$ $\displaystyle=$ $\displaystyle\sum_{j=m+1}^{r}\sum_{i=m+1}^{r}\sigma^{\prime}_{i}\sigma^{\prime}_{j}~{}trace({u^{\prime}}_{j}^{T}u^{\prime}_{i}{v^{\prime}}_{i}^{T}S_{0}^{-1}XX^{T}(S_{0}^{-1})^{T}v^{\prime}_{j})$ $\displaystyle=$ $\displaystyle\sum_{i=m+1}^{r}{\sigma^{\prime}}_{i}^{2}~{}trace({v^{\prime}}_{i}^{T}S_{0}^{-1}XX^{T}(S_{0}^{-1})^{T}v^{\prime}_{i})$ $\displaystyle=$ $\displaystyle\sum_{i=m+1}^{r}{\sigma^{\prime}}_{i}^{2}~{}||{v^{\prime}}_{i}^{T}S_{0}^{-1}X||_{F}^{2}~{},$ $$

which is still a complex function that involves the activation $X$, the diagonal matrix $S_{0}$, the singular vector $v^{\prime}_{i}$ and the singular value $\sigma^{\prime}_{i}$. As a result, compression error is not directly related to the singular value, and the conventional SVD compression by truncating the smallest singular values may lead to suboptimal compression error.

### A.2 Pseudocode for SVD-LLM

[Algorithm 1](#alg1) shows the pseudocode of SVD-LLM. Before compression, SVD-LLM randomly collects a small amount of sentences as the calibration data $C$, it then runs the truncation-aware data whitening process as shown in [Algorithm 2](#alg2) to obtain the set of whitening matrix $\text{Set}_{S}$ for the weight to compress. After that, it runs the SVD and truncation with $\text{Set}_{S}$ on each weight matrix in the LLM. Instead of directly finishing the whole compression, it stores the decomposed matrices and further utilizes these matrices to run the layer-wise closed-form update as shown in [Algorithm 3](#alg3).

Figure: Algorithm 1 Pseudocode for SVD-LLM

Figure: Algorithm 2 Pseudocode for Truncation-Aware Data Whitening

Figure: Algorithm 3 Pseudocode for Layer-Wise Closed-Form Update

**Table 9: Perplexity of LLaMA-7B compressed by SVD-LLM and ASVD (w/ and w/o LoRA) on WikiText-2.**
| Method | 20% | 30% | 40% | 50% | 60% |
| --- | --- | --- | --- | --- | --- |
| ASVD | 11.14 | 51 | 1407 | 15358 | 57057 |
| ASVD + LoRA | 7.37 | 10.16 | 14.86 | 21.83 | 44.81 |
| SVD-LLM | 7.94 | 9.56 | 13.11 | 23.97 | 53.74 |
| SVD-LLM + LoRA | 8.28 | 9.14 | 10.65 | 13.26 | 17.93 |

### A.3 Performance with LoRA Fine-Tuning.

LoRA [] is a common fine-tuning technique for LLM.
It has been applied with pruning-based LLM compression methods such as LLM-Pruner [] to mitigate accuracy drop after pruning.
LoRA can also be combined with SVD-based LLM compression methods by modifying the forward pass of a linear layer as:

$$ $\displaystyle Y$ $\displaystyle=W_{u}^{\prime}W_{v}^{\prime}X$ (8) $$

where $W_{u}^{\prime}=W_{u}+B_{u}A_{u}$, $W_{v}^{\prime}=W_{v}^{T}+B_{v}A_{v}$, and $A_{u}$, $B_{u}$, $A_{v}$, and $B_{v}$ are low-rank weights fine-tuned using LoRA.

To examine the performance of SVD-LLM in combination with LoRA, we follow the same configuration used in LLM-Pruner [] to fine-tune LLaMA-7B compressed by SVD-LLM and ASVD under the compression ratios from 20% to 80% with LoRA. The results are shown in [Section A.2](#A1.SS2).
We have three observations.
(1) Comparing SVD-LLM with SVD-LLM $+$ LoRA, under the compression ratio between 20% and 50%, the accuracy enhancement brought by LoRA is limited; as the compression ratio increases, LoRA plays a more significant role in improving the performance of SVD-LLM.
(2) Compared to ASVD $+$ LoRA, SVD-LLM $+$ LoRA consistently achieves better accuracy across all the compression ratios. In particular, under 70% compression ratio, the perplexity of SVD-LLM $+$ LoRA is 75% lower than that of ASVD $+$ LoRA.
(3) Finally, even without LoRA, SVD-LLM is able to achieve perplexity comparable to ASVD $+$ LoRA, especially under the compression ratios between 20% and 60%.

### A.4 Generated Content from the Compressed Model

We also compare some examples of sentences generated by LLaMA-7B compressed with SVD-LLM and ASVD in [Table 10](#A1.T10).
As shown, the sentences generated by the model compressed by SVD-LLM exhibit better fluency, relevance, and informativeness compared to that compressed by ASVD. More importantly, when the compression ratio is increased to 30%, the previous state-of-the-art method ASVD completely loses its generation ability. In contrast, even when the compression ratio is up to 40%, SVD-LLM is still capable of generating complete sentences.

**Table 10: An example of contents generated by the compressed LLaMA-7B by SVD-LLM and and under different compression ratios. The input is marked in bold and the normal texts are the generated sentences.**
| Ratio | SVD-LLM | ASVD |
| --- | --- | --- |
| Original | Building a website can be done in 10 simple steps: Step 1: Register an domain name; Step 2: Get a hosting package; Step 3: Develop a website; Step 4: Design the website; Step 5: Make the website live; Step 6: Perform on-page SEO; Step 7: Promote the website; Step 8: Evaluate the results; Step 9: Improve the website; Step 10: Build a website repeat … |  |
| 20% | Building a website can be done in 10 simple steps: 1) Identifying an appropriate theme. 2 ) Determining which types of Web site would be of interest and which could be implemented with minimal overhead and resources. 3 ) Identify the audience 4 ) Choose Web site software. 5 ) Select Web site developers. 6 ) Design Web site 7 ) Identify Web site technology 8 ) Design Web site layout 9 ) Define Web site aesthetics and 10 ) Define Web site access controls … | Building a website can be done in 10 simple steps: 1, Build up your website. You will need to select a name for your website. As a beginner, it is a good idea to select your company name or product name as your website name. You can use your real company or product name or you can create an unusual business name. This will make the name of your business name consistent throughout your website. 2, Decide on your business name. You will be choosing the official name of your business for all activities that represent your business. This includes your business name, products, your website, and any … |
| 30% | Building a website can be done in 10 simple steps: 1. Preliminaries and planning 2. Designing 3. Prototyping 4. Production 5. Operation… | Building a website can be done in 10 simple steps: To: With A Web You Are Into 2 Zim 40 Wh I M T If Y As An Onl You Orwice There W Os You Hcimp … |
| 40% | Building a website can be done in 10 simple steps: 2 days of website plan. If you want to set a web site. This is the 1st step. In the 2 days . It wants the company to develop a website to help the investors and investors. He will have another investor. He wants to help others. The company is working to grow, and create a name for some investors and investors. 100 million US dollars It is used to build a hotel and a hotel. In order to obtain their cooperation, investors, investors, investors in . | Building a website can be done in 10 simple steps: Front 1222ED825333 3333333333333333etennenn eenneenneSecondinjustanyanojunatajin … |

### A.5 More Experiments on compressing different LLMs

**Table 11: Zero-shot performance comparison of OPT-6.7B, LLaMA 2-7B, Vicuna-7B, Mistral-7B compressed by SVD-LLM and baselines under 20% compression ratio on seven common sense reasoning datasets (measured by both individual and average accuracy ($\uparrow$)). The best performance is marked in bold. The relative performance gain compared to the best-performing baseline is marked in green color inside bracket.**
| LLM | Method | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA | MathQA | Average$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OPT-6.7B | SVD | 0.14 | 0.27 | 0.51 | 0.25 | 0.23 | 0.54 | 0.21 | 0.31 |
| FWSVD | 0.15 | 0.26 | 0.49 | 0.26 | 0.20 | 0.52 | 0.21 | 0.30 |  |
| ASVD | 0.14 | 0.40 | 0.51 | 0.30 | 0.20 | 0.59 | 0.22 | 0.34 |  |
| SVD-LLM | 0.24 | 0.60 | 0.60 | 0.45 | 0.28 | 0.73 | 0.24 | 0.45 ($\uparrow$32%) |  |
| LLaMA 2-7B | SVD | 0.15 | 0.27 | 0.49 | 0.26 | 0.23 | 0.52 | 0.20 | 0.30 |
| FWSVD | 0.12 | 0.25 | 0.49 | 0.25 | 0.22 | 0.52 | 0.21 | 0.30 |  |
| ASVD | 0.25 | 0.31 | 0.60 | 0.41 | 0.32 | 0.72 | 0.23 | 0.41 |  |
| SVD-LLM | 0.26 | 0.50 | 0.60 | 0.41 | 0.26 | 0.66 | 0.23 | 0.41 ($\uparrow$0%) |  |
| Mistral-7B | SVD | 0.14 | 0.25 | 0.52 | 0.26 | 0.23 | 0.54 | 0.20 | 0.30 |
| FWSVD | 0.15 | 0.28 | 0.52 | 0.26 | 0.21 | 0.53 | 0.21 | 0.31 |  |
| ASVD | 0.21 | 0.51 | 0.58 | 0.42 | 0.25 | 0.50 | 0.26 | 0.39 |  |
| SVD-LLM | 0.17 | 0.55 | 0.58 | 0.36 | 0.25 | 0.67 | 0.21 | 0.40 ($\uparrow$3%) |  |
| Vicuna-7B | SVD | 0.15 | 0.26 | 0.50 | 0.26 | 0.23 | 0.52 | 0.20 | 0.30 |
| FWSVD | 0.14 | 0.27 | 0.49 | 0.26 | 0.22 | 0.53 | 0.20 | 0.30 |  |
| ASVD | 0.21 | 0.53 | 0.55 | 0.39 | 0.30 | 0.41 | 0.23 | 0.37 |  |
| SVD-LLM | 0.23 | 0.51 | 0.58 | 0.40 | 0.28 | 0.67 | 0.22 | 0.41 ($\uparrow$11%) |  |

We also evaluate the performance of different LLMs, including OPT-6.7B, LLaMA 2-7B, Mistral-7B, and Vicuna-7B under 20% compression ratio on seven common sense reasoning datasets. The results are shown in [Section A.5](#A1.SS5). SVD-LLM performs better than the best-performing baseline in most of the datasets across different LLMs and even achieves 32% higher average accuracy on OPT-6.7B.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you clarify the difference/advantage compared to Drone? Specifically, are the methods equivalent (It seems not so as I do not see a simple connection between Cholesky and eigendecomposition)? If not, can you show that your method is also optimal?
I acknowledge that the experiment is done at a different scale (Bert vs LLM), so verifying this direction works at a larger scale itself is a contribution, but seems not enough for a full paper.  
2. Can you show the performance or LoRA fine tuning without the sequential finetuning process proposed by the paper? This can be one of the novelties of the paper but requires careful verification. The failure of original finetuning might be because of bad choice of hyperparameters for training.
soundness: 3
strengths: 1. The writing is clear and easy to follow.
2. The empirical comparison is extensive. The authors not only compared with the SVD-based baselines, but also compared with other compression methods like structured pruning and quantization to demonstrate the superiority of the proposed method.
3. Real inference speedup and memory saving can be observed without specialized hardware. Unlike some unstructured pruning and/or quantization methods, which require specialized hardware for speedup, this makes the proposed method more practical and applicable.
summary: The authors investigate the compression of large language models through low rank approximation. They point out that the existing works in this direction have the following two problems. First, existing works approximate the weight matrix without considering the input activation matrix. Second, existing works do not finetune the model after the truncation. Consequently, they propose SVD-LLM, which addresses both issues by data whitening and LoRA finetuning. Their experiments show that their method not only outperforms other SVD approximation methods, but also outperforms other compression methods like structured pruning or model quantization. Furthermore, real inference speedup and memory saving can be observed without specialized hardware.
weaknesses: 1. As mentioned by SAC, the concept of data aware compression is not new and already exists in [1]. Additionally, in [1] they claim optimality for the problem considered, while this paper seems to offer weaker guarantees. (From my understanding, this paper shows that given the basis/decomposition of their choice, pruning the smallest entries of the middle diagonal matrix is optimal. On the other hand, [1] claim that both their choice of the basis and the truncation is optimal)
2. The LoRA training part of this paper also seems not to be novel as existing in LLM-pruner. In addition, the claim for the necessity of sequential finetuning for W_u, W_v seems odd. For lora finetuning of transformers, similar structure exists for key and query attention matrix W_k, W_q, but existing literature does not report any issues for fine tuning.


[1] Chen, P., Yu, H. F., Dhillon, I., & Hsieh, C. J. (2021). Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34, 29321-29334.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to the weaknesses.
soundness: 3
strengths: - The paper introduces SVD-LLM, a novel SVD-based compression method that incorporates a truncation-aware data whitening strategy to directly map singular values to compression loss, allowing for more informed truncation decisions.
- The paper provides a rigorous mathematical foundation for the proposed truncation-aware data whitening technique, supported by theoretical proofs and lemmas.
- It proposes a parameter update mechanism with sequential low-rank approximation to compensate for accuracy degradation after compression.
- The authors evaluate SVD-LLM across various datasets and models from three different LLM families at various scales, demonstrating its superiority over existing SVD-based methods, especially at high compression ratios.
summary: With the remarkable capability of large language models (LLMs) in the field of natural language processing, their large model size becomes a major obstacle for practical deployment. To this end, the paper proposes a Singular Value Decomposition (SVD)-based LLM compression method, which aims to address two key limitations faced by existing SVD methods in compression: the truncation of small singular values that may lead to higher compression loss and the lack of model parameter updating after SVD truncation.
weaknesses: - The authors should evaluate SVD-LLM on the latest models in LLaMA family, such LLaMA-3-8B.
- Authors are recommended to analyze the computation costs of SVD-LLM and provide the evaluation of its time expense.
- No comparison with the latest SOTA methods, such as FLAP: Fluctuation-based Adaptive Structured Pruning for Large Language Models. Yongqi An, et al. AAAI 2024, available on arXiv since Dec. 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The paper claims to use LoRA finetuning in Section 3.2 but it is not clear how it is done. Does this introduces additional parameters, or fused afterward? Or the finetuning is directly applied on $W_u$ and $W_v$?
2. Section 4.2, the authors discuss memory reduction for the KV cache at the expense of recomputation overhead. How does this impact throughput, and by how much?
3. In Section 4.4 and Table 8, SVD-LLM is combined with QuIP#. More details are needed regarding how this combination is executed.
4. In Algorithm 3,  there is a layer-wise closed-form update procedure that was not mentioned in the paper.
soundness: 3
strengths: 1. The activation whitening is supported by theoretical analysis, which is convincing.
2. The experiments are extensive, demonstrating improvements comparing to selected baselines.
3. SVD-LLM delivers practical benefits in terms of inference speedup and memory reduction on hardware.
summary: This paper introduces a LLM compression approach based on SVD called SVD-LLM. SVD-LLM incorporates a whitening process to maintain the orthogonality of activations and align the number of retained singular values and the compression loss. After compressed via SVD, SVD-LLM further finetune the model for better performance.
weaknesses: 1. The comparisons in the main experiments might be \textbf{unfair}, as SVD-LLM adds extra finetuning additional finetuning that is not applied to other SVD-based baselines, which could drastically improve performance under high compression ratios. Extra finetuning is not a novel technique and can be seamlessly integrated into the baselines.
2. Theorem 3.2 and Corollary 3.3 only hold when $S$ is calculated on the fly, but in practice SVD-LLM precomputes $S$ using a calibration set, and it is unclear if the orthogonality of activations  is preserved with this approach. Further experiments are needed here to address this gap, for example, the orthogonality of whitened activations using the precomputed $S$.
3. The paper lacks validation of the method on larger models, such as those at the 70B scale, which is crucial for evaluating the feasibility of model compression approaches.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: The concept of “SVD-LLM (U)” in S4.3 - "Modular Sensitivity Study" needs to be clarified. From what I understand, the core part of SVD-LLM is the truncation-aware data whitening, the concept of “SVD-LLM (U)” thus needs to be clarified since SVD-LLM will be degraded to normal truncated-SVD when "truncation-aware data whitening" is disabled.
soundness: 4
strengths: 1. SVD-LLM can achieve state-of-the-art in terms of compression efficiency and performance on downstream tasks.
2. The inference speedup that SVD-LLM can achieve is inspiring given SVD-LLM’s good performance on PPL and downstream tasks.
3. The research is well presented and it is easy to follow.
summary: This paper proposes SVD-LLM, a novel SVD-based LLM compression method that addresses efficiency issues in previous methods. It establishes a direct mapping between singular values and compression loss with theoretic proof. Extensive experiments demonstrate the effectiveness of the proposed method and support the author’s claims.
weaknesses: (a) “SVD for LLM Compression” part in “Related works” is insufficient, and it can be improved by discussing more papers related to SVD-based LLM compression and papers that relate to the Atomic Feature Mimicking (AFM) technique, where the latter is also dedicated to model decomposition for LLM compression. Papers that authors should be aware of are listed as follows.

1. Compressing Transformers: Features Are Low-Rank, but Weights Are Not!, AAAI’23

2. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML’23

3. Adaptive Rank Selections for Low-Rank Approximation of Language Models, NAACL’24

(b) The experiments on the latest model are missing. Authors should test their proposed method, i.e., SVD-LLM, and compare it with baselines on the latest model such as LLaMA-3-8B.

Meta-review: This paper proposes a low-rank compression algorithm for compressing a weight matrix in a neural network. The algorithm is truncation aware and is better than naive SVD. Experimental results show promising performance gain over existing low-rank compression algorithms on LLMs. 

The main strength of this paper is the strong empirical results, while the main concern from the reviewers is that the optimal low-rank compression derived from this paper is equivalent to the DRONE paper published a few years ago. After extensive discussions between authors and reviewers, the authors agree that the factorization formed in this paper is mathematically identical to DRONE, so the contributions of this paper are (1) Conducting a relative comprehensive experiments on recent LLMs, where DRONE was developed before ChatGPT/Llama so the experiments was mainly conducted on BERT. (2) The authors explicitly show that they can maintain the XX^T matrix in a streaming fashion to make it work for large-scale datasets/models. This was not explicitly mentioned in the DRONE paper. However, after the discussion closed, Reviewer 3PXg found that DRONE mentioned that the XX^T matrix can be maintained in the streaming manner, so this part of novelty becomes unclear. 

The paper received 8, 6, 6, 5, and reviewer 3PXg mentioned during discussion to reduce the rating from 6 to 5 due to the point (2) mentioned above, so the actual final rating is 8, 6, 5, 5 which is a borderline paper. Although the novelty may be limited compared to DRONE, given the strong empirical results the AC still votes for weak accept for this paper. 

If the paper is accepted, the AC and reviewers would like to see that the authors address point (2) mentioned above in the final version to better compare their work with DRONE.

Decision: Accept (Poster)