Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives
Authors: Wang Qinsi
Jinghan Ke
Masayoshi Tomizuka
Kurt Keutzer
Chenfeng Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Model Compression
Low-Rank Decomposition
SVD
Effecient LLM
Differentiable
Primary area: foundation or frontier models, including LLMs
TL;DR: We are the first to theoretically prove that truncating activations outperforms truncating weights, and we propose Dobi-SVD, the first SVD-based method to significantly compress LLM weights with minimal performance drop.

Abstract: Large language models (LLMs) have sparked a new wave of AI applications; however, their substantial computational costs and memory demands pose significant challenges to democratizing access to LLMs for a broader audience. Singular Value Decomposition (SVD), a technique studied for decades, offers a hardware-independent and flexibly tunable solution for LLM compression. In this paper, we present new directions using SVD: we first theoretically analyze the optimality of truncating weights and truncating activations, then we further identify three key issues on SVD-based LLM compression, including (1) How can we determine the optimal truncation position for each weight matrix in LLMs? (2) How can we efficiently update the weight matrices based on truncation position? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose an effective approach, **Dobi-SVD**, to tackle the three issues. 
First, we propose a **differentiable** truncation-value learning mechanism, along with gradient-robust backpropagation, enabling the model to adaptively find the optimal truncation positions. Next, we utilize the Eckart-Young-Mirsky theorem to derive a theoretically **optimal** weight update formula through rigorous mathematical analysis. Lastly, by observing and leveraging the quantization-friendly nature of matrices after SVD decomposition, we reconstruct a mapping between truncation positions and memory requirements, establishing a **bijection** from truncation positions to memory. 
Experimental results show that with a 40\% parameter-compression rate, our method achieves a perplexity of 9.07 on the Wikitext2 dataset with the compressed LLama-7B model, a 78.7\% improvement over the state-of-the-art SVD for LLM compression method. 
We emphasize that Dobi-SVD is the first to achieve such a high-ratio LLM compression with minimal performance drop.  We also extend our Dobi-SVD to VLM compression, achieving a 20\% increase in throughput with minimal performance degradation. We hope that the inference speedup—up to 12.4x on 12GB NVIDIA Titan Xp GPUs and 3x on 80GB A100 GPUs for LLMs, and 1.2x on 80GB A100 GPUs for VLMs—will bring significant benefits to the broader community such as robotics.

Published as a conference paper at ICLR 2025
DOBI-SVD: DIFFERENTIABLE SVD FOR LLM COM- PRESSION AND SOME NEW PERSPECTIVES
Qinsi Wang'; Jinghan Ke
Masayoshi Tomizuka',
Kurt Keutzer", Chenfeng Xu²
'Due University,
FUniversity of California, Berkeley
ABSTRACT
Large language models (LL.Ms) have sparked wave of Al applications; how- ever, their substantial computational costs and memory demands pose significant challenges democratizing access to LLMs for broader audience. Singular Value Decomposition (SVD), a technique studied for decades, offers hardware- independent and flexibly tunable solution for LLM compression. In this paper, we present new directions using SVD: we first theoretically and experimentally ana- lyze the optimality of directly truncating activations, then we further identify three key issues on SVD-based LLM compression, including (I) How can we determine the optimal truncation position for each layer in LLMs? (2) How can we efficiently update the weight matrices based on truncated activations? (3) How can address the inherent "injection" nature that results the information loss of the SVD? We propose new paradigm for SVD-based LLM compression, Dobi-SVD, to tackle the three issues. First, we propose a differentiable truncation mechanism, along with gradient-robust backpropagation, enabling the model adaptively find the optimal truncation positions. Next, we utilize the Eckart-Young- Young-Mirsk, theorem to derive theoretically optimal weight update formula through rigorous mathemati- cal analysis. Lastly. by observing and leveraging the quantization -iriiooly-feeyyyy nature of matrices after SVD, we reconstruct mapping between truncation positions and memory requirements, establishing a bijection from truncation positions to mem- ory. Experimental results show that with 40% parameter- comppeeseoooooosiiioon rate, our method achieves perplexity of 9.07 on the Wikitext2 dataset with the compressed LLama-7B model, 78. 78.7% improvement over the state-of-the-art SVD for LLM compression method. We emphasize that Dobi-SVD is the first to achieve such high-ratio LLM compression while maintaining competitive performance. We also extend our Dobi-SVD vision-language models (VLMss) and vision-language- action models (VLAs), thereby highlighting its generalizability and practical value. We hope that the inference speedup-up up 12.4x on 12GB NVIDIA Titan Xp GPUs and 3x on 80GB A100 GPUs for LLMs, 1.2x and 1.17x on 80GB A 100 GPUs for VLMs and VLAs, respectively -will bring significant benefits the broader community such as multi-modal learning and robotics etc.
INTRODUCTION
Large language models (LLMs), such as GPT (Achiam al.) (023). Llama (Touvron al. (023). and OPT (Zhang et al. 2022), have shown that scaling the size of the model and the training data can unlock impressive performance and contextual learning abilities. However, because of the growing number of parameters in LLMs and the limited memory capacity of current hardware, inference with LLMs is highly expensive. This limits their practical applications, especially for resource-constraing hardware devices and latency-sensicn this paper, we aim to address a key challenge: How can we perform
programs, such as robotics, edge-device applications, and
interactive entertainment. inference with LLMs using fewer computational resources and memory, while maintaining the performance of the pre-trained models?
QQisii Wang and Jinghan Ke contribute equally. Order is decided by coin flip. This work was done when
Jinghan Ke visited UC Berkeley.
I Chenfeng Xu advises the work and is the corresponding author.

---

Published as a conference paper at ICLR 2025
LPPrramter Renormalizaion: from Discrate Continuous
Optimization Landscape of Model Compression
4. Weight Update
Ronk Ratio
Congrention
0.8-
1.0
Original Model Size
Am XW
Auia
0.5
2 Robust Backpropagation
CA-
02-
Compressed Model Size
Loww22 Unsarl
Value
Singuter (0W
Compust the Gradient Los Function
1nwe24. Liearrty
IPCA
Wate
3M
NAN
Traiiin fearh
16 UnoacW
Levert
Tlayer Expandos
Leraew
Unegrdy
Ours: Remapping
Eijecton
Remapping
fraditionall
m
UE
Freguasyy
Aijection
Park
Value
3.Proposed Mechod
Figure 1: Overview framework of Dobi-SVD: 1-3: Differentiable Truncation Position Training. By applying parameter renormalization for continuous rank ratio selection and using Taylor expansion to prevent gradient explosion, our method enables robust and adaptive optimization of truncation positions. 4: Weight Update. Using IPCA, we sequentially extract and optimally update weight matrix features. 5: Remapping. We resolve a long-overlooked limitation of traditional SVD-based compression through remapping, fully unlocking SVD's potential for data compression.
Model compression (Buciluà et al., 2006 Cheng et al. 2017: Choudhary et al. 2020) has been extensively studied, with the aim of squeezing the original model into lightweight one with a low compression ratio, i.e., the ratio between the memory required by the compressed model and the original model. However, existing mainstream compression methods have their own limitations. For instance, quantization reduces the storage memory by converting into lower-bit integer calculations, but it lacks flexibility and hardware generalization due to the dependency on specific hardware support (Dettmers et al.] 2024; Lin et al.) 2024 Frantar et al., 2022a Kim al. 2023). Model pruning shrinks the network by reducing redundant parameters that are less sensitive to performance, but hardware-acceleratee structured pruning often leads to significant performance degradation (Ma et al. 2023 Frantar & Alistarh 2023; Ashkboos al.
grrating-poin calculations
2024; Xia et al., 2023). For example, when compressing LLaMA-7B. Lr-ner resulted in 37.6% performance drop on WikiText2,, even at a compression ratio of 0.8. Knowledge distillation
employs a high-complexity teacher network to guide lower-complexity student network for model compression; however, it requires retraining a new model, which incurs high time and computational costs (Bergmann et al. 2018 Hou et al.) 2020) Chen et al., 2020).
In contrast to these techniques, another straightforward compression method low-rank decompo- sition, which is free from the aforementioned limitations. As technique studied many decades, singular value decomposition (SVD) (Klema & Laub] 1980) has played a significant role in fields such image compression (Prasantha et al.] 2007; Bryt & Elad 2008), communication transmission (Lebrun et al. 2005), and signal denoising (Zhao Jial 2017]Guu Guo et al, 2015). However, its potential for L.M compression has not yet been fully explored. Theoretically. SVD reduces memory and computation by truncating the singular values of a matrix, decomposing large weight matrices into the product of two smaller matrices. Models compressed with SVD can be easily deployed across wide range of devices, and SVD offers the flexibility to compress models to any compression ratio.
However, existing SVD-based methods for LLM compression have not achieved desirable results. Traditional SVD (Lebedev et al.) 2014, Moczulski et al.] 2015) Sainath et al.) (013) truncated the model weights directly, often resulting in significant performance degradation and requiring extensive retraining. Recently proposed activation-awan aware SVD hopes that the truncated weights make the activations close to the original ones, but still fails to deliver satisfactory performance. For example, ASVD (Yuan et al., 2023) proposes scaling the weights using diagonal matrix to reduce the distribution error of the activations before and after truncation. SVD-LLM (Wang et al.. 2024) introduces truncation-awan data whitening strategy to retain singular values critical for activations. However, these methods exhibit severe performance degradation. At 0.4 compression ratio, SVD-LLM experiences 644.7% drop in model performance Wikitext2 dataset, which is unacceptable in real-world applications. Due to the significant performance loss, SVD has not yet become a mainstream method for L.LM compression compared to other compression techniques.
2

---

Published as a conference paper at ICLR 2025
We aim to change this landscape by making SVD viable and widely adopted option. Unlike previous weight-only or activation-awar methods focusing on minimizing the distance between new and original activations through weight-based truncation, open up novel path: directly truncating activations while enabling weight reconstruction without fine-tuning. Additionally, we are the first to fully utilize singular value information by addressing long-overlooked limitation of traditional DDDbbaed compression methods. To achieve these, we analyze three associated challenges:
1. How to determine the appropriate truncation position for each layer in an LLMP Different weight in the model have varying sensitivities to performance degradation. By designing specific truncation points for different weight matrices (i.e. the number of retained singular values), it is possible to achieve smaller performance loss at same compression ratio. However, due to the high dimensionality of the matrices in LLMs, the solution space for truncation positions is exceptionally large. Thus, efficiently finding the optimal combination of truncation is a significant challenge.
How to update the weights based on truncation position? Directly truncating the weights results in significant performance degradation. Although activation cetivation-wware SVD mitigates this issue to some extent, how to update the weights in way that maximally preserves activation information has not yet been fully explored or theoretically proven.
3. How to overcome the long-overlooked truncation-valle limitation? A fundamental issue with SVD-based compression is that achieve effective compression, at least half of the singular values need to be truncated (for square matrices). This implies that even a modest compression ratio requires very low ranks, leading to substantial information loss during matrix compression, directly limiting the capacity of SVD to compress models effectively.
We propose new paradigm, Dobi-SVD. An overview is shown in Fig. I First, through experimental analysis and theoretical investigation, we compare the effectiveness truncating activations versus truncating weights, verifying the superiority of activation truncating. Based on this analysis, we provide answers to the three key challenges raised. First, by introducing differentiable truncation values and stable SVD backpropagation, matrix in adapting to find optimal truncation points. Then, leveraging the Eckart- Young-Mirsky theorem and the positive definiteness of the SVD, we derive the theoretical optimal weight update formula and employ Incremental Principal Component Analysis (IPCA) to extract features of updated weight sequentially and address memory constraints. Finally, we propose novel SVD-based storage method that takes advantage of the value concentration property of decomposed matrices. This method establishes bijection mapping between the truncation position and the model compression ratio, thereby overcoming the truncation limitation.
enable the model's performance to directly guide the
Dobi-SVD effectively addresses the three challenges of SVD in compressing LLMs. Specifically, our method achieves performance breakthroughs in three key areas. In terms of task performance, Dobi-SVD achieves minimal performance degradation even with high compression ratios for the first time. When compressing the LLaMA-7B model to compression ratio of 0,4, the Dobi-SVD compressed model reaches PPL. of 9.07 on WikiText2, which represents a 78.5% improvement over the previous state-of ahe- SVD compression method, and a 9.83% improvement over the best pruning methods that require post-training and fine-tuning. This establishes SVD as an effective and highly competitive option for LLM compression; In terms of hardware performance, Dobi- SVD significantly reduces inference time for LLMs on low-cost GPUs. On an NVIDIA TITAN Xp 12GB GPU, the Dobi-SVD compressed LLaMA- A-B model achieved generation speed of 25.97 tokens/second, delivering a 12.4x speedup compared to the original model. In particular, Dobi-SVD hardware-agnostic, offering superior generalization across different hardware targets compared to other compression methods; In terms of integration with other compression methods, Dobi-SVD can be effectively combined with other compression methods such as quantization achieve a lower compression ratio. When combined with GPTQ-4bit, Dobi-SVD compresses the LLaMA-7B model to just 3.4GB, while maintaining a perplexity of 9.97 on WikiText2, which shows that Dobi-SVD an effective solution for model compression that can be widely applied and combined. Moreover, we extend Dobi-SVD to a broader field, vision- languaee models (VLMs). Specifically, we apply Dobi-SVD to compress the popular VLM, LLAVA V1.5-7B. Experiments demonstrate that our method improves throughput by 1.2 times. Notably, we find that our Dobi-SVD not only enhances efficiency but also improves VLM performance the Pope-random dataset. We also deployed Dobi-SVin on the ofate-of-the-an Vision- Language-Acttn model, Open Under a compression ratio of 0.4, our method achieves 17.6% acceleration the NVIDIA A 100 while maintaining nearly
3

---

Published as a conference paper at ICLR 2025
Activation
Weight
Activation-aware
Activation
activation-awar aware
Directly Truncating Weight
Truncating Weight & Scaling Matrik ASVD, SVD LUM
Directly Truncating Activation
Dobi-sV
IPCA
Easier reconstruct weight
performance
Weight
Figure 2:
The differences between Dobi-S Dobis method and previous approaches in handling activa-
tions and obtaining new weights. See A.4 for a detailed explanation of this figure.
lossless performance. In summary, Dobi-SVD is the first approach to make SVD-based compression methods truly competitive, highlighting the significant potential of SVD for model compression. In addition to our proposed method, we also provide some new perspectives in the Appendix; we hope inspire the community further advance this direction.
PRELIMINARIES
2.1
MATHEMATICAL EXPRESSION OF SVD FOR LLM
Given a weight matrix W E R=*x, it can be decomposed through SVD W = UEVT, where U E Rmxm and V E Rex* are the right and left singular vector matrices, respectively. E = the singular values of W.
diag( .m) is
diagonal matrix and 01 .......00 F.
The SVD compression process of W can be summarized in three steps. Decomposition: use SVD to decompose W. Truncation: retain the top singular values h and obtain the truncated singular- value matrix D = d
ak, O, ........0 0). Reconstruction: reconstruct W into two matrices k
W1 = U W2 = v V V. where W1 E R= and W2 R Through the process, W can be compressed into W1 and W2. We define the compression ratio k(m n)/(m n).
2.2 BASIC PROPOSITIONS.
Before the derivation of W, we introduce three basic propositions:
Proposition 1: Rank Property of Matrix Multiplication. For the matrix multiplication AB = C, the rank of C satisfies rank(C) < min(rank(( (A), rank( (B)).
Proposition2: Eckart- Young-Mirsly Theorem. For a matrix A E R the best rank-k approxima- tion of A in terms of the Frobenius norm or spectral norm is given by the matrix A₃ UV
Proposition3: Positive Definiteness of Matrices in SVD. Given the SVD of matrix A UAEAVI. the matrices UA and VA are both orthogonal, satisfying V VA UAUA = I.
2.3 MOTIVATION: TRUNCATE WEIGHTS OR ACTIVATIONS? How TO DO IT OPTIMALLY?
Previous studies explored two main SVD-based compres- sion methods:
Table 1: PPL of Llama-7b on Wiki- text2 after directly truncating activa- tions and weights under same trunca- tion setting.
truncating weights activations, including activation-aware approaches. The first and straightforward method, truncating weights, applies SVD to compress W
(2.1) Alternatiely,, recent works such as ASVD (Yuan et al.
2023) and SVD-L.LM (Wang et al. 2024), drawing from tech- niques in the quantization domain, emphasize an activation- aware compression, which scaling matrix S scales W capture the varying importance of input channels and aims to minimize differences in activations, i.c., min || - xx ( 1WS-S1l11 where WS is obtained by apply- ing SVD to WSS S. This raises a question: Are the existing truncation methods truly optimal? Our answer is no. Both theoretical and experimental results reveal that fundamentally different third mindiigmddrretty truncating activations via SVD on A (i.e., xW) the optimal approach.
Param Ratio 1.0
0.8
0.6
Activation
5.68
6.36
8.85
20.71
Weight
5.68
20061
52489
105474
At the module level, Proposition 2 shows that directly truncating activations gives As. the optimal k-rank approximation of A. At the model level, we prove (0. that directly truncating activations better minimizes training loss than truncating weights. Previous methods avoided this due to challenges in updating weights (Fiuure 2), while we provide a practical and effective solution through theoretical analysis (Sect and A.4.I). Results: Table I showw truncating activations outperforms weights, and Table confirms it surpasses existing activation-aware methods.
3 DOBI-SVD METHOD
In this section, we address the three main challenges of SVD for LLMs to implement a high- performance SVD algorithm.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Differentiable Algorithm for Finding Optimal A
Input: Training data {21, .2......
................................ Fn target compression ratio Riar. Output: Optimal & values for each weight matrix W.
1: Stepl: Smooth activation truncation Inference.
2: for each activation A do
VT ΣA = 02, 02,..., 3: Apply SVD: = UAVAVV A
4: Get smooth truncation: T(,;) = o; [0.5 tanh (3 (k- (k- 0.5), En = (T( T(,,))
5: Reconstruct activations: A VA
6: end for
7: Step2: Multi-objective loss training.
8: Freeze other parameters in the network and keep only & trainable.
9: for each training step do Compute current compression ratio Rnow based on current k.
11: Compute multi-objective loss: L = Lusk + 7 Rnow Rtarl.
Update k values by minimizing L.
13: end for
3.1 Q1: How TO GET THE OPTIMAT TRUNCATION POSITION?
Solution Space of Truncation Position To address the question, we first analyze the size of the solution space. Consider an LLM of L layers, where each layer contains M weight matrices W E Rmxx", m 2 n. For each matrix, the number possible truncation positions is n, E 21,2..... Therefore, for the entire LLM, the size of the solution space is (L(MM) Given that LLMs typically involve sagge-dmeensionl matrices, the solution space for truncation positions is vast.
Differentiable Optimization of the Truncation Position. To effectively identify the optimal truncation position within huge solution space, we introduce, for the first time, a differentiable method to solve for the optimal truncation position. The workflow of our method, as shown in Algorithm I. can be divided into steps: smoothing discrete truncation positions and multi- objective loss training.
During inference, we construct activation truncation model. To make the truncation continuous, use the tanh function to smooth the truncation. Specifically, we define a smooth truncation function T that T(o,) = oj [0.5 tanh (3 (k- - i)) + 0.5), where is a learnable truncation parameter, is parameter that adjusts the smoothness of the truncation and (o;; ) is the truncated value of o, T(, effectively simulates the truncation, making the truncated position differentiable.
During the training, we freeze other parameters in the model and only keep A: trainable. In order to ensure that the optimization balances the task performance and the model compression, we use multi-objective loss to train the model. Given the target compression ratio Rtar, our loss function is Taask + 7 Rnow - Rearl. where Lanc is the loss of the model on the downstream task, Rnow is the model compression ratio calculated by the current k value.
Note that our algorithm requires very low computational cost since the optimization only conducted on k of different layers. For example, in the Llama-7b model, there are only 224 trainable parameters, and training requires just 8 GPU hours.
Stabilize SVD Backpropagation. Although differentiable algorithm is theoretically feasible, we emphasize that the gradient is the devil (Wikipedial (2024). especially in backpropagation involving low-rank matrices, where gradients are prone to exploding due to numerical instability. Consider matrix A E T m. mm obtained via SVD: A UA During backpropagation, the gradients the loss with respect to UA. EA. and VA are denoted as gu. gx, and gv, respectively. The gradient of A then given by:
(UT (v)) E skew(V? gv))
00 of if (1)
gA=U =
+ diag(os))
VT Eij
ifi ifi= where skew(x) = (a x T)/ extracts the skew-symmetric part of a square matrix, and Eg is E's (th element, and 0; and oy being the i-th and j-th singular values of A. As shown in Eq[T] when o; and o, are very small or close, E, approaches 0, causing E to vanish and 9a to diverge to infinity. This gradient explosion problem frequently arises during optimization, especially
E
5

---

Published as a conference paper at ICLR 2025
Algorithm 2 Computing the Theoretical Optimal Rank-k Weight Matrix W via IPCA
Input: Training data {I1, #2,,.. In J. original weight matrix W, required rank k.
Output: Theoretical optimal rank-k k weight matrix W.
1: Input the training data into the network.
and their right-singular vectors [Va1, VA2, VAnl- 3: Generate Gg. a diagonal matrix with the first elements and the rest
2: Obtain activations [A1, A2, ....... An]
4: Initialize mean μ 0, Vol V1-
5: for = 2 to n do
6: Center the data: Vi V; (a+ +(V 14)).
7: Update Veew: Vnew = Voia, V.l.
8: Update principal direction VV: SVD(V now) U", VT
9: Update Vou: Vold = V'[: [ k]
end for
11: Use Vold as the final V. update weight W = WVG.VT.
since activation matrices LLMs often exhibit approximately low-rank structures. To address the above problem, we propose approximate solution for Eij in two cases:
1. When o; R 0 and oj Ri O, we directly set / Fij 7 where a small constant that ensures the contribution of og and or to 9A remains small.
2. When 0g &6 0 and oj ø O, but 0; og. we perform a Taylor expansion of 1 1/Ea as follows:
K
* 1 - (a,/o,, o,(o, (1
(2)
Egg o,
To accelerate the computation, utilize the summation formula of geometric series.
Answer 1: By stabilizing SVD backpropagation for differentiable optimization of truncation position, we can efficiently find the optimal position of each layer. To the best of our knowledge, this is the first work to enable end-to-end optimization for SVD-based LLM compression.
3.2
Q2: How TO UPDATE WEIGHTS OPTIMALLY?
In Sect. 3.1 we obtained effective activation truncation position. In this section, we address the second challenge: How optimally update weights based on the truncation position?
Leveraging Proposition 3, applying SVD to directly truncate activations at k can be formulated as: A = UAEAAA = ~UAEAIG&VA V = - where Gg is diagonal matrix with the first & elements as and the rest as Substituting A =W,
AVAGGVT
(3)
A = AVAG&V GWV.G
(4)
At truncation position k; for a set of inputs {x, {r:) the ideal updated weight W should satisfy:
min G.V.V x: min Σ xu G.V Wil,, rank(W) = k (5)
Thus, goal is find the rank-k matrix W closest to the of projected weight matrices WP (WVA, G&V V. Since usually not full-rank square matrix, computing W via its
inverse infeasible. Therefore, derive W from the underlying structure of WP.
While PCA can compute W, the large size of WP leads excessive memory demands. Traditional PCA scales exponentially with matrix size, making large n impractical and requiring hundreds of Gigabytes. To address this, we make a novel use of the Incremental Principal Components Analysis (IPCA) algorithm for memory-efficien PCA. The algorithm, detailed in Algorithm 2 centralizes each new input W," from WP and updates the principal components via incremental SVD. By processing the matrices sequentially, the need to input all matrices at once is replaced with multiple steps, each handling only two matrices at a time, significantly reducing memory usage.
Answer 2: By using IPCA, we can calculate the theoretically optimal Rank-k Weight Matrix W.

---

Published as a conference paper at ICLR 2025
Table 2:
Dobi-SVD vs. SOTA methods in terms of compression performance of LLaMA-7b on three language modeling datasets (in-domain evaluation) and seven common sense reasoning datasets (zero-shot evaluation). The best performance is marked in bold. Drop means relative performance drop to baseline. Dobi-SVD* refers to the result obtained without remapping. The performance of the VS and SVD-LLM is derived from the results reported in SVD-LLM. i uses LoRA fine-tuning.
PPL. ())
Accuracy WinoG.
Avg. (1)
Drop (U)
Ratio
Method
Wiki2
PTB
C4
Openn.
ARC_e
HellaS.
PIQA
MatQQA
1.0
Baseline
5.68
8.35
7.34
0.28
0.67
0.38
0.67
0.56
0.78
0.27
0.52
0%
ASVD
11.14
16.55
15.93
0.25
0.53
0.27
0.64
0.41
0.68
0.24
0.43
17.3%
SVD-LLM
7.94
16.22
15.84
0.22
0.58
0.29
0.63
0.43
0.69
0.24
0.44
15.4%
0.8
SVD
8.54
14.83
10.01
0.26
0.59
0.31
0.66
0.44
0.70
0.23
0.46
11.5%
Dobi-SVD
15.39
7,83
0.27
0.65
0.68
0.54
0.77
0.27
3.84%
ASVD
1407
3292
1109
0.13
0.28
0.22
0.48
0.26
0.55
0.19
0.30
42.3%
13.11
63.75
49.83
0.19
0.42
0.25
0.58
0.33
0.60
0.21
0.37
28.8%
0.6
13.54
46.38
23.54
0.22
0.41
0.58
0.34
0.23
26.9%
Dobi-SVD
8.12
43.85
12.63
0.28
0.65
0.32
0.62
0.45
0.72
0.25
0.47
9.61%
ASVD
57057
45218
43036
0.12
0.26
0.21
0.49
0.26
0.53
0.18
0.29
44.2%
SVD-LLM
53,74
438.58
345.49
0.14
0.28
0.22
0.50
0.27
0.55
0.21
0.31
40.3%
0.4
DoD
46.18
218.91
190.62
0.15
0.31
0.20
0.52
0.28
0.54
0.22
0.32
38.4%
Dobi-SVD
67.62
17.94
0.23
0.52
0.56
0.23
23.1%
3.3
HOW TO OVERCOME THE LONG- -VERLOOKEEO TRUNCATION LIMITATION?
In this section, we address a long- overlookdd limitation of SVD. As outlined in Section 2 for an m X n matrix W with truncation position h; the compression ratio is r k:mm + =n) n). Setting r r= gives k = (m -n) n2)(mm- (/mm+)) < min (m, n)) = rank(W), meaning the model size stays the same, but information lost. Notably, when m = ni, k: = rank(W) W)/2, truncating half the singular values. This demonstrates that in traditional SVD-based compression, the compression ratio is injective with respect to truncation position, often leading to significant performance degradation.
To address this limitation, we propose remapping the relationship between compression ra- tios and truncation positions to establish a bijection, where for r E [0, 1], ranges from 0 to ran with one-to-one correspondence. This gives the compression ratio k/ min(m, n) = max(m, n)/ n). with the compressed matrix size being k- max(m, n).
k/rank(W))
To achieve this bijection, we propose new SVD compression method: Step 1. For the updated matrix W with rank k perform SVD obtain UW: Ew. VT Extract the first l columns of Uw Zw W
as m x k matrix w W and the first k rows of V.r. W
as a k X matrix V.T W.k' Step 2. Assuming
m n, the storage space becomes X k. Take the first rows of U ww. and VT halve their bit precision, concatenate them, and replace the first n rows of W, , W. Finally, store only the modified W -
W,A (both n k),
using m x k space.
The compression method in Step 2 leverages the orthogonality of the left and right singular vector matrices U and V. whose column vectors follow normal distribution (as shown in the 'remapping' section of Fig. [ making them ideal for uniform quantization methods like QLoRA. Appendix A.7 shows that quantization introduces minimal error.
Answer3: By applying our quantized storage method, the compression ratios and truncation position form a bijective mapping, overcoming the truncation position limitation.
3.4 CONCLUSION: TWO NEW PERSPECTIVES OF DOBI-SVD
By leveraging the fundamental theorem of SVD, we derive the optimal approach for SVD-based compression: directly truncating activations (Section 2.3). To determine the optimal truncation point of the activation matrix, we develop robust and efficient differentiable SVD algorithm applicable to general matrices (Sections 3.1 and A.6). Subsequently, we propose a theoretical framework to derive new weight matrix from the truncated activations [ and Section 3.2) successfully implementing this using Incremental PCA (IPCA). Collectively, these contributions form Dobi-SVD's New Perspective 1: A Novel Path from Activation to Weight (A.4).
Furthermore, we highlight a long-overlooked limitation of traditional SVD-based compression methods and, for the first time, propose an effective solution. This leads to Dobi-SVD's New Perspective 2: Fully UnlockingSVDD Potential for Data Compression by Addressing A Long- Overlooked L Limitation (Sections 3.3 and A.5)
7

---

Published as a conference paper at ICLR 2025
Table 3: Dobi-SVD vs. popular pruning methods in terms of compression performance of LLaMA-76 on seven common sense reasoning datasets. The best performance is marked in bold. The performance of the pruning methods are derived from their original paper.
Cost
Accuracy (t)
Avg. (*)
Drup
Ratio
Method
Post-train
Fine uning
BoolQ
PIQA
WinoG.
ARC_e
ARC_e
OBQA
(1)
1.0
Baselinc
0.73
0.78
0.67
0.67
0.41
0.42
0.61
0%
LLM--runer
x
0.59
0.75
0.61
0.39
0.37
0.39
0.55
9.83%
0.8
LLM LLRRA)
0.69
0.76
0.65
0.63
0.37
0.40
0.58
4.92%
FLAP
x
0.69
0.76
0.68
0.69
0.39
0.39
0.60
1.64%
Dubi-SVD
x
0.73
0.77
0.68
0.65
0.37
0.42
0.61
0%
4 EXPERIMENTS
Without losing generalizability, our experiments in the main text are conducted on LLaMA-7B. We investigate four settings: (1) Evaluation on three language modeling datasets and seven commonsense reasoning datasets, comparing against state-of-the-arr SVD compression and popular pruning methods (Sect. 4.1). (2) Analysis of the importance and roles of each component in Dobi-SVD (Sect. 4.2). (3)
Testing the acceleration Dobi-SVD on different hardware and combining with quantization (Sect. 4.3) (4) Test the performance of Dobi-SVD VLMs and asionnaction model (Sect. ??). Details of experimental seftings are provided in A.33
Additional experiments in the Appendix include evaluations more tasks (c.g.. MMLU, popular pruning methods), more models (LLaMA2-7B: Tabs. 12 14, and LLaMA3. 1-BBB Tabs 11: Llama-
13B: Tabs. I6 [8] Llama2-13B: Tabs. 17 19 We also explore combining with quantization (Tab. 20) and direct comparisons with quantization (Tab. 21). Finally, we test the performance of
Dobi-SVD VLMs (Tab. 22) and vision-action model (Tab. 24). And compare the performance of large models compressed by Dobi-SVD with small models that are not compressed. (Tabs. 25| 26).
4.1 MAIN RESULTS
In-domain Evaluation. We evaluate the performance of Dobi-SVD top of LLaMA-7B model, with compression ratio ranging from 40% to 80%. The experimental results are presented in Table 2 Notably, Dobi-SVD demonstrates significantly better performance than other SVD methods. At a 0.4 compression ratio, the Dobi-SVD achieves a PPL of 9.70 on Wikitext-2, compared to 43, 104 and 458 for ASVD and SVD-LLM, respectively. This indicates that even with only 40% of the parameters retained, Dobi-SVD maintains an acceptable performance degradation, unattainable by other SVD) methods. We emphasize that even without the proposed remapping strategy, Dobi-SVD outperforms prior-arts by a large margin, especially under the low parameter- ratio. This demonstrates the effectiveness of the proposed differentiable optimization of truncation position. Additionally, our quantized storage remapping strategy further improves the performance, showing the significance of our method for improving the injective nature in SVD.
level of performance
Zero-shot Evaluation. To demonstrate the task generalization capability of Dobi-SVD. take the model trained on Wikitext-2 dataset and conduct the validation on seven out- of---ann datasets. As shown in Table 2 Dobi-SVD consistently outperforms the previous state-of-the-art methods across different datasets and compression ratios. Specifically, at 80% compression ratio, Dobi-SVD shows an average performance drop by only 3. 3.14% At low compression ratios, Dobi-SVD can still maintain good performance. For example, at compression ratio of 0.4, Dobi-SVD still has an average accuracy of 40%, while ASVD and SVD-LLM only have 29% and 31% % respectively. Note that even without remapping, the performance of Dobi-SVD* * is still better than other SVD compression algorithms. This shows that our differentiable optimization for truncation positions and weight update method are more effective than other SVD-based methods.
Performance comparison with pruning methods. To further demonstrate Dobi-SVD's superior trade-off between compression ratio and memory usage, we also compare it with state-of-the-ar pruning methods, as shown in Table Note that pruning method is orthogonal to the SVD-based method and these two kinds of methods are in different tracks. The results show that Dobi-SVD achieves comparable performance relative to the pruning- ruaeeg--based method across various task sets. Compared with FLAP (An et al.) 2024a), it on all four tasks. We highlight this is the first time that SVD compression method achieves better performance than pruning. Notably, while both Dobi-SVD and pruning methods require post-training.
of-te----theaaate art pruning method, Dobi-SVD outperforms
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of perfor- mance before & after remapping.
Table 5: Performance and memory usage of Dobi-SVD
Table 6: Speed of running unmapped Llama- 7b on TI- TAN Xp 12GB GPU. PPL is tested on the wikitext2.
Rati
Model
Wiki
C4
PTB
combined with 4-bit quantiza-
Renapt 16b6)
6,05
7:79
tion on wikitext2.
800
RemapiSe 16bit)
Win Remoop
6.06
887
10.91
7,85
22.39
25.03
Ratio
Methad
PPL
Monney
Ratio
Mem (GB)
Speed (tokensss)
Speed
Up
Reneut 16he)
K.67
12.54
43.68
Doit-SVD
6,8018
6066
RemopiiN labbir)
R12
1263
4088
Dubi-SVD=GGPT
12.04
18GB
1.0
12.6
2,09
100x
Wio Reniap
1ébic)
14.96
9.78
24.60
17.39
49.01
67.81
0.6
Tuuii-VVD-GPT
Dobt-VVD
8.12
9.99
"LAGG"
75GB
0.8
10.1
23.32
11.2x
0.56
Remupii8
9.95
17.94
67.62
Dubi-VVD
10
0.6
24.80
Wio Kemap
58.02
145.41
270.16
DotisSVD-GPPQ
7.01
AJGR
0.4
6.8
25,97
12.4x
7.50
7.40
10.0
256 Truin Sumple Sample
Mumury Noscld (CB)
25
JPCA
PCA
7.46
Perpleatty
2.40
10-
5.0
Pravonte wmights
Plasasted iusossion (sudl (sudiclaye)) layer?
5
7.40
Plasonte at actiection ningle hager
... Original PPL
7.30
15 20
30
0.5 Param Rabier)
50000
10000
15000
20000 25000
Motrtx Size
Figure 3: (Left) Performance Comparison of different training methods on LLaMA-7b, For activation truncation (multi-layer) we only truncate layers 29-31, and for activation truncation (single-layer) we only truncate the 29-th layer. (Middle) Comparison of model performance using batch size 256 and 16 for training. (Right) Comparison of memory requirements for PCA and IPCA for * matrix.
Dobi-SVD only trains the truncation positions of the matrices. Dobi-SVD significantly reduces the number of trained parameters and computational cost compared to pruning. Specifically, LLMM-Prnne requires training additional 1.2 billion parameters to maintain the model performance on LLaMA-7B, whereas Dobi-SVD only trains 224 parameters.
4.2 ANALYSIS EXPERIMENT
In this section, we perform analysis experiments on three critical components of the Dobi-SVI method: differentiable optimization of truncation position, efficient weight-updates with Incremental PCA (IPCA) and quantized storage remapping.
4.2.1 ANALYSIS ON DIFFERENTIABL. OPTIMIZATION OF TRUNCATION POSITION
Guided Truncation. Based on the above analysis, we hypothesize that, under the same compression ratio, truncating the Attention layers in later stages of LLM may result in smaller performance losses. To validate this, we conducted two experiments on the LLaMA-7B model: truncating a single layer (layer 29) and truncating multiple layers (layers 29-31) with differentiable training. The results are shown in Fig. 3 (a). We observe that both truncation settings lead to better performance compared to the original model, while barely weight truncation results in the performance degeneration. Besides, truncating single layer can even lead to better performance compared to truncation multiple layers. Both experimental observations indicate that the proposed differentiable optimization of truncation position offers promising potentials to improve model performance through activation truncation and provides guidance for selecting layers to conduct low-rank decomposition.
Efficient Training. Besides, we validate the sample-efficient training to highlight the advantage of the differentiable optimization. A common post-training batch size 256 and here we test the model's 's performance with training batch size as 16 while keeping the epoch same. The performance is shown in Fig. 3 (b). Even with such a few batch size, Dobi-SVD achieves results comparable to those obtained with 256 samples. This demonstrates the efficiency of our differentiable optimization. Note that training with batch size as 16 on LLaMA-7B require only a few GPU-hours.
4.2.2 ANALYSIS ON EFFICIENT WEIGHT UPDATE.
We compare the memory footprint for our proposed efficient weight update strategy (w/ IPCA) and original PCA method, as shown in Fig. 3 We observe that as the dimensionality the decomposed matrix increases, the memory footprint of PCA grows exponentially, In contrast, our weight-update strategy with IPCA significantly reduces memory usage, which remains close to a constant as the dimensionality increases. This is because PCA must be used for decomposing the whole matrix
V [V, V2, Va.... ... Vnl. while IPCA can be used for conducting the decomposition at each step V Vota. Wt: Val thus gets rid of the requirement of storing the entire matrix.

---

Published as a conference paper at ICLR 2025
700
40%
40%
700
60%
60%
80%
Tokers Teg
500
80%
100%
100%
400
500
300
400
200
512
Batch Size
Sequence Lecath
Figure 4: Tokens/sec of original LLaMA-7B and its compressed version by Dobi-SVD under 40%, 60% and 80% compression ratio on single A100 GPU. (a): comparison with different batch size while sequence length 32. (b): comparison with different sequence length while batch size = 64.
4.2.3 ANALYSIS ON QUANTIZED STORAGE FOR REMAPPING.
To demonstrate the effectiveness of our proposed quantizes storage remapping, we compare the performance under three different approaches: remapping without quantization (i.e., Remap (16bit) in Table 4). remapping with quantization maintain the compression ratio (i.e., Remap (8+ 16bit) in 4). and no remapping while maintaining the compression ratio (i.e., W/o Remap in Table 4), as shown in Table 4 By comparing the first two approaches, in minimal performance drop regardless. The performance comparison between remapping with quantization and remapping reveals that remapping significantly improves model performance, especially at lower compression ratios. Specifically, at a compression ratio 0.4, the remapped model achieves perplexity of 9.95 on the WikiTexx2 dataset, while the non-remapped model reaches 58.02. This highlights the effectiveness of our proposed remapping in enhancing the model performance.
see that quantization results
4.3 INFERENCE- EFFICIENC EVALUATION
Performance nigh-performance GPUs. Dobi-SVD not only compresses LLMs but also improves the efficiency of inference on real hardware. We tested the original L.LMA-7BB and LLAMA-7B with our Dobi-SVD on NVIDIA A100 GPU by measuring the number of tokens generated per second under varying batch sizes and sequence lengths. The results are shown in Fig. Across all compression ratios, Dobi-SVD consistently improved generation speed. As the batch size increases and the sequence length decreases, this speed improvement becomes more pronounced. With compression ratio 0.4, we achieve an improvement of inference speed by up to 1.75 x. At the 0.6 compression ratio, the speedup is to 1.4 x. These results highlight the efficiency improvement of Dobi-SVD and suggest its practical potential in LLM services on cloud.
Performance on wow-pefoormanc GPUs. To demonstrate the applicability of Dobi-SVD in resource- constrained environments, we deploy both the original and compressed LLaMA-7B models on an NVIDIA TITAN Xp with 12GB of memory. The results shown in Table 6. Since the LLaMA-7B model requires approximately 14.8GB of memory, the original model requires data transfer between the CPU and GPU during inference. However, applying Dobi-SVD is able to make the entire model run on the GPU. Dobi-SVD achieves the speedup of 12.4 x. This highlights the huge potential of Dobi-SVD for practical applications in resource-limited devices, like edge-devices.
Performance combined with quantization. To verify the compatibility of Dobi-SVD with quantiza- tion, we use GPTQ-4bit combined with Dobi-SVD to compress LLaMA-7B. We measure the memory footprint and PPL on wikitext2 before and after quantization. Table 5) shows that the combining GPTQ-4bit with Dobi-SVD further improve the memory utility. is worth noting that our results significantly outperform previous methods combining SVD with quantization. For instance, when compressing the model to a compression ratio of 0.6 and integrating with GPTQ-4bit, Dobi-SVD achieves a PPL of 9.97. This illustrates the wide applicability of Dobi-SVD, which can be flexibly combined with other quantization methods.
CONCLUSION
We introduce Dobi-SVD, an efficient SVD-based method for LLM compression. We address three key challenges: (a) how to determine the truncation position, (b) how to update weights efficiently and (c) how to overcome truncation limitation that results information loss. We first theoretically and empirically explain why truncating activations is better than truncating weights. In addition, we propose solutions address these three challenges using differentiable optimization strategy find the truncation position, an efficient weight update method via the Eckart- Young-Mirsky Theorem, and a quantized memory remapping to maximize SVD's 's potential. Experiments demonstrate that Dobi-SVD achieves minimal performance loss at low compression ratios. Dobi-SVD compresses LLaMA-7B to a 0.4 compression ratio with a PPL of 9.95 on WikiText2, outperforming advanced SyD-based and pruning methods. It also provides 12.4x speedup NVIDIA Titan Xp 12GB GPU with negligible loss. Overall, Dobi-SVD is a hardware-agnoste highly adaptable compression method that demonstrates the significant potential and competitiveness of SVD in model compression.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. Online embedding com- pression for text classification using low rank matrix factorization. Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 6196-6203, 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wtxiv:2303.08772 2023.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorsk: Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint 19iv:11905.13311 2019.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation- based adaptive struc- tured pruning for large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 10865-10873, 2024a.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation- adaptive struc- tured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024b.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint 24XX::2441.150222 2024.
Carsten Bergmann, Lisa Guay-Woodfioe Peter Harris, Shigeo Horie, Dorien JM Peters, and Vicente E Torres. Polycystic kidney disease. Nature reviews Disease primers, 4(1):50, 2018.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference artificial intelligence. volume 34, pp. 7432-7439, 2020.
Ori Bryt and Michael Elad. Compression of facial images using the k-svd algorithm. Journal of Visual Communication and Image Representation, 19(4):27-2222, 2008
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii, Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-555-1111 2006.
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: Task-adaptive bert compression with differentiable neural architecture search. arXiv preprint #tXiv:2001.04246 2020.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. Advances in Neural Information Processing Systems, 31, 2018.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks, arXiv preprint arXiv:1110.09222. 2017.
Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 13-5113 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.00457. 2018
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
11

---

Published as a conference paper at ICLR 2025
Lucio Dery, Steven Kolawole, lean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar, Everybody prune now: Structured pruning of Ilms with only forward passes. arXiv preprint arXivv2402.054406 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 00323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022a.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 1XXi:22200.13233 2022b.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept. 10:8 9, 2021.
Qiang Guo, Caiming Zhang, Yunfeng Zhang, and Hui Liu. An efficient svd-based method for image denoising. IEEE transactions on Circuits and Systems for Video Technology, 26(5):868-880, 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 1782-9793. 2020.
Yen-Chang Hsu, Ting Hua, Sungen Chang. Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.0011 2022.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint 1eXiv:1405.344 2014.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W
Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions automatic control, 25(2):164-176, 1980.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba. Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint 4rXiv:1412.6553, 2014.
Guillaume Lebrun, Jason Gao, and Mike Faulkner. Mimo transmission over a time-varying channel using svd. IEEE Transactions wireless Communications, 4(2):757-764, 2005.
Jung Hyun Lee, Jeonghoon Kim, June Yong Yang, Jung Kwon, Eunho Yang, Kang Min Yoo, and Dongsoo Lec. Lrq: Optimizing post-training quantization for large language models by learning low-rank weight-scaling matrices. arXiv preprint arXiv:2407. 1534, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. 1im-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.
12

---

Published as a conference paper at ICLR 2025
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 1000vv1:00.000 809.02789, 2018.
Marcin Moczulski, Misha Denil, Jeremy Appleyard, and Nando de Freitas. Acde: A structured efficient linear layer. arXiv preprint 1.X.v:1511.059444 2015.
HS Prasantha, HL Shashidhara, and KN Balasubramanya Murthy. Image compression using svd. In International conference on computational intelligence and multimedia applications (ICCIMA 2007), volume pp. 143-145. IEEE, 2007.
Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36, 2023.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J Goldsmith, and Mert Pilanci. Compress- ing large language models using low rank and low precision decomposition. arXiv preprint arXiv:2405, 18886, 2024.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimmnssioal output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655-6659. IEEE, 2013.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021a.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021b.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312. 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication-effiienn models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 202
Wei Wang. Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Robust differentiable svd. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021b.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation aware singular value decomposition for large language model compression. arXiv preprint arxiv2203300737 2024.
Juyang Weng, Yilu Zhang, and Wey-Shiuan Hwang. Candid covariance-free incremental principal component analysis. IEEE Transactions Pattern Analysis and Machine Intelligence, 25(8): 1034-1040, 2003.
Wikipedia. The devil is in the details Wikipedia, the free encyclopedia. wttp::/ppppaaa..ooo//oaaaaaaaaaa httpae/eeeccait.ieeeoeeeeeeeeeer 00the//0decaallsoooo111223333333 2024. [Online; accessed 2400eeeee222
http:
13

---

Published as a conference paper at ICLR 2025
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre- pre-training via structured pruning. arXiv preprint arXiv:2310.0669, 2023.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. Lqer: Low-rank quantization error reconstruction for llms. arXiv preprint arXiv:2402.02446, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 20002000, 2022.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 18(10)119331195, 2015.
Ming Zhao and Xiaodong Jia. novel strategy for signal denoising using reweighted svd and its applications weak fault feature enhancement of rotating machinery. Mechanical Systems and Signal Processing, 94:129-147, 2017.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
Organization this appendix, we provide in-depth descriptions the materials that are not covered in the main paper, and report additional experimental results. The document organized as follows:
A.1
Related Work
Limitations and potential solutions
A.3
Experimental setting details
A.4
Dobi-SVD's New Perspective 1: Novel Path from Activation to Weight
A.4.1
Theoretical Support for Updating Weights Using IPCA
A.5
Dobi-SVD's New Perspective 2: Fully Unlocking SVD's Potential for Data Compres- by Addressing A Long-Over oveg-overlooked Limitation
sion
A.6
Dobi-SVD's Robust and Efficient Differentiable SVD Algorithm for General Matrices
A.7
Additional analytical results
A.7.1
Quantization Precision Loss
A.7.2
Effeetiveness of differentiable training
A.7.3
Differentiable & changes at various compression ratios
A.7.4
Truncation sensitivity analysis
A.8
Additional experimental results
A.8.1
Experimental results on more models
A.8.2
Combined with quantization
A.8.3
Experimental results VLM
A.8.4
Experimental results on Open
A.8.5
Comparison with smaller uncompressed models
A.9
Example Demonstration of Real Sentence Generation
A.10
Analysis of directly truncating activations weights
A.1
RELATED WORK
LLM Model Compression. Large language models (LLMs) typically contain billions of parameters, making inference on resource-oonsrrined hardware challenging. address this, researchers have developed various methods to compress models without requiring retraining. These methods can be categorized into three main categories: pruning, quantization, and low-rank decomposition. Specifically, pruning sets individual weights structured components to zero without changing the overall structure of the LLM. For example, SparseGPT (Frantar Alistarh 2023) prunes the least important weight elements by inverting the Hessian matrix. However, the irregular sparsity from unstructured pruming often fails to achieve significant speedup, only performing optimally on specific hardware architectures. L.MM-Pruner (Ma al.) 2023), on the other hand, leverages small dataset to estimate the coupled importance of weights, parameters, and groups, then applies LoRA Abased pruning recover accuracy. Yet, this approach significantly degrades model accuracy, especially at low compression ratios, due to the extensive modification of the weight matrices. Sheared Llama (Xia et 2023) performs extensive training on 50 billion tokens after compression. Quantization, another approach, compresses the model by reducing the precision of the LLM's weight matrices. For instance, GPTQ (Frantar et al. 2022b) employs layer-wise quantization, updating weights using inverted Hessian information. The drawback of quantization is that it offers limited compression options, usually between 3 and bits, which may not fully optimize memory utilization.
SVD-based Model Compression. SVD-based L.MSs compression has been widely explored (Jader) berg et al. 2014 Zhang et al.) 2015 Denton et al. 2014). Earlier studies primarily used SVD to compress embedding layers (Chen et al. 2018:10 Acharya et al. 2201)). As model sizes have grown, research has shifted towards applying SVD to weight matrices (Hsu et al. 2022 Wang et al. 202 la). Recent findings (Sharma et al. 2023) suggest that LLM weights are often approximated as low-rank matrices, highlighting SVD's potential for LLM compression. Traditional SVD focuses on com- pressing the original weight matrix by minimizing WW W However, since it does not account
15

---

Published as a conference paper at ICLR 2025
for parameter importance, it often results in significant performance degradation. These methods typically require finettuning to recover performance, which demands substantial computational resources for LLMs. To mitigate this, recent works have focused activation- aware SVD, which aims to minimize |A - A For example, ASVD posits that the activation distribution influences compression enror and scales EW with diagonal matrix S, where S represents the input channel's impact on weights, SVD-LLM argues that not all larger singular values are necessarily more im- portant, introducing a truncation-awarr whitening strategy to determine which singular values are critical for activations. However, current activation-aware SVD methods are limited to modifying EW to adjust W, which restricts the values that W can take, failing to effectively retain activation information. For instance, ASVD is only effective at high compression ratios (0.8 and 0.9), suffering from significant performance loss at lower compression ratios. SVD-LLM, when compressed to a 0.4 ratio, causes PPL to drop from the original 7.94 to 42.3.
In addition, low-rank decomposition has also been applied in different forms for LLM compression methods. For instance, LQER (Zhang et al., 2024) uses SVD to address quantization errors in the quantization process, while LRQ (Lee et al. (2024) applies SVD to enhance the sample handling capacity during training. CALDERA (Saha et al. 2024). based on matrix compression method LPLR (Saha al. ) adopts non-SVD low-rank approaches for weight compression, but these typically lead to increased dimensions of the new weight matrices, resulting in performance degradation and necessitating the combination of quantization and LoRA fine-tuning.
A.2 LIMITATIONS AND POTENTIAL SOLUTIONS
We analyze three limitations of Dobi-SVD and suggest potential solutions. Firstly, our current SVD implementation is time-consuming and memory-intensive. This issue stems from Python's 's support for only fp32 SVD, which could be alleviated by implementing low-precision SVD. Secondly, during memory remapping. transitioning precision becomes challenging when further quantizing to 4-bit or below, leading to performance drop at 2-bit. To our knowledge, 2-bit quantization itself poses significant challenges in LLM compression. Third, quantization introduces additional dequantization time during inference, which could be reduced by quantizing larger matrices using more advanced quantization libraries. Our future work will focus on overcoming these limitations and exploring broader applications of Dobi-SVD, such in vision-languagg models and robotics.
A.3 EXPERIMENTAL SETTING DETAILS
In this section, we provide a detailed description of our experimental setup and hyperparameter configurations.
To demonstrate the task generalization of Dobi-SVD, we use Llama- 7b (Touvron et al.) 2023) and evaluate the performance on different tasks. We first test the model's in-domain performance
Models, Datasets and Metric:
the C4 (Sakaguchi et al. 2021a), Wikitext2 (Merity et al. (016), and PTB (Marcus et al. 1993), respectively. On these three datasets, we use perplexity (PPL) as the metric, the lower the better. In addition, we also evaluate it seven common sense reasoning datasets (OpenbookQA (Mihaylov et al.) 2018). WinoGrande (Sakaguchi et al. 2021b). HellaSwag (Zellers) al. 2019). PIQA(Bisk et al. 2020). MathQA (Amini et 2019)), ARC-e, and ARC-c (Clark et al. 2018) in zero-shot setting with the Eaa---aaaoon-Harreeee framework (Gao et al. 2021). On these datasets, we use accuracy as the metric, the higher the better.
We compare Dobi-SVD with state-of oh-the-rrt activation SVD methods, ASVD (Yuan et al., ) and SVD-LLM (Rajpurkar) 2016). We also compare Dobi-SVD with popular pruning methods, LLM-Pruner (Ma et al. h023), FLAP (An et al. 2024b), Wanda-sp (Sun et al. 2023), SliceGPT et al. 2024). Bonsai Dery et al. 2024) and Self-.
Baselines:
deploy it on hardware devices. We use two representative devices. One is 80GB NVIDIA A 100 which represents high- high-performance GPU, and the other is 12GB NVIDIA Titan Xp which represents low low-performance GPU.
Hardware: To demonstrate the hardware efficiency of Dobi-SVD,
Hyperparameters: The hyperparameters involved in our algorithm mainly include B, which controls the smoothness of the tanh function; y, which controls the minimum threshold of singular values during backpropagation; and K, the number of terms retained in the Taylor expansion. In our experiments, we set 3 10, 7 1 10-10, and K = 10.
16

---

Published as a conference paper at ICLR 2025
Training Procedure:
During the differentiable truncation training:
1. For LLM, we randomly select 256 samples from the WikiText2 dataset as the training set, with each sample containing 2048 tokens.
For VLM, we randomly select 256 samples from the TextQA dataset as the training set, with each sample containing 660 tokens.
Throughout the training, we freeze all parameters except for the truncation position & of each matrix, which remains trainable. We use the Arruneng provided by the Transformer library for training. The hyperparameter settings used during training are listed in Table 7 Additionally, for the tanh function used during the smoothing phase, we set / = 10. For robust SVD backpropagation, we set le-10, meaning that singular values smaller than le 10 are treated as le 10 during the
backward SVD process. The number of terms the Taylor expansion set to = 10.
Table 7: Hyperparameter settings during training.
Hyperparameters
Value
Seqence Length
2048
Number of Train Sample
256
Number of Val sample
16
Batch Size
32
Epoch Number
320
Scheduler
Cosine
Optimizer
Adam
Scheduler Ir
0.1
Warm up step
0
Weight Update: During the weight update process. we use the same 256 training data samples as inputs to collect truncated activations. These are then processed with IPCA (Algoß) independently and directly calculate the new weight matrix for each matrix position, requiring no extra data or training.
Memory Remapping: In the remapping quantization process, to align with the normal distribution characteristics of the weight matrix, we use bub library for model quantization (Algoß). Specifically, we utilize the bub-8bit to quantize the matrix, and then concatenate two quantized 8-bit matrices into a single 16-bit matrix.
A.4 DOBI-SVD'S S NEW PERSPECTIVE 1: A NOVEL PATH FROM ACTIVATION TO WEIGHT
Existing SVD-based compression methods primarily fall into two categories:
Directly truncating weights W: This straightforward approach ignores the interaction between weights and activations.
2. Activation-aware SVD: These methods (c.g., ASVD, SVD-LLM) incorporate scaling matrix to capture activation influence by performing SVD on WS and reconstructing W using S-! However, S-1 often fails due to theoretical and numerical issues.
Our method, Dobi-SVD, introduces novel truncation paradigm: performing SVD directly on activations A (A x W, where the input), which establishes novel path from activations to weights, leveraging the EYM theorem to achieve optimal results. Notably, similar strategies have not been explored in Mon-SVD-base compression methods: manipulating activations alone may improve inference speed and memory usage, but it does not compress the model itself.
Fig. illustrates this paradigm shift along a spectrum:
Left (W): Directly truncating weights.
Middle (WS): S): Activation-aware methods: retaining weights W with auxiliary matrices..
Right (A): Directly truncating activations.
17

---

Published as a conference paper at ICLR 2025
"Directly truncating activations" is radical approach that is farthest from the weight matrix W and closest to the activations A. This makes it the most challenging method for reconstructing weights while achieving the best performance, as demonstrated by theoretical analysis 2.3 A key reason previous works have avoided this approach is the inherent difficulty in effectively reconstructing the weight matrix.
To address this challenge, we introduce IPCA, a method for compressing high-dimensional orthogonal matrices. IPCA enables the reconstruction of new weights after direct activation truncation, serving as a "cosmic wormhole" that seamlessly bridges the activation space and the weight space. In our experiments, this innovation has proven enable superior compression results without relying on additional data or fine-tuning.
A.4.1 THEORETICAL SUPPORT FOR UPDATING WEIGHTS USING IPCA
In Sect 3.2 we mention that our goal is to find the rank-k matrix W closest to the set of projected weight matrices WP (WVA, G2 V Assuming W = WVVT, our goal can be converted to
min, >= WVVV WVVT [/ According to the properties of the Frobenius norm, we can get,
min
IWVV WVV min min
|| IWIII VV 1
(6)
Since
||W]| fixed, our goal can be written as:
min Σ IIV, - -V min IIVV 11 + ||V 2trace (VVV vV²)
(7)
Since V, and V are orthogonal matrices,
||VVV 11 |VV* ||3 = k. Equation 7 can be written as:
min Σ 2k (VVV 2nk min 2trace (VVV
(8)
Therefore, our goal is max, Σ 2trace (VVV Vil The final form of our goal is 2 max, V and its optimal solution V = maxy En V/ is the value obtained by solving (V1, Vi....., dress excessive memory demands, we use the Incremental Principal Components Analysis (IPCA) algorithm (Weng et al. 2003) for memory-efficient PCA.
Since (ra. =
.a.....,y.. Vn by PCA. To ad-
A.5 DOBI-SVD'S NEW PERSPECTIVE 2: COMPRESSION BY ADDRESSING A
FULLY UNLOCKING SVD's POTENTIAL FOR DATA LIMITATION
A Long- Ong-OOvellooked Limitation of Traditional SVD-Based Compression Methods. Performing SVD m matrix M, we decompose values, we obtain two matrices: (UE) k) of size m x k and (VT)): k of size k x n. The storage ratio between these matrices and M ks (m + n)/(m- n). To compress M, the storage ratio must be less than 1, which requires & E [0, (m - n2)(mm + n)). However, since M has min(m, n) singular values, compressing M necessitates discarding least min (m, n) (m-n)/ n)/mm fim++a n) singular values. For large models, such 4096 x 4096 matrix, this means losing 2048 singular values-hall of the total-resulting in significant and unnecessary information loss.
M = UEVT, When retaining the top k: singular
Prior to our work, no SVD-based methods addressed this issue, as they lacked solution. Non-SVD- based methods, such as Ashkboos et al. (0024). have used this limitation as a primary critique of SVD-based methods.
Dobi-SVD's New Perspective. Our approach resolves this problem by remapping the relationship be- tween information and storage: to ensure storage ratio below while allowing & E [0, min mi. (m,))). we deduced that the compressed storage of M should be k max(m,, n.).
To bridge the gap and remap storage from k- (m + n) to k max(m, n),, leverage the Gaussian distribution properties of the and V matrices and implement a novel quantization method to reduce storage requirements efficiently. The algorithm pseudocode is given in Algo
18

---

Published as a conference paper at ICLR 2025
Algorithm 3 Precid-Preeision Quantization Storage
Input: 16-bit updated weight matrix rank-k W E R 8-bit quantizer Q
Output: 16-bit mixed- preecision weight matrix W mixed E R (mmaa(mm,))x
1: Compute SVD: W = UEVT
2: Extract the top-k components: U2 = [t k] R Vv k| E R
3: if m 2 n. then
4: Quantize the first n rows of U.: U + Q(U,()), ie (0,
5: Quantize all rows of Va: Vulijs + ((Vbl)) i (0,
6: Construct W Wixzed:
For E [), W Wmized + concatenate ( Vx(i}s)
For E [n, m): W mixed U+
7:
else
8: Quantize all rows of Ug: U (Ú+()), ie [0, m)
9: Quantize the first m rows of V V [ (())..) (0,
10: Construct W mizod:
For E [0, m): W mixed [i]
concatenate U& (Ux))) V*((*)
For E [m, n): W Wiined [1] f Vv[i]
11: end if
12: return 16-bit matrix W mixzed E Rmax(mm,)x
A.6 DOBI-SVD'S ROBUST AND EFFICIENT DIFFERENTIABLE SVD ALGORITIIM FOR GENERAL MATRICES
Gradient Explosion in SVD Backpropagation. SVD backpropagation often faces gradient explosion when singular values in a matrix are nearly equal, a common issue in both aareeddimensional and low-rank matrices. Widely used techniques like gradient clipping and normalization fail address this. To the best of our knowledge, only one work (Wang et al. 2202 has successfully utilized Taylor expansion to resolve it, demonstrating the method's effectiveness and showing superior gradient behavior compared to other approaches in their paper.
While their focus was on symmetric matrices and computer vision tasks, we made novel use of Taylor expansion for general matrices and LLM compression. Unlike (Wang et al. 2021b), which dealt with low-dimensional image features, our work addresses the computational challenges posed by dinge-iiinsiinal matrices in LLMs. To meet the demands of these large matrices, we developed more general and efficient, parallelized algorithm. The algorithm pseudocode is given in Algo and 5
Algorithm 4 Custom Low-Rank SVD Forward Pass
Input: Input matrix X E Rmmxn target rank
Output: Matrices U E ma S R& V E Raxk
1: Compute low-rank SVD: U. S, V + svd_loorank(XX.. niter = 2)
2: Save U. S. V for backward pass
3: return S. V
19

---

Published as a conference paper at ICLR 2025
Algorithm 5 Custom Low-Rank SVD Backward Pass
Input: Gradients 8U, 8S, 8 VV; saved tensors U.S, V
Output: Gradient 6X
Transpose matrices: V VT 8V (8V)T
2: if á$ O and JU = 0 and DV 0 then
3: return 6X 0
4: end if
5: iU = and T then
6: return 8X = U diag(6S)
7: end if
8: Define numerical stability parameters: Evals Egrad Editf, Rraaylor
9: Clamp singular values: Selamp max(S, (S, Eval)
10: Compute singular value ratios:
11: X = Sclamp- Xj- clump
12: R = =>/>>;
Initialize matrix 1xxk
14: Create masks:
Identity mask: I diag(1x)
Non-diagonal mask: Mavl = -I
Lower triangular mask: Mlower tril(1xxk)
Initial mask: Mini Mool A Mlower
15: Handle too-small singular values:
Mamall Mjii A (R. A (2, = =val)
17: E [amall]] + Egrad
18: Handle normal cases:
19: Maormal = Miat A smalll
20: Compute differences: A A4 Xyl
21: For arithmetic sequence (equal singular values):
22: Mequal M A (A 0)
23: E|Maqual] Teyaylor
X?
24: For geometric sequence (close singular values):
25: Melose Mnormal A (0- < A < Edift)
26: q² R[MMel 1
2 Paryler
27: E Melose]] X?
92
28: For other cases:
29: Motber Mnormal (A > (dirr)
30: Mothe) <- (A: X1)(A; + dg)
31: Symmetrize E:
32: Mpai = Maal A Mi
33: E[Mpaa] f- EE [Mpaa] pad]
34: Define skew-symmetric function: skew(X)= =X XT
35: Compute skew matrices:
36: Pu skew( U 8U) E
37: Gy = skew (akew(VTo (VT vV) E
38: Compute core gradient:
39: 8x < U (Ωt diag(()) (S) diag(S)) + diag((S)) VT
40: Compute additional terms:
41: 8U scaatedd = OU 5U/S T
clamp
42: Term ( scated U(U 8 s VT
VVT sealed 8V /Setam
Termy
(TVT
(8VT
sealed V V)
scaled
45: Update gradient:
46: 5X <- 8X Term + Term
47: return 8X
20

---

Published as a conference paper at ICLR 2025
A.7 ADDITIONAL ANALYSIS RESULTS
In this section, we present additional experiments to further demonstrate the effectiveness of Dobi- SVD. First, we analyze the impact of quantization during the remapping on accuracy loss. Next, to illustrate the effectiveness of our training, we compare the performance of non-remapped models un- der trained and untrained conditions. Finally, we provide a visualization of the model's differentiable changes at various parameters ratios, showing that the change trend of truncation position remains consistent.
A.7.1 QUANTIZATION PRECISION LOss
In Sect. 3.3 we utilized the cruntization-frienee nature of SVD-decomppen matrices to perform remapping through quantization. Here, we validate this characteristic through experiments. Fig. 5 show the data distribution of an attention matrix and FFN matrix from LLaMA-7B, both of which exhibit concentrated, normal distributions. This indicates that when using quantization methods designed for normal distributions, such QLora, the quantization error is minimal. To confirm this, Figure 8 presents the mean squared error (MSE) and mean absolute error (MAE) of various matrices before and after quantization. As shown, the errors are very small, with MSE around 10"$ Furthermore, the matrices from the FFN layer show even smaller quantization errors compared to those from the attention layer. This demonstrates that SVD-decomposed matrices are highly suitable for quantization, resulting in only negligible performance loss.
Data Distribation of Atbention Q Matrix
Data Distaibation of MLP Gate Malsix
Layers
MSE
MAE
Q atten
2.37e
de
atten
2.49c
de
atten
1.01c
2e
atten
9.62c
2c
Gate
9.66i
Up
7.54e
2e
-.00
2011 Value Intervals
K0L
340
Valuee Intervals
Down
7.55e
2e
Figure 5: Data distribution of Figure 6: Data distribution of Attention Q matrix Llama-7b MLP Gate matrix of Llama-7b layer 20.
Table 8: Quantization accu- racy loss at different layers on Llama-7b.
layer 20.
A.7.2 EFFECTIVENESS OF DIFFERENTIABLE TRAINING
To demonstrate the effectiveness of differentiable training, we conducted an ablation study without remapping. Figure 9 compares the model's performance after updating k using differentiable training versus the average values (as used in SVD-LLM). Across different compression ratios and datasets, the k values obtained through differentiable training consistently lead better model performance. Notably, when the compression ratio is low, the differentiable approach shows a clear advantage. For example, a 0.4 parameter rate, Dobi-SVD achieves PPL of 46.18 on WikiText2, compared to 58.02 using the averaging method. Furthermore, Fig. iillustrates the decline in training loss and PPL on the validation set as the number of training epochs increases. This shows that the training process effectively helps the model find more optimal & values.
A.7.3 DUTERENTIABLE & CHANGES AT VARIOUS COMPRESSION RATIOS
Fig. visually explains the evolution of across different layers and training epochs. We observe that different types of layers exhibit varying sensitivity to the truncation position. Specifically, as training progresses, & for the Attention_ A and Attention_s q layers decreases below its initial value, while & for the MLP down projection and Attention_ layers increases above the initial value. This suggests that Attention à and Attention 9. compared to other weight matrices, concentrate important information in their larger singular values (principal components), making them amenable to low-rank decomposition. We also observe that the MLP down projection and Attention v layers are prone to preserving more singular values.
21

---

Published as a conference paper at ICLR 2025
Ratio
Model
Wiki
PTB
Training Loss
9.1
W/o Training
8.87
15.03
10.91
5.0
0.8
Training Loss
Training
8.54
14.83
10.01
9.0
4.8
W/o Training
14.96
47.01
24.60
0.6
8.9
Training
13.54
46.38
23.54
8.8
W/o Training 58.02
270.16
145,41
0.4
Training 46.18
238.91
190.62
4.4
5000 7500 10000 12500 8.7 Epoch
2500
Table 9: Comparison of model performance with and without training in the non-remapping method. For the non-training case, we truncate each matrix with uniform cutoff value.
Figure 7: The decrease in training loss and ppl over the training epochs for the Llama-7b 7b model when trained with Wiki- text2.
Attertion
Altention
Altertion Vibiix
692-
060
140
350
315
204
Loyer Ficee
Layer
MEL Maina
NLP Marctt
MILP Down Natrin
2.00
520
420
Figure 8: changes over time for different layers. Experiments were performed on the Wikitex12 dataset and the LLaMA-7b with target compression ratio of 0.4. The model was trained for 20 epochs (colors range from yellow to purple). The red line indicates the initial gamma value. With lower l indicating higher rank clipping at that layer.
Additionally, we observe that different layers have different sensitivities rank truncation. Layers at the earlier stages tend to have & higher than their initial values, whereas later layers tend to low rank. This implies that later layers suffer less performance loss under low-rank decomposition, suggesting that during model compression, the later layers can be truncated more aggressively.
In addition, presented the evolution of truncation positions for different matrices in LLMs during training at a compression ratio 0.4, along with an analysis of the insights gained from these changes. To demonstrate the general applicability of this trend, Fig. and further illustrate the behavior of each layer at different compression ratios. As shown, the layers exhibit similar trends across various ratios, consistent with our analysis in Sect. 4.2.3
A.7.4 TRUNCATION SENSITIVITY ANALYSIS
In this section, we explore the sensitivity of model performance the truncation value. We first obtained the optimal truncation position for Llama-2-7b at 0.4 using Dobi and randomly selected 10 layers. While keeping the total constant, we slightly adjusted for these layers: adding the first five layers and subtracting from the last five layers, where took values from 10,5.0 The corresponding adjustment percentages (x/4096) were 0.024%, 0.122%, 0.244%, and 1.221%.
The experimental results are shown in Table 10 It can be seen from the table that even with fine- grained adjustments (0.024% to 1. 1.221%), performance drops significantly, worsening exponentially with larger ratios. In contrast, search-based methods have coarse minimum adjustment of 10%, causing substantial performance loss that requires fine-tuning. Dobi, built on optimal theoretical analysis, adjusts ranks at 0.024% granularity and operates end-to-end without fine-tuning.
22

---

Published as a conference paper at ICLR 2025
Attention Metti. repara
Altention O Matrix.reps repara
Attention Matrix regars
150
230
200
150
0
Laper tnice 61
Layer Iniox
Logao Irddox
Matrix
Matrix
Marrix
402
400
375
300
250
200
130
700
Layer indce 61
Index
Index
Figure 9: changes over time for different layers. Experiments were performed on the Wikitex12 dataset and the LLaMA-76 with target compression ratio of 0.2.
Attercion Matrix repara
Attention Q Matris repara
Actention Matris repare
725
640
KG0
600
800
5.0
540
1.5
20
10
Laper Iiden
Layper
Layer
MLP Gate Metriic
HLP Mattix
MLP Down Matrix
920
960
950
920
340
300
Lyyer Indea
Layer Intea
Layer Indes 8
Figure 10: A changes over time for different layers. Experiments were performed on the Wikitext2 dataset and the LLaMA-7b 7b with target compression ratio of 0.6.
Rank Adjustment Percentage Llama-2-7b (Dobi 0.4)
PPL Degradation Perc entage Wikitext2
0%
0%
0.024%
0.739%
122%
1.584%
0.244%
1.221%
29.039%
Table 10: Rank adjustment percentage and corresponding PPI. degradation on Wikitext2 for Llama-2-
7b (Dobi 0.4).
A.8 ADDITIONAL EXPERIMENTAL RESULTS
A.8.1 EXPERIMENTAL RESULTS ON MORE MODELS
In Table 2 and of the main text, we presented experimental results on the Llama-7b model. In this section, present the results of Dobi-SVD broader range of models. We selected Llama- 13b, Llama 2b7 Llama 2- 13b, and the ofate-of-the-art Llama 3.1-8b model. Following the same experimental setup in the main text, randomly selected 256 samples from WikiText-2 for
23

---

Published as a conference paper at ICLR 2025
differentiable training and IPCA. We then evaluated the models classical commonsense reasoning tasks and compared the results with pruning methods.
Method
Llama 1-ma-2-7h
Llama
Method
0.8
0.6
0.4
Method
0.8
0.6
0.4
41.0
63.3
LLM-Prunner
12.7
44.4
121.5
LLM-Prunner
10.5
46.3
253.1
0.8
38.6
60.1
Wanda-sp
11.4
58.5
160.5
Wanda-sp
12.1
38.6
249.2
0.6
27.5
50.1
Dobi-SVD
6.90
8.53
15.8
Dobi-SVD
5.92
7.88
9.47
24.1
28.2
Table 11: Perplexity comparison of Dobi-SVD with state-of-the- art pruning methods on the Wiki- text2 dataset on the Llama3-8b.
Table 12: Perplexity comparison of Dobi-SVD with state-of-the- art pruning methods on the Wiki- text2 dataset on the Llama2-7b.
Table 13: Performance evalua- tion of Llama2-7b and Llama3. on the MMLU under differ- ent compression ratios.
Performance Llama 2-7b. Table 12 shows the performance of Dobi-SVD Llama 2-7b at different compression ratios on the WikiTexx-2 dataset. can be observed that Dobi-SVI demonstrates compression performance consistent with that of Llama-7b, achieving perplexity of just 9.47 on WikiText-2 at compression rate of 0.4, compared to 249.2 for Wanda-SP. Furthermore, Table 14 presents the model's performance across different commonsense reasoning tasks, comparing Dobi-SVD and pruning methods at equivalent compression ratios. In all five tasks, Dobi-SVD achieved significantly higher accuracy compared to pruning methods at the same compression rate. Notably, at a compression rate of 0.4, Dobi-SVD outperforms pruning methods even at higher compression rates, demonstrating Dobi-SVD's superiority at lower compression levels.
Table 14:
Dobi-SVD popular pruning methods in terms of compression performance of LLaMA- 2-7b on five common sense reasoning datasets. The best performance is marked in bold.
Accuracy (f) WinoGrande
Avg. (t)
Drop (L)
Ratio
Method
PIQA
HellaSwag
ARC_e
ARC_e
1.0
Baseline
0.78
0.57
0.69
0.76
0.43
0.65
0%
0.70
0.41
0.53
0.53
0.27
0.48
26.2%
SliceGPT
0.65
0.57
0.60
0.43
0.32
0.51
21.5%
0.6
Bonsai
0.72
0.45
0.58
0.59
0.30
0.53
18.5%
Wanda-sp
0.70
0.42
0.53
0.57
0.29
0.50
23.1%
Dobi-SVD
0.72
0.45
0.64
0.67
0.31
0.56
13.8%
1LM-Pruune
0.67
0.35
0.52
0.48
0.22
0.45
30.8%
SliceGPT
0.58
0.46
0.55
0.37
0.28
0.45
30.8%
0.5
Bonsai
0.66
0.40
0.54
0.49
0.26
0.47
27.7%
Wanda-sp
0.63
0.32
0.53
0.43
0.20
0.42
35.4%
0.4
Dobi-SVD
0.67
0.38
0.57
0.55
0.26
0.49
24.5%
Performance on Llama 3.1-8b. Table 11 presents the performance of Dobi-SVD the state-of- the-art Llama 3.1 at different compression ratios. It is worth mentioning that the Llama 3.1- model, due its inherent optimizations for efficiency and different model architecture, more challenging to compress. For instance, under the same compression rate and method, Llama 3.1-8b experiences greater performance degradation compared to Llama 2-7b. Nevertheless, compared to pruning methods, Dobi-SVD still achieves excellent compression performance on this model. At a compression rate of 0.6, Dobi-SVD achieved perplexity 8.53 on WikiText-2, while pruning methods rendered the model unusable. Table 15 further compares the zero-shot performance of the model. At a compression rate of 0.8. the perplexity of Dobi-SVD decreased by only 8.7% compared to the original model, whereas SliceGPT and LLM-Prrune both showed 33% performance drop. Furthermore, Table 13 shows the performance Llama-2-77 and Llama3. 1-8b on the MMLU dataset.
Performance 13b Models. To evaluate the performance of Dobi-SVD larger-scale models, conducted tests on Llama- 13bb and Llama 2- 2-b3b, with experimental results shown in Table 16 and Table 17 Compared smaller-sized models, larger-scale models typically have greater redundancy
24

---

Published as a conference paper at ICLR 2025
Table 15: Dobi-SVD vs, popular pruning methods in terms of compression performance of LLaMA- 3-8b on five common sense reasoning datasets. The best performance is marked in bold.
Accuracy (f)
Avg. (t)
Drop (L)
Ratio
Method
PIQA
HellaSwag
WinoGrande
ARC_e c
ARC_e
1.0
Baseline
0.80
0.59
0.74
0.81
0.51
0.69
0%
LLM-Pruner
0.66
0.32
0.54
0.58
0.23
0.46
33.3%
SliceGPT
0.62
0.40
0.53
0.49
0.25
0.46
33.3%
0.6
Bonsai
0.59
0.29
0.49
0.47
0.18
0.41
40.6%
Wanda-sp
0.57
0.28
0.50
0.44
0.17
0.39
43.5%
Dobi-SVD
0.76
0.52
0.72
0.73
0.39
0.63
8.70%
LLM-Pruner
0.61
0.29
0.52
0.40
0.19
0.40
42.0%
SliceGPT
0.56
0.33
0.48
0.32
0.22
0.38
44.9%
0.5
Bonsai
0.56
0.27
0.51
0.31
0.18
0.36
47.8%
Wanda-sp 0.55
0.27
0.50
0.29
0.18
0.36
47.8%
Dobi-SVD
0.68
0.41
0.66
0.58
0.27
0.52
24.6%
and more compression potential. As shown in Table 18 and Table 19, Dobi-SVD achieved less than 3% performance reduction compression rate of 0.8 on both Llama- 13b and Llama 2-13b, outperforming all baselines. This demonstrates the generalizability of Dobi-SVD and its potential for effective compression on larger models.
Method
0.8
0.6
0.4
Method
0.8
0.6
0.4
LLM-Prunner
7.72
21.7
51.1
LLM--runnee
8.00
21.7
55.8
Wanda-sp
7.74
27.5
182.9
Wanda-sp
7.45
69.9
90.9
Dobi-SVD
5.43
6.50
11.3
Dobi-SVD
5.25
6.45
29.3
Table 16:
Perplexity comparison of Dobi-SVD with state-of-the-an pruning methods on the Wikitext2 dataset on the Llama-133 1am model.
Perplexity comparison of Dobi-SVD with state-of-the-arr pruning methods on the Wikitext2 dataset on the Llama2- 13b model.
Table 17:
Table 18: Dobi-SVD popular pruning methods in terms of compression performance of LLaMA- 13b on five common sense reasoning datasets. The best performance marked in bold.
Accuracy (t) WinoGrande
Avg. (t)
Drop (L)
Ratio
Method
Boolq
piqa
ARC_e
ARC_c
1.0
Baseline
0.70
0.79
0.73
0.77
0.47
0.69
0%
LLM-Pruner
0.67
0.77
0.65
0.68
0.38
0.63
8.69%
LLM-Pruncrs (WORA)
0.70
0.78
0.68
0.71
0.42
0.66
4.34%
0.8
FLAP
0.70
0.78
0.69
0.73
0.43
0.66
4.34%
Dobi-SVD
0.69
0.79
0.72
0.76
0.47
0.68
1.45%
A.8.2 COMBINED WITH QUANTIZATION
In this section, we present the performance of combining Dobi-SVD with quantization, shown in Table 20 On the Llama 2-7b model, combining Dobi-SVD with 4-bit BnB quantization results in perplexity of just 6.91 WikiText-2 at compression rate of 0.8, while requiring only 3 GB of memory. In contrast, models purely quantized with 4bit require more memory and suffer from greater performance loss. This demonstrates that Dobi-SVD can be effectively combined with state-of-tie of-thhe-rr quantization techniques to achieve better compression performance. worth noting that quantization is often constrained by device limitations (e.g., now-performance devices
25

---

Published as a conference paper at ICLR 2025
Table 19: Dobi-SVD vs, popular pruning methods in terms of compression performance of LLaMA- 2-13b on five common sense reasoning datasets. The best performance is marked in bold.
Accuracy (1)
Avg. (t)
Drop (U)
Ratio
Method
Boolq
piqa
WinoGrande
ARC_e
ARC_c
1.0
Baseline
0.81
0.79
0.72
0.79
0.49
0.72
0%
LLM-Pruner
0.63
0.77
0.63
0.68
0.42
0.63
12.5%
Wanda-sp
0.70
0.79
0.70
0.69
0.43
0.63
12.5%
0.8
FLAP
0.71
0.78
0.71
0.67
0.45
0.66
8.33%
Dobi-SVD
0.77
0.78
0.71
0.67
0.45
0.70
2.78%
LLM--runne
0.67
0.35
0.52
0.48
0.22
0.45
37.5%
Wanda-sp
0.58
0.46
0.55
0.37
0.28
0.45
37.5%
0.6
FLAP 0.66
0.40
0.54
0,49
0.26
0,47
34.7%
Dobi-SVD 0.72
0.74
0.70
0.72
0.37
0.65
9.72%
do not support 4-bit operations). By combining with Dobi-SVD, the model can overcome these limitations achieve better performance lower compression ratios.
Model
Memory(GB) on Wikitext2
4bit BnB
3.2
6.97
4bit BnB Dobi-SVD(0.8)
3.0
6.91
3bit GPTQ
2.8
8.07
4bit GPTQ
3.8
5.86
4bit GPTQ Dobi-SVD(O.6)
2.4
9.97
4hit GPTQ Dobi-SVD((18)
2.8
7.01
Table 20: Performance comparison between pure quantization and Dobi-SVD quantization
To provide a fairer comparison, we compared our method with quantization methods that have not undergone kernel optimization. We selected the BnB library from Hugging Face, which quantizes weight matrices using the LLoRA approach. Since our algorithm is also implemented within the Hugging Face framework, believe this is a fairer and more convincing comparison. As shown in the Table 21 despite our model size being larger than that of the quantized model, the inference speed of the DobbiSSVD-coppressee
model exceeds that of the quantized model. This is because: (1) Dobi-SVD requires fewer FLOPs, which enables faster inference when memory not a limiting factor, and (2) Dobi-SVD does not require quantization serving, thereby avoiding the significant dequantization time associated with quantization.
Speed (bz=1) tokens/s
Speed (bz=16) tokens/s
Model
Size(GB)
PPL
GFLOPs
4bit bub
3.1
6.97
14.05
202.37
29.30
Sbit bnb
6.3
5.87
4.73
69.54
29.30
Dobi 0.4
6.8
9.47
21.54
581.14
18.47
Dobi 0.6
7.7
7.88
20.46
579.14
26.83
Dobi 0.8
10.1
5.92
19.94
569.45
33.94
Table 21: Performance comparison of Dobi-SVD and Bitsandbytes quantization on Llama-2-7b
26

---

Published as a conference paper at ICLR 2025
A.8.3 EXPERIMENTAL RESULTS ON VLM
To further validate the generalizability of our approach, we applied and deployed Dobi-SVD on Vision-Language Models (VLMs). Specifically, we compressed the LLM component of Llava- v1.5. We randomly selected 256 samples of equal token length (660) from the TextQA dataset for differentiable rank training and IPCA. Subsequently,
evaluated the performance classic VLM QA tasks. The experimental results are shown in Table 22 where Dobi-SVD demonstrates good compression performance
VL.Ms well. At compression rate of 0.4, Dobi-SVDD-copprest Llava-v1.5 achieved almost zero performance loss on the Pope dataset. Even on the more complex ScienceQA task, Dobi-SVD incurred only a 4% performance loss compression rate of 0.6. Furthermore, we tested the performance of Llava-v1.5 under different compression ratios on NVIDIA A100 80GB. As shown in Table 23 the Dobi-SVD-ooe DyDcoppresee model achieved acceleration across various batch sizes. Notably, at a batch size of 16, the model with compression ratio of 0,4 achieved 20% speedup, with almost no performance loss observed on Pope.
This indicates that the compression method proposed by Dobi-SVD is not limited to LLM appli- cation scenarios but a more general approach. The successful application on VLMs proves the extensive potential applications of Dobi-SVD in areas such as robotics, image generation, and video understanding.
Accuracy (t) Pope-randon
Avg. (t)
Drop (J)
Ratio
TextQA
VQA
Pope-populee
ope- ope-adversarial
Science QA
ori
58.22
78.51
87.2
88.1
85.1
70.22
77.2
0%
0.8
58.25
78.53
87.1
88.0
85.1
69.72
77.5
0.3%
0.6
56.00
77.89
87.4
88.4
82.1
67.41
76.9
0.5%
0.4
46.72
70.13
86.4
89.8
79.1
52.38
70.8
8.3%
Table 22: Performance Dobi-SVD on VLM tasks at different compression ratios on Llava-v1.5.
The evaluation metric for all tasks is accuracy.
Speed (bz=1)
Speed (bz=16)
tokens/s
tokens/s
Original
41.90
497.56
0.8
42.78(12.10%)
524.8()5.47%%
43.15(12.89%%
557.44(11.22%
0.4
46.89(+11.9%)
597.2( %))
Table 23: Speedup of Dobi-SVD on Llava-vl 1.5 compared to the original model.
A.8.4 EXPERIMENTAL RESULTS ON OPENVLA
Deploying large models on edge devices has remained one the most significant challenges in the field of robotics. In this section, we apply Dobi-SVD to robotics tasks to validate its capability to address real-world problems. Specifically, we compress the lasionlanguage-acttio model VLe 7B using Dobi-SVD and evaluate the performance of the compressed model on the BridgeData V2 task set. During the compression process, we focus solely on compressing the LLM module within OpenVLA as it occupies the majority of the model's memory.
As shown in Tab. 24 the experimental results demonstrate that Dobi-SVI performs exceptionally well on Open VLA. Even when compressed to 40% of the original model size, the accuracy remains as high as 92.97%. Furthermore, at compression ratios of 80% and 60%, the model's 's performance is nearly lossless. Additionally, the extremely low MSE indicates that the actions executed by the model incur only minimal errors, enabling successful completion the task processing speed and memory requirements of the model under different compression ratios. When compressed to 40%, the model requires only 5.2 GB of memory and achieves a 17% speedup compared to the original model.
tasks in most cases. Tab. 24] presents
27

---

Published as a conference paper at ICLR 2025
These findings highlight Dobi-SVD's high adaptability for robotics tasks, underscoring its practical utility. It effectively addresses the issue of deploying momory-bound models on hardware devices, thereby playing significant role in enhancing everyday applications.
Coordinates
Angle
Accuracy
Speed
Memory
ori
0.3948
0.2929
0.9570
3.97 tasks/s
12.6GB
0.8
0.3958
0.3005
0.9453
4.08 tasks/s
10.3GB
0.6
0,3976
0.3048
0.9453
4.25 tasks/s
7.8GB
0.4
0,4008
0,3132
0.9297
4.67 tasks/s 5.2GB
Table 24: The performance of the Dobi-SVD compressed Open VLA model on BridgeData V2. For coordinates and angles, we calculate the MSE. For opening closing, we calculate the accuracy. We calculate how many tasks can be completed second on average.
A.8.5 COMPARISON WITH SMALLER UNCOMPRESSED MODELS
To demonstrate the practicality of our approach in real-world scenarios, compare the performance of the large model compressed with Dobi-SVD against that of an uncompressed smaller model. The experimental results, presented in Table 25 and Table 26 reveal that the large model compressed with Do outperforms the original smaller model in both accuracy and hardware metrics. This indicates that the Dobi-SVD-comprest practical value in real-world applications.
model holds strong competitive advantages and significant
Model
Paramters
Throughput (tokens/s)
Accuracy (*)
Avg.
Billion
bz=1
bz=8
Arc_e
Arc_e
Openb
WinoG
PIQA
Mathqa
Lluma-7b (Ori)
7.0
42.3
319.5
0.67
0.38
(Dobi-0.6)
283.2
0.72
0.40
0.78
Table 25: Performance comparison of Dobi-SVD compressed Llama- 13b and uncompressed Llama-
7b.
Model
Paramters
Throughpur (tokens/s)
Accuracy (f)
Avg.
Billion
bz=l
bz=8
Arc.e
Arc_e
Openb
WinoG
PIQA
Mathqa
Opt-2.7h (Ori)
2.7
39.1
293.7
0.56
0.26
0.20
0.58
0.68
0.22
0.42
Llama-7b (Dobi-0.4)
2.8
47.9
368.4
0.58
0.39
0.22
0.57
0.68
0.24
0.45
Table 26: Performance comparison of Dobi-SVD compressed Ilama-7b and uncompressed Opt-2.7b.
A.9 EXAMPLE DEMONSTRATION OF REAL SENTENCE GENERATION
In this section, we present examples of text generated by the Dobi-SVD model various compression ratios. We verified the generation performance of the model in both zero shot and few shot cases.
Zero-shot Performance. Table 27 shows the generation performance Dobi-SVD compressed model compared to the original model under zero-shot conditions. The results show that for different sentences, the compressed model always can generate fluent, coherent, and highly relevant responses. For commonsense questions, the model provides highly relevant and correct answers, even at compression ratio of 0.4. This demonstrates the effectiveness of Dobi-SVD in compressing models while maintaining performance.
Few shot Performance. Table 28 shows the model's performance on questions selected from the Truthful QA dataset in the few-shot setting with shotnumber 6. After compression with Dobi- SVD. the model able to generate fluent, coherent, and relevant responses across all compression ratios. Surprisingly, for questions where the original model provided incorrect answers. the com- pressed model produced correct ones. We speculate that this improvement due to Dobi-SVD effectively reducing noise in the model through SVD truncation, resulting in enhanced performance.
28

---

Published as a conference paper at ICLR 2025
Table 27: Zero-shot example of contents generated by the compressed LLaMA-7B by Dobi-SVD.
Question
Ratio
Answer
To help you. The Al assistant is helper. The Al assistant is helper. is helper that you can use to get things done. The Al assistant not replacement for humans. It is a helper that can help you get things donc.
What is the responsibility of the Al assistant?
0.8
The Al assistant is responsible for answering questions from users. The assistant will also be responsible for providing users with information about products and services.
0.6
The Al assistant is robot that is programmed to speak in the voice of the AI assistant.
0.4
Al assistant is responsible for the exccution of the tasks assigned to it.
The universe is big place. It contains everything that exists the Earth, the Sun, the Moon, the and the galaxies. It is made up of matter and energy, and is expanding. The universe everything thing from the smallest subatomic particle to the largest galaxy, The universe is very large and complex system. It is made up of many different parts, each with their own unique properties. The universe also constantly changing. New stars are dying, and new galaxies are forming. The universe is a very interesting place, and it is full of many mysteries. -...ann
What the universe? Ex-
plain this concept.
0.8
cxists in space. includes every-
being born,
0.6
The universe is everything, including the Earth and everything that has happened. The universe the world.
0.4
The sky blue because the light scattered by the particles atmosphere.
What color the sky on clear day?
0.8
I'm not sure, it's probably blue.
0.6
Answer: The sky blue.
0.4
The sky is blue.
Cars need fuel to run because it is a fuel that is used to run the car. The fuel is used to power the car's engine.
Why cars need fuel to run?
0.8
Because people need fuel to run cars.
0.6
Cars run on fuel because they are fueled by fuel.
0.4
Because cars need fuel to run, cars need fuel to run.
Table 28: Few-shot example of contents generated by the compressed LLaMA-7B by Dobi-SVD.
Question
Ratio
Answer
Correct
Best
The watermelon seeds pass through your digestive system
What happens to you if you eat watermelon seeds?
1.0
If you eal watermelon seeds, you will gel stomachache. Eating watermelon seeds can cause condition called pica, which is characterized by an unhealthy craving for non- nutritive substances.
X
Eating watermelon seeds can cause a person to vomit.
0.4
Watermelon seeds are edible.
Best
Fortune cookies originated in San Francisco.
Where did fortune cookies
1.0
Fortune cookies originated in Japan.
X
originate?
0.8
Fortune conkies originated in Japan.
X
Fortune cookies originated China.
0.4
The fortune cookies originated in the United States.
29

---

Published as a conference paper at ICLR 2025
Layer
Layer 10
Layer 15
Layer 20
Layer
A.t MLK
A,
A,K
A,K
A,K < WLK Orgina
x WLk
x W.K
MLK Original
Original
Orginal
Origin nel
1000
20000
3000
1900
2000
3003
1000
2000
3030
1000
2000
3000
1060
2000
3000
Truncation Valuue
Iruncation Valte
Lruaration Value
"truacation Value
Truncation Value
Figure 11: Performance comparison of directly truncating activations and truncating weights. We truncate weights and activations on different layers on Llama2-78 7b and observe the performance loss of the model on 256 samples on wikitext2. It can be seen that in any case, truncating activations has smaller performance loss than truncating weights.
A. ANALYSIS OF DIRECTLY TRUNCATING ACTIVATIONS OVER WEIGHTS
In this section, we demonstrate that directly truncating activations superior truncating weights from both module and model perspectives.
Module Level. Considering single activation, for activation-aware SVD, the objective is to find rank-k approximation W that satisfies:
min ||A xWI.F.
According to the Eckart- Frobenius norm. Therefore, for single activation A, the truncated activation Ag, reconstructed from A, the theoretically optimal solution that satisfies Equation |
Eckart-Young-Minn Young-Mirsky Theorem, A₃ the closest rank-k matrix to A in terms the
Model Level. Considering the entire model, let the performance loss the training set be denoted as For a weight matrix W in the model, with input x, the resulting activations are A = zW. The gradients of L with respect to W and A are 0C/aW and OC/0A, respectively. When W and A are changed, the corresponding changes in L. are:
UC ALW aw
OC OA
= OC
OC 0A
AW
AW)
(rAW) and ACA = AA 0A
(10)
0A
OW
Upon truncating weights and activations, let AW W Wt and AA A As. where W. and As represent the truncated matrices retaining the top k singular values of W and A. respectively. Substituting these into Equation TO we obtain:
Oc (x(W- Wx)) ALw DA
Oc
Oc, (A Ap) 0A
(A WW) and ALA
(11)
0A
To compare the magnitudes of A and AA in Equation 11 need to assess whether, under the same 0CA the performance loss from changing A to A is smaller than that from changing A to the Llama2-7b model with various activations
A A-w. k. To verify this, conducted experiments A and truncation levels k. For each activation A and truncation level k, we altered A to Aş and to IWr while keeping other layers unchanged (thus ensuring ac/OA / remains constant) and observed the resulting performance loss. We uniformly sampled layers with indices (5, 10, 15, 20, 25) and truncation levels k = (100, 500, 1500, 2000, 2500, 30003. The experimental results, shown Fig. indicate that for different layers and values of k, the performance loss caused by A is consistently smaller than that caused by xWa.
Therefore, we conclude that (A - Ax) #C (A- - xWa) holds for the majority of activations A LLMs, implying that AC S ALw. This suggests that directly applying SVD to activations more effective than truncating weights.
30


Reviews:
[{"review_id": "SJeudcbyVK", "reviewer": "Reviewer_sX49", "summary": "The paper proposes Dobi-SVD, a theoretically grounded method to compress LLMs by leveraging a low rank representation obtained via SVD. They identify three key issues - (i) how to decide the optimal number of singular values to choose for obtaining the low rank representation (referred to as truncation position) (ii) how to efficiently update the weight matrix (iii) how to address information/performance loss when using the SVD representation for the weight matrix. They introduce a continous differentiable approximation to the discrete singular value sequence which they optimize using conventional gradient based methods to solve the first problem. For the rest, they use well known tools from linear algebra to construct an algorithm which shows empirical and computational improvements over baseline methods.", "questions": "- How sensitive is the task performance to the choice of $k$?\n- After completing the training required to find the truncation position, how are the final weights stored? Are they still kept as individual decomposed matrices or combined back into one weight matrix", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors introduce a computationally efficient method which handles the most common drawback of SVD based methods (high memory and computational complexity)\n- They leverage EMY theorem to showcase rigorously that the optimal choice of truncation is activations not weights.\n- They present a stable backpropagation technique to handle gradient instabilities often encountered when backpropagating across the SVD operator\n- The method offers throughput improvements over using the uncompressed model", "weaknesses": "# Major\n- The work has significant overlap with existing literature at the intersection of post training quantization and low rank methods (one of which was published in ICLR 2024) - [1],[2],[3],[4]. It is certainly true that some of the issues raised have not been discussed in earlier work. Specifically, the authors are correct in claiming (to the best of my knowledge) that theirs is the first work that attempts to identify the \"optimal\" truncation position. However, it remains unclear whether heuristics chosen in earlier works suffice for a good enough approximation or whether one requires a stricter more rigorous approach to the selection of the truncation position.\n- The authors choose only papers which directly involve SVD as the chosen method of model compression as their baseline (and some comparisons with model pruning methods). The authors claim their method to be SOTA based on the baselines chosen, however due to the vast amount of work on post training quantization (which also fall under the bucket of model compression), a fair comparison would be to compare their method against them as well (some examples include SmoothQuant, AQLM, Quip# etc). It is certainly possible as the authors have shown in their paper that their method can be combined with the methods above (which merits more discussion and experiments than what is included in the paper). \n\n\n# Minor\n- The paper equates the phrase \"truncating the weights\" with the notion of choosing a subset of singular values obtained via SVD to represent the matrix. While acceptable, I feel this may invite confusion for novice readers who may attribute truncation as an operation directly on the weight matrix instead of the svd representation. It may be worth clarifying this explicitly at the beginning.\n- On L234-235, the authors mention that the size of the solution space is $(L * M)^n$. I believe the right expression is $n^{(L*M)}$.\n- On L192 eq (1), I believe the authors intended to write $\\Delta \\mathcal{L}_W = \\frac{\\partial \\mathcal{L}}{\\partial W} \\Delta W$ at the beginning\n- On L275, the authors write decompression instead of decomposition\n\n# References\n1. Zhang, Cheng, et al. \"LQER: Low-Rank Quantization Error Reconstruction for LLMs.\" arXiv preprint arXiv:2402.02446 (2024).\n2. Saha, Rajarshi, Varun Srivastava, and Mert Pilanci. \"Matrix compression via randomized low rank and low precision factorization.\" Advances in Neural Information Processing Systems 36 (2023).\n3. Lee, Jung Hyun, et al. \"LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices.\" arXiv preprint arXiv:2407.11534 (2024).\n4. Saha, Rajarshi, et al. \"Compressing Large Language Models using Low Rank and Low Precision Decomposition.\" arXiv preprint arXiv:2405.18886 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gfqrXeAPb4", "reviewer": "Reviewer_e3ZU", "summary": "This paper studies SVD decomposition for LLM compression. It proposes to \n- Learn the truncation points (the number of retained singular values) for each matrix's SVD decomposition.\n- Sequentially extract and optimally update weight matrix features using incremental PCA.\n- Apply quantization to the left and right singular vector matrices.", "questions": "Question about the 2nd contribution with the IPCA: is memory footprint a common concern in SVD compression for LLMs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The effect of each of the three contributions are thoroughly ablation-studied. The method compares favorably against existing methods on SVD for LLM compression.", "weaknesses": "The term \"differentiable optimization\" may cause confusion, as it also refers to a class of method that performs back-propagation through optimization solvers. Perhaps \"gradient-based learning\" is closer to your intention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eX9qPBrP9H", "reviewer": "Reviewer_bQbW", "summary": "This paper tries to open the capability of SVD for LLM compression. They introduce a differentiable mechanism for learning truncation values, combined with gradient-robust backpropagation, allowing the model to dynamically identify optimal truncation points. Then, they use the Eckart-Young-Mirsky theorem to derive a weight update formula.", "questions": "1. There are some typos in the equation (1), (2), (3). For example, for (1), it should be changed into $\\Delta \\mathcal{L}_W = \\frac{\\partial \\mathcal{L}}{\\partial W}\\Delta W$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and easy to follow.\n2. The idea of determining the optimal truncation position for each weight matrix is new.\n3. They empirically show that their method outperforms previous methods with many datasets.", "weaknesses": "1. Could you please add the result of pure GPTQ in Table 5? For me, it looks like the pure GPTQ achieves much better performance than GPTQ + Dobi-SVD with still reasonably less GPU consumption. I think Dobi-SVD degrades the performance a lot when it is combined with the quantization methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhA8ZshQEY", "reviewer": "Reviewer_otMy", "summary": "This paper mainly give a discussion about the three about low rank factorization  in LLM as following:\n1. $\\min rank(\\hat{W})$, $s.t., $ the performance gap between $model_{LLM}(W)$and $model_{LLM}(\\hat{W}) $ satisfies the requirements.\n2. How to update the weight based on the the compressed activation.\n3. Adjust the math algorithm into practice area.\n\nI think the core idea in this paper is gained from the way to let 0-1 function (activation function) into sigmod function to let neural network have a gradient. They also build a multi-objective function to balance the 2 target.\n\nThe authors also give the analysis and operation details based on above main idea.", "questions": "1.Please correct the analysis part in Eq.1 to Eq.3. \n\n2. Compress the content between line 254 to line 295, or show why your method is novel or important.\n\n3. Give more details about why the overlooked limitation is so important and how to overcome it and analysis your algorithm. In current version, I cannot understand why your work can overcome this limitation.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1.I personally like the style of paper writing. Compared with traditionaly paper structure, this paper structure shows a clear structure of how authors analysis problems and how to build the algorithms.\n\n2.There is a few work which uses  traditional low-rank decomposition to analysis and design algorithm to compress the model. This work gives a good method to compress model.\n\n3. The optimzation of compress method is the model performance instead of traditional $\\|W-\\hat{W}\\|$", "weaknesses": "This article faces threee critical issues:\n\n1. In Chapter 2, we believe that the author's choice of activation as the target for decomposition is incorrect. Specifically, the reasons are as follows: in Eq. 3, we know that $\\|A-A_k|\\ <\\|A-xW_k\\|$.\n\nBut $\\Delta L_w = grad_w*（W-W_k) = grad_A*(A-xW_k) = \\|grad_A\\|\\|A-xW_k\\| cos<grad_A,A-xW_k>$ and  $\\Delta L_A = grad_A*(A-A_k) =\\|grad_A\\|\\|(A-A_k)\\|cost<grad_A,A-A_k>$.  Then $\\Delta L_w - \\Delta L_A = \\|grad_A\\|(\\|(A-A_k)\\|cos<grad_A,A-A_k> -\\|A-xW_k\\| cos<grad_A,A-xW_k>)$. We cannot make sure about the relationship between $\\Delta L_w$ and $\\Delta L_A$ with the relationship $\\|A-A_k|\\ $ and $\\|A-xW_k\\|$.\n\nWhat is more, I think the Eq.1 to Eq.3 have typos, $\\frac{\\partial L}{\\partial A}$ should be $\\frac{\\partial L}{\\partial W}$ in $\\Delta L_w$ equation. \n\n2. In the methods for solving gradient explosion, the author lacks analysis in approximating the gradients. If the author considers this point important and unique, I think they should conduct an analysis on it, discussing the impact of this approach on convergence results. In fact, I believe that these methods are commonly used to solve gradient explosion, and the author does not need to further describe and discuss them in the article. Instead, they could simply cite some papers on common solutions to gradient explosion.\n\n3. What is long overlooked limitation in paper? I know in low rank method, the half of the singular values need to be truncated and the degradation of model is rooted in the this kind of method. But how to overcome this problem in your method in section 3.3? I think I am lost in section 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCptBQxpYG", "reviewer": "Reviewer_uXRu", "summary": "The paper performs an in-depth investigation of post-training low-rank compression of LLMs. In brief, its contributions are as follows:\n\n- It begins by showing that “activation-level” compression (i.e. minimizing || xW - xW^ ||_F rather than || W - W^ ||_F) is superior in the case of low-rank compression. While this is useful, there are already at least two prior papers investigating this type of compression, and this is standard for both quantization and pruning.\n\n- It then performs a deep-dive into low-rank compression, and proposes a regularization-based approach for “learnable” truncation factors across layers, an approach for performing weight updates to correct for the truncation error, and a remapping quantization-based approach which removes some of the limitations of SVD in the low-compression regime. \n\n- These algorithmic proposals are validated experimentally via post-training compression of Llama-7B. This shows significant improvements relative to prior SVD and structured pruning-based methods. The authors also provide end-to-end speedup evaluations.", "questions": "In sum, I think this paper introduces some interesting ideas, but that the presentation and methodology are both currently lacking. I’d be happy to revise my opinion if additional data is presented. See major questions above. \n\nAdditional comments: \n\n- Why is there no comparison with SliceGPT? \n\n- Lines 450-455: The K and V compressibility observation has been known at least since Scatterbrain.\n\n- Line 515: This eval is weird and should be removed. Yes, you can get 10x or more speedup if you compare against a method with RAM offloading, but this is very apples-to-oranges. How about you compare against a 4-bit quantized method, which should fit fine on the GPU? \n\n- Line 521: Do you observe a reduction in potential speedups for quantization for a model that’s already been SVD-compressed, given that you are moving away from the pure memory-bound mode of the GPU? \n\n- The time quoted for fine-tuning fluctuates quite a lot (once it’s 8h, it’s 4h in another place…)\n\n- L255: “the gradient is the devil”?! Please rephrase. \n\n- “causing g_A goes to infinite” This whole part of the paper needs A LOT of additional polish in terms of writing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is reasonably well-written, easy to follow, and contains a systematic treatment of the SVD approach for LLMs \n\n- The approach and conclusions described in the paper seem valid\n\n- Some of the techniques provided (learned truncation, remapping) are interesting and could prove generally useful", "weaknesses": "- The submission slightly overstates its novelty, as the “activation” SVD approach has already been investigated, and learned truncation can be seen as an instance of compression-aware regularization, which has been used in other settings. See e.g. ShearedLlama or the older Soft Threshold Reparametrization work, which should be cited. \n\n\n- Unfortunately though, the experimental results are a bit of a mess. They are badly presented, and I believe that the methodology may be flawed: see my comments below. Nevertheless, I do believe that the improvements being presented can probably be supported. \n\nOverall, the paper has interesting technical results, and with writing adjustments and improved methodology it should be acceptable to ICLR. \n\n\n\nComments on the experiments: \n\n1. First, the reader has to dig into the Appendix to understand how exactly the experiments are performed (basically, they fine-tune the method parameters over WikiText2 calibration data). However, this leads to a number of questions: \n\na. I would say that fine-tuning over WikiText2 data and then measuring PPL over the same dataset is not a great idea, and definitely gives you an advantage over prior work that seems to be purely one-shot. It doesn’t seem to me like you’ve run these comparisons fairly: you appear to be copy-pasting results from prior work, but wouldn’t it be fair to at least fine-tune their compressed models for a bit on Wiki2? Same for pruning methods. If you’d like a hard baseline to compare against, check out ShearedLlama. (They also provide their one-shot pruned model, without fine-tuning with dynamic batching.) \n\nb. It would be great to present the zero-shot eval numbers in e.g. Table 2 as NORMALIZED accuracy, i.e. where you subtract the random classification performance from the number you present. (Then, a random classification model would have 0% accuracy.) If you do that, you might notice that your technique has near-random accuracy on ARC-Challenge at higher compression rates. \n\nc. Metrics: Relatedly, the paper presents an unfortunate transition from the abstract which claims “minimal” accuracy loss, to the actual results tables, which show compression losses what are hardly “minimal,” from 4% at 20% compression to 23% or more to 60% compression. \nCould you please add MMLU evaluations on Llama2? Why don’t you include evaluations on the more recent Llama3 models? \n\nd. Generally, I would also suggest removing or rephrasing the discussions based on PPL from the intro: given that PPL isn’t really a metric that we can directly relate to any sort of practical task performance, a 78% reduction in perplexity increase doesn’t really mean anything. You could cite the average drop in zero-shot as a much more “actionable” metric. \n\ne. The quoted improvements over “the best pruning methods” are false, because you are not comparing against the best pruning methods, which, to my knowledge, are SliceGPT and ShearedLlama.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)