SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
Authors: Vijay Lingam
Atula Tejaswi Neerkaje
Aditya Vavre
Aneesh Shetty
Gautham Krishna Gudur
Joydeep Ghosh
Eunsol Choi
Alex Dimakis
Aleksandar Bojchevski
sujay sanghavi
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter Efficient Fine Tuning
Large Language Models
Deep Learning
Primary area: deep_learning_architectures

Abstract: Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights $\(\mathbf{W}\)$ and inject learnable matrices $\(\mathbf{\Delta W}\)$. These $\(\mathbf{\Delta W}\)$ matrices are structured for efficient parameterization, often using techniques like low-rank approximations or scaling vectors. However, these methods typically exhibit a performance gap compared to full fine-tuning. While recent PEFT methods have narrowed this gap, they do so at the expense of additional learnable parameters. We propose SVFT, a *simple* approach that structures $\(\mathbf{\Delta W}\)$ based on the specific weight matrix $\(\mathbf{W}\)$. SVFT updates $\(\mathbf{W}\)$ as a sparse combination $\(M\)$ of outer products of its singular vectors, training only the coefficients of these combinations. Crucially, we make additional off-diagonal elements in $M$ learnable, enabling a smooth trade-off between trainable parameters and expressivity—an aspect that distinctly sets our approach apart from previous works leveraging singular values. Extensive experiments on language and vision benchmarks show that SVFT recovers up to **96%** of full fine-tuning performance while training only **0.006 to 0.25%** of parameters, outperforming existing methods that achieve only up to **{85\%}** performance with **0.03 to 0.8%** of the trainable parameter budget.

SVFT: Parameter-Efficiet Fine-Tuning with Singular Vectors
Vijay Lingam Atula Tejaswi!*
Aditya Vavre
Aneesh Shetty
Gautham Krishna Gudurt Joydeep Ghosh
Alex Dimakis
Eunsol Choii
Aleksandar Bojchevski*
Sujay Sanghavi
University of Texas at Austin
University of Cologne
$CISPA Helmholtz Center for Information Security
Abstract
Popular purameter-efficient fine-tuning (PEFT) methods, such LoRA and its variants, freeze pre-trained model weights W and inject learnable matrices A W. These AW matrices structured for efficient parameterization, often using tech- niques like low-rank approximations or scaling vectors. However, these methods typically exhibit performance gap compared full fine-tuning. While recent PEFT methods have narrowed this gap. they do so at the expense of additional learnable parameters.
propose SVFT a simple approach that structures W based on the specific weight matrix W. SVFT updates W as sparse combination M of outer products of its singular vectors, training only the coefficients of these combinations. Crucially, make additional off-diagonal elements in M learnable, enabling a smooth trade-off between trainable parameters and expressivity an aspect that distinctly sets approach apart from previous works leveraging singular values. Extensive experiments on language and vision benchmarks show that SVFT recovers up 96 % of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that achieve only up to 85% % performance with 0.03 to 0.8% of the trainable parameter budget.
Introduction
Large-scale foundation models often adapted for specific downstream tasks after pre-training. Parameter-efficicnt fine-tuning (PEFT) facilitates this adaptation efficiently by learning a minimal set of new parameters, thus creating "expert" model. For instance, Large Language Models (LLMs)
pre-trained on vast training corpora are fine-tuned for specialized tasks such as text summarization [13] 37.. sentiment analysis [27 21), and code completion (88 using instruction fine-tuning datasets. Although full fine-tuning (Full-FT) is a viable method to achieve this, it requires re-training and storing all model weights, making it impractical for deployment with large foundation models.
To address these challenges, PEFT techniques [14 (e.g., LoRA (15) were introduced to significantly reduce the number of learnable parameters compared to Full-FT, though often at the cost of perfor- mance. DoRA [19 bridges this performance gap by adding more learnale parameters and being more expressive than LoRA. Almost all these methods apply a low-rank update additively to the frozen pre-traaee trained weights, potentially limiting their expressivity. Furthermore, these adapters are agnostic to the structure and geometry of the weight matrices they modify. Finally, more expressive
"indicates equal contribution/addiiing
"Code is available at /tttps://githue wtt.e//o//////eee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

550
125
Full Fine-Tuning (2500M
Fine-Tuning (2500M params)
500
36RMe
SVFT
47.5
Accuracy (%)
45.0
SVIT...
SVFT:-.
DoRAA
32
pvP
00.0
423
Da
400
VellAwezee
97.5
LoRAuet
DoRA
95.0
350
ValkAr Loge
ba
62.5
Velt wiit
norngng
50.0
(M)
Figure I: Performance total trainable parameters for GSM-SK (left) and Commonsense Reasoning d=16 outperforms DoRAy=s/16 with 75% less trainable parameters.
(right) on Gemma-2B. SVFT
PEFT methods (e.g., LoRA, DoRA, BOFT [20) still accumulate a considerable portion of learnable parameters in their most efficient configuration (e-g., setting rank=1 in LoRA and DoRA). The storage requirements for the learnable adapters can grow very quickly when adapting to a large number of downstream tasks [17].
it possible to narrow the performance gap between PEFT and Full-FT while being highly parameter-efficient? Yes, we propose SVFT: Singular Vectors guided Fine-Tuning a simple approach that involves updating an existing weight matrix by adding to a sparse weighted combination of its own singular vectors. The structure of the induced perturbation in SVFT depends on the specific matrix being perturbed. Our contributions can summarized as follows:
We introduce SVFT, a new PEFT method. Given weight matrix W, SVFT involves adapting it with matrix AW = Eiiyee miyu, where the (u:} and {wy) are the left and right singular vectors of W. Ω is an a-priori fixed sparsity pattern, and may for (i,.j) E f2 are learnable parameters. By controlling |S2] we can efficiently explore the accuracy vs parameters trade-off.
SVFT achieves higher downstream accuracy, function of the number of trainable parameters, as compared to several popular PEFT methods (see Figure 1} and over several downstream tasks across both vision and language tasks. For instance, GSM-8K using Gemma-2B our method recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that only recover up to 85 % performance using 0.03 to 0.8 % the trainable parameter budget (see Figure T).
We introduce four simple variants for parameterizing weight updates, namely: Plain, Random, Banded, and Top-k in SVFT (which differ their choices of the fixed sparsity pattern ?)) and validate these design choices empirically. Additionally, we theoretically show that for any fixed parameters budget, SVFT can induce higher rank perturbation compared to previous PEFT techniques.
Related Work
Recent advancements in large language models (LLMs) have emphasized the development of PEFT techniques to enhance the adaptability and efficiency of large pre-trained language models.
LoRA. A notable contribution in this field is Low-Rank Adaptation (LoRA) [15], which freezes the weights pre-trained models and integrates trainable low-rank matrices into each transformer layer. For pre-trained weight matrix Wo d LoRA constraints the weight update AW to low-rank decomposition:
Wo + AWr x = Wa + AAx, where B E Rdxr A E x' and rank & min(d, nt. We underline the (trainable) parameters that updated via gradient descent.
LoRA variants. We highlight some recent approaches that further improve the vanilla LoRA architecture. Vector-based Random Matrix Adaptation (VeRA) [1 minimizes the number of

---

35
Disccton
ROLe
1/1M-
Pestrare Weighs
W gd
socn byen
Weee
y
dtt
21-mm
B-toirale
Figure 2: Schematic comparison of LoRA, VeRA, DoRA, and SVFT (left to right).
trainable parameters by utilizing pair of low-rank random matrices shared between layers and learning compact scaling vectors while maintaining performance comparable to LoRA. Formally, VeRA can be expressed as: h = W AWæ Woz+ T1, A Ar,, where A and B are initialized randomly, frozen, and shared across layers, while A, and a are trainable diagonal matrices.
An alternative approach, Weight-Decomposed Low-Rank Adaptation (DoRA) 19]. decomposes pre- trained weight matrices into magnitude and direction components, and applies low-rank updates for directional updates, reducing trainable parameters and enhancing learning capacity and training sta- Wo+
bility. DoRA can be expressed as: h m Wo+&WTe I = m [ No-BBlll" 2, where || lle denotes the vector-wise norm of matrix across each column. Similar to LoRA, Wo remains frozen, whereas the magnitude vector m (initialized to Wolle) and low-rank matrices A, contain trainable parameters.
Wo+AW
AdaLoRA 38 adaptively distributes the parameter budget across weight matrices based on their importance scores and modulates the rank of incremental matrices to manage this allocation effectively. PiSSA (Principal Singular Values and Singular Vectors Adaptation) [2 is another variant of LoRA, where matrices A, are initialized with principal components of SVD and the remaining components are used to initialize Wo. FLoRA [34 enhances LoRA by enabling each example in mini-batch to utilize distinct low-rank weights, preserving expressive power and facilitating efficient batching, thereby extending the domain adaptation benefits of LoRA without batching limitations.
Other PEFT variants. Orthogonal Fine-tuning (OFT) 26 modifies pre-trained weight matrices through orthogonal reparameterization to preserve essential information. However, it still requires a considerable number of trainable parameters due the high dimensionality of these matrices. Butterfly Orthogonal Fine-tuning (BOFT) (20 extends OFT's methodology by incorporating Butterfly factorization therehy positioning OFT as a special case of BOFT. Unlike the additive low-rank weight updates utilized in LoRA, BOFT applies multiplicative orthogonal weight updates, marking significant divergence in the approach but claims to improve parameter efficiency and fine-tuning flexibility. BOFT can be formally expressed as: h (R(m, b) Wo),, where the orthogonal matrix
R(m, b) E gdxd is composed of product of multiple orthogonal butterfly components. When 1. BOFT reduces block-diagonal OFT with block size b. When
and d, BOFT reduces to the original OFT with an unconstrained full orthogonal matrix.
SVD-based Variants. SVF 31, SVDiff [0] and SAM-Parser 25 also leverage the structure of W matrices by decomposing them into three consecutive matrices via Singular Value Decomposition (SVD). However, these methods fine-tune only the singular values while keeping other components fixed, making them comparable SVFTP. Appendix C. we present a comparison of SVFT? with SVF, confirming that their performance is similar, which supports our observations.
3 Method
In this section, we introduce Singular Vectors guided Fine-Tuning (SVFT). The main innovation SVFT lies in applying seneture/geomeeryy-aaaa weight updates through sparse weighted combination of singular vectors.
3

---

Instantiations of
Random
Top-
M
dimis
zor06
Figure 3: An Overview of SVFT. The original weights W are decomposed into U, Σ, V. Here, M contains all the trainable parameters, which can be configured into patterns such as Plain, Random, Banded, and Top-k, represented by patterns of trainable (orange) and zero (gray) elements.
3.1 SVFT Formulation
We now formally describe our method, SVFT for parameter-efficient fine-tuning pre-trained model. Let Wo E IRd1 xd2 denote weight matrix in the pre-trained model, such as key matrix, query matrix, or an MLP matrix within a transformer block. To this matrix, we add a structured, learnable update AW as follows.
As first step, we compute the Singular Value Decomposition (SVD) of the given matrix: Wo UEVT. That is, U is the d1 x d1 matrix of left singular vectors (i.e., its columns are orthonormal), V is the d2 x d2 matrix of right singular vectors (i.e., rows are orthonormal), and Σ is a dj d2 diagonal matrix. Then, we parameterize our weight update as AW = UMV where U, V are fixed and frozen, while M is a d1 d2 sparse trainable matrix with pre-determined and fixed sparsity patterr That is, we first pre-determine small fixed set of elements in M that will be allowed be non-zero and train only those elements. The forward pass for SVFT can be written as,
h Wor + AW* U(E +M)+
(1)
We explore four simple choices for Ω the pre-determined sparsity pattern of M. Plain (SVFT?). In this variant, we constrain M to be a diagonal matrix, which can be interpreted adapting singular values and reweighting the frozen singular vectors. Since only the diagonal elements are learned, this is the most parameter-efficient SVFT variant.
Banded (SVFT ). this approach, we populate M using a banded matrix, progressively making off off-diagonals learnable. Specifically, for constants 21 and 22, My if < 21 or j > i 22, to induce off-diagonal elements that
where 21, 22 2 0. In experiments, capture additional interactions beyond those represented by singular values. This banded perturbation induces local interactions, allowing specific singular values interact with their immediate neighbors. ensuring smoother transitions. This method, although deviating from the canonical form of SVD, provides a mechanism capture localized interactions.
z1 22
Random (SVFT A straightforward heuristic for populating M involves randomly selecting elements to be learnable.
Top-k SVFTT The final design choice we explore involves computing the alignment between the left and right singular vectors as uu uy" We then select the top-k elements and make them learnable. However, note that this only works when left and right singular vectors have the same size. A possible interpretation of this is we make only the top-k strong interactions between singular vector directions learnable. The subscript #p denotes the total number of learnable parameters.
illustrate these SVFT design choices in Figure 3 Our empirical results demonstrate that these simple design choices significantly enhance performance compared to state-of-the-art PEFT methods. Note that P has fixed number of learnable parameters, while the remaining variants are configurable. We hypothesize that further innovation is likely achievable through optimizing the sparsity pattern of M, including efficient learned-sparsit methods. In this paper, explore these
Learnable parameters underlined.

---

four choices to validate the overall idea: determining perturbation using the singular vectors of the matrix that is being perturbed.
3.2 Properties of SVFT
We highlight some properties of SVFT in the following lemma and provide insights into how its specific algebraic structure compares and contrasts with baseline PEFT methods.
Lemma: Let Wo be matrix of size d1 d2 with SVD given by UEVT. Consider updated
final matrix Wa + UMVT, where M is a matrix of the same size as E, which may or may not be diagonal. Then, the following holds:
(a) Structure: If M is also diagonal (i.e. the plain SVFT), then the final matrix W +UMT has as its left singular vectors and sign + M) as its right singular vectors. That is, its singular vectors are unchanged, except for possible sign flips. Conversely, if M is not diagonal (i.e., variants of SVFT other than plain), then and V may no longer be the singular directions of the final matrix Wo UMV
(b) Expressivity: Given any target matrix of size d1 d2, there exists an M such that P = Wo + UMV"", That is. if M is fully trainable, any target matrix can be realized using this method.
(c) Rank: M has k non-zero elements, then the rank of the update UMVT is at most min min(d1, ). For the same number of trainable parameters, SVFT can produce a much higher rank perturbation than LoRA (eventually becoming full rank), but a constrained structured subspace.
We provide our proofs in Appendix [A] Building on this lemma, we now compare the form of the S SVFT update with LoRA and VeRA. SVFT's AW can be written as a sum of rank-one matrices:
AW Σ
msyu,
(i.))E
where u, is the jth left singular vector, Vy is the jth right singular vector, and Q2 is the set of non-zero elements in M. Thus, our method involves adding a weighted combination of specific rank-one perturbations of the form n.v..
LoRA and VeRA updates also be expressed as sums of rank-one matrices.
A WLLAA LORA
Ea,," b," and AW
Σ aá ④ G)b
where a and by the trainable columns of A and B matrices in LoRA. In VeRA, à; and b; are random and fixed vectors, while o and 3 represent the diagonal elements of Aa and As respectively.
Note that LoRA requires di + d2 trainable parameters per rank-one matrix, while SVFT and VeRA require only one. Although LoRA can potentially capture directions different from those achievable by the fixed {u, o/''>> pairs, each of these directions incurs a significantly higher parameter cost.
VeRA captures new directions at parameter cost similar to SVFT; however, there is a key distinction: in VeRA, each vector à; or b, appears in only one of the rank-one matrices. In contrast, in SVFT, the same vector u, can appear in multiple terms the summation, depending on the sparsity pattern of M. This results in an important difference: unlike SVFT, VeRA is not universally expressive it
cannot represent any target matrix P. Moreover, à b, are random, while u, U2 depend W..
Note. SVFT requires storing both left and right singular vectors due its computation of the SVD on pre- pre-trained weights. While this increases memory usage compared to LoRA, it remains comparable to or lower than DoRA and BOFT. present a memory analysis Section [53 Further exploration of memorr-reduction techniques, such as quantization, is planned as future work. Importantly, inference time and memory consumption remain the same across all methods, including SVFT, as the weights can be fused.
5

---

Experiments
4.1 Base Models & Setup
We adapt widely-used language models, encoder-only model (DeBERTaV) V3 base and two decoder-only models (Gemma-2B// 2B/7B [32]. LLaMA-3-8B []. We also experiment with vision transformer models (ViT-B/16 and ViT-L/16) 99) pre-trained on ImageNet-216 k ( following prior
work [17. The complete details of our experimental setup and hyperparameter configurations are provided in Appendix C}
Baselines. We compare with Full Fine-Tuning (FT) updating all learnable parameters in all layers, along with LoRA [15), DoRA [9], BOFT [20] and VeRA [
Target Modules. We adapt all weight matrices for SVFT, as it does not increase trainable parameters at the same rate as baseline methods. For baselines, we adapt the target modules recommended in [19]: QKVUD matrices for LoRA and DoRA, compatible matrices for VeRA, and QV matrices for BOFT to stay within GPU memory limits. Additional details can be found in Appendix C.7 and
C.8 We also conduct experiments adapting QKVUD modules across methods and observe similar trends, as discussed in Appendix
4.2 Datasets
Language. For natural language generation (NLG) tasks, evaluate GSM-8K 77] and MATH 12] by fine-tuning on MetaMathQAA-400 [35. following [20]. We also evaluate on commonsense reasoning benchmarks (BoolQ (5, PIQA 31. SIQA [30], HellaSwag (36). Wino- grande [29], ARC-easy/challenge [6]. and OpenBookQA (23). We follow the setting outlined in prior work [19] [6]. where the training of all benchmarks are amalgamated for fine-tuning. We fine-tune on 15K examples from this training set. For natural language understanding (NLU), we evaluate on the General Language Understanding Evaluation (GLUE) benchmark consisting of classification and regression tasks, in line with 15].
Vision. Our experiments on vision tasks consist of 4 benchmarks: CIFAR-100 18]. Food101
RESISC45 [33), and Flowers 102 [24]. We follow the setup from [1]. and fine-tune on subset comprising samples from each class.
Table I: Performance (Accuracy) on Mathematical Reasoning (GSM-8K and MATH). #Params denote the number of trainable parameters. bold and underline represent the best and second best performing PEFT methods, respectively. SVFT offers superior/compettiivee performance at much lower #Params. For SVFT d. we set 16 for Gemma and = 12 for LLaMA-3 models.
Gemma-2B
Gemma-7B
LLaMA-3-SB
Method
#Params GSM-8K MATH #Params GSM-SK MATH #Params GSM-8K MATH
Full-FT
2.5B
52.69
17.94
8.5B
74.67
25.70
8.0B
64.13
16.24
LoRA,=32
26.2M
43.06
15.50
68.8M
76.57
29.34
56.6M
75.89
24.74
DoRA,
13.5M
44.27
16.18
35.5M
74.52
29.84
29.1M
75.66
24.72
BOFT
1.22M
36.01
12.13
2.90M
71.79
28.98
4.35M
67.09
21.64
DoRA,
1.19M
35.25
13.04
3.26M
74.37
26.28
2.55M
68.30
21.96
LoRA
0.82M
32.97
13.04
0.82M
72.4
26.28
1.77M
68.84
20.94
VeRA,=1024
0.63M
36.77
14.12
0.43M
71.11
27.04
0.98M
63.76
20.28
SVFT
0.19M
40.34
14.38
0.43M
73.50
27.30
0.48M
69.22
20.44
SVFT
6.35M
50.03
15.56
19.8M
76.81
29.98
13.1M
75.90
24.22
'BOFT's is approximately three times slower than LoRA. The shared matrices in VERA can become a limiting factor for models with non-uniform internal dimensions, such LLaMA-3.

---

Table 2: Evaluation results eight commonsense reasoning benchmarks with Gemma-7B. We follow [19] for hyperparameter configurations, and report accuracy for all tasks. HS and WG denote HellaSwag [6] and WinoGrande [29]. respectively. SVFT offers competitive performance at a fraction of #Params. SVFT can match =oRA,=32 with ~7x fewer parameters.
Method
#Params
BoolQ
PIQA
SIQA
HS
WG
ARC-
ARC-c
OBQA
Average
Full-FT
8.5B
72.32
87.32
76.86
91.07
81.76
92.46
82.76
89.00
84.19
LoRAr=32
68.8M
71.55
87.95
77.27
91.80
79.71
92.67
82.16
86.40
83.69
DoRA,-10
35.5M
71.46
87.59
76.35
92.11
78.29
92.00
80.63
85.60
83.00
DoRA.=1
3.3
68.22
86.72
75.23
91.14
78.13
91.87
83.19
86.20
82.59
VcRA,=204s
1.49M
64.25
86.28
74.04
86.96
69.00
92.76
82.33
82,00
79.70
LoRA.-1
0.82M
65.44
86.28
75.02
89.91
75.92
91.79
81.91
85.40
81.46
SVFT"
0.51M
67.92
86.45
75.47
86.92
74.03
91.80
81.23
83.00
80.85
SVFT
9.80M
71.90
86.98
76.28
91.55
78.76
92.80
83.11
85.40
83.35
Table 3: DeBERTa Vusse with different adaptation methods the GLUE benchmark. We report matched accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher is better for all tasks. iiniccttss values reported in 20]
Method
#Params
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
Full-FT*
184M
89.90
95.63
89.46
69.19
94.03
92,40
83.75
91.60
88.25
LoRA* F=8
1.33M
90.65
94.95
89.95
69.82
93.87
91.99
85.20
91.60
88.50
DoRA,
0.75M
89.92
95.41
89.10
69.37
94.14
91.53
87.00
91.80
88.53
BOFT 19 12
0.75M
90.25
96.44
92.40
72.95
94.23
92.10
88.81
91.92
89.89
LoRA 0.17M
90.12
95.64
86.43
69.13
94.18
91.43
87.36
91.52
88.23
VeRA
0.09M
89.93
95.53
87.94
69.06
93.24
90.4
87.00
88.71
87.73
SVFT
0.06M
89.69
95.41
88.77
70.95
94.27
90.16
87.24
91.80
88.54
SVFTR
2=2
0.28M
89.97
95.99
88.99
72.61
93.90
91.50
88.09
91.73
89.10
5
Results
5.1
Performance on Language Tasks
Natural Language Generation. We present results on mathematical question answering against baseline PEFT techniques across three base models varying from 2B 8B parameters in Table To ensure comprehensive comparison, we test baseline techniques (LoRA, DoRA) with different configurations, and varying hyper-parameters like rank cover a range of learnable parameters from low to high. Note that even when the rank is as low as 1, both methods yield more trainable parameters than S SVFT (~0.2M) shows as much as 18% relative improvement over techniques that use 6x more trainable parameters (BOFT m= LoRA,=1). Against techniques of comparable size (VeRA), SVFT achieves 15.5% relative improvement on average. Even in the default regime, SVFT. matches techniques with at least 3x more trainable parameters. Notably, on GSM-8K, SVFT again achieves 96% % of full fine-tuning performance, while DoRA,=16 recovers 86% with 2x more parameters than SVFTR
Commonsense Reasoning. Table benchmarks with Gemma-7B, and observe similar trends. In the lower and moderately parameter- ized regime (~0..43M), SVFT" shows competitive performance in comparison to LoRA,s1 and
we compare performance on commonsense reasoning

---

Table 4: Performance image classification benchmarks CIFAR- 100 (C100), Food101 (F101), Flowers102 (F102), and Resisc-45 (R45). We only adapt Q, matrices for all methods, following prior work 17
We report accuracy for all tasks.
VIT Base
VIT Large
Method
#Params
C100
F101
F102
R45
#Params
C100
F101
F102
R45
Hcad
78.58
75.14
98.71
64.42
79.14
75.66
98.89
64.99
Full-FT
85.8M
85.02
75.41
99.16
75.30
303.3M
87.37
78.67
98.88
80.17
LoRA,=8
294.9K
85.65
76.13
99.14
74.01
786.4K
87.36
78.95
99.24
79.55
VeRA,=258
24.6K
84.00
74.02
99.10
71.86
61.4K
87.55
77.87
99.27
75.92
SVFT
18.5K
83.78
74.43
98.99
70.55
49.2K
86.67
77.47
99.09
73.52
SVFT
165.4K
84.85
76.45
99.17
74.53
441.5K
87.05
78.95
99.23
78.90
SVFT
165.4K
84.65
76.51
99.21
75.12
441.5K
86.95
78.85
99.24
78.93
DoRAr=1, which have 1.9x and 7.7x more parameters, respectively. Against VeRA, which trains 3.5x more parameters, SVFT shows a relative improvement ~1.16% Similarly, SVFT 2=8 also matches or exceeds methods that use up to 7x more trainable parameters. For instance, SVFT attains average performance of 83.35% with only 9.8M parameters, closely matching LORA, (83.6%% 68.8M parameters). We observe similar trends with Gemma-2B (refer Table II).
d=8
Natural Language Understanding. Results on the GLUE benchmark are summarized Table SVFT matches LoRA,=s and DoRA,=1 which use 12-22x more trainable parameters. Similarly, when compared to OFT and BOFT, SVFT" maintains a comparable average performance despite being 12x smaller. These results highlight SVFT's ability to strike a balance between parameter efficiency and performance, making it an attractive PEFT choice for simple classification tasks.
Parameter efficiency. In Figure I
plot the perfor- mance of SVFT on mathematical reasoning and com- monsense reasoning against other PEFT techniques across a range of configurations. Across trainable parameter budgets ranging from lowest to highest, SVFT obtains the best overall performance, matching methods that re- quire significantly more trainable parameters. These re-
Jedgh: Types
GVV
44
Accuracy (9)
sults establish SVFT as pareto-dominant approach for parameter-effiieet fine-tuning.
Corfurevin
5.2 Performance on Vision Tasks
Number of Trainable
Table 4 presents a comparison between SVFT and other PEFT techniques on image classification benchmarks, ing ViT-B and ViT-L models. The results show that SVFT variants achieve strong balance between performance and parameter efficiency, often surpassing or matching the performance of other methods with fewer learnable parameters. Notably, the SVFT variant attains av- erage accuracy of 83.87% across tasks with ViT-Base, outperforming Full-FT, which achieves close 83. 72%. Additionally, it's important note that in these vision experiments, both classifier head parameters and other learnable parameters are trained.
Figure 4: Performance variation with SVFT based on the adapted weight matrices GSM-8K with Gemma-2B. Adapting more target weight types re- sults in greater gains in performance. In- terestingly, for a fixed parameter budget, adapting U and D weight types gives greater lifts than adapting Q and V.
5.3 Memory Analysis
Although VVF reduces trainable parameters, results higher overall GPU memory usage compared to LoRA. However, fewer trainable parameters lower the memory demands for gradients,

---

activations, optimizer states, and other buffers. To validate this, we used HuggingFace's internal memory profiler measure peak GPU memory usage. Our results, along with the adapted modules for all baselines, are summarized in [Table 5] We observe that SVFT uses approximately 1.2x more memory than LoRA but remains comparable to or more efficient than DoRA. We present additional analysis in Appendix C.5
Table 5: GPU Memory analysis, measured in gigabytes (GB). We report the average performance on GS and MATH. SVFT outperforms both LoRA and DoRA in terms of performance while requiring lesser GPU memory than DoRA.
Gemma-28
Gemma-7B
Method
Target Modules
Params
GPU Mem
Perf.
#Params
GPU Mem
Perf.
LoRA,
V.K.
3.28M
18.88
27.56
8.6M
63.57
51.03
DoRA,
Q.K.V.U.D
3.66M
24.58
28.44
9.72M
78.70
51.94
LoRA, 32
V.K.V.U.D 26.2M
19.06
29.28
68.8M
64.24
52.96
DoRA,=16
Q.K.V.V.D
13.5M
24.64
30.22
35,5M
78.99
52.18
SVET*
Q.K.V.DDO.GO
194K
21.90
27.36
429K
76.26
50.40
SVFT
Q.K.VUDDDDU
3.28M
22.02
31.87
9.8M
76.65
50.99
SVFT
Q.K.VUUDDDOG
6.35M
22.15
32.79
19.8M
77.04
53.40
5.4
Contribution of Each Weight Type
In Figure 4 we investigate the contribution of each weight type. Starting with the base configuration. we apply SVFT to the Q and V weights in each transformer block and report the performance. We then incrementally add the remaining weight modules (K, U, D, G) and observe the changes in performance. For each configuration, we also vary the trainable parameters by incrementing the total learnable dif-diagosals.
Note that applying SVFT to U, and G does not increase trainable parameters much as applying LoRA/DoRA to these modules (Table 8). For example, for large matrix of shape dj x d2. LoRA,-1 learns d1 + d2 parameters, while SVFT learns min(d1, parameters. We observe that adapting only U and D with SVFT yields up to 10% relative improvement over adapting Q and V for the same parameter budget (~ 0.8M). Our findings indicate that adapting more weight types enhances performance.
5.5 Impact of M's Structure on Performance
We analyze the impact of various parameterizations of M (Plain, Banded, Random, Top-k) on downstream performance. To ensure a fair comparison, we align the number of trainable coefficients across all variants whenever possible. As shown in Table 7 the Banded variant outperforms the others, followed closely by the Random variant, across different models and tasks. This trend is also evident in the average rank column of table. Based these empirical findings, we recommend using the Banded variant.
Table 6: Results GSM-8K after fine- tuning on Pythia-2.8B checkpoints at dif- ferent stages of pre-training (PT).
PT Steps
Method
#Params
APrrf
39K
143K
Full-FT
2.5B
21.00
30.09
9.09
LoRA
5.24M
11.22
18.95
7.73
5.6 Impact of Pre-trained Weight Quality
SVFT
5.56M
15.08
23.19
8.11
key feature of SVFT that the weight update de- pends the pre-trained weights W. We therefore ask the following question: Does the quality of pre-trained weights have a disproportionate impact SVFT? To

---

Table 7: Results fine-tuning with SVFT using different M parameterizations.
Gemma-2B
Gemma-7B
LLaMA-3-8B
Avg.
Structure
Rank
#Params GSM-8K MATH #Params GSM-8K MATH #Params GSM-8K MATH
Plain
0.2M
40.34
14.38
0.43M
73.50
27.30
0.48M
69.22
20.44
4
Banded
6.4M
47.84
15.68
19.8M
76.81
29.98
17.2M
75.43
24.44
Random
6.4M
50.03
15.56
19.8M
76.35
29.86
17.2M
74.07
23.78
2
Top-k
6.4M
49.65
15.32
19.8M
76.34
29.72
17.2M
73.69
23.96
3
answer this, we consider two checkpoints from the Pythia suite [2] at different stages of training, Le.. 39K steps and 143K steps, respectively. We fine-tune each of these checkpoints independently with Full-FT, LoRA, and SVFT. We then compare the increase in performance (APerf). As shown in Table 6 compared to LoRA, SVFT benefits more from better pre-trained weights. We also note that SVFT outperforms LoRA in both settings, suggesting that the benefits of inducing A W that explicitly depends on W are beneficial even when W is sub-optimal.
6 Discussion
Limitations. Despite significantly reducing learnable parameters and boosting performance, SVFT incurs some additional GPU memory usage. Unlike LoRA, SVFT necessitates computing the SVD and storing both left and right singular vectors. While memory consumption remains comparable lower than DoRA and BOFT, it's roughly 1.2x that of LoRA. However, similar the scaling explored in (34, memory usage should amortize with the increasing scale of adaptation tasks. In future work we will explore quantization and other techniques address memory concerns.
Broader Impact. Our work enables easier personalization of foundational models, which can have both positive and negative societal impacts. Since our method provides computational efficiency (smaller parameter footprint), it will be less expensive to enable personalization.
7
Conclusion
This work introduces SVFT, a novel and efficient PEFT approach that leverages the structure of pre- trained weights determine weight update perturbations. We explore four simple yet effective sparse parameterization patterns, offering flexibility in controlling the model's 's expressivity and the number of learnable parameters, Extensive experiments on language and vision tasks demonstrate SVFT's effectiveness PEFT method across diverse parameter budgets. Furthermore, we theoretically show that SVFT can induce higher-rank perturbation updates compared existing methods, for fixed parameter budget. In future work, aim to develop principled methods to generate sparsity patterns, potentially leading to further performance improvements.
Acknowledgements
We would like to thank Greg Kuhlmann for helping support this research. This work was also supported by the NSF institutes ENCORE and IFML.
References
[1] Meta AI. Introducing meta llama 3: The most capable openly available 1lm to date. April 2024.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023.
10

---

[3] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence, 2020.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 - mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[5] Christopher Clark, Kenton Lee, Ming- wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[10] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7323-7334, 2023.
[11] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: style pre-training with gradient-disentangled embedding sharing, 2023.
Improving deberta using electra-
[12] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[13] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS' 15, page 1693-1701. MIT Press, 2015.
[14] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2019.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[16] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. LIm-adapters: An adapter family for parameter-efficier fine-tuning of large language models, 2023.
[17] Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M Asano. ELORA: Efficient low-rank adaptation with random matrices. In The Twelfth International Conference on Learning Repre- sentations, 2024.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

[19] Shih- Yang Liu, Chien-Y Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min- Hinn Chen. Dora: Weight-decompont low-rank adaptation. 2024.
[20] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. The Twelfth International Conference on Learning Representations, 2024.
[21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[22] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404. 02948, 2024.
[23] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018.
[24] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
[25] Zelin Peng, Zhengqin Xu, Zhilin Zeng, Xiaokang Yang, and Wei Shen. Sam-parser: Fine-tuning sam efficiently by parameter space reconstruction. Proceedings of the AAAI Conference on Artificial Intelligence, 18:4515-4525 03 2024.
[26] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Thirty- Conference on Neural Information Processing Systems, volume 36, pages 79320-79366 2023.
[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020.
[28] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong. Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
[29] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
[30] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions, 2019.
[31] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning. In Advances in Neural Information Processing Systems, 2022.
volume 35, pages
[32] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprini arXiv:2403.08295, 2024.
[33] Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi- domain meta-dataset for few-shot image classification. In Thith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
12

---

[34] Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In The Twelfth International Conference on Learning Representations, 2024.
[35] Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023.
[36] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[37] Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328 11339. PMLR, 13-18 Jul 2020.
[38] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Appendix
The appendix is organized follows.
In Appendix A
we give proofs for the lemmas outlined in 3.2
In Appendix B we compare how the trainable parameters count for different PEFT tech- niques (LoRA, DoRA, VeRA) versus our method SVFT.
In Appendix C describe results for additional experiments and provide implementation details for all the experiments.
A Proofs
We provide brief proofs for the Structure, Expressivity and the Rank lemmas for SVFT:
(a) Structure: If M is diagonal, then the final matrix Wo + UMVT can be written UCE + M)VT since Wo UEVT, where (E + M) is also diagonal matrix. Thus, U{E+M
+ MMM is valid and unique SVD of Wo + U UM up to sign flips the singular vectors.
(b) Expressivity: Finding M for any target matrix P of size d1 x d2 such that p= Wo U UMVT the same as finding M for new target matrix P P' = UMM For full SVD, the dimension of M is d1 x d2 and since the dimension of P' also d, x d2, P UMVT a bijection and M U (P Wo) (since U and V are orthogonal).
Wo such that
(c) Rank: If M has k non-zero elements, then the rank of the update U UMVT will be upper bounded by & (since by Gaussian climination, & or less elements will remain, the best case being all elements in the diagonal). We also know that the rank is upper bounded by min min(d, (1, d2). giving an achievable upper bound the rank min min(d1, d2]).
B Parameter Count Analysis
Table 8: Parameter count analysis. Lunelt D h denote total layers being adapted, hidden dimension, rank, and additional off-diagonals respectively.
Method
Trainable Parameter Count
LoRA
2 Liuned D model Leuned X Dmndet X (2r + 1)
DoRA
VeRA
Liunet X (Diodel + r)
SVFT
Liuned X Dmodel
SVFTB
Liuned X (Dmodel le (D model k)) + 1))
C Additional Experiments and Implementation Details
All of our experiments are conducted on Linux machine (Debian GNU) with the following specifi- cations: 2x A100 80 GB, Intel Xcon CPU @ 2.20GHz with 12 cores, and 192 GB RAM. For all our experiments (including baseline experiments), we utilize hardware-level optimizations like mixed weight precision (c.g., bfloat16) whenever possible.
C.1 Comparison against SVD-based Variants
compare SVF 31 and our proposed method, SVFT, on the GSM-8K and MATH benchmarks using Gemma-2B. The results are presented in Table 9 The results indicate that SVF and SVFT ex- hibit comparable performance these benchmarks, expected due to their design equivalence. This
14

---

finding also applies to SVDiff [0] and SAM- parser highlights a significant performance improvement when comparing SVF to SVFT", demonstrating the advantage of learning the off-diagonal elements.
25 for the same reason. Additionally, the table
Table 9; Results of SVF and SVFT on GSM-8K and MATH with Gemma-2BB
Method
#Params
Target Modules
GSM-8K
MATH
SVF
120K
Q.K.V.U,
36.39
13.96
SVFT
120K
Q.K.V.U.D
36.39
13.86
SVF
194K
O.K.VUUDD.OGG
39.12
14.02
SVET
194K
O.K.VUU,D.OGG
40.34
14.38
SVFT 6.35M
Q.K.V.UDD.OG
50.03
15.56
C.2 Performance Evaluation with Fixed Target Module Adaptation
We compare SVFT to baseline methods, adapting the same target modules to ensure consistent evalu- ation. Results are presented in Table 10 showing that SVFT outperforms other methods in this setup.
Performance (Accuracy) on Mathematical Reasoning (GSM-8K and MATH). All methods are applied on the target modules (Q.K,V,U,D) with SVFT demonstrating superior performance. When applying SVFT Gemma-2B and LLaMA-3-SB use d 12 and 24 respectively.
Table 10:
Gemma-2B
LLaMA-3-8B
Method
#Params
GSM-8K
MATH #Params GSM-8K MATH
LoRA,
3.28M
40.60
14.50
7.07M
69.37
22.90
DoRA,-4
3.66M
41.84
15.04
7.86M
74.37
24.10
LoRAr=32
26.2M
43.06
15.50
56.6M
75.89
24.74
DoRA,=1s
13.5M
44.27
16.18
M9.1M
75.66
24.72
SVFT
2.98M
47.41
16.72
15.98M
75.32
25.08
C.3
Commonsense Reasoning Gemma-2B
We evaluate and compare SVFT variants against baseline PEFT methods on commonsense reasoning tasks with Gemma- 2B model and tabulate results in Table
C.4 Are All Singular Vectors Important?
To determine the importance of considering all singular vectors and singular values during fine- tuning, we reduce the rank U and V, and truncate Σ and M to an effective rank of original weight matrix W [R then after truncation, U E R V Raxr, This truncation significantly reduces the number of trainable parameters, so we compensate by increasing the number of off-diagonal coefficients (d) in M.
15

---

Table 11: Results with Gemma-2B eight commonsense reasoning benchmarks. We follow hyperparameter configurations, and report accuracy for all tasks.
19 for
Method
#Params BOOLQ PIQA SIQA HellaSwag Winogrande ARC-E ARC-C OBQA Average
Full-FT
2.5B
63.57
74.1
65.86
70.00
61.95
75.36
59.72
69
67.45
LoRA,=32
26.2M
63.11
73.44
63.20
47.79
52.95
74.78
57.16
67.00
62.43
LoRA
13.5M
62.87
73.93
65.34
53.16
55.51
76.43
59.55
68.4
64.40
BOFT 9-
1.22M
59.23
63.65
47,90
29.93
50.35
59.04
42.66
41.00
49.22
m
-eRA,-2048 2048
0.66M
62.11
64.31
49.18
32.00
50.74
58.08
42.83
42.6
50.23
LoRA,=1
0.82M
62.2
69.31
56.24
32.47
51.53
69.52
48.8
56,4
55,81
DoRA,-1
1.19M
62.17
68.77
55.93
32.95
51.22
68.81
48.72
55.6
55.52
SVFT*
0.19M
62.26
70.18
56.7
32.47
47.04
69.31
50.08
58.4
55.81
SVIT 16
6.35M
63,42
73.72
63.86
71.21
59.58
73.69
54.77
66.6
65.86
Table 12: Performance with varying rank (r) and the off-diagonal elements (d) of M. When
r 2048, the update full-rank.
Rank (r)
Diags (d)
#Params GSM-8K
MATH
128
64
1.55M
0.98
0.21
1536
0.15M
16.37
3.64
1536
0.74M
25.01
6.04
2048
0.19M
40.34
14.38
Our results, with four different configurations of and are presented in Table 12 The findings show that a very low rank (r 128) leads to poor performance, even when parameters are matched. A reasonably high rank of r = 1536, which is 75% of the full rank, still fails to match the performance of the full-rank variant that has 0.25x the trainable parameters. This indicates that all singular vectors significantly contribute to the end task performance when fine-tuning with SVFT, and that important information is lost even when truncating sparingly.
C.5 Additional Memory Analysis Experiments
We present additional memory analysis experiments for Gemma-2B and LLaMA-3-8B in Table 13 and Table 14, SVFT variants consume lesser memory than DoRA and 1.2x more memory than LoRA.
Table 13: Memory analysis for Gemma-2B.
Method
Target Modules
#Params GPU Mem (GB)
GSM-SK
MATH
LoRA,=4
Q.K.V.U,D
3.28M
18.88
40.63
14.5
DoRAy=4
V.K.
3.66M
24.58
41.84
15.04
LoRA,=32
Q.K.V.U,D
26.2M
19.06
43.06
15.5
DoRAr=16
V.K.
13.5M
24.64
44.27
16.18
SVFTR 4-12
Q.K.
2,98M
20.50
47.61
16.72
SVFT
V.K.V.U.D.OG
194K
21.90
40.34
14.38
SVFTR 2=8
V.K.V.U.D..G
3.28M
22.02
47.76
15.98
SVFTR
O.K.V.UDD.O
6.35M
22.15
50.03
15.56
16

---

Table 14: Memory analysis for LLaMA-3-8B.
Method
Target Modules
#Params
GPU Mem (GB)
GSM-8K
MATH
LoRAr=1
V.K.D
1.77M
57.82
68.84
20.94
DoRA,=1
V.K.
2.56M
70.17
68.30
21.96
LoRA,
Q.K.
56.6M
58.41
75.89
24.74
DoRA,=16
V.K.
29.1M
70.44
75.66
24.72
SVFT 8
V.K.
15.98M
71.52
75.32
25.08
24
SVFT 4=12
U,D.O.G
13.1M
70.37
75.90
24.22
SVFT P
O.K.VU.DD.OG
483K
73.95
69.22
20.44
SVFT
Q.K.VU.D..OG
15.9M
71.52
73.99
25.08
C.6 Performance Total Trainable Parameters
In addition to the experiments performed in Figure (main paper) for Gemma-2B on challenging natural language generation (NLG) tasks like GSM-8K and Commonsense Reasoning, also plot the performance vs total trainable parameters for larger state-of-the-art models like Gemma-7B and LLaMA-3-8B GSM-8K, Figure further demonstrates SVFT's aantoodominaace. On larger models, we observe that full-finttnning overfits, leading to sub-optimal performance in comparison PEFT methods.
OVFTt
78
SVTT
LoftA
76
99
DoflAucs
74
Ful Fine-Tunnen Tuning (5500M paramal
SVF
(N)
Duf
73
SVPTI.>
LeRA..
BoFT
68
SVET'
Call
DuRAv
Sor T
85
Full Fine-Turtin (2500M parema)
70
075
12
12.5
Number of Trainable Parama
Number of Trainable Panams
Figure 5: Performance versus total trainable parameters for GSM-8K on Gemma-7B (left) and LLaMA-3-8B (right).
C.7 Settings for Language Tasks
Natural Language Understanding. We fine-tune the DeBERTaV3 | model and apply SVFT all linear layers every transformer block of the model. We only moderately tune the batch size, learning rate, and number of training epochs. We use the same model sequence lengths used by [20] keep our comparisons fair. The hyperparameters used in our experiments can be found Table 15
Natural Language Generation. See the hyperparameters used in our experiments in Table 16 For
LoRA, DoRA, we adapt Q, K, V.U. D matrices. We apply BOFT on Q, V matrices since applying on multiple modules is computationally expensive. For VeRA, which enforces constraint of uniform internal dimensions for shared matrices, we apply on G, U projection matrices as it yields the highest number of learnable parameters. We apply SVFT on Q, K, V.U. U. D, O, G for the Gemma family of models, and U, D, O, G for LLaMA-3-8BB Note that applying SVFT on these modules does not increase trainable parameters the same rate as applying LoRA or DoRA on them would. We adopt the code base from https: / thub. www./e/ttmaat./aaoooooooataataae www.eettttaatttttataaaatttt
git for training scripts and
17

---

Table 15: Hyperparameter setup used for DeBERTaV V buse the GLUE benchmark.
Method
Dataset
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Optimizer
AdamW
Warmup Ratio
0.1
Schedule
Linear
Learning Rate (Head)
6E-03
Max Seq. Len.
256
128
320
64
512
320
320
128
# Epochs
10
10
30
20
10
6
15
15
Batch Size
32
32
32
16
32
SVFT"
Learning Rate
5E-02
5E-02
5E-02
8E-02
8E-02
5E-02
5E-02
5E-02
Batch Size
32
16
SVFT
Learning Rate
1B-02
1E-02
1E-02
1E-02
3E-02
1E-02
3E-02
1E-02
evaluation setups and use the fine-tuning data available at atittgg://ugginggfass./////88s8
me aaa-mathh/ettaMthhhe.
Table 16: Hyperparameter setup used for fine-tuning on MetaMathQAA40K.
Gemma-2B
Gemma-7B
LLaMA-3-8B
Hyperparameter
SVET
SVFTR
SVFT
SVFTR
SVET
SVFTR
d=16
d=16
d=12
Optimizer
AdamW
Warmup Ratio
0.1
LR Schedule
Cosine
Learning Rate
5E-02
IE-03
5E-02
1E-03
5E-02
1E-03
Max Seq. Len.
512
# Epochs
Batch Size
64
Commonsense Reasoning. See the hyperparameters used in our experiments Table We adopt the same set of matrices as that of natural language generation tasks. We use the code base from https://gtt thub. www/AAIIEEgerumnneeeee-eeeaaaeea BeGG--Eggrunnerr/// LLM ALaattttss which also contains the training and evaluation data.
C.8 Settings for Vision Tasks
For each dataset in the vision tasks, train 10 samples per class, using examples per class for validation, and test on the full test set. Similar to previous literature, we always train the classifier head for these methods since the number of classes large. The parameter counts do include the num- ber of parameters in the classification head. The hyperparameters are mentioned in Table 18] We tune the learning rates for SVFT and BOFT select learning rates for other methods from [7]. run training for 10 epochs, and report test accuracy for the best validation model. For all methods, since the classi- fication head has to fully trained, we report the parameter count other than the classification head.
18

---

Table 17: Hyperparameter setup used for fine-tuning commonsense- 15K.
Gemma-2B 2B
Gemma-7B
Hyperparameter
SVET
SVFT
SVET
SVFT
d=8
d=8
Optimizer
AdamW
Warmup Steps
100
LR Schedule
Linear
Max Seq. Len.
512
# Epochs
Batch Size
64
Learning Rate
5E-02
5E-03
5E-02
1E-03
Table 18: Hyperparameter setup used for fine-tuning on all vision tasks.
Hyperparameter
ViT-B
VIT-L
Optimizer
AdamW
Warmup Ratio
0.1
Weight Decay
0.01
LR Schedule
Linear
# Epochs
10
Batch Size
64
SVFT Learning Rate (Head)
4E-03
SVFT" Learning Rate
5E-02
SVFT 2-2
Learning Rate (Head)
4E-03
SVFT 2=2
Learning Rate
5E-02
SVFT 8=8
Learning Rate (Head)
4E-03
SVFT
Learning Rate
5E-03
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are motivated using extensive experiments (see /eccionn 4). Lemma and its proof are introduced where required. We include some limitations of our work in the Limitations section (see section 6).
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
We run extensive experiments on a range of models to study and analyze our method's performance. See section for some limitations. We include parameter count analysis in Appendxix B
Justification:
3. Theory Assumptions and Proofs
19

---

Low-Rank Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We introduce our Lemma in subsection 3.2 and its corresponding proof in Appendix A
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide our method description in section 3/ experimental evaluation in section and hyper parammeeeeeter ranges in [Appendix Cj Additionally, we provide code and scripts to replicate our experiments as part of supplementary material to provide more details.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is publicly available at attttss://gtthub cow/vijayllngeeeeocceim. svft also referenced in our abstract. We include all details on hyper-paramette ranges and methods in Appendix C All our experiments run on publicly available datasets. We provide references to the dataset source in our code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details method in section 3] We provide comprehensive details on hyper-parameter ranges and datasets/model names in [Appendiix CJ Additional available in the code submitted part
details model implementation and experiments of supplementary material. We rely on public datasets, splits, and evaluation strategies from previously published literature. We refer and cite these works in section 4
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since we experiment with large models, it is often difficult run these experiments with multiple seeds and report error bars. We follow the same setup from previously published research these research works also do not compute error bars or report them in any of their results tables. E.g. DoRA: Adaptation (ICML 2024 Accepted paper). Our limited compute resources have hindered us from rumning multiple seed experiments.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We use a Linux machine (Debian GNU/Linux 10) with the following configu- ration: 2x A100 80GB, 192GB RAM, Intel Xeon CPU @ 2.20GHz with 12 cores. Here are some more information on the compute time for our language experiments: 2B models take around hours and 7B/8B models take around 14 hours for training. For vision experiments using vision transformers (ViTs), it takes up to 4 hours for training.
20

---

9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We follow and abide by the ethics guidelines laid out by NeurIPS. We also preserve and conform to anonymity policies.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Our work can allow more easy personalization of founda- tional models which can have both positive and negative societal impact. This is because our method provides computational efficiency (smaller memory footprint), making it less expensive to enable personalization.
Justification: See Section 6
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: To the best of our knowledge, the paper poses no such risks safeguards.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers for utilizing the assets like code (in supplementary material), datasets (subsection 4.2] and models (subsection 4.T) including their appropriate versions. We also abide by different licenses for code bases, models, and datasets used in our work like CC, MIT, META LLAMA COMMUNITY LICENSE AGREEMENT, Gemma Terms of Use, and more.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide assets in the form of code and multiple scripts as a part of the supplementary material for the experiments and ablations performed in the paper. The details about model training, their respective hyperparameters, and limitations are furnished in section Appendix C and section 6 respectively. We also add the appropriate licenses for disseminating our assets, particularly code and scripts in the supplementary material.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing, nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer: [NA]
Justification: The paper does not involve crowdsourcing, nor research with human subjects.
22


Reviews:
[{"review_id": "yyfVAdT6yN", "reviewer": "Reviewer_Cyjn", "summary": "This paper proposes a parameter-efficient fine-tuning (PEFT) method named SVFT. Instead of imposing learnable parameters directly on the pre-trained weight matrices, it applies singular value decomposition to the pre-trained weight matrices and then tunes only the coefficients of the singular values. The method is validated on various vision/language tasks, showing comparable or better performance compared to multiple existing PEFT methods.", "questions": "See Weaknesses", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to follow\n* The method is novel. Studying the behavior of SVFT in more depth can be beneficial to the PEFT community.\n* On both language and vision tasks, SVFT shows comparable or better performance than LoRA, DoRA, VERA, and BOFT, demonstrating the efficacy of the method.", "weaknesses": "* The comparison between VeRA and $\\text{SVFT}^P$ is interesting. Design-wise, they are quite similar, yet $\\text{SVFT}^P$ leads to much better performance than VeRA in most cases, even though VeRA has a larger intermediate dimension (e.g., Table 2). I wonder if the authors can elaborate more on this matter.\n* In many cases, $d=2$ seems to be sufficient for $\\text{SVFT}^B$. However, increasing $d$ for $\\text{SVFT}^B$ does not necessarily improve performance (e.g., Table 4 in the main paper or Table 9 in the appendix). How should we interpret this, and how should we select the suitable $d$ in practice?\n* What advice do the authors have for choosing between $\\text{SVFT}^B$ and $\\text{SVFT}^R$? They seem to have comparable performance on different benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We address the weaknesses (W) raised by the reviewer below. \n\n**W1:** Recall that VeRA injects frozen low-rank random matrices with learnable scaling vectors resulting in weight updates independent of the pre-trained weights. SVFT$^P$ derives its weight updates directly from the pre-trained matrix $W$. \\\nVeRA differs from SVFT$^P$ in two ways: \\\ni) VeRA achieves high parameter efficiency by sharing frozen adapters across compatible layers, a choice that no other previous methods take (e.g., LoRA, DoRA). This particular design choice gives it lower expressiveness than what may have been possible for other methods with the same intermediate dimension. \\\nii) The perturbation directions that VeRA operates on (i.e., its frozen adapters) are chosen independent of the underlying matrix that it is trying to perturb. \n\nWe note that despite the sharing of parameters across layers in VeRA, it ends up having more trainable parameters per layer compared to SVFT$^P$. For example, in Table 2, we achieve better results with SVFT$^P$ using dims=4096 (0.51M params) compared to VeRA with dims=2048 (1.49M params). SVFT$^P$'s use of singular vectors gives it layer-wise specialization.\n\n\n**W2:** As '$d$' (off-diagonals) (or '$r$' for other PEFT methods) increases, performance typically improves. However, overfitting can occur with any method, especially with smaller datasets or models. Figure 5 (left) in the paper shows $DoRA_{r=16}$ underperforming DoRA$_{r=4}$ on language tasks. In Tables 4 and 9, the additional fine-tuning of the classifier head may contribute to overfitting.\n\n\n**W3:** We note that simple heuristics for sparsity patterns in $M$ significantly improve performance over previous PEFT techniques. While most variants (except SVFT$^P$) offer comparable performance, the banded pattern (SVFT$^B$) slightly outperforms others. We hypothesize that a more principled approach to finding optimal sparsity patterns for specific tasks and base models may yield further performance gains. We leave it as future work to explore task-specific and model-specific sparsity patterns. We will revise our paper to reflect these new results and findings.\n\n\n| Results on fine-tuning with SVFT using different $M$ parameterizations. |\n|:---|\n\n| Structure |    | Gemma-2B |   |  | Gemma-7B |  |  | LLaMA-3-8B |  |\n|:----------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:--------:|:-------:|:-------:|\n|           | #Trainable Params | GSM-8K | MATH | #Trainable Params | GSM-8K | MATH | #Trainable Params | GSM-8K | MATH |\n| Plain     | 0.2M    | 40.34  | 14.38 | 0.43M   | 73.50  | 27.30 | 0.48M   | 69.22  | 20.44 |\n| Banded    | 6.4M    | 47.84  | **15.68** | 19.8M   | **76.81**  | **29.98** | 17.2M   | **75.43**  | **24.44** |\n| Random    | 6.4M    | **50.03**  | 15.56 | 19.8M   | 76.35  | 29.86 | 17.2M   | 74.07  | 23.78 |\n| Top-k     | 6.4M    | 49.65  | 15.32 | 19.8M   | 76.34  | 29.72 | 17.2M   | 73.69  | 23.96 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkGgsooYcG", "reviewer": "Reviewer_BEJi", "summary": "This paper presents a new PEFT method to enhance the fine-tuning efficiency of large language models (LLMs). The approach uses SVD to decompose pre-trained weights, initializing adapters with eigenvectors and eigenvalues, and adjusts within the eigenvalue space. Extensive experiments show that the proposed method requires fewer parameters while achieving performance that matches or exceeds other PEFT methods.", "questions": "same as weaknesses", "limitations": "This paper does not reflect any potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy follow. The motivation is clear, the method is straightforward, and it has been evaluated on mainstream tasks in both NLP and CV fields.", "weaknesses": "1. Introducing the M matrix indeed allows for more ways to modify eigenvalues, but the paper does not discuss the differences and advantages between directly fine-tuning the SVD eigenvalue space and the proposed method.\n2. Decomposing the pre-trained weights directly results in a significant increase in the number of parameters during training, leading to much higher training costs compared to methods like LoRA. This is the main drawback of fine-tuning the eigenvalue space.\n3. The approach of fine-tuning the singular value space after SVD decomposition of pre-trained weights is not novel. The paper lacks a summary of such methods, such as [1] Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning and [2] Svdiff: Compact parameter space for diffusion fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We address the weaknesses (W) raised by the reviewer below. \n\n**W1:** We emphasize that our main contribution is making off-diagonal elements in $M$ learnable; this allows us to smoothly trade-off between the number of trainable parameters and expressivity (which is not possible by only fine-tuning the singular values -- the main diagonal of $M$). We justify this theoretically (Expressivity Lemma in Section 3.2) and empirically (in all our results SVFT$^B$, SVFT$^R$, SVFT$^T$ outperforms SVFT$^P$).\n \n**W2:** We acknowledge that while SVFT reduces trainable parameters, it increases overall GPU memory usage compared to LoRA.  However, reducing trainable parameters decreases the memory required for gradients, activations, optimizer state, and various buffers. To address the reviewer's concern, we used HuggingFace's internal memory profiler to log actual peak GPU memory usage. We've included these findings, along with the adapted modules for all baselines, in the table below. In summary, SVFT outperforms LoRA while using ~1.2x more memory. It is worth noting that SVFT's GPU memory usage is either similar to or lower than that of DoRA.\n\n---\n\n### **Gemma-2B**\n\n| Method       | Target Modules | #Trainable Params | GPU Mem (GB)  | GSM-8K | MATH  |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LoRA$_{r=4}$      | Q, K, V, U, D   | 3.28M  | 18.8839 | 40.63 | 14.5  |\n| DoRA$_{r=4}$      | Q, K, V, U, D          | 3.66M  | 24.5798 | 41.84 | 15.04 |\n| LoRA$_{r=32}$     | Q, K, V, U, D          | 26.2M             | 19.0558 | 43.06 | 15.5  |\n| DoRA$_{r=16}$     | Q, K, V, U, D          | 13.5M             | 24.6430 | 44.27 | 16.18 |\n| SVFT$^R_{d=12}$  | Q, K, V, U, D          | 2.98M             | 20.4988 | 47.61 | **16.72** |\n| SVFT$^P$       | Q, K, V, U, D, O, G        | 194K              | 21.8984 | 40.34 | 14.38 |\n| SVFT$^R_{d=8}$   | Q, K, V, U, D, O, G         | 3.28M             | 22.0224 | 47.76 | 15.98 |\n| SVFT$^R_{d=16}$ | Q, K, V, U, D, O, G         | 6.35M             | 22.1465 | **50.03** | 15.56 |\n\n---\n\n### **Gemma-7B**\n\n| Method      | Target Modules | #Trainable Params | GPU Mem (GB)  | GSM-8K | MATH  |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LoRA$_{r=4}$     | Q, K, V, U, D          | 8.60M  | 63.5711 | 73.76 | 28.3  |\n| DoRA$_{r=4}$     | Q, K, V, U, D         | 9.72M             | 78.7007 | 75.43 | 28.46 |\n| LoRA$_{r=32}$    | Q, K, V, U, D        | 68.8M   | 64.2403 | 76.57 | 29.34 |\n| DoRA$_{r=16}$    | Q, K, V, U, D          | 35.5M  | 78.9913 | 74.52 | 29.84 |\n| SVFT$^P$      | Q, K, V, U, D, O, G         | 429K | 76.2630 | 73.5  | 27.3  |\n| SVFT$^R_{d=8}$  | Q, K, V, U, D, O, G        | 9.80M   | 76.6506 | 73.62 | 28.36 |\n| SVFT$^R_{d=16}$ | Q, K, V, U, D, O, G         | 19.8M    | 77.0363 | **76.81** | **29.98** |\n---\n\n### **Llama-3-8B**\n\n| Method      | Target Modules | #Trainable Params | GPU Mem (GB)   | GSM-8K | MATH  |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LoRA$_{r=1}$     | Q, K, V, U, D  | 1.77M | 57.8217 | 68.84 | 20.94 |\n| DoRA$_{r=1}$     | Q, K, V, U, D | 2.56M  | 70.1676 | 68.30 | 21.96 |\n| LoRA$_{r=32}$    | Q, K, V, U, D | 56.6M  | 58.4089 | 75.89 | 24.74 |\n| DoRA$_{r=16}$    | Q, K, V, U, D | 29.1M  | 70.4383 | 75.66 | 24.72 |\n| SVFT$^B_{d=24}$ | Q, K, V, U, D | 15.98M | 71.5224 | 75.32 | **25.08** |\n| SVFT$^R_{d=12}$ | U, D, O, G | 13.1M | 70.3681 | **75.90** | 24.22 |\n| SVFT$^P$      | Q, K, V, U, D, O, G    | 483K  | 73.9495 | 69.22 | 20.44 |\n| SVFT$^R_{d=12}$ | Q, K, V, U, D, O, G | 15.9M | 71.5224 | 73.99 | **25.08** |\n---\n\n\n**W3:** We thank the reviewer for pointing us to these papers (SVF [1], SVDiff [2]). Indeed [1] and [2] are essentially equivalent to SVFT$^P$, but we note that our main insight and contribution is that significant gains can be achieved by learning *off-diagonal* interactions in $M$.\n\nDoing our due diligence, in the following Table, we compare SVFT$^P$ against SVF on GSM-8K and Math benchmarks. Results in the Table below show SVF performs similar to SVFT$^P$ and worse than SVFT$^{R}$ as we had expected. We will acknowledge these papers and add a discussion in our final version of the paper. \n\n---\n| Baseline | #Trainable Params | Target Modules | GSM-8K | MATH |\n|:----------:|:---------:|:----------------:|:-------:|:------:|\n| SVF | 120K | Q, K, V, U, D | 36.39 | 13.96 |\n| SVFT$^P$ | 120K | Q, K, V, U, D | 36.39 | 13.86 |\n| SVF | 194K | Q, K, V, U, D, O, G | 39.12 | 14.02 |\n| SVFT$^P$ | 194K | Q, K, V, U, D, O, G | 40.34 | 14.38 |\n| SVFT$^B_{d=16}$ | 6.35M | Q, K, V, U, D, O, G | 47.84 | **15.68** |\n| SVFT$^R_{d=16}$ | 6.35M | Q, K, V, U, D, O, G | **50.03** | 15.56 |\n| SVFT$^T_{d=16}$ | 6.35M | Q, K, V, U, D, O, G | 49.65 | 15.32 |\n---\n\n  \t\t\n\n**References**\\\n[1] Sun et al.: Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning. NeurIPS 2022\\\n[2] Han et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jqYMOWaSDJ", "reviewer": "Reviewer_x5z5", "summary": "The authors propose a parameter-efficient fine-tuning method with singular vectors (SVFT). SVFT modifies the parameter matrix based on the structure of the original parameter matrix $\\mathbf{W}$. By training the coefficient of a sparse combination of outer products of $\\mathbf{W}$'s singular vectors, SVFT recovers up to 96% of full fine-tuning performance while training 0.006 to 0.25% of the parameter.", "questions": "if $\\text{SVFT}^{R}_{d=2}$, $k=?$ \n\nif $\\text{SVFT}^{T}_{d=2}$, $k=?$\n\nWhat is the purpose of proposing $\\text{SVFT}^T_d$ if it has not shown any performance advantage in any experiment?\n\nHow to choose the appropriate sparsity pattern $\\Omega$?\n\nFor model performance, does randomness in $\\text{SVFT}^T_d$ lead to instability?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It is a simple approach that should be easy to reproduce.", "weaknesses": "Overstatement: the authors fine-tune the coefficient of the original parameter matrix $\\mathbf{W}$'s singular vectors to achieve that fine-grained control over expressivity, which is good - but practically this is still similar to the existing methods. Stating in the abstract that \\`\\` We propose SVFT, a simple approach that fundamentally differs from existing methods: the structure imposed on $\\Delta \\mathbf{W}$ depends on the specific weight matrix $\\mathbf{W}$ .\\'\\' suggests more extensive comparison with existing methods as well. \n\n[1] Peng, Zelin, et al. SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction, AAAI 2024.\n\n[2] Han, Ligong, et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.\n\nI could not find the comparison of $\\text{SVFT}_P$, $\\text{SVFT}_B$, $\\text{SVFT}^R_d$ and $\\text{SVFT}^T_d$ in Tables 1-4. At the moment it's unclear how to choose the sparsity pattern $\\Omega$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We address the weaknesses (W) and questions (Q) raised by the reviewer below.\n\n**W1:** We thank the reviewer for pointing us to these papers (SAM-PARSER [1], SVDiff [2]). Indeed we are not the first one to explore the singular vectors/values space. [1] and [2] are essentially equivalent to our SVFT$^P$ variant, but we note that our main insight and contribution is that significant gains can be achieved by learning *off-diagonal* interactions in $M$.\n\nDoing our due diligence, in the following Table, we compare SVFT$^P$ against SVF (which is also equivalent to SAM-PARSER and SVDiff) on GSM-8K and Math benchmarks. Results in the Table below show SVF performs similarly to SVFT$^P$ and worse than SVFT$^R$ as we had expected (the same holds true for SVFT$^B$ and SVFT$^T$ from the table in W2). We will acknowledge these papers and add a discussion in our final version of the paper. \n\n| Baseline | #Trainable Params | Target Modules | GSM-8K | MATH |\n|:----------:|:---------:|:----------------:|:-------:|:------:|\n| SVF | 120K | Q, K, V, U, D | 36.39 | 13.96 |\n| SVFT$^P$ | 120K | Q, K, V, U, D | 36.39 | 13.86 |\n| SVF | 194K | Q, K, V, U, D, O, G | 39.12 | 14.02 |\n| SVFT$^P$ | 194K | Q, K, V, U, D, O, G | 40.34 | 14.38 |\n| SVFT$^R_{d=16}$ | 6.35M | Q, K, V, U, D, O, G| **50.03** | **15.56** |\n\nTo clarify our contribution, we will revise the highlighted statement to: \\\n*SVFT enables a trade-off between the number of trainable parameters and model expressivity by allowing a flexible number of off-diagonal interactions between singular vectors in $\\Delta$W, distinguishing it from previous SVD-based methods.*\n\n**W2/Q3:** We note that simple heuristics for sparsity patterns in $M$ significantly improve performance over previous PEFT techniques. While most variants (except SVFT$^P$) offer comparable performance, the banded pattern (SVFT$^B$) slightly outperforms others. We hypothesize that a more principled approach to finding optimal sparsity patterns for specific tasks and base models may yield further performance gains. We leave it as future work to explore task-specific and model-specific sparsity patterns. We will revise our paper to reflect these new results and findings.\n\n| Results on fine-tuning with SVFT using different $M$ parameterizations. |\n|:---|\n\n| Structure |    | Gemma-2B |   |  | Gemma-7B |  |  | LLaMA-3-8B |  |\n|:----------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:--------:|:-------:|:-------:|\n|           | #Trainable Params | GSM-8K | MATH | #Trainable Params | GSM-8K | MATH | #Trainable Params | GSM-8K | MATH |\n| Plain     | 0.2M    | 40.34  | 14.38 | 0.43M   | 73.50  | 27.30 | 0.48M   | 69.22  | 20.44 |\n| Banded    | 6.4M    | 47.84  | **15.68** | 19.8M   | **76.81**  | **29.98** | 17.2M   | **75.43**  | **24.44** |\n| Random    | 6.4M    | **50.03**  | 15.56 | 19.8M   | 76.35  | 29.86 | 17.2M   | 74.07  | 23.78 |\n| Top-k     | 6.4M    | 49.65  | 15.32 | 19.8M   | 76.34  | 29.72 | 17.2M   | 73.69  | 23.96 |\n\n**Q1:** For SVFT$^R$ and SVFT$^T$ variants, we select the number of trainable parameters in each target module's weight matrix to match SVFT$^B$'s parameter count for a given 'k'. This is calculated as $k=n(2d+1) - d(d+1)$, where $d$ is the number of off-diagonals in the banded matrix and $n$ is the matrix's smaller dimension. This is to have a fair comparison between methods.  Eg: For $Q$ matrix on Gemma-2B ($n$=2048), when off-diagonals $d=2$, then $k=10,234$ for both SVFT$^T_{d=2}$ and SVFT$^R_{d=2}$ which is the same as that of SVFT$^B_{d=2}$. In practice, ‘$k$’ can be set to any value for these two variants.\n\n**Q2:** We propose SVFT$^T$ to explore a broad spectrum of possible selection criteria of $M$’s sparsity pattern. Our experiments show that simple heuristics-based sparsity patterns can improve performance. In future work, we will explore more principled approaches for finding the optimal pattern.\n\n**Q4:** We did not observe any instability in our experiments. In the worst-case scenario, SVFT$^T$ will be equivalent to SVFT$^R$, which performs well in our experiments.\n\n**References**\\\n[1] Peng, Zelin, et al. SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction, AAAI 2024.\\\n[2] Han, Ligong, et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5VoYQ1IlSH", "reviewer": "Reviewer_DEin", "summary": "- Proposes method for parameter efficient fine-tuning using Singular Value Decomposition of the pretrained weights\n- Fine-tuning is done by adding a learned residual $\\Delta W = UMV^\\top$, where $M$ is a sparse trainable matrix, which results in fine-tuned weigts $\\hat{W} = U\\Sigma V^\\top + UMV^{\\top}$\n- Proposes 4 variants for sparsity: 1) Plain: $M$ is a diagonal matrix 2) Banded: the off-diagonal elements of $M$ are non-zero in a banded pattern 3) Random: Randomly selects $k$ elements of $M$ to be non-zero 4) Top-k: Selects tok $k$ elements for which the alingement between left and rght eigenvectors is the highest\n- Provides theoretical insights on the properties of the method concerning the structure, expressivity and rank of the fine-tuning residuals", "questions": "1. In Figure 1, which weight types are adapted for each method?\n2. For Natural Language Generation tasks, why is SVFT applied to Q,K,V,U,D,O,G for the Gemma family of models, and U,D,O,G for LLaMA-3-8B?\n3. For Math and Commonsense reasoning tasks, what are the training, validation and test splits used, and on which split are the hyperparameters tuned?\n4. For visual classification tasks, why is LoRA and DoRA applied to all the linear layers instead of Q and V as done in the original VeRA paper [2], which shows better performance at a much lower parameter cost? Is the training data from the same, or different even if the number of samples per class are the same?\n    - The performance reported in [2] for VeRA is higher than that reported in this submission (the discrepancy with ViT-B for Food101 is particularly high). What is the cause of this?", "limitations": "Authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using SVD of the pretrained weights for parameter efficient fine-tuning is interesting.\n2. The number of trainable parameters increases at a lower rate than LoRA/DoRA if more layers are adapted.\n3. Provides theoretical insights on the properties of the method concerning the structure, expressivity and rank of the fine-tuning residuals\n4. Shows better performance compared to VeRA, which suggests that SVFT is better for reducing the number of trainable parameters that the random parameters of VeRA (at the cost of higher memory usage). It also does not have a restriction of requiring same dimensions across layers like VeRA.\n5. Observation that truncating the singular vectors leads to loss of performance in Appendix C.3 is interesing and informative.\n5. The code is provided for verification.", "weaknesses": "1. **Missing citation and comparison with prior work**:\n    1) The proposed method is very similar to [1], which has not been cited. [1] fine-tunes the singular values (rather than a residual addition) for few shot semantic segmentation tasks. Given how closely related SVFT $^P$ is to [1] (even though the original work was in context of segmentation methods), a comparison should be performed on the benchmarks in this paper. This work is different enough, and does show additional insights, but a comparison should be performed for completeness.\n2. **Clarity and coherence of writing**:\n    1) In Table 3 (NLU experiments on GLUE), the paper mentions that the results fo some baseline methods are obtained from prior works, but does not cite the actual works that the results are obtained from\n    2) The paper mentions the components chosen for adaptation for SVFT (in the Appendix) but does not mention anything about the baseline methods. This makes it hard for a reader to know the details of the implementation up front, and also makes the comparison between the methods difficult.\n    3) On lines 14, 15, 16 and lines 46, 47, 48, the exact benchmark and model that generate these results should be mentioned - otherwise it's difficult to follow the paper.\n    4) The weight types adapted for each benchmark is different, and is not consistently mentioned in the main text, which makes it hard to follow the paper.\n3. **Increased memory usage during training**\n    1) While it's a positive that SVFT can keep the parameter count down while adapting more weight types, this will come at the cost of increased memory requirements.\n    2) The advantage of reducing the parameters is a reduction in the memory usage due to the gradients (first and second order moments in case of AdamW). It might be possible that the actual memory usage is still comparable to LoRA if only a small number of layers are adapted (this can be verified by checking the actual memory allocated on the GPU during training) given that the modules chosen for adaptation are the same. However, if SVFT requires more weight types to be fine-tuned than LoRA/DoRA, the memory usage will be higher even if the number of parameters is lower. Hence, for all the methods, the actual memory usage must be reported for the chosen weight types the methods are applied to.\n4. **Inconsistencies in evaluations**:\n    1) Natural Language Generation: For LoRA and DoRA, Q,K,V,U,D matrices are adapted (which is inline with [3]). However, for SVFT, Q,K,V,U,D,O,G are adapted for the Gemma family of models, and U,D,O,G for LLaMA-3-8B. This means that the performance cannot be attributed to the method rather than the choice of weight types adapted.\n    2) To decouple the choice of weight types adapted from the method, a comparison with LoRA and DoRA should be performed by adapting the same weight types. The performance of SVFT when Q,K,V,U,D are adapted for all the models must be compared with LoRA and DoRA.\n    3) Relating to the point above, for SVFT, why are U,D,O,G adapted for for Llama 3-8B and Q,K,V,U,D,O,G for Gemma on the Math benchmark?\n    4) With these inconsistent evaluation choices, the effectiveness, advantages and disagvantages of the method are not clear. If SVFT works better than baselines for certain components and not others, the paper should reflect this - and also show the memory required when using the weight types chosen for the respective methods.\n5. **The results on Vision tasks are confusing**:\n    1) The results with ViT-B and ViT-L in the main text are with different benchmarks (with ViT-B: CIFAR100 and Flowers102, and with ViT-L: Food101 and Resisc45). \n    2) On closer inspection of Table 9 in the Appendix, SVFT variants perform worse than LoRA and DoRA on Resisc45 with ViT-B, and marginally better on Food101. This suggests that SVFT is not as effective as LoRA and DoRA on some benchmarks even though it reduces the parameters. This should be mentioned in the main text.\n    3) The paper replicates the settings used in [2]. However, [2] adapts the Q and V layers for LoRA, which results in a high performance at a much lower parameter cost (comparison in the Table below, taken from this submission, and from [2]) than what the authors report with their implementation of LoRA applied to all linear layers. This can mean two things:\n        1) The training data/checkpoints are different, which can cause a difference in performance\n        2) Tuning all the linear layers is not a good idea for LoRA based on these results and [2], and is not indicative of LoRA's actual performance which makes the comparison unfair\n    4) To have a proper comparison with baselines, the above two points need to be disambiguated by testing how LoRA performs by adapting Q and V with the training data used by the authors for this submission. The arguments made for NLG in point 4 are also applicable here.\n\n\n| Method     | #Params | CIFAR100 | Flowers102 | Food101 | Resisc45 | #Params | CIFAR100 | Flowers102 | Food101 | Resisc45 |\n|------------|---------|----------|------------|---------|----------|---------|----------|------------|---------|----------|\n|            |         | ViT-B    |            |         |          |         | ViT-L    |            |         |          |\n| LoRA $_{r=8}$ | 1.32M   | 84.41    | 99.23      | 76.02   | 76.86    | 0.35M   | 86.00    | 97.93      | 77.13   | 79.62    |\n| DoRA $_{r=8}$ | 1.41M   | 85.03    | 99.30      | 75.88   | 76.95    | 3.76M   | 83.55    | 98.00      | 76.41   | 78.32    |\n| LoRA $_{r=1}$ | 0.16M   | 84.86    | 96.88      | 73.35   | 76.33    | 0.44M   | 85.97    | 98.28      | 75.97   | 78.02    |\n| DoRA $_{r=1}$ | 0.25M   | 84.46    | 99.15      | 74.80   | 77.06    | 0.66M   | 84.06    | 98.11      | 75.90   | 78.02    |\n| VeRA       | 24.6K   | 83.38    | 98.59      | 75.99   | 70.43    | 61.4K   | 86.77    | 98.94      | 75.97   | 72.44    |\n| SVFT $^{P}$    | 18.5K   | 83.85    | 98.93      | 75.68   | 67.19    | 49.2K   | 86.74    | 97.56      | 75.95   | 71.97    |\n| SVFT $_{d=2}$ | 0.28M   | 84.72    | 99.28      | 75.64   | 72.49    | 0.74M   | 86.59    | 98.24      | 77.94   | 79.70    |\n| SVFT $^{B=4}_{d=4}$ | 0.50M   | 83.17    | 98.52      | 76.54   | 66.65    | 1.32M   | 87.10    | 97.71      | 76.67   | 71.10    |\n| SVFT $_{d=8}$ | 0.94M   | 85.69    | 98.88      | 76.70   | 70.41    | 2.50M   | 87.26    | 97.89      | 78.36   | 73.83    |\n| From [2]           |         | ViT-B    |            |         |          |         | ViT-L    |            |         |          |\n| LoRA $_{r=8}$ | 0.294M   | 85.9    | 98.8      | 89.9   | 77.7    | 0.786M   | 87.00    | 99.91      | 79.5   | 78.3    |\n| VeRA       | 24.6K   | 84.8    | 99.0     | 89.0   | 77.0    | 61.4K   | 87.5    | 99.2      | 79.2   | 78.6    |\n\n\nFollowing studies are needed to verify the effectiveness of the method:\n1) A comparison with [1] (I would expect this to perform similar to SVFT $^P$, and any one benchmark, say Math, should be enough to verify this).\n2) Reporting the memory usage - Point 3 in Weaknesses\n2) Ablation study for point 4 (ii) in Weaknesses.\n3) Verification for point 5 (iii) in Weaknesses.\n\n**References**:\n\n[1] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, Jingdong Wang: Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning. NeurIPS 2022\n\n[2] Dawid Jan Kopiczko, Tijmen Blankevoort, Yuki Markus Asano: VeRA: Vector-based Random Matrix Adaptation. ICLR 2024\n\n[3] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen: DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the rigorous review and valuable feedback. We have addressed all the weaknesses (W) and questions (Q) with extensive experiments and clarifications.\n\n**W1:** Please refer to our global response **G1** for a comprehensive answer.\n\n**W2:**  \n1, 4. We appreciate you highlighting this oversight. The results marked with * in Table 3 were indeed obtained from BOFT [1].\n\n2. We apologize for not clearly stating the target modules for baselines in the main paper. This information is provided in Appendix C.5. To clarify:\n   - For Table 1: LoRA and DoRA adapt QKVUD modules as per [2]; BOFT adapts QV modules (adapting more leads to OOM); VeRA adapts GU modules (maximizing trainable parameters within its compatibility constraints).\n   - For Table 3: All methods adapt all linear layers, following [1].\n\n3. The statistics mentioned in lines 14-16 and 46-48 are extracted from Figure 1 (left) for Gemma-2B on GSM-8K. We will update this in the final version for clarity.\n\n**W3:**\\\n1, 2. We acknowledge that while SVFT reduces trainable parameters, it increases overall GPU memory usage compared to LoRA. As the reviewer correctly noted, reducing trainable parameters decreases memory required for gradients, activations, optimizer state, and various buffers. Following the reviewer's suggestion, we used HuggingFace's internal memory profiler to measure actual peak GPU memory usage. We've included these findings, along with the adapted modules for all baselines (see **G2** in global response). In summary, SVFT uses ~1.2x more memory than LoRA but is comparable to or less than DoRA.\n\n**W4/Q2:** We appreciate the reviewer pointing out the inconsistency in adapted modules. For SVFT, we initially adapted more modules (QKVUDOG) compared to baselines (QKVUD) due to the lower total trainable parameters. However, for LLaMA-3-8B, adapting QKVUDOG exceeds A100-80GB GPU memory (true for baselines as well). Based on our ablation study in Section 5.3, which showed UDOG modules contributing most to performance gains, we chose to adapt UDOG while staying within memory limits. \nWe agree that this inconsistency could make it challenging to attribute gains specifically to SVFT. Following the reviewer's advice, we re-ran experiments with SVFT adapting only QKVUD, matching the baselines. We've included these new results in a table below, demonstrating that SVFT maintains similar performance gains even in this consistent setting.\n\n---\n**Gemma-2B**\n---\n\n| Method | #Params |Modules| GSM-8K | MATH |\n|:-:|:-:|-|:-:|:-:|\n| LoRA$_{r=4}$|3.28M|Q,K,V,U,D|40.6|14.5|\n| DoRA$_{r=4}$|3.66M|Q,K,V,U,D|41.84|15.04|\n| LoRA$_{r=32}$|26.2M|Q,K,V,U,D|43.06|15.50|\n| DoRA$_{r=16}$|13.5M|Q,K,V,U,D|44.27|16.18|\n| SVFT$^R_{d=12}$|2.98M|Q,K,V,U,D|47.41|**16.72**|\n| SVFT$^R_{d=8}$|3.28M|Q,K,V,U,D,O,G|**47.76**|15.98|\n\n---\n**LLaMA-3-8B**\n---\n| Method | #Params | Modules | GSM-8K | MATH |\n|:-:|:-:|:-:|:-:|:-:|\n| LoRA$_{r=4}$|7.07M|Q,K,V,U,D|69.37|22.90|\n| DoRA$_{r=4}$| 7.86M|Q,K,V,U,D|74.37|24.1|\n| LoRA$_{r=32}$|56.6M|Q,K,V,U,D|**75.89**|24.74|\n| DoRA$_{r=16}$|29.1M|Q,K,V,U,D|75.66|24.72|\n| SVFT$^R_{d=24}$|15.98M|Q,K,V,U,D|75.32|**25.08**|\n| SVFT$^R_{d=16}$|17.26M|U,D,O,G|75.43|24.44|\n---\n\n**W5/Q4:**\n\n**1.** We selected different datasets for ViT-B and ViT-L in Table 4 for two primary reasons:\n   i) To provide a balanced evaluation, we ran ViT-B on relatively simpler datasets (CIFAR100, Flowers102), while testing ViT-L on more complex ones (Food101 and Resisc45).\n   ii) To enhance the table's readability and prevent it from becoming overly large. The complete version of this table, including all dataset-model combinations, is available in Table 9 of the appendix.\n\n**2.** We acknowledge the reviewer's observation that SVFT variants may not consistently outperform LoRA/DoRA across all benchmarks, as evidenced by the Resisc-45 results with ViT-B. However, it's important to note that these results are still comparable to full fine-tuning outcomes. We will update the main paper to reflect this nuanced finding. Additionally, it's worth considering that in vision benchmarks, the classifier head is fully tunable, which may increase the potential for overfitting.\n\n**3, 4.** To clarify our methodology:\n   i) We adopted the target modules for adaptation from [1] and the data split setup from [2]. While [1] uses VTAB-1K for vision experiments, they don't release data splits or sources. Upon careful examination of [3], we discovered that unlike other datasets where evaluation is done on the test split, [3] evaluates Resisc45 on all remaining samples. For consistency, we evaluate on the test split for all vision datasets in our study.\n   ii) To address the second point, we have re-run all vision experiments, adapting only the QV modules – please refer to **G3** in global response for table, which we will include in the revised paper. In short, SVFT offers competitive performance. Adapting QV modules for these vision tasks seems sufficient for all baselines.\n\n**Discrepancy in Food101:** Table 5 in [3] shows ViT-B significantly outperforming ViT-L on Food101, contrary to trends in other datasets. We were unable to reproduce these results for ViT-B, even with full fine-tuning.\n\n**Q1:** See Appendix C.5 L451, or W2.2\\\n**Q3:** For Math reasoning, we fine-tune on the MetaMathQA-40K (L181) and evaluate on GSM-8K, MATH following [1]. For Commonsense reasoning, We follow the setting outlined in prior work [2] i.e. all the training splits from the respective datasets are amalgamated into the training set (L184). For baselines, we use the recommended hyperparams from the respective papers, and for SVFT, we do not perform hyperparam tuning and select params outlined in Appendix C.5.\n\n**References** \\\n[1] Liu et al. Parameter-efficient orthogonal finetuning via butterfly factorization. ICLR 2024. \\\n[2] Liu et al. Dora: Weight-decomposed low-rank adaptation. ICML 2024. \\\n[3] Kopiczko et. al: VeRA: Vector-based Random Matrix Adaptation. ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback. We address overlapping concerns below.

---

**G1: Missing Discussion of Related Work.**\
SVF, SVDiff, SAM-PARSER are equivalent to SVFT$^{P}$ barring minor implementation differences. We compare these methods against SVFT$^{P}$ on GSM-8K and Math benchmarks. Results are shown in the table below and observe similar performance. For completeness, we will revise our paper to include these results. 

---

### **SVF v/s SVFT$^{P}$ on Gemma-2B**
---
| Baseline | #Trainable Params | Target Modules | GSM-8K | MATH |
|:-:|:-:|:-:|:-:|:-:|
| SVF | 120K | Q, K, V, U, D | 36.39 | 13.96 |
| SVFT$^P$ | 120K | Q, K, V, U, D | 36.39 | 13.86 |
| SVF | 194K | Q, K, V, U, D, O, G | 39.12 | 14.02 |
| SVFT$^P$ | 194K | Q, K, V, U, D, O, G | 40.34 | 14.38 |
| SVFT$^R_{d=16}$ | 6.35M | Q, K, V, U, D, O, G| **50.03** | **15.56** |
---

**G2: Memory usage during Training.**\
We acknowledge that while SVFT reduces trainable parameters, it increases overall GPU memory usage compared to LoRA. However, reducing trainable parameters decreases the memory required for gradients, activations, optimizer state, and various buffers. To substantiate this, we used HuggingFace's internal memory profiler to measure peak GPU memory usage. We have included these findings along with the adapted modules for all baselines in the table below. In summary, SVFT uses ~1.2x more memory than LoRA but is comparable to or less than DoRA. Also note that since SVFT requires less trainable parameters than other methods, the cost of storing U and V can be amortized, and can eventually be cheaper than LoRA/DoRA, especially when the number of parallel adaptations is in millions [1].

---

### **Gemma-2B**

---

| Method       | Target Modules | #Trainable Params | GPU Mem (GB)  | GSM-8K | MATH  |
|:-:|:-:|:-:|:-:|:-:|:-:|
| LoRA$_{r=4}$      | Q, K, V, U, D   | 3.28M  | 18.8839 | 40.63 | 14.5  |
| DoRA$_{r=4}$      | Q, K, V, U, D          | 3.66M  | 24.5798 | 41.84 | 15.04 |
| LoRA$_{r=32}$     | Q, K, V, U, D          | 26.2M             | 19.0558 | 43.06 | 15.5  |
| DoRA$_{r=16}$     | Q, K, V, U, D          | 13.5M             | 24.6430 | 44.27 | 16.18 |
| SVFT$^R_{d=12}$  | Q, K, V, U, D          | 2.98M             | 20.4988 | 47.61 | **16.72** |
| SVFT$^P$       | Q, K, V, U, D, O, G        | 194K              | 21.8984 | 40.34 | 14.38 |
| SVFT$^R_{d=8}$   | Q, K, V, U, D, O, G         | 3.28M             | 22.0224 | 47.76 | 15.98 |
| SVFT$^R_{d=16}$ | Q, K, V, U, D, O, G         | 6.35M             | 22.1465 | **50.03** | 15.56 |

---

### **Gemma-7B**

---

| Method      | Target Modules | #Trainable Params | GPU Mem (GB)  | GSM-8K | MATH  |
|:-:|:-:|:-:|:-:|:-:|:-:|
| LoRA$_{r=4}$     | Q, K, V, U, D          | 8.60M  | 63.5711 | 73.76 | 28.3  |
| DoRA$_{r=4}$     | Q, K, V, U, D         | 9.72M             | 78.7007 | 75.43 | 28.46 |
| LoRA$_{r=32}$    | Q, K, V, U, D        | 68.8M   | 64.2403 | 76.57 | 29.34 |
| DoRA$_{r=16}$    | Q, K, V, U, D          | 35.5M  | 78.9913 | 74.52 | 29.84 |
| SVFT$^P$      | Q, K, V, U, D, O, G         | 429K | 76.2630 | 73.5  | 27.3  |
| SVFT$^R_{d=8}$  | Q, K, V, U, D, O, G        | 9.80M   | 76.6506 | 73.62 | 28.36 |
| SVFT$^R_{d=16}$ | Q, K, V, U, D, O, G         | 19.8M    | 77.0363 | **76.81** | **29.98** |

---

### **Llama-3-8B**

---

| Method      | Target Modules | #Trainable Params | GPU Mem (GB)   | GSM-8K | MATH  |
|:-:|:-:|:-:|:-:|:-:|:-:|
| LoRA$_{r=1}$     | Q, K, V, U, D  | 1.77M | 57.8217 | 68.84 | 20.94 |
| DoRA$_{r=1}$     | Q, K, V, U, D | 2.56M  | 70.1676 | 68.30 | 21.96 |
| LoRA$_{r=32}$    | Q, K, V, U, D | 56.6M  | 58.4089 | 75.89 | 24.74 |
| DoRA$_{r=16}$    | Q, K, V, U, D | 29.1M  | 70.4383 | 75.66 | 24.72 |
| SVFT$^B_{d=24}$ | Q, K, V, U, D | 15.98M | 71.5224 | 75.32 | **25.08** |
| SVFT$^R_{d=12}$ | U, D, O, G | 13.1M | 70.3681 | **75.90** | 24.22 |
| SVFT$^P$      | Q, K, V, U, D, O, G    | 483K  | 73.9495 | 69.22 | 20.44 |
| SVFT$^R_{d=12}$ | Q, K, V, U, D, O, G | 15.9M | 71.5224 | 73.99 | **25.08** |

---

**G3: Adapting QV modules for Vision Benchmarks.**\
Following a similar setup in [2], we have re-run all vision experiments, adapting only the QV modules. In short, SVFT offers competitive performance. Adapting QV modules for these vision tasks seems sufficient for all baselines.

---

| Method        | #Trainable Params | CIFAR100 | Food101 | Flowers102 | RESISC45 |
|:-:|:-:|:-:|:-:|:-:|:-:|
| **ViT Base**  |    |          |         |            |          |
| Head          | -                  | 78.58    | 75.14   | 98.71      | 64.42    |
| Full          | 85.8M         | 85.02    | 75.41   | 99.16      | 75.3   |
| LoRA$_{r=8}$      | 294.9K            | 85.65| 76.13 | 99.14     | 74.01    |
| VeRA$_{r=256}$    | 24.6K             | 84       | 74.02   | 99.1       | 71.86    |
| SVFT$^P$        | 18.5K             | 83.78    | 74.43   | 98.99      |  70.55  |
| SVFT$^R_{d=4}$    | 165.4K            | 84.85    | 76.45   | 99.17      | 74.53    |
| SVFT$^B_{d=4}$    | 165.4K            | 84.65    | 76.51   | 99.21 | 75.12    |
| **ViT Large** |                    |          |         |            |          |
| Head          | -                  | 79.14    | 75.66   | 98.89      | 64.99    |
| Full          | 303.3M        | 87.37    | 78.67   | 98.88      | 80.17|
| LoRA$_{r=8}$      | 786.4K            | 87.36    | 78.95| 99.24  | 79.55    |
| VeRA$_{r=256}$    | 61.4K             | 87.55| 77.87   | 99.27  | 75.92    |
| SVFT$^P$        | 49.2K             | 86.67    | 77.47   | 99.09      |  73.52   |
| SVFT$^R_{d=4}$    | 441.5K            | 87.05    | 78.95| 99.23     | 78.9     |
| SVFT$^B_{d=4}$    | 441.5K            | 86.95    | 78.85   | 99.24      | 78.93    |
---

**References** \
[1] He, Xu Owen. "Mixture of A Million Experts." arXiv preprint arXiv:2407.04153 (2024)\
[2] Dawid Jan Kopiczko et al. VeRA: Vector-based Random Matrix Adaptation. ICLR 2024

Meta-review: This paper presents a parameter-efficient fine-tuning (PEFT) method based on singular value decomposition (SVD) which highlights the importance of learning off-diagonal coefficients. The majority of consensus from the reviewers was that the effectiveness of finetuning off-diagonal coefficients provides a new sight in the literature of PEFT.  Reviewers (DEin, BEJi and Cyjn) are positive, while Reviewer x5z5 recommended to reject it. During the reviewer-author discussion period, there are new experimental results which seem to show the advantages of the proposed method, as confirmed by reviewers.  During the reviewer-AC discussion, there are informative discussions between reviewers in term of the novelty of the proposed method. The key discussion point is whether it is possible, and how to, design the sparsity pattern for the off-diagonal coefficients for downstream tasks in a more elegant way to go beyond the proposed three heuristic methods in the submission. Both the reviewers and the meta-reviewer agree it is an important aspect.  Overall, this meta-reviewer concurs with the overall positive feedbacks in terms of the effectiveness of the proposed method, and recommends to **accept** this paper. 

The authors are encouraged to carefully revise the paper due to **the significant updates in the rebuttal and discussion**. The authors are also encouraged to discuss the design and/or the learning of the sparsity pattern in the revision.
Decision: Accept (poster)