Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning
Authors: Chongjie Si
Zhiyi Shi
Shifan Zhang
Xiaokang Yang
Hanspeter Pfister
Wei Shen
Venue: ICLR 2025 Poster (2025)
Keywords: parameter efficient fine-tuning
low-rank adaptation
task-specific directions
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: It provides a clear definition of task-specific directions and propose a method unleashing the power of these directions.

Abstract: Large language models demonstrate impressive performance on downstream tasks, yet requiring extensive resource consumption when fully fine-tuning all parameters. To mitigate this, Parameter Efficient Fine-Tuning (PEFT) strategies, such as LoRA, have been developed. 
In this paper, we delve into the concept of task-specific directions (TSDs)—critical for transitioning large models from pretrained states to task-specific enhancements in PEFT. We propose a framework to clearly define these directions and explore their properties, and practical utilization challenges. We then introduce a novel approach, LoRA-Dash, which aims to maximize the impact of TSDs during the fine-tuning process, thereby enhancing model performance on targeted tasks. Extensive experiments have conclusively demonstrated the effectiveness of LoRA-Dash, and in-depth analyses further reveal the underlying mechanisms of LoRA-Dash.

Published as a conference paper at ICLR 2025
UNLEASHING THE POWER OF TASK- DIREC- TIONS IN PARAMETER EFFICIENT FINE-TUNING
Chongjie Si
Zhiyi Shi
Shifan Zhang'
Xiaokang Yang'
Hanspeter P
Wei Shen
Moo Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University
Harvard University
chongjiesi@ss iss @sjuu. edu.
wei. shen& s j.t.uedu.ccuco.....
ABSTRACT
Large language models demonstrate impressive performance on downstream tasks, yet requiring extensive resource consumption when fully fine-tuning all pa- rameters. To mitigate this, Parameter Efficient Fine-Tuning (PEFT) strategies. such as LoRA, have been developed. In this paper, we delve into the concept of task-specific directions (TSDs) critical for transitioning large models from pre- trained states to task-specific enhancements in PEFT. We propose a framework to clearly define these directions and explore their properties, and practical utiliza- tion challenges. We then introduce novel approach, LoRA-Dash, which aims to maximize the impact of TSDs during the fine-tuning process, thereby enhancing model performance targeted tasks. Extensive experiments have conclusively demonstrated the effectiveness of LoRA-Dash, and in-depth analyses further re- veal the underlying mechanisms of LoRA-Dash.
INTRODUCTION
Large language models (LLMs) Liu et al. (2019): Qin et al. (2023) : He et al. 2021b); Touvron et al. (2023a); Devlin et al. (2018) Touvron et al. (003b)) Al @ Meta (2024) have exhibited supe-
rior performance in a variety of natural language processing (NLP) tasks, including commonsense reasoning SSap al. (2020) and natural language understanding Wang al. (20T8). Despite the ad- vantages, the practice of fully fine-tuning these models, which often comprise hundreds of millions hundreds of billions of parameters, requires substantial computational resources and incurs sig- nificant memory costs Touvron et al. (2023a); | Ma et al. (2024); Raffel et al. (2020). This extensive resource requirement restricts the practical deployment LLMs across diverse scenarios.
To address this issue, parameter efficient fine-tuning (PEFT) Zhang et al. (2022); Hu et al. (202 I); Si et al.) (2024a); Houlsby et al. 2019); Zaken et al. (202T): Si et al. (2024b) has been a focal point of recent advancements in adapting LLMs with minimal computational and GPU memory costs. It aims to optimize the number of adjustable parameters to improve task performance without chang- ing the original structure of the model. Among various PEFT methods, LoRA (low-rank adaptation) Hu et al. (202T) stands out. Since the effectiveness of fine-tuning can be ascribed to the reparam- eterization of "low- lite-diimnsiooall manifold" [Aghajanyan al. (2020): Li et al. (2018). it suggests that the changes to linear model weights can be characterized by low-rank structure. Specifically, for each weight matrix W Raxm LoRA models its changes A AW& E B using low-rank matrices, A E Raxr and B E Brxm, with < (n, m) to achieve parameter efficiency. The change AW is added the pre-trained weights W, and only A and B are updated during training. Due to LoRA's flexibility, its applications have become widespread Rombach et al. (022 Ma et al. (2024).
LoRA underscores that AW enhances directions in W that, although not pivotal for pretraining, are indispensable for specific downstream tasks, referring to these "task-specific directions". Indeed, we suggest that these directions serve as intuitive representations of the low-dimensional manifold, which indicates the significance of task task-specific directions for the success of fine-tuning How- ever, despite that LoRA refers to or even points out the significance of these directions, no existing
"Equal contributions.
1 "Corrssonniigg author.

---

Published as a conference paper at ICLR 2025
studies have concretely defined or systematically pinpointed them (even in those studies on low- dimensional manifold), let alone outlined strategies for their effective utilization. This not only suppresses LoRA's performance potential but also hinders deeper understanding of the fundamen- tal mechanisms underlying fine-tuning.
The chief goal of this paper is bridge this gap and further unleash the potential of task-specific di- rections in fine-tuning, we accomplished two key advancements, which constitute the contributions of this paper[ First, we propose framework to provide precise definition of task-specific directions, exploring the properties and practical utilization challenges of these directions. Sec- ond, building on the comprehensive analysis, we introduce a novel method, LoRA-Dash, which identifies task-specific directions during training and proactively utilizes their influence. Extensive experiments demonstrate the effectiveness of LoRA-Dash, and detailed analytical insights provide us a more comprehensive understanding of LoRA-Dash.
2 BACKGROUND
2.1 PARAMETER EFFICIENT FINE-TUNING
LFMs' huge complexity and computational demands with billions of parameters create significant challenges for adapting them to specific downstream tasks Xu et al.] (2023): Han et al. (2024). Pa- rameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the parameters and memory needed for adaptation levels similar to full fine-tuning Si et al. (2024b); Ding et al. (2023). Current PEFT methods can be roughly divided into three distinct categories Si et al. (2024b); Liu al. (2024): adapter- bbasee Houlsby et al. ( 2011)][Chen et al.] (2022): [uo et al. (2023) He et al. (2021a); Mahabadi et al. (202))
variety of downstream tasks while maintaining performance
Karimi Mahabadi et al. (2021), prompt-based Lester et al. 2021 Razdaibiediia et (2023 y: Wang
et alll:2222222 (2023); | Shi & Lipani (2023): Fischer et al. (2024), and low-rank matrix decomposition-
(202I); ): Qiu al. (2023); Renduchintala et al. Kopiczko et al. (2023) YEH et al. (2023 y Zhang et al. (022). The first category of methods enhances performance by integrating linear modules with existing layers, either sequentially or con- currently. The second category focuses on refining trainable vectors by adding soft tokens (prompts) to the initial input. The third type, introduced by LoRA Hu et al. (202T), employs low-rank decom- position to model weight changes during fine-tuning and can be combined with pre-trained weights.
Hu al. (2021); Liu et al. (2024); ); Hycon- Woo
2.2
LORA: LOW-RANK ADAPTATION
Based on findings that updates to the weights typically exhibit a low intrinsic rank Agghjaanyan et al.
y Li et al. 20188) LoRA models the changes AW E Rexm for each layer's weights R T as AAW AB, where A E Raxr, B E Rrxm with the rank r & {n, m) to achieve parameter efficiency. For the original output h Wx, the modified forward pass is
W
h h Wx + AWx = (W + AB)x.
with Kaiming distribution He let al. (2013), and matrix B is initialized with zeros, which sets the initial AW to zero at beginning. During training. LoRA only updates the low-rank matrices A and B with W being frozen. During inference, the low-rank matrices are integrated into the W, resulting in no additional costs.
In the training initialization for LoRA, matrix A is commonly
2.3 RETHINKING
DIRECTIONS" IN LORA
LoRA highlights that AW encompasses directions within W that are deemed insignificant for initial tasks but are crucial for specific downstream tasks, which LoRA refers to "task-specific directions" (TSDs, plural). Indeed, we suggest that "TSDs" are intuitive depictions of the low- dimensional manifold AAhaaaayyan et al.] (2020); Li al. (2018). Given that the most impactful transformations occur within the low-dimensional manifold during fine-tuning, TSDs are also sig- nificant in PEFT. However, the description of TSDs in LoRA, as shown in Sec. B.I presents several contradictions that contribute to confusion. This confusion complicates the identification of TSDs and the subsequent utilization of them.
Please refer Sec. for more details on the structure and contributions of this paper.
2

---

Published as a conference paper at ICLR 2025
We believe that these issues are mainly due to unclear definition of task-specific directions. Con- sequently, to clearly define TSDs and enhance their practical application, we have opted to discard all previous definitions utilized by LoRA. We aim to undertake comprehensive reconstruction of the entire framework from seratch, ensuring that TSDs are not only well-defined but also effectively utilized in further applications.
3
BUILD FRAMEWORK: UNDERSTANDING TASK-SPECIFIC DIRECTIONS
3.1 PRELIMINARIES
Considering a matrix A E RR where n. < m, can be decomposed using SVD as A UEVT. Here, Σ = (diag("1........ on) ) is the diagonal matrix of singular values, with U and V being the left and right singular vectors, respectively. This decomposition can be further expressed as
A a...v.
Since } forms the orthogonal bases for F and {v:} for R", the matrix space R= can be spanned by the bases [u,v]). ). which are also orthogonal. Thus, matrix can be seen as a matrix in subspace of the original gaxm space, spanned by set of linearly independent bases {u.v... Definition 1. For a matrix A with its left and right singular vectors represented by matrices U and V, respectively, the bases of A are defined as follows:
Core Bases: The core bases of the matrix A are defined {u.y.), where each u.y, is rank-one matrix formed by the outer product of singular vectors u, and v.
Global Bases: The global bases the matrix A are defined as {u.v] for all i, j. covering all combinations of the left and right singular vectors.
Definition 2. The direction of a matrix A E Ruxm (where n < m) is defined based on its global bases, using an expanded set of its singular values padded with zeros, specifically as 00.,0,...,,0, 02,00
,n....,000 E Rnn,, i.e,, flattened E by rows.
Note that any global basis can be regarded as a direction of unit, since its direction is a one-hot vector. For simplicity in further discussions, we will not diffcrentiate between global bases (core bases) and global directions (core directions).
3.2 WHAT ARE TASK-SPECIFIC DIRECTIONS?
We start with the fact that for any specific task, there exists an optimal matrix W E Ruxm within the matrix space 2=xm Si et al. 2024b). For a pretrained matrix W, its optimal alteration for this specific task is defined as A W and the directions of W. Since AW" and W are established their respective bases, we initially project both A and W onto the global bases of W, capturing their directions.
W W. In PEFT, we only possess the information of W
Definition 3. Define (.).)) as projection operator that projects a direction in one coordinate system onto another coordinate system. Specifically, (A)
(p11,...,PPun PPm)) E RR is the projection of the direction of matrix A E Roxm onto the global bases of another matrix W R Ruxm, Pij = i.! Av, where u, and v, are the left and right singular vectors of W, respectively.
Established on the global bases of W. Iw (W*) represents the direction that W should evolve. W itself can only utilize up to n bases (i.e., its core bases), meaning that it can at most alter n values of its direction. Thus, we only focus tails, please refer to Sec. B.3). During the transformation, not all core directions' coordinate values change to the same extent. Given the diversity of downstream tasks and the variability in W* for different tasks, certain core directions of W undergo significant changes while others may experi- ence only minimal alterations. Specifically, the change rate ò, of the coordinate value corresponding to the i-th core direction V.v.
the changes related its
directions (for more de-
of W is represented as:
Iw (W*)
G;
u. W'v; u. Wv:
6:
(3)
0 +
0 +
0 +E
σ +
3

---

Published as a conference paper at ICLR 2025
Here, 10-6 is constant to prevent singular values from being d; is the i-th singular value the matrix W, and 6, represents the change rate of the coordinate value required in the i-th core direction V for transformation from W to Nw (W*), specifically quantifies the extent of adaptation needed for a particular task. Hence, we finally define the TSD) as:
Definition 4. For a specific task and a pre-trained weight W, considering the optimal weights for this task as W*, the task-specific directions (TSDs) of this task W W's core directions whose coordinate values exhibit significantly higher change rates ó through the alteration from W W
Inspired by Eq. 33 and Definition is obvious that ô is directly related to the projection of AW* We are now also equipped to revisit and more precisely articulate LoRA's conclusions on TSDs:
TSDs are the subset of core directions of pretrained weights W. They are specific to each task, meaning that they vary from one task to another but remain fixed for given task.
Core directions associated with larger singular values are less likely TSDs, the change rates of their coordinate values are typically smaller than those associ- ated with smaller singular values. reasonable since that larger singular values usually encapsulate more generalized information that the model acquired during pretraining.
identified as
We encourage readers to refer to Secs.
B.2 B.3
for more detailed reasons on Definition 4
0.05
0.03
5.04
Change Rate
0.02
0.6
3.02
0.2
2002
2001
3001
4002
4020 6095
1002
2001
300L
4002
Index
Index
The change rates of W's core bases, based on W vary significantly across each basis, and the bases with higher change rates tend be concentrated towards the end. Middle: The change rates for the directions corresponding to the smallest 10 singular values W. The directions associated with the smallest singular values do not always exhibit the highest change rates. Right: After sorting the change rates from highest to lowest, is evident that only a few directions have significant change rates, while most exhibit very low change rates. The weights taken from the 16th layer of LLaMA-7B, and the change rates are scaled (Sec. D.2 for more details).
Figure 1: Left:
3.3
WHAT ARE THE PROPERTIES OF TASK-SPECIFIC DIRECTIONS?
To further explore the properties of TSDs, we fully fine-tune LLaMA-77 7B Touvron et al. (2023a) on commonsense reasoning tasks, and assume that W* can be obtained through fully fine-tuning. After acquiring the fully fine-tuned weights W* we compute AW* W W, and based on A AW* calculate the change rates of core directions of W. The results shown in Fig, where can draw several conclusions (Please refer to Sec. D.2 for more details):
TSD predominantly corresponds to core directions associated with smaller singular values of W, though not the smallest.
TSD encompasses only few directions where substantial change rates occur; other core directions exhibit minimal or negligible change rates.
3.4 CHALLENGES AND EXPLORATIONS: USING TSD IS NOT EASY EXPECTED
In downstream tasks, TSDs represent the directions of greatest change, making them critical targets for fine-tuning to adapt pre-trae pre-trained models effectively to new tasks. Although we have thoroughly explored the definition and properties of TSDs, significant challenge is that both A A and W
4

---

Published as a conference paper at ICLR 2025
unknown before fine-tuning, indicating that utilizing TSD information beforehand in practical fine-tuning scenarios is nearly impossible.
Despite the apparent challenges, place confidence in the potential of A W. We hypothesized that the core directions with the highest change rates predicted by AW from LoRA are strongly associated with TSDs. To test this hypothesis, conducted extensive experiments. We strongly encourage readers to refer to Sec. [33] for detailed experimental setups. The results are shown in the Figs. and 8 where we observe significant overlap between the predicted directions and the actual TSD, highlighting their strong correlation. leads to an important conclusion:
Observation 1. Irrespective of the rank setting in LoRA, the training step. the specific layer within the model, LoRA's 's A AW consistently captures the information of the task-specific directions.
This is an exhilarating conclusion, suggesting that even without prior knowledge of TSDs, we can still capture their crucial information through the W obtained during LoRA training!
Query
Key
Value
0.90
1.0
==32
==
16
1=
0.85
Accuracy
0.80
0.70
10000
20000
30000
2
12
17
22
27
32
Step
Layer
Figure 2: We track the accuracy of predicted directions every 100 training steps in the query, key and value layers of the LLaMA-7B during LoRA fine-tuning, analyzing how well the continuous updated AW captures TSDs. Left: We compute the average accuracy across all query, key, and value layers, showing the model's ability retain task-specific knowledge for each training step. Across various rank settings of LoRA, these accuracy rates consistently exceed 0.75, indicating that AW reliably captures and integrates TSD information. Right: For a rank setting compute the average accuracy across all steps for each query, key and value layers, revealing their sensitivity to TSDs. The majority layers maintain an average accuracy above 0.75, showing the robustness to capture TSD information.
= 32, we
LORA-DASH: UNLEASH THE POWER OF TASK-SPECIFIC DIRECTIONS
To further unleash the power of TSDs in downstream tasks, this section introduces novel method: LoRA-Dash LoRA-Dash consists of two principle phases: the "pre-launch phase", where TSDs are identified, and the "dash phase", which unleash the potential of theses identified TSDs.
4.1 PRE-LAUNCH PHASE
In this phase, LoRA-Dash initially trains the matrices A and B (i.e., LoRA's A W) to capture the information of TSDs. To expedite the utilization of TSD information and based on Proposition I we posit that there is a predefined number of training steps, after which Á can consistently capture TSD information at any stage of training. We set / to 100 suggested in Sec. 3.4
Subsequently, we project AW onto the core directions W calculate the change rates:
6, = u. u.AWW: o, +
(4)
where u, v., and o; represent the i-th left and right singular vectors and singular value of W, respectively. We identify the top core directions with the highest change rates, setting s as another hyper-parameter, which is recommended be 8 suggested in Sec. 3.4 We denote ú, v. for
5

---

Published as a conference paper at ICLR 2025
i == 1,2,.... the identified "TSDs" for dash, To differentiate from the true TSDs derived from
W (or theoretically defined), we label these directions as "launched TSDs" (LTSDs)
4.2 DASH PHASE
To leverage the identified LTSDs, we directly learn the changes in their coordinates, denoted Ao; for the i-th u.v.. to further fine-tune the model for downstream tasks. Aay is initialized as zero. Mathematically, this adjustment represented by
Σ Ao.u.!!
(5)
Ultimately, the updated weight matrix is given by:
W + AWW. all W AWA AB AW dash W AB Σ Ao.u.
This equation forms the basis of LoRA-Dash, which aims to fully utilize the power of LTSDs derived from LoRA. During training, A, B, and Ao, are continuously updated. For more details on LoRA- Dash, please refer to Sec.
Pre-Launch Phase
Dash Phase
Direction
AW
Direction
A
Aack
aaw,
W
4W
4W
W
of n/aWs,
o/AWy,
Core Direction
Core Direction Identified TSD
Identified
Figure 3: Framework of LoRA-Dash. LoRA-Dash comprises two phases: the pre-launch phase for identifying TSDs, and the dash phase to maximize their potential.
HOW MUCH CAN TSD BOOST THE PERFORMANCE OF LORA-DASH?
To explore the performance gains LoRA-Dash, both numerical and visual comparisons are em- ployed. The numerical experiments focus commonsense reasoning and natural language un- derstanding tasks, assessing LoRA-Dash's performance improvements in standard natural language processing metrics. Visual comparisons focuses subject-driven generation task, involving using a text-to-image diffusion model to learn specific concepts (i.e., task-specific concept), providing clear, qualitative and intuitive understanding of the impact of TSDs performance. The implemen- tation details are shown in Sec. D.I The experiment details are shown in Secs. D.4.66 D.6
5.1 NUMERICAL RESULTS
The numerical results, as shown Tables. 12 demonstrate that LoRA-Dash significantly outper- forms LoRA across all tasks and models. This distinction is pronounced that further elaboration on performance gains might be unnecessary. Therefore, we instead focus on several key insights derived beyond mere performance gains.
(When fine-tuning LLaMA-7B) LoRA shows enhanced perfor- mance with increased parameter budget, indicating its dependence larger rank sizes. Con- versely, LoRA-Dash maintains strong performance even under limited parameter conditions, which
Robustness to Parameter Budget:
As shown in Secs. 3.4 and 6.1 LTSDs and TSDs are heavily overlapped. Given that LTSD closely represents TSD, unless explicill emphasized otherwise, we treat LTSD as equivalent to TSD for analytical purposes.

---

Published as a conference paper at ICLR 2025
Table 1: Results commonsense reasoning tasks. We fine-tune LLaMA-7B Touvron et al. (2023a)
LLaMA2-7B Touvron et al. (2023b) and LLaMA3-8D AI@ Meta (2024) on this task.
Model
Method
Params( %)
BoolQ
PIQA
SIQA
HellaS,
WinoG.
ARC-e ARC-c
OBQA
Avg.
ChatGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
Fully FT
100
69.9
84.2
78.9
92.3
83.3
86.6
72.8
83.4
81.4
LoRA,
0.10
2.3
46.1
18.3
T9.7
55.2
65.4
51.9
57.0
39.5
LoR--Dash
0.10
65.2
79.9
78.3
82.8
77.
78.6
65.4
78.4
75.7
LoRA,
0.21
31.3
57.0
44.0
11.8
43.3
45.7
39.2
53.8
40.7
LoRA-Dase
0.21
69.8
81.1
77.3
85.1
81.1
77.2
64.
79.6
76.9
LLaMA-7B
LoRA,=10
0.42
69.9
77.8
75.1
72.1
55.8
77.1
62.2
78.0
70.9
LoRA-Dash
0.42
66.9
80.2
77.8
78.8
79.2
78.0
61.9
77.4
75.0
L22
0.83
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
LoRA-Dash
0.83
69.9
82.8
78.6
84.9
81.6
82.3
66.5
80.8
78.4
L6RA,=64
1.66
66.7
79.1
75.7
17.6
78.8
73.3
59.6
75.2
65.8
LoRA-Dash
1.66
69.6
79.5
76.0
82.8
75.8
81.5
64.7
81.0
76.4
Fully
72.2
84.9
80.9
93.1
84.7
87.5
74.2
82.8
LoRA,=18
0.41
71.7
81.6
79.5
89.5
81.9
82.9
67.9
79.6
79.3
LLaMA2-7B
LoRA-Dash
0.41
70.9
82.2
80.5
90.2
80.1
83.5
68.9
80.8
79.6
LoRA,=32
0.82
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
LoRA-Dash
0.82
71.0
75.7
79.3
91.1
78.6
84.2
69.8
78.8
78.6
Fully
75.3
89.9
81.5
95.8
87.6
91.6
79.3
87.4
LoRA,=14
0.35
72.3
86.7
79.3
93.5
84.8
87.7
75.7
82.8
82.8
LLaMA3-8B
LoRA-Dash
0.35
74.8
88.0
80.6
95.2
85.6
89.0
77.4
84.8
84.4
LoRA, 0.70
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
LoRA-Dash
0.70
75.3
88.5
80.2
95.7
86.8
90.7
80.2
85.6
85.4
Table 2: Results with DeBERTaV3 fine-tuned on GLUE development set. "FT" represents fully fine- tuning, and "Base" and "Large" represent DeBERTa and DeBERTaVV3-argee respectively.
MNLI
SST-2
CoLAA
QQP
QNLI
RTE
MRPC
STS-B
All
Method
Params( %
Acc
Acc
Mcc
Acc
Acc
Acc
Acc
Corr
Avg.
Base(FT)
100%
89,90
95.63
69.19
91.87
94.03
83.75
90.20
91.60
88.27
LoRA,-2
0.18%
90.03
93.92
69.15
90.61
93.37
87.01
90.19
90.75
88.13
LoRA-Dash
0.18%
90.14
95.42
72.41
91.65
94.36
89.89
91.67
91.64
89.65
LoRA,
0.72%
89.80
93.69
69.30
91.78
92.97
86.28
90.68
91.62
88.27
LoRA-Dash
0.72%
90.55
96.99
70.78
92.39
94.22
88.09
91.18
91.91
89.52
Large(FT) 100%
91.81
96.93
75.27
93.01
96.02
92.68
92,20
92.98
91.36
LoRA,
0.20%
91.33
95.87
73.89
91.84
95.14
91.69
90.68
92.85
90.41
LoRA-Dash
0.20%
91.65
96.11
76.11
92.61
95.52
92.78
92.18
91.25
LoRA, 0.80%
91.38
96.33
74.48
92.54
95.48
92.05
91.17
92.92
90.79
LoRA-Dash
0.80%
91.18
96.45
76.88
92.81
95.85
93.51
91.91
92.86
91.43
suggests that TSDs crucial for maximizing fine-tuning efficiency. This highlights its effective- ness in harnessing the full potential of TSDs, even in constrained settings.
Surpassing Fully Fine-Tuning Performance: While LoRA typically falls short of the results achieved by fully fine-tuning, LoRA-Daa in some cases outperforms fully fine-tuning. This in- triguing observation may provide new insights into optimizing training strategies for fine-tuning.
For comparisons with other methods, please refer to Tables. 23 24
5.2
VISUAL RESULTS
The visual comparisons, as shown in Fig. demonstrate the superior fidelity of images generated by LoRA-Dash in aligning with the subjects of the input images compared to those by standard LoRA. For instance, images generated by LoRA of a dog and a vase showed significant deviations from the input, whereas LoRA-Dash' outputs maintained high consistency with the original images.
Moreover, the effectiveness of TSDs particularly evident in the response of LoRA-Dash to given prompts. TSDs effectively encapsulate the specific information of each prompt, enabling LoRA-
7

---

Published as a conference paper at ICLR 2025
LoK
LOR Dash
Lok.
LoRA-Dash
LoR
Imput Images: Clock
Clock on the sea beach
A cube shaped clock
Clock on green grass with sunflowers around
LORA
LoRA--aase
LoRA A
LoRA MARRADDaa Dash
oRA
LoR Dash
Input Images: Dog
Dog on a wooden floor
Dog wearing rainbow scart
Dog on cobblestone street
LORA
Dush.
LOR
LoRA-Dash
with wheat field in the buckground
Input Images: Vase
shiny vase
Inpat Images:
Figure 4: Comparison of generated images from LoRA and LoRA-Dask subject-driven gener- ation task. L-RA-Dash consistently aligns more closely with the subjects in the input images and adheres better the given prompts than LoRA.
Dash to precisely grasp the semantic essence of prompts such as sea beach or wheat field. This capability allows LoRA-Dash to adeptly render complex themes and details within the images.
UNDERSTANDING LORA-DAS
In this section, conduct extensive experiments better understand LoRA-Dash. We first explore the relationships between the LTSDs, the final directions after the dash phase, and TSDs to determine whether LoRA-Dash can amplify the information of TSDs (Sec. (6.1) and, if so, what extent (Sec. 6.2 We then compare the effectiveness of different directions, such as the top singular directions (Sec. 6.3). and investigate the impact of the length of the pre-launch phase and the number of directions in the dash phase on performance (Sec. 6.4]. Finally, we verify whether the information TSDs can help enhance the effectiveness of other methods (Sec. (6.5). We have also provided extensive discussions in the supplementary material, and we strongly encourage readers to consult for a more comprehensive understanding of LoRA-Dash
6.1 How DOES LORA-DASI DETERMINE AND ALIGN WITH TASK-SPECIFIC DIRECTIONS
We examine the alignment between the LTSDs identified during the pre-launch phase, the direc- tions with highest change rates pinpointed by the final trained AW all of LoRA-Dash (denoted as "DTSDs", Delta TSDs), and true TSDs derived from W*. Following the settings in Secs. 3.4 we report the percentage that:
and D.3
How many of the top 4 DTSDs directions are contained in s LTSD We denote this percentage as "DTSDs LTSDs for further convenience.
How many of the top 4 TSDs contained in LTSDs. Denote this as "TSDs n LTSDs"
How many of the top 4 TSDs in DTSDs. Denote this "TSDs n DTSDs".
The first is to explore the consistency between the directions selected during the pre-launch phase of LoRA-Dash and those showing the greatest change after the final training, and the second and third aim to validate the consistency between the directions chosen during the LoRA-Dash pre-launch phase and the final trained with the true TSDs. conducted tests on LLaMA-7B Touvron et al. (003a) and DeBERTaVe V3-ERTaV3-arrt -lrge He et al. (202bb) each with three ranks of LoRA-Dash and the results are shown in Tables. and
Here since
8, the setting equal that Sec. D.3
8

---

Published as a conference paper at ICLR 2025
Table 3: Alignment the directions of LoRA-Dash and TSDs. We report the percentage from the self-attention value (v) projection of the first, middle and last layer of two models. as well as the average percentage of all q. k, and v modules. More results are shown Tables, 21
Model
LLaMA-7B
16)
DeBERTaV3- (r 8)
Layer
First (1st)
Middle (16th) Last (32nd)
All.
First (Ist)
Middle (12nd) Last (24th)
All.
DISDs n LTSDs
0.75
1.00
0.75
0.77
1.00
1.00
1.00
0.98
TSDs n LTSDs
0.75
0.75
1.00
0.74
1.00
1.00
1.00
0.84
TSDs n DTSDs
1.00
0.75
1.00
0.82
1.00
1.00
1.00
0.89
is obvious that LoRA-Dash indeed captures great proportion of TSDs' information. Moreover, can also obtain two significant findings: First, the LTSDs and DTSDs of LoRA-Dash align closely, which indicates that the directions amplified by LoRA-Dash finally are essentially consis- tent with those selected during the pre-launch phase. Therefore. since LTSDs contain substantial information about TSDs, it is not surprising that the final DTSDs consistently include information about the true TSDs. Second, when compared with the results in Fig. where the end of each line indicates the final percentage of TSDs n DTSDs of LoRA, it becomes evident that LoRA-Dash 's trained weights capture a greater proportion of TSD information than vanilla LoRA.
6.2 How MUCH DOES LORA-DASH AMPLIFY TASK-SPECIFIC DIRECTIONS?
Since LoRA-Dash captures the information of TSDs, we further explore to what extent LoRA- Dash can amplify the information of these directions. To quantify this amplification, we define an amplification factor that measures how significantly LoR A-Dsh enhances the task-specific features. The details are shown Sec. D.7
Table 4: Amplification task-specife features of LoRA-Dash We report the amplification factor from the self-attention key (k) projection of the first, middle and last layer of LLaMA-7B, as well as the average of all k, and v modules.
Layer
r'
16
r 32
AW all
AW AW AB
AW dash
AW all
AW AB
AW
AWash dash
AW all
AW, AB
A AW dash
1st
282.15
41.01
275.59
628.60
169.96
577.64
42.00
13.83
36.71
16th
11.15
4.64
9.87
13.07
7.60
11.38
2.16
1.57
1.92
32nd
68.02
7.58
66.81
30.87
11.37
29.00
8.19
1.69
8.42
Avg.
22.85
7.33
21.07
32.12
13.36
28.44
5.28
2.53
4.57
We conduct this experiment on LLaMA-7B with three ranks. We also test the separate impacts of AW AB and AWassh on the amplification. The results shown in Table. 4 demonstrate that LoRA-Dash significantly enhances the features associated with LTSDs under all test conditions. Notably, the feature amplification contributed by AW dash accounts for the majority of the overall enhancement, whereas the amplification from A WW AB is relatively minor in comparison. LoRA- Dash effectively learns and amplifies the majority of TSD information with only parameters (i.e., Ao; in AW Adssh), reducing the learning hurden also allows A WWA AB to potentially focus on learning other significant features, further optimizing the model's 's performance on downstream tasks, We strongly encourage readers to refer to Sec. D.7 for a deeper discussion on this topic.
AW AB during fine-tuning. This efficiency
6.3 THE EFFECTIVENESS TSD COMPARED WITH OTHER CORE DIRECTIONS
We investigated the effects of TSDs during the pre-launch phase compared other locations, such as selecting top, bottom or random core directions. The results when fine-tuning LLaMA3-8B, as shown in Fig. 15 a). clearly indicate that selecting TSDs yields the best performance, followed by dash-botte bottom with top selections performing the worst. This is not difficult to understand: directly dashing TSDs maximizes the ability of W adapt downstream tasks. Since TSDs generally

---

Published as a conference paper at ICLR 2025
located towards the end of the spectrum, choosing directions from the bottom naturally results in better outcomes than selecting from the top. We have a more thorough discussion Sec. E.3
LaRA
1200
Avi
Sullor
Accumes
0EGA
MnC
755
Dircctione for Dosh
Longth of Pre Arre-Auuchhhcc Phoso
Number of Dash Dirctions
Figure 5: Ablation study of LoRA-Dash. Ablation study the directions for dash. b. Ablation study on the length of pre-launch phase t. c. Ablation study on the number of directions s for dash.
6.4 THE INFLUENCE OF THE HYPER- PAPEE-PARAMMTERS LORA-DASH
We evaluate the influence of two hyper-parameter / and in LoRA-Dash when fine-tuning LLaMA- 7B, and the results are illustrated in Fig. (c). can be observed that starting the dash phase after a longer pre-launch period tends to yield slightly inferior results compared to starting it earlier. This could be attributed to the fact that entering the dash phase earlier allows for more extensive utilization of TSDS, potentially enhancing the adaptation to downstream tasks more effectively. Ad- ditionally, a larger value of 8, representing the number of directions used the dash phase, also impacts performance. Based on our analysis in Sec. D.7 suspect that the inclusion of some irrelevant directions might introduce noise into the training process (i.e., destabilizing model con- vergence), leading to diminished performance.
6.5 CAN TSD ENHANCE THE PERFORMANCE OF OTHER METHODS?
To investigate whether TSDs are beneficial for other PEFT methods, we extended our experiments
to include AdaLoRA Zhang et al. (2022) and FLoRA [i al.) (2024a), assessing the impact of leveraging TSDs within these frameworks. The results, as shown in Table. 5 clearly indicate that incorporating TSDs significantly enhances the performance of these methods as well. This under- scores the pivotal role of TSDs in optimizing model behavior for downstream tasks, highlighting their universal applicability across various PEFT strategies.
Table 5: Results of commonsense reasoning tasks when fine-tuning LLaMA-7B.
Method
Params (%)
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-e
OBQA
Avg.
ChaiGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
AdaLoRAyuag
82.2
77.2
78.3
78.2
79.7
77.2
75.5
Adall .RR Dash
0.83
69.4
82.1
77.5
77.9
75.1
80.3
63.8
79.8
75.7
Fl a2
0.83
66.4
81.3
77.1
75.6
77.1
77.2
62.4
77.6
74.3
FLoRA-Dash
81.9
78.0
83.3
79.6
62.7
79.4
7 CONCLUSION
In this paper, we revisit LoRA's exploration of task usee-ficc directions (TSDs), highlighting its in- adequacies in understanding TSDs. We then provide a precise definition of TSD and delve into their properties to better understand their role fine-tuning large language models. Building on this foundational knowledge, we introduce LoRA-I A-Dahh, designed to fully unleash the potential of TSDs. Through comprehensive experiments and in-depth analysis, we demonstrate the significant advan- tages of LoRA-Dash over conventional methods. Our findings not only validate the effectiveness of LoRA-Dart Dash but also highlight the crucial importance of TSDs in achieving superior task task-specific performance. By pushing the boundaries of effffiier-efficient fine-tuning, aim to inspire contin- ued research and development in this vibrant area, transforming practices across diverse applications in natural language processing and beyond.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by NSFC 62322604, NSFC 62176159, and Shanghai Municipal Science and Technology Major Project 2021SHZZDZX002..
REFERENCES
Armen Aghajanyan, Luke Zettlemoee, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning, arXiv preprint arXiv::0012.13255, 2020.
Al@Meta Llama 3 model card. 2024.
URL htttp:://github.ccom//eee 11ama/
11 1lama3/blob.
M/MDEL CARD md
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 78)):1 2009.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity similarity-muliiingeen multilingual and cross- lonsullingual focused evaluation. arXiv preprint arXiv:1708.0000 00055. 2017.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16661 2022.
Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 100::1900..0044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.0000 2018
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 18.00080 2018.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter- fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-25) 235, 2023.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005). 2005.
Mare Fischer, Alexander Bartler, and Bin Yang. Prompt tuning for parameter-efffcient medical image segmentation. Medical Image Analysis, 91:103024, 2024.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 1-9, 2007.
Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy, and Guillaume Rabusseau. Rosa: Random subspace adaptation for efficient fine-tuning, arXiv preprint arXiv:2407.0788 2024.
11

---

Published as a conference paper at ICLR 2025
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Pframeter-effciiet fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpaatrick and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint 10.0:22110.03366, 2021a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 1034, 2015.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style embedding sharing. arXiv preprint 1rX1v:2111.0954,,
pre-training with 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-27999 PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 2023.
Nam Hyeon-Woo Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-eeffcieent federated learning. arXiv preprint 2rXiv:2108.0660 2021.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022- 1035, 2021.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104 10.006691 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint 1xXi::880.0888.. 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0011 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 35:1950-1965, 2022.
Shih-Yang Liu, Chien-Yt Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 000,0030, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 2019.
Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Ron- grong Ji. Towards efficient visual adaption via structural pa-arameterization. arXiv preprint arXiv:2302.08106, 2023.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, (5(1),654, 2024.
12

---

Published as a conference paper at ICLR 2025
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint
arXiv:2106.04489 2021.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.000 029448, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02755 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint crXiv:2302.06411 2023.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 656.:5606005200, 2016.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameter-
ization. arXiv preprint 2rXi::2005.0397, 2023.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter effi- ciency of Iora with weight tying. arXiv preprint arXiv:2311.09578 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer: High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. part III 18, pp. 234 241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning rext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
13

---

Published as a conference paper at ICLR 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- -00,999--,,,0 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint arXiv:1904.09725 2019.
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. Commonsense reasoning for natural language processing. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguisties: Tutorial Abstracts, pp. 27-33, 2020.
Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine- tuning. arXiv preprint arXiv:2309.055 17,, 2023.
Chongjie Si, Xuehui Wang, Xue Yang. Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xi- aokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405. 14739, 2024a.
Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. arXiv preprint arXiv:2407.05417 2024b.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-16622 2013.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLLtttps://wwent. github. www.b//..ooooogg/oooo 9ithub..ccbbloog/gwee2222 qwe.2../gwee22....
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2rXiv:2307.09288 2023b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint ar 1804 07461, 2018
Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient I1m finetuning. arXiv preprint arXiv:2406.00044, 2024.
Yaqing Wang. Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Fred- erick Liu, Yi Liang, Bo Pang. Michael Bendersky, et al. Non-intrusive adaptation: Input- centric parameter-efficient fine-tuning for versatile multimodal modeling. arXiv preprint 1rX1v:2310.12100, 2023.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Xia, Chi, Quoe Le, Denny Zhou, et al. Chain-of thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 25824784-44877, 2022.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017."
Lingling Xu, Haoran Xie. Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. 12148, 2023.
14

---

Published as a conference paper at ICLR 2025
SHIH YING YEH, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong, Navigating eett-to-imaee customization: From lycoris tinettuning to model evaluation. In The Twelfth International Conference Learning Representations, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter--fficient fine-tuning for transformer-based masked language-models. arXiv preprint (rXx::2106.10199 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint vrXiv:/95.00782. 2019.
Fangzhao Zhang and Mert Pilanci. Spectral adapter: Fine-tuning in spectral space. arXiv preprint TrXiv:2405.13952. 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for purameter-efficient fine-tuning. In The Eleventh Inter- national Conference on Learning Representations, 2022.
15

---

Contents
Introduction
2
Background
2.1
Parameter Efficient Fine-tuning
2.2
LoRA: Low-rank Adaptation
2.3
Rethinking "Task-Specite Directions" LoRA
3
Build Framework:
Understanding spec-fc Directions
3.1
Preliminaries
3.2
What are Task Directions?
3.3
What are the Properties of Task-specific Directions?
3.4
Challenges and Explorations: Using TSD is Not as Easy Expected
4
LoRA-Dash: Unleash the Power of Task-specific Directions
Pre-I Launch Phase
4.2
Dash Phase
5
How Much Can TSD Boost the Performance of
5.
Numerical Results
5.2
Visual Results
6
Understanding LoRA-Das
6.1
How Does LoRA- Dash Determine and Align with lask-specific Directions
6.2
How much does LoRA-Dash Amplify -sask-specific Directions?
6.3
The Effectiveness of TSD Compared with Other Core Directions
6.4
The influence of the Hyper-parameters LoRA-Das.
10
6.5
Can TSD Enhance the Performance of Other Methods?
10
7
Conclusion
10
Structure and Contributions of This Paper
17
Details on Task-specific Directions
17
B.1
How Do Task-specific Directions Arise in LoRA?
18
B.2
Why should TSD be Established on W?
18
B.3
Why Must TSD be in the Core Bases Rather than Other Global Bases?
18
Details on LoRA-Dash
19
Algorithm of LoRA A-Dash
19
Matrix Operation of Dash Phase
19
D Details on Experiments
19
D.1
Implementation Details
19
D.2
Details on Fully tuni-t LLaMA in Section 3.3
19
D.3
Details on Fine-tuning LLaMA with LoRA Section
20
D..4
Details on Commonsense Reasoning Task
21
D.5
Details on Natural Language Understanding Task
D.6
Details on Subject-driven Generation Task
24
D.7
Details on the Amplification TSD Sec, 6.2
25
More Experiment Results
27
E.
More Results for Subject-driven Generation Task
27
E,2
More Results for the Alignment on the Directions of LoRA-Dash
27
E.3
More Results for Ablation Study
27
Are TSDs "Task-speeiffi"??
30
E.5
Convergence Curve
31
E,6
Comparison with Other Methods
16

---

Published as a conference paper at ICLR 2025
A STRUCTURE AND CONTRIBUTIONS OF THIS PAPER
The structure this paper is as follows:
We begin by reviewing the description and exploration of TSDs within the LoRA frame- work. Our analysis reveals the significance of the clear definition of TSDs within PEFT (Sec. 2
We propose a novel framework that clearly defines what TSDs are (Sec. 3.2 and further explores their properties (Sec. 3.3). Additionally, we highlight the challenges of applying
TSDs fine-tuning processes (Sec. 3.4
To further unleash the potential of TSDs, propose method, LoRA-Dash, which leverages TSDs to enhance performance on downstream tasks (Sec. 4. We first introduce
how LoRA-Dash identifies TSDs (Sec. 4.1), followed by explanation of how it leverages this information (Sec. (.2).
We conduct various experiments to further analyse the underlying mechanisms of LoRA-
Dash (Sec. 6
The contributions of this paper are as follows:
We propose a framework, which clearly defines TSDs, and offers foundations for further analysis.
We propose novel PEFT method LoRA-Dash which identified TSDs and leverage the potential them to enhance performance on downstream tasks.
B DETAILS ON TASK-SPECIFIC DIRECTIONS
Initialisation
t,
100,
3 FLAG
delta signa (s-Parraeeterreereett (8,1)
deč forward(..):
h (M 48
result. P.1inear weight, bias)
Vanila LoRA
result + dropout (X) lora A lora scaling
10
Pre-launch
12 11 FLAG s t:
FLAG += 1
return
16
Identity TSD based on Change Rates
sigma, svd(weight)
19
projection diag(u. (lora A lora B scaling) 6 largest change rates index caleulate_chann caaitte_hhnggeeeee rate (signa, projection, u tad, bed ufi, top index], vii, sop_e)
indexes
22
25
Dash Phase
Unleash the Power 7SD
if FLAG
28
result +- dxopout(x) (u_ted a diag aaiga é v_tad.
return result
LoRA-Dash
Figure 6: Pseudo codes LoRA-Dash. LoRA is capable of identifying TSDs, but it does not utilize the corresponding information. LoRA-Dash builds on this by actively leveraging the power of these identified TSDs to enhance model performance significantly.
17

---

Published as a conference paper at ICLR 2025
B.1 How Do SASKISPECIHIC DIRECTIONS ARISE IN LORA?
To explore the relationship between the learned AW and the original weights W. LoRA initially applies SVD AW extract its left and right singular vectors. U and V. LoRA first validates that the directions corresponding to the top singular vectors of AW tend to overlap significantly across different ranks (conclusion 1). Subsequently. LoRA projects W onto the r-dimensional subspace defined by AW, calculating UW and its Frobenius norm ||U"WV]l This norm is also computed by replacing the top r singular vectors U and V derived from W or a random matrix for comparison. The results reveal that | WV F when U and V are top singular vectors derived from AW or W is much greater than when they are derived from a random matrix, suggesting a stronger correlation of AW with W than with random matrix. This indicates that the features AW amplifies already present in W.
Additionally, LoRA introduces the "feature amplification factor" to measure the extent of feature enhancement, defined as || ////=//|UTTT WVIl= The factor is significantly higher when U and V are top singular vectors derived from AW compared when they are those derived from W, suggesting that AW only boosts directions that are not emphasized in W. Moreover, a larger 64) yields much lower amplification factor than a smaller r (r 4), implying that the number of "pss-speccfii directions" is small. Through extensive experiments across various ranks and layers, LoRA determines that A W potentially amplifies important directions for specific down- stream tasks that were learned but not emphasized during general pre-training, and refers to them as "task-specific" directions (TSDs).
LoRA has drawn three conclusions related to TSDs:
1. TSDs are the directions of top singular vectors derived from the learned AW AB. TSDs consistently exhibit significant overlap across different r settings used in the learning of AW, where is the rank configuration of AW.
2. TSDs are not the directions of top singular vectors derived from the pretrained weights W.
3. TSDs are certain directions that have already been learned by W but were not emphasized.
However, only the first conclusion of LoRA already presents significant contradictions. For spe- cific task, TSDs should indeed remain consistent and not depend on learned A W. Moreover, re- visiting the first and the third conclusions, could deduce that the top singular directions of AW are among the directions of W. Nonetheless, in practice, the singular directions of A A are un- likely to align exactly with any specific directions in W. As shown in our paper, although LoRA highlights the importance of TSDs for downstream tasks, it struggles with a clear definition of TSDs which leads contradictions, and does effectively outline how they can be leveraged, leading to potential under-utilization in practical applications
For more details what they did and why they drew the conclusions, encourage readers refer to their original paper Hu et al. (202I), specifically Section 7.
B.2 WHY SHOULD TSD BE ESTABLISHED ON W?
Indeed, the most accurate direction for TSD should be the direction of A AW*. However, since we do not have access to A A W' during fine-tuning, only have information about W. Therefore, we establish TSD within the coordinate system based on the directions of W.
B.3 WHY MUST TSD BE IN THE CORE BASES RATHER THAN OTHER GLOBAL BASES?
Task- specific directions are defined as a subset of the core bases. However, why can't they be defined as a set of bases from the global bases of W instead? Indeed, other global bases might also experience significant coordinate changes. However, the primary reason for not considering these bases is the difficulty in defining them.
Firstly, TSDs were defined based on the change rates, since the coordinate values of W on the other global bases are zero, even minimal projection of AW onto these global bases could result in an excessively high change rate, which would be unreasonable. Secondly, if TSDs were defined based the absolute magnitude of coordinate changes, consider scenario where coordinate
18

---

Published as a conference paper at ICLR 2025
value in the core basis and coordinate value in the global basis change by the same amount. If the original coordinate value the core basis was significantly larger than the change, the change, even if substantial, could be viewed as a minor perturbation relative the original value. Therefore, the same magnitude of change does not necessarily imply that the corresponding directions are equally important. For these reasons, define TSDs based solely on the rate of change and restrict their definition to the core bases.
C DETAILS ON LORA-DASH
C.I ALGORITHM OF LORA-DASH
The algorithm of LoRA-Dash shown in Fig. 6
C.2 MATRIX OPERATION OF DASH PHASE
Since in Sec. 4.2 we have adopted vector calculations for change rates for clear representation. However, to enhance efficiency, we will transition to matrix computations for determining change rates.
un] E R and V = [Vi singular vectors for weight matrix W E IR (n m), we have
Let U -
Vm] mmxm represent the left and right
1 Wv
u A Wwm
U'AWV
AW V1
Vm
U AAWV,
(7)
u A
AWW
Therefore, we can derive the change rates 8 the matrix UWAWV.
[81
5m] which are the diagonal elements of
Following a similar process, can rewrite Eq.
5 as
=
(8)
where Ú i1
U.), V
M
anj AE diag([Ao
Ao.)).
D DETAILS ON EXPERIMENTS
D.1 IMPLEMENTATIO DETAILS
We compare LoRA-Dash mainly with LoRA. The ranks of LoRA-Dash, LoRA and other methods are varied among (2. 4, 8. 16, 32, 64). All methods are implemented using the publicly avail- able PyTorch Paszke et al. (2019) framework, and all experiments are conducted on NVIDIA A100 G The hyper-parameter of LoRA-Dash set 100, and 8. We fine-tune the layers of different models in all the experiments, and the detailed experimental settings are shown in the
following Secs. D.4 D.4 D.6
D.2 DETAILS ON FULLY FINE-TUNING LLAMA IN SECTION 3.3
To further explore the properties of TSDs, fully fine-tune LLaMA-7B Touvron al. (2023a) commonsense reasoning tasks. For the commonsense reasoning tasks, which consist of 8 distinct sub-tasks, each defined with specific training and testing sets, we have adopted the approach in Hu et al. (2023). Following this method, we amalgamated the training datasets from all sub-tasks to construct comprehensive final training dataset. More details can be found in Sec. D.4 The experiment was conducted on 8 A A000 GPUs.
We have also test that most of the experiments can also he conducted on consumer GPU resource such NVIDIA 3090.
19

---

Published as a conference paper at ICLR 2025
The change rates in Fig. derived from the weights obtained from the self-attention value projection in the 16th layer. However, our validation confirms that the results and conclusions drawn from the diagrams are consistent, regardless of which layer specific module within the attention mechanism is analyzed. For the original change rate 5 for a core basis, it is scaled as In(ó + 1)/3.
Additionally, it is worth noting that the phenomenon presented in Fig. [ is not isolated case. Similar behavior has been observed other tasks and models. For instance, as shown in Fig.. results obtained by training Qwen2.5-7B Team (2024) on the math reasoning task Hu al. (2023) exhibit patterns that are fundamentally consistent with our observations in Fig.
0.04
1.0
04
Change Rate
Change Rate
0.03
03
0.02
0.2
1001
2001 Index
3001
1001
2001 Index
3001
Figure 7: The results when fine-tuning Qwen2.5-7B on math reasoning task are similar to those in
Fig. 1
D.3 DETAILS ON FINE-TUNING LLAMA WITH LORA IN SECTION 3.4
Query
Key
Velue
0.90
1.0
r=
r=
16
r=
0.7
Recall
0.80
Recall
0.75
0.65
10000
20000
30000
7
22
32
Slep
Layer
Figure 8: We track the recall of TSDs every 100 training steps in the query, key and value layers of the LLaMA-7B during LoRA fine-tuning, analyzing how well the continuous updated W captures TSDs. The left and right figure settings the same as those in Fig. 2
We fine-tune LLaMA-7B using LoRA under three ranks, and for every 100 training steps we record the top core directions of W which exhibit the highest change rates based on the projection of continuously updated A AW (i.e., the directions related top largest 6; u Av., / where v; and are the corresponding left/rihht singular vectors and values of W). These directions serve as the "TSDs" predicted by LoRA. We denote these directions as "launched TSDs" (LTSDs) for differentiation. We then identify the top 4 or 16 TSDs with the highest change rates in Sec. 3.3
conduct two experimental setups to explore that:
1. How many of the top 4 TSDs are contained in the LTSDs;
2. How many LTSDs are contained in the top TSDs.
The purpose of the first experiment is more intuitive, aiming to confirm whether the TSDs with the top highest change rates can indeed be captured by AW from LoRA. The second experiment, which may seem bit more obscure, designed to address concerns that, besides capturing the top-ranked
20

---

Published as a conference paper at ICLR 2025
TSDs, the LTSDs identified might include irrelevant directions. Therefore, this second setup allows us to examine whether the LTSDs predominantly fall within the upper echelons TSDs based on their change rates, providing insight into the overall relevance and accuracy of the LTSDs identified by LoRA. Specifically, we employ two metrics to quantify the quality of LTTSD::
Accuracy: Measures how many of the 8 LTSDs correctly included in the top 16 TSDs.
Recall: Assesses how many of the top 4 TSDs are captured among the LTSDs.
The results presented in Figs 2 and 8 lead to two significant observations regarding the effectiveness of using AW identify TSDs:
The LTSDs captured by A W include several of the actual TSDs with the highest change rates, demonstrating that the predicted directions are not only accurate but pinpoint some of the most critical TSDs.
Precision in Identifying Crucial TSDs:
The entirety of the LTSDs resides within the upper echelon of TSDs in terms change rates. This suggests that while the primary directions identified are indeed the most vital, the remaining directions predicted also belong to the upper tier of important TSDs.
Overall Coverage of Significant TSDs:
These insights affirm the utility of AW in accurately capturing and representing the information of real TSDs. The method shows potential leveraging learned directional changes to identify and utilize key features specifie to given tasks, underscoring the robustness of AW predictive tool in practical applications.
For implementation details, please refer to Sec. D.4
D.4 DETAILS ON COMMONSENSE REASONING TASK
The commonsense reasoning benchmarks consist distinct sub-tasks, each with its designated dataset, i.e., BoolQ Clark et al. 2019), PIQA Bisk al. (020), SIQA Sap et al. (2019), HellaS. Zellers et al. (2019). , WinoG. (Sakaguchi et al. (2021), ARC- Clark et al. (2018). OBQA Mihaylov et al. (20T8). Adhering to the protocol outlined by Hu al. (2023). merge the training datasets from all tasks to form a comprehensive training dataset (Commonsensel 17KK
70K dataset), sub- sequently performing evaluations against each individual task's testing set. The template create
commonsense reasoning task is shown in Table. 6
We fine-tune LLaMA-7B Touvron et al. (2023a). LLaMA2-7B Touvron al. (2023b) and LLaMA3- 8B AI@ Meta (2024) on this task. Additionally, we integrate results from ChatGPT's implementation with the gpt-3.5-turbe API, particularly focusing on zero-shot Chain of Thought approaches Weil et al. 2022).
To ensure equitable comparisons, initial fine-tuning for models utilizing LoRA-Dash is conducted under the LoRA configurations while only varying the learning rate to optimize performance. The hyper-parameter settings LoRA-Das are shown in Table, 7 The results of LoRA are cited from Hu et al. (2023): Liu et al. (2024).
We also represent some answer samples with fine-tuned LLaMA-7B. The answers are shown in Tables. 8 15
D.5 DETAILS ON NATURAL LANGUAGE UNDERSTANDING TASK
For natural language understanding (NLU) task, we adopt the General Language Understanding Evaluation (GL.U) Wang et al. (2018) benchmark, which is designed to test capabilities across various tasks. This benchmark consists two single-sentence classification tasks, CoLA Warstadt et al. (2019) and SST-2 Socher et al. (2013), three similarity and paraphrase tasks, MRPC Dolan & Brockett (2003), QQP Wang et al. (2018), and STS-B B Cer et al. (2017), and three natural language inference tasks, MNLI Williams al. (2017). QNLI Rajpurkar et al. (2016). and RTE Dagan et al. (2005); Haim et al. (2006): Giappicoolo et al. (200)): Bentivogl et al. (2009). The details of these datasets are shown in Table. 16
21

---

Published as a conference paper at ICLR 2025
Table 6: The template of each dataset used create commonsense reasoning data for fine-tuning.
Dataset
Fine-tuning Data Template
BoolQ
Please answer following question with or false, question: [QUESTION]
Answer format: true/false
the correct answer is [ANSWER]
PIQA
Please choose the correct solution to the question: [QUESTION]
SolutionI: ISOLUTION 1]
Solution2: ISOLUTION 2]
Answer format: solutionl/solution2
the correct answer is [ANSWER]
SIQA
Please choose the correct solution to the question: [QUESTION]
Answerl: [ANSWER_I]
Answer2: [ANSWER.2]
Answer3: [ANSWER_3]
Answer
the correct answer is [ANSWER]
HellaSwag
Please choose the correct ending complete the given sentence: [ACTIVITY_LABEL] [CONTEXT]
Endingt: [ENDING_]]
Ending2: [ENDING.]] 2]
Ending3:
[ENDING.]
Ending4: IENDING 4]
Answer format: ending //ndding/eedddngg/nnnnnngg
the correct answer is [ANSWER]
WinuGrande
Please choose the correct answer to fill in the blank to complete the given sentence: [SENTENCE]
Optionl: [OPTION_I]
Option2: [OPTION_2]
the correct answer [ANSWER]
ARC-e ARC-c
Please choose correct solution the question: [QUESTION]
Answerl: [ANSWER 1]
Answer2: [ANSWER 2]
Answer3: [ANSWER_3]
Answer4: [ANSWER_4]
Answer format: hwwt///mmwee///nserraaaweee
the correct unswer is [ANSWER]
OBQA
Please choose the correct solution to the question: [QUESTION]
Answerl: [ANSWER_I]
Answer2: [ANSWER 2]
Answer3: [ANSWER_]]
Answer4: [ANSWER.4]
Answer format: hww.e///aawwec//sssee/awweeer
the correct answer is [ANSWER]
Table 7:
settings of LoRA-Dasi on commonsense reasoning task.
Hyper-parameters
LLaMA-7B
LLaMA2-7B
LLaMA3-8B
Rank
4
8
16
32
64
16
32
16
32
a
8
16
32
64
128
32
32
64
LR
5c-4
4e-4
5c-4
lc-4
0.9c-4
2e-4
lc-4
2c-4
0.8c-4
LR Scheduler
Linear
Dropout
0.05
Optimizer
Adam W
Batch size
16
Warmup Steps
100
Epochs
3
Where
Q, K. V. Up. Down
22

---

Published as a conference paper at ICLR 2025
Table 8: An example from BoolQ. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
was whiskey galore based on true story? true
Label
LoRA
the correct answer is false
LoRA-Dash
the correct answer is true
Table 9: An example from PIQA. The outputs of LLaMA-7B with LoRA and aash.
Question
To add chocolate and caramel to hot milk and cream.
Solution1:
Chop the chocolate into small chunks.
Bring the milk and cream simmer on a low-medium heat.
Once the mixture begins to cool, remove from the heat and add the chocolate and caramel sauce.
Solution2: Chop the chocolate into small chunks.
Bring the milk and cream simmer low medium heat.
Once the mixture begins simmer, remove from heat and the chocolate and caramel sauce.
Label
solution2
LoRA
the correct answer solution1
LoRA-Dush
the correct answer is solution2
Table 10: An example from SIQA. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
Lee challenged her to fight after she talked bad about Lee to his friends. How would Others feel result?
Answerl: happy that Lee wants fight
Answer2: angry
Answer3: fight
Label
answer2
LoRA
the correct answer is answerl
LoRA-Dash
the correct answer is answer2
Table 11: An example from HellaSwag. The outputs of LLaMA-7B with LoRA and LoRA-Dash
Question
Personal Care and Style: [header] How to dress rugged (men) [title] Shop for pants.
[step] Greatt rugged outhits start with the right pants.
Shop for jeans, khakis and canvas with straight-eut legs, and avoid any flashy graphics or lond colors.
Endingt: Also, make you try pants ensure they fit comfortably.
Rugged styles are about utility, so the pants shouldn't extremely tight or extremely baggy:
look for "relaxed fit" "straight-cut".
Ending2: Men should try slacks, dress pants, or street clothes for rugged look.
[substeps] Choose gut-length, flared, or boot-cur trousers.
Ending3: good rule of thumb to dress in the right pair of pants, balance with that.
Consider clothing tucked into your upper thighs with high-rise, thigh high, or flare
Ending4: Lightweight and comfortable feet are always major consideration when shopping for rugged shirt.
[substeps] These hiking booss. while lightweight and appropriate for longer. must.
Label
eudingl
LoRA
the correct answer ending?
LoRA-Das -ash
the correct answer is ending!
We fine-tune DeBERTaV3- base and DeBERTaV3- He et al. (202lb) models on this task. The hyper-paramettr settings for this task is shown in Table. 17
23

---

Published as a conference paper at ICLR 2025
Table 12: An example from WinoGrande. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
Lawrence was always gaining weight while Dennis was losing it as liked to eat too little.
Optionl: Lawrence
Option2: Dennis
Label
option2
LoRA
the correct answer is optionl
LoRA-Dash
the correct answer is option2
Table 13: An example from ARC-e. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
There are a variety of events that can disrupt ecosystems. Some events result in local disturbances and others can affect much larger area. Which event would most likely result in change on a global scale?
Answerl: land scarred by forest fires
Answer2: erosion of ocean shorelines
Answer3: converging plate boundaries
Answer4: destruction of tropical rain forests
Label
answer4
LoRA
the correct answer is answer3 the correct answer is answer4
LoRA-Dash
Table 14: An example from ARC-e. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
What is similarity between sound waves and light waves? Answerl: Both carry energy.
Answer2: Both travel in vacuums.
Answer3: Both are caused by vibrations.
Answer4: Both are traveling at the same speed.
Label
answerl
LoRA
the correct answer is answer3 the correct answer is answerl
LoRA-Dash
Table 15: An example from OBQA. The outputs of LLaMA-7B with LoRA and LoRA-Dash.
Question
Which best demonstrates the concept force causing an increase in speed? Answerl: skating on rough surface
Answer2: a full bag swung in circles
Answer3: computer powering on
Answer4: a baker stirring batter
Label
answer2
LoRA
the correct answer is answerl the correct answer is answer2
LoRA-Dash
D.6 DETAILS ON SUBJECT-DRIVEN GENERATION TASK
In our experiment, we fine-tune the text-to-image diffusion models specifically tailored for subject- driven generation tasks, as outlined in recent research Ruiz et al. (2023). The objective this task is to generate images that adhere closely to prompts associated with particular subject, defined by a few exemplar images. This involves initially fine-tuning text-to-image model using image- text pairs where the text contains unique identifier (e.g.. "A photo of [V] cat"). Subsequent
24

---

Published as a conference paper at ICLR 2025
Table 16: Details of GLUE dataset.
Dataset
Task
# Train
# Dev
# Test
Label
Metrics
Single-Senn Sentence Classification
CoLA
Acceptability
8.5k
1k
1k
2
Matthews corr
SST-2
Sentiment
67k
872
1.8k
2
Accuracy
Similarity and Paraphrase
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy Fl
QQP
Paraphrase
364k
40k
391k
Accuracy FI
STS-B
Similarity
7k
1.5k
1.4k
1
Pearson/ Spearman Corr
Natural Language Inference
MNLI
NLI
393k
20k
20k
3
Accuracy
QNLI
QA/NLI
108k
5.7k
5.7k
2
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
Table 17: Hyper-parameter settings of LoRA-Dash on NLU task.
Hyper-parameter
MNLI
SST-2
CoLA
QQP
QNLI
RTE
MRPC
STS-B
Optimizer
AdamW
Warmup Ratio
0.1
LR schedule
Linear
Rank r
2 &
LoRA alpha
& 16
Max Seq. Len.
256
128
64
320
512
320
320
128
Batch Size
32
32
32
32
32
32
32
32
Learning Rate
5e-4
8e-4
8e-4
le-3
5c-4
1.2c-3
1e-3
5c-4
Epochs
12
24
25
5
5
50
30
25
image generations are driven by new prompts incorporating this identifier, aiming to produce images aligned with the learned subject.
For this experiment, we use the SDXL5 model Podell et al. (2023). applying both LoRA and LoRA- Dash methods for fine-tuning. The fine-tuning process is conducted with learning rate of le-4 and a batch size of 4. We train the model over 500 steps a single 80GB A100 GPU, taking approximately 23 minutes to complete. For the generation phase, we execute 50 inference steps for each given prompt to synthesize the final images, which takes approximately 7 seconds to complete.
We mainly adopt the official data of DreamBooth Ruiz et 2023 for diffusion.
D.7 DETAILS ON THE AMPLIFICATION OF TSD IN SEC.
To explore to what extent LoRA-Dash can amplify the information the TSDs it determines, we project the pretrained weights W and the weights merged after training, i.c. W + A W all. onto the LTSDs. Let Ü [új u,] and = [ Vel. we can then define the amplification factor of AW all the Frobenius norms of these weights these directions, which is given
25

---

Published as a conference paper at ICLR 2025
by:
[U(W- + AWai)
Specifically, we also evaluate the separate impacts of AW AB and AW dash in the amplification, with the corresponding amplification factor being
| (W AWWAWR VAB and | (W
AW dash V
(10)
|
[U
The results of the amplification are shown in Table.
Moreover, as shown Sec. B.2 TSDs are indeed the projection of W direction onto the core bases of W. Therefore, we also explored the amplification factor along the direction of AWait We replaced U and V the previous amplification factor calculation with the top r left and right sin- gular vectors of A W Wal, where r is the rank setting of LoRA-Dash, We reported the corresponding results in Table. 18 where we can draw a conclusion:
Table 18: Amplification of AW... all of LoRA-Dash onto TSDs or AW wnn't direction. We report the amplification factor from the self-attention key (k) projection of the first, middle and last layer of LLaMA-7B, as well as the average of all q. k, and modules.
U and V from
r 8
7
r
32
TSD
A W all
TSD
A W alt
TSD
A W all
1st
282.15
22.75
628.60
30.13
42.00
3.07
16th
11.15
144.89
13.07
119.18
2.16
9.92
32nd
68.02
75.81
30.87
120.76
8.19
10.63
Avg.
22.85
153.49
32.12
126.26
5.28
11.31
The amplification factor the directions of A W all is significantly higher than that on the TSDs. Since TSDs can be considered as a projection of A W onto the coordinate system of W, is natural that some information lost in this projection for only select TSDs, leading to a lower amplification factor on the TSD compared to before the projection.
Additionally, observed interesting trend: the amplification factor AW all 's directions sig- nificantly decreases as the rank increases. In LoRA Hu et al. (202T), the authors claimed that the occurrence of small number of TSDs leads to this situation. However, is that higher rank corresponds to AW containing more irrelevant directions, thereby reducing the amplification factor.
plausible explanation
To validate this, consider the setting of = 32 of A The 32 directions of A AWa an include some irrelevant directions, which is why the amplification factor is lower. This implies that if the irrelevant directions among these 32 are removed, the amplification factor could potentially increase. To explore whether this the case, we recalculated the amplification factors using the top 2, and 16 singular directions of A W.it. and the results are shown in Table. 19 The observation that the amplification factor decreases as projections are made onto an increasing number of the top singular directions of A W all indicates that A W au 's singular directions do indeed contain irrelevant directions in its bottom singular directions. It is not surprising, since TSDs are only a few directions, select several top directions in pre-
shown in its properties. Therefore, it also motivates launch phase of LoRA-Dash.
However, believe that there is necessary correlation between amplification factors and the effectiveness of a model. Amplification factors simply reflect how much the model amplifies features in certain directions, and the specific degree to which this should occur is unknown.
26

---

Published as a conference paper at ICLR 2025
Table 19: Amplification of AW all of LoRA-Dash onto different number of top singular directions of A Wau. We report the amplification factor from the self-attention key (k) projection of the first, middle and last layer of LLaMA-7B, as well as the average of all q. k, and modules.
U and V are directions of top
2
4
8
16
32
1st
11.93
8.65
6.44
4.34
3.07
16th
71.82
50.09
31.98
17.89
9.92
32nd
160.18
58.17
30.82
19.15
10.63
Avg.
139.03
70.42
38.87
21.02
11.31
E MORE EXPERIMENT RESULTS
E. MORE RESULTS FOR SUBJECT-DRIVEN GENERATION TASK
We here provide more visual results for the subject-driven generation task in Fig.
We also conducted user study using the DreamBooth dataset. We selected 10 different categories and generated corresponding images using five prompts for each category. We then invited 36 ex- perts to evaluate the images based on subject similarity and prompt consistency, with each expert evaluating randomly sampled 25 pairs of images. During evaluation, the pairs of images were shuf- fled, and the experts were unaware of the source model for each image. Overall, the images LoRA- Dash generated achieved an 84.51 approval rating. This suggests that images generated by LoRA-Dash were more favorably recognized by experts in terms of subject similarity and prompt consistency.
We also evaluated the performance using the text-image score evaluated by CLIP Radford et al. (22II and the image similarity score. The CLIP text-image score measures the alignment between textual descriptions and the corresponding images, while the image similarity score quantifies the similarity between the original image and the generated image. The results are presented in Table
20 where it is evident that LoRA-Dase outperforms LoRA both metrics. The images generated by A-RSa exhibit better text alignment and a stronger correspondence with the original images, demonstrating that LoR A-Dsh achieves superior results compared to LoRA.
Table 20: Performance comparison between LoRA and LoRA-Dash CLIP text-image alignment and image similarity metrics.
Metric (%)
LoRA
LoRA-Dash
Mean CLIP Text-Image Score Mean Image Similarity
32.0
32.8
76.4
78.2
While we try to ensure objectivity and fairness, it is worth noting that, unlike numerical compar- isons, visual comparisons and user studies are inherently subjective. As for the actual effectiveness of LoRA-Dash, we look forward feedback from researchers broader range of practical appli- cations.
E.2 MORE RESULTS FOR THE ALIGNMENT ON THE DIRECTIONS OF LORA- -DASSH
More results for Sec. 6.1 are shown in Tables. 21
E.3 MORE RESULTS FOR ABLATION STUDY
We here provide more results when fine-tuning DeBERTaVe VeBaRTaV3-large the ablation study in Fig. b)-(c) in Sec. 6..4 The experimental results observed align well with the trends analyzed in Sec.
6.4 indicating consistency between theoretical insights and practical outcomes.
27

---

Published as a conference paper at ICLR 2025
LoRA
LORA Dash
ORA
LoRA Dash
Lokt
LoRA Dash
Input Images Bear Plushie
puple bear plushie
A bear plushie on a minor
A bear plushie on white rug
LOBA Dash
LORA
LORA- Dash
LORA
Cocd
Input Images Duck Toy
A duck toy on the beach
A duck toy with mountain in the background
shiny duck toy
"LOA
LoRA Dash
LoRA
LORA Dash
LORA
LoRA ash
Input Images Teapot
teapot with blue house in the background
teapot with the Eiffel Tower in the background
A leapot in the jungle
LORA
LoRA- Dash
LORA
LORA
Dash
LORA
LoRA- Dash
Input Images Backpack
A backpack on the wooden floor
A backpack with a mountain in the background
A backpack wheat field
OROAA
LORA
Dash
LORA
LORA Dash
DRA
LORA Dashi
Input Images C Candle
A candle on top of pupple rug in forest
candle with a tree and autumn leaves the background
candle white rug
WILL LoRA
LORA
LoRA Dash
LORA
LoRA- -aRA-Dash
LoRA-Das
Input Images Fancy Boot
A fancy boot top of pink fabric
fancy boot on the sidewalk in crowded street
A purple fancy boot
Figure 9: More results for subject-driven generation task.
We here further discuss the impact of selecting different dash directions. The effectiveness of choos- ing top, bottom, random directions has been topic frequently mentioned various studies. However, find differing conclusions in different works, which complicates the understanding. For instance, Meng et al. (2024) and ZZhan & Pilanci (2024) suggest that selecting top directions vields the best results, while Hameed al. (2024) advocates for random selections, and reference Wang et al.) (2024) supports choosing the bottom. This inconsistency is puzzling and a matter we aim to explore further through research.
Indeed, fine-tuning leverages model's broad pre-trained knowledge base, adapting to learn spe- cific, nuanced details relevant to particular downstream tasks. The essence of TSDs lies in their
28

---

Published as a conference paper at ICLR 2025
Table 21: Alignment the directions of LoRA-Dash with TSDs. We report the percentage from the self-attention value (v) projection of the first, middle and last layer of two models, as well as the average percentage of all q. k, and v modules.
Model
LLaMA-7B (r 8) 8)
DeBERTaV3-large (r 2)
Layer
First (Ist)
Middle (16th)
Last (32nd)
All.
First (Ist)
Middle (12nd)
Last (24th)
All.
AWau L
1.00
0.75
1.00
0.74
1.00
1.00
1.00
0.99
TSD n LTSD
1.00
1.00
1.00
0.78
0.75
0.75
0.50
0.73
TSD AAWW
1.00
0.75
1.00
0.83
0.75
0.75
0.50
0.76
Table 22: Alignment on the directions of LoRA-Dash with TSDs. We report the percentage from the self-attention value (v) projection of the first, middle and last layer of two models, well as the average percentage of all k, and v modules.
Model
LLaMA-7B (r 32)
DeBERTaV3
32)
Layer
First (Ist)
Middle (16th) Last (32nd)
All.
First (Ist)
Middle (12nd)
Last (24th)
All.
AWau LTSD
0.75
0.75
0.75
0.75
0.75
1.00
1.00
0.97
TSD n LTSD
0.75
0.75
0.75
0.77
0.75
0.75
1.00
0.78
TSD n AW.l
1.00
1.00
1.00
0.81
0.75
1.00
1.00
0.82
representation of the singular directions where the original model weights exhibit the most signifi- cant changes during adaptation new tasks. Intuitively, these TSDs, being the most altered, should indeed offer the most impactful directions for adjustment during fine-tuning, potentially yielding the best performance.
Analyzing the properties of TSDs reveals that they typically reside along the directions correspond- ing to smaller, yet not the smallest, singular values. If we choose to dash in the directions associated with the smallest singular values-the "bottom" directions-itf plausible that some of these might overlap with the TSDs. Hence, dashing in the bottom directions might produce results inferior to directly using the TSDs.
As for the top singular directions, which encapsulate the most significant information from pre- training, these generally involve more generalized knowledge. Since such information is less task- specific, adjusting these directions could potentially degrade performance rather than enhance it, especially if the task at hand requires nuanced understanding that diverges from the general pre- training data.
Lastly, random selection of directions introduces unpredictability in outcomes. However, there is a chance that such random choices might inadvertently select appropriate directions or even TSDs, thus performing better than top selections but generally worse than bottom choices due to the ran- domness. This method could strike balance between leveraging underlying model strengths and adapting new specifics, albeit inconsistently.
In our empirical tests, the results largely align with our predictions. However, there are some ex- ceptions where the effectiveness of top directions surpasses random or bottom, or where the bot- tom directions approach or even slightly exceed the performance of the TSDs. Please look for counterexample the Fig. a).. We consider these variations to be within the expected range, acknowledging that it's unrealistic to expect uniform trends across all tasks and models. The trends we have identified represent a more general, broadly applicable pattern.
In addition the standard experimental setup, we conducted two additional experiments: adjusting the coordinate value changes all directions or both top and bottom directions collectively. Neither of these configurations resulted in improved outcomes. This aligns with analysis in Sec. 6.4 which suggested that the inclusion of irrelevant directions could degrade training performance due to increased noise in the model updates.
29

---

Published as a conference paper at ICLR 2025
60
80
50
78
78
78
Acuuweyy
70-
76-
76-
74
L005
tonddoato
50
100
200
902
1002
2300
5000
10000
16
Length
Figure 10: More results on CoLA when fine-tuning DeBERTaV3-large for ablation study of LoRA- Dash. a. Ablation study on the directions for dash. Beyond the top, random, and bottom directions, we also explored adjusting all directions and a combined adjustment both bottom and top direc- tions. b. Ablation study on the length of pre-launch phase t. Ablation study the number of directions s for dash.
Dog
Dog
Dog Cat
Dog A & Clock
Dog A & Dog B
Cat & Clock
Car & Dog B
Clock & Dog I
Dog A Cat
Dog A & Clock
Dog A & Dog B
Cat & Clock
Cat & Dog B
Clock & Dog
The results of the TSDs similarity between different tasks (subjects). For the first row: Images from different subjects. The second and third rows contain results extracted from dif- ferent layers of U-net Ronneberger et al. (2015). Each subplot illustrates the similarity of TSDs captured for different subjects. The title of each subplot indicates the respective subjects, with the directional order of the first subject represented by columns and the second by rows. The deep blue blocks highlight where the directions for the two subjects coincide.
Figure 11:
E.4 ARE TSDS "TASK-SPRCIFICC"
We sought to verify whether TSDs are genuinely task-specific, that is, whether the most critical directions differ for each task. To investigate this, we employed a prototypical example: the subject- driven generation task. In this setting, tracked the TSDs captured by LoRA-Dash when different subjects served as targets and analyzed the relationships between these TSDs. The results are pre- sented in the Fig. (Please refer to the caption of Fig. for detailed information).
The experimental results affirm the task-specific nature of TSDs; indeed, each task exhibits distinct TSDs with minimal overlap between tasks. Moreover, even when tasks share the same direction,
30

---

Published as a conference paper at ICLR 2025
its importance varies significantly between them. For instance, depicted in the central subfigure of the first column in Fig. [I] both "Dog A" and "Cat" share a common direction, yet it ranks as the seventh most significant for "Dog A" and the second most significant for "Cat". Furthermore, tasks that are closely related tend to share more directions, as observed between "Dog A" and "Dog B"; however, the importance of these shared directions still differs for each task. This variability underscores the fundamentally task-specific property of TSDs, illustrating their unique and variable impact across different contexts.
E.5 CONVERGENCE CURVE
We present loss curves for both LoRA-Dash and LoRA when fine-tuning LLaMA-7B. We ensure consistency in all training apper-paramters, and the results are shown in Fig. [2 We can ob- serve that during the pre-launch phase, the loss trajectories of LoRA-Dash and LoRA are aligned. However, as the training progresses into the dash phase, LoRA-Dash consistently exhibits a slightly lower loss compared to standard LoRA.
One may concern that a lower training loss could lead to overfitting. Therefore, we also tracked the validation loss throughout the training process, with the results presented in Fig. b). It is clear that the validation loss for L-RA-Dash is much lower than that for LoRA. This suggests that LoRA- Dash not only avoids overfitting but also enhances its learning efficacy by effectively incorporating lask-specific knowledge.
LoRA-Dash
LoRA-Dash
LoRA
LoRA
Loss
1-
100 200 300
1000
2000
3000
4000
5000
6000
0
5000 10000 15000 20000 25000 30000
Step
Step
Step
Figure 12: a. Training loss when fine-tuning LLaMA-7B. We record the training loss every 10 steps. Validation loss when finettuning LLaMA-7B. We record the training loss every 80 steps.
E.6 COMPARISON WITH OTHER METHOSS
In the main text, we only mainly compared the results of LoRA-Dash and LoRA because LoRA- Dash is designed further hamesss the potential of TSDs captured by LoRA. Here, further compare the performance of LoRA-Dase with other methods.
Specifically, we compare BitFit Zaken al. (2021), )), Prompt Learning (Prefix) Li Liang (202 Series Adapter (Series) Houlsby et al. (2019). Adapter proposed by Pfeiffer et al. (2020) (PAdapter), Parallel Adapter (Parallel) He et al. (2021a), (IA) Liu et al. (2022), SSL Si et al. (2024b), SSB Si et al.) (2024b), AdaLoRA Zhang et al. (002)). PISSA Meng et al. (202)). MiLoRA Wang et al. (2024), FLORA et al.) (2024a) and DoRA et al. (2024). We follow the settings detailed in our
paper, and the results are shown in Tables. 23 24
The results clearly show that even compared to state-of-the-art (SOTA) methods like DoRA and Fl FLORA, LoRA-Dash holds its ground impressively. Additionally, two phenomena are noteworthy:
When fine-tuning LLaMA-7B, at a rank of 4, all methods except LoRA-Dast perform poorly under low parameter budgets, while LoRA-Dash achieves results comparable to LoRA- -Dssh,
those with higher parameter settings. This further highlights the superiority of which stimulates performance downstream tasks by unleashing the potential of TSDs.
31

---

Published as a conference paper at ICLR 2025
Table 23: Results commonsense reasoning tasks of LoRA-Dash compared with other methods.
For LoRA derivatives, we report the parameter gains over LoRA.
Method
Params BoolQ
PIQA SIQA HellaS. WinoG.
ARC-e
ARC-
OBQA
Avg.
ChatGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
Fine tuning LLaMA-7B
Fully FT
100%
69.9
84.2
78.9
92.3
83.3
86.6
72.8
83.4
81.4
Prefix
0.11% 1%
64.3
76.8
73.9
42.
72.1
72.9
54.0
60.6
64.6
Series
0.99%
63.0
79.2
76.3
67.9
75.7
74.5
57.1
72.4
70.8
Parallel
3.54%
67.9
76.4
78.8
69.8
78.9
73.7
57.3
75.2
72.2
LoRA,
0.10%
2.3
46.1
18.3
19.7
55.2
65.4
51.9
57.0
39.5
AdaLoR ,oRA, A
0.6k
78.1
74.3
34.0
74.4
57.5
71.2
FLORA,
2.6k
67.2
78.0
72.9
65.4
73.8
73.8
55.3
71.8
69.8
DoRA,=1
+ 877k
51.3
42.2
77.8
25,4
78.8
78.7
62.5
78.6
61.9
LoRA-Dash,
1.3k
65.2
79.9
78.3
82.8
77.1
78.6
65.4
78.4
75.7
LORA,
0.83%
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
AdaLoRA,=3y
5.1k
69.1
82.2
77.2
78.3
78.2
79.7
61.9
77.2
75.5
FLORA, (0ORA,=32 =322
+ 164k
66.4
81.3
77.1
75.6
77.1
77.2
62.4
77.6
74.3
DoRA, A-
+ 877k
69.7
83.4
78.6
87.2
81.0
81.9
66.2
79.2
78.4
LoRA-Dash, Araa-Dsh,-30
1.3k
69.9
82.8
78.6
84.9
81.6
82.3
66.5
80.8
78.4
Fine-tuning LLaMA3-8B
LoRA,
0.35%
72.3
86.7
79.3
93.5
84.8
87.7
75.7
82.8
82.8
AdaLoRA,
2.6k
90.4
85.0
76.7
79.1
83.3
86.4
75.1
75.4
81.4
FLORA, 1
+ +1k
90.2
84.2
79.9
79.3
85.1
86.7
74.8
93.9
84.2
ORRA-Dash-=a
1.3k
74.8
88.0
80.6
95.2
85.6
89.0
77.4
84.8
84.4
LoRA,
0.70%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
PISSA 1OSSA,330
67.1
81.1
77.2
83.6
78.9
77.7
63.2
74.6
75.4
MiLoRA,=30
+0
68.8
86.7
77.2
92.9
85.6
86.8
75.5
81.8
81.9
DoRA
+ 784k
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
LoRA-Dash,
1.3k
75.3
88.5
80.2
95.7
86.8
90.7
80.2
85.6
85.4
Table 24: Results with DeBERTaV3-base fine-tuned on GLUE development set. "FT" represents
fully finettunin.
MNLI
SST-2
CoLAA
QQP Ave
QNLI Ave
RTE Acc
MRPC
STS-B
Method
% Params
Acc
Acc
Acc
Curr
Avg.
FT 100%
89.90
95.63
69.19
91.87
94.03
83.75
90.20
91.60
88.27
(IA)3
0.03%
89.44
95.52
67.01
89.01
91.80
79.42
88.23
90.79
86.40
SSL
0.02%
88.35
95.07
66.64
88.19
90.10
82.31
88.68
90.13
86.18
SSB
0.05%
89.86
95.53
67.82
89.87
93.41
83.75
88.72
90.94
87.49
BitFit
0.05%
89.37
94.84
66.96
88.41
92.24
78.80
87.75
91.35
86.21
Series
0.17%
90.10
95.41
67.65
91.19
93.52
83.39
89.25
91.31
87.73
PAdapter
0.16%
89.89
94.72
69.06
91.05
93.87
84.48
89.71
91.38
88.02
LoRA
0.18%
90.03
93.92
69.15
90.61
93.37
87.01
90.19
90.75
88. 13
AdaLoRA
0.18%
90.66
95.80
70.04
91.78
94.49
87.36
90.44
91.63
88.86
FLoRA
0.18%
90.60
96.00
70.20
91.40
94.46
88.09
90.93
91.96
89.21
DoRA
0,22%
90.21
94.38
69.33
90.84
93.26
86.94
90.19
91.34
88.31
LoRA-Dash
0.18%
90.14
95.42
72.41
91.65
94.36
89.89
91.67
91.64
89.65
Compared with LoRA derivatives, LoRA-Dash introduces significantly fewer additional parameters, such as nearly 130 times fewer than FLoRA and 700 times fewer than DoRA rank when fine-tuning LLaMA-7B, yet achieves better or comparable results.
Revisiting the objective of Earameter-Efficient Fine- Tine-Tuning, the key question is whether it is possi- ble achieve performance comparable to fully fine-tuning while training with significantly fewer parameters. Through extensive experimentation and comparison with other methods, LoRA-Dash maintains excellent performance with minimal parameter budget, where most methods would
32

---

Published as a conference paper at ICLR 2025
be constrained by the amount of parameters available. This demonstrates that LoRA-Dash signifi- cantly surpasses other methods terms of parameter utilization efficiency. Therefore, we conclude that LoRA- R-R- exemplifies ideal realization of PEFT goals.
Reflecting on the entirety of this research, we arrive at a final conclusion:
"LoRA-Dash epitomizes the essence of parameter efficiency, maximizing the ef- ficacy of each parameter deployed. With significantly fewer parameters trained, LoRA-Dash sustains or even exceeds the performance set by fully fine-tuning."
This marks the end of our current exploration, but it merely pause in the ongoing quest for deeper understanding. The path ahead remains rich with potential, inviting further inquiry into the intricate dance of knowledge...
The End
33


Reviews:
[{"review_id": "5BHqV49Vsw", "reviewer": "Reviewer_93T4", "summary": "This paper proposes a parameter-efficient fine-tuning (PEFT) approach, LoRA-Dash, that explicitly exploits task-specific directions to improve parameter-efficiency. The proposed method consists of two phases: in the pre-launch phase, LoRA fine-tuning is performed for a certain number of steps to identify task-specific directions, which are estimated as the core directions of the pre-trained matrix most amplified by the LoRA update matrix. In the dash phase, changes in these directions are explicitly parameterized alongside the LoRA matrices to further enhance task alignment. The authors empirically demonstrate that LoRA-Dash outperforms standard LoRA and recent PEFT methods across a range of tasks and base models and show the robustness of the estimated task-specific directions.", "questions": "1. In Figure 5, the performance on OBQA with TSD in the left-most subplot does not match with Length of Pre-launch Phase set to 100 in the middle subplot. Why is this the case since both settings should correspond to the setting used in the main experiment?\n2. For subject-driven generation tasks, how do the task-specific directions correspond to the objects or features in the image?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq.\n1. This paper introduces a novel framework for identifying task-specific directions in pre-trained models by projecting various task-related matrices onto the subspace of the original weight matrix, providing a consistent basis for analyzing task-specific adaptations.\n2. The proposed method effectively leverages task-specific directions, achieving significant improvements over LoRA with minimal computational overhead, demonstrating its practical advantage as a PEFT method.\n3. Empirical evaluations show that the task-specific directions derived from this approach align with those amplified by full fine-tuning, indicating that the method successfully identifies important directions for task adaptation.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.\n1. [Reliance on Preliminary Experiments for Hyperparameter Selection]\nThe proposed method relies on preliminary experiments to set key hyperparameters, such as the length of pre-launch phase and the number of dash directions. Although the chosen hyperparameters work well on the evaluated benchmarks, there is no guarantee that these settings generalize to other tasks and models. Moreover, the authors do not provide a systematic or principled approach for selecting these hyperparameters in varying scenarios.\n2. [Lack of Objective Evaluation for Diffusion Models]\nThe evaluation of diffusion models relies solely on subjective assessments, which introduces potential biases and limits reproducibility. Incorporating quantitative metrics for diffusion model performance would strengthen the evaluation and provide a more comprehensive view of the model’s effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rQOz3VUOFE", "reviewer": "Reviewer_B9L3", "summary": "This paper critiques LoRA's prior TSD exploration and provides a precise TSD definition to better understand their role in fine-tuning large language models. The authors introduce LoRA-Dash, which fully exploits TSDs potential, and demonstrate its significant advantages over conventional methods through extensive experimentation. The findings validate LoRA-Dash's effectiveness and highlight the importance of TSDs in parameter-efficient fine-tuning, aiming to inspire further advancements in natural language processing and beyond.", "questions": "Thanks for submitting to ICLR'25, I really enjoy reading your paper. I think your paper focuses on a hot and important topic. However, I have several further questions:\n\n1. The paper lacks evaluation of the algorithm's overhead on end-to-end fine-tuning time. Can LoRA-Dash accelerate the fine-tuning phase as suggested by its name?\n2. The benefits of the LoRA-Dash algorithm are unclear. Although it outperforms the vanilla LoRA approach when r is small, its improvement over the best fine-tuned LoRA model is minimal. Please clarify the real benefits of the LoRA-Dash algorithm.\n3. Can you provide insights on applying TSD observations to other parameter-efficient fine-tuning methods, such as adapter/P-tuning? Are these methods suitable for this approach?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors observe that TSD can be predicted from delta W in LoRA-based fine-tuning, offering a new perspective on task-specific directions.\n2. This paper provides a precise definition of task-specific directions and explores their application in LoRA-based parameter-efficient fine-tuning.\n3. The authors propose the LoRA Dash algorithm, which proactively utilizes these influences to enhance the model fine-tuning process.", "weaknesses": "See detailed questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCvkhprNHF", "reviewer": "Reviewer_KtSW", "summary": "This paper addresses the resource-intensive issue of fully fine-tuning large language models. It focuses on task-specific directions (TSDs) in parameter-efficient fine-tuning (PEFT). LoRA is examined, and its lack of a clear TSD definition is noted. A framework is then introduced to define TSDs based on the relationship between pre-trained weights and optimal weights for specific tasks. The properties of TSDs are explored, and challenges in using them are identified. Despite the unknowns in practical fine-tuning, it's found that LoRA's ∆W can capture TSD information. The novel LoRA-Dash method is proposed, comprising pre-launch and dash phases. Experiments show that LoRA-Dash outperforms LoRA, is robust to parameter budgets, and excels in various tasks. It also enhances other PEFT methods and provides valuable insights for optimizing model fine-tuning.", "questions": "Mostly the above comments. Also some minors:\n\n- Is Lora-Dash applicable to multimodal models, such as LLaVA?\n\n- Why does the performance of LoRA-Dash decline when the rank increases beyond a certain point?\n\n- How can we address the issue of excessive memory requirements for SVD in the application of this method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A task-aware approach for identifying critical gradient directions is proposed.\n\n- An effective task-specific fine-tuning scheme Lora-Dash has been introduced.\n\n- Theoretical analysis and extensive experiments have been conducted, providing comprehensive validation of the effectiveness of the proposed algorithm.", "weaknesses": "- More justification is expected. For example, does fine-tuning a general-purpose model for a specific task risk compromising its generalization capability? It is essential to explore strategies that enable a model, even after fine-tuning, to retain the flexibility to handle a range of general tasks. Striking a balance between task-specific adaptation and broad applicability remains a key challenge in fine-tuning.\n\n- The experiments can use more perspectives. For example, the observations in Figure 1 are based solely on commonsense reasoning tasks in the LLama model. Further comparisons with other relevant methods (such as Lora-ga，dora) and other models (such as Qwen, Mistral) should be included for a more comprehensive analysis. \n\n- In the visualization part, as shown in Figure 4 and Figure 8, a comparison with the results from full fine-tuning should also be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTI9hsC4dj", "reviewer": "Reviewer_27u4", "summary": "To enhance the prevalent parameter efficient fine-tuning (PEFT) strategies (e.g., LoRA) for large language models (LLMs), this paper delves into the concept of task-specific directions (TSDs) which are crucial for the adaptation of pre-trained LLMs to downstream tasks. Based on the proposed mathematical description of TSDs, a simple and effective scheme is designed to approximately identify TSDs from the parameter difference matrix obtained through LoRA. The model parameters derived from LoRA are then aligned with the estimated TSDs to enhance LoRA's performance. The results of extensive experiments confirm the validity of the mathematical description and the effectiveness of the proposed PEFT framework across diverse learning settings.", "questions": "Please refer to weaknesses 1-5 listed in the section above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well organized and easy to follow.\n2. In contrast to the conceptual discussion of task-specific directions (TSDs) in existing parameter-efficient fine-tuning methods, this paper offers a deeper understanding by providing a mathematical description of TSDs, which is crucial for enhancing the performance of these parameter-efficient fine-tuning methods.  \n3. Te proposed TSDs identification and utilization method are simple and effective.\n4. Experimental results across diverse settings validate the significance of task-specific directions in fine-tuning large language models (LLMs), and demonstrate the effectiveness of the proposed LoRA-Dash compared to the prevalent LoRA algorithm.", "weaknesses": "1. As a formal definition, the description of task-specific directions (i.e., “directions whose coordinate values exhibit significantly higher change rates $\\delta$ through alteration” in Definition 4) is somewhat vague. The term “significantly higher change rate” would benefit from a more precise definition, such as being specified with reference to a particular threshold.\n2. More discussions about how to select the value of hyper-parameter $s$ should be included, as the choice of $s$ directly determines the exact task-specific directions.\n3. The statement in Proposition 1 is supported solely by observations from experiments and is not accompanied by rigorous proof.\n4. The evaluation results of Fully FT method regarding LLaMA2-7B and LLaMA3-7B models are missed in Table 1.\n5. To thoroughly evaluate and compare the effects of task-specific directions and other core directions, it is recommended to include the vanilla LoRA as a baseline method in Figure 5a.\n6. The primary contribution of this paper is the proposal of a simple and effective improvement scheme for the LoRA algorithm and its variants. Although this contribution is commendable for acceptance, its impact on the broader research community is still limited, which hinders me from assigning a higher rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)