Unveiling LoRA Intrinsic Ranks via Salience Analysis
Authors: Wenjun Ke
Jiahao Wang
Peng Wang
Jiajun Liu
Dong Nie
Guozheng Li
Yining Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-Efficient Fine-Tuning
Low-Rank Adaptation
LoRA
Primary area: optimization_for_deep_networks

Abstract: The immense parameter scale of large language models underscores the necessity for parameter-efficient fine-tuning methods. Methods based on Low-Rank Adaptation (LoRA) assume the low-rank characteristics of the incremental matrix and optimize the matrix obtained from low-rank decomposition. Although effective, these methods are constrained by a fixed and unalterable intrinsic rank, neglecting the variable importance of matrices. Consequently, methods for adaptive rank allocation are proposed, among which AdaLoRA demonstrates excellent fine-tuning performance. AdaLoRA conducts adaptation based on singular value decomposition (SVD), dynamically allocating intrinsic ranks according to importance. However, it still struggles to achieve a balance between fine-tuning effectiveness and efficiency, leading to limited rank allocation space. Additionally, the importance measurement focuses only on parameters with minimal impact on the loss, neglecting the dominant role of singular values in SVD-based matrices and the fluctuations during training. To address these issues, we propose SalientLoRA, which adaptively optimizes intrinsic ranks of LoRA via salience measurement. Firstly, during rank allocation, the salience measurement analyses the variation of singular value magnitudes across multiple time steps and establishes their inter-dependency relationships to assess the matrix importance. This measurement mitigates instability and randomness that may arise during importance assessment. Secondly, to achieve a balance between fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series window, which adaptively controls the size of time-series for significance measurement and rank reduction during training, allowing for rapid rank allocation while maintaining training stability. This mechanism enables matrics to set a higher initial rank, thus expanding the allocation space for ranks. To evaluate the generality of our method across various tasks, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks. Experimental results demonstrate the superiority of SalientLoRA, which outperforms state-of-the-art methods by 0.96\%-3.56\% on multiple datasets. Furthermore, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5\% compared to AdaLoRA. The code is publicly available at https://github.com/Heyest/SalientLoRA.

Unveiling LoRA Intrinsic Ranks via Salience Analysis
Wenjun Keli2, Jiahao Wang', Peng Wang! Jiajun Liu', Dong Nie², Guozheng Li', and Yining Li'
Schooo of Computer Science and Engineering, Southeast University
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education 'Meta Inc.
(kewenjun, wang_jh, pwang, jiajii,, gzli, liyining) @ @euu.ddu.en, dongnie@cs.uncc.o
Abstract
The immense parameter scale of large language models underscores the necessity for parameter-efficient fine-tuning methods. Methods based on Low-Rank Adap- tation (LoRA) assume the low-rank characteristics of the incremental matrix and optimize the matrix obtained from low-rank decomposition. Although effective, these methods are constrained by a fixed and unalterable rank, neglecting the vari- able importance of matrices. Consequently, methods for adaptive rank allocation are proposed, among which AdaLoRA demonstrates excellent fine-tuning perfor- mance. AdaLoRA conducts adaptation based on singular value decomposition (SVD), dynamically allocating ranks according importance. However, it still struggles to achieve a balance between fine-tuning effectiveness and efficiency, leading to limited rank allocation space. Additionally, the importance measure- ment focuses only on parameters with minimal impact on the loss, neglecting the dominant role of singular values in SVD-based matrices and the fluctuations during training. To address these issues, the intrinsic ranks the weight matrix via salience measurement and adaptively optimizes ranks of LoRA. This method measures the salience of rank within a time-series by constructing ieperdeepesssscces among the correlations of singular values and prune ranks with low salience while retaining those with high signifi- cance. Additionally, an adaptive adjustment the time-series window enhances the speed of rank allocation while ensuring training stability. This mechanism enables matrics to set a higher initial rank, thus expanding the allocation space for ranks. To evaluate the generality of method across various tasks, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks. Experimental results demonstrate the superiority of SalientLoRA, which outperforms state-of-the methods by 0.96%-3.56 on multiple datasets. Furthermore, as the rank alloca- tion space expands, our method ensures fine-tuning efficiency, achieving a speed improvement 94.5% compared AdaLoRA. The code is publicly available http:://gttttbboom//yyyyes////st
propose SalientLoRA, which unveils
Introduction
Large language models (LLMs) 23
28
2 4 exhibit robust generative and inferential capabilities,
excelling in various downstream tasks 26
12 14 [6]. However, the vast number of parameters
in LLMs makes fine-tuning computationally demanding and time-consuming. Notably, LLaMA
23 encompasses parameters ranging from 7 billion to 65 billion, and fine-tuning LLaMA 65B necessitates substantial 780GB of GPU memory [5].
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Loss Fluctuations
CoLA
0.06
70.5
Mee
0.055
Singular Value
Runtime
Regu Loss
0.05
2.2K
Runtime (8)
70
0.05
Mce
0.04
1.9K
0.03
0.045
69.5
1.6K
Regu Loss Singular Value
0.02
0.04
69
1.3K
2
5
8
1.5rt
3rt
4.5rt 6 7.5rz
Intial Rank ri Figure 1: (Left) Fluctuations of regularization loss and singular values across multiple time steps. (Right) Performance and runtime of fine-tuning DeBERTaVa ---bse model on the CoLA dataset with increased initial rank AdaLoRA. Here, the metric Mee means Matthews Correlation Coefficient.
Time Step
To improve fine-tuning efficiency, numerous parameter-efficient fine-tuning (PEFT) methods [9]
13
have been developed with only a minimal number of trainable parameters, significantly reducing computational resources. Incremental methods [9] [3
18
I0
29 17
[8] introduce extra trainable parameters into the existing architecture of LLMs. However, the increased model depth introduces time delays during inference. Reparameterization methods [TO] 29 17
34 [5], based on
low-rank adaptation (LoRA) [10], assume the low-rank nature of the incremental matrix and perform low-rank dccomposition on it. Although effective, they still constrained by fixed intrinsic rank, potentially limiting their adaptability to the dynamic significance of different matrix elements.
To address the aforementioned limitations, dynamic rank allocation methods 25
6 29
have been
proposed, which can be seamlessly incorporated into LoRA-based PEFT approaches to significantly enhance adaptability. Among these methods, AdaLoRA (29 stands out, having demonstrated
excellent fine-tuning performance, as evidenced by its widespread adoption numerous research [29] introduces singular value decomposition (SVD) to the incremental matrices, employing sophisticated importance-aware methods (30 3 to adaptively allocate intrinsic ranks. This method improves the alignment of ranks with the dynamic importance of matrix elements. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and fine- tuning efficiency in the following two aspects: (1) The importance assessment focuses only parameters with minimal impact on loss, thereby neglecting the dominant role of singular values in the SVD matrix. This assessment also fails to consider the variability of parameters across multiple time steps, rendering it vulnerable to training fluctuations and probabilistic instability. F (left) illustrates significant fluctuations in singular values and the regularization loss of orthogonality, with a sharp increase and decrease of regularization loss from steps 3 to 8. Such variability compromises the reliability of the importance measurement. (2) AdaLoRA starts with specific initial rank and adjusts rank allocation by trimming unimportant parameters in groups during the fine-tuning process. limited Despite this, it struggles balance fine-tuning efficiency and effectiveness, resulting rank allocation space. Figure [ (right) illustrates that increasing the initial rank from 1.5 to 7.5 times the target rank r; improves fine-tuning performance by 0.78%. However, the rank allocation space expands, the fine-tuning time nearly doubles.
studies 33 AdaLoRA
In this paper, we argue that in LoRA optimization, the inter-dependencies among multidimensional ranks are crucial for unveiling the intrinsic ranks, especially when a specific rank significantly influences others. As consequence, we propose SalientLoRA, method that conducts salience analysis within a time-series LoRA ranks, pruning ranks with low salience while retaining those with high significance. Following AdaLoRA, we decompose the incremental matrix using SVD through regularization constraints. The salience measurement calculates the correlation between singular values across modules in time-series and constructs a dependency graph of correlation relationships. This graph reveals the inter-dependencies of ranks, assigning higher salience to rank with a broader influences domain. The salience measurement also examines the variation of the regularization loss of orthogonality and singular values in the time-series, mitigating instability and randomness that may arise during training. Moreover, balance fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series windows. This mechanism dynamically controls the size of time-series for salience measurement and rank reduction, facilitating rapid rank allocation while preserving training stability. This mechanism allows matrices to set higher initial rank, thereby expanding the rank allocation space with greater efficiency. To validate the generality of our method, conduct experiments on the GLUE dataset for NLU tasks, the XSum and CNN

---

datasets for NLG tasks, and the MT dataset for instruction tuning, separately fine-tuning encoder- only (DeBERTaV3-base) encoder-decoder (BART-large and T5-base,, and decoder-only (LLaMA) models. Experimental results demonstrate the superiority of our approach, which outperforms other existing fine-tuning methods by 0.96%-3.566 across multiple datasets, achieving state-of-the-ae results. Moreover, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving speed improvement of 94.5% 5 compared to AdaLoRA.
2 Related Work
2.1 Parameter-Efficient Fine-Tuning
Parameter-efficient fine-tuning methods (PEFT) can be divided into two categories; incremental
13 and reparameterization approaches TO 29 17]. Incremental methods add small neural modules into the existing architecture of LLMs, focusing solely on training these newly integrated modules. Adapter- Tuning [9] inserts simple Adapter modules after the feedforward layer of each Transformer architecture. Prefix- - T'nnig [33 and Prompt- Tunpng [18] introduce additional trainable prefix vectors before the input layer or hidden layers. However, these approaches introduce time delays during inference due to either increasing model depth or reducing the available input sequence length. Additionally, they still exhibit performance discrepancies compared to full- full-parameter fine-tuning approaches. Reparameterization techniques update the incremental weight matrix in a parameter- efficient manner, without altering the existing architecture of models. LoRA [10] assumes the low-rank nature of the incremental matrix and performs low-rank decomposition on it. DoRA 17 decomposes pre-trained weights into magnitude and direction for fine-tuning, using low-rank decomposition for directional updates. Although the LoRA-based reparameterization approaches effective, they assign the same rank to all weight matrices, failing to consider the varying importance of weight matrices across different layers and modules.
2.2 Dynamic Rank Allocation
Intuitively, more important matrices are capable of learning more complex knowledge and thus require higher rank. Therefore, the dynamic rank allocation methods 25 29] are necessary, which can be seamlessly applied to the LoRA-based approaches. To achieve the dynamic low-rank adaptation, DyLoRA 225 trains LoRA blocks for range of ranks by sorting the representation
leamned by the adapter module at different ranks during training. SoRA [6] controls rank cardinality under gate sparsity by integrating gate unit optimized through the proximal gradient. AdaLoRA [29] performs adaptation through singular value decomposition (SVD), dynamically adjusting intrinsic ranks based on significance. Among these methods, AdaLoRA has demonstrated its effectiveness, as proved by numerous studies and experiments. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and fine-tuning efficiency.
Overview
As illustrated in Figure 2 the overall architecture of our method comprises three parts: incremental matrix SVD, salience measurement and adaptive rank allocation.
Firstly, given the pretrained weight Wo d xda the update process be formalized as W Wo A, where A represents the incremental updates. To reduce the parameter count and facilitate rank allocation, we decompose A into PVQ, where V Rexr is diagonal matrix, P E Rd and Q E RrXd2, with rank r < {d,, d2). Following AdaLoRA, the orthogonality of P and Q is regularized through the auxiliary loss to simulate singular value decomposition (SVD). The regularization loss R(P, Q) defined as follows:
R(P. Q) |P' 1| +|Q00 - -I
(I)
Through this decomposition approach, the rank of the incremental matrix can be easily controlled by zeroing out the singular values in V. Secondly, we measure the salience of singular values within time-series through orthogonality-aware singular value magnitudes and the influence domain, unveiling the inter-dependencies among ranks. Thirdly, singular values are sorted by their salience at the last step of cach time-scries, and those with low significance are zeroed out to achieve the
3

---

Incremental Matrix
Moasuroment
Adaptive
Salience of Singulor Value
A
P
Q
P
AE
Query
Query
71
Q
Ao...
Singuler
Dependency Graphh
P
Step
Of 1
Step
A+
Query
pd
X5
Q4
pd
Ag
Q
02 90
pk
Ag
Q**
1
Donse
Denee
Key
Orinogonality
Ormogonaltyy
Figure 2: The overall framework SalientLoRA. First, the incremental matrix is decomposed using SVD to facilitate rank allocation. During fine-tuning, the salience of singular values within time-series measured, which composed of orthogonality-awar singular value magnitudes and the influence domain of the dependency graph. Finally, in the rank allocation process, singular values with lower significance are progressively trimmed.
adaptive rank allocation. The rank allocation space is determined by the initial total rank r; and the target total rank Tt, both of which are hyperparameters. Upon completion of the rank allocation, the most critical r, ranks retained. In this process, adaptive time-series window is introduced to accelerate the rank allocation process and maintain stability during training. Section 4 elaborrae on the Salience Measurement and the Adaptive Time-series Window, respectively.
Salience Measurement
4.1 Orthogonality-Aware Singular Value Magnitudes
Following the objective of SVD, which utilizes singular values to represent the primary charac- teristics of matrix, we assess their importance based on the magnitudes. However, the relia- bility of SVD heavily relies the orthogonality of P and Q. Inadequate orthogonality under- mines the efficacy of importance assessment with magnitudes. Therefore, we consider variations of regularization losses within time-series, assigning weights to singular value magnitudes ac- cording to the orthogonality. Specifically, for the singular value Xa, we record its magnitude (m)) and regularization losses of the corresponding incremental matrix R(P!" ). R(P! Q )..... R(P(") Q!")) within the n time steps. A high regularization loss indicates low reliability of the SVD at that step, and consequently, the weight assigned to it
Xa = (x(1) (1) x
should be small. Therefore, the weight of step calculated follows:
up ( E. R(P! QQ R(P!)) Q!")
(2)
Subsequently, we normalize the weights with Min-Max normalization, then calculate the orthogonality-aware singular values magnitudes:
(
ma
4.2 Influence Domain of Singular Value
During training, there exist dependency relationships among singular values, wherein the variation of one singular value can influence multiple others. In light of this, we construct the dependency graph of the correlations between singular values and determine their significance through the influence domain. Initially, the correlation is calculated between any pair of singular values within the time

---

windows Xo (1/ 100 ..... ( and X, (A/ .1.0 ....., .1a0) by the Pearson coefficient:
Cov(Aa, A6)
Pob vD(Aa:>b)
(4)
If the Pob exceeds B, it indicates correlation between X and Xg. For correlated pair of singular values, the slope hab between them is further computed through linear regression fitting to determine the extent of influence of d As- larger slope kab indicates greater influence of Aa Ab, signifying dependency relationship between them. Thus, we obtain the dependency matrix D Rmxm and the element Das of D calculated follows:
Kab 10
Pub > 3 and kab > >g
Dan
(S)
clse
where denotes the total number of singular values and is the threshold for filtering dependency relationships. Note that the Pearson correlation coefficients and slopes are calculated using matrix parallel multiplication, incurring only minimal computational and time costs. Subsequently, we trans- form matrix D) into dependency graph, with singular values as nodes and dependency relationships as edges. The dependency graph among singular values forms directed cyclic graph, containing numerous redundant dependencies within the cycles. Therefore, employ a depth-first search (DFS) algorithm for de-cycling eliminate these redundancies. The pseudocode of de-cycling algorithm provided in Algorithm
Finally, based the dependency graph, the influence domain of singular values computed:
node i without subsequent nodes 4 = Ex Dark node i with subsequent node A
(6)
As illustrated in the dependency graph of Figure some singular value determines multiple others, with its variation can consequently lead to changes in several other singular values, thus signifying higher level of importance.
Ultimately, we combine the influence domain with orthogonality-aware magnitudes to obtain the salience for each singular value. Here, the Å is hyperparameter that controls the contribution of two components to the salience measurement. The salience of the A is denoted as s.:
Am, + (1 A)
Algorithm De-Cycling Algorithm for Dependency Graphs
Input: Dependency graph G
Output: Directed acyclic graph G'
I: Path stack()
& Initialize stack record traversal path Record nodes that have heen visited
Visited =
3: for each node v in do
4: if not in Visited then
5: DFS VISIT(w)
end if
7: end for
8:
9: def DFS_DISSSI VISIT(v)
Visited.add(e)
11: Path.push(v))
12:
for each node adjacent to v do
13:
if not in Visited
b If u is in the path, cycle has been detected.
14:
DFS VISIT(u)
else
Path
16:
Remove the edge with the smallest weight in the cycle end if
17:
18:
end for
19:
Path.pop(v)
5

---

5 Adaptive Time-Series Window
During the rank allocation for ft steps, we implement an adaptive adjustment of time-series window. Within each time-series, the salience of singular values is assessed and the unimportant singular values are dropped at the last step of the time-series. This mechanism follows the principle that during the early stages training, rank adjustment can be swiftly executed, while in the later stages, rank allocation should be approached more cautiously. Specifically, we initially maintain small time window during the carly stages of training, facilitating rapid rank reduction to enhance rank allocation efficiency. The pruned singular values and their corresponding singular vectors excluded from gradient updates and salience calculations to expedite fine-tume tuning. As training progresses, the time window expands, and the allocation becomes more cautious. The broad time window reduces the frequency of importance calculations, thereby further accelerating the fine-tuning process. The adjustment of the time window is follows:
T2 ns
T
(8)
where T, Tú and T represent the current time window size, the initial time window size, and the final time window size during the rank allocation phase, respectively, with T, Ty. ne denotes the current training step during the rank allocation phase.
The model starts with relatively high initial rank r; and zeros certain number of singular values within each time window to allocate rank. At the end of the rank allocation process, the rank is pruned the target total rank rt- The total rank count maintained in each time window is:
T Tj
T=
x (r: - -r))
(9)
Experiment
6.1 Experimental Settings
Datasets To evaluate the applicability of our fine-tuning approach across multiple tasks and various models, we conduct experiments natural language understanding (NLU), natural language genera- tion (NLG), and large-scale model instruction fine-tuning tasks, respectively. The specific datasets chosen for each task and the statisties will be detailed in Section 6.2 6.3 6.4
and Appendix [
Baselines We choose full fine-tuning (Full FT) and several existing incremental and reparameter- ization methods baselines. Full FT denotes fine-tuning all parameters of the model, requiring significant computational resources. Adapter [9] integrates additional neural network layers into the model, only fine-tuning the newly added parameters. AdapterFusion (20] propose more efficient design with adapters only applied after FFN and LayerNorm modules [3]. LoRA (10) performs low-rank decomposition
the incremental parameter matrices, substantially reducing the number of parameters for fine-tuning. SoRA 6 introduces gating module with a proximal gradient decent update to control the sparsity of the updated matrices. AdaLoRA 29) dynamically distributes the parameter budget among weight matrices by evaluating the importance. DoRA [17] decomposes the pre-trained weights into magnitude and direction for fine-tuning, using LoRA for directional updates.
Settings Our experiments are conducted on four NVIDIA RTX 3090Ti GPUs for NLU and NVIDIA Ampere 1100 for NI. and instruction tuning tasks. During salience measurement, the slope threshold for dependency calculation 7 2, the correlation threshold 3 = 0.9, and the À is set to 0.7. For adaptive tine-window adjustment, the initial time window size T; = 10, the final time window size T' 200, and the initial total rank r; is set to 7.5 times the target total rank Tt. The selected weight matrix of transformer layer for fine-tuning includes query/key/value projection (Wg. W.. W.), output projection (W.) in the self-attention, and two weight matrices (W,, , Wy.) in two feedforward layers (FFNs). We select the learning rate from (8 x 10-5, 5 10-5, x 10-5, 1 x 10-1, 10-1.5 x 10-1,8 x 10-4,1 1 x 10-3), and pick the rest-performing learning rate for every method. Further details other hyperparameters shown in Appendix

---

6.2 Natural Language Understanding
benchmark utilizing the encoder-only model DeBeRTaV3-base 77]. The GLUE benchmark consists of eight datasets: CoLAA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We use Matthew's correlation coefficient, Spearman's 's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. To compare under the same parameter budget, we set the hidden dimensions d of adapters to and 32, the rank r of LoRA and DoRA to 2 and and the target total ranks ry for AdaLoRA and SalientLoRA to 144 and 276, respectively.
For NLU, we evaluate our method on the GLUE 27
Table 1: Experimental results of SaRientLoRA and other baselines on the GLUE benchmark across varying parameter budgets. The bold scores indicate the best results. We report the average perfor- mance over runs using different random seeds, with SalientLoRA significantly better than AdaLoRA and DoRA with p-value < 0.05 based on paired t-test.
Method
#Params
MNLI
SST-2
CoLAA
QQP
QNLI
RTE
MRPC
STS-B
Ave.
Full FT
184M
89.98
95.64
69.21
92.05
93.78
82.49
89.22
91.59
88.00
Adapter
0.31M
90.10
95.41
67.65
91.54
93.52
83.39
89.25
91.31
87.60
AdapterFusion/ss
0.30M
89.89
94.72
69.06
91.40
93.87
84.48
89.71
91.38
87.90
LoRA,=2
0.32M
90.04
94.95
68.71
91.61
94.01
85.31
89.65
91.58
88.23
DoRA,
0.34M
90.14
95.78
70.21
91.77
94.17
87.48
90.27
91.24
88.88
AdaLoRA,,EMn
0.32M
90.22
95.76
70.04
91.78
94.13
87.36
90.13
91.21
88.83
SalientLoRA,
1,,
0.32M
90.94
96.23
71.87
91.97
94.83
88.43
91.68
92.36
89.79
Adapter 2-32
1.21M
90.13
95.53
68.64
91.91
94.11
84.48
89.95
91.48
88.28
AdapterFusion,
1.18M
90.33
95.61
68.77
92.04
94.29
85.20
89.46
91.54
88.41
SoRA
0.91M
90.35
95.64
71.48
92.39
94.28
87.77
91.98
92.22
89.36
LoRA,=S
1.27M
90.47
95.67
69.73
91.95
93.76
85.32
89.71
91.86
88.56
DoRA,=8
1.29M
90.37
96.02
71.46
92.36
94.47
87.74
91.15
92.12
89.46
AdaLoRA, 2,72
1.27M
90.27
95.95
70.86
92.13
94.28
87.36
90.22
91.39
89.06
SalientLoRA, ,,
1.27M
91.07
96.58
72.68
92.41
95.04
88.93
92.34
92.76
90.23
The results are shown in Table [ Firstly, compared to incremental methods, reparameterization approaches generally yield better results. Under the parameter budget of 0.3M, LoRA and DoRA outperform adapters by average scores of 0.63% and 1.28% respectively. This demonstrates the superiority of reparameterization methods, achieving enhanced fine-tuning performance without adding extra inference latency. Secondly, both AdaLoRA and SalientLoRA consistently outperform LoRA across all eight datasets, showing an average improvement of 0.6% and 1. 1.56% in 0.32 million parameters, and 0.5% and 1.67% 1.27 million parameters. respectively. This suggests that adaptively adjusting the rank based on the importance of matrices indeed leads to better fine-tuning performance. Third, SalientLoRA surpasses all other fine-tuning methods with margins of 0.91 %- 1.79% and 0. 0.77%% under two parameter budgets, achieving state-of-the-ar results. Specifically, SalientLoRA outperforms AdaLoRA by 0.96% and 1.17% on average, which indicates the superiority of our method for salience measurement.
6.3 Natural Language Generation
For the NL. task, we finetune the encoder-decoder model BART-large 11 and T5-base 21 on the text summarization datasets XSum [19 and CNN/DailyMail (8), with evaluation metrics ROUGE 1/2/L scores. We set the rank of LoRA and DoRA to 2, and the target total ranks r; for AdaLoRA and SalientLoRA to 144, respectively. Fine-tuning results on datasets are shown in Table 2 The experimental results indicate that under the same parameter budget, SalientLoRAA achieves the best performance when fine-tuning T5-base and BART-large, surpassing other methods by margins of 0.36%-2.2%. Specifically, when fine-tuning the BART-large model on the XSum dataset, SalientLoRA exceeds A by 0.64%, 0.86% and 1 % ROUGE 1/2/L scores, respectively, and outperforms DoRA by 0.54%, 0.36%, and 0.65%. Moreover, SalientL.oRA A even surpasses the performance of full fine-tuning on the CNN/DailyMail dataset, with improvements of 1 1.42%, 0.1 1.18% and 1.51% % by b5-base and 0. 0.78%, 0.58%, and 0.88% by BART-large, respectively.
7

---

Table 2: Performance comparison of different fine-tuning methods NLG tasks. The three metrics on each dataset are ROUGE 1/2/1. scores.
Model
Method
#Params
XSum
CNN/DailyMail
Full FT LoRA
212.6M
38.81
16.50
31.27
42.05
20.34
39.40
0.34M
36.35
14.16
28.88
41.27
19.33
38.76
T5-base
AdaLoRA
0.34M
36.68
14.27
28.13
41.53
19.52
39.01
SoRA 0.46M
36.87
14.54
28.32
41.78
19.86
40.18
DoRA
0.36M
36.89
14.68
28.49
42.92
20.03
40.28
SoR
0.34M
37.36
15.03
29.14
43.47
20.52
40.91
Full FT
375.5M
45.49
22.33
37.26
44.16
21.28
40.90
LoRA
0.60M
42.81
19.68
34.73
43.68
20.63
40.71
BART-large
AdaLoRA
0.60M
43.29
19.95
35.04
43.94
20.83
40.96
SoRA
0.72M
43.46
20.27
35.28
44.21
21.21
41.18
DoRA
0.64M
43.39
20.45
35.39
44.35
21.34
41.34
SalAentLoRA
0.60M
43,93
20.81
36.04
44.94
21.86
41.78
6.4 Instruction Tuning
To evaluate the effectiveness of method in fine-tuning large language models of the decoder-only architecture, we fine-tune the LLaMA-7B [23] and LLaMA2-7B [24] model on the Alpaca [22] instruction dataset, which consists of 52k instances generated by GPT-4 based on inputs from Alpaca. The fine-tuned
Table 3: The average score on MT-Bench and train- able parameter count of L.LMA-7BB after instruc- tion tuning by different fine-tuning methods.
Model
Method
#Params.
Score
Full FT
6426.3M
5.12
LoRA
159.9M
4.89
LLaMA and LLaMA2 are evaluated on the MT-
LLaMA
AdaLoRA
1.599M
5.06
Bench 32 generating model responses to pre-defined set of 80 high-quality, multi-turn questions. These responses are then assessed using GPT-4, which assigns quantitative score on scale of 10 to each answer. We set the rank of LoRA and DoRA to 64, and the target total ranks r, for AdaLoRA and SalientLoRA to 12280, respectively. We present the aver- age scores alongside the number of trainable parameters in Table All compared fine-tuning methods utilized only 159.9M to 163.7M trainable parameters, which is merely 2.4% of Full FT. Experimental results indicate that LoRA still lags behind Full FT by 0.23 and 0.2 points. Notably, Salientl A surpasses all fine-tuning methods, even outperforming Full FT by 0.12 and 0.17 points, demonstrating the superior performance of SoR.
SoRA
163.7M
5.11
DoRA
161.2M
5.17
SalientLoRA
159.9M
5.28
Full FT
6426.3M
5.25
LoRA
159.9M
5.05
AdaLoRA
159.9M
5.17
LLaMA2
SoRA
163.7M
5.24
DoRA
161.2M
5,31
SalientLoRA
159.9M
5.42
6.5
Ablation Results
To validate the effectiveness of the components in our salience measurement, we conduct experiments on the full SalientLoRA and its variants. Here, OAM represents the orthogonality-wwar magnitude and ID represents the influence domain. SalientL.RRA w/o OAM indicates using the mean of magnitudes within a time-series without adjusting weights based on orthogonality. We follow the experimental setup for the NLU task, setting the target total rank r; of SalientLoRA 144. As shown in Table we observe that both components of salience measurement (OAM and ID) significantly contribute the final results. SalientLoRA w/o OAM experiences a performance decrease ranging from 18% to 0.73% across the eight datasets, with an average performance drop of 0.48%. This indicates that adaptively controlling the weights of magnitudes based on the matrix orthogonality effectively alleviates the problem of unreliable salience measurement caused by the orthogonality fluctuation during training. Additionally, SalientL.oR w/o ID shows a performance decline ranging from 0.32% 1.13% across the eight datasets, with average decrease of 0.8%. This highlights the crucial role of ID in performance enhancement. ID constructs dependency between singular values and further effectively measures importance by calculating the influence domain.

---

Table 4: The results of ablation experiments. Here, 4 represents the performance declines of variants.
Method
MNLI
SST-2
CoLA
QQP
QNLI
RTE
MRPC
STS-B
Ave.
SalientLoRA,
90.94
96.23
71.87
91.97
94.83
88.43
91.68
92.36
89.79
w/o OAM
90.36
95.87
71.14
91.79
94.28
88.04
91.05
91.94
89.31
0.48
w/o ID
90.12
95.59
70.56
91.65
94.11
87.52
90.63
91.76
88.99
0.8
6.6 Analysis Space Allocation and Time Consumption
We remark that as the rank allocation space expands, both SalientLoRA and AdaLoRA exhibit performance improvements, and SalientL.oRA achieves a better balance between performance and fine-tuning time due to the adaptive time-series window and parallelized computation. Given the resource-intensive nature of the experiments, from NLU tasks for validation. We set the target total rank Tt at 144 and gradually increase the average initial rank for each matrix from 3 to 18. Here, of matrices fine- tunedt and r; denotes the initial total rank. The results in Figure 3 indicate that as the initial rank increases, AdaLoRA shows performance improvements by 0.39% 1.44%, while SalientLoRA improves by 0.55% 1.5%. Furthermore, the fine-tuning time of AdaLoRA significantly rises by 108% on average, whereas SalientLoRR, experiences only marginal increases, ranging from 7% to 9%. This highlights the superiority of our method in balancing fine-tuning performance and efficiency by the adaptive time-series window and parallelized computation during salience measurement, resulting in reduced fine-tuning time.
choose the CoLA, RTE, STS-B, and MRPC datasets
f. where represents the number
CoLA
RTE
STS-B
MRPC
88.5
6.0K
92.4
3.7K
-4.8K
71.5
71
3.0K
2.4K
-4.0K OKOKK
5.0K
92.1
3.IK
2.5K
91.5
4,2K
3.6K
Runtime
70.5
1.8K
87.5
3.0K
91.8
3.0K
1.KK
2.0K
91.5
1.3K
90
1.8K
3
9 12
6
9
12
15
18
12
18
3
6
12
15
18
Average Initial Rank
Average Initial Rank
Average Initial Rank
Initial Rank
AdaLoRA
SalientLoRA
AdaLoRA
SalienLoRRA
Figure 3: Comparison of fine-tuning effectiveness and runtime for AdaLoRA and SalientLoRA across multiple datasets as the rank space increases. The line graph illustrates their the fine-tuning performance, while the bar chart depicts the fine-tuning time.
6.7 Analysis of Hyperparameter
This section investigates the impact of the different contributions of two components salience measurement on the final fine-tuning results, namely orthogonality magnitude and influence domain. The degree of contribution is controlled by the hyperparameter magnitude to the salience measurement. Conversely, 4sow
higher À indicates
greater contribution of orthogonality- as \ decreases, the influence domain plays a greater role. The results in Figure 4 show an upward trend in fine-tuning performance across CoLA, RTE, STS-B, and MRPC datasets as À increases, reaching a peak when À reaches 0.7. This suggests that a greater contribution of orthogonality-aware magnitude leads to more effective salience measurement. The primary reason lies in the SVD decomposition of the incremental matrix, where the magnitude of the singular values still represents the most fundamental factor of matrix characteristics. However, when À further rises, fine-tuning performance declines. This suggests that when the contribution of orthogonality- aware magnitude slightly outweighs that of the influence domain (i.c., when À approaches 0.7), salience measurement of singular values can achieve optimal performance, yielding the best fine-tuning results. This also demonstrates the effectiveness of both components in the salience measurement.
6.8 Salience Illustration across Different LoRA Ranks
In this section, we visualize the salience of singular values and analyze their distribution across different layers and modules in models. Here, we primarily illustrate the influence domain of singular values regarding inter-dependencies. Due to the presence of multiple singular values in each weight matrix, average and normalize the influence domains for easier presentation. Figure 5 depicts the

---

CoLA
RTE
STS-B
MRPC
73
93
72.7
Rlas AR 0588
92 2024
92.2
71.8
71.2
70.9
72.4
71.5
72.1
71.99
72.23
71.51
71.07
--,,
Tc
276
87.8
88.1
88.7
88.4
55.37
276
Coff
91.8
92.7
01.96
92.86
92.15
02.27
-- 7 276
-- Tc 144
92.2
91
91.2
91.6
91.8
91
91.36
90.51
92.13
91.6
rc-
276
0.3 0.4
0.8
0.3
0.4
0.5
0.6 0.7 0.8
0.4
0.6 0.7 0.8
0.3 0.4 0.7 0.8
Hypeppammeter
Hyperperameter À
Hyperparameter
Hyperperameterr
Figure 4: Variations of fine-tur fine-tuning performance with different X, which controls the contributions of two components in salience measurement: orthogonality magnitude and influence domain.
CoLA
Wa
3.44
10:74
0.36
W
0.33
0.64
12.67
W.
0.67
0.38
0.72
01:72
0.5
W.
0.7
0.38
9.47
0.5
0.56
0.77
0.84
0.81
0.8
0.5
0.76
0.77
W
0.37
0.43
0.29
0.53
0.53
0.63
9.76
0.71
0.
0.74
0.73
0.71
W.
0:58
0.79
-0.52
0.57
0:75
0.77
0.7
W/
0.52
0.66
0.69
0.27
0.3
0.55
0,85
0.82
0.84
0.83
0,89
0.87
Wx
0.58
0.73
9.71
0.82
0.74
0.85
0.82
0.35
0.55
0.85
0.89
0.53
W2
8.19
0.13
0.34
0.46
0.57
0.775
00.84
0.77
0..3
0.87
0.85
0.77
W/
0.25
0.27
0.36
0.58
0.64
0.72
0.81
0:55
0.94
0.366
0.88
0.35
10
12
10
12
Layer
Luyer MR
STS-B
W.
015
0.34
0.17
0:25
0.46
0.550
0.51
0.53
0.51
0.49
0.45
0.27
Wg
0.12
0.25
0.17
0.57
0.49
0.54
0.61
0.6
0.55
0.72
0.43
0.444
0.4
W.
023
0:51
0.2
0.39
0.32
0.57
0.62
0.66
58
0.6
1.67
0.64
W:
0.22
0.18
0.59
0.45 0.69
0.67
0.
74
0.77
0.63
0.55
0.52
0.69
0.76
0.71
0.3
W₂
AA6
0.38
0.47
0.52
0.65
0.:90
0.79
0.71
89
0.68
85
0.80
W
4.
0:75
0.53
0.57
6R
0.76
0.72
0.88
85
0.86
8.59
W/
Wr
0.74
0.84
0:85
0.50
01300
W,
6.79
0:81
Wg
0.26
0.59
0:75
0:77
0.9
0.87
2
8
6
8
10
12
2
6
10
12
Luyer
Layar
Figure 5: Distribution of influence domain across different layers and modules when fine-tuning DeBERTaV3-base model on different datasets.
distribution of the influence domains after the completion of rank allocation for DeBERTaV3-bass on different datasets. The results reveal that deeper layers and Feedforward Network (FFN) modules generally possess larger influence domains compared to shallower layers and other components. This indicates that the variations of singular values in these modules can significantly influence other parameters, therefore possessing a higher importance. This observation also aligns with the empirical conclusions AdaLoRA that weight matrices of FFN modules and deep layers are more crucial for model performance. Therefore, this validates the effectiveness of the influence domain for salience measurement identify important parameters.
7
Conclusion
This paper proposes SalientLoRA, novel adaptive rank allocation method for LoRA-based PEFT by salience analysis. This method measures the salience rank within time-series by constructing inter-dependencies among the correlations of singular values and prune ranks with low salience while retaining those with high significance, thereby unveiling the intrinsic ranks of the weight matrix. Additionally, an adaptive adjustment of the time-series window enhances the speed of rank allocation while ensuring training stability. Extensive experiments conducted natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks demonstrate that our method achieves state-of-the-art fine-tuning performance, effectively balancing fine-tuning efficiency and performance.
10

---

Acknowledgments
We thank the anonymous reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos. (33.67370057) the Start-up Research Fund of South- east University (RF1028623234) and the Fundamental Research Funds for the Central Universi- lies(22420244300355, All opinions are of the authors and do not reflect the view of sponsors.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2303::2300008874, 2023.
[2] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint 2rXiv:2305.00000 2023.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint ar0iv 06450, 2016.
[4] Mann Benjamin et al. Brown, Tom. Language models are few-shot learners. In NeurlPS, 2020.
[5] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuming of quantized lims. arXiv preprint arXiv:2305.1433) 2023.
[6] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In EMNLP, 2023.
[7] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhann enhanced bert with disentangled attention. ICLR, 2021.
[8] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. In NeurIPS, 2015.
[9] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
[10] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In ICLR, 2022.
[11] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequencceto-seqqeeee pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
[12) Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation extractors. In EMNLP, 2023.
[13] Xiang Lisa Li and Percy Liang, Prefix-tuning: Optimizing continuous prompts for generation. ACL, 2021.
[14] Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, and Yanhe Liu. Towards continual knowledge graph embedding via incremental distillation. In AAAI, 2024.
[15] Jiajun Liu, Wenjun Ke, Peng Wang. Jiahao Wang, Jinhua Gao, Ziyu Shang. Guozheng Li, Zijie Xu, Ke Ji, and Yining Li. Fast and continual knowledge graph embedding via incremental lora. In IJCAI, 2024.
[16] Jiajun Liu, Peng Wang, Ziyu Shang, and Chenxiao Wu. Iterde: an iterative knowledge distillation framework for knowledge graph embeddings. In AAAI, 2023.

---

[17] Shih- Yang Liu, Chien-Y Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min- Minn Chen. Dora: Weight-decompoosed low-rank adaptation. arXiv preprini 00X2v32402.09333, 2024.
[18] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022.
[19] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In EMNLP, 2018.
[20] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In EACL, 2021.
[21] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn, Res., 21:140:1-140:66 2019.
[22] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: an instruction-following llama model (2023). URL https://github. com/hatsu-lab/stanffor pllpaa 2023.
[23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09220 2023.
[25] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In EACL, 2023.
[26] Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. GPT-RE: In-context learning for relation extraction using large language models. In EMNLP, 2023.
[27] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[28] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In ICLRR 2023.
[29] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, 2023.
[30] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Super tickets in pre-trained language models: From model compression to improving generalization. In ACI., 2021.
[31] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In ICMI., 2022.
[32] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLMe with MT- and chatbot arena. In NeurlPS, 2023.
12

---

[33] Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. AutoPEFT: Automatic configuration search for parameter-effiien fine-tuning, Transactions of the Association for Computational Linguisties, 12:525-544 542, 2024.
[34] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint 20Xhv:2309.02411, 2023.
13

---

A Dataset Statistics
The GLUE benchmark consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, B, MNLI, QNLI, and RTE. We use Matthew's 's correlation coefficient, Spearman's 's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. Table S displays the statistics of the GLUE benchmark.
Table 5:
The statistics of GLUE benchmark. Here, Mec denotes Matthews correlation coefficient and
See denotes Spearmans correlation coefficient.
Dataset
#Train
#Test #Dev Metrics
Single-Sentence Classification
COLA
8.5k
1K
1k
Mec
SST
67K
872
1.8K
Accuracy
Pairwise Text Classification
MNLI
393K
20K
20K
Accuracy
RTE
2.5K
276
3K
Accuracy
QQP
364K
40K
391K
Accuracy
MRPC
3.7K
408
1.7K
Accuracy
QNLI
108K
5.7K
5.7K
Accuracy
Text Similarity
STS-B
7K
1.5K
1.4K
See
The CNN/Daily Mail dataset serves as a corpus for single-document generative summarization, comprising news articles collected from CNN and Daily Mail, with each summary containing multiple summary sentences. XSum dataset consists of BBC articles and accompanying single sentence summaries. The statisties for the CNNi and XSum datasets shown in Table 6
Table The statistics of XSum and CNN/Daily datasets.
Dataset
#Train
#Test
#Dev
CNN//DilyMail
286k
13K
11k
XSum
204K
11k
11K
B Experimental Analyses on Hy Hyperparameter
We conduct additional experimental analyses on two sets of hyperparameters 3, 7, T; and Ty to explore their impact. These experiments are conducted the CoLA and MRPC datasets to fine-tune the DeBERTaV3-bazz model. The total target rank is set to 144, and all the other parameters are consistent with the main experiments.
and y are thresholds that control the relevance and de- pendency relationships of singular values, respectively. To explore their effects, we keep other hyperparameters constant and change the values of 3 and separately to observe the timing and effects of SalientLoRA fine-tuning. The results are presented in the T where MCC repre- sents Matthew's 's Correlation Coefficient and Acc denotes accuracy. The performance exhibits minimal sensitivity
Table 7: Experimental results with differ- ent values of hyperparameters and %
CoLA
MRPC
3
7
MCC
Time
Acc
Time
0.5
2
71.87
29
91.65
42
0.7
2
71.82
24
91.57
34
to variations in the values of 3 and 7. This insensitivity
0.9
2
71.87
21
91.68
33
stems from the decycling operation for the dependency graph, which effectively eliminates the majority of redun- dant dependency relationships. However, setting exces- sively low values for 3 and can lead to a large number of redundant dependencies in the graph, which increases the time cost of the decycling process and thus impacts the efficiency of fine-tuning, There- fore, we select hyperparameter values that ensure high efficiency of fine-tuning, with 3=0.9 and 7=2.
0.9
2
71.78
27
91.63
39
0.9
1.5
71.83
34
91.67
47
14

---

T. and Ty control the sizes of the initial and final time windows in the adaptive time-series window mechanism, respectively. The results in Table indicate that the im- pact of 7; on performance is minimal, with differences only ranging from 0.02% to 0.09%. However, as 7; in- creases, the fine-tuning time significantly lengthens. This due to slower reduction in rank during the early stages of fine-tuning, which impacts the efficiency of rank al- location. Moreover, when T; remains constant and T increases from 100 to 250, there is slight improvement in performance, while the fine-tuning time remains rela-
Table 8: Experimental results with differ- ent values of hyperparameters T; and y
CoLA
MRPC
T: Ty
MCC
Time
Acc
Time
10
200
71.87
21
91.68
33
30
200
71.81
23
91.59
37
50
200
71.85
27
91.64
41
10
100
71.52
22
91.32
34
10
150
71.71
21
91.56
34
tively unchanged. This improvement can be attributed to the significance analysis in the later stages of fine-tuning, which incorporates singular values under more time steps, yielding more reliable allocation outcomes. Therefore, set 10 and Ty 200 to achieve balance between performance and efficiency.
10
250
71.84
23
91.64
35
C Hyperparameter Statistics
The hyperparameter settings for NLU, NLG and instruction tuning tasks are shown Table 9 Here, denotes the target total rank, n, represents the number of steps to warm up the training before rank allocation, and n indicates the number of steps during the rank allocation phase.
Table 9: The hyperparameter settings for NLU, NLG and instruction tuning tasks.
Task
Dataset
Learning Rate
Batch Size
Epochs
Tr
T1;
f1,
MNLI
5 x 10-4 44
32
7
144/276
8000
35000
RTE
8 X 10
32
50
144/276
600
1500
QNLI
8 x 10-1
32
5
144/276
2000
8500
MRPC
10-1
32
30
144/276
600
1000
NLU
QQP
3 10-4
32
5
144/276
8000
25000
SST-2
8 10-5
32
24
144/276
6000
20000
CoLA
1 10-3
32
25
144/276
800
2000
STS-B
8 10-1
32
25
144/276
800
2500
XSum
8 x 10
64
25
144
6000
25000
NLG
CNN/DailyMail
8 x 10-4
32
15
144
5000
90000
Instruction Tuning
Alpaca
5 x 10
12280
2000
6000
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. phippscccpubb... ic/i wwt.guuide//ooeeiiiieeeee
wwriiisssoooooiccyccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http////eeriispeeee http:///wwwpp...cp/oo///ccccttaa
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We respect the license of LLaMA and so on.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersii thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
20

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "VsvtqH4uNo", "reviewer": "Reviewer_xsy4", "summary": "This paper proposes SalientLoRA, a new method for adaptively optimizing the intrinsic ranks of low-rank adaptation (LoRA) matrices. The key ideas are:\n\nUsing singular value decomposition (SVD) to decompose the LoRA matrices and measure the salience/importance of each singular value based on its magnitude, orthogonality constraints, and influence on other singular values within a time window during training.", "questions": "1. Taking the last row of Table 1 as an example, initially, the model uses a total rank that is 7.5 times the target rank, so the gpu memory usage is roughly equivalent to that of LoRA with r=8*7.5=60. Although the memory usage may decrease during the model optimization process, can you consider comparing with methods like LoRA and DoRA with r=60?\n2. Based on my understanding, the constructed influence domains form an undirected simple graph. If this graph forms a single cycle, how do you perform de-cycling?\n3. Do you calculate influence domains starting from vertices with a degree of 0, similar to topological sorting, and then update the degrees of the vertices connected to it, then repeat the process?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel salience measurement technique that considers singular inter-dependencies and temporal variations.\n- Comprehensive evaluation across many datasets and model types (encoder, decoder, encoder-decoder).\n- Achieves new state-of-the-art results on multiple benchmarks while being more efficient than prior LoRA methods.", "weaknesses": "- The article contains some details that are not clearly explained, such as how the R function on line 145 is calculated, and what specifically is done in the de-cycling process introduced on line 165.\n- More analysis could be provided to interpret why the salience measurement works well. For example, are the average of influence domains consistent across models fine-tuned on different types of datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive feedback provided by the reviewer. We provide a detailed explanation and experimental analysis as follows.\n\n***W1: The article contains some details that are not clearly explained, such as how the R function on line 145 is calculated, and what specifically is done in the de-cycling process introduced on line 165.***\n\n(1) Function $R(\\cdot)$\n\nIn our paper, we have elaborated on the function $R(\\cdot)$ for calculating the orthogonalization loss in Equation 1. Specifically,  $R(\\cdot)$ measures the degree of orthogonality of $\\textbf{P}$ and $\\textbf{Q}$. The equation is as follows:\n\n$R(\\textbf{P},\\textbf{Q})=||\\textbf{P}^T\\textbf{P}-\\textbf{I}||^2_F+||\\textbf{Q}\\textbf{Q}^T-\\textbf{I}||^2_F$\n\nwhere $\\textbf{P},\\textbf{Q}$ denote the left and right singular matrices, respectively. \n\n(2) The De-cycling Process\n\nDue to space constraints, detailed content and analysis of the de-cycling process are provided in the Global Response.\n\n\n***W2：More analysis could be provided to interpret why the salience measurement works well. For example, are the average of influence domains consistent across models fine-tuned on different types of datasets?***\n\nDue to space constraints, detailed content and analysis are provided in the Global Response.\n\n***Q1: Taking the last row of Table 1 as an example, initially, the model uses a total rank that is 7.5 times the target rank, so the gpu memory usage is roughly equivalent to that of LoRA with r=8\\*7.5=60. Although the memory usage may decrease during the model optimization process, can you consider comparing with methods like LoRA and DoRA with r=60?***\n\nWe set the average target rank of each matrix in SalientLoRA to 8, and the initial rank to 60, to compare memory usage and fine-tuning performance with $\\text{LoRA}\\_{r=8}, \\text{DoRA}\\_{r=8}, \\text{LoRA}\\_{r=60}, \\text{DoRA}_{r=60}$. Specifically, we fine-tune the DeBERTaV3-base and BART-large models on the CoLA and XSum datasets using the aforementioned settings. \n\nThe results, as presented in the table below, demonstrate that small changes in rank settings do not significantly impact memory usage.  Memory usage of SalientLoRA peaks at only 5.41GB and 23.22GB, which are merely 0.33GB and 0.86GB higher than that of $\\text{LoRA}\\_{r=8}$.  Furthermore, the memory consumption of SalientLoRA rapidly decreases due to rank pruning, ultimately aligning closely with that of $\\text{LoRA}\\_{r=8}$. Notably, under this setting, our method still achieves the best results, surpassing those of $\\text{LoRA}\\_{r=60}$ and $ \\text{DoRA}\\_{r=60}$ by 0.9% and 0.57%, respectively. This also underscores the superior performance of our approach.\n\n| model       | GPU Mem (CoLA) | GPU Mem (XSum)  | Mcc (COLA) | Rough-L (XSum) |\n| ----------- | -------------- | --------------- | ---------- | -------------- |\n| LoRA r=8    | 5.08GB         | 22.26GB         | 69.73      | 34.84          |\n| DoRA r=8    | 5.37GB         | 22.87GB         | 71.46      | 35.47          |\n| LoRA r=60   | 5.28GB         | 22.98GB         | 69.78      | 34.92          |\n| DoRA r=60   | 5.58GB         | 23.61GB         | 71.78      | 35.65          |\n| SalientLoRA | 5.41GB -> 5.12GB    | 23.12GB -> 22.43GB  | 72.68      | 36.22          |\n\n***Q2: Based on my understanding, the constructed influence domains form an undirected simple graph. If this graph forms a single cycle, how do you perform de-cycling?***\n\nIn practice, the constructed dependency graph of singular values is a directed graph, where nodes represent singular values and edges represent their dependencies. This can be explained from two aspects.\n\n(1) The dependencies are quantified through slopes between two singular values within a time-series, where the choice of independent and dependent variables significantly influences the slope calculations. For instance, consider two singular values, $\\lambda_a$ and $\\lambda_b$. If $\\lambda_a$ is chosen as the independent variable and $\\lambda_b$ as the dependent variable, the resulting slope $k_{ab}$ quantifies the influence of $\\lambda_a$ on $\\lambda_b$. Conversely, selecting $\\lambda_b$ as the independent variable results in the slope $k_{ba}$. Since $k_{ab}$ and $k_{ba}$ are numerically distinct, the weights of the edges from $\\lambda_a$ to $\\lambda_b$ and $\\lambda_b$ to $\\lambda_a$ also differ, thus forming a directed graph.\n\n(2) Intuitively, the dependency relationship between two singular values is unidirectional: if a singular value is of significant importance, the variations of its magnitude can impact changes in the other one, reverse is not necessarily true.\n\n\n***Q3：Do you calculate influence domains starting from vertices with a degree of 0, similar to topological sorting, and then update the degrees of the vertices connected to it, then repeat the process***\n\nThe calculation of influence domains resembles a reverse topological sort. We calculate the influence domains of all vertices by proceeding backwards through the graph. Specifically, the process initiates from vertices with an out-degree of zero. These vertices do not influence any other nodes, so their influence domain is directly assigned a value of 1. Subsequently, the influence domain of a node that has subsequent nodes is calculated as the weighted sum of the influence domains of these downstream nodes. This iterative procedure continues until the influence domains for all nodes in the graph have been determined.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wDnA8DDVSQ", "reviewer": "Reviewer_i7vj", "summary": "The paper introduces SalientLoRA, an approach designed to optimize the intrinsic ranks of LoRA components in LLMs through salience measurement. The method first utilizes salience measurement to analyze the variations and inter-dependencies of singular value magnitudes over time, which helps assess matrix importance while mitigating instability and randomness. This analysis informs the adaptive adjustment of the time-series window used for significance measurement and rank reduction during training. This adaptive mechanism allows for rapid and stable rank allocation, permitting an initially higher rank setting to expand the allocation space for ranks.", "questions": "To fully evaluate the robustness of the proposed method, could you provide detailed ablation studies and analyses for the hyperparameters, including $\\beta$, $\\gamma$, $T_i$, and  $T_f$?", "limitations": "There are no potential negative societal impacts.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. SalientLoRA's use of salience measurement to analyze and utilize the variations of singular values effectively addresses the challenges of instability and randomness in rank optimization. The adaptive adjustment of the time-series window for significance measurement during training enhances the efficiency and stability of rank allocation.\n\n2.  Demonstrating substantial performance gains over state-of-the-art methods on diverse NLU and NLG tasks highlights the effectiveness of SalientLoRA in practical applications.", "weaknesses": "The proposed method incorporates a sophisticated multi-stage process that involves several critical hyperparameters, such as $\\beta$, $\\gamma$, $T_i$, and  $T_f$. However, the paper currently lacks a detailed analysis of these hyperparameters, which is crucial for understanding their roles and optimal settings within the methodology. Systematically exploring how each hyperparameter impacts the model's performance, including sensitivity analyses or hyperparameter tuning results, would greatly enhance the paper's scientific rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question: To fully evaluate the robustness of the proposed method, could you provide detailed ablation studies and analyses for the hyperparameters, including β, γ, Ti, and Tf?***\n\nWe appreciate the insightful suggestions provided by the reviewer. In response, we conduct additional experimental analyses on two sets of hyperparameters to explore their impact. These experiments are conducted on the CoLA and MRPC datasets to fine-tune the DeBERTaV3-base model. The total target rank is set to 144, and all the other parameters are consistent with the main experiments. \n\n(1) $\\beta$ and $\\gamma$ \n\n$\\beta$ and $\\gamma$ are thresholds that control the relevance and dependency relationships of singular values, respectively. To explore their effects, we keep other hyperparameters constant and change the values of $\\beta$ and $\\gamma$ separately to observe the timing and effects of SalientLoRA fine-tuning. The results are presented in the table below, where MCC represents Matthew’s Correlation Coefficient and Acc denotes accuracy.\n\n| $\\beta$ | $\\gamma$ | MCC (CoLA ) | Acc (MRPC) | Time (CoLA ) | Time (MRPC ) |\n| ------- | -------- | ----------- | ---------- | ------------ | ------------ |\n| 0.5     | 2        | 71.87       | 91.65      | 29min        | 42min        |\n| 0.7     | 2        | 71.82       | 91.57      | 24min        | 34min        |\n| 0.9     | 2        | 71.87       | 91.68      | 21min        | 33min        |\n| 0.9     | 1.5      | 71.78       | 91.63      | 27min        | 39min        |\n| 0.9     | 1        | 71.83       | 91.67      | 34min        | 47min        |\n\n\nThe performance exhibits minimal sensitivity to variations in the values of $\\beta$ and $\\gamma$. This insensitivity stems from the decycling operation for the dependency graph , which effectively eliminates the majority of redundant dependency relationships. However, setting excessively low values for $\\beta$ and $\\gamma$​ can lead to a large number of redundant dependencies in the graph, which increases the time cost of the decycling process and thus impacts the efficiency of fine-tuning. Therefore, we select hyperparameter values that ensure high efficiency of fine-tuning, with $\\beta$=0.9 and $\\gamma$=2.\n\n\n\n(2) $T_i$ and $T_f$\n\n$T_i$ and $T_f$ control the sizes of the initial and final time windows in the adaptive time-series window mechanism, respectively. The results indicate that the impact of $T_i$ on performance is minimal, with differences only ranging from 0.02% to 0.09%. However, as $T_i$ increases, the fine-tuning time significantly lengthens. This is due to a slower reduction in rank during the early stages of fine-tuning, which impacts the efficiency of rank allocation. \n\nMoreover, when $T_i$ remains constant and $T_f$ increases from 100 to 250, there is a slight improvement in performance, while the fine-tuning time remains relatively unchanged. This improvement can be attributed to the significance analysis in the later stages of fine-tuning, which incorporates singular values under more time steps, yielding more reliable allocation outcomes. Therefore, we set $T_i=10$ and $T_f=200$ to achieve a balance between performance and efficiency.\n\n| $T_i$ | $T_f$ | MCC (CoLA ) | Acc (MRPC) | Time (CoLA ) | Time (MRPC ) |\n| ----- | ----- | ----------- | ---------- | ------------ | ------------ |\n| 10    | 200   | 71.87       | 91.68      | 21min        | 33min        |\n| 30    | 200   | 71.81       | 91.59      | 23min        | 37min        |\n| 50    | 200   | 71.85       | 91.64      | 27min        | 41min        |\n| 10    | 100   | 71.52       | 91.32      | 22min        | 34min        |\n| 10    | 150   | 71.71       | 91.56      | 21min        | 34min        |\n| 10    | 250   | 71.84       | 91.64      | 23min        | 35min        |\n\n#", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijb6DR2EGY", "reviewer": "Reviewer_SXAJ", "summary": "The work presents an algorithm for adapting the rank of the LORA matrices according to a novel “saliency metric”  assigned to each singular value of the LORA matrices. \n\nThe saliency measure is computed taking into account a sequence of steps (time window) during training and computing two quantities at the end of each time window: the orthogonality-aware singular values and the domain influence of each singular value. The orthogonality-aware singular value is a weighted average of the singular value where the weight takes into account the orthogonality of the SVD decomposition at that step. The domain influence takes into account the correlation between singular values within each time window. \nAt the end of each step sequence, the ranks are adjusted based on this salience measurement.", "questions": "> 1. *line 143 to 146: The authors claim that the weight assigned to singular values of high loss should be small. However, from equation (2), it seems that the steps with higher loss receive the larger weight in the time window.* \n\n> 2.  *line 148. What do the authors mean by “we normalize the weights from 0 to 1”? An equation would be helpful to clarify the operation here.*\n\nA key performance measure that is not discussed is memory consumption. \n> 3. *Given a fixed parameter budget what is the amount of VRAM consumed by SalientLORA compared to for instance AdaLora?*\n\nOther minor remarks:\n\nWhen the authors state that AdaLORA has been adopted *in numerous research studies* (line 50) they should cite at least the most relevant ones to support the claim.\n\nline 56 The sentence is somewhat obscure. \n> 4. *What do the authors mean by “dominant role of singular values in the SVD matrix”? What is the precise meaning of “dominant role” in this context?*", "limitations": "No.\nThe authors do NOT seem to discuss the limitations of their method in the current version of the manuscript.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors propose a novel and interesting algorithm. The chosen setup speeds up the LoRA fine-tuning while maintaining accuracy or slightly outperforming other methods on the reported benchmarks. The experimental evaluation is convincing since the authors compare the proposed algorithm with other LoRA improvements on a reasonable number of tasks.", "weaknesses": "The main weakness of the work is the clarity of the exposition, which is obscure in some parts. \n\nFor example, one of the methods' core building blocks is the decycling operation of the dependency graph mentioned between lines 164-165: the authors must reference the algorithm they use for “de-cycling” the graph, describing its steps at least in the appendix.\\\nI leave other statements that require clarification in the question section below.\n\nIn addition, the paper does not discuss the limitations of the methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive suggestions provided by the reviewer. We provide a detailed explanation and experimental analysis as follows.\n\n***W1: the authors must reference the algorithm they use for “de-cycling” the graph, describing its steps at least in the appendix.***\n\nWe provide a detailed description of the de-cycling process in the Global Response and present the algorithm's pseudocode in the PDF. This detailed description will be included in the final version of the paper.\n\n***Q1: line 143 to 146: The authors claim that the weight assigned to singular values of high loss should be small. However, from equation (2), it seems that the steps with higher loss receive the larger weight in the time window.***\n\n We sincerely apologize for the error of weight calculation formula presented in our paper. The correct formula  should be $w^{(i)}=\\frac{\\sum^n_{j=0}R(P^{(j)}_a,Q^{(j)}_a)}{R(P^{(i)}_a,Q^{(i)}_a)}$.  By calculating in this way, singular values with higher losses are assigned lower weights, thereby enhancing the effectiveness of importance assessment with magnitude in a time-series. We thank the reviewer for identifying this correction, and we will amend it in the final camera-ready version of the manuscript.\n\n\n\n***Q2: Line 148. What do the authors mean by “we normalize the weights from 0 to 1”? An equation would be helpful to clarify the operation here.***\n\nWe appreciate the reviewer’s suggestion. Here, we normalize the weights of the singular values to prevent large discrepancies in weight values from affecting the evalution. We use Min-Max normalization, and the calculation formula is as follows,\n\n$w^{(i)}_{norm}=\\frac{w^{(i)}-w\\_{min}}{w\\_{max}-w\\_{min}}$\n\n\nwhere $ w_{min}$ and $w_{max}$ represent the minimum and maximum values among $w^{(0)}$ to $w^{(n)}$, respectively. This normalization method ensures a uniform scale for all weights, thereby enhancing the robustness of our results. A comprehensive description of this normalization technique will be included in the camera-ready version of the manuscript.\n\n\n\n***Q3: Given a fixed parameter budget what is the amount of VRAM consumed by SalientLORA compared to for instance AdaLora?***\n\nWe appreciate the constructive suggestions provided by the reviewer. To compare the memory usage of AdaLoRA and SalientLoRA, we fine-tune the DeBERTaV3-base and BART-large models on the CoLA and XSum tasks respectively. We set the target rank $r_t$ to 144 and the initial total rank $r_i$ to 7.5 times $r_t$. The results in the table below indicate that the training overheads for both fine-tuning methods are quite close, with SalientLoRA exhibiting a marginal increase in memory occupancy of only 0.11%-0.22%. This slight difference arises because SalientLoRA needs to store all the singular values within the time window for salience assessment. However, as each matrix retains only 2-15 singular value, the impact on memory is minimal.\n\n|                       | CoLA    | XSum     |\n| --------------------- | ------- | -------- |\n| AdaLoRA $r_t=144$     | 5.084GB | 22.411GB |\n| SalientLoRA $r_t=144$ | 5.095GB | 22.436GB |\n| AdaLoRA $r_t=276$     | 5.399GB | 23.362GB |\n| SalientLoRA $r_t=276$ | 5.410GB | 23.387GB |\n\n \n\n***Q4: When the authors state that AdaLORA has been adopted in numerous research studies (line 50) they should cite at least the most relevant ones to support the claim.***\n\nWe are grateful for the reviewer's reminder. Numerous studies have shown that applying AdaLoRA achieves excellent fine-tuning performance in various scenarios, including speech models [3] and several other languages models [1, 2]. We will include these references in the final version of the paper. \n\n\n[1] Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Xu L et al., 2023. In arXiv preprint.\n\n[2]  AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning. Zhou et al., 2024. In Transactions of the Association for Computational Linguistics.\n\n[3] Whisper-med_15k. 2024. https://huggingface.co/sin2piusc/whisper-med_15k.\n\n\n\n***Q5: What do the authors mean by “dominant role of singular values in the SVD matrix”? What is the precise meaning of “dominant role” in this context?***\n\nAfter performing singular value decomposition (SVD), the singular values represent the primary characteristics of a matrix. Larger singular values contain more information from the matrix and capture more of its essential features, thereby possessing greater importance. Consequently, singular values play a dominant role in the SVD matrix. In light of this, our method utilizes the magnitude of these singular values to assess their significance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
***1. A Detailed Explanation of the De-cycling Process.***

Since the dependency graph between singular values is a directed cyclic graph, we use a depth-first search (DFS) algorithm to detect and remove cycles. Specifically, we begin by performing a depth-first traversal of each node in the graph, recording the path in a stack. If a node is encountered that is already present in the path, a cycle is detected. At this stage, we remove the edge with the smallest weight from the cycle. This process is repeated until all nodes have been traversed.

 The pseudocode is provided in the following PDF.


***2. A Thorough Analysis of Salience Measurement***

The salience measurement consists of two components: the magnitude of singular values and the influence domain.

(1) Magnitude of Singular Values

After performing singular value decomposition (SVD), the singular values represent the primary characteristics of a matrix. Larger singular values contain more information from the matrix and capture more of its essential features, thereby possessing greater importance. Consequently, singular values play a dominant role in the SVD matrix. In light of this, our method utilizes the magnitude of these singular values to assess their significance, and combines this with the orthogonality loss at each timestep to produce a more robust evaluation.

(2) Influence Domain

Numerous studies [1, 2] demonstrate that there exists the interdependencies among the different structures of models. Constructing a dependency graph of these structures is instrumental in analyzing the importance of model structures. Inspired by this, we assume that the intrinsic ranks among different matrices also possess certain dependencies and influence relationships. The intrinsic rank of important  matrices can exert influence over other matrices. Therefore, beyond the magnitude of singular values, we analyze the intrinsic dependencies among singular values in different matrices to measure their importance. These dependencies reveal how variations in singular values influence each other across multiple timesteps. If variations in a singular value can induce changes in several others, it indicates that the singular value has a broader influence domain and is of higher importance. 

To intuitively reveal the influence domain of singular values, we visualize their distributions across various datasets, as shown in the following PDF. These distributions exhibit similar characteristics, with deeper layers and FFN (Feedforward Neural Network) layers having a larger influence domain. This aligns with findings from previous studies [3, 4], which indicate that deeper layers and FFNs are more significant in the model.

------
[1] LLM-Pruner: On the Structural Pruning of Large Language Models. Xinyin Ma and Gongfan Fang and Xinchao Wang, 2023. In NeurIPS. 

[2] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. Chen et al., 2023. In arXiv preprint. 

[3] LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. Zhang et al., 2024. In ACL.

[4] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. Zhang et al., 2023. In ICLR.

Meta-review: This paper proposes a new method, SalientLoRA, for adaptively optimizing the intrinsic rank of low-rank adaptation (LoRA) matrices. Experiments on NLU, NLG, and instruction tuning tasks show that SalientLoRA achieves state-of-the-art fine-tuning performance and balances fine-tuning efficiency and performance. The proposed SalientLoRA method is novel and the experimental results are encouraging.

This paper received borderline scores and the scores did not improve after discussion between the authors and reviewers. The reviewers' main concerns include: 1) the paper lacked a detailed analysis of hyperparameters and 2) the paper was not clearly written (e.g., lack of details, with clarity issues). The authors attempted to address these issues by providing some analysis and details in the rebuttal. The reviewers believe that by adding the responses to reviewer i7vj to Section 6.7, the analysis of the impact of various hyperparameters can be considered satisfactory. However, even with the inclusion of the details provided by the authors at the rebuttal stage (e.g., an explanation of the de-cycling method), the writing of this paper still does not reach the level that reviewers expect for a top-level paper. If this paper is accepted, the writing needs to be significantly revised and improved.
Decision: Accept (poster)