Title: Unveiling LoRA Intrinsic Ranks via Salience Analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Parameter-Efficient Fine-Tuning

Low-Rank Adaptation

LoRA
Abstract: The immense parameter scale of large language models underscores the necessity for parameter-efficient fine-tuning methods. Methods based on Low-Rank Adaptation (LoRA) assume the low-rank characteristics of the incremental matrix and optimize the matrix obtained from low-rank decomposition. Although effective, these methods are constrained by a fixed and unalterable intrinsic rank, neglecting the variable importance of matrices. Consequently, methods for adaptive rank allocation are proposed, among which AdaLoRA demonstrates excellent fine-tuning performance. AdaLoRA conducts adaptation based on singular value decomposition (SVD), dynamically allocating intrinsic ranks according to importance. However, it still struggles to achieve a balance between fine-tuning effectiveness and efficiency, leading to limited rank allocation space. Additionally, the importance measurement focuses only on parameters with minimal impact on the loss, neglecting the dominant role of singular values in SVD-based matrices and the fluctuations during training. To address these issues, we propose SalientLoRA, which adaptively optimizes intrinsic ranks of LoRA via salience measurement. Firstly, during rank allocation, the salience measurement analyses the variation of singular value magnitudes across multiple time steps and establishes their inter-dependency relationships to assess the matrix importance. This measurement mitigates instability and randomness that may arise during importance assessment. Secondly, to achieve a balance between fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series window, which adaptively controls the size of time-series for significance measurement and rank reduction during training, allowing for rapid rank allocation while maintaining training stability. This mechanism enables matrics to set a higher initial rank, thus expanding the allocation space for ranks. To evaluate the generality of our method across various tasks, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks. Experimental results demonstrate the superiority of SalientLoRA, which outperforms state-of-the-art methods by 0.96\%-3.56\% on multiple datasets. Furthermore, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5\% compared to AdaLoRA. The code is publicly available at https://github.com/Heyest/SalientLoRA.

Unveiling LoRA Intrinsic Ranks via Salience Analysis

Abstract

The immense parameter scale of large language models underscores the necessity for parameter-efficient fine-tuning methods. Methods based on Low-Rank Adaptation (LoRA) assume the low-rank characteristics of the incremental matrix and optimize the matrix obtained from low-rank decomposition. Although effective, these methods are constrained by a fixed and unalterable rank, neglecting the variable importance of matrices. Consequently, methods for adaptive rank allocation are proposed, among which AdaLoRA demonstrates excellent fine-tuning performance. AdaLoRA conducts adaptation based on singular value decomposition (SVD), dynamically allocating ranks according to importance. However, it still struggles to achieve a balance between fine-tuning effectiveness and efficiency, leading to limited rank allocation space. Additionally, the importance measurement focuses only on parameters with minimal impact on the loss, neglecting the dominant role of singular values in SVD-based matrices and the fluctuations during training. To address these issues, we propose SalientLoRA, which unveils the intrinsic ranks of the weight matrix via salience measurement and adaptively optimizes ranks of LoRA. This method measures the salience of rank within a time-series by constructing inter-dependencies among the correlations of singular values and prune ranks with low salience while retaining those with high significance. Additionally, an adaptive adjustment of the time-series window enhances the speed of rank allocation while ensuring training stability. This mechanism enables matrics to set a higher initial rank, thus expanding the allocation space for ranks. To evaluate the generality of our method across various tasks, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks. Experimental results demonstrate the superiority of SalientLoRA, which outperforms state-of-the-art methods by 0.96%-3.56% on multiple datasets. Furthermore, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5% compared to AdaLoRA. The code is publicly available at https://github.com/Heyest/SalientLoRA .

1 Introduction

Large language models (LLMs) [23, 28, 2, 4] exhibit robust generative and inferential capabilities, excelling in various downstream tasks [26, 12, 14, 16]. However, the vast number of parameters in LLMs makes fine-tuning computationally demanding and time-consuming. Notably, LLaMA [23] encompasses parameters ranging from 7 billion to 65 billion, and fine-tuning LLaMA 65B necessitates a substantial 780GB of GPU memory [5].

1.5

r

t

3

r

t

4.5

r

t

6

r

t

7.5

r

t

r

69

69.5

70

70.5

Mcc

CoLA

Mcc

1.3K

1.9K

2.2K

Runtime (s)

Runtime

1

2

3

4

5

6

7

8

9

10

0.02

0.03

0.04

0.05

0.06

Loss Fluctuations

0.04

0.045

0.05

0.055

Singular Value

Singular Value

To improve fine-tuning efficiency, numerous parameter-efficient fine-tuning (PEFT) methods [9, 13, 18, 10, 29, 17, 34] have been developed with only a minimal number of trainable parameters, significantly reducing computational resources. Incremental methods [9, 13, 18] introduce extra trainable parameters into the existing architecture of LLMs. However, the increased model depth introduces time delays during inference. Reparameterization methods [10, 29, 17, 34, 15], based on low-rank adaptation (LoRA) [10], assume the low-rank nature of the incremental matrix and perform low-rank decomposition on it. Although effective, they are still constrained by a fixed intrinsic rank, potentially limiting their adaptability to the dynamic significance of different matrix elements.

To address the aforementioned limitations, dynamic rank allocation methods [25, 6, 29] have been proposed, which can be seamlessly incorporated into LoRA-based PEFT approaches to significantly enhance adaptability. Among these methods, AdaLoRA [29] stands out, having demonstrated excellent fine-tuning performance, as evidenced by its widespread adoption in numerous research studies [33]. AdaLoRA [29] introduces singular value decomposition (SVD) to the incremental matrices, employing sophisticated importance-aware methods [30, 31] to adaptively allocate intrinsic ranks. This method improves the alignment of ranks with the dynamic importance of matrix elements. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and finetuning efficiency in the following two aspects: (1) The importance assessment focuses only on parameters with minimal impact on loss, thereby neglecting the dominant role of singular values in the SVD matrix. This assessment also fails to consider the variability of parameters across multiple time steps, rendering it vulnerable to training fluctuations and probabilistic instability. Figure 1 (left) illustrates significant fluctuations in singular values and the regularization loss of orthogonality, with a sharp increase and decrease of regularization loss from steps 3 to 8. Such variability compromises the reliability of the importance measurement. (2) AdaLoRA starts with a specific initial rank and adjusts rank allocation by trimming unimportant parameters in groups during the fine-tuning process. Despite this, it struggles to balance fine-tuning efficiency and effectiveness, resulting in a limited rank allocation space. Figure 1 (right) illustrates that increasing the initial rank r i from 1.5 to 7.5 times the target rank r t improves fine-tuning performance by 0.78%. However, as the rank allocation space expands, the fine-tuning time nearly doubles.

In this paper, we argue that in LoRA optimization, the inter-dependencies among multidimensional ranks are crucial for unveiling the intrinsic ranks, especially when a specific rank significantly influences others. As a consequence, we propose SalientLoRA, a method that conducts salience analysis within a time-series LoRA ranks, pruning ranks with low salience while retaining those with high significance. Following AdaLoRA, we decompose the incremental matrix using SVD through regularization constraints. The salience measurement calculates the correlation between singular values across modules in a time-series and constructs a dependency graph of correlation relationships. This graph reveals the inter-dependencies of ranks, assigning higher salience to rank with a broader influences domain. The salience measurement also examines the variation of the regularization loss of orthogonality and singular values in the time-series, mitigating instability and randomness that may arise during training. Moreover, to balance fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series windows. This mechanism dynamically controls the size of time-series for salience measurement and rank reduction, facilitating rapid rank allocation while preserving training stability. This mechanism allows matrices to set a higher initial rank, thereby expanding the rank allocation space with greater efficiency. To validate the generality of our method, we conduct experiments on the GLUE dataset for NLU tasks, the XSum and CNN

datasets for NLG tasks, and the MT dataset for instruction tuning, separately fine-tuning encoderonly (DeBERTaV3-base), encoder-decoder (BART-large and T5-base), and decoder-only (LLaMA) models. Experimental results demonstrate the superiority of our approach, which outperforms other existing fine-tuning methods by 0.96%-3.56% across multiple datasets, achieving state-of-the-art results. Moreover, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5% compared to AdaLoRA.

2 Related Work

2.1 Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning methods (PEFT) can be divided into two categories: incremental [9, 13, 18] and reparameterization approaches [10, 29, 17]. Incremental methods add small neural modules into the existing architecture of LLMs, focusing solely on training these newly integrated modules. Adapter-Tuning [9] inserts simple Adapter modules after the feedforward layer of each Transformer architecture. Prefix-Tuning [13] and Prompt-Tuning [18] introduce additional trainable prefix vectors before the input layer or hidden layers. However, these approaches introduce time delays during inference due to either increasing model depth or reducing the available input sequence length. Additionally, they still exhibit performance discrepancies compared to full-parameter fine-tuning approaches. Reparameterization techniques update the incremental weight matrix in a parameterefficient manner, without altering the existing architecture of models. LoRA [10] assumes the low-rank nature of the incremental matrix and performs low-rank decomposition on it. DoRA [17] decomposes pre-trained weights into magnitude and direction for fine-tuning, using low-rank decomposition for directional updates. Although the LoRA-based reparameterization approaches are effective, they assign the same rank to all weight matrices, failing to consider the varying importance of weight matrices across different layers and modules.

2.2 Dynamic Rank Allocation

Intuitively, more important matrices are capable of learning more complex knowledge and thus require a higher rank. Therefore, the dynamic rank allocation methods [25, 6, 29] are necessary, which can be seamlessly applied to the LoRA-based approaches. To achieve the dynamic low-rank adaptation, DyLoRA [25] trains LoRA blocks for a range of ranks by sorting the representation learned by the adapter module at different ranks during training. SoRA [6] controls rank cardinality under gate sparsity by integrating a gate unit optimized through the proximal gradient. AdaLoRA [29] performs adaptation through singular value decomposition (SVD), dynamically adjusting intrinsic ranks based on significance. Among these methods, AdaLoRA has demonstrated its effectiveness, as proved by numerous studies and experiments. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and fine-tuning efficiency.

3 Overview

As illustrated in Figure 2, the overall architecture of our method comprises three parts: incremental matrix SVD, salience measurement and adaptive rank allocation.

Firstly, given the pretrained weight W 0 ∈ R d 1 × d 2 , the update process can be formalized as W = W 0 + ∆ , where ∆ represents the incremental updates. To reduce the parameter count and facilitate rank allocation, we decompose ∆ into PVQ , where V ∈ R r × r is a diagonal matrix, P ∈ R d 1 × r and Q ∈ R r × d 2 , with rank r ≪ { d 1 , d 2 } . Following AdaLoRA, the orthogonality of P and Q is regularized through the auxiliary loss to simulate singular value decomposition (SVD). The regularization loss R ( P , Q ) is defined as follows:

Through this decomposition approach, the rank of the incremental matrix can be easily controlled by zeroing out the singular values in V . Secondly, we measure the salience of singular values within a time-series through orthogonality-aware singular value magnitudes and the influence domain, unveiling the inter-dependencies among ranks. Thirdly, singular values are sorted by their salience at the last step of each time-series, and those with low significance are zeroed out to achieve the

Salience Measurement

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Step 1

Query

...

...

Incremental Matrix SVD

Orthogonality of P and Q

Step 1

Step

...

=

？

1

...

=

？

90°

Adaptive Rank Allocation

Salience of Singular Value

...

=

？

1

=

？

1

=

？

1

90°

=

？

...

...

...

Dependency Graph

Correlation of Singular Value

Orthogonality of P and Q

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Step

Query

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Key

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Key

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Dense

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Dense

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Key

...

Query

.

...

adaptive rank allocation. The rank allocation space is determined by the initial total rank r i and the target total rank r t , both of which are hyperparameters. Upon completion of the rank allocation, the most critical r t ranks are retained. In this process, an adaptive time-series window is introduced to accelerate the rank allocation process and maintain stability during training. Section 4 and 5 elaborate on the Salience Measurement and the Adaptive Time-series Window, respectively.

4 Salience Measurement

4.1 Orthogonality-Aware Singular Value Magnitudes

Following the objective of SVD, which utilizes singular values to represent the primary characteristics of a matrix, we assess their importance based on the magnitudes. However, the reliability of SVD heavily relies on the orthogonality of P and Q . Inadequate orthogonality undermines the efficacy of importance assessment with magnitudes. Therefore, we consider variations of regularization losses within a time-series, assigning weights to singular value magnitudes according to the orthogonality. Specifically, for the singular value λ a , we record its magnitude λ a = { λ (1) a , λ (2) a , . . . , λ ( n ) a } and regularization losses of the corresponding incremental matrix R ( P (1) a , Q (1) a ) , R ( P (2) a , Q (2) a ) , . . . , R ( P ( n ) a , Q ( n ) a ) within the n time steps. A high regularization loss indicates low reliability of the SVD at that step, and consequently, the weight assigned to it should be small. Therefore, the weight of step i is calculated as follows:

Subsequently, we normalize the weights with Min-Max normalization, then calculate the orthogonality-aware singular values magnitudes:

4.2 Influence Domain of Singular Value

During training, there exist dependency relationships among singular values, wherein the variation of one singular value can influence multiple others. In light of this, we construct the dependency graph of the correlations between singular values and determine their significance through the influence domain. Initially, the correlation is calculated between any pair of singular values within the n time

windows λ a = { λ (1) a , λ (2) a , . . . , λ ( n ) a } and λ b = { λ (1) b , λ (2) b , . . . , λ ( n ) b } by the Pearson coefficient:

If the p ab exceeds β , it indicates a correlation between λ a and λ b . For a correlated pair of singular values, the slope k ab between them is further computed through linear regression fitting to determine the extent of influence of λ a on λ b . A larger slope k ab indicates a greater influence of λ a on λ b , signifying a dependency relationship between them. Thus, we obtain the dependency matrix D ∈ R m × m and the element D ab of D is calculated as follows:

where m denotes the total number of singular values and γ is the threshold for filtering dependency relationships. Note that the Pearson correlation coefficients and slopes are calculated using matrix parallel multiplication, incurring only minimal computational and time costs. Subsequently, we transform matrix D into a dependency graph, with singular values as nodes and dependency relationships as edges. The dependency graph among singular values forms a directed cyclic graph, containing numerous redundant dependencies within the cycles. Therefore, we employ a depth-first search (DFS) algorithm for de-cycling to eliminate these redundancies. The pseudocode of de-cycling algorithm is provided in Algorithm 1.

Finally, based on the dependency graph, the influence domain of singular values is computed:

As illustrated in the dependency graph of Figure 2, some singular value determines multiple others, with its variation can consequently lead to changes in several other singular values, thus signifying a higher level of importance.

Ultimately, we combine the influence domain with orthogonality-aware magnitudes to obtain the salience for each singular value. Here, the λ is a hyperparameter that controls the contribution of two components to the salience measurement. The salience of the λ a is denoted as s a :

Algorithm 1 De-Cycling Algorithm for Dependency Graphs

5 Adaptive Time-Series Window

During the rank allocation for n f steps, we implement an adaptive adjustment of time-series window. Within each time-series, the salience of singular values is assessed and the unimportant singular values are dropped at the last step of the time-series. This mechanism follows the principle that during the early stages of training, rank adjustment can be swiftly executed, while in the later stages, rank allocation should be approached more cautiously. Specifically, we initially maintain a small time window during the early stages of training, facilitating rapid rank reduction to enhance rank allocation efficiency. The pruned singular values and their corresponding singular vectors are excluded from gradient updates and salience calculations to expedite fine-tuning. As training progresses, the time window expands, and the allocation becomes more cautious. The broad time window reduces the frequency of importance calculations, thereby further accelerating the fine-tuning process. The adjustment of the time window is as follows:

where T, T i , and T f represent the current time window size, the initial time window size, and the final time window size during the rank allocation phase, respectively, with T i < T f . n t denotes the current training step during the rank allocation phase.

The model starts with a relatively high initial rank r i and zeros out a certain number of singular values within each time window to allocate rank. At the end of the rank allocation process, the rank is pruned to the target total rank r t . The total rank count maintained in each time window is:

6 Experiment

6.1 Experimental Settings

Datasets To evaluate the applicability of our fine-tuning approach across multiple tasks and various models, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large-scale model instruction fine-tuning tasks, respectively. The specific datasets chosen for each task and the statistics will be detailed in Section 6.2, 6.3, 6.4 and Appendix A.

Baselines We choose full fine-tuning (Full FT) and several existing incremental and reparameterization methods as baselines. Full FT denotes fine-tuning all parameters of the model, requiring significant computational resources. Adapter [9] integrates additional neural network layers into the model, only fine-tuning the newly added parameters. AdapterFusion [20] propose a more efficient design with adapters only applied after FFN and LayerNorm modules [3]. LoRA [10] performs low-rank decomposition on the incremental parameter matrices, substantially reducing the number of parameters for fine-tuning. SoRA [6] introduces a gating module with a proximal gradient decent update to control the sparsity of the updated matrices. AdaLoRA [29] dynamically distributes the parameter budget among weight matrices by evaluating the importance. DoRA [17] decomposes the pre-trained weights into magnitude and direction for fine-tuning, using LoRA for directional updates.

Settings Our experiments are conducted on four NVIDIA RTX 3090Ti GPUs for NLU and NVIDIA Ampere A100 for NLG and instruction tuning tasks. During salience measurement, the slope threshold for dependency calculation γ = 2 , the correlation threshold β = 0 . 9 , and the λ is set to 0.7. For adaptive time-window adjustment, the initial time window size T i = 10 , the final time window size T f = 200 , and the initial total rank r i is set to 7.5 times the target total rank r t . The selected weight matrix of transformer layer for fine-tuning includes query/key/value projection ( W q , W k , W v ), output projection ( W o ) in the self-attention, and two weight matrices ( W f 1 , W f 2 ) in two feedforward layers (FFNs). We select the learning rate from { 8 × 10 -5 , 5 × 10 -5 , 3 × 10 -5 , 1 × 10 -4 , 3 × 10 -4 , 5 × 10 -4 , 8 × 10 -4 , 1 × 10 -3 } , and pick the best-performing learning rate for every method. Further details on other hyperparameters are shown in Appendix C.

6.2 Natural Language Understanding

For NLU, we evaluate our method on the GLUE [27] benchmark utilizing the encoder-only model DeBERTaV3-base [7]. The GLUE benchmark consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We use Matthew's correlation coefficient, Spearman's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. To compare under the same parameter budget, we set the hidden dimensions d of adapters to 8 and 32, the rank r of LoRA and DoRA to 2 and 8, and the target total ranks r t for AdaLoRA and SalientLoRA to 144 and 276, respectively.

The results are shown in Table 1. Firstly, compared to incremental methods, reparameterization approaches generally yield better results. Under the parameter budget of 0.3M, LoRA and DoRA outperform adapters by average scores of 0.63% and 1.28% respectively. This demonstrates the superiority of reparameterization methods, achieving enhanced fine-tuning performance without adding extra inference latency. Secondly, both AdaLoRA and SalientLoRA consistently outperform LoRA across all eight datasets, showing an average improvement of 0.6% and 1.56% in 0.32 million parameters, and 0.5% and 1.67% in 1.27 million parameters, respectively. This suggests that adaptively adjusting the rank based on the importance of matrices indeed leads to better fine-tuning performance. Third, SalientLoRA surpasses all other fine-tuning methods with margins of 0.91%1.79% and 0.77%-1.95% under two parameter budgets, achieving state-of-the-art results. Specifically, SalientLoRA outperforms AdaLoRA by 0.96% and 1.17% on average, which indicates the superiority of our method for salience measurement.

6.3 Natural Language Generation

For the NLG task, we finetune the encoder-decoder model BART-large [11] and T5-base [21] on the text summarization datasets XSum [19] and CNN/DailyMail [8], with evaluation metrics of ROUGE 1/2/L scores. We set the rank r of LoRA and DoRA to 2, and the target total ranks r t for AdaLoRA and SalientLoRA to 144, respectively. Fine-tuning results on two datasets are shown in Table 2. The experimental results indicate that under the same parameter budget, SalientLoRA achieves the best performance when fine-tuning T5-base and BART-large, surpassing other methods by margins of 0.36%-2.2%. Specifically, when fine-tuning the BART-large model on the XSum dataset, SalientLoRA exceeds AdaLoRA by 0.64%, 0.86%, and 1% in ROUGE 1/2/L scores, respectively, and outperforms DoRA by 0.54%, 0.36%, and 0.65%. Moreover, SalientLoRA even surpasses the performance of full fine-tuning on the CNN/DailyMail dataset, with improvements of 1.42%, 0.18%, and 1.51% by T5-base and 0.78%, 0.58%, and 0.88% by BART-large, respectively.

6.4 Instruction Tuning

To evaluate the effectiveness of our method in fine-tuning large language models of the decoder-only architecture, we fine-tune the LLaMA-7B [23] and LLaMA2-7B [24] model on the Alpaca [22] instruction dataset, which consists of 52k instances generated by GPT-4 [1] based on inputs from Alpaca. The fine-tuned LLaMA and LLaMA2 are evaluated on the MTBench [32], generating model responses to a pre-defined set of 80 high-quality, multi-turn questions. These responses are then assessed using GPT-4, which assigns a quantitative score on a scale of 10 to each answer. We set the rank r of LoRA and DoRA to 64, and the target total ranks r t for AdaLoRA and SalientLoRA to 12280, respectively. We present the average scores alongside the number of trainable parameters in Table 3. All compared fine-tuning

methods utilized only 159.9M to 163.7M trainable parameters, which is merely 2.4% of Full FT. Experimental results indicate that LoRA still lags behind Full FT by 0.23 and 0.2 points. Notably, SalientLoRA surpasses all fine-tuning methods, even outperforming Full FT by 0.12 and 0.17 points, demonstrating the superior performance of SalientLoRA.

6.5 Ablation Results

To validate the effectiveness of the two components in our salience measurement, we conduct experiments on the full SalientLoRA and its variants. Here, OAM represents the orthogonality-aware magnitude and ID represents the influence domain. SalientLoRA w/o OAM indicates using the mean of magnitudes within a time-series without adjusting weights based on orthogonality. We follow the experimental setup for the NLU task, setting the target total rank r t of SalientLoRA to 144. As shown in Table 4, we observe that both components of salience measurement (OAM and ID) significantly contribute to the final results. SalientLoRA w/o OAM experiences a performance decrease ranging from 0.18% to 0.73% across the eight datasets, with an average performance drop of 0.48%. This indicates that adaptively controlling the weights of magnitudes based on the matrix orthogonality effectively alleviates the problem of unreliable salience measurement caused by the orthogonality fluctuation during training. Additionally, SalientLoRA w/o ID shows a performance decline ranging from 0.32% to 1.13% across the eight datasets, with an average decrease of 0.8%. This highlights the crucial role of ID in performance enhancement. ID constructs dependency between singular values and further effectively measures importance by calculating the influence domain.

6.6 Analysis of Space Allocation and Time Consumption

We remark that as the rank allocation space expands, both SalientLoRA and AdaLoRA exhibit performance improvements, and SalientLoRA achieves a better balance between performance and fine-tuning time due to the adaptive time-series window and parallelized computation. Given the resource-intensive nature of the experiments, we choose the CoLA, RTE, STS-B, and MRPC datasets from NLU tasks for validation. We set the target total rank r t at 144 and gradually increase the average initial rank ¯ r i for each matrix from 3 to 18. Here, ¯ r i = r i n , where n represents the number of matrices fine-tuned, and r i denotes the initial total rank. The results in Figure 3 indicate that as the initial rank increases, AdaLoRA shows performance improvements by 0.39% - 1.44%, while SalientLoRA improves by 0.55% - 1.5%. Furthermore, the fine-tuning time of AdaLoRA significantly rises by 108% on average, whereas SalientLoRA experiences only marginal increases, ranging from 7% to 9%. This highlights the superiority of our method in balancing fine-tuning performance and efficiency by the adaptive time-series window and parallelized computation during salience measurement, resulting in reduced fine-tuning time.

3

6

9

12 15 18

Average Initial Rank

70

70.5

71

71.5

Mcc

CoLA

AdaLoRA

SalientLoRA

1.2K

1.8K

2.4K

3.0K

AdaLoRA

SalientLoRA

3

6

9

12 15 18

Average Initial Rank

87

87.5

88

88.5

Acc

RTE

2.0K

3.0K

4.0K

5.0K

6.0K

3

6

9

12 15 18

87

87.5

88

88.5

Acc

RTE

3

6

9

12 15 18

91.5

91.8

92.1

92.4

Corr

STS-B

1.3K

1.9K

2.5K

3.1K

3.7K

3

6

9

12 15 18

Average Initial Rank

91.5

91.8

92.1

92.4

Corr

STS-B

3

6

9

12 15 18

90

90.5

91

91.5

Mcc

MRPC

1.8K

2.4K

3.0K

3.6K

4.2K

4.8K

Runtime (s)

3

6

9

12 15 18

Average Initial Rank

MRPC

6.7 Analysis of Hyperparameter λ

This section investigates the impact of the different contributions of two components in salience measurement on the final fine-tuning results, namely orthogonality-aware magnitude and influence domain. The degree of contribution is controlled by the hyperparameter λ . A higher λ indicates a greater contribution of orthogonality-aware magnitude to the salience measurement. Conversely, as λ decreases, the influence domain plays a greater role. The results in Figure 4 show an upward trend in fine-tuning performance across CoLA, RTE, STS-B, and MRPC datasets as λ increases, reaching a peak when λ reaches 0.7. This suggests that a greater contribution of orthogonality-aware magnitude leads to a more effective salience measurement. The primary reason lies in the SVD decomposition of the incremental matrix, where the magnitude of the singular values still represents the most fundamental factor of matrix characteristics. However, when λ further rises, fine-tuning performance declines. This suggests that when the contribution of orthogonality-aware magnitude slightly outweighs that of the influence domain (i.e., when λ approaches 0.7), salience measurement of singular values can achieve optimal performance, yielding the best fine-tuning results. This also demonstrates the effectiveness of both components in the salience measurement.

6.8 Salience Illustration across Different LoRA Ranks

In this section, we visualize the salience of singular values and analyze their distribution across different layers and modules in models. Here, we primarily illustrate the influence domain of singular values regarding inter-dependencies. Due to the presence of multiple singular values in each weight matrix, we average and normalize the influence domains for easier presentation. Figure 5 depicts the

0.3

0.4

0.5

0.6

0.7

0.8

Hyperparameter

70.9

71.2

71.5

71.8

72.1

72.4

72.7

73

Mcc (%)

71.29

71.51

71.67

71.87 71.78

70.94

71.89

72.23

72.53

72.6872.66

71.61

CoLA

r

t

=144

r

t

=276

0.3

0.4

0.5

0.6

0.7

0.8

Hyperparameter

87.8

88.1

88.4

88.7

89

Acc (%)

88.21

88.31 88.37 88.43

88.32

88.04

88.63

88.79 88.85

88.9388.92

88.43

RTE

r

t

=144

r

t

=276

0.3

0.4

0.5

0.6

0.7

0.8

Hyperparameter

91.8

92.1

92.4

92.7

93

Corr (%)

91.96

92.15

92.27

92.36 92.31

91.85

92.34

92.53

92.66

92.76

92.67

92.15

STS-B

r

t

=144

r

t

=276

0.3

0.4

0.5

0.6

0.7

0.8

Hyperparameter

91

91.2

91.4

91.6

91.8

92

92.2

92.4

MCC (%)

91.36

91.51

91.61 91.68 91.65

91.12

91.9

92.13

92.26

92.34

92.26

91.72

MRPC

r

t

=144

r

t

=276

1

2

3

4

5

6

7

8

9

10

11

12

Layer

Wq

Wk

Wv

Wo

Wf

1

Wf

2

0.18

0.24

0.44

0.57

0.69

0.86

0.63

0.79

0.83

0.74

0.55

0.36

0.26

0.26

0.34

0.33

0.55

0.63

0.74

0.82

0.81

0.81

0.78

0.67

0.7

0.38

0.47

0.5

0.66

0.77

0.84

0.81

0.8

0.8

0.76

0.77

0.58

0.47

0.66

0.79

0.78

0.82

0.82

0.82

0.81

0.83

0.85

0.79

0.52

0.66

0.69

0.77

0.8

0.84

0.85

0.82

0.84

0.83

0.89

0.87

0.19

0.13

0.34

0.46

0.57

0.75

0.84

0.78

0.83

0.87

0.86

0.77

CoLA

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1

2

3

4

5

6

7

8

9

10

11

12

Layer

Wq

Wk

Wv

Wo

Wf

1

Wf

2

0.14

0.18

0.24

0.33

0.48

0.51

0.59

0.64

0.68

0.67

0.65

0.42

0.23

0.34

0.19

0.38

0.42

0.72

0.65

0.73

0.74

0.72

0.68

0.6

0.37

0.43

0.29

0.53

0.53

0.63

0.76

0.71

0.8

0.74

0.73

0.71

0.52

0.47

0.53

0.57

0.7

0.75

0.72

0.73

0.76

0.76

0.8

0.75

0.58

0.73

0.71

0.82

0.74

0.83

0.82

0.85

0.86

0.85

0.89

0.88

0.28

0.27

0.36

0.58

0.64

0.72

0.8

0.83

0.84

0.86

0.88

0.85

RTE

1

2

3

4

5

6

7

8

9

10

11

12

Layer

Wq

Wk

Wv

Wo

Wf

1

Wf

2

0.15

0.34

0.17

0.25

0.46

0.69

0.51

0.53

0.61

0.49

0.46

0.27

0.23

0.31

0.2

0.39

0.32

0.57

0.62

0.66

0.68

0.61

0.67

0.64

0.27

0.66

0.23

0.43

0.48

0.6

0.66

0.61

0.7

0.63

0.65

0.59

0.66

0.38

0.47

0.52

0.65

0.69

0.79

0.71

0.69

0.68

0.86

0.89

0.58

0.65

0.7

0.79

0.71

0.82

0.72

0.8

0.83

0.81

0.91

0.85

0.33

0.29

0.38

0.51

0.6

0.7

0.79

0.83

0.84

0.87

0.89

0.81

STS-B

1

2

3

4

5

6

7

8

9

10

11

12

Layer

Wq

Wk

Wv

Wo

Wf

1

Wf

2

0.12

0.25

0.17

0.37

0.49

0.54

0.61

0.6

0.65

0.72

0.43

0.44

0.27

0.22

0.18

0.39

0.45

0.69

0.67

0.69

0.74

0.77

0.63

0.55

0.32

0.43

0.28

0.52

0.59

0.64

0.71

0.69

0.79

0.76

0.74

0.71

0.46

0.75

0.53

0.57

0.68

0.76

0.72

0.88

0.85

0.86

0.89

0.74

0.58

0.74

0.72

0.84

0.74

0.85

0.82

0.86

0.87

0.86

0.92

0.89

0.32

0.26

0.37

0.59

0.62

0.73

0.89

0.77

0.88

0.9

0.78

0.87

MRPC

distribution of the influence domains after the completion of rank allocation for DeBERTaV3-base on different datasets. The results reveal that deeper layers and Feedforward Network (FFN) modules generally possess larger influence domains compared to shallower layers and other components. This indicates that the variations of singular values in these modules can significantly influence other parameters, therefore possessing a higher importance. This observation also aligns with the empirical conclusions in AdaLoRA that weight matrices of FFN modules and deep layers are more crucial for model performance. Therefore, this validates the effectiveness of the influence domain for salience measurement to identify important parameters.

7 Conclusion

This paper proposes SalientLoRA, a novel adaptive rank allocation method for LoRA-based PEFT by salience analysis. This method measures the salience of rank within a time-series by constructing inter-dependencies among the correlations of singular values and prune ranks with low salience while retaining those with high significance, thereby unveiling the intrinsic ranks of the weight matrix. Additionally, an adaptive adjustment of the time-series window enhances the speed of rank allocation while ensuring training stability. Extensive experiments conducted in natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks demonstrate that our method achieves state-of-the-art fine-tuning performance, effectively balancing fine-tuning efficiency and performance.

Acknowledgments

We thank the anonymous reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057), the Start-up Research Fund of Southeast University (RF1028623234) and the Fundamental Research Funds for the Central Universities(2242024k30035). All opinions are of the authors and do not reflect the view of sponsors.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Mann Benjamin et al. Brown, Tom. Language models are few-shot learners. In NeurIPS , 2020.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 , 2023.

Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In EMNLP , 2023.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In ICLR , 2021.

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NeurIPS , 2015.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML , 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR , 2022.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL , 2020.

Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation extractors. In EMNLP , 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL , 2021.

Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, and Yanhe Liu. Towards continual knowledge graph embedding via incremental distillation. In AAAI , 2024.

Jiajun Liu, Wenjun Ke, Peng Wang, Jiahao Wang, Jinhua Gao, Ziyu Shang, Guozheng Li, Zijie Xu, Ke Ji, and Yining Li. Fast and continual knowledge graph embedding via incremental lora. In IJCAI , 2024.

Jiajun Liu, Peng Wang, Ziyu Shang, and Chenxiao Wu. Iterde: an iterative knowledge distillation framework for knowledge graph embeddings. In AAAI , 2023.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 , 2024.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL , 2022.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In EMNLP , 2018.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In EACL , 2021.

Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2019.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: an instruction-following llama model (2023). URL https://github. com/tatsu-lab/stanford_alpaca , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In EACL , 2023.

Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. GPT-RE: In-context learning for relation extraction using large language models. In EMNLP , 2023.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR , 2019.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In ICLR , 2023.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR , 2023.

Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Super tickets in pre-trained language models: From model compression to improving generalization. In ACL , 2021.

Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In ICML , 2022.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS , 2023.

Han Zhou, Xingchen Wan, Ivan Vuli´ c, and Anna Korhonen. AutoPEFT: Automatic configuration search for parameter-efficient fine-tuning. Transactions of the Association for Computational Linguistics , 12:525-542, 2024.

Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Deltalora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411 , 2023.

A Dataset Statistics

The GLUE benchmark consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We use Matthew's correlation coefficient, Spearman's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. Table 5 displays the statistics of the GLUE benchmark.

The CNN/Daily Mail dataset serves as a corpus for single-document generative summarization, comprising news articles collected from CNN and Daily Mail, with each summary containing multiple summary sentences. XSum dataset consists of BBC articles and accompanying single sentence summaries. The statistics for the CNN/DailyMail and XSum datasets are shown in Table 6.

B Experimental Analyses on Hyperparameter

We conduct additional experimental analyses on two sets of hyperparameters β , γ , T i and T f to explore their impact. These experiments are conducted on the CoLA and MRPC datasets to fine-tune the DeBERTaV3-base model. The total target rank is set to 144, and all the other parameters are consistent with the main experiments.

β and γ are thresholds that control the relevance and dependency relationships of singular values, respectively. To explore their effects, we keep other hyperparameters constant and change the values of β and γ separately to observe the timing and effects of SalientLoRA fine-tuning. The results are presented in the Table 7, where MCC represents Matthew's Correlation Coefficient and Acc denotes accuracy. The performance exhibits minimal sensitivity to variations in the values of β and γ . This insensitivity stems from the decycling operation for the dependency graph , which effectively eliminates the majority of redundant dependency relationships. However, setting exces-

sively low values for β and γ can lead to a large number of redundant dependencies in the graph, which increases the time cost of the decycling process and thus impacts the efficiency of fine-tuning. Therefore, we select hyperparameter values that ensure high efficiency of fine-tuning, with β =0.9 and γ =2.

T i and T f control the sizes of the initial and final time windows in the adaptive time-series window mechanism, respectively. The results in Table 8 indicate that the impact of T i on performance is minimal, with differences only ranging from 0.02% to 0.09%. However, as T i increases, the fine-tuning time significantly lengthens. This is due to a slower reduction in rank during the early stages of fine-tuning, which impacts the efficiency of rank allocation. Moreover, when T i remains constant and T f increases from 100 to 250, there is a slight improvement in performance, while the fine-tuning time remains relatively unchanged. This improvement can be attributed to the significance analysis in the later stages of fine-tuning, which incorporates singular values under more time steps,

Table 8: Experimental results with different values of hyperparameters T i and T f .

yielding more reliable allocation outcomes. Therefore, we set T i = 10 and T f = 200 to achieve a balance between performance and efficiency.

C Hyperparameter Statistics

The hyperparameter settings for NLU, NLG and instruction tuning tasks are shown in Table 9. Here, r t denotes the target total rank, n i represents the number of steps to warm up the training before rank allocation, and n f indicates the number of steps during the rank allocation phase.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We respect the license of LLaMA and so on.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents an algorithm for adapting the rank of the LORA matrices according to a novel “saliency metric”  assigned to each singular value of the LORA matrices. \n\nThe saliency measure is computed taking into account a sequence of steps (time window) during training and computing two quantities at the end of each time window: the orthogonality-aware singular values and the domain influence of each singular value. The orthogonality-aware singular value is a weighted average of the singular value where the weight takes into account the orthogonality of the SVD decomposition at that step. The domain influence takes into account the correlation between singular values within each time window. \nAt the end of each step sequence, the ranks are adjusted based on this salience measurement.\n\nSTRENGTHS:\nThe authors propose a novel and interesting algorithm. The chosen setup speeds up the LoRA fine-tuning while maintaining accuracy or slightly outperforming other methods on the reported benchmarks. The experimental evaluation is convincing since the authors compare the proposed algorithm with other LoRA improvements on a reasonable number of tasks.\n\nWEAKNESSES:\nThe main weakness of the work is the clarity of the exposition, which is obscure in some parts. \n\nFor example, one of the methods' core building blocks is the decycling operation of the dependency graph mentioned between lines 164-165: the authors must reference the algorithm they use for “de-cycling” the graph, describing its steps at least in the appendix.\\\nI leave other statements that require clarification in the question section below.\n\nIn addition, the paper does not discuss the limitations of the methods.\n\nQUESTIONS:\n> 1. *line 143 to 146: The authors claim that the weight assigned to singular values of high loss should be small. However, from equation (2), it seems that the steps with higher loss receive the larger weight in the time window.* \n\n> 2.  *line 148. What do the authors mean by “we normalize the weights from 0 to 1”? An equation would be helpful to clarify the operation here.*\n\nA key performance measure that is not discussed is memory consumption. \n> 3. *Given a fixed parameter budget what is the amount of VRAM consumed by SalientLORA compared to for instance AdaLora?*\n\nOther minor remarks:\n\nWhen the authors state that AdaLORA has been adopted *in numerous research studies* (line 50) they should cite at least the most relevant ones to support the claim.\n\nline 56 The sentence is somewhat obscure. \n> 4. *What do the authors mean by “dominant role of singular values in the SVD matrix”? What is the precise meaning of “dominant role” in this context?*\n\nLIMITATIONS:\nNo.\nThe authors do NOT seem to discuss the limitations of their method in the current version of the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SalientLoRA, an approach designed to optimize the intrinsic ranks of LoRA components in LLMs through salience measurement. The method first utilizes salience measurement to analyze the variations and inter-dependencies of singular value magnitudes over time, which helps assess matrix importance while mitigating instability and randomness. This analysis informs the adaptive adjustment of the time-series window used for significance measurement and rank reduction during training. This adaptive mechanism allows for rapid and stable rank allocation, permitting an initially higher rank setting to expand the allocation space for ranks.\n\nSTRENGTHS:\n1. SalientLoRA's use of salience measurement to analyze and utilize the variations of singular values effectively addresses the challenges of instability and randomness in rank optimization. The adaptive adjustment of the time-series window for significance measurement during training enhances the efficiency and stability of rank allocation.\n\n2.  Demonstrating substantial performance gains over state-of-the-art methods on diverse NLU and NLG tasks highlights the effectiveness of SalientLoRA in practical applications.\n\nWEAKNESSES:\nThe proposed method incorporates a sophisticated multi-stage process that involves several critical hyperparameters, such as $\\beta$, $\\gamma$, $T_i$, and  $T_f$. However, the paper currently lacks a detailed analysis of these hyperparameters, which is crucial for understanding their roles and optimal settings within the methodology. Systematically exploring how each hyperparameter impacts the model's performance, including sensitivity analyses or hyperparameter tuning results, would greatly enhance the paper's scientific rigor.\n\nQUESTIONS:\nTo fully evaluate the robustness of the proposed method, could you provide detailed ablation studies and analyses for the hyperparameters, including $\\beta$, $\\gamma$, $T_i$, and  $T_f$?\n\nLIMITATIONS:\nThere are no potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SalientLoRA, a new method for adaptively optimizing the intrinsic ranks of low-rank adaptation (LoRA) matrices. The key ideas are:\n\nUsing singular value decomposition (SVD) to decompose the LoRA matrices and measure the salience/importance of each singular value based on its magnitude, orthogonality constraints, and influence on other singular values within a time window during training.\n\nSTRENGTHS:\n- Novel salience measurement technique that considers singular inter-dependencies and temporal variations.\n- Comprehensive evaluation across many datasets and model types (encoder, decoder, encoder-decoder).\n- Achieves new state-of-the-art results on multiple benchmarks while being more efficient than prior LoRA methods.\n\nWEAKNESSES:\n- The article contains some details that are not clearly explained, such as how the R function on line 145 is calculated, and what specifically is done in the de-cycling process introduced on line 165.\n- More analysis could be provided to interpret why the salience measurement works well. For example, are the average of influence domains consistent across models fine-tuned on different types of datasets?\n\nQUESTIONS:\n1. Taking the last row of Table 1 as an example, initially, the model uses a total rank that is 7.5 times the target rank, so the gpu memory usage is roughly equivalent to that of LoRA with r=8*7.5=60. Although the memory usage may decrease during the model optimization process, can you consider comparing with methods like LoRA and DoRA with r=60?\n2. Based on my understanding, the constructed influence domains form an undirected simple graph. If this graph forms a single cycle, how do you perform de-cycling?\n3. Do you calculate influence domains starting from vertices with a degree of 0, similar to topological sorting, and then update the degrees of the vertices connected to it, then repeat the process?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}