Unveiling LoRA Intrinsic Ranks via Salience Analysis

Authors: wenjun  ke
jiahao  wang
peng  wang
jiajun  liu
dong  nie
guozheng  li
yining  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 19:58:54

Abstract: The immense parameter scale of large language models underscores the necessity for parameter-efficient fine-tuning methods. Methods based on Low-Rank Adaptation (LoRA) assume the low-rank characteristics of the incremental matrix and optimize the matrix obtained from low-rank decomposition. Although effective, these methods are constrained by a fixed and unalterable intrinsic rank, neglecting the variable importance of matrices. Consequently, methods for adaptive rank allocation are proposed, among which AdaLoRA demonstrates excellent fine-tuning performance. AdaLoRA conducts adaptation based on singular value decomposition (SVD), dynamically allocating intrinsic ranks according to importance. However, it still struggles to achieve a balance between fine-tuning effectiveness and efficiency, leading to limited rank allocation space. Additionally, the importance measurement focuses only on parameters with minimal impact on the loss, neglecting the dominant role of singular values in SVD-based matrices and the fluctuations during training. To address these issues, we propose SalientLoRA, which adaptively optimizes intrinsic ranks of LoRA via salience measurement. Firstly, during rank allocation, the salience measurement analyses the variation of singular value magnitudes across multiple time steps and establishes their inter-dependency relationships to assess the matrix importance. This measurement mitigates instability and randomness that may arise during importance assessment. Secondly, to achieve a balance between fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series window, which adaptively controls the size of time-series for significance measurement and rank reduction during training, allowing for rapid rank allocation while maintaining training stability. This mechanism enables matrics to set a higher initial rank, thus expanding the allocation space for ranks. To evaluate the generality of our method across various tasks, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks. Experimental results demonstrate the superiority of SalientLoRA, which outperforms state-of-the-art methods by 0.96\%-3.56\% on multiple datasets. Furthermore, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5\% compared to AdaLoRA. The code is publicly available at https://github.com/Heyest/SalientLoRA.

Main paper: 
 introduction: 
 Large language models (LLMs) [23, 28, 2, 4] exhibit robust generative and inferential capabilities, excelling in various downstream tasks [26, 12, 14, 16] . However, the vast number of parameters in LLMs makes fine-tuning computationally demanding and time-consuming. Notably, LLaMA [23] encompasses parameters ranging from 7 billion to 65 billion, and fine-tuning LLaMA 65B necessitates a substantial 780GB of GPU memory [5] . To improve fine-tuning efficiency, numerous parameter-efficient fine-tuning (PEFT) methods [9, 13, 18, 10, 29, 17, 34] have been developed with only a minimal number of trainable parameters, significantly reducing computational resources. Incremental methods [9, 13, 18] introduce extra trainable parameters into the existing architecture of LLMs. However, the increased model depth introduces time delays during inference. Reparameterization methods [10, 29, 17, 34, 15] , based on low-rank adaptation (LoRA) [10] , assume the low-rank nature of the incremental matrix and perform low-rank decomposition on it. Although effective, they are still constrained by a fixed intrinsic rank, potentially limiting their adaptability to the dynamic significance of different matrix elements.To address the aforementioned limitations, dynamic rank allocation methods [25, 6, 29] have been proposed, which can be seamlessly incorporated into LoRA-based PEFT approaches to significantly enhance adaptability. Among these methods, AdaLoRA [29] stands out, having demonstrated excellent fine-tuning performance, as evidenced by its widespread adoption in numerous research studies [33] . AdaLoRA [29] introduces singular value decomposition (SVD) to the incremental matrices, employing sophisticated importance-aware methods [30, 31] to adaptively allocate intrinsic ranks. This method improves the alignment of ranks with the dynamic importance of matrix elements. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and finetuning efficiency in the following two aspects: (1) The importance assessment focuses only on parameters with minimal impact on loss, thereby neglecting the dominant role of singular values in the SVD matrix. This assessment also fails to consider the variability of parameters across multiple time steps, rendering it vulnerable to training fluctuations and probabilistic instability. Figure 1 (left) illustrates significant fluctuations in singular values and the regularization loss of orthogonality, with a sharp increase and decrease of regularization loss from steps 3 to 8. Such variability compromises the reliability of the importance measurement. (2) AdaLoRA starts with a specific initial rank and adjusts rank allocation by trimming unimportant parameters in groups during the fine-tuning process. Despite this, it struggles to balance fine-tuning efficiency and effectiveness, resulting in a limited rank allocation space. Figure 1 (right) illustrates that increasing the initial rank r i from 1.5 to 7.5 times the target rank r t improves fine-tuning performance by 0.78%. However, as the rank allocation space expands, the fine-tuning time nearly doubles.In this paper, we argue that in LoRA optimization, the inter-dependencies among multidimensional ranks are crucial for unveiling the intrinsic ranks, especially when a specific rank significantly influences others. As a consequence, we propose SalientLoRA, a method that conducts salience analysis within a time-series LoRA ranks, pruning ranks with low salience while retaining those with high significance. Following AdaLoRA, we decompose the incremental matrix using SVD through regularization constraints. The salience measurement calculates the correlation between singular values across modules in a time-series and constructs a dependency graph of correlation relationships. This graph reveals the inter-dependencies of ranks, assigning higher salience to rank with a broader influences domain. The salience measurement also examines the variation of the regularization loss of orthogonality and singular values in the time-series, mitigating instability and randomness that may arise during training. Moreover, to balance fine-tuning performance and efficiency, we propose an adaptive adjustment of time-series windows. This mechanism dynamically controls the size of time-series for salience measurement and rank reduction, facilitating rapid rank allocation while preserving training stability. This mechanism allows matrices to set a higher initial rank, thereby expanding the rank allocation space with greater efficiency. To validate the generality of our method, we conduct experiments on the GLUE dataset for NLU tasks, the XSum and CNN datasets for NLG tasks, and the MT dataset for instruction tuning, separately fine-tuning encoderonly (DeBERTaV3-base), encoder-decoder (BART-large and T5-base), and decoder-only (LLaMA) models. Experimental results demonstrate the superiority of our approach, which outperforms other existing fine-tuning methods by 0.96%-3.56% across multiple datasets, achieving state-of-the-art results. Moreover, as the rank allocation space expands, our method ensures fine-tuning efficiency, achieving a speed improvement of 94.5% compared to AdaLoRA.2 Related Work 
parameter-efficient fine-tuning: 
 Parameter-efficient fine-tuning methods (PEFT) can be divided into two categories: incremental [9, 13, 18] and reparameterization approaches [10, 29, 17] . Incremental methods add small neural modules into the existing architecture of LLMs, focusing solely on training these newly integrated modules. Adapter-Tuning [9] inserts simple Adapter modules after the feedforward layer of each Transformer architecture. Prefix-Tuning [13] and Prompt-Tuning [18] introduce additional trainable prefix vectors before the input layer or hidden layers. However, these approaches introduce time delays during inference due to either increasing model depth or reducing the available input sequence length. Additionally, they still exhibit performance discrepancies compared to full-parameter fine-tuning approaches. Reparameterization techniques update the incremental weight matrix in a parameterefficient manner, without altering the existing architecture of models. LoRA [10] assumes the low-rank nature of the incremental matrix and performs low-rank decomposition on it. DoRA [17] decomposes pre-trained weights into magnitude and direction for fine-tuning, using low-rank decomposition for directional updates. Although the LoRA-based reparameterization approaches are effective, they assign the same rank to all weight matrices, failing to consider the varying importance of weight matrices across different layers and modules.Intuitively, more important matrices are capable of learning more complex knowledge and thus require a higher rank. Therefore, the dynamic rank allocation methods [25, 6, 29] are necessary, which can be seamlessly applied to the LoRA-based approaches. To achieve the dynamic low-rank adaptation, DyLoRA [25] trains LoRA blocks for a range of ranks by sorting the representation learned by the adapter module at different ranks during training. SoRA [6] controls rank cardinality under gate sparsity by integrating a gate unit optimized through the proximal gradient. AdaLoRA [29] performs adaptation through singular value decomposition (SVD), dynamically adjusting intrinsic ranks based on significance. Among these methods, AdaLoRA has demonstrated its effectiveness, as proved by numerous studies and experiments. However, AdaLoRA still exhibits limitations in assessing the importance of LoRA ranks and fine-tuning efficiency.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 As illustrated in Figure 2 , the overall architecture of our method comprises three parts: incremental matrix SVD, salience measurement and adaptive rank allocation.Firstly, given the pretrained weight W 0 ∈ R d1×d2 , the update process can be formalized as W = W 0 + ∆, where ∆ represents the incremental updates. To reduce the parameter count and facilitate rank allocation, we decompose ∆ into PVQ, where V ∈ R r×r is a diagonal matrix, P ∈ R d1×r and Q ∈ R r×d2 , with rank r ≪ {d 1 , d 2 }. Following AdaLoRA, the orthogonality of P and Q is regularized through the auxiliary loss to simulate singular value decomposition (SVD). The regularization loss R(P, Q) is defined as follows:R(P, Q) = ||P T P -I|| 2 F + ||QQ T -I|| 2 F (1)Through this decomposition approach, the rank of the incremental matrix can be easily controlled by zeroing out the singular values in V. Secondly, we measure the salience of singular values within a time-series through orthogonality-aware singular value magnitudes and the influence domain, unveiling the inter-dependencies among ranks. Thirdly, singular values are sorted by their salience at the last step of each time-series, and those with low significance are zeroed out to achieve the The overall framework of SalientLoRA. First, the incremental matrix is decomposed using SVD to facilitate rank allocation. During fine-tuning, the salience of singular values within time-series is measured, which is composed of orthogonality-aware singular value magnitudes and the influence domain of the dependency graph. Finally, in the rank allocation process, singular values with lower significance are progressively trimmed. adaptive rank allocation. The rank allocation space is determined by the initial total rank r i and the target total rank r t , both of which are hyperparameters. Upon completion of the rank allocation, the most critical r t ranks are retained. In this process, an adaptive time-series window is introduced to accelerate the rank allocation process and maintain stability during training. Section 4 and 5 elaborate on the Salience Measurement and the Adaptive Time-series Window, respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
orthogonality-aware singular value magnitudes: 
 Following the objective of SVD, which utilizes singular values to represent the primary characteristics of a matrix, we assess their importance based on the magnitudes. However, the reliability of SVD heavily relies on the orthogonality of P and Q. Inadequate orthogonality undermines the efficacy of importance assessment with magnitudes. Therefore, we consider variations of regularization losses within a time-series, assigning weights to singular value magnitudes according to the orthogonality. Specifically, for the singular value λ a , we record its magnitudeEQUATIONa , . . . , λ (n) a } and regularization losses of the corresponding incremental matrix R(P(1) a , Q (1) a ), R(P (2) a , Q (2) a ), . . . , R(P (n) a , Q (n) a )within the n time steps. A high regularization loss indicates low reliability of the SVD at that step, and consequently, the weight assigned to it should be small. Therefore, the weight of step i is calculated as follows:EQUATIONSubsequently, we normalize the weights with Min-Max normalization, then calculate the orthogonality-aware singular values magnitudes:m a = n j=1 w (j) λ (j) a (3)During training, there exist dependency relationships among singular values, wherein the variation of one singular value can influence multiple others. In light of this, we construct the dependency graph of the correlations between singular values and determine their significance through the influence domain. Initially, the correlation is calculated between any pair of singular values within the n time windows λ a = {λ EQUATIONIf the p ab exceeds β, it indicates a correlation between λ a and λ b . For a correlated pair of singular values, the slope k ab between them is further computed through linear regression fitting to determine the extent of influence of λ a on λ b . A larger slope k ab indicates a greater influence of λ a on λ b , signifying a dependency relationship between them. Thus, we obtain the dependency matrix D ∈ R m×m and the element D ab of D is calculated as follows:EQUATION)where m denotes the total number of singular values and γ is the threshold for filtering dependency relationships. Note that the Pearson correlation coefficients and slopes are calculated using matrix parallel multiplication, incurring only minimal computational and time costs. Subsequently, we transform matrix D into a dependency graph, with singular values as nodes and dependency relationships as edges. The dependency graph among singular values forms a directed cyclic graph, containing numerous redundant dependencies within the cycles. Therefore, we employ a depth-first search (DFS) algorithm for de-cycling to eliminate these redundancies. The pseudocode of de-cycling algorithm is provided in Algorithm 1.Finally, based on the dependency graph, the influence domain of singular values is computed:EQUATIONAs illustrated in the dependency graph of Figure 2 , some singular value determines multiple others, with its variation can consequently lead to changes in several other singular values, thus signifying a higher level of importance.Ultimately, we combine the influence domain with orthogonality-aware magnitudes to obtain the salience for each singular value. Here, the λ is a hyperparameter that controls the contribution of two components to the salience measurement. The salience of the λ a is denoted as s a :EQUATIONAlgorithm 1 De-Cycling Algorithm for Dependency GraphsInput: Dependency graph G Output: Directed acyclic graph G ′ 1: P ath = stack() ▷ Initialize a stack to record the traversal path 2: V isited = ∅ ▷ Record the nodes that have been visited 3:for each node v in G do 4: if v not in V isited then 5: DFS_VISIT(v) 6:end if 7: end for 8: 9: def DFS_VISIT(v) 10:V isited.add(v) 11:P ath.push(v) 12:for each node u adjacent to v do 13:if u not in V isited ▷ If u is in the path, a cycle has been detected. 14:DFS_VISIT(u) 15:else if u in Path 16:Remove the edge with the smallest weight in the cycle 17:end if 18:end for 19:P ath.pop(v) 
adaptive time-series window: 
 During the rank allocation for n f steps, we implement an adaptive adjustment of time-series window.Within each time-series, the salience of singular values is assessed and the unimportant singular values are dropped at the last step of the time-series. This mechanism follows the principle that during the early stages of training, rank adjustment can be swiftly executed, while in the later stages, rank allocation should be approached more cautiously. Specifically, we initially maintain a small time window during the early stages of training, facilitating rapid rank reduction to enhance rank allocation efficiency. The pruned singular values and their corresponding singular vectors are excluded from gradient updates and salience calculations to expedite fine-tuning. As training progresses, the time window expands, and the allocation becomes more cautious. The broad time window reduces the frequency of importance calculations, thereby further accelerating the fine-tuning process. The adjustment of the time window is as follows:EQUATIONwhere T, T i , and T f represent the current time window size, the initial time window size, and the final time window size during the rank allocation phase, respectively, with T i < T f . n t denotes the current training step during the rank allocation phase.The model starts with a relatively high initial rank r i and zeros out a certain number of singular values within each time window to allocate rank. At the end of the rank allocation process, the rank is pruned to the target total rank r t . The total rank count maintained in each time window is:EQUATION6 Experiment 
experimental settings: 
 Datasets To evaluate the applicability of our fine-tuning approach across multiple tasks and various models, we conduct experiments on natural language understanding (NLU), natural language generation (NLG), and large-scale model instruction fine-tuning tasks, respectively. The specific datasets chosen for each task and the statistics will be detailed in Section 6.2, 6.3, 6.4 and Appendix A.Baselines We choose full fine-tuning (Full FT) and several existing incremental and reparameterization methods as baselines. Full FT denotes fine-tuning all parameters of the model, requiring significant computational resources. Adapter [9] integrates additional neural network layers into the model, only fine-tuning the newly added parameters. AdapterFusion [20] propose a more efficient design with adapters only applied after FFN and LayerNorm modules [3] . LoRA [10] performs low-rank decomposition on the incremental parameter matrices, substantially reducing the number of parameters for fine-tuning. SoRA [6] introduces a gating module with a proximal gradient decent update to control the sparsity of the updated matrices. AdaLoRA [29] dynamically distributes the parameter budget among weight matrices by evaluating the importance. DoRA [17] decomposes the pre-trained weights into magnitude and direction for fine-tuning, using LoRA for directional updates.Settings Our experiments are conducted on four NVIDIA RTX 3090Ti GPUs for NLU and NVIDIA Ampere A100 for NLG and instruction tuning tasks. During salience measurement, the slope threshold for dependency calculation γ = 2, the correlation threshold β = 0.9, and the λ is set to 0.7. For adaptive time-window adjustment, the initial time window size T i = 10, the final time window size T f = 200, and the initial total rank r i is set to 7.5 times the target total rank r t . The selected weight matrix of transformer layer for fine-tuning includes query/key/value projection (W q , W k , W v ), output projection (W o ) in the self-attention, and two weight matrices (W f1 , W f2 ) in two feedforward layers (FFNs). We select the learning rate from {8 × 10 -5 , 5 × 10 -5 , 3 × 10 -5 , 1 × 10 -4 , 3 × 10 -4 , 5 × 10 -4 , 8 × 10 -4 , 1 × 10 -3 }, and pick the best-performing learning rate for every method. Further details on other hyperparameters are shown in Appendix C.For NLU, we evaluate our method on the GLUE [27] benchmark utilizing the encoder-only model DeBERTaV3-base [7] . The GLUE benchmark consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We use Matthew's correlation coefficient, Spearman's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. To compare under the same parameter budget, we set the hidden dimensions d of adapters to 8 and 32, the rank r of LoRA and DoRA to 2 and 8, and the target total ranks r t for AdaLoRA and SalientLoRA to 144 and 276, respectively. The results are shown in Table 1 . Firstly, compared to incremental methods, reparameterization approaches generally yield better results. Under the parameter budget of 0.3M, LoRA and DoRA outperform adapters by average scores of 0.63% and 1.28% respectively. This demonstrates the superiority of reparameterization methods, achieving enhanced fine-tuning performance without adding extra inference latency. Secondly, both AdaLoRA and SalientLoRA consistently outperform LoRA across all eight datasets, showing an average improvement of 0.6% and 1.56% in 0.32 million parameters, and 0.5% and 1.67% in 1.27 million parameters, respectively. This suggests that adaptively adjusting the rank based on the importance of matrices indeed leads to better fine-tuning performance. Third, SalientLoRA surpasses all other fine-tuning methods with margins of 0.91%-1.79% and 0.77%-1.95% under two parameter budgets, achieving state-of-the-art results. Specifically, SalientLoRA outperforms AdaLoRA by 0.96% and 1.17% on average, which indicates the superiority of our method for salience measurement.For the NLG task, we finetune the encoder-decoder model BART-large [11] and T5-base [21] on the text summarization datasets XSum [19] and CNN/DailyMail [8] , with evaluation metrics of ROUGE 1/2/L scores. We set the rank r of LoRA and DoRA to 2, and the target total ranks r t for AdaLoRA and SalientLoRA to 144, respectively. Fine-tuning results on two datasets are shown in Table 2 . The experimental results indicate that under the same parameter budget, SalientLoRA achieves the best performance when fine-tuning T5-base and BART-large, surpassing other methods by margins of 0.36%-2.2%. Specifically, when fine-tuning the BART-large model on the XSum dataset, SalientLoRA exceeds AdaLoRA by 0.64%, 0.86%, and 1% in ROUGE 1/2/L scores, respectively, and outperforms DoRA by 0.54%, 0.36%, and 0.65%. Moreover, SalientLoRA even surpasses the performance of full fine-tuning on the CNN/DailyMail dataset, with improvements of 1.42%, 0.18%, and 1.51% by T5-base and 0.78%, 0.58%, and 0.88% by BART-large, respectively. To evaluate the effectiveness of our method in fine-tuning large language models of the decoder-only architecture, we fine-tune the LLaMA-7B [23] and LLaMA2-7B [24] model on the Alpaca [22] instruction dataset, which consists of 52k instances generated by GPT-4 [1] based on inputs from Alpaca. The fine-tuned LLaMA and LLaMA2 are evaluated on the MT-Bench [32] , generating model responses to a pre-defined set of 80 high-quality, multi-turn questions. These responses are then assessed using GPT-4, which assigns a quantitative score on a scale of 10 to each answer. We set the rank r of LoRA and DoRA to 64, and the target total ranks r t for AdaLoRA and SalientLoRA to 12280, respectively. We present the average scores alongside the number of trainable parameters in Table 3 . All compared fine-tuning methods utilized only 159.9M to 163.7M trainable parameters, which is merely 2.4% of Full FT. Experimental results indicate that LoRA still lags behind Full FT by 0.23 and 0.2 points. Notably, SalientLoRA surpasses all fine-tuning methods, even outperforming Full FT by 0.12 and 0.17 points, demonstrating the superior performance of SalientLoRA.To validate the effectiveness of the two components in our salience measurement, we conduct experiments on the full SalientLoRA and its variants. Here, OAM represents the orthogonality-aware magnitude and ID represents the influence domain. SalientLoRA w/o OAM indicates using the mean of magnitudes within a time-series without adjusting weights based on orthogonality. We follow the experimental setup for the NLU task, setting the target total rank r t of SalientLoRA to 144. As shown in Table 4 , we observe that both components of salience measurement (OAM and ID) significantly contribute to the final results. SalientLoRA w/o OAM experiences a performance decrease ranging from 0.18% to 0.73% across the eight datasets, with an average performance drop of 0.48%. This indicates that adaptively controlling the weights of magnitudes based on the matrix orthogonality effectively alleviates the problem of unreliable salience measurement caused by the orthogonality fluctuation during training. Additionally, SalientLoRA w/o ID shows a performance decline ranging from 0.32% to 1.13% across the eight datasets, with an average decrease of 0.8%. This highlights the crucial role of ID in performance enhancement. ID constructs dependency between singular values and further effectively measures importance by calculating the influence domain. We remark that as the rank allocation space expands, both SalientLoRA and AdaLoRA exhibit performance improvements, and SalientLoRA achieves a better balance between performance and fine-tuning time due to the adaptive time-series window and parallelized computation. Given the resource-intensive nature of the experiments, we choose the CoLA, RTE, STS-B, and MRPC datasets from NLU tasks for validation. We set the target total rank r t at 144 and gradually increase the average initial rank ri for each matrix from 3 to 18. Here, ri = ri n , where n represents the number of matrices fine-tuned, and r i denotes the initial total rank. The results in Figure 3 indicate that as the initial rank increases, AdaLoRA shows performance improvements by 0.39% -1.44%, while SalientLoRA improves by 0.55% -1.5%. Furthermore, the fine-tuning time of AdaLoRA significantly rises by 108% on average, whereas SalientLoRA experiences only marginal increases, ranging from 7% to 9%. This highlights the superiority of our method in balancing fine-tuning performance and efficiency by the adaptive time-series window and parallelized computation during salience measurement, resulting in reduced fine-tuning time. This section investigates the impact of the different contributions of two components in salience measurement on the final fine-tuning results, namely orthogonality-aware magnitude and influence domain. The degree of contribution is controlled by the hyperparameter λ. A higher λ indicates a greater contribution of orthogonality-aware magnitude to the salience measurement. Conversely, as λ decreases, the influence domain plays a greater role. The results in Figure 4 show an upward trend in fine-tuning performance across CoLA, RTE, STS-B, and MRPC datasets as λ increases, reaching a peak when λ reaches 0.7. This suggests that a greater contribution of orthogonality-aware magnitude leads to a more effective salience measurement. The primary reason lies in the SVD decomposition of the incremental matrix, where the magnitude of the singular values still represents the most fundamental factor of matrix characteristics. However, when λ further rises, fine-tuning performance declines. This suggests that when the contribution of orthogonality-aware magnitude slightly outweighs that of the influence domain (i.e., when λ approaches 0.7), salience measurement of singular values can achieve optimal performance, yielding the best fine-tuning results. This also demonstrates the effectiveness of both components in the salience measurement.In this section, we visualize the salience of singular values and analyze their distribution across different layers and modules in models. Here, we primarily illustrate the influence domain of singular values regarding inter-dependencies. Due to the presence of multiple singular values in each weight matrix, we average and normalize the influence domains for easier presentation. Figure 5 distribution of the influence domains after the completion of rank allocation for DeBERTaV3-base on different datasets. The results reveal that deeper layers and Feedforward Network (FFN) modules generally possess larger influence domains compared to shallower layers and other components. This indicates that the variations of singular values in these modules can significantly influence other parameters, therefore possessing a higher importance. This observation also aligns with the empirical conclusions in AdaLoRA that weight matrices of FFN modules and deep layers are more crucial for model performance. Therefore, this validates the effectiveness of the influence domain for salience measurement to identify important parameters. 
conclusion: 
 This paper proposes SalientLoRA, a novel adaptive rank allocation method for LoRA-based PEFT by salience analysis. This method measures the salience of rank within a time-series by constructing inter-dependencies among the correlations of singular values and prune ranks with low salience while retaining those with high significance, thereby unveiling the intrinsic ranks of the weight matrix. Additionally, an adaptive adjustment of the time-series window enhances the speed of rank allocation while ensuring training stability. Extensive experiments conducted in natural language understanding (NLU), natural language generation (NLG), and large model instruction tuning tasks demonstrate that our method achieves state-of-the-art fine-tuning performance, effectively balancing fine-tuning efficiency and performance. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We respect the license of LLaMA and so on. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The GLUE benchmark consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We use Matthew's correlation coefficient, Spearman's correlation coefficient, and overall accuracy to evaluate the CoLA, STS-B, and MNLI datasets. For the remaining datasets, we apply accuracy as the evaluation metric. Table 5 displays the statistics of the GLUE benchmark. We conduct additional experimental analyses on two sets of hyperparameters β, γ, T i and T f to explore their impact. These experiments are conducted on the CoLA and MRPC datasets to fine-tune the DeBERTaV3-base model. The total target rank is set to 144, and all the other parameters are consistent with the main experiments. β and γ are thresholds that control the relevance and dependency relationships of singular values, respectively. To explore their effects, we keep other hyperparameters constant and change the values of β and γ separately to observe the timing and effects of SalientLoRA fine-tuning.The results are presented in the Table 7 , where MCC represents Matthew's Correlation Coefficient and Acc denotes accuracy. The performance exhibits minimal sensitivity to variations in the values of β and γ. This insensitivity stems from the decycling operation for the dependency graph , which effectively eliminates the majority of redundant dependency relationships. However, setting excessively low values for β and γ can lead to a large number of redundant dependencies in the graph, which increases the time cost of the decycling process and thus impacts the efficiency of fine-tuning. Therefore, we select hyperparameter values that ensure high efficiency of fine-tuning, with β=0.9 and γ=2. T i and T f control the sizes of the initial and final time windows in the adaptive time-series window mechanism, respectively. The results in Table 8 indicate that the impact of T i on performance is minimal, with differences only ranging from 0.02% to 0.09%. However, as T i increases, the fine-tuning time significantly lengthens. This is due to a slower reduction in rank during the early stages of fine-tuning, which impacts the efficiency of rank allocation. Moreover, when T i remains constant and T f increases from 100 to 250, there is a slight improvement in performance, while the fine-tuning time remains relatively unchanged. This improvement can be attributed to the significance analysis in the later stages of fine-tuning, which incorporates singular values under more time steps, yielding more reliable allocation outcomes. Therefore, we set T i = 10 and T f = 200 to achieve a balance between performance and efficiency.The hyperparameter settings for NLU, NLG and instruction tuning tasks are shown in Table 9 . Here, r t denotes the target total rank, n i represents the number of steps to warm up the training before rank allocation, and n f indicates the number of steps during the rank allocation phase. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes a new method, SalientLoRA, for adaptively optimizing the intrinsic rank of low-rank adaptation (LoRA) matrices. Experiments on NLU, NLG, and instruction tuning tasks show that SalientLoRA achieves state-of-the-art fine-tuning performance and balances fine-tuning efficiency and performance. The proposed SalientLoRA method is novel and the experimental results are encouraging.

This paper received borderline scores and the scores did not improve after discussion between the authors and reviewers. The reviewers' main concerns include: 1) the paper lacked a detailed analysis of hyperparameters and 2) the paper was not clearly written (e.g., lack of details, with clarity issues). The authors attempted to address these issues by providing some analysis and details in the rebuttal. The reviewers believe that by adding the responses to reviewer i7vj to Section 6.7, the analysis of the impact of various hyperparameters can be considered satisfactory. However, even with the inclusion of the details provided by the authors at the rebuttal stage (e.g., an explanation of the de-cycling method), the writing of this paper still does not reach the level that reviewers expect for a top-level paper. If this paper is accepted, the writing needs to be significantly revised and improved.

Reviews: review: paper_summary: paper_summary: This paper proposes SalientLoRA, a new method for adaptively optimizing the intrinsic ranks of low-rank adaptation (LoRA) matrices. The key ideas are:

Using singular value decomposition (SVD) to decompose the LoRA matrices and measure the salience/importance of each singular value based on its magnitude, orthogonality constraints, and influence on other singular values within a time window during training.
strength_weakness: strength_weakness: - The article contains some details that are not clearly explained, such as how the R function on line 145 is calculated, and what specifically is done in the de-cycling process introduced on line 165.
- More analysis could be provided to interpret why the salience measurement works well. For example, are the average of influence domains consistent across models fine-tuned on different types of datasets?
questions: questions: 1. Taking the last row of Table 1 as an example, initially, the model uses a total rank that is 7.5 times the target rank, so the gpu memory usage is roughly equivalent to that of LoRA with r=8*7.5=60. Although the memory usage may decrease during the model optimization process, can you consider comparing with methods like LoRA and DoRA with r=60?
2. Based on my understanding, the constructed influence domains form an undirected simple graph. If this graph forms a single cycle, how do you perform de-cycling?
3. Do you calculate influence domains starting from vertices with a degree of 0, similar to topological sorting, and then update the degrees of the vertices connected to it, then repeat the process?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces SalientLoRA, an approach designed to optimize the intrinsic ranks of LoRA components in LLMs through salience measurement. The method first utilizes salience measurement to analyze the variations and inter-dependencies of singular value magnitudes over time, which helps assess matrix importance while mitigating instability and randomness. This analysis informs the adaptive adjustment of the time-series window used for significance measurement and rank reduction during training. This adaptive mechanism allows for rapid and stable rank allocation, permitting an initially higher rank setting to expand the allocation space for ranks.
strength_weakness: strength_weakness: The proposed method incorporates a sophisticated multi-stage process that involves several critical hyperparameters, such as $\beta$, $\gamma$, $T_i$, and  $T_f$. However, the paper currently lacks a detailed analysis of these hyperparameters, which is crucial for understanding their roles and optimal settings within the methodology. Systematically exploring how each hyperparameter impacts the model's performance, including sensitivity analyses or hyperparameter tuning results, would greatly enhance the paper's scientific rigor.
questions: questions: To fully evaluate the robustness of the proposed method, could you provide detailed ablation studies and analyses for the hyperparameters, including $\beta$, $\gamma$, $T_i$, and  $T_f$?
limitations: limitations: There are no potential negative societal impacts.
paper_summary: paper_summary: The work presents an algorithm for adapting the rank of the LORA matrices according to a novel “saliency metric”  assigned to each singular value of the LORA matrices. 

The saliency measure is computed taking into account a sequence of steps (time window) during training and computing two quantities at the end of each time window: the orthogonality-aware singular values and the domain influence of each singular value. The orthogonality-aware singular value is a weighted average of the singular value where the weight takes into account the orthogonality of the SVD decomposition at that step. The domain influence takes into account the correlation between singular values within each time window. 
At the end of each step sequence, the ranks are adjusted based on this salience measurement.
strength_weakness: strength_weakness: The main weakness of the work is the clarity of the exposition, which is obscure in some parts. 

For example, one of the methods' core building blocks is the decycling operation of the dependency graph mentioned between lines 164-165: the authors must reference the algorithm they use for “de-cycling” the graph, describing its steps at least in the appendix.\
I leave other statements that require clarification in the question section below.

In addition, the paper does not discuss the limitations of the methods.
questions: questions: > 1. *line 143 to 146: The authors claim that the weight assigned to singular values of high loss should be small. However, from equation (2), it seems that the steps with higher loss receive the larger weight in the time window.* 

> 2.  *line 148. What do the authors mean by “we normalize the weights from 0 to 1”? An equation would be helpful to clarify the operation here.*

A key performance measure that is not discussed is memory consumption. 
> 3. *Given a fixed parameter budget what is the amount of VRAM consumed by SalientLORA compared to for instance AdaLora?*

Other minor remarks:

When the authors state that AdaLORA has been adopted *in numerous research studies* (line 50) they should cite at least the most relevant ones to support the claim.

line 56 The sentence is somewhat obscure. 
> 4. *What do the authors mean by “dominant role of singular values in the SVD matrix”? What is the precise meaning of “dominant role” in this context?*
limitations: limitations: No.
The authors do NOT seem to discuss the limitations of their method in the current version of the manuscript.

Comments: title: Explaination of the Innovations and Contributions of Paper ID 7458: Unveiling LoRA Intrinsic Ranks via Salience Analysis
We will include the following limitations in the camera-ready version.
comment: Dear Chairs, SACs,  ACs, and Reviewers,

We appreciate the opportunity to discuss our NeurIPS submission titled "Unveiling LoRA Intrinsic Ranks via Salience Analysis". We are grateful for the evaluations provided by the review committee. As a foundational study, we would like to highlight the key strengths and innovations of our work, as we hope that our work will not only advance the field of parameter-efficient fine-tuning (PEFT), but also offer valuable insights for future research.

**1. Novelty and Contribution:** We propose a novel intrinsic rank allocation and fine-tuning method, SalientLoRA. This method adaptively allocates ranks based on matrix importance, revealing the intrinsic rank of LoRA via salience analysis, and achieves noteworthy fine-tuning performance with high efficiency. We validate the fine-tuning effects on models with different architectures (encoder-only, encoder-decoder, decoder-only) across multiple tasks, demonstrating performance improvements of 0.96%-3.56% over state-of-the-art (SOTA) baselines. Notably, compared to other rank allocation methods like AdaLoRA [1], our method reduces fine-tuning time by up to 94.5%. This high efficiency aligns with the primary goals of parameter-efficient fine-tuning methods.

**2. Generalizability of Insight:** Our work suggests promising potential for generalizability, presenting a foundational study that could enhance both the efficiency and memory usage of models during fine-tuning. Specifically, as an adaptive rank allocation approach, our method can be combined with other LoRA-based fine-tuning methods, such as DoRA [2], which applies LoRA to different architectures. By integrating our SalientLoRA with these approaches, our method is capable of uncovering the intrinsic rank of LoRA, thereby further enhancing fine-tuning performance. Given the broad applicability, we anticipate that our work could make a meaningful contribution to the research field.

We appreciate the efforts of Chairs, SACs, ACs and reviewers in guiding this process. We hope that the innovative aspects and the potential impact of our work can be fully recognized in the final deliberations.

Thank you once again for your consideration and the opportunity to contribute to NeurIPS.

Best regards,

The authors

---

**References**

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. Zhang et al., 2023. In ICLR.

[2] DoRA: Weight-Decomposed Low-Rank Adaptation. Liu et al., 2024. In ICML.
Thank you very much for your feedback. We appreciate your thorough and responsible review. 

Best regards.
I fully understand your situation and will consider your comment. However, addressing the reviewers' concerns ensures that the mark won't decrease, but it doesn't guarantee an increase. I still believe that this contribution is a borderline accept submission. I will discuss it further with the area chairs to help them make a decision.
Dear Reviewer SXAJ,

Thank you for your feedback.

In your review comments, you indicated that the main drawback of the paper is the clarity of the presentation, with some details not being sufficiently explained. We have provided a detailed clarification for each of these aspects in our rebuttal, including the decycling process, normalization, and so on. These revisions will be incorporated into the final version of the paper to enhance its clarity. With these improvements, we believe the paper demonstrates better soundness and we hope you will consider raising your score. Your evaluation is very important to us, and we sincerely appreciate your consideration.
I thank the authors for their response. I am inclined to keep my score.
Dear Reviewer xsy4,

Thank you for your thorough and valuable feedback. We have carefully addressed each of your concerns. We hope that our clarifications provide satisfactory answers to your questions. We are committed to improving our work based on your insights, and we look forward to your response.

Best regards, 

The Authors
We are glad that our responses have addressed most of your concerns. We will incorporate this section into the final version of the paper to further enhance its rigor. If you find that the detailed hyperparameter analysis has contributed to improving the paper's soundness, we would be deeply grateful if you could consider raising the score of Soundness or the Overall Rating. Your evaluation is very important to us, and we sincerely appreciate your consideration.
Dear Reviewer SXAJ,

Thank you for your thorough and valuable feedback. We have carefully addressed each of your concerns. We hope that our clarifications provide satisfactory answers to your questions. We are committed to improving our work based on your insights, and we look forward to your response.

Best regards, 
The Authors
Thank you for your responses! The responses address most of my concerns related to the hyperparameters. Hence, I will keep my original score.
Dear Reviewer i7vj,

Thank you for your thorough and valuable feedback. We have carefully addressed each of your concerns. We hope that our clarifications provide satisfactory answers to your questions. We are committed to improving our work based on your insights, and we look forward to your response.

Best regards,
The Authors
In the final version of the paper, we will include the following limitations.

The proposed mechanism of adaptive time-series window effectively accelerates and stabilizes the fine-tuning process. However, this mechanism employs a relatively rigid adaptive approach, considering only the time step information and focusing on progressively cautious pruning as the process advances. Future work will integrate the consideration of singular values' detailed dynamics, including momentum and interdependencies, to enable a more flexible and adaptive adjustment process.