Decoupling Angles and Strength in Low-rank Adaptation

Authors: Massimo Bini, Leander Girrbach, Zeynep Akata
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: parameter efficient finetuning, transfer learning, computer vision, NLP

Abstract:
Parameter-Efficient FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained models. These methods allow for quick adaptation to downstream tasks with minimal computational cost. However, popular finetuning methods such as LoRA exhibit limited robustness when it comes to hyperparameter choices or extended training regimes, preventing optimal out-of-the-box performance. In contrast, bounded approaches, such as ETHER, provide greater robustness but are limited to extremely low-rank adaptations and fixed-strength transformations, reducing their adaptation expressive power. In this work, we propose Decoupled Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and scales learnable low-rank matrices. By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance. Through evaluations on subject-driven image generation, natural language understanding, and instruction tuning, we show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness. Code is available at https://github.com/ExplainableML/DeLoRA.

Decision: accept (poster)
Reviewer scores: 6, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730044840441

thread_type:
official_review

<thread_object>
creator: reviewer_ekmu

<content>
summary:
Motivated by the lack of robustness of LoRA fine-tuning with respect to hyperparameter choice and catastrophic forgetting phenomena, the authors propose a novel PEFT method trying to tackle these problems. Their proposal merges the advantages of LoRA and ETHER, being able thus to control rank and maximal distance between fine-tuned and frozen weights.

strengths:
The paper is well presented and the proposed methods is both efficient and easy to implement.

weaknesses:
While I find no evident weakness in the manuscript and its content, I believe a broader experimental setting should be proposed to test the method (see questions for more details). Moreover, given the motivation of the work, I encourage the authors to include additional evidence concerning the robustness of their proposed method with respect to hyperparameters (and maybe something more about the avoidance of forgetting phenomena).

questions:
I leave here some questions/comments concerning the work:
1) Would it be possible to test the method on other common benchmarks (e.g. GLUE Deberta V3)?
2) The only concern I have about the new parametrization is that if a column of $A$ and $B$ is driven towards zero during training, normalization may cause instabilities. Did the authors ever observe this in practice? I would be worried of this phenomenon happening especially when choosing a high rank for the correction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730450985561

thread_type:
official_review

<thread_object>
creator: reviewer_vsop

<content>
summary:
This paper addresses some limitations in current parameter-efficient fine-tuning (PEFT) methods, particularly the robustness and flexibility challenges faced by additive methods like LoRA and multiplicative approaches like ETHER. The authors introduce DeLoRA, a fine-tuning method that normalizes and scales learnable low-rank matrices to improve hyperparameter robustness without sacrificing performance. Through evaluations on subject-driven image generation and large language model (LLM) instruction tuning tasks, DeLoRA demonstrates consistent improvements over existing PEFT methods, offering a balanced solution that enhances adaptability and performance in fine-tuning applications.

strengths:
1.	Overall, this paper is well-written and presents its ideas in a clear, accessible manner, making it easy to follow.
2.	The authors provide insightful reviews of two primary categories of fine-tuning methods: additive techniques such as LoRA and multiplicative approaches like ETHER, effectively highlighting the limitations inherent to each approach.
3.	The bounded approach constrains the updating scheme within a Frobenius norm ball, promoting a more robust selection of learning rates and potentially enhancing stability across different training settings.

weaknesses:
Overall, the contribution of this paper appears somewhat incremental, as it focuses on constraining weight updates within a restricted range—a concept that has also been examined in the DoRA paper. Additionally, the performance comparison with LoRA and DoRA is challenging to assess directly, as the experiments were conducted on different datasets.

1.	[Key Issue] The concept of decoupling the angles and scales in LoRA has been explored extensively in the DoRA method. Consequently, the contribution of this paper feels somewhat incremental in comparison to DoRA. Similar to DeLoRA, DoRA can limit its scale term within a fixed range, thereby preventing over-drifting in downstream adaptations. Would this lead to the same robustness effect?
2.	[key issue] LoRA and DoRA algorithms are typically evaluated on well-established NLP and NLU tasks, such as the GLUE benchmarks and Commonsense Reasoning, where their performance characteristics are widely understood. Thus, a fairer performance comparison would be achievable if DeLoRA were also tested on these datasets.
3.	In Eq (13), the updating is limited to |H-I| <= r, effectively limiting weight changes to within a Frobenius norm ball. Then it is not surprising that the algorithm is not sensitive to the change of learning rates. But a fundamental question is: is this restriction always necessary? For instance, in cases where the pretrained model diverges significantly from the target task, imposing a strict parameter update range might inadvertently hamper overall performance.
4.	In line 172, the authors state that 'multiplicative fine-tuning methods show stronger performance in generative models.' However, the results in Table 2 suggest that additive methods like LoRA and DoRA outperform OFT approaches, which appears contradictory and requires clarification.
5.	In Table 2, the performance improvement over DoRA appears marginal. Could the authors clarify if DoRA was evaluated with r=16? To ensure statistical reliability, it would be beneficial to repeat this experiment multiple times and report the standard deviation, as the observed improvement may fall within the range of experimental error.

questions:
See the above "weakness".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674722615

thread_type:
official_review

<thread_object>
creator: reviewer_pmk9

<content>
summary:
This paper proposes a new PEFT method based on LoRA called DeLoRA. 
In specific, the authors propose to decompose the learned low-rank update in LoRA into a learnable low-rank matrix with a unit norm, and a learnable magnitude.
The authors also draw a connection between the proposed DeLoRA and multiplicative ETHER which learns low-rank matrices to be *multiplied* instead of *added to* the original frozen parameters. 
The authors evaluated the proposed method on a variety of language and vision tasks and demonstrated fine-tuning performance improvement from DeLoRA.

strengths:
1. The writing of this paper is good and it is pleasant to read. 
 
2. The proposed method is well-presented. The connection to existing methods ETHER and DoRA is also amenable. 

3. The experiments look sufficient to me.

weaknesses:
After reading through the paper, my main concerns lie in the concept of DeLoRA. Namely, 

1. While an original main motivation of DeLoRA, as presented in Introduction, is to "introduce a boundary on the weight updates" to LoRA, the addition of *learnable* parameter $\lambda$ (without any regularization/constraint, if I read the paper correctly) removes this guarantee. As a result, learnable DeLoRA solution is just a strict subset of LoRA's --- the proposed parameterization introduces some inductive bias. Its underlying rationale, which cannot be fully explained by the presented motivation, is unclear. 

2. On the other hand, as the idea of decomposing the learning into magnitude and direction has already been proposed in DoRA, I feel that how different DeLoRA really is from DoRA, e.g., if they are in fact equivalent, needs more elaboration and careful discussion. Currently the authors only provide a equation-by-equation comparison, but no insights is provided in this form. 
In addition, I think this difference should be highlighted in Introduction and Method, instead of just in Related Work.

questions:
Besides above questions in Weakness, I also wonder:

1. In Table 1,2 how was controllable boundary combined with LoRA? 

2. In Fig 3, DeLoRA's distance to initialization was in fact decreased, how was this happened? 

3. Can you elaborate more on Fig 4? I didn't find any discussion in the main body. 

4. Can you test the proposed method on more language tasks, such as math reasoning?  

I am willing to adjust my score if these questions can be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676679257

thread_type:
official_review

<thread_object>
creator: reviewer_nywf

<content>
summary:
Two existing fine-tuning methods, LoRA and ETHER, have shown widespread success across the field, especially the former. However, some issues with LoRA from hyperparameter tuning and length of fine-tuning arise, leading to degradation in performance. The proposed method, DeLoRA, separates the magnitude and direction of the weight updates from LoRA. This is reminiscent of multiplicative weight updates such as ETHER and OFT with constrained magnitudes. DeLoRA is shown empirically to frequently achieve better accuracy than either LoRA or ETHER.

strengths:
- The ablation studies provide a convincing argument that somewhere in the continuum between LoRA and ETHER+ is a more powerful method. The experiments showing that different settings have different optimal locations in this continuum show that there are some directions forward with regard to which settings require different method choices, as well as defend that this continuum often contains a better method than the extremes.

- DeLoRA shows promise in having strong robustness similar to ETHER with explicit weight constraints, even when learning rates are high.

weaknesses:
- It took some time to figure out what exactly the DeLoRA method is. For clarity, what DeLoRA is could be written in its own dedicated section, not just embedded in Figure 1 without explaining the parameters or at the end of Section 2.2.1, since the two equations describing the method seem like all the others in the derivation at first glance. 

- It's mentioned in the introduction that one downside to LoRA is performance degradation during extended fine-tuning. This reads as though DeLoRA will overcome this issue, however, the top right of Figure 3 seems to show DeLoRA having the same issue. Are there other instances that can be added to show that DeLoRA does robustly train over many iterations?

- Figure 3 could use some repetitions. The single training run makes some of the training dynamics, especially DoRA's, hard to trust that they weren't due to randomness.

questions:
- In the derivations section, the LoRA and ETHER derivations seem to create different methods. This may come from the step of changing the ETHER derivation from a multiplicative adaptation to an additive one, but how this is achieved is not clear. Adding at least a few more steps to bridge between the two would be appreciated. 

Small things:
- "Figure 5: Average..." instead of "Figure 5: Avergae..."
</content>
</thread_object>

</thread>
<thread>
creation date:
1732560380070

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Authors Response [1/3]

comment:
We would like to thank all the reviewers for the detailed and useful feedback.

We appreciate the acknowledgement for our proposed robust parameter-efficient finetuning (PEFT) method (nYwf, pmk9, vsop, EKmU), alongside its derivation from LoRA and ETHER methods (pmk9, nYwf, EKmU) and our proposal for solving their limitations (nYwf, vsop), while highlighting that the paper is well presented and easy to read (pmk9, vsop, EKmU).

For common questions and concerns we reply below here, while for each individual discussion point we refer to their respective response below. We also modify the main manuscript, highlighting in blue the relevant changes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733097813229

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their detailed feedback and constructive inputs, which we believe led to extensive improvements, and substantially strengthened our final manuscript, as reflected by multiple increases in reviewer scores (pmk9,vsop,EKmU).

During the rebuttal phase we added to the main paper 
- a new evaluation of our proposed DeLoRA method on the GLUE natural language understanding benchmark
- an extensive discussion on the core differences with DoRA, supported by DeLoRA’s peculiar training dynamics and better decoupling 
- a new Euclidean distance plot to our robustness analysis, further showing DeLoRA’s non-divergent nature

In the Appendix, we further added 
- a new ablation where we tested keeping DoRA’s magnitude term fixed, which interestingly didn’t lead to the same properties of DeLoRA
- a new robustness ablation on DeLoRA by separately varying its angular and strength components, which showed no loss of robustness
- new qualitative examples showing DeLoRA’s smaller degradation during prolonged finetuning

In addition, following the reviewers' feedback and concerns, we enhanced the manuscript's overall clarity. Part of these revisions include 
- explicit mathematical formulations for our main ablation studies (in DeLoRA’s derivation from LoRA and ETHER)
- dedicated sections for DeLoRA's setup, the comparison with DoRA, and the robustness analysis
- additional remarks on DeLoRA’s initialization and stability

With the discussion period coming to an end, we believe that we have addressed all the comments from the reviewers which helped us improve our manuscript. We hope that includes the constructive and helpful comments of the reviewer nYwf whom we didn’t get to interact so far.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a variation on LoRA called DeLoRA which uses a learnable low-rank matrix with a unit norm, and a learnable magnitude. The authors also draw a connection between the proposed DeLoRA and multiplicative low-rank update schemes. On language and vision tasks, performance improvement from DeLoRA is reported.

Strengths: Clear, easy writing presenting a simple, practical approach to improving LoRA. Reviewer nTwf summarizes it well: "The ablation studies provide a convincing argument that somewhere in the continuum between LoRA and ETHER+ is a more powerful method."

Weaknesses: The idea of magnitude and direction decomposition for learning low-rank updates has already been proposed in DoRA - hence the novelty is not super clear. Well-established NLP and NLU tasks where LoRA has been frequently reported, such as the GLUE benchmarks and Commonsense Reasoning, is missing.