Decoupling Angles and Strength in Low-rank Adaptation
Authors: Massimo Bini
Leander Girrbach
Zeynep Akata
Venue: ICLR 2025 Poster (2025)
Keywords: parameter efficient finetuning
transfer learning
computer vision
NLP
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a new PEFT method, which by normalizing and scaling learnable low-rank matrices, it enables effective decoupling of the angular learning from the adaptation strength, further leading to robustness properties.

Abstract: Parameter-Efficient FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained models. These methods allow for quick adaptation to downstream tasks with minimal computational cost. However, popular finetuning methods such as LoRA exhibit limited robustness when it comes to hyperparameter choices or extended training regimes, preventing optimal out-of-the-box performance. In contrast, bounded approaches, such as ETHER, provide greater robustness but are limited to extremely low-rank adaptations and fixed-strength transformations, reducing their adaptation expressive power. In this work, we propose Decoupled Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and scales learnable low-rank matrices. By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance. Through evaluations on subject-driven image generation, natural language understanding, and instruction tuning, we show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness. Code is available at https://github.com/ExplainableML/DeLoRA.

Published as a conference paper at ICLR 2025
DECOUPLING ANGLES AND STRENGTH IN NOW-RANK ADAPTATION
Massimo Bini 12,31. Leander Girrbach 2.3, Zeynep Akata
Uniiversity of Tübingen, Tübingen AI Center. "Helmholtz Munich, Technical University of Munich, Munich Center for Machine Learning, MDSI mmsssimo. wwwiiiniiiii -tuebingen.eee
ABSTRACT
Parameter-Efficien FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained mod- els. These methods allow for quick adaptation to downstream tasks with min- imal computational
However, popular finetuning methods such LoRA exhibit limited robustness when it comes to hyperparameter choices or extended training regimes, preventing optimal out-of-the-box performance. In contrast, bounded approaches, such as ETHER, provide greater robustness but are limited extremely low-rank adaptations and fixed- -trennhhength transformations, reducing their adaptation expressive power. In this work, we propose Decoupled Low- rank Adaptation (oeL,)RA), a novel finetuning method that normalizes and scales learnable low-rank matrices. By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance. Through evaluations on subject-driven image generation, natural language understanding, and instruc- tion tuning, show that DeLoRA matches or surpasses performance of com- peting PEFT methods, while exhibiting stronger robustness. Code is available at htttp:://github.co wwt.cc/mmmpeppeeecm nableM//Dei.RR.
INTRODUCTION
The rapid advancement of deep learning has led to the development of large-scale pretrained mod- els in various domains, especially in computer vision and natural language processing (Touvron let al. 2023a b; Radford et al., 2021; Rombach et al. 2022). However, the enormous size of these models, reaching billions of parameters, presents significant challenges when adapting them to spe- cific downstream tasks, particularly in terms of computational cost and efficiency. To address these challenges, Parameter Efficient FineTuning (PEFT) methods have emerged. PEFT methods are characterized by their introduction of small set of learnable parameters, in contrast to the extensive parameter updates required in full finetuning. Notable examples include adapters (Houlsby et al., 2019) and prompt tuning (Lester et al. (022)) In this work, we focus on enhancing LoRA (Hu al. 2022). a widely adopted finetuning method known for its simplicity and effectiveness. However, despite its success, LoRA is sensitive hyperparameter choices (Biderman et al. 2024) and of- ten exhibits performance degradation during extended finetuning (Qiu et al.) 2023). While robust finetuning approaches such as ETHER and +THER+ (Bini et al.) (024) address some of these limi- tations, they are constrained to extremely low-rank adaptations and fixed-strength transformations.
Therefore, we propose DeLoRA, an enhanced version of LoRA that introduces boundary on the weight updates through normalization, decoupling the angular learning from the adaptation strength. This enhances adaptability across diverse settings while preserving capabilities for personalization and merging at inference time. We motivate DeLoRA from two distinct perspectives: an extension of LoRA through the introduction additional normalization, and as an evolution of ETHER by enabling high-rank updates. We conduct ablation studies the design choices and demonstrate improvements over both LoRA and ETHER. Additionally, we validate the advantages of DeLoRA by evaluating across diverse tasks in image generation and LLM adaptation.

---

Published as a conference paper at ICLR 2025
LoRA
Output
DeLoRA
Output
+
W
W. Aplp EA,
ACal
A oF
1 Input
Input
loarnalie
Figure 1: Visualizations (Left) the original LoRA (Hu al. 2022) and (Right) of our proposed method DeLoRA. In addition to the low-rank matrices B, A, we introduce normalization = and scaling factor X, which effectively decouple the angular learning from the adaptation strength.
In summary, we make the following contributions in this work: (i) we thoroughly review the for- mulations of LoRA and ETHER and derive novel PEFT method, DeLoRA; (ii) demonstrate DeLoRA enhanced robustness and decoupling compared to alternatives; (iii) we extensively ablate the formulation of DeLoRA by deriving it from both LoRA and ETHER; (iv) evaluate DeLoRA on both vision and language benchmarks, matching or surpassing the performance of competing PEFT methods.
2 DECOUPLED LOW-RANK ADAPTATION (DELORA)
Our decoupled low-rank adaptation approach, by introducing learnable boundaries on the weight updates, effectively combines the strengths of LoRA and ETHER methods, allowing for high ex- pressivity and finetuning robustness. In the following sections, we will (i) present an overview of the PEFT methods LoRA and ETHER, focusing their respective limitations (Section (.1 and (ii) describe how we derive our proposed DeLoRA method from both perspectives (Section (.2 along with comparison with DoRA (Liu et al. 2024a), method that also targets decoupling angular and magnitude components.
2.1 PRELIMINARIES: LORA& & ETHER, AND THEIR LIMITATIONS
Here, provide detailed review of LoRA (Hu et al. 2022) and ETHER (Bini al. 2024), with particular focus on their limitations.
Low-rank Adaptation (LoRA). Hu et al. (002 proposed Low-rank Adaptation (LoRA), which parametrizes the update of pretrained weights W E Rdxʃ during finetuning as
(w + a BA b
where Brxd and B E xr are the learnable matrices, is a scaling factor, and r is the rank of the final BA matrix. When min(d, f). LoRA substantially reduces the number of parameters required for finetuning compared to full finetuning. Furthermore, BA matrices can be integrated into W at inference time, eliminating additional latency.
However, LoRA known to be highly sensitive to hyperparameter choices (Biderman et al. 2024), and it is prone to deterioration with over-training (Qiu et al. 2023). thus requiring careful tuning and experimentation to achieve an optimal balance between a sufficienlly high learning rate and avoiding catastrophic overwriting of the pretrained weights. In our proposed DeLoRA. mitigate this behavior by introducing boundary to the weight updates, which results in robust performance across a broad range of learning rates.
Finetuning with Hyperplane Reflections (ETHER). Following efficiency and robustness argu- ments, Bini et al. (024) propose to employ bounded transformations for finetuning, namely ETHER
2

---

Published as a conference paper at ICLR 2025
and ETHER+. ETHER (left side in Eq. (2) and ETHER+ (right side) introduce multiplicative trans- formations H or H respectively, which act on the pretrained weights as follows:
(HW)
(
x 1 b.
(2)
Here, H = -- 2un H = uut vut, H+ = ūū + T (where u, U, ū, ú are unit vectors) are bounded terms of their distance to the identity transformation, as per
F = | 1. 2.
(3)
where the subscript F denotes the Frobenius norm. This upper bound on the transformation distance prevents weight changes that cause catastrophic overwriting, as shown by Bini et al. (2024).
However, enforcing constant boundary on the transformation distance can limit the finetuning performance, as the boundary may be too strict to adapt the layer or pretrained model at hand to the respective task. Furthermore, by rewriting the formulations in Eq. (2) in residual form, we can show that the weight updates are intrinsically limited to be low-rank (sce Appendix(A) which limits the finetuning capacity of such methods. In DeLoRA, by introducing normalization and scaling factor to LoRA matrices, we show how to achieve robustness comparable to ETHER while enabling control over both boundary and rank, ultimately enhancing model expressivity and performance.
2.2 DELORA
While both LoRA and ETHER demonstrate valuable properties, namely parameter efficiency and robustness, they also exhibit notable limitations. Our proposed PEFT method, DeLoRA, addresses these shortcomings by synthesizing the strengths of both approaches. In this regard, DeLoRA can be thought of as an extension of LoRA that incorporates ETHER's robustness properties or, ternatively, an enhancement of ETHER that adopts LoRA's more expressive paradigm. In the following, we will present both derivations and finally summarize in a concise way our proposed DeLoRA formulation.
Deriving DeLoRA from LoRA. In order to achieve robustness to learning rates, we first observe that in LoRA's Eq. (T) the norm of the weight updates AW proportional ABA, which in turn is proportional to the learning rate. This means that the update strength at each training step is directly driven by the learning rate, which can lead to catastrophic overwriting in high learning rate regimes. In order to mitigate this behavior, we want to introduce normalization term. To do this, we start by decomposing the BAA matrix into the sum of its rank-1 components, i.e.
BA Σ baa
Controllable Boundary. Similarly to ETHER, we normalize each rank-1 entry, making the Frobe- nius norm of each single rank-1 component equal 1. This normalization can be introduced as in
baag
|Blll|a BEA
(5)
aill for i 1, Ex, = 0 for i,j = 1,... r, o The final update distance with respect the pretrained weights thus is bounded as
where E is diagonal matrix with entries Bi
Tox
|BEA| = Σ bra
|bba
(6)
Most importantly, the boundary is independent of the used learning rate. Next, control the bound- ary and remove its rank dependency, we scale BEA by factor A as in
BEA A.
Now, the boundary equal to À and can be chosen arbitrarily to better the pretrained network task at hand. To enable greater flexibility and layer-specific boundaries, we make each distinct À
3

---

Published as a conference paper at ICLR 2025
learable,, allowing finetuning to adapt their values accordingly. Hence, we effectively decouple the angular learning (the normalized BEA matrices) from the adaptation strength, as measured by the boundary A. Furthermore, introducing single additional learnable parameter À to each finetuned matrix creates only negligible overhead in terms of overall trainable parameters and training speed.
Weights-norm Scaling. Previous works suggest that when finetuning image generative models such as Stable Diffusion, multiplicative finetuning methods exhibit stronger performance (Qiu et al.) 2023 Liu et al., 2024b) than additive counterparts. We argue this may arise because multiplicative methods induce weight updates relative to the pretrained weights W, meaning updates are inherently layer-specifie. This might be especially relevant when adapting a diverse set of layers, which the case for our Stable Diffusion adaptations (see Fig. 3) To mimic this approach, in our additive proposed method DeL, we introduce a scaling factor equal to the pretrained weights norm. This can be formally stated as
I|[WI| BEA.
AW
(8)
Our ablation studies Stable Diffusion finetuning tasks demonstrate such performance improve- ments empirically (see Section 3.2).
Initialization. To initialize the finetuning process from the pretrained model, DeLoRA's normal- ization operation does not allow to simply follow LoRA's zero initialization of the B matrix. From preliminary experiments, we find that introducing small epsilon to avoid division by 0, would sometimes lead to unstable results, Therefore, we instead follow (Meng et al.] 2024: Bini et al., 2024) and subtract a copy of the hhiming-randonee initialized matrices to the frozen pretrained weights, as in
X/IWII
W W
BEA
(9)
where W is the original pretrained matrix, and (IlIW BEA), is the update matrix time 0.
Deriving DeLoRA from ETHER So far, we showed how to derive DeLORA from LoRA. Al-
ternatively, it is possible derive DeLoRA by introducing properties of LoRA to ETHER. We find this to be insightful to understand the impact of each individual component from a theoretical perspective. In addition, we quantitatively ablate all innovations of DeLoRA in Section 3.2
Controllable Boundary. One of the primary limitations of ETHER and ETHER+ is their fixed boundary (see Section 2. which fixed and thus cannot be adapted the pretrained model in use. We address this limitation by introducing scaling parameter X as in
X
A
H - Auu
H = I
2 uu +
2 UUT.
(10)
Consequently, the boundaries on the distances of and H from the identity matrix become || - F X, and ||/ - -|I X. In Section 3.2 we show that this modification, i.c. in- troducing a controllable bound, leads to the largest increase in performance.
Increasing the rank. In Appendix A demonstrate that ETHER and ETHER+ are restricted to rank- 1 and rank-4 weight updates respectively. In order to arbitrarily control the rank, we extend the H parameter of ETHER+ to H, which allows for an arbitrary number of weight reflection operations:
r/2 H-
r/2 Σ
Σ
+
T.o.
(11)
1=1
We can rewrite A by gathering the u and v unit vectors into two rank- matrices, as in
T=I-UEU + VOVT,
(12)
where Σ and θ are diagonal normalization matrices with entries Σ₁,- T.. O entries the diagonals of Σ and θ constructed normalize and to unit vectors. Thus, the distance from the identity matrix becomes
Tog 7. The
r/2
r/2
r/2
r/2
- 1|
Σ
u4u
Σ
<
I ||
+
Σ
Now
= r.
(13)
i=1
4

---

Published as a conference paper at ICLR 2025
As above, we can control the boundary the distance, and remove the rank dependency, by intro- ducing a scaling factor as in
--I-AUSUT =
(14)
U.V Relaxation. Finally, we relax UEUT, VeVT and replace them with distinct trainable ma- trices BEA and D&C respectively, which leads to H = A (BEA D&C)W. We emphasize how this formulation resembles multiplicative analog of our proposed DeLoRA method, and we include this variant in our ablation study. We ablate all alternatives in Section 3.2 There, we find that DeLoRA, combined with weights-norm scaled updates, as in multiplicative finetuning, achieves overall stronger performance.
DeLoRA formulation.
Summarizing, our proposed DeLoRA finetuning method consists in learn- ing a normalized low-rank matrix BEA and scale A, updating the pretrained weights as in
(w
W| BEA)
x +
(15)
T
This formulation inherently constrains the learnable finetuning updates in [lizzed ball, where W is the norm of the pretrained weights, achieving a decoupling of the transformation strength from the angular learning.
In more detail, the key components are:
Normalization: E is r-dimensional diagonal matrix that normalizes LoRA's inner low- dimensional bottleneck (Eq. (5), bounding the Frobenius norm of BEA to r (Eq. (6).
Scaling Factors: (i) 1/r is used to remove the rank dependency the boundary dimensionality, (ii) ||W]| to make the weight updates proportional to the pretrained weights, and (iii) A to control the adaptation strength and allow for layer-specific boundary adaptation (Eq. 7
Initialization: Pretrained initialization follows by merging to the pretrained weights frozen copy of the initialized finetuning adaptation matrices (Eq.
DoRA vs DeLoRA discussion. DoRA (Liu et al. 2024a), similarly to our work, addresses fine- tuning targeting the decoupling of angular and magnitude components, by using formulation that W+ WWAW We can summarize the key differences between DoRA leads to weight updates W" = m JW +AWT- and our proposed method in two main aspects: (i) DoRA applies normalization and scaling oper- ations on the fully finetuned weights, and (ii) these operations are performed on the column space of the weight matrices, which significantly differs from our approach. In contrast,
argue that DeLoRA finetuning has two key advantages: (i) by introducing the normalization and scaling opera- tions directly the weight updates AW, it more effectively prevents divergence from the pretrained model, and (ii) by normalizing the inner low-dimensional space (as opposed to the column space), it implicitly enforces Frobenius norm boundary, providing a mathematical guarantee against di- vergence. These ultimately result in (i) peculiar training dynamics (as depticted in Fig. 3 whereas DoRA and LoRA exhibit similar behavior), and (ii) enhanced decoupling, supported by the robust- ness performance in Fig. 2 and in Appendix C In this regard, we notice that although DeLoRA's learnable boundary theoretically allows an unbounded Frobenius norm, divergence from the pre- trained weights does not happen in practice, as also shown in Appendix [D] This demonstrates that during finetuning, DeLoRA's learnable boundary is able to effectively adjust and avoid divergence from the pretrained weights- behaavir that is not observed with DoRA.
3 EXPERIMENTS
In this section, we evaluate our proposed DeLoRA method for image generation, natural language understanding, and instruction tuning tasks. We begin by providing a detailed description of these tasks and their relevance. To justify our design choices, we present comprehensive ablation study that highlights the key innovations of DeLoRA. Finally, we demonstrate that DeLoRA not only matches or exceeds the performance of LoRA and other -ote----trtt-tttt- art methods but also exhibits superior robustness. This enhanced stability is particularly evident in two aspects: reduced sen- sitivity to learning rate selection and improved performance retention during extended finetuning periods.
5

---

Published as a conference paper at ICLR 2025
3.1 TASKS
Subject-driven Image Generation. Following (Qiu et al. 2023: Bini et al. 2024), we assess the
effectiveness of our proposed methods in the DrotmBooth setting (Ruiz et al. 2023), specifically by adapting Stable Diffusion (Rombach et al. (022) to recontextualize a subject shown in a set of images according to given prompt. The dataset, sourced from (Ruiz et al. 2023), comprises 30 subjects, each paired with 25 prompts. The task is to finetune Stable Diflusion to generate images portraying the given subject in the context defined by the prompts. We report an example in in Appendix E (Fig. 7 left side). For each combination of image and prompt, after finetuning, we generate four images and measure the subject-fideling by DINO (Caron et al., 202I) and CLIP (Radford et al. 2021), as proposed by (Ruiz et al. 2023). Here, the score represents the similarity of generated and given images, measuring the faithfulness of generating images of the given subject to the provided real images. Among the two metrics, the DINO score is more significant since it is more sensitive to subject-unique features (Ruiz et al. 2023).
Semantic Map to Image Following QQu et al. 2023 Bini al. 2024), we evaluate the ability of our proposed methods in finetuning Stable Diffusion to generate realistic images based on given segmentation maps. The image should follow the spatial structure laid out in the segmentation map as closely as possible. Examples of segmentation maps and their corresponding generated images presented in Appendix (Fig. 7 right side). For the control signal, use the pretrained encoder from ControlNet (Zhang et al. 2023a). For training and evaluation, we utilize semantic maps and images from the ADE20K dataset (Zhou et al., 2019). After training, we generate images for 2000 segmentation masks from the ADE20K validation set and report the mean Intersection-over-Ure
(mloU) and accuracy of semantic maps as predicted by UperNet-101 101 (Xiao et al. 2018). Note that we only use the Semantic Map to Image task to ablate our method design decisions.
Natural Language Understanding We evaluate DeLoRA's performance in adapting small-scall language models by finetuning and evaluating pretrained RoBERTa- base model (Liu et al. 2020 on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018). GLUE tasks have been extensively used to measure natural language understanding performance, compris- ing inference tasks (MNLI, QNLI, RTE), sentiment classification (SST-2), and correct identification of English grammatical structures (CoLA). CoLA results refer to Matthews correlation coefficient, MNLI to matched accuracy, and STS-B to average correlation, while all other tasks are evaluated accuracy. For proper evaluation on the validation set, adopt the setup proposed by Wu et al. (2024b), and split the validation set into two subsets, guarded by a pre-defined seed, that will be used for model selection and evaluation. We provide more details in Section 3.3
Instruction Tuning. We evaluate how effectively D can adapt LLMs to follow user-given instructions, finetuning LLaMA-2-7B (Touvron et al., 2023b) on the Alpaca dataset (Taori et al., 2023). Following Bini et al. 2024, we evaluate the zero-shot performance of instruction-tumm models four different tasks, namely (1) Massive Multitask Language Understanding (MMLU) (Henrrycks et al. 2021). which features 57 tasks in different categories such as STEM, Humani- ties, and Social Sciences; (2) A12 Reasoning Challenge (ARC) (Clark et al. 2018), which contains over 7000 grade-school science questions; (3) TruthfulQA (Lin et al. 2022). which contains 817 questions representing common misconceptions in 38 categories like health, law, finance and pol- itics. TruthfulQA additionally features two separate sub-tasks, namely single-true and multi-true. In single-true, only one of the provided answers is correct, and the model has to select the unique correct answer. In multi-true, several of the provided answers may be correct, and the model has to assign a high probability to correct answers and low probability to incorrect answers.
3.2 ABLATION OF DELORA DESIGN CHOICES
In this section, we ablate the incremental design choices that transform LoRA and ETHER+ into DeLoRA, evaluating these on the subject-driven generation and semantic map-to-image tasks. From the LoRA derivation (lop-down in Tables 22) we show how incorporating normalization with controllable boundary and weight scaling into pretrained matrices enhances performance. From the
ETHER+ derivation (bottom-up in Tables 12 . show how introducing controllable scale,
higher-rank formulation, relaxed learnable matrices, and an additive finetuning tranformation, in- crementally improves performance.

---

Published as a conference paper at ICLR 2025
Method
AW formulation
DINO
CLIP-I
LoRA [rank-r]
BA
0.674
0.785
4 + normalize w/ controllable boundary
À HEA
0.682
0.809
controllable boundary + high rank relaxed additive (DeLoRA)
normalize w/ controllable boundary weights-sealing
IWI BEA
0.701
0.825
t + controllable scale high rank relaxed
(BEA D&CJW
0.696
controllable boundary high rank
(UEUT VOLTJW
0.685
0.840
controllable boundary
Muut oyt
0.678
0.810
ETHER+ (one-sided) [rank-2, boundary equal to 21
(uwt
0.624
0.746
Table I: Ablation of A innovations the Subject-driven Image Generation task. We show how different components affect performance from both LoRA and ETHER derivation.
Method
A Formulation
mloU
Acc. 1
FID J
LoRA [rank-r]
BA BEA
25.13
64.95
31.35
+ normalize w/ controllable boundary
25.66
65.82
31.01
controllable houndary high rank relaxed additive FT (DeLoRA)
normalize w/ controllable boundary weights-
1WT BEA
26.10
65.08
30.71
controlluble boundary high rank relaxed
A ABEA DEC
25.55
65.16
29.89
controllable boundary
X(ua - MnTW
24.56
62,70
31,28
ETHER+ (one-sided) [rank-2, boundary equal 2]
(wat vrTiW
23.46
62.26
31.18
Table 2: Ablation of DeLoRA innovations on the Semantic Map to Image task. We show how different components from both LoRA and ETHER derivations incrementally improve performance.
Results for subject-driven image generation are in Table For this ablation we use a small-scale version of the setting proposed by (Ruiz al., 2023). finetuning 3 subjects over 25 prompts each (10% of the data). Among all modifications, we notice how the introduction of a controllable bound- ary +THER+ (one-sided) has the highest impact, raising the DINO score from 0.624 to 0.678 and the CLIP score from 0.746 to 0.810. This shows how the lack of strength is the hindering factor for (onesddonn-)ddee)) as already noted by (Bini et al., 2024). Starting from LoRA, we notice how the weights- scaling has the largest impact on performance, raising the DINO score from 0.682 0.701 and the CLIP score from 0.809 to 0.825. Additionally, we note that DeLoRA's performance without the weights-norm scaling falls short compared to its multiplicative counterpart.
For the Semantic Map to Image ablation study, we run a small-scale grid search by finetuning Stable Diffusion for 10 epochs on ADE20K in bfloat16 precision. Results are reported in Table Table 2 We note how DeLoRA achieves best controllability among different variations. In addition, we also note the increase in Accuracy when increasing the rank of ETHER+, hinting that it could have been a limiting factor.
3.3 BENCHMARK RESULTS
Subject- -Drven Image Generation Results are in Table 3 For comprehensive benchmark per- formance comparison, we report low-rank results from Bini et al. (2024). while rumning and eval- uating LoRA, DoRA, and DeLoRA methods at consistent rank. For each method, we conduct grid search to identify optimal hyperparameters using the same 3 subjects as in the ablation stud- ies, then evaluate the top-performing configurations the full 30-subject benchmark, testing each across three distinct seeds. 'The best and average results are reported in Table 3 We notice that LoRA, DoRA, and DeLoRA, all achieve comparable average performance terms of DINO and CLiP-Image, all outperforming lower-rank baselines. This shows that DeLoRA is able to effectively combine ETHER+ robustness properties with superior performance.
Natural Language Understanding Results are in Table 4 For proper evaluation on the GLUE validation set, we follow Wu et al. (2024a c) and split the validation set into two subsets (determined by pre-defined seeds), and use the first subset to tune hyperparameters, and the second subset to evaluate method performance. For fair comparisons In addition, in order to compare with LoRA's implementation, we simply apply DeLoRA to Q. attention layers with rank 8, which is likely sub-optimal with respect to applying lower-rank modules
same seeds as Wu al. 2024a

---

Published as a conference paper at ICLR 2025
Method
#param
DINO
CLIP-I
Real Images
0.703
0.864
DreamBooth
ct
2023
859.5M
0.644
0.793
OFT,-4
Qni et al.
2025
11.6M
0.652
0.794
ETHER+
Bini ct a
2024
0.4M
0.666
0,800
LoRA,
Hu el al.
2022
0.8M
0.660
0.796
LoRA,
Hu ct al.
2022
3.2M
0,686
0.818
10RA,-16
Liu al. 2024a
3.2M
0.687
0.819
DeLoRA,=16
(ours)
3.2M
0.686
0.820
LoRA
et al.
2022
3.2M
0.688
0.818
DoRA
11
Liu al.
2024a
3.2M
0.689
0.819
DeLoRA
(ours)
3.2M
0.693
0.820
Table 3: Results for evaluating DeLoRA subject-driven image generation. indicates experi- ments with tuned hyperparameters.
Method
#param
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg
Full Finet.
125M
87.3
94.4
87.9
62.4
92.5
91.7
78.3
90.6
85.6
BitFit
Zaken al.
2022
0.1M
84.7
94.0
54.0
91.0
87.3
69.8
89.5
82.3
Lau ct al.
2022
0.06M
85.4
93.4
86.4
57.8
91.1
88.5
73.5
83.1
LoReFT
Wu al.
2024K
0.02M
93.4
60.4
91.2
87.4
79.0
84.2
RED
Wu cl al.
20244
0.02M
93.9
61.0
90.7
87.2
78.0
90.4
84.3
LoRA
Hu ét al.
2022
0.3M
93.9
59.7
92.6
90.4
75.3
84.7
Adapter
Preilfer et al.
202 2021
0.3M
87.1
93.0
88.8
58.5
92.0
90.2
77.7
90.4
84.7
Adapter
Houlshy ct al. 2019
0.4M
87.0
93.3
88.4
60.9
92.5
90.5
76.5
90.5
85.0
DeLoRA
(ours)
0.3M
86.9
93.7
88.6
64.7
92.6
90.2
77.3
90.6
85.6
Table 4: Comparisons of different methods finetuning RoBERTa-base on GLUE benchmark. Re-
sults of all baselines are taken from Wu al. (2024a) and Wu et al. (2024c).
larger set of layers (Hu et al., 2022 We notice how DeLoRA achieves better performance on CoLA, QNLI and STS-B, and an overall significantly better average score with respect to all baselines, demonstrating efficacy in adapting language models for NLU tasks.
Results for all methods but DoRA and DeLoRA are reported from Bini (2024). For these two, a proper grid search has been run following the same setup of Bini et al. (2024). Further details cab be found in B We can see that DeL.RA achieves best results on three out of four tasks. This confirms the effectiveness our improvements, which lead to optimal average performance in this setup. On the MMLU task, ETHER and ETHER+ outperform other methods, but fall short on other tasks, achieving lower average performance compared to DeLoRA. This might be due to the limited capacity of ETHER methods from their rank limitation.
Instruction Tuning Results are in Table 5
3.4 INSIGHTS
In this section we analyze (i) the learning rate robustness properties, and (ii) the training dynamics, with focus prolonged training setting, of DeLoRA with respect other finetuning methods. Then, we analyze (iii) how weights norms differ in a pretrained model, to better understand the weights-norm scaling effect in DeLoRA.
Learning Rate Robustness. We conducted a comprehensive learning rate robustness analysis in the setting of the Surject-driven Generation task of Section 3 Evaluation done reporting DINO scores (Fig(2 Left) and Euclidean distance between finetuned and pretrained weights of a projection layer in an attention module 2 Right) across multiple methods, using a range of learning rates derived from each method's base learning rate. Our analysis shows that D is able to achieve the same robustness of ETHER+. while improving performance, whereas both LoRA and DoRA performance degrade 4x the base learning rate. We also notice how LoRA updates' distance grows at higher learning rates, while interestingly DoRA, after 8x, does not diverge further, likely thanks to its magnitude control. However this does not lead to better performance these regimes.
8

---

Published as a conference paper at ICLR 2025
Method
#param
MMLU
ARC
Tru-1
Tru-2
Avg
LLaMA-2-7B
41.81
42.92
25.21
38.95
37.22
ETHER n=32
Bini 2024
0.26M
44.57
45.14
27.91
41.83
39.86
ETHER+o-8
1.04M
44.87
46.50
29.38
43.51
41.07
LoRA,=3
et al.
2022
4.19M
43.61
46.16
28.76
42.21
40.19
DoRA,
Liu ct al.
2024a
4.19M
43.24
47.18
29.01
43.47
40.73
DeLoRA,=s
(ours)
4.19M
44.21
47.70
29.62
44.14
41.42
Table 5: Results for Instruction Tuning MMLU, ARC, and TruthfulQA benchmarks. Values represent accuracy scores achieved by different finetuning methods, Best scores are highlighted in bold, and second-best scores are underlined.
DeLoRA (Ir-Be33)
(r=6e-4)
DoRA (Ir-6e-4
ETHER. (f--6e33)
Norm to pretrained
10"
DINO Score
100
100
10a
1.0
1/1
1.32
x1
26
22
1/1000
1/33
1/16
1/8
x1
18
x30
10000
*
LR Multiplier
Multiplier
Figure 2: Learning rate robustness plots in Subject-driven generation task in terms of DINO scores (Left) and Euclidean distance between a finetuned vs pretrained projection layer weights (Right). Learning rates used for robustness evaluation were derived by multiplying the base learning rate in a range of factors.
Finetuning Regime and Prolonged Training. We further investigate the behavior of weight up- dates across different methods by measuring the Euclidean distance between finetuned weight ma- trices (after merging) and the pretrained corresponding matrices during fine-tuning. This provides us quantitative measure of the shift and rate at which fine-tuned weight matrices diverge from the pretrained weights. In Fig. (Left), we show this analysis for the out-projection matrix in one of StableDiffussions 's Unet self-attention layers. We find that LoRA- and DoRA --tained weights con- tinuously depart from the pretrained weights over the course of training, passing through an optimal regime but eventually overshooting and ending in a diverging regime (notice that best performance are typically found between 1000 and 1400 steps). In contrast, DeLoRA-trainee weights exhibit a peculiar behavior, quickly moving away from the pretrained weights, until they reach the boundary, from which they cannot diverge further. We argue that this leads to prolonged training robustness, effectively avoiding catastrophic overwriting. Qualitative examples and in Appendix E Additionally, we highlight that by adjusting the boundary parameter X, one can easily control the maximum allowable shift and, therefore, the level of finetuning robustness.
provided in Fig. 3 (Right)
Weights Norms Heterogeneity. In Fig. 4 we show the mean of column norms for weight matrices in different attention blocks of the U-Net in Stable Diffusion v1.5. By doing so, we highlight the effect of weights-none norm scaling introduced in Section We find that different modules, well as different positions in the U-Net, show systematic differences with respect to weights norms. This points differences within the pretrained model which finetuning methods should account for. Our proposed scaling is one possibility to accomplish this. Exploring more sophisticated methods to include layer-wise wise differences is interesting direction for future research.
RELATED WORK
Parameter efficient finetuning (PEFT) is an active field research, encompassing methods such as adapters (Houlsby al. 2019), prompt- and nreix-tunning variations (Lester et al. 2021 & Liang,
2021 Liu et al. 2023), , and more specialized methods such as BitFit (Zaeen et al. 2022), FourierFT
(Gao al. 2024). and LayerNorm Tuning (Zhao et al. 2024). In this paper, we propose improved

---

Published as a conference paper at ICLR 2025
Distance Intitaizatioo
UNLurrA
LOKA
DURA
300
2250
1000
2100
Step
Figure 3: (Left) Euclidean Distance of finetuned weights pretrained weights function of the number of training steps. (Right) Qualitative examples show that LoRA exhibits significant artifacts earlier in the process compared to DeLoRA, which maintains better image quality.
Avg. Norn
Block Type
Out
down Morken
down, Modks
blocks
blscks
mid, black adenndons
Layer Position
Figure 4: Average column norms of parameters in the attention modules of Stable Diffusion's Unet
PEFT method based on low-rank adapters (LoRA) first described by (Hu et al. 2022). Therefore, we focus our review of previous work on LoRA variants and refer to recent surveys (Han et al.) [024]
Xin et al. 2024) regarding PEFT methods in general. LoRA is a popular finetuning approach for
large models, featuring advantages such as low-memory footprint and no additional inference cost
(Hu et al. 2022). Compared to full-finetuning, LoRA is also less prone to catastrophic forgetting
Biderman et al.
2024).
However, beyond falling behind in performance on downstream tasks compared to full finetuning (Biderman et al. 2024), previous work has identified and attempted to address different limitations
of the original LoRA method. Lialin et al. (2023): Zi et al. (2023): y; Xia et al.) (024); y Ren et al. (2024) propose methods overcome the low-rank limitation without sacrificing memory efficiency. Simi-
larly, VeRA (Kopiczko et al., 2024) keeps the original LoRA setup but reduces trainable parameters further by only scaling the randomly initialized matrices. which are shared across layers. To ac- count for differences between layers, (Zhang et al. 2023b DDng et al. [023 ZZhag et al.) [024 Liu jet al. all,,200244 (024c) describe methods to dynamically adapt the rank of different LoRA adapters. Instead of changing the rank, in this work, propose to dynamically change the scaling of LoRA matrices for different layers, highlighting the need for layer-adaptive methods. PiSSA (Meng et al. 2024) and
MiLoRA (Wang et al. 2024) show how improved initialization of LoRA can lead to better perfor- mance and faster convergence. Zhu et al. (2024) and Hayou et al. (2024) show that LoRA matrices behave differently in terms of optimal initialization and learning rate. Our work is complementary to these findings, as we also argue for different treatments of LoR LoRAs As, but regarding different layers within model, not within the same adapter. DoRA (Liu et al., 2024a), similarly to work, tar- gets decoupling of angles and magnitudes, normalizing and scaling the full updated weight matrix W + AW on the column space, controlling each singular column of the finetuned matrices, whereas we propose to normalize the inner r-dimensional space of each AW update matrix.
CONCLUSIONS
In this work, we proposed novel parameter efficient finetuning method, DeLoRA, which com- bines the strengths of LoRA -controllable rank- and ETHER -bounded updates- to address their respective limitations. We showed that by normalizing and scaling low-rank updates, DeLoRA able to effectively decouple the angular learning from the adaptation strength, leading to competitive performance and enhanced robustness. Beyond showing the advantages of DeLoRA, we provided detailed insights into its derivation, from both perspective of LoRA and ETHER, ablating the intro- duction of each incremental innovation. Finally, we investigated DeLoRA's robustness to learning rate variations and extended training, demonstrating that its decoupled update mechanism is critical for preventing divergence from the pretrained weights. These findings offer valuable perspectives for adapting pretrained models, by addressing key limitations of current PEFT approaches.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially funded by the ERC (853489- DEXIM) and the Alfried Krupp von Bohlen und Halbach Foundation, which thank for their generous support. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. www.gauus--cnttre.co project by providing computing time the GCS Supercomputer JUWELS (Alvarez) [022]) at Jülich Supercomputing Centre (JSC). We would also like to thank Otniel-Bogdar Bogdan Mercea for the helpful feedback and discussions.
for funding this
REPRODUCIBILITY STATEMENT
To facilitate deployment and further research DeL.RR, release implementation code com rameter choices for all experiments in Appendix
as well as ablation studies and hyperpa-
REFERENCES
Damian Alvarez. Juwels cluster and booster: Exascale pathfinder with modular supercomputing architecture at juelich supercomputing centre. Journal of large-scale research facilities JLSRF, 7, 10 2021. doi: 10. 10.17815/jlsrf-7-1
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. In TMLR, 2024.
Massimo Bini, Karsten Roth, Zeynep Akata, and Anna Khoreva. large-scale models with hyperplane reflections. In ICMI, 2024.
ETHER: Efficient finetuning of
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. In arXiv, 2018
Ning Ding, Xingtai L.v, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In EMNLP, 2023.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. In ICML, 2024.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efffcient fine-tuning for large models: A comprehensive survey. In arXiv, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialiization lora finetuning dynamics. In arXiv, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-efffiient transfer learning for nlp. In ICML, 2019.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
11

---

Published as a conference paper at ICLR 2025
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In ICML, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACI, 2022.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-Shot Parameter-Efffiien Fine-Tuning is Better and Cheaper than In-Context Learning, August 2022. http:://aa...ooog//bbbb///////// 2ttp///220000068888 05638 arXiv:2205.05638 [es].
Shih-yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. JCML, 2024a.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024b.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. In A/ Open, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro A robustly optimized (bert) pre- training approach, 2020. http:httts///ppneeeetttt
eww net. fot ftttt//fooummm1115ss055---
Zequan Liu, Jiawen Lyn, Wei Zhu, and Xing Tian. Alora: Allocating low-rank adaptation for fine- tuning large language models. In NAACI, 2024c.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In arXiv, 2024.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Don-Destructive Task Composition for Transfer Learning. Proceedings the the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, URL https::::: /ttp::t/ccaaaattt: //a//aaathoolgglooolaaaaaaa lor// 20212021.a0 eaaclcl ma imain in. 39 Conference Name: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume Place: Online Publisher:
pp. 487-503 2021. doi: 10.18653/v 2tt/33/v1122222 1e122..acl-maii
Association for Computational Linguistics.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurlPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, and Jiahuan Pei. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine- tuning. In ACL, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to- text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2023.
12

---

Published as a conference paper at ICLR 2025
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-oolloiing llama model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. In arXiv, 2023b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Har- nessing minor singular components for parameter-efficient Ilm finetuning. In arXiv, 2024.
Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang, Advancing parameter efficiency in fine-tuning via representation editing. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 13445-133 13464, Bangkok, Thailand, August 2024a. Asso- URL https:
ciation for Computational Linguistics. doi: 10.18653/v1220 12224////002....loooog...
//aclanthology. wwwpo//y....//2220/..// aee44acl--aae acl-oonn.... 726
Muling Wu, Wenhao Liu, Xiaohua Wang. Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency fine- tuning via representation editing. 2024b. /ttpppp://ax....o/////aaaa////
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Man- ning, and Christopher Potts. Reft: Representation finetuning for language models, 2024c. URL https: org/ abs/
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. In arXiv, 2024.
Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-effictee fine-tuning for pre-trained vision models: A survey. In arXiv, 2024.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficien fine-tuning for transformer-bused masked language-modee models. In ACL, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In CVPR, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. In ICLR, 2023b.
Ruiyi Zhang, Rushi Qiang. Sai Ashish Somayajula, and Pengtao Xie. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In NAACL, 2024.
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal 1lm finetuning. In ICLR, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Tor- ralba. Semantic understanding scenes through the ade20k dataset. In International Journal of Computer Vision, 2019.
13

---

Published as a conference paper at ICLR 2025
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In JCML, 2024.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. In arXiv, 2023.
14

---

Published as a conference paper at ICLR 2025
A ETHER AND ETHER+ LOW-RANK LIMITATION
In ETHER and ETHER+, even if the applied transformation matrices I - wut are full-rank, the resulting weight updates to the pretrained layers are limited to low-rank. We can show this by rewriting the transformation result in residual form.
For ETHER the matrix multiplication can be written as:
HW (I W
W 2uu
where the second term on the right-hand side, by multiplying the pretrained matrix with rank-1 transformation, restricts the learnable weight updates, which are driven by u, to be rank-1.
Similarly, for ETHER+:
H #+
(W- uwTW T)
= W - -w + To - (W TW + +v WoTWWaä + (W - Tw + cu CoTW)) W)os
where the rank-1 residual matrices on the right-hand side will lead rank-4 overall weight updates.
This simple mathematical derivation, demonstrates that ETHER and ETHER+ methods are limited to be low-rank, arguably limiting the expressivity and the learning capacity of the methods.
B EXPERIMENTAL DETAILS
In this section we report further details about experiments Section 3 along with hyperparameter choices, and standard deviation results.
DDiive Generation. To find the best hyperparameters. we trained and evaluated on the first 3 subjects (10% of the data) for each method among LoRA, DoRA and DeLoRA, all with rank 16. Then, we used best hyperparameters to evaluate each method on all 30 subjects, for 3 different seeds. For LoRA and DoRA we followed best practices and fixed lambda twice the rank during hyperparemeter search. Optimal learning rate both methods is 6e-4. For DeLoR we fixed the scaling parameter to 1c-3, and found optimal learning rate of 2e-2 for the BAA matrices. Results with standard deviations are reported in Table 6
Method
DINO
CLIP-I
LoRA,=10 DoRA, 16
Hu et al.
2022
0.6864.a0m .0122
0,818 33184.0017
et al.
2024a)
0.687+.0011
0.819+.0015
DeLoRA,
(ours)
0.686..00m
0.820
Table Results with standard deviation for subject-driven image generation trained methods. Best scores are highlighted in bold, and second-best scores underlined.
GLUE. Following Wu et al. (2024c), for each benchmark task, we split the publicly available val- idation two subsets reported in Table 7 When validation sets are larger than 2K, 1K subset is used as new validation set, and the remaining as test set, otherwise the validation is split two equally sized subsets. We use the new validation set tune the hyperparameters on seed 42. Then, best hyperparameters are used to evaluate test performance for seeds 42, 43, 44, 45, 46. For each training run, we use checkpointing to save the best training run, and evaluate with that. For all experiments we use max sequence length of 512. For larger datasets (MNLI, SST-2, QNLI, QQP) we fix the À scaling learning rate to 3e-3, while for smaller datasets we fix it to le-2. For other hyperparameters
small grid search. Best values reported in Table 9 We highlight that with respect to Wu et al. (2024c). we don't 't discard any underperforming seed. Experiments with standard deviation details are reported in Table g
15

---

Published as a conference paper at ICLR 2025
Splits Sizes
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Training Set
393K
67K
3.7K
8.5K
105K
364K
2.5K
5.7K
New Validation Set
1K
436
204
522
IK
IK
139
750
New Test Set
8K
436
204
521
4.5K
39K
138
750
Table 7: GLUE dataset sizes, with new validation and test splits following Wu al.
2024c) setup.
#param
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg
Finet.
125M
87.3 4
94.4
87.9 91
62.443329
92.5
91.7
78.3
90.64.50
85.6
8..7
94.0 57
1.67
54.0
91.0+.05
87.3
1..8
4..5
82.3
IA3
0.06M
85.4
93.4
86.4
57.8
91.1=-
88.5
73.5
88.54
83.1
LoReFT
0.02M
83.1
93.4=
89.2
60.4422.0
91.21.25
87.4=
29.0
90.0
84.2
RED
83.9
93.9 sl
89.2
61.042.00
90.7+.35
87.2=
78.042.06
90.4+.a2
84.3
LoRA
0.3M
86.62.23
93.9
88.7
59.7
92.62.10
90.4=.08
75.3 79
90.3
84.7
Ada
0.3M
87.1
93.0 06
88. 1..8
58.5
92.0
90.2
77.7
90.4
84.7
Adapter
0.4M
87.0
93.3 40
88, 1.54
60.945.09
92.5
90.5 280
76.5
90.5
85.0
Del DelooR((urr))
0.3M
86.9
93.72.79
88.6
64.7
92.6
90.2-
27.3
90.6g.08
85.6
Table 8: GLUE benchmark. Comparisons of different methods finetuning RoBERTa-buee with
standard deviations. Results of all baselines are taken from Wu et al. (2024a) and Wu et al. (2024c).
Hyperparameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B B
À
12
12
4
4
12
4
12
12
Learning Rate
le-3
le-3
3e-2
le-2
3e-3
le-3
le-2
le-2
Batch Size
32
32
8
32
256
Num. Epochs
30
30
40
80
25
25
80
40
Dropout
0
0.1
0.2
0.2
0.25
0.25
0.2
Table GLUE benchmark hyperparameters.
Instruction Tuning, To assess the performance of DeLoRA in finetuning LLMs for Instruction Tuning, we adopted the experimental setup from Bini et al. (2024), finetuning Llama-2-78 (Touvron et al. 2023b) the Alpaca dataset (Taori al. 2023) for epoch, and searching for hyperparam- eters that deliver the best average performance across MMLJ, ARC, and TruthfulQA. For DoRA used learning rate of 3e-4, batch size of 8, and 100 warmup steps. For DeLoRA used an initial scaling À of 8, learning rates of 1e-2 for BA and 5e-3 for X, and other hyperparameters as DoRA. All additional reported results are sourced from Bini et al. (2024).
C FIXING THE MAGNITUDE TERM IN DoRA
In the following section we provide preliminary experiments testing fixing the magnitude in DoRA could lead to similar robustness properties as DeLoRA.
Performance. We first evaluate if fixing the magnitude term could be detrimental in terms of performance. Following the setting of our small-scale ablation Section 3.2 we run a small scale experiment comparing DoRA with its variation.
Method
DINO
CLIP-I
DoRAr=ie(fieed maginttde)
0.681
0.822
DoRA,=16
0.683
0.820
Table 10: Subject-driven Image Generation small-scale ablation
16

---

Published as a conference paper at ICLR 2025
We notice how DoRA results without updating the magnitude term seem lead only slightly underperforming results with respect to standard DoRA.
Robustness. We then the same robustness analysis reported in Fig. 2 We how fixing the magnitude term does not lead to a behavior similar to DeLoRA, but rather still follows DoRA behavior.
Plots in Fig. 5 show that simply fixing the magnitude term does not alter DoRA robustness proper- ties (Fig. Left), while actually in higher learning rate regimes seems lead further divergence (Fig. Right), not allowing the magnitude to counterbalance the divergent trend. This behavior sug- gests that keeping column norms constant might not be restrictive enough. In this regard, DeLoRA inner normalization in terms of Frobenius distance seems to be more promising strategy to avoid model divergence.
DoRA (r=6e-4)
DoRA fixed-magritude (Ir:6-44)
1.0
Norm to pretrained
0.6
DRO
0.4
40
20
1//00
1120
110
25
232
1/1000
x176
1/8
2/14
18
236
237
LR Multiplier
Muttipleer
Figure 5: Robustness analysis between DoRA with and without magnitude updates, with respect to learning rate changes from the optimal learning rate.
D ROBUSTNESS ABLATION ON DELORA'S BOUNDARY AND ANGLES
We additionally conducted an ablation DeLoRA's setting, where we run the same robustness anal- ysis of Section 3.4 by varying the learning rate of the scaling term À (affecting the boundary). and the weights BA (angular component). We notice how all methods lead to convergence, additionally demonstrating DeLoRA's robustness properties.
DeL.oRA Ir sweep A ((=iTe-))
DeLoRA sweep (Ir=6o-3
DeLoRA sweep both
Nonn to gretrained
10*
107
102
301
209
1/100
1/37
11/
1/8
25
1/200
1284
1/22
1/18
1/
2/4
LR Multiplier
Multiplier
Figure 6: Learning rate robustness plots for DeLoRA in Subject-driven generation task in terms of DINO scores (Left) and Euclidean distance finetuned vs pretrained weights of a projection layer (Right), Ablation testing impact of increasing learning rate for boundary (A) or angular weights (B
17

---

Published as a conference paper at ICLR 2025
E QUALITATIVE EXAMPLES
We report in Fig. 7 qualitative examples generated by our propopsed DeL.RAA finetuning Stable Diffusion for the tasks of Subject-driven Generation and Semantic Map to Image. While in Figure we report qualitative examples of prolonged genearation with DeLoRA, LoRA and DoRA methods.
Figure 7: Examples generated by DeLoRA- fineetnned on a small set of subject-speche specific images (left), and for semantic map to image on ADE20K (right).
DeLoRA-finetueed Stable Diffusion for personalized generation
200
600
1000
1400
1800
2200
2600
DeLoRA
LoRA
DoRA
DeLoRA
LoRA
DoRA
DeLoRA
LoRA
DoRA
Figure 8: Prolonged finetuning generated examples generated by DeLoRA, LoRA, and DoRA meth- ods, up to time step 2600.
18


Reviews:
[{"review_id": "ARuONO8Hqe", "reviewer": "Reviewer_nYwf", "summary": "Two existing fine-tuning methods, LoRA and ETHER, have shown widespread success across the field, especially the former. However, some issues with LoRA from hyperparameter tuning and length of fine-tuning arise, leading to degradation in performance. The proposed method, DeLoRA, separates the magnitude and direction of the weight updates from LoRA. This is reminiscent of multiplicative weight updates such as ETHER and OFT with constrained magnitudes. DeLoRA is shown empirically to frequently achieve better accuracy than either LoRA or ETHER.", "questions": "- In the derivations section, the LoRA and ETHER derivations seem to create different methods. This may come from the step of changing the ETHER derivation from a multiplicative adaptation to an additive one, but how this is achieved is not clear. Adding at least a few more steps to bridge between the two would be appreciated. \n\nSmall things:\n- \"Figure 5: Average...\" instead of \"Figure 5: Avergae...\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The ablation studies provide a convincing argument that somewhere in the continuum between LoRA and ETHER+ is a more powerful method. The experiments showing that different settings have different optimal locations in this continuum show that there are some directions forward with regard to which settings require different method choices, as well as defend that this continuum often contains a better method than the extremes.\n\n- DeLoRA shows promise in having strong robustness similar to ETHER with explicit weight constraints, even when learning rates are high.", "weaknesses": "- It took some time to figure out what exactly the DeLoRA method is. For clarity, what DeLoRA is could be written in its own dedicated section, not just embedded in Figure 1 without explaining the parameters or at the end of Section 2.2.1, since the two equations describing the method seem like all the others in the derivation at first glance. \n\n- It's mentioned in the introduction that one downside to LoRA is performance degradation during extended fine-tuning. This reads as though DeLoRA will overcome this issue, however, the top right of Figure 3 seems to show DeLoRA having the same issue. Are there other instances that can be added to show that DeLoRA does robustly train over many iterations?\n\n- Figure 3 could use some repetitions. The single training run makes some of the training dynamics, especially DoRA's, hard to trust that they weren't due to randomness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S83u4IEPJ8", "reviewer": "Reviewer_pmk9", "summary": "This paper proposes a new PEFT method based on LoRA called DeLoRA. \nIn specific, the authors propose to decompose the learned low-rank update in LoRA into a learnable low-rank matrix with a unit norm, and a learnable magnitude.\nThe authors also draw a connection between the proposed DeLoRA and multiplicative ETHER which learns low-rank matrices to be *multiplied* instead of *added to* the original frozen parameters. \nThe authors evaluated the proposed method on a variety of language and vision tasks and demonstrated fine-tuning performance improvement from DeLoRA.", "questions": "Besides above questions in Weakness, I also wonder:\n\n1. In Table 1,2 how was controllable boundary combined with LoRA? \n\n2. In Fig 3, DeLoRA's distance to initialization was in fact decreased, how was this happened? \n\n3. Can you elaborate more on Fig 4? I didn't find any discussion in the main body. \n\n4. Can you test the proposed method on more language tasks, such as math reasoning?  \n\nI am willing to adjust my score if these questions can be addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing of this paper is good and it is pleasant to read. \n \n2. The proposed method is well-presented. The connection to existing methods ETHER and DoRA is also amenable. \n\n3. The experiments look sufficient to me.", "weaknesses": "After reading through the paper, my main concerns lie in the concept of DeLoRA. Namely, \n\n1. While an original main motivation of DeLoRA, as presented in Introduction, is to \"introduce a boundary on the weight updates\" to LoRA, the addition of *learnable* parameter $\\lambda$ (without any regularization/constraint, if I read the paper correctly) removes this guarantee. As a result, learnable DeLoRA solution is just a strict subset of LoRA's --- the proposed parameterization introduces some inductive bias. Its underlying rationale, which cannot be fully explained by the presented motivation, is unclear. \n\n2. On the other hand, as the idea of decomposing the learning into magnitude and direction has already been proposed in DoRA, I feel that how different DeLoRA really is from DoRA, e.g., if they are in fact equivalent, needs more elaboration and careful discussion. Currently the authors only provide a equation-by-equation comparison, but no insights is provided in this form. \nIn addition, I think this difference should be highlighted in Introduction and Method, instead of just in Related Work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbEvfazafg", "reviewer": "Reviewer_vsop", "summary": "This paper addresses some limitations in current parameter-efficient fine-tuning (PEFT) methods, particularly the robustness and flexibility challenges faced by additive methods like LoRA and multiplicative approaches like ETHER. The authors introduce DeLoRA, a fine-tuning method that normalizes and scales learnable low-rank matrices to improve hyperparameter robustness without sacrificing performance. Through evaluations on subject-driven image generation and large language model (LLM) instruction tuning tasks, DeLoRA demonstrates consistent improvements over existing PEFT methods, offering a balanced solution that enhances adaptability and performance in fine-tuning applications.", "questions": "See the above \"weakness\".", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tOverall, this paper is well-written and presents its ideas in a clear, accessible manner, making it easy to follow.\n2.\tThe authors provide insightful reviews of two primary categories of fine-tuning methods: additive techniques such as LoRA and multiplicative approaches like ETHER, effectively highlighting the limitations inherent to each approach.\n3.\tThe bounded approach constrains the updating scheme within a Frobenius norm ball, promoting a more robust selection of learning rates and potentially enhancing stability across different training settings.", "weaknesses": "Overall, the contribution of this paper appears somewhat incremental, as it focuses on constraining weight updates within a restricted range—a concept that has also been examined in the DoRA paper. Additionally, the performance comparison with LoRA and DoRA is challenging to assess directly, as the experiments were conducted on different datasets.\n\n1.\t[Key Issue] The concept of decoupling the angles and scales in LoRA has been explored extensively in the DoRA method. Consequently, the contribution of this paper feels somewhat incremental in comparison to DoRA. Similar to DeLoRA, DoRA can limit its scale term within a fixed range, thereby preventing over-drifting in downstream adaptations. Would this lead to the same robustness effect?\n2.\t[key issue] LoRA and DoRA algorithms are typically evaluated on well-established NLP and NLU tasks, such as the GLUE benchmarks and Commonsense Reasoning, where their performance characteristics are widely understood. Thus, a fairer performance comparison would be achievable if DeLoRA were also tested on these datasets.\n3.\tIn Eq (13), the updating is limited to |H-I| <= r, effectively limiting weight changes to within a Frobenius norm ball. Then it is not surprising that the algorithm is not sensitive to the change of learning rates. But a fundamental question is: is this restriction always necessary? For instance, in cases where the pretrained model diverges significantly from the target task, imposing a strict parameter update range might inadvertently hamper overall performance.\n4.\tIn line 172, the authors state that 'multiplicative fine-tuning methods show stronger performance in generative models.' However, the results in Table 2 suggest that additive methods like LoRA and DoRA outperform OFT approaches, which appears contradictory and requires clarification.\n5.\tIn Table 2, the performance improvement over DoRA appears marginal. Could the authors clarify if DoRA was evaluated with r=16? To ensure statistical reliability, it would be beneficial to repeat this experiment multiple times and report the standard deviation, as the observed improvement may fall within the range of experimental error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xt650otvMN", "reviewer": "Reviewer_EKmU", "summary": "Motivated by the lack of robustness of LoRA fine-tuning with respect to hyperparameter choice and catastrophic forgetting phenomena, the authors propose a novel PEFT method trying to tackle these problems. Their proposal merges the advantages of LoRA and ETHER, being able thus to control rank and maximal distance between fine-tuned and frozen weights.", "questions": "I leave here some questions/comments concerning the work:\n1) Would it be possible to test the method on other common benchmarks (e.g. GLUE Deberta V3)?\n2) The only concern I have about the new parametrization is that if a column of $A$ and $B$ is driven towards zero during training, normalization may cause instabilities. Did the authors ever observe this in practice? I would be worried of this phenomenon happening especially when choosing a high rank for the correction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well presented and the proposed methods is both efficient and easy to implement.", "weaknesses": "While I find no evident weakness in the manuscript and its content, I believe a broader experimental setting should be proposed to test the method (see questions for more details). Moreover, given the motivation of the work, I encourage the authors to include additional evidence concerning the robustness of their proposed method with respect to hyperparameters (and maybe something more about the avoidance of forgetting phenomena).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)