ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws

## Abstract

Abstract Low-Rank Adaptation (LoRA) is the bread and butter of Large Language Model (LLM) finetuning. LoRA learns an additive low-rank perturbation, 𝑨 ⁢ 𝑩 𝑨 𝑩 {\bm{A}}{\bm{B}} bold_italic_A bold_italic_B , of a pretrained matrix parameter 𝑾 𝑾 {\bm{W}} bold_italic_W to align the model to a new task or dataset with 𝑾 + 𝑨 ⁢ 𝑩 𝑾 𝑨 𝑩 {\bm{W}}+{\bm{A}}{\bm{B}} bold_italic_W + bold_italic_A bold_italic_B . We identify three core limitations to LoRA for finetuning–a setting that employs limited amount of data and training steps. First, LoRA employs Dropout to prevent overfitting. We prove that Dropout is only suitable for long training episodes but fails to converge to a reliable regularizer for short training episodes. Second, LoRA’s initialization of 𝑩 𝑩 {\bm{B}} bold_italic_B at 0 0 creates a slow training dynamic between 𝑨 𝑨 {\bm{A}} bold_italic_A and 𝑩 𝑩 {\bm{B}} bold_italic_B . That dynamic is also exacerbated by Dropout that further slows the escape from 0 0 for 𝑩 𝑩 {\bm{B}} bold_italic_B which is particularly harmful for short training episodes. Third, the scaling factor multiplying each LoRA additive perturbation creates “short-sighted” interactions between the LoRA modules of different layers. Motivated by principled analysis of those limitations, we find an elegant solution: a Dropout-free, scaling-free, LoRA with Adaptive Learning rate–coined ALLoRA. By scaling the per sample and per parameter gradients with a coefficient inversely proportional to parameters’ ℓ 2 subscript ℓ 2 \ell_{2} roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT norm, ALLoRA alleviates those three limitations. As a by-product, ALLoRA removes two hyper-parameters from LoRA: the scaling factor and the dropout rate. Empirical results show that ALLoRA admits better accuracy than LoRA on various settings, including against recent LoRA variants such as Weight-Decomposed Low-Rank Adaptation (DoRA). Ablation studies show our solution is the optimal in a family of weight-dependent / output-dependent approaches on various LLMs including the latest Llama3.

## 1 Introduction

Large Language Models (LLMs) (Hoffmann et al., ; Touvron et al., ; Jiang et al., ) are Deep Neural Networks (DNNs)–commonly built from Transformer with self-attention–built for sequence processing, e.g., Natural Language Processing (NLP). LLMs have radically changed the way we approach NLP (Chowdhary, ) by removing the need for handcrafted feature engineering such as bags of words (Zhang et al., ). Instead, current solutions directly operate on the input data–or a lossless compression known as tokens (Shibata et al., ). Because we now have access to humongous amount of text data, the standard training pipeline for LLMs take the following form. First, the LLM is pretrained on a large text corpus through next-token prediction. That autoregressive pretext-task enables the LLM to learn the underlying dynamic of the language. Commonly, RLHF is also employed after pretraining to make the model’s behavior shift from autoregressive to agentic. Then, the LLM is fine-tuned on a more specific downstream task or dataset. That fine-tuning is user-specific and plays a fundamental role in making LLMs practically useful but relies on much more limited datasets, as we formalize below.

Premise: The training regime involved in pretraining and fine-tuning are drastically different. The former employs limitless data and abundant training steps while the latter employs limited data and few training steps.

That premise is now widely accepted upon as the latest state-of-the-art LLM solutions stem from the numerous open-source industry groups such as Meta’s Llama (Dubey et al., ), Google’s Gemma (Team et al., ), Apple’s OpenELM (Mehta et al., ), or Cohere’s Aya. Hence, as LLM practitioners, most of the attention is now turning into deriving fine-tuning strategies that meet the very particular needs of fine-tuning LLMs.

To tackle that paradigm shift introduced by the pretraining-finetuning strategy, specialized methods have been developed, such as the eponymous Low-Rank Adaption (LoRA). LoRA has fueled countless deployment of LLMs–as it took a gigantic leap in accommodating for the fine-tuning regime. In short, LoRA proposes to fine-tune a LLM by learning an additive low-rank matrix perturbation to some of the LLM’s internal parameter matrices. Core to its design, LoRA leverages (i) Dropout (Srivastava et al., ) as a mean to prevent overfitting to the fine-tuning task, and (ii) zero-initialization to ensure that training starts from the LLM’s pretrained mapping, and (iii) a scaling factor that rescales the LoRA’s matrix factorization. While the impact of LoRA is ubiquitous, we nonetheless believe that LoRA could be further improved based on three observations.

LoRA’s three fatal flaws for finetuning: First, Dropout–a stochastic regularizer–whose benefits quickly vanish when considering fine-tuning, and can in fact introduce detrimental additional variance to the training. Second, the zero-initialization which is difficult to escape from due to Dropout’s implicit regularization. Third, the scaling parameter that introduces nonlinear interactions between LoRA modules of different layers.

While each of those three design choices are well-motivated when considering long training, e.g., pretraining, it becomes harder to prove their benefits when considering fine-tuning that only employs a minimal amount of training steps. That is why, after carefully bringing to light and studying the above flaws of LoRA–in the context of fine-tuning–in section 3, we will propose a novel variation of LoRA that we coin ALLoRA for Adaptive Learning rate LoRA (section 4). ALLoRA proposes to remove the Dropout regularizer and the scaling factor while adding an adaptive learning rate for the low-rank matrices entries. As depicted in LABEL:code:allora, the implementation is straightforward with theoretical and practical benefits. First, by removing the Dropout regularization and the scaling factor, ALLoRA is simpler to employ as it no longer requires cross-validation of those parameters. Second, we demonstrate how ALLoRA improves performances over LoRA and alternatives such as DoRA. In short, our adaptive learning rate strategy is able to prevent over-fitting, learn competitive solutions, and converge more quickly than alternatives–all while employing less hyper-parameters.

We summarize our contributions below:

- 1.
We identify three inefficiencies (sections 3.1, 3.2 and 3.3) in the current LoRA design making it unfit for short training, i.e., finetuning, that we empirically validate in section 3.4.
- 2.
We propose a novel adaptive learning rate variation of LoRA–coined ALLoRA–free of two of the original LoRA’s complicated designs: the Dropout regularizer and the scaling factor (section 4).
- 3.
We empirically validate the benefits of ALLoRA over numerous datasets and model architectures including the latest Llama3 family (sections 4.4 and 4.5). We obtain that despite ALLoRA employing less hyperparameters than LoRA, it is able to outperform its counter part and recent variants such as DoRA consistently.

The full codebase to reproduce figures and tables is publicly available(^1^11[http://github.com/rbalestr-lab/allora](http://github.com/rbalestr-lab/allora)) along side the finetuned models’ weights(^2^22[http://bit.ly/allora-weights](http://bit.ly/allora-weights)).

## 2 Related Works

LoRA is a type of Parameter Efficient Fine Tuning (PEFT) designed to reduce the cost of finetuning, especially with LLMs. As LLMs typically have large number of parameters–in the scale of billions–one can not afford to finetune all those parameters on a particular downstream task or dataset. Existing PEFT can be divided into three categories, namely Adapter-based Methods, Prompt-based Methods, and LoRA.

Adapter-based methods (Houlsby et al., ; He et al., ; Karimi Mahabadi et al., ; ) introduce additional trainable modules, a.k.a. the adapters, into the original backbone whose weights are frozen during the finetuning. In Houlsby et al. (), linear modules were added in sequence to the existing layer, while in He et al. (), they were added in parallel to the existing layer for the sake of better performance.

Prompt-based methods (Lester et al., ; Razdaibiedina et al., ; Wang et al., ) introduce soft tokens as trainable parameters and prepend them to the prompt. This category is the least intrusive as the finetuning can be done by only prompting the LLMs.
However, prompt-based methods are in general sensitive to initialization and their overall effectiveness is affected.

LoRA (Hu et al., ) uses low-rank matrices to simulate weight changes of the pretrained weights. Since low-rank matrices can be merged back to original weights, LoRA does not incur any additional cost at inference, which is a significant advantage over the other two categories. Many variants were proposed lately. For example, in Zhang et al. (), SVD decomposition was employed to determine
significance of singular values, and less important ones are pruned. Hyeon-Woo et al. () applies low-rank Hadamard product to federated learning. Qiu et al. () and Liu et al. () adopt orthogonal factorization and applied to diffusion models. Renduchintala et al. () introduces weight tying and realizes more savings on number of parameters. A unified LoRA family was introduced for Stable diffusion in Yeh et al. (). Different combinations of LoRA are chosen for different tasks in Ponti et al. (). A scaling vectors is learnt to adjust a pair of frozen random matrices shared across layers in Kopiczko et al. ().

More recently, Liu et al. () proposes decomposing the weights into directional and magnitude components to boost accuracy.
Hayou et al. () studies the optimal initialization of the low-rank matrices, and a follow-up work (Hayou et al., ) proposes to apply different learning rate to different low-rank matrices. Superficially this is similar to one of our idea to adapt learning rate, though our idea is inspired by a principled study of dropout (Srivastava et al., ).

More broadly, Zhao et al. () applies the low-rank concept to compute low-rank gradients directly. Jang et al. () provides a study on the existence and convergence of LoRA solutions. And Zhang & Pilanci () is a study of the potential ill conditioned low-rank matrices.

## 3 A Critical Analysis Of LoRA for Finetuning

PEFT is the main bridge between large pretrained LLMs and specialized practical use-cases. Hence, PEFT research is extremely active which led to numerous variations of LoRA being developed. Our theoretical study builds on the seminal version formalized below, other LoRA variants will be compared in our experimental sections 4.4 and 4.5.

###### Definition 1 .

(Low Rank Adapters (LoRA) from Hu et al. ()). For any weight matrix ${\bm{W}}\in\mathbb{R}^{n_{1}\times n_{2}}$
in the pretrained model, we constrain its update in the finetuning process by representing the latter with a low-rank
decomposition $\tilde{{\bm{W}}}={\bm{W}}+\frac{\alpha}{r}{\bm{B}}{\bm{A}}$. Here, only the weight matrices ${\bm{B}}\in\mathbb{R}^{n_{1}\times r}$, ${\bm{A}}\in\mathbb{R}^{r\times n_{2}}$
are trainable. The rank $r\ll\min(n_{1},n_{2})$ and $\alpha\in\mathbb{R}$ are tunable constants.

Despite LoRA’s simplicity, very little attention was put on its core premises within a finetuning context, i.e., with limited amount of training steps. We propose here a principled and critical study of LoRA, culminating with our finding of three core flaws of LoRA for short training (sections 3.1, 3.2 and 3.3). The following section 4 will investigate our solution, ALLoRA.

###### Definition 1 .

### 3.1 First Flaw: A Stochastic Regularization that Will not Converge

LoRA’s regularization comes from Dropout (Srivastava et al., ), i.e., applying a multiplicative random binary mask at the bottleneck of the matrix factorization. The use of Dropout would seem justified since Wager et al. () showed in the linear regime that Dropout acts as a variant of $\ell_{2}$ regularization on normalized design matrix (inputs), a result also found in the previous study of Wang & Manning (). However, those theoretical results deal with expectations, i.e., infinite training time. Similarly, previous empirical studies of Dropout showed great regularization benefits, but all those studies only considered full training, i.e., long training episodes that ultimately converge to their expectation. We argue that those beneficial findings do not hold during fine-tuning which only employs limited training steps.

To understand the impact of Dropout in terms of regularization, we will consider a few variations of models and study the discrepancy between the expected benefit of Dropout with its empirical realisations. We will conduct validation on real dataset and LLMs to support our theory throughout the section, and in particular in section 3.4. Also, we consider here and in section 3.2 $\alpha/r$ to be $1$ without loss of generality, none of our results are impacted by that constant scaling factor.

Linear model with full training. Let’s first consider the original setting of a linear model with Dropout applied to its predictions. That is, we consider the following Ordinary Least Squares (OLS) setting $\|{\bm{Y}}-({\bm{X}}{\bm{W}})\odot{\bm{V}}\|_{F}^{2}$,
with ${\bm{Y}}\in\mathbb{R}^{N\times C},{\bm{X}}\in\mathbb{R}^{N\times D},{\bm{W}}
\in\mathbb{R}^{D\times C}$ and the random realization of dropout matrix ${\bm{V}}\in\{0,\frac{1}{p}\}^{N\times C}$. We note that such parametrization of Dropout is commonly employed in the literature to ensure that its expectation is equal to $1$, and it is also PyTorch’s official implementation. We have the following property that has motivated the use of Dropout through more than a decade by now:

$$ $\displaystyle\mathbb{E}$ $\displaystyle\left[\|{\bm{Y}}-({\bm{X}}{\bm{W}})\odot{\bm{V}}\|_{F}^{2}\right]$ $\displaystyle=\|{\bm{Y}}\|_{F}^{2}+\mathbb{E}\left[-2Tr\left({\bm{Y}}\left({ \bm{V}}^{\top}\odot({\bm{X}}{\bm{W}})^{\top}\right)\right)+Tr\left(\left({\bm{ V}}^{\top}\odot({\bm{X}}{\bm{W}})^{\top}\right)\left(({\bm{X}}{\bm{W}})\odot{ \bm{V}}\right)\right)\right]$ $\displaystyle={\color[rgb]{0,0,1}\|{\bm{Y}}\|_{F}^{2}-2Tr\left({\bm{Y}}({\bm{X }}{\bm{W}})^{\top}\right)+Tr\left(({\bm{X}}{\bm{W}})^{\top}({\bm{X}}{\bm{W}}) \right)}+{\color[rgb]{1,.5,0}\left(\frac{1}{\lambda}-1\right)Tr\left(({\bm{X}} {\bm{W}})^{\top}({\bm{X}}{\bm{W}})\right)},$ $$

which is solved for ${\bm{W}}=p({\bm{X}}^{\top}{\bm{X}})^{-1}{\bm{Y}}^{\top}{\bm{X}}$. We note that we decomposed the loss into the terms that happen in the original (Dropout-free) setting, in blue, and the Dropout induces terms in orange. Comparing that with the usual Tikhonov regularization that produces ${\bm{W}}=({\bm{X}}^{\top}{\bm{X}}+\lambda{\bm{I}})^{-1}{\bm{Y}}^{\top}{\bm{X}}$ we see then whenever the eigenvalues of ${\bm{X}}^{\top}{\bm{X}}$ are all identical to a positive constant $c$ (e.g. when ${\bm{X}}$ is whitened), ${\bm{W}}=\frac{c}{c+\lambda}({\bm{X}}^{\top}{\bm{X}})^{-1}{\bm{Y}}^{\top}{\bm{
X}}$ hence recovering the dropout solution. Assuming $c=1$ without loss of generality, we obtain that applying Dropout with rate $p=\frac{1}{1+\lambda}$ is equivalent to applying Tikhonov regularization with rate $\lambda$. As a result, we see that if training for long enough, Dropout can efficiently replace explicit forms of regularization such as weight decay. While the above results recovers known theoretical analysis of Dropout–showing its benefits as an implicit regularizer–those derivations only emerge from taking expectation of the loss, i.e., considering infinite training steps.
For us, the question thus turns into the following: what are the benefits of Dropout as a LoRA regularizer for very short training regime such as finetuning?

Figure: Figure 1: We depict the absolute difference (y-axis) between the empirical and expected finetuning LoRA loss with varying Dropout rates (rows) on different datasets (columns) as a function of the number of Dropout realisation (x-axis). We observe that regardless of the dataset and Dropout probability, the empirical error is a poor estimate of the true expected loss even after hundreds of averaged realisations. Hence, finetuning with Dropout produces a large amount of random noise that go well beyond its regularization benefit which only emerges after a large number of steps. That finding is also confirmed by the LLM experiment in fig. 5.
Refer to caption: extracted/5922338/convergence_lora.png

LoRA with Dropout fails to converge. Let’s denote the LoRA linear finetuning setting as follows

$$ $\displaystyle\mathcal{L}\triangleq\|{\bm{Y}}-{\bm{X}}LoRA_{{\bm{A}},{\bm{B}}}( {\bm{W}})\|_{F}^{2}=\|{\bm{Y}}-{\bm{X}}{\bm{W}}-(({\bm{X}}{\bm{A}})\odot{\bm{V }}){\bm{B}}\|_{F}^{2}.$ (1) $$

In the above setting, we can slightly modify the above analysis to obtain the following result

$$ $\displaystyle\mathbb{E}$ $\displaystyle\left[\mathcal{L}\right]={\color[rgb]{0,0,1}\|{\bm{Y}}-{\bm{X}}{ \bm{W}}\|_{F}^{2}-2Tr\left(({\bm{Y}}-{\bm{X}}{\bm{W}})^{\top}({\bm{X}}{\bm{A}} {\bm{B}})\right)+\|{\bm{X}}{\bm{A}}{\bm{B}}\|_{F}^{2}}{\color[rgb]{1,.5,0}+ \frac{1-\lambda}{\lambda}\sum_{r=1}^{R}\left\|{\bm{X}}({\bm{A}})_{.,r}({\bm{B} })_{r,.}\right\|_{F}^{2}},$ $$

where we recall that $\frac{1}{\lambda}-1>0,\forall\lambda\in(0,1)$. As for the full training setting, we decomposed the loss into the terms that happen in the original (Dropout-free) setting, in blue, and the Dropout induces terms in orange. Before diving into the impact of Dropout in terms of regularization, it is interesting to understand how many samples or training steps are needed for the empirical estimate to converge to the above expectation. We propose that analysis in fig. 1. We clearly see that even for small Dropout value $0.05$, the convergence is quite slow on the various settings we explored.

Figure: Figure 2: Depiction of the distribution of standard deviation of gradients (y-axis) w.r.t. the second layer of a MLP trained for MNIST (left) and CIFAR10 (right) classification, equipped with Dropout. At each training epoch (x-axis), we consider a single mini-batch and compute the gradients under numerous Dropout realisation. For each entry in the matrix of gradients, we compute the standard deviation and report the distribution over entries. We clearly see that while the average variance of the gradient decreases slightly during training, the tail significantly increases, leading to unstable training in finetuning regimes.
Refer to caption: extracted/5922338/dropout_std.png

To better understand those trends, we propose a first simple bound on “how far off” is the expectation to the empirical estimator as a function of the number of realisations, or, steps. The detailed derivations as provided in appendix A.We obtain

$$ $\mathbb{E}\left[\left|\frac{1}{N}\sum_{n=1}^{N}\|{\bm{Y}}-{\bm{X}}{\bm{W}}-(({ \bm{X}}{\bm{A}})\odot{\bm{V}}_{n}){\bm{B}}\|_{F}^{2}-\mathbb{E}\left[\|{\bm{Y} }-{\bm{X}}{\bm{W}}-(({\bm{X}}{\bm{A}})\odot{\bm{V}}){\bm{B}}\|_{F}^{2}\right] \right|\right]\\ \leq\frac{Std\left[\|{\bm{Y}}-{\bm{X}}{\bm{W}}-(({\bm{X}}{\bm{A}})\odot{\bm{V} }){\bm{B}}\|_{F}^{2}\right]}{\sqrt{N}}.$ $$

As a result, the benefit of $N$, in our case the number of training steps, only appears for large $N$ as depicted in fig. 1. This is particularly true as training progresses where the value of the standard deviation of the error ($Std\left[\|{\bm{Y}}-({\bm{X}}{\bm{W}})\odot{\bm{V}}\|_{F}^{2}\right]$) may increase under the Dropout. While this could be dataset and model specific, we illustrate the distribution of that random variable during training in fig. 2. We look at the standard deviation of the gradients as a function of Dropout realisation in a simple MLP with MNIST classification task as training progresses and observe that the distribution becomes more and more heavy-tailed as training progresses, indicating that some mini-batch may receive highly noisy loss and gradient updates, hence making it hard to recover during short finetuning settings.

### 3.2 Second Flaw: Zero Initialization and Unfair Regularization

The second flaw we uncover comes from the zero initialization of ${\bm{B}}$. As we will see, that is a limitation regardless of employing Dropout or not, but Dropout exacerbates it.

Zero initialization implies imbalanced training dynamics. A peculiarity of LoRA compared to most other deep learning framework lies in its asymmetric initialization. While ${\bm{A}}$ is initialized with random entries, ${\bm{B}}$ is initialized at 0 0. This initialization is intuitive when looking at the output of LoRA being 0 0 at first, i.e., one starts from the original model and then moves away from that if needed for the finetuning task. However, this seemingly reasonable initialization creates a strong imbalance in the training dynamic of ${\bm{A}}$ and ${\bm{B}}$. To see that, we can extend our derivation to obtain the derivative of the expected loss as

$$ $\displaystyle\nabla_{{\bm{A}}}\mathbb{E}\left[\mathcal{L}\right]=$ $\displaystyle-{\bm{X}}^{\top}\left({\bm{Y}}-{\bm{X}}{\bm{W}}\right){\bm{B}}^{ \top}+{\bm{X}}^{\top}{\bm{X}}{\bm{A}}\left(\frac{1-\lambda}{\lambda}\text{diag }(\|{\bm{B}}_{1,:}\|_{2}^{2},\dots,\|{\bm{B}}_{r,:}\|_{2}^{2})+{\bm{B}}{\bm{B} }^{\top}\right),$ $\displaystyle\nabla_{{\bm{B}}}\mathbb{E}\left[\mathcal{L}\right]=$ $\displaystyle-{\bm{A}}^{\top}{\bm{X}}^{\top}\left({\bm{Y}}-{\bm{X}}{\bm{W}}\right)$ $\displaystyle+\left({\bm{A}}^{\top}{\bm{X}}^{\top}{\bm{X}}{\bm{A}}+\frac{1- \lambda}{\lambda}\text{diag}(\|({\bm{X}}{\bm{A}})_{1,:}\|_{2}^{2},\dots,\|({ \bm{X}}{\bm{A}})_{r,:}\|_{2}^{2})\right){\bm{B}},$ $$

hence the effective gradient norm for ${\bm{A}}$ is 0 0 during the first few steps because of ${\bm{B}}$ being 0 0, while the gradient norm for ${\bm{B}}$ will be high $\|{\bm{A}}^{\top}{\bm{X}}^{\top}\left({\bm{Y}}-{\bm{X}}{\bm{W}}\right)\|_{F}^{
2}\gg 0$–regardless of the Dropout rate employed. While such issue only concerns the first few training steps, it is clear that it will be detrimental in a finetuning regime where the total number of training steps is limited. We note that if ${\bm{A}}$ is zero-initialized instead of ${\bm{B}}$, our entire argument still holds as the same slow training dynamic appears albeit with respect to ${\bm{A}}$ instead of ${\bm{B}}$. Our findings support the conclusion of Hayou et al. () that demonstrated how ${\bm{A}}$ and ${\bm{B}}$ should receive different learning rates to improve LoRA’s performances.

Figure: Figure 3: Depiction of the norm of ${\bm{A}}$ (left), the norm of ${\bm{B}}$ (middle) and training loss (right) for a MNIST LoRA finetuning experiment. We see that as training progresses (x-axis) as the impact of increased Dropout probability (colors) has a disproportionate regularization impact on ${\bm{B}}$ while barely impacting the norm of ${\bm{A}}$, indicating an asymmetry in Dropout’s implicit regularization that makes LoRA slow to train.
Refer to caption: extracted/5922338/lora_regularization.png

Dropout further slows down the escape from 0 0. An additional issue arises when using Dropout. In that setting, the escape of ${\bm{B}}$ from 0 0, which is needed for ${\bm{A}}$ to also learn and for the LoRA module to be effective, will be further slowed down. To better characterize that effect, let’s study the close-form regularization impact of Dropout using the expected loss derived in section 3.1.
As per the full training regime, we see that Dropout acts as a regularization on ${\bm{A}}$ and ${\bm{B}}$ based on their alignment with ${\bm{X}}$. In fact, taking derivative with respect to the loss, we see that the orange contributing term’s gradient is given by

$$ $\displaystyle\nabla_{{\bm{A}}}\left(\frac{1}{\lambda}-1\right)\sum_{i=1}^{r} \left\|{\bm{X}}({\bm{A}})_{:,i}({\bm{B}})_{i,:}\right\|_{F}^{2}$ $\displaystyle=\left(\frac{1}{\lambda}-1\right){\bm{X}}^{\top}{\bm{X}}{\bm{A}} \text{diag}(\|{\bm{B}}_{1,:}\|_{2}^{2},\dots,\|{\bm{B}}_{r,:}\|_{2}^{2}),$ $\displaystyle\nabla_{{\bm{B}}}\left(\frac{1}{\lambda}-1\right)\sum_{i=1}^{r} \left\|{\bm{X}}({\bm{A}})_{:,i}({\bm{B}})_{i,:}\right\|_{F}^{2}$ $\displaystyle=\left(\frac{1}{\lambda}-1\right)\text{diag}(\|({\bm{X}}{\bm{A}}) _{1,:}\|_{2}^{2},\dots,\|({\bm{X}}{\bm{A}})_{r,:}\|_{2}^{2}){\bm{B}}.$ $$

Hence, Dropout with LoRA regularizes the matrices ${\bm{A}}$ and ${\bm{B}}$ using a weight decay type of regularization but weighted by the squared $\ell_{2}$ norm of the rows of ${\bm{B}}$ and the squared $\ell_{2}$ norm of the columns of ${\bm{X}}{\bm{A}}$, respectively. Those findings bring to light an unfair regularization impact of Dropout with LoRA. At initialization, i.e., when ${\bm{B}}=0$, the strength of the regularizer on ${\bm{A}}$ will be null. Yet, the strength of the regularizer on ${\bm{B}}$ is large as it is equal to the norm of ${\bm{X}}{\bm{A}}$ which, even at initialization, will be largely greater than 0 0. This is concerning since not only ${\bm{B}}$ starts from a zero-norm initialization opposed to ${\bm{A}}$, but also its regularization is stronger than that of ${\bm{A}}$. We depict that dynamic in a practical scenario in fig. 3. We observe that the impact of Dropout’s regularization, as measured by varying the value of $\lambda$, is minimal on the training dynamic of ${\bm{A}}$, only altering the final norm by about 30%, while the impact of Dropout’s regularization on ${\bm{B}}$ is drastic, producing solutions with norms varying by more than 500%.

### 3.3 Third Flaw: Ripple Effect Of Scaling Factor

The third and last flaw we investigate deals with the scaling factor. While section 3.1 and section 3.2 considered it to be $1$ without loss of generality, we now use back the value from definition 1, i.e., using $\eta=\frac{\alpha}{r}$ as the Scaling Factor.

The scaling factor plays an important role to match $||{\bm{B}}{\bm{A}}||$ to a comparable level with $||{\bm{W}}||$. Despite its effectiveness, the scaling factor creates a ripple effect across layers of a LLM and may make finetuning unstable. Hu et al. () discussed the importance of the scaling factor and suggest to tune it carefully to prevent ${\bm{B}}{\bm{A}}$ from overwhelming ${\bm{W}}$. From a different perspective, Houlsby et al. () empirically showed the scale of the initialization of ${\bm{B}}{\bm{A}}$ can negatively impact validation accuracy. Later, Hayou et al. () argued that for the best performance, either ${\bm{A}}$ or ${\bm{B}}$ must be initialized at 0 0.

To illustrate the ripple effect, we adopt a multi-linear model which is a simplified version of the toy model in Hayou et al. ().

$$ $f_{l}({\bm{x}})={\bm{W}}_{l}f_{l-1}({\bm{x}}),\quad l\in[L]$ (2) $$

where $L\geq 1$ is the number of layers. Applying LoRA at each layer gives

$$ $f_{l}({\bm{x}})=({\bm{W}}_{l}+\eta{\bm{B}}_{l}{\bm{A}}_{l})f_{l-1}({\bm{x}})$ $$

Expanding the equation, we have
$f_{L}({\bm{x}})=({\bm{W}}_{L}+\eta{\bm{B}}_{L}{\bm{A}}_{L})...({\bm{W}}_{1}+
\eta{\bm{B}}_{1}{\bm{A}}_{1}){\bm{x}}$.
Let $||\cdot||_{M}$ be a matrix norm induced by a vector norm $||\cdot||_{v}$. Applying Cauchy-Schwartz and the triangle inequality, we have

$$ $\displaystyle||f_{L}({\bm{x}})||$ $\displaystyle\leq||f_{L}({\bm{x}})||_{v}=||({\bm{W}}_{L}+\eta{\bm{B}}_{L}{\bm{ A}}_{L})...({\bm{W}}_{1}+\eta{\bm{B}}_{1}{\bm{A}}_{1})||_{M}\cdot||{\bm{x}}||_ {v}$ $\displaystyle\leq(||{\bm{W}}_{L}||+\eta||{\bm{B}}_{L}{\bm{A}}_{L}||)...(||{\bm {W}}_{1}||+\eta||{\bm{B}}_{1}{\bm{A}}_{1}||)||{\bm{x}}||$ $\displaystyle=C(1+\eta\frac{||{\bm{B}}_{L}{\bm{A}}_{L}||}{||{\bm{W}}_{L}||}).. .(1+\eta\frac{||{\bm{B}}_{1}{\bm{A}}_{1}||}{||{\bm{W}}_{1}||})||{\bm{x}}||$ $\displaystyle\leq C(1+\eta\bar{m})^{L}||{\bm{x}}||=\Theta((1+\eta)^{L})$ $$

where $C=||{\bm{W}}_{L}||...||{\bm{W}}_{1}||$ is a constant, and $\bar{m}=\frac{1}{L}\sum_{l\in L}{\frac{||{\bm{B}}_{l}{\bm{A}}_{l}||}{||{\bm{W}
}_{L}||}}$ is also a constant in a single forward pass. Notice that all the inequalities are tight.

###### Proposition 1 .

(Ripple Effect) In the worst case, a constant scaling factor $\eta$ may cause the final output of a single forward pass of a LoRA finetuned model to grow exponentially w.r.t. the number of layers in the model.

We also note that proposition 1 is especially limiting for LLMs that most commonly resort to increased depth rather than increased width to scale up their capacity (Hestness et al., ).

Figure: Figure 4: Test set performance gap (y-axis) between the close-form Dropout regularization and its empirical estimate as a function of training steps (x-axis). We observe that the benefit of Dropout as a regularizer falls short for finetuning (small number of training steps) compared to pretraining regimes (large number of trainign steps).
Refer to caption: extracted/5922338/lora_gap.png

Figure: Figure 5: Left: LoRA with varying Dropout rates: High value of Dropout provides the strongest performance after long fine-tuning and the weakest performance after short fine-tuning. Each line is an average of 3 runs. X-axis is epochs, and Y-axis is accuracy. Right: ALLoRA escapes from 0 rapidly, and then tapers off into a measured move. The starting phase matches that of LoRA with a much higher learning rate. LoRA with a lower learning rate can reach the same level of $L^{2}$ norm but much slower. This finding echoes fig. 1 that showed how Dropout’s induced noise does not converge until long training is employed.
Refer to caption: extracted/5922338/fig-accuracy-epoch-dropout.png

###### Proposition 1 .

### 3.4 Empirical validation of the harmful impact of Dropout for short fine-tuning

Because we were able to derive the expectation in close-form, we can now perform LoRA fine-tuning on the expected loss to see how the idealised performance varies. To that end, we propose a simple experiment with a 3-layer MLP on MNIST. We pretrain the model on a subset of $4096$ training set images, and then perform LoRA fine-tunining on another set of $512$ training images. We measure the gap in the test loss when training on the idealized (expected) loss and on the empirical loss on the test set throughout finetuning in fig. 4. We clearly observe that not only the gap consistently increases with the Dropout rate, but also that the gap culminates after a few hundred training steps and then slowly goes down. In short, the detrimental impact of Dropout is maximum during short finetuning episodes.

Moving to LLMs, we also confirm that simplified model’s intuition with LLM experiments below. We run experiments with Qwen2-0.5B on Bias in Bios, a classification task to predict the job of an employee given the job description, for various dropout rates
and up to 10 epochs. We note that 10 epochs is already a large number of finetuning iterations for practical scenarios. It shows that large dropout rates successfully avoid overfitting, but at the cost of lower accuracy at lower number
of epochs, while no dropout sees higher accuracy at lower number of epochs, but will overfit at later epochs. See fig. 5 Left and table 5.

Having concluded our brief tour of LoRA’s possible shortcomings when it comes to fine-tuning LLMs in a few shots, we now propose to study our attempt at improving LoRA through a novel parametrization.

## 4 ALLoRA: Escaping LoRA’s Flaws for Fine-Tuning

### 4.1 Deconstructing LoRA

Section 3 summarized three flaws of LoRA, which we show can be addressed by a single solution: ALLoRA.

First we establish the underlying links among dropout, scaling factor, and learning rate. Consider the LoRA finetuning of a single layer as in eq. 2, $f({\bm{x}})=({\bm{W}}+\eta{\bm{B}}{\bm{A}}){\bm{x}}$. Following Hayou et al. () and without loss of generality, we can simplify the model by assuming ${\bm{W}}=0$, which is equivalent to defining $\tilde{{\bm{y}}}={\bm{y}}-{\bm{W}}{\bm{x}}$, and rewriting the loss function by $\tilde{{\bm{y}}}$. Also assuming $\eta=1$, we have $f({\bm{x}})={\bm{B}}{\bm{A}}{\bm{x}}$. The goal is to minimize loss $\mathcal{L}$ whose gradient is $g=\frac{\partial\mathcal{L}}{\partial({\bm{B}}{\bm{A}})}$. $f({\bm{x}})\in\mathbb{R}^{n_{1}}$ is a column vector. Expanding it per row gives

$$ $(f({\bm{x}}))_{i}=({\bm{B}}{\bm{A}})_{i,:}{\bm{x}},\quad i\in[n_{1}]$ (3) $$

The effect of dropping out $(f({\bm{x}}))_{i}$ for a given $i$ is equivalent to applying a per-row scaling factor $\eta_{i}=0$ to $({\bm{B}}{\bm{A}})_{i,:}$. Note that this is true only for $({\bm{B}}{\bm{A}})_{i,:}$, the effect on $({\bm{B}}{\bm{A}})_{j,:},j\neq i$ is slightly different. Since $\frac{d\eta\cdot f(x)}{dx}=\eta\frac{df(x)}{x}$, $\eta_{i}=0$ is implicitly applied to the $i$-th row of the gradient $g_{i}=(\frac{\partial\mathcal{L}}{\partial({\bm{B}}{\bm{A}})})_{i,:}$, which is again a scaling factor applied to the learning rate $l$.

The observation reveals that both scaling factor and dropout are adaptions on LoRA output $f(x)$, and both have effects on gradient. We are inspired to formalize a general framework that subsumes both, within which we can use a principled approach to systematically discover novel solutions.

###### Definition 2 .

(Adaptive Learning) Consider a single layer linear model $f({\bm{x}})={\bm{B}}{\bm{A}}{\bm{x}}$ with gradient
$g({\bm{x}})=\frac{\partial\mathcal{L}(f({\bm{x}}))}{\partial({\bm{B}}{\bm{A}})}$. Let Output Adaptor be a function $f_{o}:\mathbb{R}^{n_{1}}\rightarrow\mathbb{R}^{n_{1}}$, Gradient Adaptor be a function
$f_{g}:\mathbb{R}^{n_{1}\times n_{2}}\rightarrow\mathbb{R}^{n_{1}\times n_{2}}$. Define adapted output $\tilde{f}$ and adapted gradient $\tilde{g}$ by

$$ $\begin{cases}\tilde{f}=f_{o}\circ f\\ \tilde{g}=f_{g}\circ g\end{cases}$ (4) $$

Adaptive learning is to use the adapted $\tilde{f}$ and $\tilde{g}$ in place of $f$ and $g$ respectively in the learning process.

Let $I:x\mapsto x$ be the identity function. Then a natural corollary is that all learning is adaptive learning (when $f_{o}=I$ and $f_{g}=I$). Note that $\mathcal{L}$ is a function of $f({\bm{x}})$, hence $g({\bm{x}})=\frac{\partial(\mathcal{L}\circ f)}{\partial({\bm{B}}{\bm{A}})}$. Use $\tilde{f}$ in place of $f$ defines a naturally adapted gradient $\tilde{g}=\frac{\partial(\mathcal{L}\circ\tilde{f})}{\partial({\bm{B}}{\bm{A}}
)}=\frac{\partial(\mathcal{L}\circ f_{o}\circ f)}{\partial({\bm{B}}{\bm{A}})}$. When it’s clear from the context, we omit $\tilde{g}$ if it’s naturally defined by a non-trivial $\tilde{f}$.

###### Definition 2 .

### 4.2 ALLoRA: Less Hyper-Parameters and More Stability for Finetuning

Under the Adaptive Learning framework, scaling factor is define by $f_{o}=\kappa:x\mapsto\eta x$.

$$ $\begin{cases}\tilde{f}=\kappa\circ f=\eta f\\ \tilde{g}=\kappa\circ f=\eta g\end{cases}$ (5) $$

One idea to reduce the ripple effect while keeping the positive effect of scaling factor is to force $f_{o}=I$, while keeping $\tilde{g}$ intact, which is to use a larger learning rate $\eta\cdot l$. Nonetheless, a fixed learning rate cannot simultaneously achieve both fast escape from 0 and, once away from 0, measured discovery of optimal direction. We think an adaptive learning rate that is inversely proportional to $||({\bm{B}}{\bm{A}})_{i,:}||$ is a good candidate to realize our idea. We use the function $1/\sqrt{||({\bm{B}}{\bm{A}})_{i,:}||+1/\eta^{2}}$ which reaches maximum $\eta$ at $||({\bm{B}}{\bm{A}})_{i,:}||=0$ and then tapers down when $||({\bm{B}}{\bm{A}})_{i,:}||$ increases (fig. 6).

Formally, ALLoRA is defined by

$$ $\begin{cases}\tilde{f}=I\circ f\\ \tilde{g}_{i}=1/\sqrt{||({\bm{B}}{\bm{A}})_{i,:}||+1/\eta^{2}}\cdot g_{i}, \quad i\in[n_{1}]\end{cases}$ $$

where $\eta$ is a hyperparameter. Note that this does not introduce a new hyperparameter. We split
learning rate into a constant base learning rate $l_{b}$ and $\eta$, and the effective learning rate is $\eta\cdot l_{b}$.

One more implementation detail is the backward pass computes, in addition to the gradients of ${\bm{A}}$ and ${\bm{B}}$, also the gradient of the input from layer below, and propagate which back to the layer below. We only modify the gradients of ${\bm{A}}$ and ${\bm{B}}$, but not that of the input. This helps further restrict the changes within each layer and reduce ripple effect.

To quickly verify our idea, we add probing code to trace the $L^{2}$ norm of row vectors of ${\bm{B}}{\bm{A}}$. As shown in fig. 5 Right, adaptive learning rate escapes from 0 0 rapidly, the speed matches with LoRA with a learning rate that is $\eta\cdot l_{b}$. Then it finds an approriate level and enters measured discovery of optimal directions. LoRA with a learning rate lower than $\eta\cdot l_{b}$ can reach the same level, but at a much slower pace. The experiment is with Snowflake Arctic XS and Rotten Tomatoes.

Note that ALLoRA multiplies different scaling factors to different rows of ${\bm{B}}{\bm{A}}$’s gradient stochastically (because ${\bm{A}}$ is stochastically initialized and ${\bm{B}}{\bm{A}}$ is stochastically learnt). Intuitively it is a generalization of Dropout which multiplies binary factors (0 0 or $\frac{1}{1-p}$) to different rows stochastically. We hypothesize that ALLoRA may recover some regularisation effect of Dropout and invite researchers to find a theoretical proof.

### 4.3 A Family of Adaptive Solutions

We adopt a principled approach to explore other reasonable designs that fall into the adaptive learning framework defined by eq. 4.

First notice that in $(f({\bm{x}}))_{i}=({\bm{B}}{\bm{A}})_{i,:}{\bm{x}}$, $(f({\bm{x}}))_{i}$ and $({\bm{B}}{\bm{A}})_{i,:}$ define each other. So instead of adapt the learning by $({\bm{B}}{\bm{A}})_{i,:}$,
we can also adapt it by $(f({\bm{x}}))_{i}$, which is Output-Dependent, or ALLoRA-OD, defined by

$$ $\begin{cases}\tilde{f}=I\circ f\\ \tilde{g}_{i}=1/\sqrt{|(f({\bm{x}}))_{i}|+1/\eta^{2}}\cdot g_{i},\quad i\in[n_ {1}]\end{cases}$ $$

Note that $(f({\bm{x}}))_{i}$ is a scalar, hence we use its absolute value. Qualitatively, ALLoRA-OD subjects to the stochastic noise in ${\bm{x}}$ because $(f({\bm{x}}))_{i}=({\bm{B}}{\bm{A}})_{i,:}{\bm{x}}$. According to Smith et al. (), this type of stochastic noise is an implicit regularization. Our conjecture is that it may drag down the accuracy just as dropout does, and therefore ALLoRA-OD may not be as good as ALLoRA.

Given the link between learning rate and scaling factor, we may achieve similar effect by switching from adaptive learning rate to Adaptive Scaling Factor, or ASF-LoRA, defined by

$$ $\tilde{f}_{i}=1/\sqrt{|(f({\bm{x}}))_{i}|+1/\eta^{2}}\cdot f_{i},\quad i\in[n_ {1}]$ $$

Note that $\tilde{g}$ is naturally defined by using $\tilde{f}$ in place of $f$.
The potential downside is that it introduces ripple effect across layers, which may blur accuracy. And our conjecture is again ASF-LoRA may not be as good as ALLoRA.

One more caveat of ASF-LoRA is that we cannot merge ${\bm{B}}{\bm{A}}$ back to ${\bm{W}}$ as we need to apply $f_{o}$ to the LoRA output.

### 4.4 Empirical Validation: Perception Tasks

Our first set of experiments gauges the performance of ALLoRA on perception tasks. Mainstream LLMs nowadays are mostly pretrained by next token prediction, which is good for generative tasks, but may not be a good fit for perception tasks such as Natural Language Understanding (NLU) and Sentiment Analysis (SA). In fact, we observe subpar accuracy when finetuning popular open-weight models for NLU and SA tasks (see table 6). We hope to show that ALLoRA may help boost the accuracy for perception tasks.

For our experiments, we pick three midsized LLMs: Qwen2-0.5B, Snowflake-Artic-L, and OpenELM-450M, and four NLU and SA datasets: Bias in Bios, Emotion, Rotten Tomatoes, and Yelp Review. To demonstrate the stability of ALLoRA, we run the experiments with various $\eta^{2}\in\{1,2,4\}$ with a fixed baseline learning rate $l_{b}=1e-4$. To be fair for LoRA, we run LoRA at learning rate $l\in\{1e-4,\sqrt{2}e-4,2e-4\}$, respectively. We finetune for $2$ epochs. Each experiment is run $5$ times and we report average final accuracy. Table 2 Left shows the accuracy gap between ALLoRA and LoRA, where positive numbers indicate ALLoRA has better accuracy. The result shows that ALLoRA in general admits better accuracy over plain LoRA. Average improvement over all cases is $0.3\%$.

In the experiment, the dropout rate for ALLoRA is $0.0$ and that for LoRA is $0.05$. We also run ALLoRA+D, the version of ALLoRA with Dropout, also at dropout rate $0.05$. Table 2 Right shows that there is no evident difference between ALLoRA and ALLoRA+D, matching our theoretical result from section 3.1.

Since $\eta$ originates from $\frac{\alpha}{r}$ in definition 1, we also run at various LoRA ranks $r$, and the results show that ALLoRA’s advantage is consistent across different $r$ (table 1).

**Table 1: Accuracy comparison of various LoRA ranks on Qwen2-0.5B and Emotion. LoRA’s and DoRA’s learning rate $l=1e-4$, ALLoRA’s base learning rate is $l_{b}=1e-4$, and $\eta=1$. Each cell is an average over 5 runs.**
| Method | $r=4$ | $r=8$ | $r=16$ | $r=32$ | $r=64$ |
| --- | --- | --- | --- | --- | --- |
| LoRA | 33.09 | 34.01 | 35.19 | 35.59 | 38.13 |
| DoRA | 33.34 | 34.14 | 35.50 | 36.81 | 38.07 |
| ALLoRA | 33.45 | 34.80 | 35.61 | 37.18 | 38.27 |

**Table 2: Accuracy gap between ALLoRA and LoRA. Each cell is an average of 5 runs. Left: ALLoRA admits better accuracy than that of LoRA. Right: ALLoRA+D, the version with $0.05$ dropout rate, admits comparable accuracy than that of ALLoRA.**
|  |  |
| --- | --- |

### 4.5 Empirical Validation: Commonsense Reasoning

We also compare ALLoRA with DoRA (Liu et al. ()), a recent LoRA variant that demonstrated superb performance over a range of PEFT methods. Since DoRA results are universally better than other PEFT methods, we only compare ALLoRA to DoRA. We run experiments on LLaMA-7B, LLaMA2-7B, and LLaMA3-8B on 8 Commonsense tasks. Following DoRA’s setup, for each model, we run both ALLoRA and ALLoRA+D with LoRA rank $r\in\{16,32\}$ and for $3$ epochs. Table 3 shows that for all cases, either ALLoRA or ALLoRA+D has the best average accuracy. On average, ALLoRA and ALLoRA+D each boosted accuracy by $0.3\%$ over DoRA.

Note that we run experiments with various $\eta^{2}\in\{1,2,4\}$ and report the best accuracy, this follows DoRA’s practices to run with various learning rate $l\in\{1e-4,2e-4\}$ and report the best.

In table 3 we also report the number of trainable parameters as a percentage of the number of pretrained parameters. Since ALLoRA does not introduce additional trainable parameters, its trainable parameters are slightly lower than that of DoRA.

**Table 3: Accuracy comparison of LLaMA 7B, LLaMA2 7B, and LLaMA3 8B between ALLoRA and DoRA on eight commonsense reasoning datasets. DoRA results are taken from Liu et al. (). ALLoRA+D is the version of ALLoRA with $0.05$ dropout rate.**
| Model<br>LoRA Rank | Method | # Params<br>% | BoolQ | PIQA | SIQA | HSwag | WGrande | ARC-e | ARC-c | OBQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-7B<br>16 | DoRA | 0.43 | 70.0 | 82.6 | 79.7 | 83.2 | 80.6 | 80.6 | 65.4 | 77.6 | 77.5 |
| ALLoRA+D (ours) | 0.41 | 69.4 | 82.7 | 78.3 | 84.8 | 80.0 | 80.9 | 65.7 | 79.2 | 77.6 |  |
| ALLoRA (ours) | 0.41 | 69.2 | 80.8 | 78.5 | 83.9 | 81.1 | 80.8 | 65.2 | 78.2 | 77.2 |  |
| LLaMA-7B<br>32 | DoRA | 0.84 | 69.7 | 83.4 | 78.6 | 87.2 | 81.0 | 81.9 | 66.2 | 79.2 | 78.4 |
| ALLoRA+D (ours) | 0.83 | 70.0 | 82.3 | 78.1 | 84.6 | 82.2 | 81.0 | 67.9 | 81.0 | 78.4 |  |
| ALLoRA (ours) | 0.83 | 70.2 | 82.6 | 78.6 | 83.8 | 81.1 | 81.0 | 66.3 | 82.6 | 78.3 |  |
| LLaMA2-7B<br>16 | DoRA | 0.43 | 72.0 | 83.1 | 79.9 | 89.1 | 83.0 | 84.5 | 71.0 | 81.2 | 80.5 |
| ALLoRA+D (ours) | 0.41 | 71.7 | 83.7 | 79.5 | 91.4 | 82.4 | 84.3 | 69.2 | 81.2 | 80.4 |  |
| ALLoRA (ours) | 0.41 | 72.4 | 83.9 | 80.0 | 90.8 | 83.0 | 84.7 | 71.3 | 80.2 | 80.8 |  |
| LLaMA2-7B<br>32 | DoRA | 0.84 | 71.8 | 83.7 | 76.0 | 89.1 | 82.6 | 83.7 | 68.2 | 82.4 | 79.7 |
| ALLoRA+D (ours) | 0.83 | 72.2 | 83.1 | 79.6 | 91.2 | 84.5 | 84.5 | 71.0 | 80.0 | 80.8 |  |
| ALLoRA (ours) | 0.83 | 72.3 | 83.8 | 79.3 | 91.4 | 83.0 | 85.0 | 71.2 | 82.2 | 81.0 |  |
| LLaMA3-8B<br>16 | DoRA | 0.35 | 74.5 | 88.8 | 80.3 | 95.5 | 84.7 | 90.1 | 79.1 | 87.2 | 85.0 |
| ALLoRA+D (ours) | 0.35 | 75.2 | 88.9 | 80.8 | 95.6 | 84.7 | 90.2 | 80.6 | 85.8 | 85.2 |  |
| ALLoRA (ours) | 0.35 | 74.5 | 89.1 | 80.4 | 95.5 | 85.8 | 90.7 | 80.3 | 86.0 | 85.3 |  |
| LLaMA3-8B<br>32 | DoRA | 0.71 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 |
| ALLoRA+D (ours) | 0.70 | 74.5 | 88.9 | 81.8 | 95.9 | 86.3 | 90.4 | 80.5 | 87.6 | 85.8 |  |
| ALLoRA (ours) | 0.70 | 75.1 | 88.7 | 81.8 | 95.8 | 85.4 | 91.0 | 81.1 | 86.6 | 85.7 |  |

**Table 4: Accuracy gap between ALLoRA and other adaptive approaches. Each cell is an average of 5 runs. Top left: ALLoRA admits better accuracy than that of ALLoRA-OD, where learning rate is LoRA Output-Dependent. Top right: ALLoRA admits better accuracy than that of ASF-LoRA, where an Adaptive Scale Factor is applied to LoRA output. Bottom left: ALLoRA admits better accuracy than that of LoRA with a fixed scaling factor. Bottom right: Adaptive scale factor is not better than a fixed scale factor.**
|  |  |
| --- | --- |
|  |  |

## 5 Ablation Study

Using the same setup in section 4.4, we run experiments with ALLoRA-OD, the output-dependent variant, and ASF-LoRA, the adaptive scaling factor variant. We also run LoRA with comparable fixed scaling factors to form an objective baseline for ASF-LoRA.

### 5.1 ALLoRA-OD

Table 4 Top Left shows the accuracy gap between ALLoRA and ALLoRA-OD. A positive number indicates that ALLoRA has better accuracy. Overall speaking, ALLoRA has better accuracy than ALLoRA-OD. But the difference is moderate, as average improvement over all cases is $0.4\%$.

The result matches our conjecture that stochastic noise experienced by ALLoRA-OD might have dragged down accuracy at early epochs.

### 5.2 ASF-LoRA and LoRA with Fixed Scaling Factor

Since a scaling factor on output is implicitly also a scaling factor on gradient, we use the same $\eta$ when comparing between ALLoRA and ASF-LoRA, i.e., the gradient adaptor $f_{g}$ in ALLoRA and the output adaptor $f_{o}$ in ASF-LoRA use the same $\eta$.

Table 4 Top Right shows the accuracy gap between ALLoRA and ASF-LoRA. A positive number indicates that ALLoRA has better accuracy. ALLoRA has a significant advantage over ASF-LoRA as average improvement over all cases is $1.1\%$. Since we know that ALLoRA-OD is only slightly worse than ALLoRA, the evidence leans toward that Adaptive Learning Rate is in general a better solution family than Adaptive Scaling Factor.

We also run LoRA at comparable fixed scaling factors $\frac{\alpha}{r}\in\{1,\sqrt{2},2\}$. The results, as shown in table 4 Bottom, show that

- •
ASF-LoRA is not a competitive method, as over half of cases see ASF-LoRA’s accuracy significantly lower than LoRA with a comparable fixed scaling factor (positive numbers in table 4 Bottom Right).
- •
ALLoRA is significantly better than LoRA with a comparable fixed scaling factor, as average improvement over all cases is $0.9\%$.

## 6 Conclusion and Future Work

This paper identifies three major flaws of LoRA, namely dropout, zero-initialization, and scaling factor. We conducted principled analysis and proved that dropout is not a must-have in the finetuning regime. After uncovering the hidden connection between dropout, scaling factor, and learning rate, we proposed a unified adaptive learning framework to address them all: ALLoRA. Empirical results show that ALLoRA admits better accuracy than plain LoRA over multiple backbones, datasets, and learning rates; and better accuracy than recent successful LoRA variants such as DoRA. Ablation study shows that ALLoRA is the optimal in a family of adaptive methods.

We list a few interesting research directions and invite researchers to explore the frontier opened-up by our research:

- •
The adaptive learning framework introduced by this paper is generic and may find broad applications beyond LoRA. Other use cases such as pretraining may not have the constraints that the weight matrix must be initialized at 0 0. But they may have other types of constraints, which may be solved by adaptive learning with a different adaptor function.
- •
Within the LoRA use case, we only examine one particular adaptor function, there could be other adaptor functions that have superior performance.
- •
We only provide empirical evidence that ALLoRA admits better accuracy and hypothesize that ALLoRA is implicitly a regularization. Theoretical guarantee is needed, especially for the convoluted case where the base model has multiple layers.
- •
Starting from 0 0 weights may avoid the lottery ticket hypothesis (Frankle & Carbin ()), for good or bad, where adaptive learning rate can be a handy tool.

## Appendix A Proof of upper bound

###### Proof.

To simplify the derivations, we will denote ${\bm{Y}}-{\bm{X}}{\bm{W}}-(({\bm{X}}{\bm{A}})\odot{\bm{V}}_{n}){\bm{B}}$ by ${\bm{Z}}_{n}$.

$$ $\displaystyle\mathbb{E}\left[\left|\frac{1}{N}\sum_{n=1}^{N}\|{\bm{Z}}_{n}\|_{ F}^{2}-\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]\right|\right]$ $\displaystyle=$ $\displaystyle\mathbb{E}\left[\sqrt{\left(\frac{1}{N}\sum_{n=1}^{N}\|{\bm{Z}}_{ n}\|_{F}^{2}-\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]\right)^{2}}\right]$ $\displaystyle\leq$ $\displaystyle\sqrt{\mathbb{E}\left[\left(\frac{1}{N}\sum_{n=1}^{N}\|{\bm{Z}}_{ n}\|_{F}^{2}-\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]\right)^{2}\right]}$ $\displaystyle=$ $\displaystyle\sqrt{\mathbb{E}\left[\left(\frac{1}{N}\sum_{n=1}^{N}\|{\bm{Z}}_{ n}\|_{F}^{2}\right)^{2}\right]-\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]^{2}}$ $\displaystyle=$ $\displaystyle\sqrt{\frac{1}{N^{2}}\sum_{n=1}^{N}\mathbb{E}\left[\|{\bm{Z}}_{n} \|_{F}^{4}\right]-\frac{1}{N}\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]^{2}}$ $\displaystyle=$ $\displaystyle\sqrt{\frac{1}{N^{2}}\sum_{n=1}^{N}\left(\mathbb{E}\left[\|{\bm{Z }}_{n}\|_{F}^{4}\right]-\mathbb{E}\left[\|{\bm{Z}}\|_{F}^{2}\right]^{2}\right)}$ $\displaystyle=$ $\displaystyle\sqrt{\frac{1}{N}Var\left[\|{\bm{Z}}\|_{F}^{2}\right]}$ $\displaystyle=$ $\displaystyle\frac{Std\left[\|{\bm{Z}}\|_{F}^{2}\right]}{\sqrt{N}}$ $$

∎

###### Proof.

## Appendix B Finetuning Accuracy At Various Dropout Rates

Table 5 contains accuracy of various dropout rates at different number of epochs.

**Table 5: Reprise of fig. 5 Left depicting that the epoch at which the LoRA fine-tuned model reaches the best accuracy increases with the Dropout rate, i.e., the larger the probability to drop dimensions, the more regularization is applied and the better the final performance–but only after very long fine-tuning.**
| dropout | 0.0 | 0.05 | 0.1 | 0.2 | 0.4 |
| --- | --- | --- | --- | --- | --- |
| Acc. at epoch=3 | 77.92 | 77.96 | 77.68 | 77.44 | 77.22 |
| Acc. at epoch=10 | 78.00 | 79.95 | 80.50 | 80.77 | 80.65 |
| Max acc. | 79.81 | 80.27 | 80.54 | 80.78 | 80.65 |
| Epoch of max acc. | 6 | 8 | 8 | 9 | 10 |

## Appendix C Adaptive Function

Figure 6 is an adaptive function that provides output value when $|x|=0$, and then tapers down when $|x|>0$.

Figure: Figure 6: Adaptive function $1/\sqrt{|x|+1/\eta^{2}}$ for $\eta^{2}=1,2,4$.
Refer to caption: extracted/5922338/fig-1-sqrt-abs-eta2.png

## Appendix D Perception Tasks

Table 6 shows the accuracy data of all of our experiments on perception tasks. Each cell is an average of 5 runs. ALLoRA is universally better than LoRA in terms of accuracy.

**Table 6: Accuracy comparison of various models, datasets, and learning rates between ALLoRA and plain LoRA. ALLoRA+D is the version of ALLoRA with $0.05$ dropout rate. LoRA’s learning rate is $\eta\times 1e-4$, ALLoRA’s and ALLoRA+D’s base learning rate is $1e-4$. Each cell is an average over 5 runs.**
| Learning Rate | Method | Qwen2-0.5B | Snowflake-Artic-L | OpenELM-450M |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| b-in-b | emotion | rotten | emotion | rotten | yelp | emotion | rotten | yelp |  |  |
| $\eta^{2}=1.0$<br>$l_{b}=1e-4$ | LoRA | 70.81 | 35.59 | 53.19 | 87.06 | 77.71 | 63.61 | 84.98 | 87.95 | 71.41 |
| ALLoRA+D (ours) | 71.29 | 37.18 | 53.64 | 86.93 | 78.29 | 63.55 | 86.07 | 88.20 | 71.32 |  |
| ALLoRA (ours) | 71.53 | 37.35 | 54.33 | 86.98 | 78.19 | 63.52 | 86.04 | 88.11 | 71.36 |  |
| $\eta^{2}=2.0$<br>$l_{b}=1e-4$ | LoRA | 73.70 | 37.76 | 54.32 | 88.30 | 79.34 | 64.26 | 90.01 | 88.74 | 71.62 |
| ALLoRA+D (ours) | 73.77 | 38.52 | 54.67 | 88.23 | 79.44 | 64.24 | 90.01 | 88.95 | 71.63 |  |
| ALLoRA (ours) | 73.99 | 38.09 | 55.37 | 88.52 | 79.16 | 64.33 | 89.93 | 89.04 | 71.68 |  |
| $\eta^{2}=4.0$<br>$l_{b}=1e-4$ | LoRA | 75.60 | 38.68 | 54.90 | 88.95 | 80.04 | 64.76 | 91.33 | 89.55 | 71.79 |
| ALLoRA+D (ours) | 75.72 | 39.00 | 55.83 | 89.08 | 80.24 | 64.61 | 91.27 | 89.61 | 71.83 |  |
| ALLoRA (ours) | 75.52 | 39.47 | 55.68 | 88.91 | 80.24 | 64.76 | 91.43 | 89.47 | 71.82 |  |

**Table 7: Accuracy comparison of various models, datasets, and learning rates between ALLoRA and other adaptive approaches. for adaptive learning rate approaches, i.e., ALLoRA and ALLoRA-OD, base learning rate is $1e-4$. For adaptive scaling factor, i.e., ASF-LoRA, learning rate is fixed at $1e-4$, an adaptive scaling factor $1/\sqrt{|x|+1/\eta^{2}}$ is applied. For LoRA, a fixed scaling factor $\eta$ is applied, and learning rate is fixed at $1e-4$. Each cell is an average over 5 runs.**
| $\eta^{2}$ | Method | Qwen2-0.5B | Snowflake-Artic-L | OpenELM-450M |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| emotion | rotten | emotion | rotten | emotion | rotten |  |  |
| $1.0$ | ALLoRA (ours) | 37.35 | 54.33 | 86.98 | 78.19 | 86.04 | 88.11 |
| ALLoRA-OD | 36.91 | 52.31 | 87.24 | 77.47 | 86.31 | 88.11 |  |
| ASF-LoRA | 36.40 | 53.43 | 87.30 | 78.03 | 83.44 | 87.58 |  |
| LoRA $\times\eta$ | 35.59 | 53.19 | 87.06 | 77.71 | 84.98 | 87.95 |  |
| $2.0$ | ALLoRA (ours) | 38.09 | 55.37 | 88.52 | 79.16 | 89.93 | 89.04 |
| ALLoRA-OD | 38.15 | 53.55 | 88.22 | 79.16 | 89.86 | 89.02 |  |
| ASF-LoRA | 37.20 | 54.17 | 87.74 | 78.18 | 86.26 | 88.22 |  |
| LoRA $\times\eta$ | 36.61 | 54.47 | 87.62 | 78.72 | 88.77 | 88.39 |  |
| $4.0$ | ALLoRA (ours) | 39.47 | 55.68 | 88.91 | 80.24 | 91.43 | 89.47 |
| ALLoRA-OD | 38.67 | 54.95 | 88.97 | 80.21 | 91.27 | 89.38 |  |
| ASF-LoRA | 38.45 | 54.41 | 87.97 | 79.23 | 89.64 | 88.80 |  |
| LoRA $\times\eta$ | 37.63 | 53.70 | 88.49 | 79.31 | 90.25 | 89.08 |  |

## Appendix E Ablation

Table 7 shows the accuracy data of all of our ablation study on perception tasks. Each cell is an average of 5 runs. ALLoRA is universally better than the rest in the family.

## Appendix F Code

Figure: Listing 1: ALLoRA Code

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Figure 3, the authors compare LoRA, ALLoRA-0 (ALLoRA without dropout), and ALLoRA. From the previous sections, it is apparent that dropout is detrimental to LoRA’s performance given the authors only fine-tune for two epochs. I am curious about the comparison between ALLoRA-0 and LoRA without dropout.
2. In Section 3.1, why does the average variance of the gradient decrease slightly during training while the worst-case scenario increases? Does this observation hold for other models, including LLMs?
3. In page 2, the authors called the three issues of LoRA "fatal flaws". I think the wording here is not appropriate. Vanilla LoRA has been also used extensively in many applications and it worked just fine.
soundness: 2
strengths: 1. The concept of adaptive learning for LoRA is somewhat innovative, although the formulation resembles adaptive optimizers in full fine-tuning, such as AdaFactor.
2. The experiments encompass a diverse range of model types and sizes, addressing both vision and language models.
summary: This paper first identifies three main challenges in fine-tuning with LoRA: dropout, zero initialization, and the difficulty of setting an appropriate scaling factor. The authors propose solutions to these issues by adaptively setting the learning rate. Through various experiments, they demonstrate that their methods outperform LoRA, DoRA, and other ALLoRA-like baselines.
weaknesses: The primary issue with this paper lies in its clarity and coherence, particularly in Sections 3.1 and 3.2. The mathematical reasoning presented is often inconsistent with the conclusions drawn. 

For instance, in Section 3.2, the claim that “as the training set size (n) decreases, the optimization landscape for V becomes degenerate” is not convincingly supported by the preceding discussion on the Hessian product.

In page 4, it is unclear to me how the formula (the "how far off" expectation bound) helps to argue the main conclusion of this section. The paragraph after the formula just does not make sense to me.

Additionally, there is a notable lack of comparison with relevant prior work that addresses similar challenges outlined in Sections 2 and 3. Specifically, regarding initialization, existing studies such as PiSSA[1], MiLoRA[2], and LoRA-GA[3] have already highlighted the benefits of non-zero initialization for LoRA. The discussion on scaling factor selection has also been explored by RSLoRA[4]. 

Furthermore, while the authors cite LoRA+[5] in relation to learning rate adjustments, there is no direct comparison between the fixed learning rate strategy of LoRA+ and the adaptive approach of ALLoRA. A thorough comparison with these established methods is necessary to substantiate ALLoRA’s claims and enhance the paper’s credibility. I agree that some of the mentioned works are very recent and may be considered concurrent work considering ICLR policy. However, given the pace of the development of this direction, and many of these recent papers dealing with very similar issues and adopting similar ideas (some from arguably more principled way), it is difficult to assess the novelty and contribution of the present submission without comparison to any of them. Even if I ignore all these recent works, I am not super exicited with the theoretical reasoning and empirical results of the paper. Hence, I think the submission may not pass the bar of ICLR acceptance.

[1]	 Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. "Pissa: Principal singular values and singular vectors adaptation of large language models." arXiv preprint arXiv:2404.02948 (2024).
[2]	 Wang, Hanqing, et al. "MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning." arXiv preprint arXiv:2406.09044 (2024).
[3]	 Wang, Shaowen, Linxi Yu, and Jian Li. "LoRA-GA: Low-Rank Adaptation with Gradient Approximation." arXiv preprint arXiv:2407.05000 (2024).
[4]	Kalajdzievski, Damjan. "A rank stabilization scaling factor for fine-tuning with lora." arXiv preprint arXiv:2312.03732 (2023).
[5]	 Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "Lora+: Efficient low rank adaptation of large models." arXiv preprint arXiv:2402.12354 (2024).

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: Q1. It seems that the size of the adapter needs to be calculated for each batch to determine the adaptive learning rate. How much additional computation does this require? Specifically, how much does it increase the actual runtime compared to LoRA?

Q2. Equation 3 assumes a multi-linear model. However, in most large language models (LLMs), several activations, FFNs, and normalizations are applied after the LoRA adapter. In such cases, the formula in line 279 may not hold. For instance, if normalization is applied after the layer, $\Vert f_L(x) \Vert$ could be smaller than the calculated value. This raises the question of whether the Ripple Effect could still occur in LLMs under these circumstances.

Q3. Figure 3 illustrates performance improvements through figures for comparison with the baseline. However, in my opinion, given the relatively modest performance gains, this does not effectively demonstrate the model's superiority. Could you present the performance more clearly by indicating the absolute values?
soundness: 2
strengths: - The novel aspect lies in identifying and addressing the issues commonly accepted in LoRA training, such as dropout, initialization with zero, and the scaling factor.
- By removing two types of hyperparameters and replacing them with an adaptive learning rate, there is an advantage in significantly reducing the need for grid search in model tuning.
summary: The authors point out three core issues when fine-tuning LLM with LoRA: i) the instability of dropout in short-term training such as fine-tuning, ii) poor optimization landscape due to the zero initialization of the adapter, and iii) the scaling factor causes nonlinear interactions between LoRA layers. To solve these issues, the authors remove two hyperparameters, dropout and scaling factor, and introduce an adaptive learning rate. To address this issue, the authors removed two hyperparameters, dropout and scaling factor, and instead introduced an adaptive learning rate approach. This approach helps the model quickly move away from the initial zero state and gradually reduces the learning rate as training progresses, promoting stable convergence.
weaknesses: W1. Figure 1 illustrates the distribution of the gradient's standard deviation as training progresses. While it is clear that the expectation in an OLS setting is affected by the standard deviation, the relationship between this formula and dropout remains unclear. For instance, the spiking gradients in Figure 1 could be due to out-of-distribution inputs. However, this does not seem to be directly related to dropout. In my view, to understand any potential connection between Figure 1 and dropout, it would be necessary to show that with dropout set to 0, the gradients remain stable, exhibiting low standard deviation without any spikes.

W2. The authors highlighted instability in LoRA-based fine-tuning methods with fewer epochs. However, through empirical experiments, it was not observed that training had actually stabilized or that the convergence speed had improved.

W3. The experiments lack baseline comparisons. For Table 1, experiments should include basic baselines, such as LoRA, and reference studies [1,2].

W4. (Minor) The ablation results in Figure 4 are not entirely clear. Personally, I believe that instead of using four separate plots to show relative improvement rates for the same data and parameters, it would be more effective to present this information in a single table.

>[1] Zhang, Qingru, et al. "AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning." *arXiv preprint arXiv:2303.10512* (2023).

>[2] Jiang, Ting, et al. "MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning." *arXiv preprint arXiv:2405.12130* (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - I suggest expanding the experimental setup to include standard benchmarks like GLUE and additional SOTA LoRA variants beyond DoRA, for a broader evaluation of ALLoRA’s effectiveness.

- Seems like ALLoRA uses same dropout rate as LoRA, 0.05, and 0 dropout rate is suboptimal even under short fine-tuning paradigm, as shown in Table 2. Any explanation why?

- Could you conduct ablation studies on the same model and dataset configurations used in the primary performance evaluations to ensure consistency across experimental settings?
soundness: 2
strengths: The paper suggest simple yet efficient implementation of LoRA methods by strategically adjusting the existing elements like learning rates, dropout, and scaling factors. They also provide empirical validations on the benefits of their design choice over other LoRA variants such as DoRA.
summary: The paper points out three limitations of existing LoRA methods, unnecessary Dropout for short fine-tuning session, suboptimal zero initialization that results in poor optimization landscape, and uniform scaling factor across layers. Authors resolve these problems by removing Dropout regularization and the scaling factor out of the LoRA design. Additionally, they add an adaptive learning rate strategy that adaptively scales the update inversely proportional to the weight norm. With these modifications, ALLoRA improves performance over LoRA and other recent variants.
weaknesses: 1. ALLoRA relies on a heuristic approach that seems more like a refined hyper-parameter adjustments rather than a fundamental improvement in LoRA architecture. The core changes such as adapting learning rates based on parameter norms, removing Dropout, and eliminating the scaling factor can be viewed as incremental modifications to existing techniques, not a novel structural design. 


2. Limited experiments in terms of datasets, baseline models, comparison with other recent LoRA variants. 
- Their experimental setup doesn't align with the standard setting. No GLUE benchmark experiments and the downstream task is limited to the commonsense reasoning task on LLaMA variants.
- Missing comparison to the various SOTA LoRA variants, only DoRA is used to compare the performance of ALLoRA on commonsense reasoning datasets. 


3. Ablation study done in non-ideal settings.
- Ablation study not done on the same settings used for performance report, only done on midsized LLMs Qwen2-0.5B, Snowflake-Artic-L, and OpenELM-450M.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - How does the computational cost of ALLoRA compare to standard LoRA?
- Are there any scenarios where the adaptive learning rate approach might be disadvantageous?
- How sensitive is the method to the choice of η² hyperparameter?
- Could the approach be combined with other recent LoRA variants for additional benefits?
- Why was the GLUE benchmark not included in the evaluation?
- How does ALLoRA compare to LoRA+ in terms of performance?
- What criteria were used to determine the optimal number of epochs for different datasets? For instance, RTE requires 50 epochs for good performance (according to literature), would your framework still apply?
- Can you provide statistical significance analysis of the performance improvements in terms of standard deviations?
soundness: 2
strengths: - Identifies potential limitations in the current LoRA approach.
- Novel adaptive learning rate solution that addresses these issues.
- Reduction in hyperparameters while maintaining or improving performance.
- Clear ablation studies demonstrating the effectiveness of different components.
- Comprehensive empirical validation across different models and tasks.
summary: The paper identifies three key limitations of Low-Rank Adaptation (LoRA) in the context of fine-tuning large language models: the ineffectiveness of dropout for short training epochs, poor optimization landscape due to zero initialization, and problematic interactions due to the scaling factor. The authors propose ALLoRA, which addresses these issues through an adaptive learning rate approach that scales gradients inversely proportional to L2 norm. While the paper presents interesting ideas, there are significant concerns about the depth of analysis and justification of claims.
weaknesses: - Their analysis relies on the fact that we only fine-tune for a small number of epochs. This is highly subjective as the number of epochs required for a good fine tuning depends on the model and the dataset.
- Limited theoretical analysis of the adaptive learning rate’s convergence properties.
- The ripple effect argument lacks mathematical rigor, only establishing upper bounds without lower bounds or empirical validation.
- Use of simplified models may not accurately represent LLM fine-tuning dynamics.
- Absence of standard deviations in results makes it difficult to assess statistical significance.
- Lack of comparison with well-established benchmarks (e.g., GLUE).
- No comparison with similar approaches like LoRA+.
- Arbitrary choice of epoch numbers without clear justification.
- No discussion of computational overhead compared to standard LoRA.
- Unclear visualization of dropout’s impact in initial epochs.
- Limited justification for why/how ALLoRA ensures faster or more reliable convergence.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please see the above comments for questions.
soundness: 2
strengths: 1. The investigated topic in this paper is quite interesting and critical as fine-tuning large models have now been very popular in many domain applications.
2. The proposed method is technically precise and straightforward, which may ease the real deployments.
3. This work is well motivated and the technical issues are well studied.
4. The proposed method is validated through extensive results.
summary: This work presents a novel low-rank adaptation algorithm that tries to resolve the issues of dropout, poor optimization landscape, and scaling factor. Specifically, the authors propose a dropout-free, scaling-free LoRA with adaptive learning rate, termed ALLoRA. By scaling the per sample and per parameter gradients with a coefficient inversely proportional to parameters' $l_2$ norm, ALLoRA alleviates the above issues. Also, ALLoRA can remove two hyperparameters from LoRA, the scaling factor and the dropout rate. To validate the proposed approach, the authors utilize different models and datasets to show the competitive performance of ALLoRA with the comparison with LoRA and DoRA.
weaknesses: 1. The novelties in this work are quite marginal. Although the authors have proposed the novel LoRA variant, it lacks of in-depth theoretical analysis on ALLoRA. The authors should try to have more comprehensive technical analysis why ALLoRA allows for improvement compared to vanilla LoRA and CoRA.

2. In Figure 1, the authors use two datasets MNIST and CIFAR10 to demonstrate the flaw of dropout. They plot the distribution of standard deviation of gradients. However, during the training, they consider a single mini-batch, instead of the regular one pass over the whole dataset in practice. This may cause the worse variance of the gradient during the training. That way, the claim that this is attributed to Dropout does not make much sense in this context. Also, in the caption, they have the conclusion "while the average variance of the gradient decreases slightly during training, the worst case increases, hence leading to unstable training in finetuning regimes". But this conclusion is not well supported by the two plots.

3. In Figure 2, I am confused how the authors use these two plot to demonstrate the dropout can hurt the fine-tuning of large models. Particularly, what is the purpose to show the right plot? Shouldn't it be a direct comparison between LoRA and ALLoRA in terms of accuracy, indicating that without dropout, ALLoRA can perform better? Also, the authors claim that 10 epochs is already a large number of finetuning iterations for practical scenarios. How to justify this? Is there any a standard to tell what number of iterations is large in finetuning?

4. In Section 3.2, the authors would like to show the second flaw of finetuning LLMs when the zero-initialization is used, which can cause poor optimization landscape. But how can we see this issue from this section? The authors claim that as the training set size decreases, the optimization landscape for $V$ will become degenerate. I am confused about this. Just from the Eqs. (1) and (2), how did the authors make this claim? They should at least show an example for the poor optimization landscape.

5. In Section 3.3, from Line 275 - 280, the derivation here is confusing. I understand that the authors would like have $(1+\eta)^L$ in the final term. However, when they define $C$ , how can the second inequality follow from? I believe $C$ is a consecutive product among all matrix norms of $W^*_1$,...,$W^*_L$. Also, can the authors tell in the paper why they need $\bar{m}$ in the paper? Please derive step by step clearly in the paper. 

6. Notations are confusing. From the beginning, the authors didn't really define $W^*$. Then in line 306, they say $l$ is a learning rate, but in previous context, $l$ indicates the layer. Also, what does it mean by $f_o=\kappa:x\mapsto\eta x$? Please check thoroughly all notations in the paper to make sure they are clearly and properly defined. 

7. In line 334, the authors mention that "We think a function that is inversely proportional to $||(BA)_{i,:}||$ is a good candidate to realize our idea". Why? You need justification here, not just saying that.

8. The Adaptive Learning in this work seems to have scaling factor in the functional, instead of directly applying to model parameters. However, to me, their effects are similar. That way, what is the exact difference and novelty in this study ALLoRA brings, compared to LoRA. Also, I didn't see how the second issue, poor optimization landscape has been addressed through ALLoRA. 

9. The experimental results are not that promising. Though the authors present many results, from Section 4.4, the average improvement over all cases is 0.3%, which is even within the standard deviation, in my opinion. The same to Section 4.5. With such marginal performance improvement, what values does ALLoRA bring about?

10. Overall, though the topic in this work looks interesting, it requires a substantial amount of efforts from the authors to make it technically solid and sound, including more in-depth theoretical analysis, clarification on technical discussion, and more convincing experimental results.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - For lines 193-197, the authors need to provide more elaboration on the derivation. Specifically, it is unclear how ( p ) must be equal to ( 1/(1+\lambda) ). Additionally, there is no explanation to ensure that the final solution of ( W ) should be equivalent for the cases of dropout and Tikhonov regularization.
- What is the difference between ALLoRA and adaptive gradient methods?
soundness: 2
strengths: 1. This paper is generally well-motivated, where the identified three flaws of standard LoRA may be of independent interest for readers.
2. This paper provide thorough theoretical explanations, trying to support the three core limitations of standard LoRA. These theoretical justifications may also be of independent interest for readers.
summary: Motivated by the identification of three core limitations in the standard Low-Rank Adaptation (LoRA) method, this paper introduces ALLoRA, a variant designed to address and overcome these flaws. The authors first conduct theoretical justifications, utilizing toy examples to elucidate the issues inherent in LoRA, particularly within the context of fine-tuning with limited training steps. Building upon these analyses, the paper proposes a solution in the form of LoRA with Adaptive Learning Rate (ALLoRA). Empirical experiments are then conducted to validate the effectiveness of ALLoRA, demonstrating its performance compared to both the standard LoRA method and other recent variants.
weaknesses: 1. In general, the theoretical justifications in this paper do not fully support the three flaws of standard LoRA as mentioned. Consequently, the motivation of this paper may not be convincing, at least to me.
    - For the first flaw, the connection between expectation and training steps may be incorrect. The expectation is over the randomness of V, whereas infinite training steps imply that W approaches the closed-form solution W*. To demonstrate that dropout is not suitable for fine-tuning with limited training steps, the authors need to prove that infinite training steps can lead to the expectation of the results. More importantly, how is "limited training steps" defined? Dropout with large rates outperforms that with small rates even for 8-epoch training, which is also considered limited compared to pre-training.
    - For the second flaw, it is unclear how the degenerate optimization landscape is related to the zero-initialization of LoRA. More importantly, why does the landscape become degenerate as the training set size (n) decreases? As far as I can see, this is related to X^TX, which may remain the same even after n decreases.
    - For the third flaw, it is unclear how the output's potential exponential growth with respect to the number of layers is connected to the performance of LoRA. If this connection is not made clear, we cannot conclude that it is harmful for the final fine-tuning performance.

2. Supposing the "flaws" are true, the paper fails to explain why ALLoRA is a good or necessary way to overcome them. More specifically, one could also directly remove dropout, the scaling factor, and so on to address these issues.

3. This paper primarily compares with DoRA and lacks comparisons with other more recent LoRA techniques, such as those in (Hayou et al., 2024a, 2024b), rsLoRA, Flexora, and others. Importantly, the improvement is quite marginal.

4. This paper lacks an ablation study to show how ALLoRA can help overcome each flaw independently. This is crucial for understanding the efficacy of ALLoRA.

5. The citation format needs to be refined. There are many places where the authors should use \citep to cite a paper, but they have applied \citet to cite the authors.

Meta-review: This work first analyzed the limitations of the vanilla LoRA. In total, the vanilla LoRA has three weak points, some of which were already known, e.g., the scaling effect of LoRA. In addition, reviewers pointed out (potentially) confusing parts in their proofs and analyses. However, the authors did not answer appropriately. I consider that their proofs and analyses are not correct after seeing the silence of the authors during the rebuttal phase. As reviewers said, this work lacks theoretical analyses and comparison with existing work. There have been recently proposed many LoRA enhancements and the authors need to consider them seriously before resubmission.

Decision: Reject