LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

## Abstract

Abstract Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.
Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.
In this paper, we first uncover a fundamental connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization is mathematically equivalent to full fine-tuning using a low-rank gradient for parameter updates.
And this low-rank gradient can be expressed in terms of the gradients of the two low-rank matrices in LoRA.
Leveraging this insight, we introduce LoRA-Pro, a method that enhances LoRA’s performance by strategically adjusting the gradients of these low-rank matrices.
This adjustment allows the low-rank gradient to more accurately approximate the full fine-tuning gradient, thereby narrowing the performance gap between LoRA and full fine-tuning.
Furthermore, we theoretically derive the optimal solutions for adjusting the gradients of the low-rank matrices, applying them during fine-tuning in LoRA-Pro.
We conduct extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification tasks, demonstrating that LoRA-Pro substantially improves LoRA’s performance, effectively narrowing the gap with full fine-tuning.
Code is publicly available at https://github.com/mrflogs/LoRA-Pro .

## 1 Introduction

Foundational models (Radford et al., ; Brown et al., ; Achiam et al., ; Kirillov et al., ; Rombach et al., ; Touvron et al., ) have become the cornerstone of modern deep learning.
By undergoing pre-training on massive datasets, these models typically exhibit excellent generalization and versatility.
Remarkably, some foundation models even demonstrate emergent properties (Hoffmann et al., ; Kaplan et al., ).
Due to these advantages, foundational models have been widely applied to various downstream applications.

Nevertheless, it still requires additional fine-tuning when applied to downstream tasks, where the huge parameter size of foundation models result in high cost in this stage.
To address this issue, recent research has focused on parameter-efficient fine-tuning (PEFT) methods (Hu et al., ; Houlsby et al., ; Lester et al., ).
PEFT methods reduce the fine-tuning cost by keeping the foundation models frozen and only fine-tuning small, additional lightweight adapters.
With the majority of parameters frozen, PEFT enables faster fine-tuning and requires fewer resources.

Low-rank adaptation (Hu et al., ), also known as LoRA, is one of the most famous PEFT methods, which has been widely adopted across various domains.
Inspired by previous works (Aghajanyan et al., ; Li et al., ), LoRA hypothesizes that the changes in weights during model adaptation exhibit a low-rank structure.
To capture this, LoRA re-parameterizes these changes by expressing them as the product of two low-rank matrices: $W=W_{0}+\Delta W\approx W_{0}+sBA$, where $s$ is a scaling factor, and $A\in\mathbb{R}^{r\times n}$ and $B\in\mathbb{R}^{m\times r}$ are low-rank matrices with rank $r\ll\min(m,n)$.
LoRA reduces the number of trainable parameters from $m\times n$ to $r\times(m+n)$, thereby decreasing the cost of fine-tuning.
However, despite its efficiency, LoRA’s fine-tuning performance often falls short compared to full fine-tuning (Hu et al., ; Liu et al., ; Ding et al., ).

In this paper, we propose a novel PEFT method, LoRA-Pro, aimed at bridging the gap between LoRA and full fine-tuning.
To begin with, we uncover a crucial connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization is equivalent to full fine-tuning using a low-rank gradient for parameter updates.
In LoRA, we discover that the change in weight $W$ is connected to the changes in matrices $A$ and $B$, expressed as $\mathrm{d}W=\frac{\partial W}{\partial A}\mathrm{d}A+\frac{\partial W}{
\partial B}\mathrm{d}B$.
This relationship implies that updating matrices $A$ and $B$ with gradients $g^{A}$ and $g^{B}$ is equivalent to updating $W$ with a low-rank equivalent gradient $\tilde{g}$ in full fine-tuning, where:

$$ $\tilde{g}=\frac{\partial W}{\partial A}g^{A}+\frac{\partial W}{\partial B}g^{B }=sBg^{A}+sg^{B}A.$ (1) $$

Leveraging this insight, our goal is to bridge LoRA’s gap with full fine-tuning by minimizing the discrepancy between the low-rank equivalent gradient $\tilde{g}$ and the full fine-tuning gradient $g$, by adjusting the gradients of matrices $A$ and $B$, i.e., $\min_{g^{A},g^{B}}\|\tilde{g}-g\|^{2}_{F}$.
Furthermore, we theoretically demonstrate that this optimization problem admits an optimal closed-form solution, as shown in Theorem 2.1.
Notably, the optimal gradients for the low-rank matrices do not explicitly depend on the full fine-tuning gradient.

Our main contributions are summarized as follows:

- •
We first uncover a crucial connection between LoRA and full fine-tuning in the optimization process: optimizing with LoRA is mathematically equivalent to full fine-tuning using a low-rank gradient for updating.
- •
We propose a novel PEFT method called LoRA-Pro.
Our approach minimizes the distance between the true gradient and the low-rank gradient by adjusting the gradients of matrices A and B.
We theoretically prove the optimal gradients and optimize using these gradients.
- •
Extensive experiments across tasks in natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification demonstrate the effectiveness of our method.

## 2 Method

In this section, we begin by revisiting LoRA (Hu et al., ) in Section 2.1.
Following this, we conduct a comparison between LoRA and full fine-tuning, and point out their connection in the optimization process in Section 2.2.
Finally, in Section 2.3, we introduce LoRA-Pro as a solution to bridge the gap between LoRA and full fine-tuning.

### 2.1 Revisiting Low-Rank Adaptation

First of all, let’s dive back into Low-Rank Adaptation (LoRA) (Hu et al., ).
LoRA’s core idea revolves around recognizing the low-rank structure of the change matrix $\Delta W$ in the standard fine-tuning process.
This insight allows LoRA (Hu et al., ) to re-parameterize the change matrix into the product of two low-rank matrices,

$$ $W=W_{0}+\Delta W\approx W_{0}+sBA.$ (2) $$

Here, $W_{0}\in\mathbb{R}^{m\times n}$ represents the pre-trained weight matrix, $B\in\mathbb{R}^{m\times r}$ and $A\in\mathbb{R}^{r\times n}$ are the low-rank matrices, and $s$ is a scaling factor.
For LoRA (Hu et al., ), $s=\frac{\alpha}{r}$, while for rsLoRA (Kalajdzievski, ), $s=\frac{\alpha}{\sqrt{r}}$.
Here, $\alpha$ is the hyper-parameter and $r\ll min(m,n)$ denotes the rank.
Consequently, LoRA significantly reduces the number of fine-tuning parameters from $m\times n$ to $r\times(m+n)$, thereby decreasing the computational cost of fine-tuning.

### 2.2 LoRA v.s. Full Fine-Tuning

Despite its widespread applications across various domains, LoRA’s performance still falls short when compared to full fine-tuning.
In this part, we compare LoRA and full fine-tuning in the optimization process.
Then, we demonstrate that optimizing using LoRA is equivalent to using a low-rank gradient in full fine-tuning for updating the parameters.

Full fine-tuning.
In full fine-tuning, we utilize differential to analyze the relationship between changes in the loss and changes in the weights:

$$ $\mathrm{d}L=\langle\frac{\partial L}{\partial W},\mathrm{d}W\rangle_{F},$ (3) $$

where $\mathrm{d}L$ and $\mathrm{d}W$ denotes the changes of the parameter $W$ and the loss $L$, and $\langle\cdot,\cdot\rangle_{F}$ is the Frobenius inner product.
To minimize the loss function, we typically set $\mathrm{d}W=-\frac{\partial L}{\partial W}\triangleq-g$ (omitting the learning rate for simplicity), which results in $\mathrm{d}L=-\|\frac{\partial L}{\partial W}\|^{2}_{F}\leq 0$.

Low-rank adaptation.
In LoRA optimization, given that $W=W_{0}+sBA$, we compute the differential using the chain rule:

$$ $\displaystyle\mathrm{d}L$ $\displaystyle=\langle\frac{\partial L}{\partial W},\mathrm{d}W\rangle_{F}$ (4) $\displaystyle=\langle\frac{\partial L}{\partial W},\frac{\partial W}{\partial A }^{T}\mathrm{d}A+\frac{\partial W}{\partial B}^{T}\mathrm{d}B\rangle_{F}$ $\displaystyle=\langle\frac{\partial L}{\partial W}\frac{\partial W}{\partial A },\mathrm{d}A\rangle_{F}+\langle\frac{\partial L}{\partial W}\frac{\partial W} {\partial B},\mathrm{d}B\rangle_{F}$ $\displaystyle=\langle\frac{\partial L}{\partial A},\mathrm{d}A\rangle_{F}+ \langle\frac{\partial L}{\partial B},\mathrm{d}B\rangle_{F}.$ $$

Similarly, LoRA sets $\mathrm{d}A=-\frac{\partial L}{\partial A}\triangleq-g^{A}_{lora}$ and $\mathrm{d}B=-\frac{\partial L}{\partial B}\triangleq-g^{B}_{lora}$, and thus $\mathrm{d}L=-\|\frac{\partial L}{\partial A}\|^{2}_{F}-\|\frac{\partial L}{
\partial B}\|^{2}_{F}\leq 0$.
Moreover, employing the chain rule, we derive:

$$ $g^{A}_{lora}=\frac{\partial L}{\partial W}\frac{\partial W}{\partial A}=sB^{T} g,\qquad g^{B}_{lora}=\frac{\partial L}{\partial W}\frac{\partial W}{\partial B }=sgA^{T}.$ (5) $$

Why LoRA performs worse than full fine-tuning.
With Equation (3) and (4), we observe a critical connection between full fine-tuning and LoRA in the optimization process.
In LoRA, changes in matrices $A$ and $B$ are inherently linked to changes in matrix $W$, i.e., $\mathrm{d}W=\frac{\partial W}{\partial A}^{T}\mathrm{d}A+\frac{\partial W}{
\partial B}^{T}\mathrm{d}B$.
This indicates that updating $A$ and $B$ with gradient $g^{A}$ and $g^{B}$ is equivalent to performing full fine-tuning on $W$ via the following update:

$$ $\mathrm{d}W=\frac{\partial W}{\partial A}^{T}\mathrm{d}A+\frac{\partial W}{ \partial B}^{T}\mathrm{d}B=-(sBg^{A}+sg^{B}A).$ (6) $$

Equation (6) reveals that LoRA optimization is equivalent to full fine-tuning using a low-rank gradient $\tilde{g}=sBg^{A}+sg^{B}A$ (which rank is at most to $2r$(^1^11We provide the proof in Appendix B.1)) for optimization.
Consequently, the performance gap between LoRA and full fine-tuning may stem from differences between $\tilde{g}$ and the full gradient $g$.
The low-rank gradient $\tilde{g}$ may lose important information contained in $g$, leading to distinct optimization trajectories and ultimately causing LoRA to converge to a sub-optimal solution.

### 2.3 Low-Rank Adaptation with Equivalent Gradient

###### Definition 1 (Equivalent Gradient) .

In this part, we introduce our LoRA-Pro method, which bridges the performance gap by minimizing the discrepancy between the gradients.
For convenience, we define the concept of equivalent gradient in Definition 1.
Equivalent gradient describes the virtual low-rank gradient of the matrix $W$ in LoRA optimization process, despite $W$ not being a trainable parameter.
To narrow the performance gap, our goal is to carefully adjust $g^{A}$ and $g^{B}$ of matrices $A$ and $B$ to minimize the distance between the equivalent gradient $\tilde{g}$ and the full gradient $g$ in full fine-tuning.
Hence, our objective is:

$$ $\displaystyle\min_{g^{A},g^{B}}\|\tilde{g}-g\|_{F}^{2}$ (7) s.t. $\displaystyle\tilde{g}=sBg^{A}+sg^{B}A,$ $\displaystyle\mathrm{d}L\leq 0.$ $$

Here, $\|\cdot\|_{F}$ denotes the Frobenius norm, and $\mathrm{d}L$ denotes the change in loss when updating with gradients $g^{A}$ and $g^{B}$.
The objective aims to minimize the distance of the gradients while ensuring a decrease in loss using the solutions for $g^{A}$ and $g^{B}$.

Figure: Figure 1: Illustration of LoRA-Pro. LoRA (Hu et al., ) reduces the trainable parameter by re-parameterizing the weight into the product of two low-rank matrices, i.e., $W=W_{0}+sBA$. We have discovered a connection between the optimization processes of full fine-tuning and LoRA. Updating matrices $B$ and $A$ using gradients $g^{B}$ and $g^{A}$ is equivalent to updating weight $W$ using a virtual low-rank gradient $\tilde{g}=sBg^{A}+sg^{B}A$. Therefore, in LoRA-Pro, we aim to adjust gradients $g^{B}$ and $g^{A}$ to minimize the distance between the equivalent gradient $\tilde{g}$ and the full fine-tuning gradient $g$, thereby reducing their performance gap. In Theorem 2.1, we provide the optimal update gradients, and in Appendix C, we present the pseudo-code for the optimization algorithm.
Refer to caption: x1.png

Closed-form solution.
Fortunately, we prove that the optimization problem (7) admits an optimal closed-form solution, as stated in Theorem 2.1.
Additionally, an interesting observation arises from Theorem 2.1: while the full gradient $g$ serves as the ground truth in the objective, it does not necessarily explicit appear in the closed-form solution.
Instead, the closed-form solution for the optimal gradients can be expressed in terms of the gradients of LoRA.
This allows for an efficient gradient adjustment process, where we backpropagate using the standard LoRA and adjust the gradients of matrices $A$ and $B$ based on the closed-form solution presented in Theorem 2.1.
(^2^22We provide detailed algorithms in Appendix C.)

###### Theorem 2.1 .

###### Proof.

Loss minimization.
While Theorem 2.1 offers a closed-form solution to the optimization problem $\min_{g^{A},g^{B}}\|\tilde{g}-g\|^{2}_{F}$, it’s crucial to understand that this solution does not inherently guarantee a decrease in loss when updating the matrices $A$ and $B$.
To address this concern, we introduce Theorem 2.2.
In this theorem, we prove that the change in loss $\mathrm{d}L$ can be expressed as a negative sum of two Frobenius norms, which leads to $\mathrm{d}L<0$.
This result ensures that the optimization process consistently drives towards a lower loss.

Selection of matrix X.
Although the equivalent gradient itself is not directly related to the matrix $X$, the presence of $X$ plays a significant role in the updates of matrices $A$ and $B$.
We select an appropriate $X$ such that $g^{A}$ and $g^{B}$ remain close to $g^{A}_{lora}$ and $g^{B}_{lora}$ respectively.
To achieve this, we minimize their Frobenius norm, as demonstrated in Equation (14).
In practical terms, $B^{T}B$ and $-AA^{T}$ do not share common eigenvalues.
Therefore, according to Theorem 2.3, we can determine a unique optimal $X$ for updating matrices $A$ and $B$.

###### Theorem 2.2 .

###### Proof.

###### Theorem 2.3 .

###### Proof.

###### Definition 1 (Equivalent Gradient) .

###### Theorem 2.1 .

###### Proof.

###### Theorem 2.2 .

###### Proof.

###### Theorem 2.3 .

###### Proof.

## 3 Experimental Results

In this section, we present extensive experiments to evaluate the effectiveness of LoRA-Pro across various tasks and models.
First, we assess natural language understanding capabilities using the GLUE benchmark by fine-tuning the T5-base (Raffel et al., ) model in Section 3.1.
Next, we evaluate its capabilities in dialogue generation, mathematical reasoning, and code generation using the Llama-2-7B model (Touvron et al., ), covered in Section 3.2.
We then examine LoRA-Pro’s effectiveness on image classification tasks using the CLIP-ViT-B/16 (Radford et al., ) model in Section 3.3.
Finally, we conduct an ablation study of LoRA-Pro in Section 3.4.

Training details.
To ensure a fair comparison, we align our experimental setup with that of LoRA-GA (Wang et al., ).
By default, we fine-tune the model using the AdamW optimizer (Loshchilov & Hutter, ) with hyper-parameters $\beta_{1}=0.9$, $\beta_{2}=0.999$, and weight decay set to 0.
We implement a cosine learning rate schedule with a warmup ratio of 0.03.
LoRA is applied to all linear modules, excluding the embedding layer, normalization layer, and classification head.
By default, we set the rank $r=8$ and $\alpha=16$.

For natural language understanding tasks, we fine-tune T5-base (Raffel et al., ) model with learning rate 1e-4.
The sequence length is set to 128, and the training batch size is 32.
For dialogue generation, mathematical reasoning and code generation tasks, we fine-tune the Llama-2-7B (Touvron et al., ) model with a learning rate of 2e-5.
We set the sequence length to 1024 and the macro batch size to 32.
For image classification tasks, we fine-tune the CLIP-ViT-B/16 (Radford et al., ) model with learning rate 1e-4.
The classifier is obtained using prompts such as “a photo of a {class}” and kept frozen during fine-tuning.
And the training batch size is set to 64.

All experiments are conducted on NVIDIA RTX A6000 GPUs.
To obtain a reliable estimate of model performance, we perform three runs with different random seeds and report the average and standard deviation of the results.

### 3.1 Results on Natural Language Understanding Tasks

**Table 1: Results of fine-tuning T5-base using full fine-tuning and various LoRA variants on a subset of the GLUE datasets. The LoRA rank is set to 8 by default. Bold and underline indicate the highest and second-highest scores, respectively.**
| Method | MNLI | SST2 | CoLA | QNLI | MRPC | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Full FT | 86.33±0.00 | 94.75±0.21 | 80.70±0.24 | 93.19±0.22 | 84.56±0.73 | 87.91 |
| LoRA | 85.30±0.04 | 94.04±0.11 | 69.35±0.05 | 92.96±0.09 | 68.38±0.01 | 82.08 |
| PiSSA | 85.75±0.07 | 94.07±0.06 | 74.27±0.39 | 93.15±0.14 | 76.31±0.51 | 84.71 |
| rsLoRA | 85.73±0.10 | 94.19±0.23 | 72.32±1.12 | 93.12±0.09 | 52.86±2.27 | 79.64 |
| LoRA+ | 85.81±0.09 | 93.85±0.24 | 77.53±0.20 | 93.14±0.03 | 74.43±1.39 | 84.95 |
| LoRA-GA | 85.70±0.09 | 94.11±0.18 | 80.57±0.20 | 93.18±0.06 | 85.29±0.24 | 87.77 |
| DoRA | 85.67±0.09 | 94.04±0.53 | 72.04±0.94 | 93.04±0.06 | 68.08±0.51 | 82.57 |
| AdaLoRA | 85.45±0.11 | 93.69±0.20 | 69.16±0.24 | 91.66±0.05 | 68.14±0.28 | 81.62 |
| LoRA-Pro | 86.03±0.19 | 94.19±0.13 | 81.94±0.24 | 93.42±0.05 | 86.60±0.14 | 88.44 |

In this section, we evaluate our LoRA-Pro across various natural language understanding datasets.
To provide a comprehensive comparison, we include several baseline methods: 1) full fine-tuning and the standard LoRA (Hu et al., ). 2) LoRA variants maintaining the original structure, such as rsLoRA (Kalajdzievski, ), LoRA+ (Hayou et al., ), PiSSA (Meng et al., ), and LoRA-GA (Wang et al., ), 3) LoRA variants with modified structures, including DoRA (Liu et al., ) and AdaLoRA (Zhang et al., ).

The results are presented in Table 1.
We fine-tune the T5-base model (Raffel et al., ) with the baseline methods on a subset of GLUE datasets: MNLI, SST2, CoLA, QNLI, and MRPC.
As shown in Table 1, we observe that LoRA-Pro achieves the highest scores on 3 out of 5 datasets and the highest average score across all 5 datasets, and achieves the highest accuracy on average over the 5 datasets.
Specifically, on average over 5 datasets, LoRA-Pro surpasses standard LoRA (Hu et al., ) with a margin of 6.36.
And LoRA-Pro even achieve higher than full fine-tuning.
This superior performance may be attributed to overfitting in full fine-tuning, where optimizing all model parameters can lead to overfitting on the training data, thus reducing the model’s generalization to the test set.
This effect is particularly pronounced on small datasets, such as MRPC, which contains only 3.7k training data.
These results validate the effectiveness of our methods.

**Table 2: Fine-tuning results of Llama-2-7B model. We fine-tune the Llama-2-7B model using full fine-tuning and LoRA variants on subsets of the WizardLM (Xu et al., ), MetaMathQA (Yu et al., ), and CodeFeedback (Zheng et al., ) datasets, respectively. And we assess dialogue generation, mathematical reasoning, and coding abilities on MT-Bench, GSM8K, and HumanEval datasets. Bold and underline indicate the highest and second-highest scores, respectively.**
|  | MT-Bench | GSM8K | HumanEval |
| --- | --- | --- | --- |
| Full FT | 5.30±0.11 | 59.36±0.85 | 35.31±2.13 |
| LoRA | 5.61±0.10 | 42.08±0.04 | 14.76±0.17 |
| PiSSA | 5.30±0.02 | 44.54±0.27 | 16.02±0.78 |
| rsLoRA | 5.25±0.03 | 45.62±0.10 | 16.01±0.79 |
| LoRA+ | 5.71±0.08 | 52.11±0.62 | 18.17±0.52 |
| DoRA | 5.97±0.02 | 53.07±0.75 | 19.75±0.41 |
| AdaLoRA | 5.57±0.05 | 50.72±1.39 | 17.80±0.44 |
| LoRA-GA | 5.95±0.16 | 53.60±0.30 | 19.81±1.46 |
| LoRA-GA (rank=32) | 5.79±0.09 | 55.12±0.30 | 20.18±0.19 |
| LoRA-GA (rank=128) | 6.13±0.07 | 55.07±0.18 | 23.05±0.37 |
| LoRA-Pro | 5.86±0.06 | 54.23±0.79 | 22.76±0.35 |
| LoRA-Pro (rank=32) | 6.01±0.05 | 55.14±1.73 | 28.05±0.00 |
| LoRA-Pro (rank=128) | 5.68±0.14 | 56.48±0.23 | 34.55±2.46 |

### 3.2 Results on Large Language Models

In this section, we evaluate the performance of LoRA-Pro on large language models, focusing on dialogue generation, mathematical reasoning, and code generation capabilities.
Our experimental setup follows the configuration used in LoRA-GA (Wang et al., ).

- •
For the dialogue generation task, we fine-tune the Llama-2-7B (Touvron et al., ) model on a 52k subset of the WizardLM dataset (Xu et al., ) and evaluate it using the MT-Bench dataset (Zheng et al., ).
GPT-4 is used to assess the quality of the model’s responses, and we report the first-turn score as the metric.
- •
For the math task, we fine-tune the Llama-2-7B (Touvron et al., ) model on a 100k sample from the MetaMathQA dataset (Yu et al., ).
The model is then evaluated on the GSM8K test set (Cobbe et al., ), and we report the accuracy as the metric.
- •
For the coding task, we fine-tune the Llama-2-7B (Touvron et al., ) model on a 100k subset of the CodeFeedback dataset (Zheng et al., ) and test it on the HumanEval dataset (Chen et al., ), reporting the PASS@1 metric.

We compare LoRA-Pro with several baselines, including full fine-tuning, LoRA (Hu et al., ), PiSSA (Meng et al., ), rsLoRA (Kalajdzievski, ), LoRA+(Hayou et al., ), DoRA(Liu et al., ), AdaLoRA (Zhang et al., ), and LoRA-GA (Wang et al., ).
By default, we set the rank to 8 and $\alpha=16$.
Following LoRA-GA (Wang et al., ), we initialize the scaling factor as in rsLoRA (Kalajdzievski, ), i.e., $s=\frac{\alpha}{\sqrt{r}}$.

Table 2 presents our experimental results, which demonstrate LoRA-Pro’s superior performance.
With a rank of 8, LoRA-Pro achieves notable improvements over the original LoRA: 0.25 on MT-Bench, 11.98 on GSM8K, and 8.00 on HumanEval.
When compared to the second-best PEFT method, LoRA-GA, LoRA-Pro shows consistent gains: 0.46 on GSM8K and a substantial 2.95 on HumanEval.
These results validate the effectiveness of our LoRA-Pro method.

Interestingly, we observe that full fine-tuning unexpectedly underperforms on MT-Bench.
We attribute this to potential discrepancies between the WizardLM training data distribution and the MT-Bench evaluation set.
The extensive learning capacity of full fine-tuning may lead to overfitting on the training distribution, compromising generalization to MT-Bench.
Since LoRA-Pro aligns more closely with full fine-tuning during optimization, its relatively poor performance on MT-Bench may also be attributed to overfitting.

To further explore the scalability of our method, we increase the rank in LoRA-Pro from 8 to 128.
Our observations reveal a clear trend: as the rank increases, the performance gap between LoRA-Pro and full fine-tuning narrows rapidly.
Notably, LoRA-Pro consistently outperforms LoRA-GA at the same ranks on both GSM8K and HumanEval datasets.
At rank 32, LoRA-Pro surpasses LoRA-GA by 0.02 on GSM8K and 7.87 on HumanEval.
This performance disparity becomes even more pronounced at rank 128, where LoRA-Pro outperforms LoRA-GA by 1.41 on GSM8K and 11.5 on HumanEval.
These results demonstrate the superior scalability and effectiveness of LoRA-Pro across various rank settings.

### 3.3 Results on Image Classification Tasks

**Table 3: Fine-tuning results of CLIP-ViT-B/16 on image classification tasks. We fine-tune CLIP-ViT-B/16 using full fine-tuning and LoRA variants across StanfordCars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets. Bold indicates the highest results, while underline represents the second-highest results.**
| Method | Cars | DTD | EuroSAT | GTSRB | RESISC45 | SUN397 | SVHN | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | 63.75 | 44.39 | 42.22 | 35.22 | 56.46 | 62.56 | 15.53 | 45.73 |
| Full FT | 84.23±0.06 | 77.44±0.19 | 98.09±0.03 | 94.31±0.28 | 93.95±0.0 | 75.35±0.10 | 93.04±0.18 | 88.06 |
| LoRA | 72.81±0.13 | 73.92±0.38 | 96.93±0.07 | 92.40±0.10 | 90.03±0.14 | 70.12±0.18 | 88.02±0.07 | 83.46 |
| rsLoRA | 82.38±0.20 | 78.03±0.76 | 98.06±0.08 | 95.04±0.11 | 93.96±0.18 | 75.38±0.24 | 92.74±0.18 | 87.94 |
| LoRA+ | 72.87±0.18 | 74.07±0.45 | 97.01±0.02 | 92.42±0.18 | 89.96±0.11 | 70.17±0.15 | 88.08±0.05 | 83.51 |
| DoRA | 73.72±0.06 | 73.72±0.33 | 96.95±0.01 | 92.38±0.17 | 90.03±0.08 | 70.20±0.19 | 88.23±0.05 | 83.48 |
| LoRA-Pro | 85.87±0.08 | 78.64±0.25 | 98.46±0.03 | 95.66±0.05 | 94.75±0.21 | 76.42±0.14 | 94.63±0.20 | 89.20 |

In this section, we assess the performance of LoRA-Pro on image classification tasks.
To provide a comprehensive comparison, we compare it with several baselines: zero-shot CLIP (Radford et al., ), full fine-tuning, vanilla LoRA (Hu et al., ), rsLoRA (Kalajdzievski, ), LoRA+ (Hayou et al., ), and DoRA (Liu et al., ).

We fine-tune the CLIP-ViT-B/16 (Radford et al., ) model on various datasets, including StanfordCars (Krause et al., ), DTD (Cimpoi et al., ), EuroSAT (Helber et al., ), GTSRB (Houben et al., ), RESISC45 (Cheng et al., ), SUN397 (Xiao et al., ), and SVHN (Netzer et al., ).
Accuracy is used as the evaluation metric.
During fine-tuning, only the visual backbone of the CLIP-ViT-B/16 model is updated, while the classifier, derived from prompts such as “a photo of a {class}”, remains frozen.

The results are presented on Table 3.
LoRA-Pro achieves the highest accuracy across all seven datasets.
Specifically, on average, LoRA-Pro surpasses zero-shot classification by 43.47, outperforms LoRA (Hu et al., ) by 5.74, and exceeds rsLoRA (Kalajdzievski, ) by 1.26.
These results validate the effectiveness of our LoRA-Pro method.

### 3.4 Ablation Study

**Table 4: Ablation study on the selection of different X in LoRA-Pro.**
| choice of X | MT-Bench | GSM8K | HumanEval |
| --- | --- | --- | --- |
| Zero | 5.76±0.02 | 53.83±1.16 | 22.96±1.96 |
| Sylvester (Thm. 2.3) | 5.86±0.06 | 54.23±0.79 | 22.76±0.35 |
| Symmetry (Eq. (16)) | 5.63±0.12 | 54.46±0.88 | 22.56±1.06 |

Ablation study on the selection of X.
Based on Theorem 2.1, in LoRA-Pro, the matrix $X$ can be chosen arbitrarily.
While its selection does not affect the equivalent gradient, it does influence the updates of matrices $A$ and $B$ in LoRA.
Here, we conduct an ablation study on the choice of $X$.

We compare three possible values for $X$.
1) Zero solution: In this simplest case, we set $X=\mathbf{0}$.
2) Sylvester solution: Here, $X$ is obtained by solving the Sylvester equation, as described in Theorem 2.3.
3) Symmetry solution: This approach aims to balance the contributions of both terms in the equation $\tilde{g}=sg^{B}A+sBg^{A}$, enforcing the condition $g^{B}A=Bg^{A}$.
For the symmetry solution, solving for $X$ yields:

$$ $X=-\frac{1}{2s}B(B^{T}B)^{-1}B^{T}gA(A^{T}A)^{-1}A=-\frac{1}{2s^{2}}B(B^{T}B)^ {-1}B^{T}g^{B}_{lora}(A^{T}A)^{-1}A.$ (16) $$

The comparison of the selection of $X$ is presented in Table 4.
As shown in the table, the results obtained from different choices of $X$ are similar, and, except on MT-Bench, they outperform the other PEFT baselines listed in Table 2.
However, considering the high variance in the results for both the zero solution and the symmetry solution, we ultimately select the sylvester solution as the default choice for the matrix $X$.

Ablation study on the full-rank assumption.
In Theorem 2.1, we assume that the matrices $A\in\mathbb{R}^{r\times n}$ and $B\in\mathbb{R}^{m\times r}$ are full-rank during training.
Our goal here is to verify whether this assumption holds in practice.
We track the rank changes of all $A$ and $B$ matrices during the fine-tuning process of Llama-2-7B on the MetaMathQA (Yu et al., ) dataset.

In Figure 2, we illustrate the rank changes of matrices $A$ and $B$ from the $q$ projection of layer 9 during the training process, with rank set to 8 and 32, resipectively.
We observed that, although $A$ and $B$ do not initially satisfy the full rank assumption (with matrix $B$ initialized as a zero matrix), both matrices achieve full rank after the first update step.
The rank behavior of $A$ and $B$ in other layers also exhibits similar results.

Figure: Figure 2: Visualization of matrix ranks of A and B during training, with ranks set to $8$ and $32$, respectively.

This observation provides practical evidence that the assumption in Theorem 2.1 is reasonable and supports the validity of the proposed solutions.

**Table 5: We compare LoRA, LoRA-Pro, and Full Fine-Tuning in terms of memory cost, training time, and performance on the MT-Bench, GSM8K, and HumanEval datasets. Memory cost is measured using a single A6000 GPU with a batch size of 1. Training time is recorded on the WizardLM dataset using 8 A100 GPUs with DeepSpeed ZeRO-2 stage optimization.**
|  | Memory Cost | Training Time | MT-Bench | GSM8K | HumanEval |
| --- | --- | --- | --- | --- | --- |
| Full FT | $>$ 48 GB | 2h 33min | 5.30±0.11 | 59.36±0.85 | 35.31±2.13 |
| LoRA | 22.26 GB | 1h 22min | 5.61±0.10 | 42.08±0.04 | 14.76±0.17 |
| LoRA-Pro | 23.05 GB | 1h 23min | 5.86±0.06 | 54.23±0.79 | 22.76±0.35 |

Memory footprint and training time.
Here, we evaluate the additional costs associated with using LoRA-Pro compared to LoRA.
We primarily focus on comparing the differences in memory cost and training time between LoRA-Pro, LoRA, and full fine-tuning.
The results of the experiments are presented in Table 5.
We measure memory cost using a single A6000 GPU with a batch size of 1.
Training time is recorded on the WizardLM dataset using 8 A100 GPUs with DeepSpeed (Rasley et al., ) ZeRO-2 stage optimization.

The results are shown in Table 5.
From the table, we observe the following: 1) LoRA-Pro requires approximately 0.8GB more GPU memory compared to LoRA.
This difference likely stems from the need to compute $B^{T}B$, $AA^{T}$, and their inverses during the calculation of the optimal gradients.
2) Surprisingly, the training time for LoRA-Pro is nearly identical to that of LoRA, with only about a 1% increase in additional training time.
We attribute this to the fact that the matrices $A$ and $B$ in LoRA are low-rank.
Consequently, the extra computations required by LoRA-Pro (such as matrix inversion and the calculation of $X$) are performed on small $r\times r$ matrices, resulting in negligible computational overhead.

Considering that LoRA-Pro uses less memory and trains faster than full fine-tuning, while also providing performance improvements over LoRA, we believe that the additional memory and training time costs are acceptable.

## 4 Related Work

Parameter-Efficient Fine-Tuning.
Given the huge size of foundation models, recent research has focused on developing parameter-efficient fine-tuning methods (Hu et al., ; Liu et al., ; Ding et al., ; Houlsby et al., ; Liu et al., ; Lester et al., ).
These methods aim to reduce the cost of fine-tuning by adjusting only a small portion of the model’s parameters.
Generally, these methods fall into two main categories.
The first category is adapter tuning (Houlsby et al., ; Sung et al., ; He et al., ; Zhang et al., ; Bapna & Firat, ; Hu et al., ), which involves inserting small neural network modules, called adapters, into specific layers of the model.
During fine-tuning, we keep the model frozen and only fine-tune the lightweight adapter modules, significantly reducing the memory footprint for fine-tuning.
The second category is prompt tuning (Lester et al., ; Zhou et al., ; Li & Liang, ; Liu et al., ).
Prompt tuning adapts the models to specific tasks by adding specially designed prompts or learnable tokens to the input data, rather than directly modifying the internal parameters of foundation models.
In this paper, we focus on LoRA (Hu et al., ), a prominent method within the realm of adapter tuning.

Low-Rank Adaptation.
Low-rank adaptation, initially referred to as LoRA (Hu et al., ), has evolved into a broad category encompassing parameter-efficient fine-tuning methods based on low-rank approximations (Hu et al., ; Liu et al., ; Hayou et al., ; Kalajdzievski, ; Zhang et al., ; Kopiczko et al., ; Hyeon-Woo et al., ; Zhang & Pilanci, ; Wang et al., ; Zhao et al., ; Wang et al., ).
LoRA (Hu et al., ) assumes that the changes in the weights of pre-trained models exhibit a low-rank structure.
Consequently, it re-parameterizes these changes as the product of low-rank matrices, thereby reducing the cost during fine-tuning.

Several variants of LoRA have been proposed to address different aspects of this approach.
For example, DoRA (Liu et al., ) improves LoRA (Hu et al., ) by incorporating a learnable magnitude vector to re-scale the normalized product of low-rank matrices.
Another variant, rsLoRA (Kalajdzievski, ), introduces a new scaling factor to stabilize training in high-rank scenarios.
LoRA+(Hayou et al., ) improves upon LoRA by applying different learning rates to the two low-rank matrices.
Additionally, Galore (Zhao et al., ) employs SVD to project the gradients and its first and second momentum of full training into a low-rank space, thereby reducing the memory footprint during pre-training and fine-tuning.

## 5 Conclusion

In this paper, we introduce LoRA-Pro, a novel approach designed to bridge the performance gap between LoRA and full fine-tuning.
We have discovered that using LoRA for fine-tuning is equivalent to fine-tuning the original weights with a virtual equivalent low-rank gradient.
Based on this insight, we propose adjusting the gradients of matrices $A$ and $B$ to make the equivalent gradient match the true full fine-tuning gradient, thereby reducing their performance gap.
Fortunately, we theoretically prove that there exists an optimal closed-form solution for updating matrices $A$ and $B$, which are applied during fine-tuning in LoRA-Pro.
To validate the effectiveness of our method, we conduct extensive experiments across various domains, including natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification tasks.
The results demonstrate that LoRA-Pro significantly improves LoRA performance and narrows the performance gap with full fine-tuning.

## Appendix A Notation

In Table 6, we detail the notations utilized in our paper.

**Table 6: Description of notations used in the paper.**
| Notation | Description |
| --- | --- |
| $s$ | scaling factor in LoRA |
| $B\in\mathbb{R}^{m\times r}$, $A\in\mathbb{R}^{r\times n}$ | low rank matrices in LoRA |
| $g=\frac{\partial L}{\partial W}\in\mathbb{R}^{m\times n}$ | gradients of full fine-tuning |
| $g^{A}_{lora}=\frac{\partial L}{\partial A}=sB^{T}g\in\mathbb{R}^{r\times n}$ | gradients of matrix A in LoRA |
| $g^{B}_{lora}=\frac{\partial L}{\partial B}=sgA^{T}\in\mathbb{R}^{m\times r}$ | gradients of matrix B in LoRA |
| $\mathrm{d}L$ | differential of the loss function |
| $\mathrm{d}A$ | differential of the matrix A |
| $\mathrm{d}B$ | differential of the matrix B |
| $\|\cdot\|_{F}$ | Frobenius Norm |
| $\langle\cdot,\cdot\rangle_{F}$ | Frobenius inner product |

## Appendix B Proof of Theoretical Results

### B.1 Proof that the Equivalent Gradient is Low-Rank

###### Lemma .

###### Proof.

Since matrix rank satisfies the property of subadditivity, we have:

$$ $rank(\tilde{g})=rank(sg^{B}A+sBg^{A})\leq rank(g^{B}A)+rank(Bg^{A}).$ (18) $$

Furthermore, for any matrices $A$ and $B$, $rank(AB)\leq\min(rank(A),rank(B))$.
Therefore, we can bound the ranks as follows:

$$ $\displaystyle rank(g^{B}A)$ $\displaystyle\leq\min(rank(g^{B}),rank(A))\leq r$ (19) $\displaystyle rank(Bg^{A})$ $\displaystyle\leq\min(rank(B),rank(g^{A}))\leq r$ (20) $$

Thus, in conclusion, the equivalent gradient has a rank of at most $2r$:

$$ $\displaystyle rank(\tilde{g})$ $\displaystyle\leq rank(g^{B}A)+rank(Bg^{A})$ (21) $\displaystyle\leq\min(rank(g^{B}),rank(A))+\min(rank(B),rank(g^{A}))$ (22) $\displaystyle\leq 2r.$ (23) $$

∎

###### Lemma .

###### Proof.

### B.2 Proof of Theorem 2.1

###### Theorem .

###### Proof.

For simplicity, we denote $L=\|sBg^{A}+sg^{B}A-g\|_{F}^{2}$.
To solve the optimization problem, we need to satisfy the following conditions:

$$ $\displaystyle\frac{\partial L}{\partial g^{A}}$ $\displaystyle=2sB^{T}(sBg^{A}+sg^{B}A-g)=0$ (27) $\displaystyle\frac{\partial L}{\partial g^{B}}$ $\displaystyle=2(sBg^{A}+sg^{B}A-g)sA^{T}=0$ (28) $$

Given that matrices $A$ and $B$ are full-rank, $AA^{T}$ and $B^{T}B$ are invertible.
And from Equation (28), we derive:

$$ $g^{B}=\frac{1}{s}gA^{T}(AA^{T})^{-1}-Bg^{A}A^{T}(AA^{T})^{-1}.$ (29) $$

Substituting this into Equation (27), we obtain the following linear equation:

$$ $g^{A}[I-A^{T}(AA^{T})^{-1}A]=\frac{1}{s}(B^{T}B)^{-1}B^{T}g[I-A^{T}(AA^{T})^{- 1}A].$ (30) $$

Here, we notice that the matrix $P=I-A^{T}(AA^{T})^{-1}A$ is a projection matrix with rank $n-r$.
The solution to the linear equation (30) is:

$$ $g^{A}=\frac{1}{s}(B^{T}B)^{-1}B^{T}g+XA,$ (31) $$

where $X\in\mathbb{R}^{r\times r}$ represents an arbitrary matrix.
We take the solution (31) into Equation (29), we derive:

$$ $g^{B}=\frac{1}{s}[I-B(B^{T}B)^{-1}B^{T}]gA^{T}(AA^{T})^{-1}-BX$ (32) $$

While we have obtained closed-form solutions for $g^{A}$ and $g^{B}$, these solutions explicitly depend on the gradient of the matrix $W$, i.e., $g$, which is undesirable since $g$ is unknown during LoRA optimization.
Fortunately, the solutions can be transformed into the forms of the gradients of standard LoRA, where the gradients are:

$$ $\displaystyle g^{A}_{lora}=sB^{T}g,\quad g^{B}_{lora}=sgA^{T}.$ (33) $$

Therefore, the solutions to the optimization problem can be written as:

$$ $\displaystyle g^{A}$ $\displaystyle=\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{lora}+XA,$ (34) $\displaystyle g^{B}$ $\displaystyle=\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^{B}_{lora}(AA^{T})^{-1}-BX.$ (35) $$

In our method, we perform the standard forward and backward passes of LoRA, then adjust the gradients of A and B using Solutions (34) and (35), and subsequently update them.
∎

###### Theorem .

###### Proof.

### B.3 Proof of Theorem 2.2

###### Theorem .

###### Proof.

In summary, the proof of Theorem 2.2 is divided into two distinct parts.
To begin with, we demonstrate that $\mathrm{d}L$ can be expressed in the following form:

$$ $\mathrm{d}L=-\gamma\{\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{ lora}\rangle_{F}+\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^ {B}_{lora}(AA^{T})^{-1}\rangle_{F}\}.$ (39) $$

In the second part, we prove that this expression for $\mathrm{d}L$ is always less than or equal to zero: $\mathrm{d}L\leq 0$.

Part I.
Therefore, in this part, we first prove Equation (39).
During the optimization process, the differential change in the loss function, $\mathrm{d}L$, can be expressed in terms of the differentials $\mathrm{d}A$ and $\mathrm{d}B$ as follows:

$$ $\mathrm{d}L=\langle\frac{\partial L}{\partial A},\mathrm{d}A\rangle_{F}+ \langle\frac{\partial L}{\partial B},\mathrm{d}B\rangle_{F}.$ (40) $$

From Equation (36) and (37), we can derive that:

$$ $\mathrm{d}A=-\gamma g^{A},\quad\mathrm{d}B=-\gamma g^{B}.$ (41) $$

Given that $\frac{\partial L}{\partial A}=g^{A}_{lora}$ and $\frac{\partial L}{\partial B}=g^{B}_{lora}$, it follows that:

$$ $\displaystyle\mathrm{d}L$ $\displaystyle=-\gamma(\langle g^{A}_{lora},g^{A}\rangle_{F}+\langle g^{B}_{ lora},g^{B}\rangle_{F})$ (42) $\displaystyle=-\gamma(\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{ lora}\rangle_{F}+\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^ {B}_{lora}(AA^{T})^{-1}\rangle_{F}$ $\displaystyle+\langle g^{A}_{lora},XA\rangle_{F}-\langle g^{B}_{lora},BX \rangle_{F}).$ $$

And we have the following equation:

$$ $\displaystyle\langle g^{A}_{lora},XA\rangle_{F}-\langle g^{B}_{lora},BX\rangle _{F}$ (43) $\displaystyle=$ $\displaystyle\langle g^{A}_{lora}A^{T},X\rangle_{F}-\langle B^{T}g^{B}_{lora}, X\rangle_{F}$ $\displaystyle=$ $\displaystyle\langle g^{A}_{lora}A^{T}-B^{T}g^{B}_{lora},X\rangle_{F}$ $\displaystyle=$ $\displaystyle\langle(sB^{T}g)A^{T}-B^{T}(sgA^{T}),X\rangle_{F}$ $\displaystyle=$ $\displaystyle 0.$ $$

Therefore, we have:

$$ $\mathrm{d}L=-\gamma\{\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{ lora}\rangle_{F}+\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^ {B}_{lora}(AA^{T})^{-1}\rangle_{F}\}.$ (44) $$

Part II.
In this part, we aim to prove $\mathrm{d}L\leq 0$.
Given that the learning rate $\gamma>0$, it suffices to show the following inequalities:

$$ $\displaystyle\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{lora} \rangle_{F}\geq 0,$ (45) $\displaystyle\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^{B}_ {lora}(AA^{T})^{-1}\rangle_{F}\geq 0.$ (46) $$

By proving these inequalities, we can establish that $\mathrm{d}L\leq 0$ as derived from Equation (39).

① Proof of $\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{lora}\rangle_{F}\geq 0$.

To begin with, we need to show that $(B^{T}B)^{-1}$ is positive definite.
To establish this, it is sufficient to show that $B^{T}B$ is positive definite, as the inverse of a positive definite matrix is also positive definite.
To achieve this, consider any non-zero vector $x$, and noting that $B$ is full-rank, we have,

$$ $\langle x,B^{T}Bx\rangle=\langle Bx,Bx\rangle=\|Bx\|^{2}>0.$ (47) $$

This shows that $B^{T}B$ is positive definite.
Consequently, $(B^{T}B)^{-1}$ is positive definite as well.
Since $(B^{T}B)^{-1}$ is positive definite, and thus we can apply Cholesky decomposition, and $(B^{T}B)^{-1}=UU^{T}$.
With this, we have,

$$ $\displaystyle\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{lora} \rangle_{F}$ $\displaystyle=\frac{1}{s^{2}}\langle g^{A}_{lora},UU^{T}g^{A}_{lora}\rangle_{F}$ (48) $\displaystyle=\frac{1}{s^{2}}\langle U^{T}g^{A}_{lora},U^{T}g^{A}_{lora} \rangle_{F}$ $\displaystyle=\frac{1}{s^{2}}\|U^{T}g^{A}_{lora}\|_{F}^{2}\geq 0$ $$

② Proof of $\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^{B}_{lora}(AA^{T}
)^{-1}\rangle_{F}\geq 0$.

Similarly, we can prove that matrix $(AA^{T})^{-1}$ is positive-definite.
By employing Cholesky decomposition, we express $(AA^{T})^{-1}=UU^{T}$, where $U$ is a lower-triangle matrix.
Subsequently, we define $P=I-B(B^{T}B)^{-1}B^{T}$.
It can be shown that $P^{2}=P$ and $P$ is symmetry, indicating that $P$ is a projection matrix.
Consequently, the eigenvalues of $P$ are either 0 or 1, which implies that
$P$ is positive semi-definite.
Utilizing the Cholesky decomposition, we derive that $P=VV^{T}$, where $V$ is a lower-triangle matrix.
Finally, we have:

$$ $\displaystyle\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^{B}_ {lora}(AA^{T})^{-1}\rangle_{F}$ $\displaystyle=\frac{1}{s^{2}}\langle g^{B}_{lora},VV^{T}g^{B}_{lora}UU^{T} \rangle_{F}$ (49) $\displaystyle=\frac{1}{s^{2}}\langle V^{T}g^{B}_{lora}U,V^{T}g^{B}_{lora}U \rangle_{F}$ $\displaystyle=\frac{1}{s^{2}}\|V^{T}g^{B}_{lora}U\|_{F}^{2}\geq 0$ $$

In summary, based on the above proofs, we have demonstrated that:

$$ $\mathrm{d}L=-\gamma\{\underbrace{\langle g^{A}_{lora},\frac{1}{s^{2}}(B^{T}B)^ {-1}g^{A}_{lora}\rangle_{F}}_{{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\text{$\geq 0$ as shown in \char 172}}}+\underbrace {\langle g^{B}_{lora},\frac{1}{s^{2}}[I-B(B^{T}B)^{-1}B^{T}]g^{B}_{lora}(AA^{T })^{-1}\rangle_{F}}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{1,0,0}\text{$\geq 0$ as shown in \char 173}}}\}\leq 0$ (50) $$

∎

###### Theorem .

###### Proof.

### B.4 Proof of Theorem 2.3

###### Theorem .

###### Proof.

For simplicity, we denote $L=\|g^{A}-g^{A}_{lora}\|_{F}^{2}+\|g^{B}-g^{B}_{lora}\|_{F}^{2}$.
To solve the optimization problem, we need to satisfy the following conditions:

$$ $\frac{\partial L}{\partial X}=0.$ (53) $$

Since $g^{A}$ and $g^{B}$ are solutions in Theorem 2.1 and $g^{A}_{lora}=sB^{T}g$ and $g^{B}_{lora}=sgA^{T}$, we obtain that:

$$ $\displaystyle 2(g^{A}-g^{A}_{lora})A^{T}$ $\displaystyle-2B^{T}(g^{B}-g^{B}_{lora})=0,$ (54) $\displaystyle\Rightarrow\quad g^{A}A^{T}-B^{T}g^{B}$ $\displaystyle=g^{A}_{lora}A^{T}-B^{T}g^{B}_{lora},$ $\displaystyle\Rightarrow\quad B^{T}BX+XAA^{T}$ $\displaystyle=-\frac{1}{s^{2}}(B^{T}B)^{-1}g^{A}_{lora}A^{T},$ $$

which is a Sylvester equation. This equation has a unique solution for $X$ if and only if $B^{T}B$ and $-AA^{T}$ have no shared eigenvalues.

∎

###### Theorem .

###### Proof.

## Appendix C Optimization Algorithms

In this section, we present the pseudo-codes for implementing our LoRA-Pro method using the SGD (Sutskever et al., ) and AdamW (Loshchilov & Hutter, ) optimizers.
The details are provided in Algorithm 1, Algorithm 2, and Algorithm 3, respectively.

LoRA-Pro with SGD optimizer. In the standard SGD algorithm, as illustrated in Algorithm 1, all we need to do is adjusting the gradients of matrices $A$ and $B$ with the solutions in Theorem 2.1.

Figure: Algorithm 1 LoRA-Pro with SGD optimizer

LoRA-Pro with AdamW optimizer.
In AdamW optimizer, the implementation becomes more complex.
We offer two scalable implementation modes for LoRA-Pro with AdamW: efficient mode and full mode.
We use the efficient mode by default.
In efficient mode, the memory usage of LoRA-Pro’s optimizer remains the same as that of standard LoRA, but this mode tends to be less stable.
In full mode, LoRA-Pro aligns more closely with full fine-tuning during the optimization process, resulting in higher memory usage but improved stability.

In efficient mode, we only need to modify the gradients of matrices $A$ and $B$.
Then, we separately compute the first- and second-order gradient information for $A$ and $B$ before performing the update.
The detailed algorithm is shown in Algorithm 2.
In this mode, memory usage remains consistent with that of LoRA (Hu et al., ), delivering better results, though not as stable as full mode.

In full mode, we aim to closely approximate full fine-tuning during optimization.
Several modifications are necessary.
Firstly, in order to mimic full fine-tuning, after adjusting the gradients of matrices $A$ and $B$, we need to compute the equivalent gradient,

$$ $\tilde{g}=sg^{B}A+sBg^{A}.$ (55) $$

Subsequently, we calculate the first and second moments of this equivalent gradient to derive the corresponding AdamW gradient, $\tilde{g}^{AdamW}$.
Secondly, we determine the gradients with respect to matrices $A$ and $B$ as follows:

$$ $\tilde{g}^{A}=sB^{T}\tilde{g}^{AdamW},\quad\tilde{g}^{B}=s\tilde{g}^{AdamW}A^{ T}.$ (56) $$

Thirdly, the weight decay process must be adjusted. In line with full fine-tuning, the weight decay is given by:

$$ $W\leftarrow(1-\gamma\lambda)(W_{0}+sBA).$ (57) $$

This can be decomposed into:

$$ $W_{0}\leftarrow(1-\gamma\lambda)W_{0},\quad B\leftarrow\sqrt{1-\gamma\lambda}B ,\quad A\leftarrow\sqrt{1-\gamma\lambda}A$ (58) $$

Ablation study on the AdamW optimizer.
We conducted experiments to compare the memory cost differences between the two implementations of the AdamW optimizer, as well as their performance on MT-Bench, GSM8K, and HumanEval.
The results of the study are presented in Table 7.
As shown in the table, the full mode achieves better performance compared to the efficient mode.
We believe this is because the full mode more closely aligns with full fine-tuning during the optimization process.
However, as a trade-off, full mode incurs a higher GPU memory cost due to the calculation of the first- and second-order momentum information for the equivalent gradients $\tilde{g}$.
Despite this, its memory usage is still significantly lower than that of full fine-tuning (which theoretically requires 56GB of memory).
Therefore, we use efficient mode as the default setting for LoRA-Pro.

**Table 7: Comparison of the AdamW optimizer implementations with LoRA-Pro. We present two types of AdamW optimizers for LoRA-Pro: the efficient mode, which maintains the same memory cost as the vanilla LoRA optimizer, and the full mode, which is more aligned with full fine-tuning and offers greater stability during training.**
|  | Memory Cost | MT-Bench | GSM8K | HumanEval |
| --- | --- | --- | --- | --- |
| Full FT | $>$ 48 GB | 5.30±0.11 | 59.36±0.85 | 35.31±2.13 |
| LoRA-Pro (Efficient) | 23.05 GB | 5.86±0.06 | 54.23±0.79 | 22.76±0.35 |
| LoRA-Pro (Full) | 43.87 GB | 6.05±0.25 | 54.06±0.46 | 23.98±0.35 |

Figure: Algorithm 2 LoRA-Pro with AdamW optimizer (Efficient Mode)

Figure: Algorithm 3 LoRA-Pro with AdamW optimizer (Full Mode)

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: •	Please explain why LoRA-Pro is much better than Full FT in Table 3. Is there overfitting phenomenon that gives LoRA-Pro an advantage on these specific tasks?
•	Please explain why the training speed of LoRA-Pro is slightly worse than LoRA and much better than Full FT in Table 5. It will be better to provide a breakdown of the additional operations required by LoRA-Pro compared to standard LoRA, and to quantify the overhead in terms of FLOPs
•	Please provide more choices of X in Table 4.   For example, the analytical solution of X can be calculated in Theorem 2.1. 
•	LoRA-GA[1] has the same purpose as the work in this paper. Please discuss how LoRA-Pro differs conceptually or methodologically from LoRA-GA and supplement the results of LoRA-GA in Table 3 and Table 5.
•	If available, please add discrepancy curves between the low-rank equivalent gradient and the full fine-tuning gradient. Please supplement the training loss curves to illustrate the convergence rate of the proposed method.

[1]Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.
soundness: 3
strengths: In this paper, the optimal solution of adjusting the gradient of low rank matrix is deduced theoretically and applied to the fine tuning process of LORA-PRO, which narrows the performance gap between LoRA and full fine tuning. Extensive experiments in multiple tasks have been carried out to prove that LORA-Pro greatly improves the performance of LoRA.
summary: LoRA reduces the parameters required during training by introducing a low-rank matrix, thereby reducing computational requirements and memory footprint while maintaining model performance. This paper introduces LoRA-Pro to enhance LoRA’s performance by strategically adjusting the gradients of the two low-rank matrices, allowing the low-rank gradient to more accurately approximate the full fine-tuning gradient.
weaknesses: 1. The method is not applied to the latest LLMs.
2. LoRA-Pro brings in extra memory cost and computation cost in the training.
3. It is not clear why the optimal gradients for the low-rank matrices do not explicitly depend on the full fine-tuning gradient.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - It would be interesting to explore how the equivalent gradients differ from the low-rank gradients in LoRA. A quantitative analysis could provide valuable insights into this relationship. For instance, how does the difference between these gradients vary across different models, deeper layers, and various types of weights? Conducting such a study could yield interesting findings, and further strengthen the motivation behind the gradient adjustment.

Minor:

- Please provide metrics for the datasets used in Table 1. I believe not all metrics used are “Accuracy”. For instance, CoLA typically usually uses Matthews Correlation Coefficient - kindly specify for all datasets.
- How exactly is the *efficient* mode less stable than the *full* mode? To clarify, all the results in the main paper are derived using the *efficient* mode, is that correct?
soundness: 3
strengths: - The motivation is clearly articulated, and the paper is well-written with well-explained ideas.

- The proposed method enhances LoRA, a highly relevant technique for fine-tuning, in a straightforward manner, supported by both theoretical proofs and experimental evidence.

- The experiments and ablation studies are thorough, demonstrating the effectiveness of the proposed method across diverse settings.
summary: The authors show that demonstrating that optimizing with LoRA is mathematically equivalent to performing full fine-tuning using a low-rank gradient for updates. They introduce a novel parameter-efficient fine-tuning (PEFT) method named LoRA-Pro, which minimizes the divergence between the true gradient and the low-rank gradient by adjusting the gradients of matrices A and B. The authors provide theoretical proof of the optimal gradients and detail the optimization process utilizing these gradients. They conduct extensive experiments across various tasks, including natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification, showcasing the effectiveness of their proposed method.
weaknesses: - The authors conducted all experiments for the different methods (for a given task) using the same learning rates. However, this approach may not provide a fair representation of optimal conditions, as the selected learning rate could work well for one method but not for others. To strengthen their findings, it would be beneficial for the authors to perform a learning rate sweep for each method. If this is too resource-intensive for all datasets, they might consider limiting the sweep to a single task, which would further increase confidence in the effectiveness of their proposed method.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: - Do LoRA-Pro have some failure cases? (e.g., potential limitations or edge cases where LoRA-Pro might not perform as well)
soundness: 4
strengths: - The paper presents a novel approach by mathematically linking LoRA with full fine-tuning through gradient adjustments.
- The theoretical foundation is robust, with clear derivations and optimal solutions provided for gradient adjustments. The experiments are comprehensive, covering multiple domains such as language understanding, dialogue generation, mathematical reasoning, and image classification.
- The paper is well-organized.
summary: The paper introduces LoRA-Pro, a novel method aimed at improving the performance of LoRA in fine-tuning foundation models. LoRA-Pro enhances LoRA by adjusting the gradients of low-rank matrices to better approximate the full fine-tuning gradient. The paper establishes a mathematical equivalence between LoRA and full fine-tuning, and uses this insight to optimize LoRA's performance. Extensive experiments across various tasks demonstrate that LoRA-Pro substantially narrows the performance gap between LoRA and full fine-tuning.
weaknesses: - It will be better to test LoRA-Pro in real large models (e.g., 70B).

Meta-review: This paper proposes an improved training method for LoRA. The main insight is that the current way of optimizing LoRA is a low-rank approximation of the full fine-tuning gradient. Using this insight, the authors derive the theoretically optimal gradient to approximate the full fine-tuning gradient. Then, they evaluated this method on various NLP and CV datasets and showed improved performance.

Main strengths
- Simple observations that lead to improvements
- Solid experimental evaluation 

Main weaknesses 
- Limited novelty (e.g., Lora-GA)
- Didn’t do hyperparameter tuning (addressed during rebuttal)
- Limited ablation studies

While this paper has a high score, more thorough studies could significantly improve the paper. Thus, I recommend acceptance as a poster.

Decision: Accept (Spotlight)