LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
Authors: Zhengbo Wang
Jian Liang
Ran He
Zilei Wang
Tieniu Tan
Venue: ICLR 2025 Spotlight (2025)
Keywords: Parameter Efficient Fine-Tuning
Large Language Models
Low-Rank Adaptation
Primary area: foundation or frontier models, including LLMs

Abstract: Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.
Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.
In this paper, we first uncover a fundamental connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization is mathematically equivalent to full fine-tuning using a low-rank gradient for parameter updates.
And this low-rank gradient can be expressed in terms of the gradients of the two low-rank matrices in LoRA.
Leveraging this insight, we introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of these low-rank matrices.
This adjustment allows the low-rank gradient to more accurately approximate the full fine-tuning gradient, thereby narrowing the performance gap between LoRA and full fine-tuning.
Furthermore, we theoretically derive the optimal solutions for adjusting the gradients of the low-rank matrices, applying them during fine-tuning in LoRA-Pro.
We conduct extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification tasks, demonstrating that LoRA-Pro substantially improves LoRA's performance, effectively narrowing the gap with full fine-tuning.
Our code is publicly available at https://github.com/mrflogs/LoRA-Pro.

Published as a conference paper at ICLR 2025
PORA-PRO: ARE LOW-RANK ADAPTERS PROPERLY OPTIMIZED?
Zhengbo Wang 1.2 Jian Liang Ran He Zilei Wang'
Tieniu Tan
University of Science and Technology of China
2 NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences
3 School of Artificial Intelligence, University of Chinese Academy of Sciences Nanjing University shengbowangg@maii 1, .uwc..eu..u............... cn, liangjiant 92 gma 1 com
ABSTRACT
Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-eeffciiet fine-tuning of foundation models. Despite its computa- tional efficiency, LoRA still yields inferior performance compared to full fine- tuning. In this paper, first uncover a fundamental connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimiza- tion is mathematically equivalent to full fine-tuning using low-rank gradient for parameter updates. And this low-rank gradient can be expressed in terms of the gradients of the two low-rank matrices in LoRA. Leveraging this insight, in- troduce LoRA-Pro, method that enhances LoRA's performance by strategically adjusting the gradients of these low-rank matrices. This adjustment allows the low-rank gradient to more accurately approximate the full fine-tuning gradient, thereby narrowing the performance gap between LoRA and full fine-tuning. Fur- thermore, we theoretically derive the optimal solutions for adjusting the gradients of the low-rank matrices, applying them during fine-tuning in LoRA-Pro. We con- duct extensive experiments across natural language understanding, dialogue gen- eration, mathematical reasoning, code generation, and image classification tasks, demonstrating that LoRA-Pro substantially improves LoRA's performance, effec- tively narrowing the gap with full fine-tuning. Our code is publicly available at https: / wtt.ttt.t.uugggg com / /r logs/ LoRA-Pro
INTRODUCTION
Foundational models (Radford et al., 2021; Brown et al., 2020; Achiam et al., 2023; Kirillov al., 2023; Rombach al., 2022; Touvron et al., 2023) have become the cornerstone of modern deep learning. By undergoing pre-training on massive datasets, these models typically exhibit excellent generalization and versatility. Remarkably, some foundation models even demonstrate emergent properties (Hoffmann et al., 2022; Kaplan et al., 2020). Due to these advantages, foundational models have been widely applied to various downstream applications.
Nevertheless, it still requires additional tin-ttunig when applied to downstream tasks, where the huge parameter size of foundation models result in high cost in this stage. To address this issue, recent research has focused on parameter-efficient fine-tuning (PEFT) methods (Hu et al., 2022; Houlsby al., 2019; Lester al., 2021). PEFT methods reduce the fine-tuning cost by keeping the foundation models frozen and only fine-tuning small, additional lightweight adapters. With the majority of parameters frozen, PEFT enables faster fine-tuning and requires fewer resources.
Low-rank adaptation (Hu et al., 2022). also known as LoRA, is one of the most famous PEFT meth- ods, which has been widely adopted across various domains. Inspired by previous works (Agha- janyan et al., 2021; Li et al., 2018), LoRA hypothesizes that the changes in weights during model adaptation exhibit low-rank structure. To capture this, LoRA re-parameterizes these changes by expressing them as the product of two low-rank matrices: W = Wo + AW ~ Wo + s sBA, where
"Corresponding author.

---

Published as a conference paper at ICLR 2025
scaling factor, and A R**" and B R are low-rank matrices with rank < min,m, n). LoRA reduces the number of trainable parameters from m x n to r X (m + n), thereby decreasing the cost of fine-tuning. However, despite its efficiency. LoRA's fine-tuning performance often falls short compared to full fine-tuning (Hu et al., 2022; Liu et al., 2024; Ding et al., 2023).
In this paper, we propose novel PEFT method, LoRA-Pro, aimed bridging the gap between LoRA and full fine-tuning. To begin with, we uncover a crucial connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization equivalent full fine-tuning using a low-rank gradient for parameter updates. In LoRA, we discover that the change in weight W is connected the changes in matrices A and B. expressed as dW 8A relationship implies that updating matrices and with gradients g and g is equivalent to updating W with a low-rank equivalent gradient ÿ in full fine-tuning, where:
OW @W dA + 0B dB. This
sBg + sg aw aW g
g 0A
+ AB
(1)
OB
Leveraging this insight, our goal is to bridge LoRA's gap with full fine-tuning by minimizing the discrepancy between the low-rank equivalent gradient 9 and the full fine-tuning gradient g. by ad- justing the gradients of matrices A and B, i.e., m llğ gll.. Furthermore, we theoretically demonstrate that this optimization problem admits an optimal closed-form solution, as shown in Theorem 2.1. Notably, the optimal gradients for the low-rank matrices do not explicitly de- pend on the full fine-tuning gradient.
Our main contributions summarized as follows:
We first uncover a crucial connection between LoRA and full fine-tuning in optimization process:
optimizing with LoRA is mathematically equivalent to full fine-tuning using low-rank gradient for updating.
We propose novel PEFT method called LoRA-Pro. Our approach minimizes the distance between the true gradient and the low-rank gradient by adjusting the gradients of matrices A and B. We theoretically prove the optimal gradients and optimize using these gradients.
Extensive experiments across tasks in natural language understanding, dialogue genera- tion, mathematical reasoning, code generation, and image classification demonstrate the effectiveness of our method.
2 METHOD
In this section, we begin by revisiting LoRA (Hu et al., 2022) in Section 2.1. Following this, we conduct comparison between LoRA and full fine-tuning, and point out their connection in the optimization process in Section 2.2. Finally, in Section 2.3, we introduce LoRA-Pro as a solution to bridge the gap between LoRA and full fine-tuning.
2.1 REVISITING LOw-RANK ADAPTATION
First of all, let's dive back into Low-Rank Adaptation (LoRA) (Hu et al., 2022). LoRA's core idea revolves around recognizing the low-rank structure of the change matrix AW in the standard fine- tuning process. This insight allows LoRA (Hu et al., 2022) to re-parameterize the change matrix into the product of low-rank matrices.
W Wo AW Wo sBA.
Here, Wo mR represents the pre-trained weight matrix, B mxx and A E Rexn the low-rank matrices, and is scaling factor. For LoRA (Hu et al., 2022), = 2, while for rsLoRA (Kalajdzievski, 2023),
Vr Here, is the hyper-parameter and
min(m, n)
denotes the rank. Consequently, LoRA significantly reduces the number of fine-tuning parameters from m to r X (m + n), thereby decreasing the computational cost of fine-tuning.
2.2 LORA V.S. FULL FINE-TUNING
Despite its widespread applications across various domains, LoRA's performance still falls short when compared to full fine-tuning. In this part, compare LoRA and full fine-tuning in the op-
2

---

Published as a conference paper at ICLR 2025
timization process. Then, we demonstrate that optimizing using LoRA is equivalent to using low-rank gradient in full fine-tuning for updating the parameters.
Full fine-tuning, In full fine-tuning, utilize differential to analyze the relationship between changes in the loss and changes in the weights:
aL dL = aw d.),.
(3)
where dL and dW denotes the changes of the parameter W and the loss L, and
(. ) is the Frobe-
nius inner product. To minimize the loss function, we typically set dW OL A (omitting ow
the learning rate for simplicity), which results in dl -Il at
aw || S O.
Low-rank adaptation. In LoRA optimization, given that W Wo + sBA, we compute the differential using the chain rule:
aL dW), dL = ow
àL OW T
dA +
0B dB)
OW 0A
(4)
θL OW
ØL oW
ddA)
.BB) r
OW 0A
OW OB
0L
0L
dA) F
dB)
0A
0B"
Similarly, LoRA sets dA = 8L - nara and dB
8L
gara and thus dl = -1 0L 1|
84
| 1 < O. Morcover, employing the chain rule, derive:
0L aw
ÖL OW
Moora aw 0A
sB g,
B Htora
sgA".
oW 0B
Why LoRA performs worse than full fine-tuning. With Equation (3) and (4), observe critical connection between full fine-tuning and LoRA in the optimization process. In LoRA, changes in aw 7
matrices and are inherently linked changes in matrix W, i.c. dW ew dA + az dB.
aλ
This indicates that updating A and with gradient g and g is equivalent to performing full fine-tuning on W via the following update:
Ow
OW OB dB === A.
dw
dA
(6)
0A
Equation (6) reveals that LoRA optimization equivalent to full fine-tuning using a low-rank gra- dient ğ = BB Sy A (which rank is at most to 2r') for optimization. Consequently, the per- formance gap between LoRA and full fine-tuning may stem from differences between ğ and the full gradient g. The low-rank gradient may lose important information contained in g. leading to distinct optimization trajectories and ultimately causing LoRA to converge to a sub-optimal solution.
2.3 LOW-RANK ADAPTATION WITH EQUIVALENT GRADIENT
Definition (Equivalent Gradient). In the context of LoRA oplimization, we define the equiv- alent gradient as,
g A sBg + sg A, where is the scaling factor, and A and g gradients with respect to A and B, respec- tively.
In this part, we introduce our LoRA-Pro method, which bridges the performance gap by minimizing the discrepancy between the gradients. For convenience, we define the concept equivalent gradi- ent in Definition 1. Equivalent gradient describes the virtual low-rank gradient of the matrix W in
We provide the proof in Appendix B.1

---

Published as a conference paper at ICLR 2025
min 11 glf
Re-
X
aw
parameter
W
8
aw
W
9-
+sg
sB g8g
9
88
A
0
Gradients
Frozen
Tunable
Low-rank Subspace
Figure I: Illustration of LoRA-Pro. LoRA (Hu et al., 2022) reduces the trainable parameter by re- parameterizing the weight into the product of two low-rank matrices, i.e., W Wo + sAA. We have discovered a connection between the optimization processes full fine-tuning and LoRA. Updating matrices B and A using gradients gd and g is equivalent updating weight W using virtual low-rank gradient ÿ = BBg + sg A. A. Therefore, in LoRA-Pro, we aim to adjust gradients g and A minimize the distance between the equivalent gradient and the full fine-tuning gradient g. thereby reducing their performance gap. In Theorem 2.1, we provide the optimal update gradients, and in Appendix C, present the pseudo-code for the optimization algorithm.
LoRA optimization process, despite W not being trainable parameter. To narrow the performance gap, our goal is to carefully adjust 9 and 8 of matrices A and to minimize the distance between the equivalent gradient and the full gradient y in full fine-tuning. Hence, our objective is:
min 1| g² g
g
= BB + $g A, dL 5 0.
(7)
Here, || || denotes the Frobenius norm, and dL denotes the change in loss when updating with gradients 9dd and g". The objective aims to minimize the distance of the gradients while ensuring decrease loss using the solutions for A and gB.
prove that the optimization problem (7) admits optimal closed-form solution, as stated in Theorem 2.1. Additionally, an interesting observation arises from Theorem 2.1: while the full gradient g serves the ground truth in the objective, it does neces- sarily explicit appear in the closed-form solution. Instead, the closed-form solution for the optimal gradients be expressed in terms of the gradients of LoRA. This allows for an efficient gradient adjustment process, where we backpropagate using the standard LoRA and adjust the gradients of matrices and B based on the closed-fore solution presented in Theorem 2.1.
Closed-form solution. Fortunately,
Theorem 2.1. Assume matrices B E Rmxr A E Rex are both full rank. For the objective mingage g || - glli., the optimal solutions are given by:
( B) XA
(B" TBT B) -)-1111000 Nlora + XAA
I B(BB B 9 (AA")) BX
(9)
$2 | B(B" BJ o
1o (AAT)-1 BX.
(10)
Here, X RRr represents an arbitrary matrix.
Proof. See Appendix B.2.
Loss minimization. While Theorem 2.1 offers closed-form solution the optimization problem
minga g || gl
it's crucial to understand that this solution does not inherently guarantee a
We provide detailed algorithms in Appendix C.
4

---

Published as a conference paper at ICLR 2025
decrease in loss when updating the matrices A and B. To address this concern, introduce The- orem 2.2. In this theorem, we prove that the change in loss dL can be expressed as a negative sum of two Frobenius norms, which leads to dL 0. This result ensures that the optimization process consistently drives towards a lower loss.
Selection of matrix X. Although the equivalent gradient itself is not directly related to the matrix X, the presence of X plays a significant role in the updates of matrices and B. We select an appropriate X such that gA and وو remain close gore and o respectively. To achieve this, minimize their Frobenius norm, demonstrated in Equation (14). In practical terms, B² B and -AA do not share common eigenvalues. Therefore, according to Theorem 2.3, we can determine unique optimal X for updating matrices A and B.
Theorem 2.2. When updating matrices A and B using the closed-form solution from Theo- rem 2.1, we proceed as follows:
A A ~g
(11)
+ B 7g
(12)
where 7, 2 denotes the learning rate. Our method ensures decrease in the loss, akin to the standard gradient descent algorithm, expressed by:
1 dL - -L---(l qe (B=B)- giora) p Moraa T II-BBB"BB B) B" (AA")- "))))
1
}r} s 0.
(13)
Proof. See Appendix B.3.
Theorem 2.3. Consider the optimization problem,
min X
l1g
-iore 1 + 11g ylora
(14)
where - and gd are the optimal solutions as stated in Theorem 2.1. The optimal X can be determined by solving the Sylvester equation:
BTBX + XAA - (BB (BTBB-'o00 Nora AT,
(15)
which has unique solution X provided that BTB and AAT do have any shared eigenvalues.
Proof. See Appendix B.4.
3 EXPERIMENTAL RESULTS
In this section, we present extensive experiments to evaluate the effectiveness of LoRA-Pro across various tasks and models. First, assess natural language understanding capabilities using the GLUE benchmark by fine-tuning the T5-base (Raffel et al., 2020) model in Section 3.1. Next, we evaluate its capabilities dialogue generation, mathematical reasoning, and code generation using the Ilama-2-7B model (Touvron et al., 2023), covered in Section 3.2. We then examine A-RA-ro'' effectiveness on image classification tasks using the C1 (Radford et al., 2021) model in Section 3.3. Finally, we conduct ablation study of LoRA-Pro in Section 3.4.
Training details. To ensure a fair comparison, we align our experimental setup with that of LoRA-GA (Wang et al., 2024a). By default, we fine-tune the model using the AdamW optimizer (Loshchilov & Hutter, 2019) with crrper-parameter. 31 = 0.9, B2 = 0.999, and weight decay set to 0. We implement cosine learning rate schedule with warmup ratio of 0.03. LoRA is applied to all linear modules, excluding the embedding layer, normalization layer, and classification head. By default, set the rank 8 and 16.
5

---

Published as a conference paper at ICLR 2025
For natural language understanding tasks, fine-tune T5-base (Raffel al., 2020) model with learning rate lc-4. The sequence length is set to 128, and the training batch size is 32. For dialogue generation, mathematical reasoning and code generation tasks, fine-tune the Llama-2-77 2-7B (Tou- vron et al., 2023) model with a learning rate of 2e-5. We set the sequence length to 1024 and the macro batch size to 32. For image classification tasks, et al., 2021) model with learning rate lc-4. The classifier is obtained using prompts such as "a photo of {class)" and kept frozen during fine-tuning, And the training batch size is set to 64.
fine-tune the
(Radford
All experiments are conducted on NVIDIA RTX A6000 GPUs. To obtain a reliable estimate of model performance, we perform three runs with different random seeds and report the average and standard deviation of the results.
3.1 RESULTS ON NATURAL LANGUAGE UNDERSTANDING TASKS
Table Results of fine-tuning T5-base using full fine-tuning and various LoRA variants on a subset of the GLUE datasets. The LoRA rank is set to 8 by default. Bold and underline indicate the highest and second recond-highest scores, respectively.
Method
MNLI
SST2
CoLA
QNLI
MRPC
Average
Full FT 86.33+0.00
94.75+0.21
80.70
93.1940.2.
84.56+0.73
87.91
LoRA
85.3010.04
94.04+0.11
69.35=00.5
92.96+0.09
68.38+0.01
82.08
PiSSA
85.75+0.07
94.07+0.06
74.27+0..99
93.15+0.14
76.31+0.51
84.71
LsLoRA
85.73+0.10 10
94. 04.1930.23
72.32+1.12
93.1240.09
52.86+2.27
79.64
LoRA+
85.81+0.09
93.85+0.24
77.53+0.20
93.1440.03
1..4
84.95
LoRA-GA
85.70+0.09
94.11+0.18
80.57+0.20
93.18+0.06
85,29+0.24
87.77
DoRA
85.67+0.09
94.0420.53
72.04
93.0440.06
68. 08.08+0.51
82.57
AdaLoRA
85,45+0.11
93.69#0.20
69. 19.224
91.66+0.05
68. 14.008
81.62
LoRA-Pro
86.03+0.19
94.19+0.13
81.9420.24
93.42+0.05
86.60#0.14
88.44
In this section, evaluate our LoRA-Pro across various natural language understanding datasets. To provide a comprehensive comparison, we include several baseline methods: 1) full fine-tuning and the standard LoRA (Hu al., 2022). 2) LoRA variants maintaining the original structure, such as rsLoRA (Kalajdzievski, 2023). LoRA+ (Hayou et al., 2024), PiSSA (Meng et al., 2024), and LoRA-GA (Wang et al., 2024a), 3) LoRA variants with modified structures, including DoRA (Liu et al., 2024) and AdaLoRA (Zhang et al., 2023).
The results are presented in Table 1. We fine-tune the T5-base model (Raffel al., 2020) with the baseline methods on a subset of GLUE datasets: MNLI, SST2, CoLA, QNLI, and MRPC. As shown in Table 1, we observe that LoRA-Pro achieves the highest scores 3 out of 5 datasets and the highest average score across all datasets. and achieves the highest accuracy on average over the 5 datasets. Specifically, on average over datasets, LoRA-Pro surpasses standard LoRA (Hu et al., 2022) with margin of 6.36. And LoRA-Pro even achieve higher than full fine-tuning. This superior performance may be attributed to overfitting in full fine-tuning, where optimizing all model parameters can lead to overfitting on the training data, thus reducing the model's generalization to the test set. This effect is particularly pronounced on small datasets, such as MRPC, which contains only 3.7k training data. These results validate the effectiveness of our methods.
3.2 RESULTS ON LARGE LANGUAGE MODELS
In this section, we evaluate the performance of LoRA-Pro on large language models, focusing on dialogue generation, mathematical reasoning, and code generation capabilities. Our experimental setup follows the configuration used in LoRA-GA (Wang et al., 2024a).
For the dialogue generation task, we fine-tune the Alama-2--B (Touvron et al., 2023) model 52k subset of the WizardLM dataset (Xu et al., 2024) and evaluate it using the MT- Bench dataset (Zheng et al., 2024a). GPT-4 used to assess the quality of the model's responses, and report the first-turn score the metric.

---

Published as a conference paper at ICLR 2025
Table 2: tunent results of Llama-2-7B model. We fine-tune the Llama-2-7B model using full fine-tuning and LoRA variants on subsets of the WizardLM (Xu et al., 2024), MetaMathQA (Yu et al., 2024), and CodeFeedback (Zheng al., 2024b) datasets, respectively. And we assess dialogue generation, mathematical reasoning, and coding abilities MT-Bench, GSM8K, and HvaanEval datasets. Bold and underline indicate the highest and second-highess scores, respectively.
Method
MT-Bench
GSM8K
HumanEval
Full FT
5.3020.11
59.3640.85
35.31+2.13
LoRA
5.61+0.10
42.08=0.04
14.76+0.17
PiSSA
5.30+0.02
44.5410.27
16.02+0.78
rsLoRA
5.25+0.03
45.62++0.10
16.01=0.79
LoRA+
5.7140.08
52.11
18. 18:022
DoRA
5.97+0.02
53.07=0.75
19.75+0.41
AdaLoRA
5.57+0.05
50.72+1.39
17.80+0.44
LoRA-GA
5.95=0.16
53.60=0.30
19.81
LoRA-GA (rank=32)
5.79+0.09
55.12+0.30
20. 18.10. 19
LoRA-GA (rank=128)
6.13+0.07
55.07=0.18
23.05+0.37
LoRA-Pro
5.72+0.03
57.57+0.50
22.97+0.35
LoRA-Pro (rank=32)
5.57+0.51
57.97=0.50
26.63+0.35
LoRA-Pro (rank=128)
5.67+0.11
61.08+0.19
30.28
For the math task, we fine-tune the Llama-2-7B (Touvron et al., 2023) model on 100k sample from the MetaMathQA dataset (Yu et al., 2024). The model then evaluated on the GSM8K test set (Cobbe et al., 2021), and we report the accuracy as the metric.
For the coding task, fine-tune the Llama-2-7B (Touvron al., 2023) model on 100k subset of the CodeFeedback dataset (Zheng et al., 2024b) and test it on the HumanEval dataset (Chen al., 2021), reporting the PASS@ I metric.
We compare LoRA-Pro with several baselines, including full fine-tuning, LoRA (Hu et al., 2022), PiSSA (Meng et al., 2024), rsLoRA (Kalajdzievski, 2023), LoRA+ (Hayou et al., 2024), DoRA (Liu al., 2024), AdaLoRA (Zhang et al., 2023), and LoRA-GA (Wang et al., 2024a). By default, we set the rank to 8 and a 16. Following LoRA-GA (Wang et al., 2024a), we initialize the scaling factor as in rsLoRA (Kalajdzievski, 2023), i... +
Table presents our experimental results, which demonstrate LoRA-Pro's superior performance. With rank of 8, LoRA-Pro achieves notable improvements over the original LoRA: 0.11 on MT- Bench, 15.49 on GSM8K, and 8.21 on HumanEval. When compared to the second-best PEFT method, LoRA-GA, LoRA-Pro shows consistent gains: 3.97 GSM8K and substantial 3.16 on HumanEval. These results validate the effectiveness of our LoRA-Pro method.
Interestingly, observe that full fine-tuning unexpectedly underperforms on MT-Bench. We tribute this to potential discrepancies between the WizardLM training data distribution and the MT- Bench evaluation set. The extensive learning capacity of full fine-tuning may lead to overfitting on the training distribution, compromising generalization to MT-Bench. Since LoRA-Pa - aligns more closely with full fine-tuni munittning during optimization, its relatively poor performance on MT-Bench may also be attributed to overfitting.
To further explore the scalability of our method, we increase the rank in LoRA-Pro from 128. Our observations reveal a clear trend: as the rank increases, the performance gap between LoRA- Pro and full fine-tuning narrows rapidly. Notably, LoRA-Pro consistently outperforms LoRA-GA at the same ranks on both GSM8K and HumanEval datasets. At rank 32, LoRA-Pro surpasses LoRA- GA by 2.85 on GSM8K and 6.45 on HumanEval. This performance disparity becomes even more pronounced at rank 128, where LoRA-Pro outperforms LoRA-GA by 6.01 on GSM8K and 7.23 on Hvma These results demonstrate the superior scalability and effectiveness LoRA-Pro across various rank settings.
7

---

Published as a conference paper at ICLR 2025
3.3 RESULTS ON IMAGE CLASSIFICATION TASKS
Table 3: Fine-tuning results of CLIP- ViT-B/16 on image classification tasks. We fine-tune CLIP-
ViT-B/16 using full fine-tuning and LoRA variants across StanfordCars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets. Bold indicates the highest results, while underline rep- resents the second highest results.
Method
Cars
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Average
Zero-shot
63.75
44.39
42.22
35.22
56.46
62.56
15.53
45.73
Full FT
84.23+0.06
77.4420.19
98.09#0.03
94.31
93.95a0.0
75.35+0.10
93.04=0.18
88.06
LoRA
72.81+0.13
73.92+0.38
96.93+0.07
92.40=0.10
90.03+0.14
70.1220.18
88.02=0.07
83,46
ISLoRA
82.38+0.20
78.03=0.76
98. 98.06+0.08
95, 95.04#0.11
93,96+0.18
75.38+0.24
92.74=0.18
87.94
LoRA+
72.87a0.18
74.0740.45
97.01=0.02
92.4240.1
89.96a0.11
70.17+0.15
88 08.08=0.05
83.51
DoRA
73.72+0.06
73.7240.33
96.95+0.01
92.38+0.17
90.03+0.08
70.20 19
88. 88.23+0.05
83.48
LoRA-GA
85.1840.41
77.50:0.12
98.05+0.27
95.28+0.10
94.43+0.19
75.44=0.06
93.68+0.35
88.51
LoRA-Pro
85.87+00088
78.64+0.25
98.46+0.03
95.6620.05
94.75+0.21
76.42+0.14
94.63=0.20
In this section, assess the performance of LoRA-Pro on image classification tasks. To pro- vide a comprehensive comparison. we compare it with several baselines: zero-shot CLIP (Rad- ford et al., 2021), full fine-tuning, vanilla LoRA (Hu et al., 2022), rsLoRA (Kalajdzievski, 2023), LoRA+ (Hayou et al., 2024), DoRA (Liu et al., 2024), and LoRA-GA (Wang et al., 2024a).
We fine-tune the VVT--V6 (Radford al., 2021) model various datasets, including Stan- fordCars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), GT- al., 2017), SUN397 (Xiao al., 2010), and SVHN (Netzer et al., 2011). Accuracy is used as the evaluation metric. During fine-tuning, only the visual backbone of the CLIP-ViT-B/16 model is updated, while the classifier, derived from prompts such as "a photo of {class}", remains frozen.
SRB (Houben al., 2013), RESISC45 (Cheng
The results are presented on Table 3. LoRA-Pro achieves the highest accuracy across all seven datasets. Specifically, on average, LoRA-Pro surpasses zero-shot classification by 43.47, outper- forms LoRA (Hu et al., 2022) by 5.74, and exceeds rsLoRA (Kalajdzievski, 2023) by 1.26. These results validate the effectiveness of our LoRA- method.
3.4 ABLATION STUDY
Ablation study on the full-rank assumption. In Theorem 2.1, we assume that the matrices A E IR"xn and B E Raxr are full-rank dur- ing training. Our goal here is to verify whether this assumption holds in practice. We track the rank changes of all A and B matrices during the fine-tuning process of Lllma-2-7B on the Meta- MathQA (Yu al., 2024) dataset.
Mobrit Rank
matrix
500
1590
2002
3000
30
Matmo Rank
In Figure 2, we illustrate the rank changes of ma- trices A and from the g projection of layer during the training process, with rank set to 8 and 32, respectively. We observed that, although A and B do not initially satisfy the full rank assump- tion (with matrix B initialized as a zero matrix), both matrices achieve full rank after the first up- date step. The rank behavior of A and B in other layers also exhibits similar results.
20
20
matrix
520
1500
Training
Figure 2: Visualization of matrix ranks of A and B during training, with ranks set to and 32, respectively.
This observation provides practical evidence that the assumption in Theorem 2.1 is reasonable and supports the validity of the proposed solutions.
Memory footprint and training time. Here, we evaluate the additional costs associated with using
LoRA-Pro compared to LoRA. We primarily focus comparing the differences in memory cost and training time between LoRA-Pro, LoRA, and full fine- minituning. The results the experiments presented in Table 4. We measure memory cost using A6000 GPUs with batch size of 1.
8

---

Published as a conference paper at ICLR 2025
Table 4: We compare LoRA, LoRA-Pro, and Full Fine-Tuning in terms of memory cost, train- ing time, and performance on the MT-Bench, GSM8K, and HumanEval datasets. Memory cost is measured using a single A6000 GPU with batch size of Training time is recorded on the Meta- MathQA dataset using 8 A A100 GPUs with DeepSpeed ZeRO-2 stage optimization.
Method
Memory Cost
Training Time
MTT-Bench
GSM8K
HumanBval
Full
x GB
4h 20min
5.3020.11
59.3620.85
35.31+2.13
LoRA
~ 8 x 17 GB
1h 30min
5.6140.10
42.08:0.04
14.7620.17
LoRA-GA
~ 8 x 17 GB
1h 31min
5.9540.16
53.60=0.30
19.81=1.46
LoRA-Pro
8 x 21 GB
Ih 41min
5.7220.03
57.5720.50
22.97=0.35
Training time is recorded on the WizardLM dataset using A100 GPUs with DeepSpeed (Rasley et al., 2020) ZeRO-2 stage optimization.
The results are shown in Table 4. From the table, we observe the following: 1) LoRA-Pro requires approximately 4GB more GPU memory compared LoRA. This difference likely stems from the need to compute Bd B, AAT, and their inverses during the calculation of the optimal gradients. 2) Surprisingly, the training time for LoRA-Pro is nearly identical to that of LoRA, with only about 10 minutes increase in additional training time. We attribute this to the fact that the matrices A and B in LoRA are low-rank. Consequently, the extra computations required by LoRA-Pro (such as matrix inversion and the calculation of X) are performed on small r x matrices, making the extra computational overhead manageable.
Considering that LoRA-Pro uses less memory and trains faster than full fine-tuning, while also pro- viding performance improvements over LoRA, we believe that the additional memory and training time costs are acceptable.
Wizzrd LM
MetaMathQA
CodeFeedback
LoRA
LoftAcfro
Loll/-rr
LoRA-SA
LoRA
LoRA
Full Fine Turing
Full Fine Tuning
Full Fine Tuning
2.4
Figure 3: Training loss curves of LoRA, LoRA-GA, LoRA-Pro and Full Fine-tuning on WizardLM, MetaMathQA, and CodeFeedback.
Training curves LoRA-Pro. this part, we present the training loss curves for LoRA, LoRA- GA, LoRA-Pro, and Full Fine-tuning across WizardL MetaMathQA, and CodeFeedback datasets. As illustrated in Figure 3, LoRA-Pro demonstrates a faster convergence speed compared to LoRA and LoRA-GA. Furthermore, LoRA-Pro achieves lower final loss value upon convergence, indi- cating its improved efficiency and effectiveness.
4 RELATED WORK
Parameter-Efficiee Fine-Tuime TieeTTnnigg Given the huge size of foundation models, recent research has focused on developing parameter-effiiiet fine-tuning methods (Hu al., 2022: Liu et al., 2024; Ding et al., 2023; Houlsby et al., 2019; Liu et al., 2023: Lester et al., 2021; Wang et al., 2024c). These methods aim to reduce the cost of fine-tuning by adjusting only a small portion of the model's parameters. Generally, these methods fall into two main categories. The first category is adapter tuning (Houlsby et al., 2019; Sung et al., 2022; He et al., 2021; Zhang et al., 2024; Bapna & Firat, 2019; Hu et al., 2022), which involves inserting small neural network modules, called adapters, into specific layers of the model. During fine-tuning, we keep the model frozen and only fine-tune the lightweight adapter modules, significantly reducing the memory footprint for fine-tuning. The second category is prompt tuning (Lester et al., 2021; Zhou et al., 2022; Li & Liang, 2021; Liu et al., 2022; Wang al., 2023; 2024b; Liang al., 2024). Prompt tuning adapts the models to specific

---

Published as a conference paper at ICLR 2025
tasks by adding specially designed prompts or learnable tokens to the input data, rather than directly modifying the internal parameters of foundation models. In this paper, we focus on LoRA (Hu et al., 2022), prominent method within the realm of adapter tuning.
Low-Rank Adaptation. Low-rank adaptation, initially referred to as LoRA (Hu et al., 2022), has evolved into broad category encompassing parameter-effiient fine-tuning methods based on low- rank approximations (Hu et al., 2022; Liu et al., 2024; Hayou et al., 2024; Kalajdzievski, 2023; Zhang et al., 2023; Kopiczko et al., 2024; Hyeon-Woo et al., 2022; Zhang & Pilanci, 2024; Wang et al., 2024a; Zhao et al., 2024; Wang et al., 2024a). LoRA (Hu et al., 2022) assumes that the changes in the weights of pre-trained models exhibit low-rank structure. Consequently, it re-par re-parameterizes these changes as the product of low-rank matrices, thereby reducing the cost during fine-tuning.
Several variants of LoRA have been proposed to address different aspects of this approach. For example, DoRA (Liu et al., 2024) improves LoRA (Hu et al., 2022) by incorporating a learn- able magnitude vector to re-scale the normalized product of low-rank matrices. Another variant, rsLoRA (Kalajdzievski, 2023), introduces new scaling factor stabilize training in high-rank sce- narios. LoRA+ (Hayou et al.. 2024) improves upon LoRA by applying different learning rates to the two low-rank matrices. Additionally, Galore (Zhao et al., 2024) employs SVD to project the gradients and its first and second momentum of full training into a low-rank space, thereby reducing the memory footprint during pre-training and fine-tuning.
5 CONCLUSION
In this paper, we introduce LoRA-Pro, a novel approach designed to bridge the performance gap between LoRA and full fine-tuning. We have discovered that using LoRA for fine-tuning is equiv- alent to fine-tuning the original weights with a virtual equivalent low-rank gradient. Based on this insight, we propose adjusting the gradients of matrices A and B to make the equivalent gradient match the true full fine-tuning gradient, thereby reducing their performance gap. Fortunately, we theoretically prove that there exists an optimal closed-form solution for updating matrices and B, which are applied during fine-tuning in LoRA-Pro. To validate the effectiveness of our method, we conduct extensive experiments across various domains, including natural language understand- ing, dialogue generation, mathematical reasoning, code generation, and image classification tasks. The results demonstrate that --RA-Pro significantly improves LoRA performance and narrows the performance gap with full fine-tuning.
Limitations. LoRA- Pro still have some limitations: (1) LoRA-Pro still adheres to LoRA's assump- tion that AW is of low rank. However, this assumption may break down in cases of pre-training or when there is a large amount of fine-tuning data, potentially leading to suboptimal results. (2) So far, we have only applied LoRA-Pro to variants that have structure similar to LoRA. It currently cannot be applied to structurally different LoRA models, such as DoRA (Liu et al., 2024) or FLoRA (Wen & Chaudhuri, 2024). We plan to explore these directions in future research.
ACKNOWLEDGEMENT
This work was funded by the National Natural Science Foundation of China under Grants (62276256, U2441251) and the Young Elite Scientists Sponsorship Program by CAST (2033NNRC00I. We thank Jie Cheng and Yongcan Yu for providing computational resources critical this work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In ACL-IJCNLP. 2021.
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In EMNI.-IJCNILP. 2019.
10

---

Published as a conference paper at ICLR 2025
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state the art. Proceedings of the IEEE, 105(10):1855-188,, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.1110.411,,. 2021.
Ning Ding. Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Parameter-cfficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22) 2350 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprini arXiv:240..1235. 2024.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In ACL-IJCNLP, 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-223) 2019.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In NeurIPS, 2022.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN 2013.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efffiient transfer learning for nlp. In ICML, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Nam Hyeon- -oo,, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-effin
federated learning. In ICLR, 2022.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint N.X2:22112.03332, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In ICCV, 2023.
11

---

Published as a conference paper at ICLR 2025
Dawid Jan Kopiczko. Tijmen Blankevoort, and Yuki M Asano. Vera: Vector bastod random matrix adaptation. In ICLR, 2024.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshop. 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt luning. In EMNLP, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, 2021.
Jian Liang, Lijun Sheng, Zhengbo Wang, Ran He, and Tieniu Tan. Realistic unsupervised clip fine-tuning with universal entropy optimization. In ICML, 2024.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35. 2023.
Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. ACL, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.0224 022448 2024.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NeurlPS workshop, 2011. 1.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 1-67, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. SIGKDD, pp. 3505-3506, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. CVPR, 2022.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In /CML., 2013.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In NewrIPS, 2024a.
12

---

Published as a conference paper at ICLR 2025
Zhengbo Wang. Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In ICCV, 2023.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Connecting the dots: Collaborative fine-tuning for black-box vision-language models. In ICML, 2024b.
Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free clip-based adaptation. In ICLR, 2024c.
Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In ICLR, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardIm: Empowering large pre-trained language models to follow complex instructions. In ICLR, 2024.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. ICLR, 2024.
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation mod- els. ICML, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. alama-adapter. Efficient fine-tuning of large language models with cero-initialized attention. In ICLR, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory--fficcint Ilm training by gradient low-rank projection. In ICML, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurlPS, 2024a.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCoodenteeppeeter: Integrating code generation with execution and refinement, In Findings of ACL, 2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-234) -3488, 2022.
13

---

Published as a conference paper at ICLR 2025
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
Appendix
The structure of Appendix is follows,
Appendix A contains the notation usage in our paper.
Appendix B contains the proofs of the theorems in the main manuscript.
Appendix details the optimization algorithms of the proposed method.
Appendix D presents additional experimental results.
A NOTATION
In Table 5, we detail the notations utilized our paper.
Table 5: Description of notations used in the paper.
Notation
Description
scaling factor in LoRA low rank matrices in LoRA gradients of full fine-tuning gradients of matrix in LoRA LoRA
gxxr E grxn
9 aw
E mmxn
9oora
24
ØL
sB" g E R2
Slora
an
sgA" E gamxr
gradients of matrix differential of the loss function differential of the matrix A differential of the matrix B Frobenius Norm Frobenius inner product
dL
AA
dB
(F
B PROOF OF THEORETICAL RESULTS
B. PROOF THAT EQUIVALENT GRADIENT IS LOW-RANK
Lemma. Assume B E Rmxr A E Rrxn and g Rmxx . RR represent matrices and their corresponding gradients in LoRA oplimization. We demonstrate that the equivalent gradient:
g = sg A sBg
(16)
where s > is the scaling factor, has matrix rank at most 2r.
Proof. Since matrix rank satisfies the property subadditivity, we have:
rank(ğ) rank(sg" =B S rankly A) + rank(Bg
(17)
Furthermore, for any matrices A and B, rank(AB) min(rank(A)) rank (B)). can bound the ranks as follows:
Therefore,
r A) < ) <
(18)
rank(Bg* S min(rank( B), rank(g S
(19)
14

---

Published as a conference paper at ICLR 2025
Thus, in conclusion, the equivalent gradient has rank of at most 2r: rank(ī) rank(y A) + (rnk(Byy))
(20)
min(rank(g rank (A)) + min(rank(B)) rank(g"))
(21)
S 2r.
(22)
B.2 PROOF OF THEOREM 2.1
Theorem. Assume matrices B lR E grx" are both full rank. For the objective
g g l1ğ - gll;r, the solutions are given by:
g 1 [B" 9+ g B)²BB) glora + XA
(23)
9 - B(B" ("A"")"""""A"AAAA"""" BX
(24)
Ioroa { BX.
(25)
Here, X Rr represents an arbitrary matrix.
Proof. For simplicity, we denote L = ||=Bg* + sg A -ll... To solve the optimization problem,
need to satisfy the following conditions:
= 2BBT(( (sBga + 8g A--A-9 g) =
(26)
0y
0L
0g 2(sBgA + sg A g) 0
(27)
Given that matrices A and B are full-rank, AA and B² B are invertible. And from Equation (27), we derive:
88 9 (AA")- AT(AA""")--
(28)
Substituting this into Equation (26), we obtain the following linear equation:
AAlI A (AA") 11A] A] = (B" g|l - A (AA"))-))))
(29)
Here, we notice that the matrix P = I AT (TAAAT)-1A is projection matrix with rank r. The solution to the linear equation (29) is:
g = _ (B" B) + XA,
(30)
where X E R' represents an arbitrary matrix. We take the solution (30) into Equation (28), we derive:
n" 3 - -F
BX
(31)
While we have obtained closed-form solutions for gd and - these solutions explicitly depend on the gradient of the matrix W. i.e., g, which undesirable since g is unknown during LoRA opti- mization. Fortunately. the solutions can be transformed into the forms of the gradients of standard LoRA, where the gradients are:
Nera = S g,
B f = SgAT.
(32)
Therefore, the solutions to the optimization problem can be written as:
32 3
1
(B=B)=
B)
g XA,
(33)
8 -T-
a "AA") BX.
(34)
22
In our method, we perform the standard forward and backward passes LoRA, then adjust the gradients of A and B using Solutions (33) and (34), and subsequently update them. n
15

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF THEOREM 2.2
Theorem. When updating matrices A and B using the closed-form solution from Theo-
rem 2.1, we proceed as follows:
A A A-79
(35)
+ B - -19
(36)
where y 2 denotes the learning rate. Our method ensures decrease in the loss, akin to the standard gradient descent algorithm, expressed by:
dP --lllbor J/2 (B"B) z2
B) Diora) " + 32
1
B(B"
"B""' (AA") "))
(37)
Proof. summary, the proof Theorem 2.2 is divided into two distinet parts. begin with, we demonstrate that dL can be expressed in the following form:
1 dL - -- 32 (BT
1 Kora' + (gloa Eell B(DBB)-BB" Isfora BT (AA")-"))} (38)
In the second part, prove that this expression for dL is always less than equal zero: dL s 0.
Part Therefore, in this part, we first prove Equation (38). During the optimization process, the differential change in the loss function, d', can be expressed in terms of the differentials dAA and dB as follows:
aL 4a dA)F +
OL
dL
dB).
(39)
0B
From Equation (35) and (36), we can derive that:
dA 7 dB -
(40)
0L = girn and Given that 0A
0L OB
_ it follows that:
d7. - ( A + (glora (
1
(BT Diordd F + (olor 1 / [T- B(BBBBB B) Bdd 1" (AAT)-1)a (41)
=->((gior
52
+ (giora XA) (of BX)
And we have the following equation:
Wiroa: XA) (slora: BX)
Loro AT. .)rr (BT Aiora X)F
(ofora AT -B8 glora: X)r
(42)
=(<B²* A) - -B" (sg ) ),X), X)F
=0.
Therefore, we have:
1 11- B ( (AA")-2))) (43)
dL = --llaar s2 (B" ()= B) toraa
Part H. In this part, we aim to prove dL < O. Given that the learning rate > 0, it suffices to show the following inequalities:
L 5z (B B) A 2 0,
(44)
(g
B(B" -) Po (AA*) 2 0.
(45)
By proving these inequalities, establish that dl. 0 derived from Equation (38).
16

---

Published as a conference paper at ICLR 2025
① Proof of (g
/F 2 0.
To begin with, we need to show that (B" B)-1 is positive definite. To establish this, it is sufficient to show that B B is positive definite, as the inverse a positive definite matrix is also positive definite. To achieve this, consider any non-zero vector I, and noting that B is full-rank, have,
(x, B" Bz) (Bx, Br)) || 0.
(46)
This shows that Ba B is positive definite. Consequently, (B² B)- is positive definite as well. Since [BT B)-1 is positive definite, and thus we can apply Cholesky decomposition, and (B" B)-1 = UUT With this, have.
(groa (BTB) "Bora) F
z2 Liora UU giora)
g (UT Norr U giorad
(47)
- giorall? 20 20
Proof of (gfora: 1 - B(B BB 1o " (AA")"))=
Similarly, we can prove that matrix (AAAT)-² is positive-definite. By employing Cholesky decom- position, express (AAT)-1 UUT where is lower-triangle matrix. Subsequently, define P B(B B) BT can be shown that p2 = and is symmetry, indicating that P projection matrix. Consequently, the eigenvalues of either 0 or which implies that P is positive semi-definite. Utilizing the Cholesky decomposition, we derive that = VV where V is lower-triangle matrix. Finally, have:
(glora 2 B(B" B) B) B" 1 (AAra(AAT)TT) (AAT)-") (oooa VV B UUT)
22 (VT g U, V Ulr (48)
2 ||V plora Ul? 20
In summary, based on the above proofs, have demonstrated that:
1
dL =->(ggora (B" B) B) gloral F + (glora g2
[I B)B" B Ilooa (AAT)~1>F)
( S
0 shown
shown in
(49)
B.4 PROOF OF THEOREM 2.3
Theorem. Consider the optimization problem,
min llg - 1l +
11g g 1|?;
(50)
where gd and gd are the optimal solutions as stated in Theorem 2.1. The optimal X can be determined by solving the Sylvester equation:
B' XAA $2
garn AT 1)= -
(51)
which has unique solution X provided that BTT B and eigenvalues.
AAT do not have any shared
Proof. For simplicity, we denote L = problem, we need satisfy the following conditions:
~ o 12 + 119
g. 1. To solve the optimization
ØL.
0.
(52)
ax
17

---

Published as a conference paper at ICLR 2025
Since gd and 9 are solutions in Theorem 2.1 and D sB" g and glora sgA", obtain that:
2(g Nora) 2B (g" Mlora 0,
g gAA A Bd gB Flora
A² BB Piora:
(53)
1
BX BX XAA = 5
(B" B) Giora A
which is a Sylvester equation. This equation has a unique solution for X if and only if Bd B and -AA have no shared eigenvalues.
C OPTIMIZATION ALGORITHMS
In this section, we present the pseudo-codes for implementing our LoRA-Pro method using the SGD (Sutskever et al., 2013) and AdamW (Loshchilovv & Hutter, 2019) optimizers. The details are provided in Algorithm 1 and Algorithm respectively.
LoRA-Pro with SGD optimizer. In the standard SGD algorithm, as illustrated in Algorithm 1, all we need to do adjusting the gradients matrices and with the solutions Theorem 2.
Algorithm LoRA-Pro with SGD optimizer
Require: Given initial learning rate y, scaling factor 8.
1: Initialize time step +- 0, low-rank matrices Ao E Rrx and Bo R
2: repeat
3:
(-1+1
4: Noor: Foora + (y gradients
Select batch and return the corresponding
5:
A, B + AL-1, B
P. Obtain the low-rank matrices and
6:
X SolveSylvester BTT BX XAAT = - (BT l
B) "oora AT) to Compute by solving
the sylvester equation
7:
g - Siora + XA
Adjust the gradients of LoRA with Theorem 2.1 g n -- -B(BB B) B² B" Igtrr (AAT)-1 BX
8:
9:
A, + A:-1 79
10:
B. t 12-1-193
11: until stopping criterion is met
12: return optimized parameters A, and B,
LoRA-Pro with AdamW optimizer. In AdamW optimizer, the implementation becomes more complex. We aim to closely approximate full fine-tuning during optimization. Several modifications necessary. Firstly, order mimic full fine-tuning, after adjusting the gradients of matrices A and B, we need to compute the equivalent gradient,
g ==
" A sBga
(54)
Subsequently, we calculate the first and second moments of this equivalent gradient to derive the corresponding A gradient, g AdomW Secondly, we determine the gradients with respect to matrices and B follows:
sB" Adam B si AdamW AT
(55)
Thirdly, the weight decay process must be adjusted. line with full fine-tuning, the weight decay is given by:
W - -1 -1) + +BA).
(56)
This can be decomposed into:
Wo + (1 (1- (1->A)Wo
B+ VI-ABB At VA-
(57)
18

---

Published as a conference paper at ICLR 2025
Algorithm 2 LoRA-Pro with AdamW optimizer
Require: Given initial learning rate 7, scaling factor 8, original weight matrix Wo I mm and B1 0.9, 32 = 0.999, c 10 8 À E R
1: Initialize time step & low-rank matrices Ao R and Bo mxxy first momentum
mo E 1mxx,, second momentum E Rmxx
2: repeat
3: t+
4: Norr: Fiora t SelectBatch A,-,, B(-1)
Select batch and return the corresponding
gradients
5:
A, + A,1, B
4 Obtain the low-rank matrices and B
6:
X +-1 0
X's value does not affect equivalent gradient
7:
g
7 (B" B) B)
giora + XA
> Adjust the gradients of LoRA with Theorem 2.1
8:
g B = x² - -B(B"B)
BTT
Jotora (AAT)
BX
9:
î + sg A +BBBAA
D> Compute equivalent gradient
my 3 (1)
11:
+ -2 + (1 32)9
TH I-3
13:
E. + 21
1-33
14:
Po.te ÿ AdamW
ih
15:
T t sB" o Adamm W
16:
=
&- sÿ AdamW AT
17:
+ SolveSylvester BT BX XAAT = -B -grra AT) D Compute by solving
the sylvester equation
18:
g + (B² B) Trora + XA
> Adjust the gradients of LoRA with Theorem 2.1 Mora (AAT)- -B
19:
ÿ = I [L--BB'' B(B
1
B
20:
A + VI-
Weight Decay
21:
B+ B V1 TAB
22:
Wo + (1 W
23:
A A,11 - A1-1--199 79"
24:
B, B--1- -9
25:
until stopping criterion is met
return optimized parameters A, and B,
19

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
D.1 ABLATION STUDY OF THE SELECTION OF X
Based on Theorem 2.1, in LoRA-Pro, the matrix X can be chosen arbitrarily. While its selection does not affect the equivalent gradient, it does influence the updates of matrices A and B in LoRA. Here, we conduct an ablation study on the choice of X.
We compare three possible values for X. 1) Zero solution: In this simplest case, we set X O. 2) Sylvester solution: Here, is obtained by solving the Sylvester equation, as described in Theorem 2.3. 3) Symmetry solution: This approach aims to balance the contributions of both terms in the equation O = sg A + ABgA, enforcing the condition و Bg For the symmetry solution, solving for X yields:
B" -BB glrr A)
1 (B"BB 25²
X
(58)
2s
The comparison of the selection of X is presented in Table 6. As shown in the table, the choice of X significantly impacts LoRA-Pro's 's performance, particularly evident in the GSM8K dataset. Differ- ent X selections influence the subspaces of A and B, ultimately affecting the optimization process described in Theorem 2.1. Our experiments demonstrate that the Sylvester solution consistently outperforms both the zero and symmetry solutions across all three evaluation tasks. We attribute the superior performance of the Sylvester solution to its ability to select subspaces for A and B that enable faster gradient descent (i.e., maximizing the approximation between the modified gradients g go and the LoRA gradients g Iorag).
Table 6: Ablation study on the selection of different X in LoRA-Pro.
Choice of X
MT-Bench
GSM8K
HumanEval
Zero
31.7410.69 42.81+0.62 5.72+0.03
5.58#0.14
17.28=0.35
Symmetry (Eq. (58))
5.71+0.11
17.88+0.35
Sylvester (Thm. 2.3)
57.57+0.50
22.97+0.35
D.2 VISUALIZATION OF DIFFERENCES BETWEEN EQUIVALENT GRADIENTS AND FULL GRADIENTS
In this section, we fine-tune Llama-2-7B on the MetaMathQA dataset and visualize the dis- crepancies between the equivalent gradients of LoRA and LoRA-Pro and the full gradients during training, i.e., the differences before and after gradient adjustments. We present visualizations for different optimization modules, including the Q. K. V, O, Up, Down, and Gate layers, and provide results for these modules across the shallow (1), medium (15), and deep (31) layers of Llama-2-7B
The results are shown in Figure 4. From the figure, we can draw the following conclusions:
After gradient adjustments in LoRA-Pro, we observe a significant reduction in the distance between the equivalent gradients and the full gradients.
In certain layers, the discrepancy between LoRA's equivalent gradients and the full gradi- ents continues to increase (e.g.. Layer O, Up, Gate projections; Layer 15 Up and Gate projections; and Layer 31 O projection). However, in these layers, the discrepancy for LoRA-Pro remains stable, indicating that LoRA-Pro can consistently align with the full gradients during training, preventing the model from settling into sub-optimal solutions.
In deep layers, the discrepancy between equivalent gradients and full gradients decreases as training progresses, whereas in shallow and medium layers, the discrepancy first in- creases and then stabilizes. The cause of this phenomenon is not yet clear, and we plan to investigate further in future research.
These findings highlight that LoRA-Pro effectively reduces the distance between LoRA and full gradients during training and ensures continuous alignment with full gradients, underscoring the efficacy of LoRA-Pro.
20

---

Published as a conference paper at ICLR 2025
Dr. Snav 0e. Lave 31
0.664
6410
0.500
0803
BRT Haper,
8.575
Jwn.erne...... de Mone Jarei
waw. JLCCS
6435
Lone Ma
0202
0411
0.510
0.24
0.30
0.7
080
0.20
635
0:24
0.00
DP k0. ww.. 1
140 (10, M00.MML.. MAR TI..
0.2
DIT keaa www..................k..
0.4
0.2
081
640
6.14
0.20
0351
0.30
D3 lit Layer ALAALRn
0.31
0.32
89 attopp pai
044
ott re..ooo..oppop...mm. 1ews aog
INT 28.2. ALWLLR
0.0
0.44
180
6.00
030
0.50
6.50
0.24
633
0.50
1200
250
2000
Figure 4: Visualization of the differences between the equivalent gradients of LoRA, LoRA-Pro, and the full-parameter gradients during training, i.e., -lğ--9 -gli F. The rows illustrate the differences across various modules, including Q. K. V. O, Up. Down, and Gate. The columns show the differences at different depths, categorized as shallow (1), medium (15), and deep layers (31).
21

---

Published as a conference paper at ICLR 2025
D.3 EXPERIMENTS RESULTS WITH DIFFERENT LEARNING RATES
To demonstrate the effectiveness of LoRA-Pro, we evaluated its performance on GSM8K under learning rates of le-5 and 5e-5, comparing it with LoRA and LoRA-GA. The results, presented in Table 7, show that LoRA-Pro maintains its advantages under both learning rates, highlighting its robustness to variations in learning rate.
Table Performance comparison of LoRA, LoRA-GA, LoRA-Pro on GSM8K with learning rates le-5, 2e-5, and 5e-5.
GSM8K
LoRA
LoRA-GA
LoRA-Pro
le-5
36.65+0.82
50.25+0.62
56.48+0.27
2c-5
42.08+0.04
53.60+0.30
57.57+0.50
5e-5
46.41+0.16
52.89+0.19
58.76+1.88
D.4 ADDITIONAL. EXPERIMENTS ON LATEST MODELS
Table 8: Performance comparison of LoRA, LoRA-GA, and LoRA-Pro with Llama-2-7B and Llama-3.1-8B.
GSM8K
LoRA
LoRA-GA
LoRA-Pro
Llama-2-7B 42.08+0.04
53.60+0.30
54.23+0.79
Llama-3.1-8B
71.0440.26
72.20+1.15
75.4940.42
To further demonstrate the effectiveness of LoRA-Pro, conducted additional experiments using the latest model, 1--MA-3.118B (Dubey et al., 2024). We fine-tuned the model using these three mothods,LoRA, LoRA-GA, and LoRA-Pro.on the MetaMath1001 dataset and evaluated its perfor- mance on the GSM8k dataset. All results are averaged over three different random seeds.
As shown in Table 8, LoRA-Pro demonstrates clear advantage over both LoRA and LoRA-GA when applied to the 1 1-8B model, further highlighting its effectiveness.
22


Reviews:
[{"review_id": "MJPrZFxyQm", "reviewer": "Reviewer_j8va", "summary": "LoRA reduces the parameters required during training by introducing a low-rank matrix, thereby reducing computational requirements and memory footprint while maintaining model performance. This paper introduces LoRA-Pro to enhance LoRA’s performance by strategically adjusting the gradients of the two low-rank matrices, allowing the low-rank gradient to more accurately approximate the full fine-tuning gradient.", "questions": "•\tPlease explain why LoRA-Pro is much better than Full FT in Table 3. Is there overfitting phenomenon that gives LoRA-Pro an advantage on these specific tasks?\n•\tPlease explain why the training speed of LoRA-Pro is slightly worse than LoRA and much better than Full FT in Table 5. It will be better to provide a breakdown of the additional operations required by LoRA-Pro compared to standard LoRA, and to quantify the overhead in terms of FLOPs\n•\tPlease provide more choices of X in Table 4.   For example, the analytical solution of X can be calculated in Theorem 2.1. \n•\tLoRA-GA[1] has the same purpose as the work in this paper. Please discuss how LoRA-Pro differs conceptually or methodologically from LoRA-GA and supplement the results of LoRA-GA in Table 3 and Table 5.\n•\tIf available, please add discrepancy curves between the low-rank equivalent gradient and the full fine-tuning gradient. Please supplement the training loss curves to illustrate the convergence rate of the proposed method.\n\n[1]Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In this paper, the optimal solution of adjusting the gradient of low rank matrix is deduced theoretically and applied to the fine tuning process of LORA-PRO, which narrows the performance gap between LoRA and full fine tuning. Extensive experiments in multiple tasks have been carried out to prove that LORA-Pro greatly improves the performance of LoRA.", "weaknesses": "1. The method is not applied to the latest LLMs.\n2. LoRA-Pro brings in extra memory cost and computation cost in the training.\n3. It is not clear why the optimal gradients for the low-rank matrices do not explicitly depend on the full fine-tuning gradient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oOOHUfBsXz", "reviewer": "Reviewer_Aox8", "summary": "The authors show that demonstrating that optimizing with LoRA is mathematically equivalent to performing full fine-tuning using a low-rank gradient for updates. They introduce a novel parameter-efficient fine-tuning (PEFT) method named LoRA-Pro, which minimizes the divergence between the true gradient and the low-rank gradient by adjusting the gradients of matrices A and B. The authors provide theoretical proof of the optimal gradients and detail the optimization process utilizing these gradients. They conduct extensive experiments across various tasks, including natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification, showcasing the effectiveness of their proposed method.", "questions": "- It would be interesting to explore how the equivalent gradients differ from the low-rank gradients in LoRA. A quantitative analysis could provide valuable insights into this relationship. For instance, how does the difference between these gradients vary across different models, deeper layers, and various types of weights? Conducting such a study could yield interesting findings, and further strengthen the motivation behind the gradient adjustment.\n\nMinor:\n\n- Please provide metrics for the datasets used in Table 1. I believe not all metrics used are “Accuracy”. For instance, CoLA typically usually uses Matthews Correlation Coefficient - kindly specify for all datasets.\n- How exactly is the *efficient* mode less stable than the *full* mode? To clarify, all the results in the main paper are derived using the *efficient* mode, is that correct?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clearly articulated, and the paper is well-written with well-explained ideas.\n\n- The proposed method enhances LoRA, a highly relevant technique for fine-tuning, in a straightforward manner, supported by both theoretical proofs and experimental evidence.\n\n- The experiments and ablation studies are thorough, demonstrating the effectiveness of the proposed method across diverse settings.", "weaknesses": "- The authors conducted all experiments for the different methods (for a given task) using the same learning rates. However, this approach may not provide a fair representation of optimal conditions, as the selected learning rate could work well for one method but not for others. To strengthen their findings, it would be beneficial for the authors to perform a learning rate sweep for each method. If this is too resource-intensive for all datasets, they might consider limiting the sweep to a single task, which would further increase confidence in the effectiveness of their proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "91UF7vtCSS", "reviewer": "Reviewer_9Wt3", "summary": "The paper introduces LoRA-Pro, a novel method aimed at improving the performance of LoRA in fine-tuning foundation models. LoRA-Pro enhances LoRA by adjusting the gradients of low-rank matrices to better approximate the full fine-tuning gradient. The paper establishes a mathematical equivalence between LoRA and full fine-tuning, and uses this insight to optimize LoRA's performance. Extensive experiments across various tasks demonstrate that LoRA-Pro substantially narrows the performance gap between LoRA and full fine-tuning.", "questions": "- Do LoRA-Pro have some failure cases? (e.g., potential limitations or edge cases where LoRA-Pro might not perform as well)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper presents a novel approach by mathematically linking LoRA with full fine-tuning through gradient adjustments.\n- The theoretical foundation is robust, with clear derivations and optimal solutions provided for gradient adjustments. The experiments are comprehensive, covering multiple domains such as language understanding, dialogue generation, mathematical reasoning, and image classification.\n- The paper is well-organized.", "weaknesses": "- It will be better to test LoRA-Pro in real large models (e.g., 70B).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)