RaSA: Rank-Sharing Low-Rank Adaptation
Authors: Zhiwei He
Zhaopeng Tu
Xing Wang
Xingyu Chen
Zhijie Wang
Jiahao Xu
Tian Liang
Wenxiang Jiao
Zhuosheng Zhang
Rui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: parameter-efficient fine-tuning
large language model
low-rank adaptation
Primary area: foundation or frontier models, including LLMs

Abstract: Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at: https://github.com/zwhe99/RaSA.

Published as a conference paper at ICLR 2025
RASA: RANK- LOW-RANK ADAPTATION
Zhiwei Hel*
Zhaopeng Tu
Xing Wang² Wenxiang Jiao
Xingyu Chen Zhuosheng Zhang'
Zhijie Wang 1x Rui Wang
Jiahao Xu
Tian Liang?
Shanghai Jiao Tong University
Tencent AI Lab
(zwhe, cs, galaxychen, violetevergarden, zhangzs, wangrui 12 @s ieutuu.dd. edu. cn (zptu, wwwghtxwwngg.tno.......nngnn...e jettexu, tti lang, joe jow.:olwwaoleeeeeeeeeeee @tencent.coom
ABSTRACT
Low-rank adaptation (LoRA) has been prominently employed for parameter- efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the ex- pressive capacity of LoRA by leveraging partial rank sharing across layers. By forming shared rank pool and applying layer-specific weighting, RaSA effec- tively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core ad advantages of LoRA but also significantly boosts per- formance in challenging code and math tasks. Code, data and scripts are available at: https::/ thub. com/ wwwe99/RaSaa...
INTRODUCTION
Low-rank adaptation (LoRA, Hu et al. (2022) has become a de facto parameter-efficient fine-tuning (PEFT) method for adapting large language models (LLMs) to specific downstream tasks. Its core idea is to constrain the parameter updates to be low-rank, which significantly reduces the number of trainable parameters and allows them to be merged back into the original model, thereby avoiding additional inference latency. Despite its advantages, recent studies have shown that LoRA still lags behind full fine-tuning (FFT). particularly in scenarios involving large training datasets and complex tasks such mathematical reasoning and code generation (Jiang et al. 2024 Biderman et al. 2024). A plausible explanation for this performance gap is that the low-rank constraint limits the expressive capacity of LoRA. For instance, Biderman al. (2024) empirically found that the effective rank required for FFT is 10-100x higher than typical LoRA configuration. and Zeng & Lee) (2024) theoretically demonstrated that a Transformer network (Vaswani al. 2017) requires rank at least half the size of the model dimension to approximate another model of similar size.
Although the limited number of trainable parameters results in limited expressive capacity, recent studies still indicate redundancy in LoRA's parameters. For example, Kopiczko et al. (2024); ; Song al. (2024); Renduchintala al. (2024): Li al. (2024) further reduced the number of LoRA's parameters by sharing them across layers and modules with only slight performance loss. Brüel
Gabrielsson et al. (2024) compressed 1, 1,000 LoRAs trained from different tasks by sharing their
parameter spaces. This contradiction suggests that LoRA's parameters are still not being fully utilized.
Combining the above two observations, we propose Shaa--h Low-Rank Adaptation (RaSA), an approach that boosts the expressive capacity of LoRA by enabling partial rank sharing across layers. Specifically, given an LLM with L layers, RaSA extracts k ranks from each layer's LoRA update to form rank pool of L x l ranks, which is shared across all layers with layer-specific weighting. RaSA retains the core advantages of LoRA- - keeping the same parameter overhead and allowing for easy merging back into the model. Moreover, since modern LLMs typically have deep architectures (i.e., large L), RaSA greatly increase the effective rank of the parameter update by (L. 1) x k.
"Work was done when Zhiwei He, Xingyu Chen, and Zhijie Wang were interning at Tencent AI Lab. Zhaopeng Tu and Rui Wang are co-corresponding authors.

---

Published as a conference paper at ICLR 2025
However, higher rank does not necessarily lead to better expressive capacity. To rigorously assess the benefits of RaSA, we analyze its capacity to reconstruct high-rank matrices compared to LoRA. Theoretically, of LoRA. Empirically, we show that when k is relatively small, RaSA can be easily optimized to achieve a significantly lower reconstruction error than LoRA. Finally, conducted experiments on mathematical reasoning and code generation, demonstrating that the lower reconstruction error translates improved downstream task performance.
prove that RaSA's minimum reconstruction error is bounded by that
Our contributions are summarized as follows:
We propose RaSA, a novel extension LoRA by by allowing partial rank sharing across layers, which significantly improves the method's efficiency and expressiveness ($ 2
We provide a comprehensive analysis both theoretical and empirical - showcasing RaSA's superior capacity for matrix reconstruction ($ and its resultant improved performance on rigorous downstream tasks (e.g. code and math) ($
2
METHOD
layer-specific
shared across layers
B
X
A,
B
B
X
X
As
D:
Lkk
(a) LoRA
(b) RaSA
Figure 1: Decomposition of the update matrix AW, in LoRA and RaSA, where is the layer index.
2.1 FORMULATION
Given a pre-trained weight matrix W E gbxe LoRA constrains its update to a low-rank form by decomposing the update matrix AW gbxe into product of two rank-r matrices:
W + AW W
a BA
(BER) R A RR)x))
(1)
where rank < min(b, a) serves bottleneck dimension, reducing the number of trainable scaling factor. an LLM with L layers, LoRA assigns distinct trainable
parameters, and matrices to each layer-i: (B.A,) )iAr)) (Figure RaSA, on the other hand, mitigates the low-rank bottleneck of LoRA through rank sharing. Specifically, RaSA takes out & ranks in each layer and shares them across all layers. This process can be conceptualized follows:
1. Split the matrices B, and A, into layer-specific parts (B,, A and layer- parts (B,, A.):
B
B
A
A
ATT..
(2)
(A-(F-A)
Bbok
Rex(r
M
Baxk
2. Concatenate the layer-shared parts across all layers to form shared rank pools (Bs and As):
Bs = B B2
Br. (1X(T As [A]
A
AT T
E
(Lxx)xx
Therefore, the update for layer-i is given by:
W: + AW, = W, + a (B,,, + BsAs)
7
= W, + B Bs diag( ())
A
As
2

---

Published as a conference paper at ICLR 2025
To enable layer-specific weighting, replace the constant diagonal matrix with trainable diagonal matrix D; = diag(d, d1, d2.-.. .d.... dr-k+Lk), yielding the final RaSA update (Figure Ib))
A As
W = W; + B Bs
D
(5)
Ri k1
2.2 ANALYSIS & IMPLEMENTATION DETAILS
Rank of AW Comparing Equations ① and 5 RaSA increases the rank AW from r to r-k-Lk Since modern LLMs are deep, RaSA significantly boosts the model's expressive capacity by enabling a higher effective rank, on which we have a detailed discussion 3 Each layer in RaSA maintains the same rank for AW AW, which sets it apart from methods that dynamically assign ranks across layers, such as AdaLoRA (Zhang et al. 2023) and PriLoRA (Benedek & Wolf 2024).
Additional Parameters RaSA introduces the diagonal matrix D; additional parameters. Since D; is diagonal and operates only at the bottleneck dimension, the added parameters are negligible. In practice, D; contributes less than 0.001 % of the total model parameters.
Initialization Following LoRA, we use Kaiming initialization (He al. 2015) for Á, and As, and initialize B, and Bg zero. For D, we differentiate between the layer-specific and layer-shared parts by scaling proportionally by their respective ranks:
i k.
d
(6)
x
LI ifj k.
Same Dimension Assumption RaSA assumes that all layers share the same dimensionality. This holds for the vast majority of models (c.g. Llama (Dubey et al. 2024). Mistral (Jiang et al. 2023
RECONSTRUCTION ERROR ANALYSIS
While RaSA increases the effective rank of AW, higher rank does not necessarily guarantee improved expressive capacity. For instance, a full-rank identity matrix can only perform the identity transformation. To assess the expressive capacity of LoRA and RaSA, we compare their abilities to reconstruct a set of high-rank matrices (M.). where rank( = R >> r. Under the Frobenius norm, the minimum reconstruction error (MRE) LoRA is defined as:
Clora
min B.
Σ M. B.A.
According to the Eckart- Bouug-Missky theorem (Eekart & Young 1936), we can perform singular value decomposition (SVD) on M,;
SVD(M)) 20
(0
(6)
)
LoRA's optimal approximation is given by the first components of SVD(M,) and Clora becomes the sum of squares of the discarded singular values (those beyond the r-th one):
1. Cjora Σ II 2o
L
R
Σ
Σ
(0)
(9)
1=1 j=r-1
Similarly, when each layer shares lt ranks out, define the MRE of RaSA as:
A
Crasa(k)
min
DIM. [B,
B3]
D
(10)
B.
As
For simplicity, in this section we consider that D, operates only on the shared matrices Bg and As. which does not affect the value of Crasa(k):
1. M. (B.A, + BsD,
Crasa(k)
min As
Σ
(11)

---

Published as a conference paper at ICLR 2025
3.1 THEORETICAL ANALYSIS
Theorem 3.1. Crasa(k) I Clora
Proof. To prove this, we construct feasible solution RaSA that achieves the same reconstruction error as LoRA's minimum error. This is done by distributing the ranks shared across layers in RaSA such that they cover the same rank range as the optimal LoRA solution.
For each layer-i, we take the last components (corresponding the indices r - k + through r) from LoRA's optimal approximation (Equation ()) forming the following matrices:
u
(1)
(
u
r-
u
V
++1
v r-2
0
(12)
(0)
r-k+1
k+2
The shared matrices Bs and As are constructed by stacking (+) and vi(i) from each layer:
B5 (Ua)
U
U
As [v())
V
volj"
(13)
Similarly, we define the diagonal matrix D; for each layer-i by placing the corresponding singular values (() in their appropriate positions:
D; = diag(( (O
Σ
0)).
(14)
Finally, the matrices B, and A, are formed from the first k components of SVD(M,):
B,A, D
(15)
Substituting Equations to 15 into Equation
we derive the following:
M (B, Bg BsD,As)
Crasa(k)
R
(T)
Σ
(0
u
T
)|
1=1
-=1
1=1
j-r-k-1
L
(16)
(
12
j=1
7
Clora-
Thus, conclude that Grasa(k) < Glora proving that RaSA achieve equal or lower minimum reconstruction error compared to LoRA.
3.2 EMPIRICAL ANALYSIS
While the previous theoretical analysis guarantees that RaSA can at least match the MRE of LoRA, does not quantify how much RaSA improves upon LoRA. To provide more intuitive understanding of how RaSA achieves lower reconstruction error, we turn to an optimization based analysis using coordinate descent.
4

---

Published as a conference paper at ICLR 2025
RaSA
LoRA
Reconsturction Error
320000
Reconsturction lirror
230000
Reconsturtion Error
400000
312500
220000
390000
305000
210000
380000
297500
200000
370000
290000
190000
360000
lteration
Iteration
Iteration
(a) W
(b) Woowe
(c) gute
Reconsturctin Error
64000
Reconsturction Error
10000
61500
8750
9500
Reconsturction
$9000
8000
9000
56500
7250
8500
$4000
6500
8000
0
10
20
30
40
50
0
10
20
30
40
50
0
10
20
30
40
50
Iterution
Iteration
Iteration
(d) W
(c) W,
(8 W
Figure 2: Reconstruction error curves of RaSA (r (r= 8, 1) during coordinate descent. We also
plot the minimum reconstruction error of LoRA (Equation (9) for comparison.
Empirical Validation Specifically, we instantiate the set of high-rank matrices {Mr hetz) with the actual weight updates from model fine-tuning: ( and iteratively minimize the recon- struction error in Equation [1] by adjusting the parameters of RaSA (r 8, 1), namely the B., A. Bg, As. and D; (details can be found in appendix A We apply this procedure to various kinds of linear modules within Llama-3.1-88 until convergence, and compute étora using Equation (9) as baseline values.
Figure shows that RaSA requires ~0 iterations to achieve a significantly lower reconstruction error than LoRA's minimum. This pattern is consistent across all linear modules in the model, demonstrating the enhanced expressive capacity RaSA
Selection of l: RaSA introduces only one additional ranks are taken from each layer to be shared across all layers. When & O, RaSA reduces to LoRA, where no ranks are shared. On the other hand, when k = r, RaSA shares all ranks across layers, eliminating layer-specific low-rank updates and making the adaptation fully shared. While this maximizes the effective rank of update, it may diminish layer diversity and the ability to capture layer-specific nuances. We traversed from 0 to 8, and presents the converged reconstruction error from the previous coordinate descent experiment in Figure 3 The results indicate that small value of k. around r/8, achieves the minimum error. Further increasing & lead to a rise in reconstruction error, even exceeding that of LoRA. This finding also indicates that some current methods that share all ranks across layers, such as VeRA (Kopiczko et al.) 2024) and Tied-L (Renduchintalal et al. 2024), might be sub-optimal and challenging to be optimized.
which controls how many
4
EXPERIMENT
4.1
SETUP
Tasks Our experiments generally align with those reported by Biderman et al. (2024). We applied all the methods to instruction fine-tuning and evaluated their performance on challenging tasks: code generation and mathematical reasoning. While Biderman et al. (224) use Humaneval (Chen et al.) 202T) and GSMSK (Cobbe et al. (022)) as test sets, these two benchmarks have become saturated with the rapid growth of L.MMs. To provide more rigorous evaluation, adopted two challenging
5

---

Published as a conference paper at ICLR 2025
RaSA
LoRA
Roconstruction Eivor
155000
Recon struction Eiror
84000
Roconstruction Ervor
30900
150000
81000
30600
145000
78000
30300
140000
75000
30000
2
k
(a)
(b) 16
(a) 32
Figure 3: Reconstruction error comparison between RaSA and LoRA as a function of the shared rank parameter k. We also plot the minimum reconstruction error of LoRA (Equation (9) for comparison. The results are average across all linear modules in the model.
benchmarks as test sets. Since these two benchmarks lack validation sets, in addition to reporting the results from the last checkpoint, we also report the best results reference for the upper bound of each method. Prompt templates for evaluation are provided in appendix B
10k (Wei al. 2024) as the training data, a collection of programming question-answe (QA) pairs, which is reproduced and decontaminated version of WizardCoder (Luo et al., 2024). We used Humaneval+ (Liu et al. (023) as the test set, extension of the Humaneval benchmark that scales the number of test cases by 80 We used the Bigcode Evaluation Harness (Ben Allal et al.) 2022) as the evaluation tool, sampling 50 solutions per problem with temperature of 0.2, and report both Pass and Pass 10.
Code Generation: We used
Mathematical Reasoning: We used MetaMathQA (Yu et al. 2024) the training data, which comprises 395K QA pairs derived from the training sets of GSM8K (Cobbe et al.) 2021 and MATH (Hendrycks et al. 202I), rewritten by GPT-3.5. We used MATH (Hendrycks et al. 1021 as the test set, which consists 5K competition-level mathematics problems covering subjects and 5 diffieulty levels. We followed the evaluation protocol from LLMs Evaluation Harness Gao
et al. 2024), using sympy to verify correctness and employing greedy search for generation.
Baselines
We compare RaSA to the several representative PEFT methods:
LoRA (Huet al.
2022 that learns only a low-rank perturbation to the pretrained weight matrix.
MoRA
Jiang et al.
2024 that uses block diagonal matrices instead of low-rank matrices.
VeRA (Kpiczkoo et al. 2024) that fully shares low-rank matrices across all layers with layer- specific weighting, and freeze the low-rank matrices during training to achieve extreme parameter efficient fine-tuning. Therefore, VeRA can set higher rank-r than LoRA.
LLMs & Training Details We conducted experiments two open-sourced LLMs: Llama-3. 8B (Dubey et al. (0244 and Mistral-0.3-7B (Jiang et al. (003). Following common practice (Kopiczko) al. [024 Jiang al. 2024), used pre-trained models rather than instruction-tuned ones. We applied PEFT's on all linear modules from attention (W,, W.. W., W.) and feed-forward networks (W up: W down: W gate We set the model syper-parameter based on the optimal configurations from Biderman et al. 2024), employing the decoupled LionW optimizer with batch size of 192, and training for 8 epochs with learning rate of 5e-4 by default. For RaSA, we k max(r (8, 1)
based on the analysis in
3.2 More details are provided in appendix
4.2
MAIN RESULTS
In this section, we compare RaSA and baselines on two challenging domains - code and math.
Code Generation Table presents the results the Humaneval+ test set. We compare RaSA and prior LoRA variants in terms of both efficiency and effectiveness. Although VeRA adds only 1.6M extra parameters for 1024, it results in training time increase of between 13% and 16%

---

Published as a conference paper at ICLR 2025
Table Performance on the code generation task (i.e. Humaneval+). We italicize the best result for each rank, and bold the best result for each model. We also present training cost for each setting in terms of trainable parameters (# Trainable Param.) and training time (Time). Note that for MoRA and RaSA, r does not correspond to the effective rank of the update matrix.
Llama-3.1-BB
Mistral-0.3-7B
Trainable Param.
r
Method
PASS@
PASS@ 10
PASS@1
PASS@ I0
Time
Time
BEST
LAST
BEST
LAST
BEST
LAST
BEST
LAST
1024
VeRA
1.6M
11.3h
48.8
48.8
66.5
64.2
12.5h
42.5
39.5
57.3
54.4
LoRA
21.0M
9.6h
56.1
53.0
71.2
68.5
10.7h
42.6
39.7
57.7
54.8
MoRA
21.0M
12.0h
54.6
52.1
68.4
66.9
13.4h
45.2
38.6
64,4
48.6
RaSA
21.0M
11.2h
57.9
56.9
72.6
69.6
12.1h
50.0
49.0
66.0
64.2
LoRA
41.9M
9.8h
54.5
53.4
68.9
67.6
10.7h
46.0
40.6
61.2
54.9
16
MoRA
41.9M
12.7h
56.3
52.9
69.5
65.6
14.0h
43.4
41.0
59.4
56.0
RaSA
42.0M
11.2h
57.3
56.4
72.1
68.1
12.1h
53.6
51.3
68.5
63.7
LoRA
83.9M
10.0h
57.9
56.9
69.8
69.2
10.8h
50.2
44.4
64.4
57.0
32
MoRA
83.9M
12.4h
55.6
53.0
69.0
68.3
14.0h
42.2
42.2
56.4
56.0
RaSA
83.9M
11.5h
59.5
56.2
72.5
71.4
12.5h
55.7
55.7
70.0
65.7
LoRA with 32. VeRA, however, is the least effective among all the variants due to its extreme strategy for parameter efficiency. Both MoRA and RaSA add a comparable number of additional parameters LoRA, yet MoRA requires more time due to the use of block diagonal matrices.
In terms of model performance, MoRA shows performance on with LoRA, aligning with the findings reported in the original paper (Jiang et al. 2024). Our proposed RaSA surpasses all baseline models in nearly all scenarios. Like LoRA, RaSA's performance improves with rank, and at rank 32, RaSA typically delivers the strongest performance for both the Llama and Mistral models. RaSA achieves maximum Humaneval+ of 59.5% PASS@ @ with Llama-3.1-88
Table Performance on mathematical reasoning task (i.e. MATH). We also present extra parameters (# Extra Param.) used by AdaLoRA and PriLoRA for estimating parameter importance.
Llama-3.1-8B
Mistral-0,3-7B
Trainable Param.
Extra Param.
Method
ACC
ACC
Time
Time
BEST
LAST
BEST
LAST
7-8B
35.5
34.6
28.1
26.6
1024
VeRA
1.6M
22.4h
27.4
25.6
17.6h
19.9
19.4
LoRA
21.0M
20.1h
28.3
20.1
MoRA
21.0M
23.66
29.2
28.9
19.6h
21.4
21.4
OLoRA
21.0M
29.3h
28.4
27.8
31.1h
22.5
22.5
AdaLoRA
31.5M
63.0M
27.9h
30.4
28.9
17.7h
22.5
21.6
PriLoRA
21.3M
24.1h
22.3
RaSA
21.0M
23.8h
30.3
29.1
15.9h
24.3
23.8
LoRA
41.9M
20.2h
28.8
27.1
14.7h
20.9
19.5
16
MoRA
41.9M
24.5h
30.2
26.5
21.1h
20.5
19.4
OLoRA
41.9M
29.6h
28.6
28.4
35.9h
22.5
22.2
AdaLoRA
125
29.7
17.6h
23.5
PriLoRA
42.6M
21.3M
24.5h
28.6
29.4
15.9h
22.7
21.6
RaSA
42.0M
24.4h
31.4
29.8
15.8h
25.9
25.1
LoRA
83.9M
20.6h
28.9
27.2
14.8h
21.8
20.4
32
MoRA
83.9M
24.7h
28.6
25.8
20.5h
18.4
OLoRA
83.9M
31.3h
23.8
23.5
AdaLoRA
125.9M
251. 8M
28.7h
30.2
29.4
17.9h
23.5
23.5
PriLoRA
85,2M
42.6M
24.5h
30.0
28.8
16.4h
24.8
24.2
RaSA
31.7
16.5h
26.1
25.1
Mathematical Reasoning
We add FFT and more PEFT baselines in math task:
7

---

Published as a conference paper at ICLR 2025
OLoRA Büyükakyüz
2024 that uses QR decomposition initialize the LoRA adapters.
AdaLoRA
Zhang et al. 2023 that dynamically allocates ranks among parameter matrices.
PriLoRA Benedek & Wolf 2024) that allocates different rank for each layer in an increasing manner, and performs pruning throughout the training process.
The math results presented in Table FFT outperforms all PEFT methods, aligning the findings
from Biderman et al. (2024). Considering both training cost and accuracy, RaSA demonstrates consistent superiority over all PEFT baselines across various configurations. Mistral notably falls short of its Llama counterpart, exhibiting a performance deficit of approximately 8%, which RaSA is capable of narrowing down 5%. We also observe that directly increasing the hyper-parameter yields only marginal performance gains, but at the cost of doubling the number of training parameters. contrast, RaSA greatly outperforms LoRA with the same or even fewer parameters (RaSA, , > LoRA, This supports the notion introduced in $ [ that LoRA's parameters are underutilized. RaSA, on the other hand, improves the utilization parameters by sharing them across layers.
4.3 RASA LEARNS MORE AND FORGETS LESS THAN LORA
O
LoRA (m-)
LoRA (1-16)
(r
D RaSA (m-8)
RaSA (m)))
RaSA (2=
Training Loss
Training Loss
0.6
0.12
0.5
0.08
0.3
0.04
2
3
4
5
6
7
8
2 3
4
5
6
7
Epoch
Epoch
Epoch
Epoch
(a) Lluma-3.1-88
Code
(b) Mistral-0.3-7B Code
(c) Llama-3.1-88 Math
(d) Mistral-0.3-7B Math
Figure 4: RaSA learns more and faster than LoRA. Training curves of LoRA and RaSA with different ranks. RaSA consistently outperforms LoRA with the same rank across models and tasks.
RaSA learns more and faster than LoRA Fllusttts the training curves of the fine-tuning process. Generally, training losses for both RaSA and LoRA decrease as the rank increases. Notably, RaSA consistently outperforms its LoRA counterpart in terms of both learning effectiveness and efficiency across all cases, aligning with our empirical analysis presented in Section 3.2 These results collectively underscore the efficacy and universal applicability of the proposed RaSA method. One interesting finding is that RaSA is specifically effective for Mistral: RaSA achieves comparable potentially superior training outcomes LoRA with significantly lower rank requirement of compared to LoRA's rank of 32.
O Lr
(F
(r D RoSA (r-8) 8)
RaSA (6
RoSA 32)
70
Training Loss
Accuracy
60
55
55
50
2
4 5 6
8
1 2 3 4
5
6
7
2
3
4
5
6 7
2
6
8
Epoch
Epoch
Epoch
Epoch
(a) Llama-3,1188 Code
(b) Mistrul-0.3-7B Code
(c) Llama-3.1-8B Math
(d) Mistral-0.3-7B Math
Figure 5: RaSA forgets less than LoRA. Y-axis shows the average of prediction accuracy three
benchmarks to evaluate model's forgetting. Higher prediction accuracy denotes less forgetting.
8

---

Published as a conference paper at ICLR 2025
RaSA forgets less than LoRA We follow Biderman et al. (2024) to investigate the extent of forgetting as degradation of base model capabilities. Specifically, y. we calculate prediction accuracies on the following three benchmarks: (1) HellaSwag (Zellers et al., 2019); inference the most plausible continuation for daily events (70K problems); (2) WinoCirande (Sakaguchi et al. 2019): assesses commonsense reasoning (44K problems); (3) ARC-Challenge (Clark et al. 2018): complex reasoning and understanding of scientific concepts (7.8K problems).
Figure presents the averaged forgetting curves of the three benchmarks, clearly showing that RaSA experiences less forgetting than LoRA, with RaSA's forgetting levels being less affected by rank changes compared to LoRA. The difference in performance between r and r 32 epoch 8 stands at an average of 2.83% for LoRA and 0.75% for RaSA, indicating a smaller performance variation for RaSA. LoRA more prone to forgetting in math than code, while RaSA displays greater domain robustness. Specifically, with = 32, Mistral scores 58.7% in code and 52.8% % in math using LoRA, whereas RaSA shows a reduced performance difference between code (64.6%) and math (66.5%) domains, underscoring RaSA's robustness.
4.4 SCALING PERFORMANCE ANALYSIS
This section investigates the scaling characteristics of the RaSA approach by varying both the model size and the dataset size to assess its robustness.
Model Scaling Initially, we evaluate RaSA's performance on an expanded scale by examining larger-scale models, including Llama-3.1-70B and Mixtral-8 7BB Due to computational con- straints, employ rank of r 4 for both models specifically in the domain of mathematical reasoning. For an equitable com- parison, we present results for smaller models configured with
LoRA RaSA
MATH Accuracy
25
= 4. Each model is trained over 2 epochs using both LoRA and RaSA techniques, with performance measured terms of LAST accuracy. The results in Figure
15
Llama-SB
Llama 70B
Mistral-7F
Mixtral-8x7h
O reveal that increasing the model size substantially enhances performance for both LoRA and RaSA, across all model types. Noteworthy is the performance of larger Llama and Mistral models using LoRA, achieving MATH accura- cies of 40.4% and 32.6%, respectively. These results significantly exceed those of their smaller counterparts under identical configurations and even surpass outcomes from variants with extended training (i.e., 8 epochs). Notably, RaSA consistently outperforms LoR on these larger-scale models, underscoring RaS A'S robustness in handling models of increased scale.
Figure 6: MATH performance of scaled models.
Data Scaling Subsequently, we explore the influence of training data size on RaSA's performance. We experiment with the Llama-3.1-881 model, applying a rank of r = 8 to facilitate efficient training. The examination involves random sampling of 25% and 50% instances from the SFT data for the mathematics reasoning task. Each model is trained over epochs, with performance assessed through the LAST accuracy. As illustrated in Figure 7 LoRA's performance seems contingent on the volume of training data, with no noticeable improvement when data is increased from 25% to 50%. This finding is consistent with the results in Biderman al. (2024). In contrast, RaSA demonstrates remarkable ability to enhance performance with increase in training data volume. Impressively, with just 25% of the training data, RaSA outperforms LaSA even when the latter utilizes the entire dataset, high- lighting A'SA's exceptional efficiency in leveraging training data for performance improvement.
30
MATH Accuracy
LoRA RaSA
28
28.1
26.7
2
26
24.9
24
25%
50%
100%
Training Data
Figure 7: MATH perfor- mance of scaled data.
RELATED WORK
Parameter-Effiee Fine- Fine-Tuning (PEFT) PEFT methods aim to minimize the number trainable parameters needed for fine-tuning large models, thus reducing memory and computational require-

---

Published as a conference paper at ICLR 2025
ments. Pioneering methods include adapter-based (Houlsby et al. 2019) and prompt based (Lester et al. 2021 Li & Liang 2021) that introduce additional tunable adapter or prefix tokens to enable efficient fine- tineituning while keeping the original model parameters fixed. However, these approaches can slow down inference speed due to the extra components introduced. LoRA overcomes this drawback by introducing low-rank matrices directly into the weight update process during fine-tuning effectively reducing trainable parameters without increasing inference latency. Due to its robust performance. LoRA and its variants have been widely used to adapt LLMs for specific tasks (Yu et al. 2024 Xu et al., 2023 Biderman et al. [0224] Chen et al. 20244] Meng et al. 2024: yang Liu al. 2024). Benedek Wolf (2024) and Zhang et al. (2023) show that the number of ranks required for each parameter matrix across the model's 's layers is not uniform. Therefore, they propose dynami- cally assigning ranks based on the importance of parameters during training. These rank-allocating approaches typically involve real-time estimation of parameter importance and pruning during the training process. In contrast, RaSA uses a shared rank pool combined with layer-specific weighting, climinating the need for complex importance estimation pruning. [iderman et al. (2024) conduct a comprehensive empirical study on LoRA, and reveal that while LoRA still lags behind FFT, exhibits less catastrophic forgetting. We show that our proposed RaSA forgets even less than LoRA, and learns more and faster.
Parameter Redundancy of LoRA Although LoRA has significantly reduced the number of trainable parameters, recent research suggest that it is possible to further minimize these parameters without compromising performance. KKopizko et al. (2024) achieve 99% reduction in LoRA parameters by fully sharing a pair of low-rank, frozen random matrices across all layers, adjusted with learnable scaling vectors. Koohpayegani et al. (2024) propose learning linear combinations of set of random matrix bases, while Li ct al. (2024) push this further by replacing the matrix bases with a vector bank. [Song et (2224) and Renduchintala al (2024) explore the effects of different sharing and selective fine-tuning strategies. By sharing parameter spaces, Gabrieson et al. (2024) compress 1,000 LoRAs trained from different task, enabling more efficient serving. These findings collectively suggest that LoRA's parameter has not been fully utilized and that different LoRAs exhibit similarities across layers, modules, and even different tasks. Rather than focusing on extreme parameter reduction, this work aims to maintain the same parameter count while exploring how inter- layer sharing can enhance parameter utilization. We theoretically and empirically demonstrate that sharing ranks across layers leads to lower reconstruction error and thus better expressive capacity.
6 CONCLUSION
In this study, we introduced RaSA, novel extension LoRA through innovative partial rank sharing across layers. RaSA maintains the parameter efficiency and seamless integration into existing models characteristic of LoRA while substantially increasing the model's expressiveness. Through theoretical analysis, we established RaSA's superior capability in matrix reconstruction compared to traditional LoRA, underpinning sults on complex tasks such as code generation and mathematical reasoning have demonstrated its effectiveness over LoRA high-demand scenarios. Future research directions may explore further optimization of rank-sharing schemes and the potential of RaSA in broader range of applications, paving the way for the development of even more powerful and efficient PEFT strategies.
improved performance in downstream tasks. Empirical re-
ACKNOWLEDGMENTS
This paper is supported by the General Program of National Natural Science Foundation of China (62176153), the CCF-Tencee Rhino-Bird Open Research Fund (RAGR20240107), and the Tencent Al Lab Fund (RBFR2024002). The authors gratefully acknowledge the financial support from these funding agencies.
REFERENCES
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https://g thub. com/ 2022.
10

---

Published as a conference paper at ICLR 2025
Nadav Benedek and Lior Wolf. PRILoRA: Pruned and rank-increasine low-rank adaptation. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguis- tics: EACL 2024, pp. 263,263, St. Julian's ss Malta. March 2024. Association for Computational Linguistics.
UtRptttss: htttps://aclantthll..goooo
2024. findings-eacl.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings. Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024.
Rickard Brüiel-Gabrielsson,, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of lora adapters with little overhead, 2024. htttts/s:///aii.oo/////////////e
00066
Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01775, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- gloRA: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
6PMJoRfdak
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. / 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (3):211-218, 1936.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h Haonan Li, Kyle MeDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024. URL httt::////eeodo.orr/errrrr/////
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenct classification. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10. 19//CC.20155....
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790- 2799. PMLR, 09-15 Jun 2019. URL https: http::///ooceeedngs..o mlaaiinnnsmmmm.r....mmmmmm. preesssss//
v97/houlsby19a.hee html
11

---

Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-LoRA: Enhaneing parameter efficiency of LoRA with weight tying. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8694-8705, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 11116653/11/2224.aacc-oooo. Published as a conference paper at ICLR 2025
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen, LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://oppnnreview net nett:forummiddnnneweerrr
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06825, 2023.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang. Haizhen Huang. Furu Wei, Weiwei Deng. Feng Sun, Qi Zhang, Deqing Wang. al. Mora: High-rank updating for parameter-efficient fine-tuning, arXiv preprint arXiv:2405.. / 130, 2024.
Soroush Abbasi Koohpayegani, Navaneet K L, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Compressing lora using linear combination of random basis. In The Twelfth International Conference on Learning Representations, 2024. URL htt.htttts://opereviiee net/ forum?2dd=1fXccDyyz
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreview.. httpr//wwww.eetefooommmm/o/n//fe
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-effiiient prompt tuning. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/11/0221.com https://aclanthol.ggr org/ org/ wwn.l-mminn.n.... 243
main 243.. URL
Xiang Lisa Li and Percy Liang. Prefix Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pp. 4582-45977 Online, August 2021. Association for Computational Linguistiss. doi: 10. 1/226
URL httpttt://aalaathhtto
org/ 201.222211ccc1cc.
353
Yang Li, Shaobo Han, and Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. arXiv preprint arXiv:240.. 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Shirth-seventh Conference on Neural Information Processing Systems, 2023. URL https ///peneeview.net/eeeeee wttp://ooommmmm//m/x611111111111
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevieet wwwp://nwwweeeeeeeeououriiiiiiie
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.0 02948, 2024.
URL/https://accaathhooo http:///w.....g/22/aaaaaa-a----- naacl-lono 481
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019.
Yurun Song, Junchen Zhao, Ian G Harris, and Sangeetha Abdu Jyothi. Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation. arXiv preprint a40iv:2406. 10785, 2024.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL Flo0e2225444eee1eoooo0ooocc1oo3
Guyon,
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with OSS-instruct. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the s International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 0666322-667.. PMLR, 21-27 Jul 2024. URL 2235/wei24h, html
mlr press/
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models, 2023.
Shih yang Liu, Chien-Ys Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024. URL httpp:://pperreiieeeee netp/ierwwee/foorrmu id=3d5CIRG1n2
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelffth International Conference Learning Representations, 2024. URL URL /ttp:::s:///ppeeeveeee eview. http:/ooummiidNNNooaaaee
Rowan Zellers, Ari Holtzman. Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 10t/53//11119111/// 1472. URL httphhss/aaaaaanhlllleeeoo
org/P19-1472
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL forum?id=likkjjmha
net/
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, In The Eleventh International Conference on Learning Representations, 2023. URL, forum?id==162uuMJJi.
13

---

Published as a conference paper at ICLR 2025
A COORDINATE DESCENT EXPERIMENT
This section details the derivation of the coordinate descent experiment discussed in 3.2 inspired by Brbbie-Gan et al. (2024).
Given the parameters of RaSA: defined as:
[B,, A., Bs: Di, (s])L)) the reconstruction error of RaSA is
E Σ M. (B.A, BsD.As)
(17)
Clearly, B; and A; are independent across layers. By applying the Yokart-Young-Mirkky theo- rem (Eekart & Young 1936), we first compute the SVI) of the residual matrix:
SVD(M, BsD,As) -UV
(18)
Therefore, the update rules for B, and A, are:
3 B, = Upr-rK k) 2 for
(19)
Á, Σ (r-kk v((i--))*
Let the low-rank decomposition of M, be: M; = B,A,, where B, E R and A, E Rdxxa Next, we compute the following gradients:
B.A (B.., D
VB E
(20)
i=1
1
VAg = Σ -2 B.Á (B.A. +BsDDsss)s) BsD.,
(21)
i=1
D = -2
B.A. - (B.A, AS.
(22)
V (D.) E E).
(23)
By setting these gradients to zeros, we obtain the following update rules:
Bs
B
BB]]
A
A.D.
Σ D.ASAAD
(24)
A
]
T
As
-B:l
[ {
BsD.
[ D. BsD;
(25)
A As
diag(D) Bs AsA
B¹ BB
[B.
- B,
1.
(26)
Á
In coordinate descent, we iteratively apply Equations and 24 to (66 until convergence.
14

---

Published as a conference paper at ICLR 2025
B PROMPT TEMPLATES
Below is an instruction that describes a task. Write response that appropriately completes the request.
### Instruction:
(QUESTION)
### Response:
Let's think step by step.
Figure 8: Evaluation prompt for mathematical reasoning.
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
[QUESTION]
### Response: (IMPORT SECTION)
(FUNCTION SIGNATURE) (DOCSTRING)
Figure 9: Evaluation prompt for code generation.
C TRAINING AND DATA DETAILS
Training We mostly aligned our training configurations with the optimal configurations from Biderman et al. (2024). For LoRA, we used the decoupled LionW optimizer with a batch size of 192, training for epochs with learning rate of 5e-4. A cosine learning rate scheduler
applied, with the first 10% of training steps used for warmup, and weight decay set to zero. Training and evaluation were conducted using bíloat16 precision. While Biderman et (2024) set 32 for both math and code tasks, we reduced a to 8 for the math task due to convergence issues observed with the Mistral model when using a 32. RaSA training fully inherits all seper-parameters from LoRA training. For MoRA, we used a learning rate of 3e-4, reported in the original work (Jiang et al. 2024). For VeRA, following the original paper, we the learning rate 10 times that of LoRA. resultngg in 5e-3 (Kopiczko et al.) 2024). All experiments for the 7-8B models were conducted on node 8 x A A100-40G GPUs. For the 70B and MoE models, we used 8 nodes.
Data During training, grouped data by length, which significantly accelerated the training process. All math training data ends with "The answer is: {ANSWER]" helping answer extraction during evaluation.
15


Reviews:
[{"review_id": "NAGYOqjgMc", "reviewer": "Reviewer_7i6n", "summary": "The paper introduces an enhancement to existing fine-tuning methods for large language models (LLMs). It builds on Low-rank adaptation (LoRA), commonly used for efficient fine-tuning, but limited by its low-rank constraint in handling complex tasks like code generation and mathematical reasoning. To address this, the authors propose Rank-Sharing Low-Rank Adaptation (RaSA), a novel method that increases expressive capacity by sharing ranks across layers, allowing for more flexibility without adding parameter overhead. RaSA thus extends LoRA’s efficiency, achieving substantial performance gains in rigorous tasks. Code and data resources accompany the work.", "questions": "1. Could the efficiency of the proposed method be further optimized to reduce computational time?\n\n2. The performance improvement appears to be notably greater on Mistral-0.3-7B compared to Llama-3.1-8B. Do you have any insights into the reasons behind this difference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Technical Soundness: RaSA is a technically robust extension of LoRA, overcoming its expressive limitations through partial rank sharing. Theoretical and empirical results confirm its lower reconstruction error and improved task performance.\n\n2. Paper Presentation: The paper is clearly organized and easy to follow, with well-explained motivations, methods, and contributions, making RaSA's innovations accessible and engaging.\n\n3. Theoretical Analysis: Theoretical bounds and empirical evidence validate RaSA’s improved capacity over LoRA, providing a solid foundation for its enhanced performance in high-rank tasks.", "weaknesses": "1. Lack of Baseline Comparisons:\nThe paper lacks a comprehensive comparison with other LoRA variations, such as EM-LoRA, AdaLoRA, and Orthonormal LoRA. Without these baselines, it’s difficult to fully assess the advantages of RaSA, as the performance improvements over the original LoRA may not necessarily hold when compared with these alternative methods.\n\n2. Efficiency Concerns:\nThe proposed method is considerably less efficient than the original LoRA, with a noticeable increase in computational time (at least 1.4 hours). Despite this added overhead, the performance gains are minimal, with PASS@1 improvements generally below 2%. This limited payoff questions the practical viability of RaSA, especially for real-world applications where computational efficiency is crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwvjwYntRh", "reviewer": "Reviewer_SwqF", "summary": "This paper highlights that the low-rank constraint limits the expressive capacity of LoRA, and that LoRA’s parameters are not fully utilized. To address this, the authors propose RaSA, an approach enabling partial rank sharing across layers with minimal additional parameters. Theoretical and empirical analyses demonstrate that RaSA achieves a minimum reconstruction error bounded by LoRA, and can be easily optimized. Experiments show that RaSA achieves great performance in complex math and code tasks.", "questions": "- In line 103, the authors claim that introducing a trainable diagonal matrix enables layer-specific weighting, actually, I am confused about this, maybe it could be explained specifically.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- RaSA can efficiently increase the rank of the original LoRA, and compared to directly expanding the rank, RaSA achieves this with almost no increase in parameter count.\n- The authors validate the better expressive capacity of RaSA through detailed reconstruction error analysis, empirical analysis shows that RaSA can achieve much lower reconstruction error and reveal the relationship between rank and the hyperparameter k.\n- Experiments on code and math tasks show that RaSA achieves much better performance with only a 10% increase in time, highlighting RaSA's promising advantages.", "weaknesses": "- As written in Lines 30-31, LoRA still lags behind full fine-tuning, particularly on complex tasks, however, the authors only compare RaSA with existing peft methods, not including full fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjjMLuj604", "reviewer": "Reviewer_bzbg", "summary": "Low-rank adaptation (LoRA) is limited in expression ability due to low rank constraints. This paper propose Rank-Sharing Low-Rank Adaption (RaSA) to enhance the expressive capacity of LoRA by leveraging partial rank sharing across layers. Specifically, RaSA shares $k$ ranks across all layers and assign $r-k$ for layer-specific parameters. After that, RaSA greatly increase the effective rank of the parameter update by $(L-1)\\times k$ without introducing additional parameters. The experimental results on multiple datasets with different model architectures validate the effectiveness of the proposed approach.", "questions": "I am curious why the performance is best when $k$ is around $r/8$ (line 249-250), and if possible, is there any theoretical explanation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The problem studied in this paper is valuable. \n2. The proposed RaSA approach is interesting and effective.\n3. The paper is well organization. \n4. The theoretical and empirical analysis are insightful. \n5. The experiment is sufficient, including the study of Reconstruction error, the study of $k$, the performance comparison of parameters and time, the study of forgetable.", "weaknesses": "A great paper with almost no weaknesses. A small suggestion, the distinction between the different methods in Figures 4, 5, and 7 is only based on different color (red and blue), as printing on paper cannot distinguish them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0vnYdUHN9O", "reviewer": "Reviewer_tu4z", "summary": "The paper presents a method to improve the expressivity of LoRA, while keeping the same number of parameters. This is by sharing a portion of each learnable adaptor across all layers.", "questions": "* In Figure 1b, the name of matrices A,B are in the square, and the dimension is outside. However, in matrix D the name is outside and there are no shapes. Maybe you want to consider putting the name D_i inside, and adding shape, for readability.  \n* In line 208, it says: ‘actual weight updates from model fine-tuning’. Do you mean ‘full fine-tuning’ (FFT)?\n* In Table 1, why do you show both LAST and BEST? Why does BEST not suffice?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The authors tackle an important and meaningful topic that would be of interest to the community. PEFT is gaining more and more attention as the size of language models continues to grow.\n* The description of the proposed method is clear and concise. Writing is easy to understand and follow.\n* The method is simple, effective, and easy to implement.", "weaknesses": "* It seems that unlike vanilla LoRA, that acts independently on each layer, in RaSA there is an underlying assumption that all shared layers have the same dimension. If so, it is worth mentioning it explicitly in the paper.\n* There are LoRA extensions that allow allocating a different rank for each layer, according to its importance (for example PRILoRA and others). It seems that in RaSA the rank should be similar across layers. If so, it is worth mentioning this fact and citing the relevant papers that do allow different rank allocation.\n* There are no comparisons in Table 1 to other LoRA extensions (AdaLoRA, PRILoRA), which may be of importance to the reader. Furthermore, it is not clear why the VeRA method is compared against, when its number of parameters is x13 less.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)