BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Authors: Yuhua Zhou
Ruifeng Li
Changhai Zhou
Fei Yang
Aimin PAN
Venue: ICML 2025 poster (2025)
Keywords: Parameter-efficient fine-tuning
parameter-sharing
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models (LLMs) to adapt to downstream tasks. However, in scenarios where multiple LoRA models are deployed simultaneously, standard LoRA introduces substantial trainable parameters, resulting in significant memory overhead and inference latency, particularly when supporting thousands of downstream tasks on a single server. 
While existing methods reduce stored parameters via parameter sharing, they fail to capture both local and global information simultaneously. To address this issue, we propose the Bi-Share LoRA (BSLoRA), which extends local LoRA with intra-LoRA and inter-LoRA parameter sharing to better capture local and global information. This approach reduces trainable parameters while maintaining or even enhancing model performance. Additionally, we design three transformation methods to improve the compatibility and collaborative efficiency of shared parameters with varying shapes, enhancing overall adaptability.
Experiments on the 7B, 8B, and 13B versions of Llama show that BSLoRA, with only 44.59% of the parameters of standard LoRA,  outperforms LoRA by approximately 0.33% on commonsense reasoning and 2.08% on MMLU benchmarks. Code is available at https://github.com/yuhua-zhou/BSLoRA.git.

BSLoRA:
Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Yuhua Zhou 12
Ruifeng Li 12 Changhai Zhou
Fei Yang?
Aimin Pan
Abstract
1. Introduction
Large language models (LLMs), such as GPT-4o (Openai, 2023) and Claude-3 (Anthropie, 2024), have recently shown remarkable generalization capabilities across wide range of natural language tasks (Raiaan et al., 2024; Chang et al., 2024; Zhang et al., 2023), primarily due to the increase in model parameters. For example, GPT-3 (Brown et al., 2020) has 175 billion parameters, while the largest ver- sion of Llama 3 (Touvron al., 2023) reaches 405 bil- lion. However, the ever-increasing size of these models presents significant challenges for fine-tuning, full pa- rameter fine-tuning becomes computationally expensive and memory-intensive. To address this issue, Parameter- Efficient Fine-Tuning (PEFT) methods have been intro- duced, achieving performance comparable to full fine-tuning by adjusting only a small subset of parameters (Han et al., 2024) while keeping the majority of the model parameters frozen. Among these methods, LoRA (Hu et al., 2022) stands out by approximating parameter updates using the product of two low-rank matrices and has gained increasing popularity.
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models (LLMs) to adapt to downstream tasks. However, in scenarios where multiple LoRA models are deployed simultane- ously, standard LoRA introduces substantial train- able parameters, resulting in significant memory overhead and inference latency, particularly when supporting thousands of downstream tasks a single server. While existing methods reduce stored parameters via parameter sharing, they fail to capture both local and global information simul- taneously. To address this issue, we propose the Bi-Share LoRA (BSLoRA), which extends local LoRA with A A and inter-LoRA parame- ter sharing to better capture local and global infor- mation. This approach reduces trainable parame- ters while maintaining even enhancing model performance. Additionally, we design three trans- formation methods to improve the compatibility and collaborative efficiency of shared parameters with varying shapes, enhancing overall adaptabil- ity. Experiments on the 7B, 8B, and 13B ver- sions of Llama show that BSLORA, with only 44.59% of the parameters of standard LoRA, out- performs LoRA by approximately 0.33% on com- monsense reasoning and 2.08% on MMLU bench- marks. Code is available at https: //g thub. com/ com/yuhua- yuhuauuuaaaa caceeehhhuuueee Bt////LLLLAAL..
However, as model parameters grow, LoRA fine-tuning of LLMs still introduces a large number of additional parame- ters, even with low rank. For example, using LoRA rank of 64 on Llama 70B adds approximately 360 million pa- rameters (1.4GB of memory), increasing training difficulty, which worsens when multiple LoRA services are deployed simultaneously (Wang et al., 2024). Additionally, LoRA parameters consume significant memory during inference, which increases loading and task-switching latency. There- fore, reducing LoRA's trainable parameters has become an urgent necessity.
Several existing methods aim to reduce LoRA's parameters through parameter-sharing (Mao et al., 2024: Sun et al., 2022b), with inter-layer sharing proving effective in reduc- ing redundancy across different layers. For example, VeRA (Kopiczko et al., 2024) and VB-LoRA (Li al., 2024) lever- age inter-layer sharing to reduce memory and computation costs while capturing global patterns. However, these meth- ods often overlook the local information and intra-layer parameter redundancy (Lin et al., 2024). In Transformer models, for instance, the attention heads and forward
'Equal contribution 'College of Computer Science and Technology, Zhejiang University "Zhejiang Lab "School of Computer Science, Fudan University. Correspondence to: Aimin Pan <panaimin babeiiiagglab.org>,, Fei Yang phejingllii.ngl...o.g....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
networks in the same layer often process similar features, leading redundant parameters. Therefore, there is need to design new sharing technique that captures both local and global features, while ensuring the shared parameters can be adapted to all modules with different shapes.
2. Background and Motivation
2.1. Low-Rank Adaptation
LoRA fine-tuning is employed to recover performance with minimal parameter updates. For an LLM consisting of I layers, the weight matrix of each layer W is adjusted using an update matrix AW RRmx,, This matrix is factorized into two low-rank matrices, A and B, where A E grxm and B E R with r being a hyperparameter shared by all layers. The effectiveness of fine-tuning is highly dependent on rank selection. In this approach, the original weight matrix W remains frozen, while only A W, represented by the product AB, is updated. The forward computation can be expressed as:
In this paper, we analyze LoRA parameters and identify a high degree of redundancy (Figure 1). Meanwhile, our design follows MultiLoRA (Wang et al., 2023), which demonstrates that multi-LoRA structure can yield better results. Preliminary studies (Table 5) further confirm that intra-layer and inter-layer parameter sharing can reduce pa- rameter count while maintaining model performance and effectively capturing global information. Accordingly, we propose Bi-Share LoRA (BSLoRA), a Multi-LoRA archi- tecture that extends LoRA parameters into multiple smaller LoRAs to enhance fine-tuning expressiveness, where two smaller LoRAs are shared within and across layers to re- duce parameters. Specifically, we decompose the LoRA matrices into three components: local parameters, which capture module-specific information; intra-layer shared pa- rameters, which are shared within the same layer to capture local consistent features; and inter-layer shared parameters, which are shared across layers to capture global patterns. This enables BSLoRA to learn both local and global in- formation efficiently. Additionally, to tackle the challenge of adapting shared parameters to all modules with differ- ent shapes, we present three shape transformation methods: Slice Sharing, Gate Transformation, and Kronecker Exten- sion. To validate the effectiveness of BSLORA, we conduct extensive experiments on the Llama model family across multiple commonsense reasoning and MMLU benchmarks. Our results demonstrate that BSLoRA achieves significant parameter savings of about 50% while maintaining or even improving the model's performance compared to standard LoRA and other existing methods. We also conduct experi- ments to analyze the benefits of rank value and contributions of different configurations for local and shared weights. In summary, our contributions are as follows:
[(x) = (W + AW) Wa + BAæ
(1)
Given that the rank r is typically much smaller than the dimension d. the computational cost is significantly reduced from de to 2dr. This optimization can reduce the trainable parameters during the learning process. Typically, the matrix A is initialized by a Gaussian distribution with small standard deviation, and B is initialized as zero matrix. Hence, at the beginning of fine-tuning, the model behaves identically to the pre-trained model.
2.2. Motivation
In large language models (LLMs), parameter redundancy is a common issue. Redundancy commonly occurs within the same transformer block where the attention layer and MLP layer have overlapping functions or learn similar patterns (Lin et al., 2024), and the parameters across different blocks where similar feature representations might be learned in multiple layers (Li et al., 2024). We also plot the parame- ter similarity in Figure 1, which shows the high similarity across different modules (details refer to Appendix A.2). Existing methods aim to address the parameter redundancy through simple parameter sharing techniques (Kopiczko et al., 2024; Li et al., 2024; Song et al., 2024). However, our preliminary experiments (results shown in Table 5 (Indi- vidual)) indicate that simple intra-layer or inter-layer shar- ing may result in some performance degradation. It may need modulc-specific parameters to learn the local features. Therefore, we consider introducing local parameters com- bined with shared parameters to fine-tune the LLM.
We propose BSLoRA, a unified sharing method that combines local parameters with intra-layer and inter- layer shared parameters to effectively capture both lo- cal and global information. This approach significantly reduces the number of trainable parameters while main- taining performance.
Inspired by Wang et al. (2023) and Tian et al. (2024), which identified that decomposing a large LoRA weight into mul- tiple LoRA modules in parallel can enhance the model's adaptability and flexibility. Therefore, we intend to decom- posed the entire set of LoRA parameters into three smaller LoRA parameter blocks: the first part acts on individual modules, the second part is shared among modules within the same layer, and the third part is shared by all modules.
We introduce three shape transformation techniques to handle varying parameter shapes, thus increasing the flexibility and effectiveness of parameter sharing.
We conduct extensive experiments on multiple tasks, demonstrating the effectiveness of BSLoRA in reduc- ing parameter redundancy and improving parameter efficiency.
2

---

BSLoRA; Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
thereby improving the coherence of the information pro- cessed within each layer.
Average Similarity Within Layers
2.00
1.03
Inter-Layer Module denotes global parameter sharing, where the same LoRA update matrix is shared across dif- ferent layers of the transformer. This facilitates better infor- mation flow and feature interaction between layers, which in turn enhance the overall expressiveness and depth of the model's representations.
Similartty
6.9
0.90
Layer
Local Module refers to the traditional LoRA configura- tion, where the LoRA update matrix is applied only the current module, allowing the model to learn highly specific local features. During training and inference, the parame- ters for each module are updated according to the following equation:
Similarity Across Layers
12 10
Layer Index
20 28 26 24 22 20 10 16
[(x) Wx (BA) (BA) iutro x (BA)
(2)
where the distinct rank values of the aforementioned three sub-LoRA matrices are denoted as Tlocals Tintras and Tinters respectively. Specifically, Blocal
10 32
18 20 22 25 Layer Index
0.55
Rtocala Alocal
I Rricett B Bintra E IRnXTantro, Aintra E B Binter E RnxXiinter and Ainter E R. (Rinceer Xm.
Rinttr Xm
Figure 1. The entropy similarity (Lin al., 2024) of LoRA pa- rameters for each module within the same layer (top) and across different layers (bottrom). It shows that different modules within the same layer exhibit high entropy similarity, and this high sim- ilarity is also present across different layers. This indicates that LoRA parameters have significant degree of redundancy.
By introduciig this novel decomposition, our BSLoRA fine- tuning approach is capable of learning both localized fea- tures and global interactions, providing balance between task-specific adaptability and overall model robustness. Fig- ure 2 presents overview of our BSLoRA. By decompos- ing LoRA into multiple smaller LoRA modules, we can assign a higher rank to the shared parameters, allowing the final parameters to achieve a greater overall rank, as discussed further in Section 4.4.
We assign different ranks for each smaller LoRA, the figuration and results are shown in Table (Joint). The experimental results indicate that sharing parameters not only reduces the number of parameters but also improves model performance.
3.2. Shape Transformation
We initially followed the LoRA setup (Hu et al., 2022) by applying LoRA parameters to the q and v modules in the attention layer, matching the shared parameter size to the qu module, which yielded preliminary results (Table 5), Previous studies suggest that applying LoRA to more mod- ules improves performance (Dettmers et al., 2023), extended it to the FFN layers. However, this caused param- eter shape mismatch. In the Transformer block (Vaswani et al., 2017). the givo modules in Llama's 's attention layer have a consistent shape of (4096, 4096). Meanwhile, the FFN's up- and down-projection modules have dimensions of (4096, 11008) and (11008, 4096), respectively, making it difficult to apply single shared AB parameters across these varying shapes. Similarly, in Llama3, the Grouped- Query Attention (GQA) changes the shape of k module, further complicating the parameter sharing. To address this issue, we develop three transformation methods that adjust shared parameters to the size of target modules.
3. Method
3.1. Parameter Sharing
In this paper, we apply parameter sharing to enhance pa- rameter efficiency in LoRA fine-tuning by leveraging the benefits of parameter sharing (Han et al., 2024). Our ap- proach first decomposes the LoRA module horizontally, expanding the optimization search space and allowing for more efficient learning. We decompose the LoRA matrix into three Lub-LoRA modules: local, intra-layer. and inter- layer which collectively contribute to the parameter updates of the target module.
Intra-Layer Module refers to the sharing of parameters within the same transformer layer, such that all modules within single layer (e.g., attention and MLP) share the same LoRA update matrix. This enables the model to cap- ture consistent patterns and correlations within the layer,
3

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Frozen
Tunable
Sharing
Transformer Layer +
Shape Transformation
P
Add & Norm
transform
Apply to intra-layer modules
Apply inter-layer modules
Feed-Forward
down
Shared Weights
Target Module
B Suaee
B E
Add & Norm
F-
A
aljefu
&.W..
W
Multi-Head Attention
B
B
B Four
0
W
pretrain
A
A
Transformer Layer
W
A
X
Transformer Layer i-
Figure overview of BSLoRA. The LoRA weights are decomposed into three sub-LoRA weights: local learns modulc-speccife information, intra capture the shared features within the same transformer block, and inter learns the global information to interact with each module. The shape transformation enables the shared weights adaptive to different shapes of modules. By combining local, intra, and inter, the model can learn both local and global information during fine-tuning, so that improves the performance and generalizability.
3.2.1. SLICE SHARING
introduce the Gate Transformation (GT), which applies an input gate G; E Rmxd: and an output gate G. E RRdx For an input æ E I G; transforms it to (b, m). and the shared matrix W. E -=* processes it to produce an intermediate result (b, n). Finally, G, outputs the final shape (b, do).
The straightforward method is to slice larger trainable pa- rameter matrix and train only the sliced portions (see Figure 3 (a)), which we refer to as the Slice Sharing (SS) method. Specifically, we determine the maximum input dimension dim and output dimensions dom: among all fine-tuning pa- rameter modules. The shared matrices then defined with dimensions A, Brxxdmm and B, Rdomxr xr During for- ward computation, the shared matrix is automatically sliced to match the parameter dimensions of the target module. The calculation is expressed as:
However, defining these transformation matrices introduces many learnable parameters, leading to high memory con- sumption for large inputs and outputs. To mitigate this, we apply one-rank decomposition to the input and output gates, reducing them to the product of two small rank-one matrices (see Figure (b)). The final computation is follows:
AW B, do) A.|: d:.:]
(3)
AW = G. N.WWCCO W,G; (Gou ood) A. (G
(4)
where de and d, represent the input and output dimensions of the target module, [: d,] and A, [: d represent the sliced parts of the shared weights of B and A,. Algorithm 1 provides the pseudocode for Slice Sharing.
where the Gid TR 1xd; projects down the input dimension into a lower-dimensional space, and then Gia E R=x1 scales it up into the dimension that is comparable to the input dimension of the shared weights W.. Similarly, God e R Ixn and G E Rd.x1 x1 are applied to transform the output. By setting the input and output gates, the size of our shared parameters be flexibly changed. Algorithm 2 provides the pseudocode for Gate Transformation.
3.2.2. GATE TRANSFORMATION
The simple slicing method enables parameter sharing, but only subset of the shared parameters is used by all modules, while the remaining parameters only utilized by larger modules. This limits the efficiency of parameter sharing. To address this, we propose matrix multiplication for dynamic size transformation of shared parameters. By multiplying matrices M, E m= and M, E R"*P, the resulting matrix M. TRxxp transforms the shape. Based on this,
3.2.3. KRONECKER EXTENSION
While utilizing Gate Transformation allows us to define shared parameters of arbitrary shapes, low-rank decomposi- tion may lead to information loss in both input and output

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
(a) Slice Sharing
(b) Gate Transformation
Kronecker Extension
dj
Input Gate
B
m
LoRA
@
pm
Output Gate
d
qu
parameters (unselect)
Shared parameters (active)
Module-specifie parameters
@
Outer Product
Kromscker
Figure 3. Three methods for shape transformation: (a) Slice Sharing, slices parameters from a large shared trainable parameters: (b) Gate Transformation, an imput gate and output gate transforms input dimension and output dimension match the shape of shared parameters, and they are implemented by one-rank decomposition; (c) Kronecker Extension, modulc-specific kernel are used to extend the shared weights into target shape through the Kronecker Product.
transformations, Another approach concatenate multi- ple small shared parameters from identical copies form larger shared parameter (Wang al., 2024; Edalati al., 2022), but this limits the overall expressiveness. To address this, inspired by Karimi Mahabadi et al. (2021), we apply Kronecker matrix multiplication to expand the dimensions of the shared matrix by integer multiples, a method we term Kronecker Extension (KE). The Kronecker product between matrices X mm and Y [Rxx4, denoted X & Y pmpxxg, is mathematically defined as:
4. Experiments
4.1. Settings
LLMs. To demonstrate how BSLoRA performed on dif- ferent models, we conduct experiments on Llama families (Touvron et al., 2023): Llama and Llama 3. In particular, we fine-tune the 7B and 13B models of Llama 1 and the 8B model Llama 3, with the specific versions detailed in Appendix B.2.
Benchmark. We conduct experiments for these LLMs on two different benchmarks. The first benchmark is Com- monsense Reasoning, which includes BoolQ (Clark et al., 2019), PIQA (Bisk al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-easy (Clark et al., 2018), ARC-challenge (Clark et al., 2018), OpenbookQA (Mihaylov et al., 2018), and SIQA (Sap et al., 2019). The second benchmark is Massively Multitask Lan- guage Understanding (MMLU) (Hendrycks et al., 2021). Dataset details are presented in Appendix B.3. We em- ployed Im-eval-harness (Gao al., 2023) create open prompts for the benchmarks and produce the results.
F11
X Y
(5)
Tml
Jmfz
where mr shows the element in the tt row and 7th column of X.
We assign a module-specific kernel K Rdxr each module in the model (see Figure 3(c)). By applying the Kronecker product, we expand the shared parameter A, E R and B, E R xr to match the size A, E Rx and B, E Txx of the target module. Here, r is the rank value set for the shared parameter. Finally, according to our Kronecker Extension, AW is calculated as:
Baselines. We compare against several recently proposed LoRA-based PEFT methods: (1) LoRA (Hu al.. 2022). we set the rank 8 for the standard LoRA to fine-tune the model. (2) VeRA (Kopiczko et al., 2024), we adopt the default setting where the rank is set to 64. (3) VB-LORA (Li et al., 2024), we follow the setting of VB-LoRA, where the vector length is set to 256 and there are 90 vectors to be trained. Moreover, we set the of the top-k to 2. (4) Tied- LoRA (Renduchintala al., 2024), we follow the setting
AW (K B & (.) @ As)
where Kn E R and KA E 1 represent module- specific kernels for the B and A matrices in the LoRA mod- ule. Algorithm 3 provides the pseudocode for Kronecker Extension.
5

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Table 1. Results of Zero-shot performance on LJama 1-7B, Ilama 3-8B, and Llama 1-13B in oSRR and baselines on Commonsense
Reasoning benchmark. We report the number of trainable parameters params) and the corresponding ratio for each method.
Methods
params
ratio
OBQA
ARC-c
HellaSwag
ARC-e
PIQA
WinoG.
BoolQ
SIQA
Avg.
LoRA,=s
14.02M
0.21%
44.80
47.10
77.35
76.47
80.25
69.77
77.98
48.21
65.24
Liama 1-7B
VeRA,=64
0.89M
0.01%
44.60
44.80
76.47
75.88
79.27
70.24
75.14
46.37
64.09
VB-LORA,,4
2.49M
0.04%
46.00
47.35
77.28
77.44
79.98
70.48
76.79
48.77
65.51
ShareLoRA,...
7.23M
0.11%
44.40
47.53
77.49
77.36
80.03
70.48
78.04
48.06
65.42
Tied-LoRAr-s
0.44M
0.01%
44.40
44.88
76.19
75.25
79.22
70.09
75.06
45.70
63.85
BSLoRA (SS)
7.03M
0.10%
46.20
46.93
77.23
76.77
80.52
69.85
78.13
49.39
65.63
BSLORA (GT)
8.22M
0.12%
45,20
47.27
77.47
77.06
80.14
70.17
78.93
48.93
65.64
BSLoRA (KE)
3.66M
0.05%
45.40
47.87
77.32
77.57
80.20
70.24
77.52
48,62
65.59
LoRA,=s
14.16M
0.18%
46,20
57.34
80.04
82.95
81.88
73.72
82.32
48.67
72.42
Liama 3-8B
VeRAp-64
0.80M
0.01%
45.00
54.01
79.27
80.51
81.23
73.32
81.07
47.34
70.96
VB-LORA,
2.51M
0.03%
46.40
56.06
79.85
81.27
81.39
74.51
81.62
46.93
71.66
Sharel LoRys
8.06M
0.11%
45.60
56.57
79.89
82.79
81.99
73.80
82.69
48.46
72.31
Tied-LoRA,es
0.44M
0.01%
45.00
53.16
79.17
80.09
80.85
73.24
81.07
47.24
70.69
BSLoRA (SS)
7.67M
0.10%
46.40
57.17
79.96
82.95
81.94
74.74
83.09
49.03
72.70
BSLoRA (GT)
8.03M
0. 10%
46.20
56,83
79.89
82.87
81.94
74.27
82.97
48.36
72.45
BSLORA (KE)
3,83M
0.05%
46.40
56.57
80.04
83.08
82.15
73.64
82.60
48.98
72.44
LoRA,=8
21.95M
0.17%
45.40
51.71
80.21
79.21
80.90
72.69
81.13
48.87
67.52
Llama 1-13B
VeRAr=125
1.40M
0.01%
44.80
47.87
79.30
77.61
80.25
72.85
78.07
46.88
65.95
VB-LORA,
3.88M
0.04%
47.20
51.11
80.91
78.66
80.58
72.38
80.18
49.49
67.56
ShareLoRA
11.25M
0.09%
45.80
51.71
80.38
79.34
80.58
72.85
80.89
49.03
67.57
Tied-LoRA
0.55M
0.01%
44.80
47.87
79.08
77.40
80.25
72.85
77.92
56.65
65.85
BSLoRA (SS)
10.13M
0.08%
45.80
51.28
80.11
79.21
80.74
72.69
81.59
49.13
67.57
BSLoRA (GT)
12.14M
0.9%
46.00
51.02
80.11
79.04
80.90
72.53
80.95
48.72
67.41
BSLORA (KE)
5,94M
0.05%
45.00
51.79
80.32
79.21
80.74
72.85
80.83
48.93
67.46
of T'L (5) SoareLoRA (Song al., 2024), adopt the ShareA configuration to finetune the LLM.
4.2. Results on Commonsense Reasoning
We evaluate the zero-shot performance of BSLoRA on Com- monsense Reasoning tasks using Llama 1-7B, Llama 3-8B, and Llama 1-13B models. In Table 1, the results show that o consistently outperforms the baselines in terms of average performance across these datasets. Specifically, the Kronecker Extension (KE) method introduces fewer train- able parameters while achieving comparable performance to the SS and GT methods, indicating superior parameter efficiency. More details in Appendix A.4.
Fine-tuning Dataset. We utilized publicly available sam- ples from the Alpaca dataset (Taori et al., 2023) to fur- ther fine-tune the LLM, which contains 52k instruction- following demonstrations generated by OpenAl's text- davinci-003 engine.
Hyper-parameters and Training Details. We apply the LoRA weights to the W., Wx. W,, v. W up, and W down modules of each Transformer block. For each shape trans- formation method, we set different rank configurations
4.3. Results on MMLU Benchmark
We evaluate the zero-shot and five-shot performance of BSLoRA on the MMLU benchmark using Llama 1-7B, Llama 3-8B, and Llama 1-13B models. The results, as shown in Table 3, demonstrate that BSLoRA consistently outperforms the baseline models in terms of average per- formance across both zero-shot and five-shot settings. This highlights the effectiveness our approach diverse see- narios (see more in Appendix A.4.2).
r r= =rtoeatt, Tintra; Finttra:Tinter) We set = {2, 4,32) for Slice Share (SS). We set the shape of shared weights for Gate Transformation (GT) to (1024, T and r {2, 8, 16). For the Kronecker Extension (KE), we set r [2, 4, 16) to adapt the shared weights' shape of (256, (share) We use the same training configurations to fine-tune the LLM with BSLoRA and baseline methods. Specifically, we use AdamW (Loshchilov & Hutter, 2019) as the optimizer with 100 warm-up steps and a learning rate of 1 10-4 and set the batch size to 64, For all the experiments, train for one epoch.
4.4. Analysis
Rank Analysis. According to matrix rank theory, the rank of the sum of two matrices given by R(A R(A+B) S R(A)+ R(B). This implies that decomposing a large LoRA matrix into multiple sub-LoRAs does not increase the overall rank
htttps:/ http://hgggggggggggeee htt.:/accc.co/ataatttttete al papaaa- cleaned
6

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Table 2. Results of zero-shot and five-shot performance on Llama 1-7B, Llama 3-8B, and Llama 1-13B in BSLORA A and baselines on MML.U benchmark. We report the number of trainable parameters (# params) and the corresponding ratio for each method.
MMLU (0-shot)
MMLU (5-shot)
Method
# params
ratio
Hums.
STEM
Social
Other
Avg.
Humss
STEM
Social
Other
Avg.
LoRAr=s
14.02M
0.21%
34.67
31.24
37.21
39.36
35.62
34.86
32.57
40.36
40.81
37.15
Liama 1-7B
V4RA,=64
0.89M
0.01% %
32.22
28.32
32.40
36.72
32.42
33.58
31.27
38.58
38.85
35.57
VB-LoRA,=4
2.49M
0.04%
30.10
28.51
29.25
33.31
30.29
34.11
30.54
40.14
40.52
36.33
ShareLoRA
7.32M
0.11%
33,01
30.61
33.57
37.62
33.70
33.88
30.73
38,93
39.78
35.83
Tied-LoRA,-s
0.44M
0.01 %
31.92
28.20
32.11
36.31
32.13
33.84
31.56
38.64
38.53
35.64
BSLoRA (SS)
7.03M
0.10%
36.34
32.98
40.59
42.55
38.12
36.62
33.46
42.48
43.35
38,98
BSLORA (GT)
8.22M
0.12%
36.30
32.92
40.75
42.13
38.03
35.15
32.19
41.44
41.94
37.68
BSLoRA (KE)
3.66M
0.05%
35,56
32.00
38.12
40.01
36.42
35.32
32.41
41.40
41.29
37.61
Lo
14.16M
0.18%
56.77
53.89
72.77
70.36
63.44
59.81
55.92
76.21
72.10
66.01
Llama 3-8B
VeRA,=64
0,80M
0.01%
54.88
54.17
73.35
70.58
63.25
59.85
55.69
76.15
72.80
62
VB-LoRA,=4
2.51M
0.03%
55.83
52.74
71.86
71.00
62.86
59.11
55.22
74.68
72.26
65.32
ShareLoRA
7.67M
0.11%
56,81
54.17
73.16
70.87
63.75
59.68
55.63
76.18
72.61
66.02
Tied-LoRA
0.44M
0.01 %
54,77
54.01
74,29
70.58
64.16
59.96
55.82
76.05
72.64
66.12
BSLoRA (SS)
7.67M
0.10%
58.53
54.61
73.35
71.29
64.45
59.57
56.26
75.95
72.35
66.04
BSLORA (GT)
8.03M
0.10%
58.30
54.39
73.29
71.48
64.37
59.68
56.23
76.31
72.48
66.18
BSLoRA (KE)
3,83M
0.05%
57.98
54.49
73.35
70.84
64.16
59.38
55.98
76.08
72.16
65.90
LoRA,=s
21.95M
0.17%
43.66
38.31
54.44
53.33
47.43
45.06
37.46
$5.64
55.42
48.39
Llama 1-13B
VeRA,-128
1.40M
0.01 %
41.66
36.50
48.75
48.73
43.91
44.23
37.08
53.92
53.59
47.20
VB-LoRA,=s
3,88M
0.04%
41.02
35.81
49.66
49.76
44.06
44.27
37.14
54.27
53,91
47.40
ChareLoRA,.-
11.25M
0.09%
43.78
37.33
52.75
52.78
46.77
44.61
36.63
54.01
54.23
47.37
Tied-LoRApes
0.55M
0.01%
41.72
36.38
48.52
48.44
43.76
44.46
37.01
53.85
53.59
47.23
BSLoRA (SS)
10.13M
0.08%
44.76
37.46
53.04
53.72
47.24
44.78
37.81
54.50
55.65
48.18
BSLORA (GT)
12.19M
0.09%
43.61
37.77
53.33
53.14
46.96
44.46
37.52
53.88
55.07
47.73
BSLoRA (KE)
5.94M
0.05%
44.48
38.98
54.34
54.49
48.07
45.29
37.84
55.25
55.36
48.43
of the matrix. Consequently, the rank of the combined LoRA matrices remains bounded by the sum of their individual ranks. We validate the actual rank of each module and calculate the average ranks across layers for each Shape Transformation method using rank configurations of {2, 4, 16). The results, shown in Figure 4(a), indicate that the Slice Sharing (SS) method achieves a combined rank equal to the sum of local and shared ranks (22), while the Kronecker Extension (KE) reaches approximately 21.53. In contrast, the Gate Transformation (GT) method yields rank value equivalent to the local rank plus 2, likely due to one-rank gates causing some information loss.
W E R can be obtained by applying the Kronecker product to M E R xr and K E R By adjusting their shapes to M E R#x1 and K E R the same matrix shape for W can be achieved. Furthermore, since one di- mension is a constant I, the resulting matrix rank equals r. To explore its effect, we modify this constant to 2 to test whether it enhances the rank of W. We denote config- urations as a b, where r 1 means the shared matrix has a dimension r and a module -peecffc dimension of 1, while 2.r has a shared dimension of 2 and a module module-specific di- mension of r. The results in Figure 4(c) show that modifying the constant improves both expressiveness and information content. Comparing 1 and 1_r, we observe that each con- figuration excels in different metrics, with r_1 performing better overall while introducing fewer trainable parameters.
Contribution Analysis. We conduct further experiments to explore the contribution of each sub-LoRA matrix. By setting the rank of one sub-LoRA matrix to 8 and the others to O, we examine the individual impact of each component. The Kronecker Extension method is used reshape the shared parameters. In Figure 4(b), the results reveal the performance preferences of each component across datasets. Specifically, the local component of LoRA performs best on HellaSwag, ARC-e, BoolQ, and SIQA, while the intra-layer shared component excels on PIQA and ARC-c. Overall, the combination of local, lntre- and inter-layer shared parameters yields the best performance across all datasets.
4.5. Ablation Study
We conduct ablation studies to examine the impact of indi- vidual components of our method. All subsequent experi- ments focus on the MMLU (zero-shot) (Hendrycks et al., 2021) and GSM8K (5-shot) (Cobbe et al., 2021) bench- marks, utilizing the Llama 1-7B model. We also conduct experiments to examine the impact of different rank alloca- tions in Appendix A.4.5.
Extension Analysis. We further analyze the scalability of the Kronecker Extension method. The shared matrix
Shared Size of Gate and Kronecker Kernel. The Gate
Transformation (GT) and Kronecker Extension (KE) meth-
7

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
heleswag
+00.0
way
sige
65.5
Park Vhi005
Aveage Nerormance
65.4
pign
openbookge
65.3
05.2
65.1
winogrande
Shape Transfo mution Mathads
bookc
65.0
011
022
1
2.r
(a) Rank Anlysis
(b) Contribution Analysis
(c) Extension Analysis
Figure 4. Analysis. (a) Rank Analysis: the final rank benefit from different Shape Transformation methods. (b) Contribution Analysis: the different contribution across the sharing weights and local weights. (c) Extension Analysis: the rank extension analysis for different shape of share weights' size and kernel size.
ods can flexibly adapt the shared weights to arbitrary shapes for different modules. We examine various sizes of the shared matrix, setting the rank r {2, 8, 16). For the GT, test shared matrix sizes of [512, 1024, 2048], while for the KE, we test sizes of [64, 128, 256]. The results pre- sented in Table 3 and Table 4. We found that the parameters introduced by GT increase with the growth of the shared parameter size, which also leads to an improvement in per- formance on the MMLU benchmark. However, we observe a decrease performance on the GSMSK dataset as the shared parameter size increases. In contrast, the parame- ters introduced by the KE method decrease with increasing shared parameter size, while the average performance im- proves, Overall, we select the shared parameter size of 1024 for the GT method and 256 for the KE method.
vectors to differentiate the modules. However, its require- ment for identical matrix shapes (appliable for q, k, and v matrices) limits flexibility. ShareLoRA (Song et al., 2024) devises to share A, B, or AB matrices across layers to reduce parameters, but it is only applicable for the weights with the same shape. In contrast, PRoLoRA (Wang et al., 2024) employs an intra-layer sharing mechanism with learnable parameters, but it only reduces parameters without capturing global features. Additionally, VB-LoRA (Li et al., 2024) introduces sdivide-and-share approach that partitions shared vectors into vector bank, addressing the limitations of traditional low-rank decomposition across matrix dimen- sions, modules, and layers. However, this approach selects vectors uniformly without accounting for the internal struc- ture of the model, leading to suboptimal utilization of the model's internal information.
Table Size of Shared weights of GT
Shared Size
params
railo
MMLU
GSM8K
5.2. Multi-LoRA Architecture
512
7.62M
0.11%
36.03
10.54
1,024
8.22M
0.12%
35.61
10.77
LoRA has demonstrated exceptional resource efficiency and performance in adapting LLMSs for specific tasks, driv- ing the demand for a single model capable handling multiple tasks (Agiza et al., 2024). Several approaches have been proposed to improve their multi-task adaptabil- ity. In particular, LoraHub (Huang et al., 2023) assembles LoRA modules trained on different tasks to eliminate the need for human expertise and assumptions, enabling ef- fective cross-task generalization. Similarly, MultiLoRA (Wang al., 2023) improves adaptability by horizontally expanding LoRA modules and reducing the dominance of top singular vectors observed in LoRA. Building on these advancements, HydraLoRA (Tian et al., 2024) introduces asymmetric architecture that shares common matrix across tasks while using task-specific matrices for differ- ent sub-domains, further enhancing both fine-tuning and inference efficiency.
2,048
9.44M
0.13%
34.70
11.90
Table 4. Size of Shared weights of KE
Shared Size
params
raito
MMLU
GSMSK
4.07M
0.06%
36.47
10.84
128
3.82M
0.05%
35.34
3.72M
0.05%
37.01
11.22
5. Related Work
5.1. Parameter Sharing of LoRA
Recent advances in LoRA-based fine-tuning methods have explored various strategies for sharing LoRA weights to enhance parameter efficiency across multiple tasks. VeRA (Kopiczko et al., 2024) proposes sharing random matrices across all layers, reducing the number of parameters, but it results in some performance trade-offs and increased infer- ence latency due to its high-rank requirements. In addition, Tied-LoRA (Renduchintala et al., 2024) takes a different ap- proach by sharing LoRA matrices specifically across query, key, and value projection layers, using additional scaling
6. Discussion
Recent work on S-LORA (Sheng et al., 2024) addresses the deployment side of LoRA by keeping a unified GPU
8

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
buffer for adapter weights, dynamically paging seldom-used adapters from CPU to GPU, and fusing heterogeneous re- quests into a single batched GEMM. Although orthogonal in scope, S-LoRA and BSLoRA are highly complemen- tary: (1) Reduced paging overhead. BSLoRA removes up to 56% of the trainable weights through hierarchical shar- ing. With fewer parameters per adapter, S-LoRA has to move considerably less data across the CPU-GPU bound- ary, which shortens task-switch latency and allows many more adapters to remain resident chip. (2) Shape-aware batching. BSLORA organises its shared components by transformation Sipe-Slice, Gate, or Kronecker. Because each type yields identical weight shapes across tasks, S- LoRA's scheduler can co-group requests that reuse the same shared tensor and issue single large GEMM, further im- proving arithmetic intensity. (3) Persistent shared cores. In BSLoRA the intr-llaerr and inter-layer matrices are tiny. Under S-LoRA these small cores can remain permanently pinned in GPU memory, while only the larger local blocks are paged to and from CPU. Consequently, the amount of data transferred per context switch is minimised, further reducing end-to-end latency. We believe a unified BSLoRA + S-LoRA serving stack will offer both model-level and system-level efficiency, enabling the simultaneous deploy- ment of thousands of personalised adapters on a single GPU. Implementing and benchmarking this joint design is left for future work.
consequences of our work, none of which we feel must be specifically highlighted here.
References
Agiza, A., Neseem, M., and Reda, S. Mtlora: Low-rank adaptation approach for efficient multi-task learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16196-16205, 2024.
Anthropic.
The claude 3 model fam-
ily: Opus, sonnet, haiku.
2024.
URL
https: ptrps://o htt////www//e www.: com/ de8ba hbt.:aacccbccbbbba3333333e 0b8.066676688188787777 18857627/
Model_ Caa___Carr__aauudeee 3.d..
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E.,, Litwin, M., Gray, S., Chess, B.. Clark, J., Berner, C., McCan- dlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
7. Conclusion
This paper presents BSLoRA, which enhances the param- eter efficiency of LLMs by combining shared intra-layer, inter-layer parameters, and local parameters. This approach reduces the number of trainable parameters while boosting model efficiency. Experiments on various Llama models demonstrate that BSLoRA significantly cuts down parame- ter usage by 56.40% and improves average performance by 0.33% on commonsense reasoning tasks and 2.08% on MMLU benchmark. BSLoRA reduces redundancy, hances model adaptability, and offers flexible sharing strate- gies with potential high-rank benefits.
Chang. Y., Wang. X., Wang, J., Wu, Y., Yang, L.., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 (3):1-45, 2024.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S., Dai. Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A.. Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang. Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J.. Devlin, J., Roberts, A., Zhou, D., Le. Q. V., and Wei. J. Scaling instruction -inettuunn- language models, 2022.
Acknowledgements
This project is based upon work supported by National Natu- ral Science Foundation of China Grant No. U22A6001, Na- tional Key Research and Development Program of China No. 0023YFE0108600 Pioneer" and "Leading Goose" R&D Program of Zhejiang E..2024SSYS0003
Clark, C.,, Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 2924 2936, 2019.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
question answering? try are, the ai2 reasoning challenge. arXiv preprint 1rXivv1883.0553.00 2018.
Huang, C., Liu, Q. Lin, B. Y., Du, C., Pang, T., and Lin, M. Lorahub: Efficient cross-task generaliza- tion via dynamic loRA composition. In Proceedings of Fo-o:oooRossss of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems, 2021.
Jiang. T., Huang, S., Luo, S., Zhang, Z., Huang, H., Wei, F. Deng, W., Sun, F., Zhang, Q. Wang, D., and Zhuang, F. Mora: High-rank updating for parameter-efficien fine- tuning, 2024.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettle- moyer, L.. Qlora: Efficient finetuning of quantized Ilms. In Proceedings of Advances in Neural Information Processing Systems, volume 36, pp. 10088-10115, 2023.
Karimi Mahabadi, R., Henderson, J., and Ruder, S. Com- pacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems. volume 34, pp. 1022-1035, 2021.
Ding, N., Qin, Y., Yang. G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, (()):220-335, 2023.
Koohpayegani, S. A., L, N. K., Nooralinejad, P., Kolouri, S., and Pirsiavash, H. NOLA: Compressing loRA us- ing linear combination of random basis. In The Twelfth International Conference on Learning Representations, 2024.
Edalati, A., Tahaci, M., Kobyzev, I. Nia, V. Clark, J. J., and Rezagholizadeh, M. Krona: Parameter efficient tun- ing with kronecker adapter, 2022.
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-based random matrix adaptation. In Proceedings of International Conference on Learning Representations, 2024.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, I.,, Hsu, J., Le Noac'h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 2023.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
Guan, C., Wang, X., Zhang, Q. Chen, R., He, D., and Xie, X. Towards deep and unified understanding of deep neu- ral models in NLP. In Proceedings of the International Conference on Machine Learning, volume 97, pp. 2454 2463, 2019.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing con- tinuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on 2021.
Natural Language Processing, pp.
Han, Z. Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey. Transactions Machine Learning Research, 2024.
Li, Y., Han, S., and Ji, S. Vb-lora: Extreme parameter efficient fine-tuning with vector banks, 2024.
Lin, S., Lyu, P., Liu, D., Tang, T., Liang, X., Song, A., and Chang, X. Mlp can be good transformer learner. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 19489-19498, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In Proceedings of International Conference on Learning Representations, 2021.
Liu, S.-Y., Wang, C.-Y,, Yin, H., Molchanov, P., Wang, Y.-.. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight- decomposed low-rank adaptation. In Proceedings of International Conference Machine Learning, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly. S. Parameter-efficiae transfer learning for NLP. In Proceedings of the International Conference on Machine Learning, volume 97, pp. 2790-2799, 2019.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In Proceedings of International Conference on Learning Representations, 2019.
Hu, F. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations, 2022.
Mao, Y., Ge, Y., Fan, Y., Xu, W., Mi, Y., Hu, Z., and Gao, Y. A survey on lora of large language models. arXiv preprint arXiv:2407.1 11046, 2024.
10

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
Sun, Z., Ge, C., Wang, J., Lin, M., Chen, H., Li, H., and Sun, X. Entropy-driven mixed-precision quantization for deep network design. In Proceedings of Advances in Neural Information Processing Systems, volume 35, pp. 21508-21520 2022a.
Openai. Gpt-4 technical report. 2023. URL https:// cdn. hwt.hhopenaccoopeeeepppeeeee
Sun, Z., Yang. H., Liu, K., Yin, Z.,, Li, Z and Xu, W. Recent advances in lora: A comprehensive survey, ACM Transactions on Sensor Networks, 18(4): 44, 2022b.
Raiaan, M. A. K., Mukta, M. S. H., Fatema, K., Fahad, N. M., Sakib, S., Mim, M. M. J., Ahmad, J., Ali, M. E., and Azam, S. A review on large language models: Ar- chitectures, applications, taxonomies, open issues and challenges. IEEE Access, 2024.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stan- ford alpaca: An instruction-follooing llama model, 2023. URL https: /eg//gttuuut com/tatsu-lat st stanford_alpaca
Ren, P., Shi, C., Wu, S., Zhang, M., Ren, Z., Rijke, M., Chen, Z., and Pei, J. MELORA: Mini-ensemble low- rank adapters for parameter-efficient fine-tuning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 3052-3064, 2024.
Tian, C., Shi, Z., Guo, Z., Li, L., and Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024.
Touvron, H.. Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models, 2023.
Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied- lora: Enhancing parameter efficiency of lora with weight tying. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 8692- 8704, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. At- tention is all you need. In Proceedings of Advances in Neural Information Processing Systems, volume 30, 2017.
Sakaguhhi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wang. S., Xue, B., Ye, J., Jiang, J., Chen, L., Kong, L., and Wu, C. Prolora: Partial rotation empowers more parameter-efficient lora, 2024.
Sap, M.. Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about cial interactions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 4463 4473, 2019.
Wang, Y., Lin, Y., Zeng, X., and Zhang, G. Multilora: Democratizing lora for better multi-task learning, 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 4791448000 2019.
Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J. E., and Stoica, I. S-lora: Serving thousands of concurrent lora adapters, 2024.
Zhang, B., Haddow, B., and Birch, A. Prompting large language model for machine translation: case study. Proceedings of International Conference on Machine Learning, pp. 11092-41110, 2023.
Shi, S., Huang, S., Song, M., Li, Z., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., and Zhang, Q. ResLoRA: Iden- tity residual mapping in low-rank adaption. In Findings of the Association for Computational Linguistics, pp. 8870 8884, 2024.
Zhu, X., Liang, D.,, Jiang, X., Guan, Y., Liu, Y., Zhu, Y., and Bai, X. Layerlink: Bridging remote sensing object de- tection and large vision models with efficient fine-tuning. Pattern Recognition, 165:1 11583, 2025.
Sirignano, J. and Spiliopoulos, K. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics, 80(2):725-752 2020.
Song. Y., Zhao, J., Harris, I. G., and Jyothi, S. A. Sharelora: Parameter efficient and robust large language model fine- tuning via shared low-rank adaptation, 2024.

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
A. Appendix
A.1. Parameter-Efficien Fine- Tine-Tuning (PEFT)
Crameter-Efficienn Fine-Tuning (PEFT) is crucial for large language models (LLMs) as it reduces computational costs while preserving performance (Ding et al., 2023). Specifically, Adapters (Houlsby et al., 2019; Zhu al., 2025), inserted between layers, allow task-specific fine-tuning with minimal trainable parameters while keeping most weights fixed. Similarly, Prefix Tuning (Li & Liang, 2021) adds learnable tokens at each transformer layer to guide task-specific behavior without modifying core parameters. In contrast, Prompt Tuning (Lester et al., 2021) optimizes a small set of prompts attached to the input, leaving the model's 's architecture unchanged. LoRA (Hu et al., 2022) reparameterizes weight matrices into low-rank forms, significantly reducing trainable parameters while maintaining performance. Some studies build upon LoRA to further improve and optimize its performance (Shi et al., 2024; Ren et al., 2024; Jiang et al., 2024; Liu et al., 2024; Koohpayegani et al., 2024). Likewise, we propose BSLoRA, building on LoRA, which shares parameters both within and across layers to further enhance efficiency and adaptability.
A.2. Parameter Similarity
1. ENTROPY QUANTIFICATION
Information entropy is a key concept in information theory, used to quantify the uncertainty or randomness within a dataset or signal. It provides a measure of the average information content per symbol or event in a message or sequence of messages. As defined by Guan al. (2019), entropy can be employed to assess the information capacity of a network. Consequently, the entropy of a given layer can be calculated based the probability distribution of its features:
H(F) = f df)
Nonetheless, it is difficult to directly measure the probability distribution of feature map: p(J), E F. Following (Sirignano & Spiliopoulos, 2020; Sun et al., 2022a), we use the Gaussian distribution as the probability distribution of the intermediate feature in a layer. Therefore, the entropy of a certain layer is approximated as the mathematical expectation of F N(,,0))))
((F) E [log.N (1,
- E log (2TO²) -2/2 exp ( 1-22(0 (f u)3
(~
20
(8)
log(o) + log(2n + 1
1
2
where a is the standard deviation of the feature set f E F.
A.2.2. ENTROPY SIMILARITY
The information content of LoRA weight matrices can be measured by their entropy. Two matrices with similar entropy values typically contain similar amounts of information, indicating a high degree of redundancy. We assess the correlatio between different weight matrices by calculating mutual information, which quantifies the relationship between them and can be computed using the following formula:
1(X; ((;Y)) - H(X) H(Y) H(X,Y)
Here, (X, Y represents the entropy of the joint distribution of matrix X and matrix Y. To compute this, we flatten the matrices X and Y, concatenate them, and then calculate the entropy of the resulting joint distribution. A large mutual information between two weight matrices indicates significant overlap in their information. implying redundancy. The calculated mutual information T(x; Y) is an absolute value, which needs to be converted into a relative value. Therefore, we use Relative Mutual Information (RMI) to represent the similarity between matrices, calculated as follows:
T(X; Y)
RMT m H(Y))
(10)
If RMM > 0.8, the mutual information is considered high, indicating significant redundancy between the two matrices. 0.5 < RMI < 0.8, the mutual information is moderate, suggesting notable shared information but with some degree of independence. If RMI S 0.5, the mutual information is low, indicating minimal redundancy between the two matrices.
12

---

BSLoRA; Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Average Similarity Within Layers
Similarity Across Layers
0.9
09
0.90
Layer Similarity
12 10
0.80
Layer
16
0.6
18
0.75
0.70
0.4
15 Layer Index
10
16 18 Layer Index
22
24
28
30
Figure 5. The entropy similarity of LoRA parameters for each module within the same layer (lcft) and across different layers (right). It shows that different modules within the same layer exhibit high entropy similarity, and this high similarity is also present across different layers. This indicates that LoRA parameters have significant degree of redundancy.
We fine-tuned the Llama-7B model using LoRA with a rank r 64, isolating the LoRA matrices and calculating A AW. The information entropy of AW for each module computed based Equation 8, and the Relative Mutual Information (RMI) between modules was calculated both within the same layer and across different layers, with the results shown in Figure 1. We observed that different modules within the same layer exhibit high entropy similarity, and this similarity also extends across layers. Additionally, following the method in Lin et al. (2024), we calculated the entropy similarity the activation values for each module. As illustrated Figure the AW different parameter modules also shows high similarity in terms of activation values, further indicating the significant redundancy the parameters used in LoRA fine-tuning.
measure of similarity using Equations and 10. We obtained the RMI values
Table Performance instruction tuning with Alpaca 50K (Taori al., 2023), evaluated with MMLU (Hendrycks al., 2021) and GSMBK (Cobbe et al., 2021). Boldface indicates the best performance.
Method
Plocat
Tintra
Tinter
params
ratio
MMLU
GSMSK
Individual
LoRA (no-share)
8
4.19M
0.06%
35.12
10.84
LoRA (intra-share)
2.10M
0.03%
34.23
10.54
LoRA (inter-share)
8
0.07M
0.01% %
32.20
10.08
Joint
LoRA (share-intra)
4
4
3.14M
0.05%
34.63
10.92
LoRA (share-inter)
4
4
2.13M
0.03%
35.00
10.54
LoRA (share-intra-inter)
4
2
2
2.64M
0.04%
35.89
10.08
A.3. Shape Transformation Algorithms
We provide detailed pseudocode for each of the Shape Transformation methods used in BSLoRA. Specifically, Algorithm I outlines the steps for the Slice Sharing method, which slices larger matrices to adapt to different parameter dimensions. Algorithm 2 demonstrates the Gate Transformation method, which utilizes input and output gates to dynamically adjust shared parameters for varying module shapes. Finally, Algorithm 3 explains the Kronecker Extension method, which expands shared matrices using Kronecker products to maintain consistency across modules with diverse dimensions. These algorithms collectively contribute to enhancing parameter efficiency and flexibility in model fine-tuning.
13

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Algorithm 1 Pseudocode of Slice Sharing.
Input: The rank shared weights and the input æ;
Output: The output calculated by shared weights;
1: # Initialize the Shared weights:
2: mdin, mdout &- Find max InFeatures and OutFeatures across all modules;
3: A E R mdxan <- Normal Randomly Initialization;
4: B E mnnduutx + Zero Initialization;
5:
6: # Training and Inference Stage:
7: din, dout < InFeatures and OutFeatures of current module;
8: AW < B|: dout, :]. din]
9: result aW*
10:
11: return result
A.4. Experiments
1. RESULTS ON COMMONSENSE REASONING
We evaluate BSLoRA for zero-shot performance on Commonsense Reasoning tasks using Llama 1-7B, Llama 1-13B, and Llama 3-8B. The results are shown in Table We observe that BSLORA consistently outperforms the baselines in terms of the average performance of Commonsense Reasoning datasets. Specifically, the Gate Transformation (GT) method of BSLoRA achieves the best average performance on the Llama 1-7B, while the Slice Sharing (SS) method achieves the best average performance on the Llama 3-8B. Furthermore, BSLoRA outperforms the baselines Llama 1-7B, and 5 out of 8 datasets on Llama 3-8B. It is worth noting that large improvements are achieved on ARC-e, PIQA, BooiQ, and SIQA datasets. BSLoRA A also achieves decent performance on the remaining datasets, including OBQA, HellaSwag, and WinG, which proves that BSLoRA is stable and reliable across different datasets. Compared to the standard LoRA with a rank of 8, BSLoRA can save about 50% trainable parameters and achieve better performance. Specifically, the Kronecker Extension (KE) method introduces fewer trainable parameters, meanwhile outperforms the baselines, and achieves on-par performance with the SS and GT methods, indicating the more parameter-efffiient method.
out of datasets
A.4.2. RESULTS ON MMLU BENCHMARK
We evaluate BSLoRA for zero-shot and five-shot performance of MMLU tasks based on Llama 1-7B, Llama 1-13B, and Llama 3-8B. We demonstrate the results in Table 2. From the results, we observe that BSLoRA consistently outperforms the baselines terms of the average performance of MMLU datasets both on zero-shot and five-shot. Specifically, the Slice Sharing (SS) method of BSLORA achieves the best average performance on the Llama 1-7B, while the Gate Transformation (GT) method achieves the best average performance on the Llama 3-8B. Notice that the SS method on Llama 1-7B achieves the best performance 9 out of 10 metrics and achieves the second-best performance for the remaining the GT and Kronecker Extension (KE) BSLoRA achieve the second-best performance compared to SS and consistently outperform the baselines. Compared to the standard LoRA with rank of 8, BSLORA can save about 50% trainable parameters and achieve better performance. Specifically, the KE method outperforms the baselines and achieves on-par performance with the SS and GT methods, indicating the most parameter- effficient method.
Furthermore,
A.4.3. EXPERIMENTS ON FLAN V2 DATASET
To further demonstrate the effectiveness BSLoRA, we also conduct experiments on Llama 3-8B on the FLAN-V2 instruction dataset (Chung et al., 2022) 2, which is another dataset for instruction tuning. We conduct experiments on the Chain Of Thought task and evaluate its performance on the Commonsense Reasoning task datasets; the experiment results are shown in Table 6.
14

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Algorithm 2 Pseudocode of Gate Transformation.
Input: The rank of shared weights and the input z;
Output: The output caleulated by shared Weights;
1: # Initialize the Shared Weights:
2: dins, douts < InFeatures and OutFeatures of shared weights;
3: A, E I dinxd & Normal Randomly Initialization;
4: B, E Iedoutexr <- Zero Initialization;
5:
6: # Initialize Gate weights:
din, dout InFeatures and OutFeatures of current module;
8: Gid E Rlxdin <- Uniform Randomly Initialization;
9: Giu E [Rdinsx1 Uniform Randomly Initialization;
10: God E BIxxdouts +- Uniform Randomly Initialization;
11: Gou E Idooux1 < Uniform Randomly Initialization;
12:
13: # Training and Inference Stage:
G; IR Indinsxdin GinGia
Go E gddout xdouts Gov. Godi
16: AW +- G.B.A.G.
17: result AWæ
18:
19: return result
Table Results Zero-shot performance on Ilama 3-8B, in BSLoRA and baselines on Commonsense Reasoning benchmark. We report
the number of trainable parameters (# params) and the corresponding ratio for each method.
Methods
params
ratio
OBQA
ARC-e
HellaSwag
ARC-e
PIQA
WinoG.
BoolQ
SIQA
Avg.
LoRArss
14.16M
0.18%
45.40
53.41
79.20
80.43
79.82
74.35
83.27
47.39
67.91
Llama 3-8B
VeRA,=61
0.80M
0.01%
44.80
53.92
79.14
79.92
79.54
72.69
80.95
47.03
67.25
VB-LoRA,-4
2.51M
0.03%
44.00
54.01
78.72
80.26
78.62
74.66
81.19
46.01
67.18
SoareL.RRA,...
8.006M
0.11%
44.80
54.27
79.04
80.89
81.23
74.19
83.00
48.57
68.25
Tied-LLRRA-
0.44M
0.01% %
45.00
53.07
79.11
80.01
80.79
73.88
81.25
47.03
67.52
BSLoRA (SS)
7.67M
0.10%
56.23
79.00
81.14
80.79
74.27
83.52
48.77
68.79
BSLORA (GT)
8.03M
0.10%
45.20
55.38
79.22
81.99
81.28
74.98
83.21
48.52
68.72
BSLORA (KF)
3.83M
0.05%
44.40
55.03
79.11
82.41
81.18
74.90
84.10
49.54
68.83
A.4.4. ABLATION STUDY
Slicing Method. We evaluate three slicing methods for the shared matrix: top-left slice, bottom-right slice, and center slice. We adopt the rank configuration r = {2, 4, 8). The results are shown Table 7. It indicates that the center slice outperforms the other two slicing methods both on MMLU and GSM8K benchmark.
Gate Initialization. We initialize the input and output gates of the Gate Transformation using three schemes: Kaiming normal, Kaiming uniform, and constant one initialization. The rank configurations for different sub-LoRA weights are set for r = {2,& 8, 16). As shown Table 8, Kaiming uniform outperforms Kaiming normal initialization. Additionally, the constant ones initialization leads to gradient explosion or vanishing issues, making unsuitable for gate initialization.
Initialization of Kronecker Kernel. We apply Kaiming normal, Kaiming uniform, and constant ones initialization to the Kronecker kernel to examine the impact of different initialization schemes. We set the ranks of (2, 4, 16} for local, intra, and inter sub-LoRA matrices, respectively. From Table 9, the constant ones initialization performs better on the MMLU benchmark, while the Kaiming normal initialization outperforms the other two methods on the GSMSK benchmark. Overall, the Kaiming normal initialization performs best.
15

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Algorithm 3 Pseudocode of Kronecker Extension.
Input: The rank of shared weights and the input æ;
Output: The output caleulated by shared weights;
1: # Initialize the Shared Weights:
2: dins, douts < InFeatures and OutFeatures of shared weights;
3: A, E IR"xdins & Normal Randomly Initialization;
4: B, E Iedoutexr &- Zero Initialization;
5:
6: # Initialize Kernel weights:
7: din, dout <- InFeatures and OutFeatures of current module;
8: ha dins r;;
9: b₁ + douts r/;;
10: KA RIIx <- Uniform Randomly Initialization;
11: KB E Rkox1 t Uniform Randomly Initialization;
12:
13: # Training and Inference Stage:
A Brxd;
A @ As;
E dr KB Bgi
16: AW + BA
17: result AWæ a
18:
19: return result
Table Split Position
Table Gate Initialization
Table 9, Kernel Initialization
Method
MMLU
GSM8K
Matrix Init.
MMLU
GSMSK
Matrix Init.
MMLU
GSMSK
Top-Left
36.27
10.69
Kaiming Unif.
36.36
11.90
Kaiming Unif.
35.34
11.14
Right-Down
35.84
10.92
Kaiming Norm.
36.09
10.31
Kaiming Norm.
35.15
12.05
Center 36.58
11.14
Ones
NAN
NAN
Ones
35.84
11.14
A.4.5. RANK ALLOCATION
We designed comparative experiments to explore the impact adjusting different rank ratios on fine-tuning performance. Specifically, we varied the rank value of one sub-LoRA module while keeping the ranks of the other fixed. We conducted experiments the rank configurations for local, intra-sharing, and inter-sharing. The results shown in Table 10. Assigning a lower rank to the local component and higher rank to the shared parameters yielded better performance, further illustrating the redundancy in the standard LoRA parameters.
A.5. Analysis
A. PARAMETER COUNT ANALYSIS
In this section, compare the number of parameters of BSLoRA with the standard LoRA. Considering LLM with L layers, where each layer contains M modules with hidden dimension d, the number of trainable parameters is equal to the model size (i.e., LMdP) in full fine-tuning. LoRA reduces this number to 2LMdr, where is the rank of two low-rank decomposed matrices. In BSLoRA, the trainable parameters consist of two parts: local parameters C, computed as 2LM tocals and shared parameters, which include intra-parameter sharing Sinra and inter- parameee sharing Sinte- Therefore, the stored parameters can be represented by triplet {C, Sintra Sinter). Note that the different Shape Transformation methods would result in different parameters of Sintra and Sinter-
Slice Sharing. The parameters of S in Slice Sharing are shared across all modules. Specifically, the parameters of intra-sharing Sintra = 2T.dr and the parameters of inter-sharing Sintra 2drinter.
16

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Table Different configurations of ranks for our BSLoRA (KE). and underline means the second-bes best performance.
Boldface denotes the results terms of corresponding metrics,
Method
Ranks
OBQA
ARC-c
HellaSwag
ARC-e
PIQA
WinoG.
BoolQ
SIQA
Avg.
2,4,16
45.00
47.78
77.45
76.85
79.92
69.69
78.07
48.57
65.42
4,4,16
44.80
47.61
77.35
77.02
80.30
69.50
77.74
48.31
65.33
Adjust local
8,4,16
44.80
47.70
77.33
77.19
80.30
69.53
77.80
48.82
65.43
16,4,16
44.60
47.18
77.38
76.68
80.20
69.69
78.16
48.52
65.30
2,2,16
45.40
46.93
77.43
76.56
80.14
69.77
78.20
48.93
65.42
2,4.16
45.00
47.78
77.45
76.85
79.92
69.69
78.07
48.57
65.42
Adjust intra
2,8.16
44.80
47.44
77.36
77.06
79.43
70.24
77.43
48.52
65.29
2,16,16
45.20
46.84
77.38
76.52
80.41
69.61
78.38
48.77
65.39
2,4,8
45.40
47.01
77.32
76.73
80.09
70.17
78.17
48.41
65.41
2,4,16
45.00
47.78
77.45
76.85
79.92
69.69
78.07
48.57
65.42
Adjust inter
2,4,32
44.20
47.35
77.28
76.94
79.92
70.48
77.86
48.31
65.29
2,4.64
44.80
46.67
77.31
76.89
80.30
69.93
77.77
48.36
65.26
Gate Transformation. Different modules within and between layers share the parameters with the hidden dimension of dg. Besides, each module contains an input gate and an output gate to transform the dimensions. Specifically, the parameters of intra-sharing Sintra = U[Ld,rimene + ((d+ddd)) and the parameters of inter-sharing Sintra = ad Tanter"""" + L(
Kronecker Extension. Different modules within and between layers share the parameters with the hidden dimension of de. Besides, each module contains Kronecker kernel K e Ri*k to transform the dimensions. Specifically, the parameters of intra-sharing Sintra 22((d intra Mk), and the parameters of inter-sharing Sintra = 2(d + 2MM..
A.5.2. MULTI-LORA SERVING ANALYSIS
Memory Usage in Multi-lora Serving
Multi-LoRA deployment, tasks share a common pre-trained model, adapted each task by loading specific LoRA param- eters. Typically, all LoRA parameters are preloaded into GPU memory to minimize task-switching latency and maximize GPU utilization. However, with many tasks, only frequently used parameters remain in GPU memory, while others are stored on the CPU, causing delays during frequent task switching. To mitigate this, we reduce the parameter size through sharing, allowing more parameters to in GPU memory.
31000
28000
25000
22000
19000
16000
Out Of Memory
LoRA
13000
BSLoRA(KE)
10000
200
600
1000
Number of Serving LoRA
We conducted comparative experiment analyze the GPU memory footprint of deploying different numbers of down- stream tasks using the Llama model. Specifically, we deployed a Llama 1-7B base model on an V100 32G GPU and measured the memory usage for 100 to 100 LoRA parameters and BSLoRA (kE) parameters during inference. The results, shown Figure 6, indicate that BSLoRA reduces memory usage by approximately 60% compared standard LoRA under the same number of parameters. Furthermore, when deploying more than 350 LoRA parameters, the memory usage exceeds the GPU's capacity. Furthermore, determine the maximum deployment capacity, incrementally added parameters until the memory limit reached. Our findings reveal that standard LoRA can load up 341 parameters, while BSLoRA supports up 1095, significantly enhancing deployment efficiency.
Figure 6. Memory usage comparison of LoRA and BSLoRA (KE) for serving different numbers of tasks.
A.5.3. CONTRIBUTION ANALYSIS
We further investigate how performance affected when intra-layer and inter-layer parameter sharing are applied indepen- dently under different rank settings. Using the Llama 1-7B model fine-tuned on the Alpaca dataset, we configured various
17

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
shared rank settings for different shape transformation modalities and evaluated the results the CommonSense task. The findings, presented in Table demonstrate that intra-layer sharing alone outperforms inter-layer sharing alone in terms of performance. However, inter-layer sharing is significantly more parameter-efficient, achieving comparable results with fewer parameters. This trade-off underscores the complementary benefits of combining both strategies.
Additionally, we observed that using the GT approach to increase the rank for either intra-layer or inter-layer sharing alone does not lead to significant performance improvements and may even degrade performance. In contrast, for the SS and KE methods, increasing the rank consistently enhances model performance. further emphasizing the benefits of tailored sharing strategies.
Table 11. Different configurations of ranks for our BSLoRA. Boldface denotes the best results in terms the corresponding metrics, and underline means the becond-best performance.
Method
Mode
Ranks
OBQA
ARC-c
HellaSwag
ARC-e
PIQA
WinoG.
BoolQ
SIQA
Avg.
SS
0,8,0
45.00
47.70
77.32
76.64
80.25
70.01
78.10 10
48.41
65.43
SS
0,16,0
45,20
47.10
77.31
77.02
80.14
70.09
78.01
48.82
65.46
SS
0,32,0
45.40
47.35
77.18
76.47
80.09
69.69
78.50
49.18
65.48
GT 0,8,0
43.20
45.99
76.52
75.63
79.11
69.85
74.98
46.16
63.93
Adjust intra
GT 0,16,0
43.80
45.48
75.71
75.55
79.38
69.53
74.16
45.70
63.66
GT 0,32,0
43.60
44.88
76.03
75.34
79.00
70.01
73.00
46.11
63.50
KE 0,8,0
45.40
48.04
77.16
76.35
80.20
70.72
75.90
46.78
65.07
KE 0,16,0
45.80
46.93
77.48
76.73
80.03
70.56
75.93
47.44
65.11
KE
0,32,0
45.20
47.10
77.53
76.39
79.82
70.24
75.96
47.13
64.92
0,0,8
44.60
48.29
77.23
77.02
80.20
69.85
77.74
48.31
65.40
0,0,16
48.72
77.35
76.64
80.09
70.32
78.23
47.95
65.46
SS
0,0,32
44.40
47.78
77.36
76.94
80.03
70.80
78.75
48,21
65.53
GT
0,0,8
44.40
45.56
76.41
75.67
79.38
69.85
75.84
45.75
64.11
Adjust inter
GT
0,0.16
44.00
46.33
76.87
76.14
79.11
69.30
74.40
46.93
64.13
GT 0,0,32
44.40
44.97
76.00
72.73
79.11
69.38
75.11
45.75
63.43
KE 0,0,8
43.80
45.65
75.70
75.63
79.38
70.01
75.66
46.47
64.04
KE 0,0,16
45,80
46.93
76.59
76.77
79.16
70.64
76.57
47.75
65.03
KE
0,0,32
46.00
45.56
76.17
76.30
79.16
70.01
76.79
47.80
64.72
A.5.4. BSLORA PLACEMENT ANALYSIS
To ensure fair evaluation of parameter sharing contributions across module types, conducted controlled ablation study comparing three configurations: (1) applying BSLoRA exclusively to attention modules, (2) restricting BSLoRA to MLP modules, and (3) jointly applying BSLoRA to both attention and MLP modules.
Our experiments results (shown in Table 12) reveal two critical findings. First, BSLoRA achieves significantly higher performance when applied to attention modules alone compared to MLP-only adaptation, suggesting that attention layers exhibit more pronounced parameter redundancy. Second, the combined application of BSLoRA to both attention and MLP modules yields superior results over attention-only tuning, implying that redundancy exists not only across layers (inter-layer) but also across module types within individual layers (intra-layer).
B. Future Work
In the future, we will explore more intelligent parameter sharing modes that can selectively select different modules or different layers for parameter sharing, ultimately further improving the performance of the model.
B.I. LLM versions.
We provide the Hugging Face link of LLMs used in the experiment: Llama 1-7B: 2eo2//eccapda--- research- -arrchh--lma--- 1almmma-e 7B-hf;
hf; Llama 3-8B:
co/meta-
18

---

BSLoRA; Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
Table 12. Different placement modules for BSLoRA. Boldface denotes results terms of the corresponding metrics, and
underline means the second-bes best performance.
Shared Modules
Methods
OBQA
ARC-c
HellaSwag
ARC-e
PIQA
WinoG.
BoolQ
SIQA
Avg.
Overall
BSLoRA (SS)
46.00
56.48
83.09
82.91
81.83
73.72
83,09
48.00
68.39
Attention
BSLoRA (GT)
46.00
57.34
79.78
83.04
81.99
73.80
81.38
48.62
68.99
69.09
BSLoRA (KE)
45.80
56.74
79.90
82.53
81.83
74.03
81.56
48.72
68.89
BSLoRA (SS)
46.20
57.42
79.95
83,33
81.88
73.95
81.83
48.52
69.14
MLP
BSLoRA (GT)
45.80
56.91
79.72
83.25
81.94
73.64
81,80
48.77
68.98
69.01
BSLoRA (KE)
46.40
56.91
79.81
82.95
81.94
73.48
81.50
48.41
68.93
BSLoRA (SS)
46,40
57.17
79.96
82.95
81.94
74.74
83.09
49.03
69.41
Attention
BSLoRA (GT)
46.20
56.83
79.89
82.87
81.94
74.27
82.97
48.36
69.17
69.25
& MLP
BSLoRA (KE)
46.40
56.67
80.04
83.08
82.15
73.64
82.60
48.98
69.18
Meta-LLama-3.. BBB Llama 1-13B: https::/hucginggfaee
wttp:///gggac.....aaa//aa/a/ wttpe/ecyyyaaaaaaaaaaaaaaa11 Qwen2.5-7B: https://huggin waw.ce..innnc.....oo.mmm co /ww//wwn///// (ww.e//wwe2....eemmmmommm Qwen2. 5-
Instrrucct...
B.2. Software and hardware configuration.
Our implementation utilizes the following configurations: PyTorch version 2.1.2, Transformers library version 4.41.0, PEFT
(Purameter-Effiiien Fine-Tuning Tuning) library version 0.11.1, CUDA version 12.4, GPU: NVIDIA V100 GPU with 32GB of memory, NVIDIA A100 GPU with 80GB, Operating System: Ubuntu.
B.3. Datasets and Benchmarks
BoolQ (Clark al., 2019) is dataset for yes/no question answering. It consists of naturally occurring questions paired with passages extracted from Wikipedia. It is part of the SuperGLUE benchmark, suite of challenging NLP tasks. It is used to assess a model's ability to perform reading comprehension and binary classification based on the context provided.
PIQA (Bisk et al., 2020) is dataset designed to evaluate commonsense reasoning about physical interactions. It contains multiple-choice questions related to everyday physical tasks, asking models to choose the most plausible way of completing or describing an action. It used as a benchmark for evaluating the commonsense reasoning abilities of language models, particularly the context of tasks requiring physical understanding.
HellaSwag (Zellers al., 2019) is large-scale dataset for evaluating commonsense reasoning and natural language inference. The task involves selecting the most plausible continuation of a given story event description from multiple choices. It is used to benchmark models on their ability to perform commonsense reasoning, particularly in cases where the correct answer requires understanding context, sequencing, and implications.
WinoGrande (Sakaguchi et al., 2021) is a large-scale dataset for commonsense reasoning, specifically designed to address the limitations of the Winograd Schema Challenge. The task involves resolving pronoun references in sentences, where the correct interpretation requires commonsense knowledge. It is used benchmark for evaluating models on their ability perform pronoun resolution and commonsense reasoning.
ARC-easy (Clark et al., 2018) and ARC-challenge (Clark et al., 2018) are part of the A12 Reasoning Challenge, designed to evaluate model's ability to answer multiple-choice questions that require complex reasoning and background knowledge. They are used as a benchmark for testing advanced question-answering systems, especially those requiring sophisticated reasoning, knowledge integration, and inference capabilities.
OpenbookQA (Mihaylov al., 2018) a multiple-choice question-answerii dataset that focuses on elementary science questions. The dataset comes with "open book" of scientific facts, and models must combine this knowledge with reasoning to answer the questions correctly. It is used benchmark for evaluating model's ability to perform open- domain question answering, where success requires not just knowledge retrieval but also reasoning and application of that knowledge.
Social QA (Sap et al., 2019). often abbreviated as SIQA, is composed of guestion-answer pairs that simulate roal-world ceoemaation-seeiinn dialogues. This dataset is designed to assess the capability of models to engage in coormattion-eekinng
19

---

BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing
conversations, where the model must ask clarifying questions human gather information and then provide answer the original query.
MMLU (Hendrycks et al., 2021) is benchmark designed to assess model's s world knowledge and problem-solving abilities across wide range of subjects. It evaluates models in both zero-shot and few-shot settings, making the tasks more challenging and aligned with human evaluation methods. The benchmark spans 57 subjects, including STEM, humanities, social sciences, and other fields, with difficulty levels ranging from elementary to advanced professional. Each question presents four answer choices, and the task is predict the correct one based on the given instruction.
20


Reviews:
[{"review_id": "Se5QuuTkMM", "reviewer": "Reviewer_BXVq", "summary": "In this paper, the authors proposed BSLoRA that add intro-layer and inter-layer sharing for LoRA to reduce trainable parameters while maintenance the performance. Multiple experiments were conducted on several benchmark datasets and showed slightly better performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment:** \"*Given that LoRA already reduced the trainable parameters dramatically compared to full training, the additional reduction from the proposed intra-layer and inter-layer sharing is relatively small.*\"\n\n**Answer to C1**: Thank you for your insightful comment. While it is true that LoRA significantly reduces trainable parameters compared to full fine-tuning, the memory footprint of LoRA adapters becomes increasingly non-negligible as the size of the pre-trained model and the number of downstream tasks grow. In real-world inference systems, **LoRA adapters are typically not merged with the base model** to support dynamic multi-task serving. For instance, applying LoRA with rank 64 on LLaMA 70B introduces over 1.4 GB of additional memory. As more tasks are deployed simultaneously, e.g., **100 tasks would take about 140G memory, this overhead quickly becomes a major bottleneck.** \n\nTo address this, recent works such as Vera, ShareLoRA, and Tied-LoRA explore further parameter reduction. Building on these efforts, BSLoRA introduces **a more fine-grained parameter sharing** framework that combines local, intra-layer, and inter-layer components. To enable sharing across weights of different shapes, we also propose three shape transformation strategies (SS, GT, KE). This design not only reduces trainable parameters but also improves model performance, as demonstrated in our experiments. From our experiment's results, **BALoRA can reduce over 50% trainable parameters on average and achieve an average 1.25% performance improvement.** Importantly, BSLoRA can be seen as **a flexible superset of LoRA**, offering a tunable architecture that adapts to both performance and efficiency needs.\n\n---\n\n**Comment:** \"*Given the complexity of hyper-parameter tuning and shape transformation (SS, GT, KE), I am not fully convinced that the gain would outweigh the effort.*\"\n\n**Answer to C2**: We appreciate your concerns regarding the complexity of hyperparameter tuning and shape transformation. In practice, **BSLoRA does not introduce additional tuning overhead compared to standard LoRA**—the only tuning required is selecting appropriate rank values for the chosen sharing strategy. Importantly, **all three proposed sharing strategies (SS, GT, KE) consistently reduce parameter count while maintaining or improving performance**, making the selection process flexible and low-effort. In our experiments, we empirically chose rank settings that best fit each sharing method, but our **ablation studies (Tables 7, 8, and 9) show that BSLoRA performs robustly across different configurations**, demonstrating that extensive hyperparameter tuning is not necessary for effective deployment. By default, we adopt the **Kronecker Extension (KE)** strategy in our setup, as it offers the best trade-off between **parameter efficiency and performance**. Overall, BSLoRA is a simple yet effective solution that achieves higher efficiency **without adding tuning complexity**.\n\n---\n**Comment**: \"*Typo in the caption of Figure 1: (left) and (right) should be (top) and (bottom)*\"\n\n**Answer to C3**: Thank you for pointing this out. We will correct the caption of Figure 1 from \"(left)\" and \"(right)\" to \"(top)\" and \"(bottom)\" in the revised version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, the main experimental results, analysis, and ablation study.", "relation_to_broader_scientific_literature": "The introduce intra-layer and inter-layer sharing approach further reduced the trainable parameters of LoRA and could enable fine-tuning of much larger models.", "essential_references_not_discussed": "No, essential references are adequate.", "other_strengths_and_weaknesses": "***Strengths***\n1. The idea of adding intra-layer and inter-layer sharing to further reduce trainable parameters of LoRA makes sense and is technically sound to me.\n2. The authors conducted extensive on experiments, analysis, and ablation study to validate the proposed approach.\n3. Writing is good and easy to follow.\n\n*** Weaknesses***\n1. Given that LoRA already reduced the trainable parameters dramatically compared to full training, the additional reduction from the proposed intra-layer and inter-layer sharing is relatively small. \n2. Given the complexity of hyper-parameter tuning and shape transformation (SS, GT, KE), I am not fully convinced that the gain would outweigh the effort.", "other_comments_or_suggestions": "1. Typo in the caption of Figure 1: \"(left)\" and \"(right)\" should be \"(top)\" and \"(bottom)\".", "questions_for_authors": "Please refer to the \"Other Strengths And Weaknesses\" for the detailed comments and provide additional justification regarding the balance between benefits gained and required effort.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y7SwBxF9Ex", "reviewer": "Reviewer_oFSX", "summary": "This paper introduce a method, sharing lora parameters across local, intra-layer, inter-layer. To address the shape mismatch issues, shape transformation are introduced including slice sharding, gate transformation, KRONECKEREXTENSION. Results on different datasets show the effectiveness of the method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment:** \"*Fairness of Comparison with Baselines: The comparison between BSLoRA and baselines such as ShareLoRA and Tied LoRA may be unfair. ... To evaluate the true effectiveness of BS LoRA’s design, how does applying LoRA to both attention and MLP modules compare to applying it to only one module type (e.g., attention-only or MLP-only)? ....*\"\n\n**Answer to C1:** Thank you for the thoughtful suggestion. We agree that evaluating the contribution of parameter sharing across different module types is important for a fair comparison. To address this, we conducted an additional ablation study, comparing BSLoRA applied to **only the attention modules**, **only the MLP modules**, and **both modules simultaneously**. \n\nThe results show that applying BSLoRA to the **attention modules alone yields better performance** than applying it only to the MLP modules, indicating that **redundancy in attention layers is more significant**. Moreover, applying BSLoRA to **both attention and MLP modules further improves performance** over attention-only tuning. This suggests that **redundancy exists not only across layers but also across different module types within the same layer**, as discussed in Section 2.2. These findings support our core claim: combining **intra-layer and inter-layer sharing** strategies enables more effective reduction of parameter redundancy, while also contributing to better overall performance. The results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ .\n\n---\n\n**Comment:** \"*Rationale for Tied-LoRA Variant Selection*\"\n \n**Answer to C2:** Thank you for raising this important question. After reviewing the original Tied-LoRA paper, we observed that **both TL5 and TL6 perform well** across various tasks. In our own experiments, we replicated both the TL5 and TL6 configurations. Based on the specific training setup and evaluation methods used in our study, **we found that TL5 outperformed TL6 under these conditions**. Consequently, we decided to present the results for TL5 in the main text of the paper. We will clarify this choice in the revised version of the paper and provide a detailed comparison between TL5 and TL6 in the appendix to ensure transparency and consistency in the reported findings. The results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ .\n\n**Inconsistent Optimal Methods Across Setups** Tab 2 results show that no single config among SS, GT, KE consistently outperforms others across tasks, model sizes, or ranks.\n\n**Answer to C3 :**  Thank you for your thoughtful observation. Indeed, the optimal configuration of **SS**, **GT**, and **KE** varies across tasks, model sizes, and ranks. This variability arises due to the **task-specific nature of the benchmarks** and the **differences in model layer architectures**. Our approach aims to provide **flexibility** by offering multiple methods that can be tailored to the requirements of different tasks and model sizes.\n\nFor example, as shown in **Table 2**, the **SS method performs best** on **Llama 1-7B**, while the **KE method performs best on Llama 3-8B**. This flexibility allows users to select the method that fits their deployment context. Moreover, the performance variation across different methods is minimal, highlighting the **robustness of BSLoRA** across diverse settings.\nWe believe this **flexibility** is one of BSLoRA’s strengths, enabling it to be adapted for various deployment scenarios without the need for extensive hyperparameter tuning for each task.\n\n---\n\n**Comment:** \"*The improvements compared with ShareLoRA and Tied LoRA are marginal.*\"\n\n**Answer to C4:** Thank you for your comment. While the performance gains of BSLoRA over other parameter-sharing methods such as ShareLoRA and Tied-LoRA may appear modest, our method introduces a **more fine-grained parameter sharing strategy**. As discussed in Section 2.2 of our paper, redundancy in LoRA parameters exists not only across layers but also **within the same layer across different parameter modules**. However, existing approaches focus only on inter-layer sharing for modules at the same relative position, **overlooking the redundancy among intra-layer components**. \n\nBSLoRA addresses this limitation by jointly implementing **intra-layer and inter-layer parameter sharing**, enabling a more comprehensive reduction in trainable parameters. To overcome the challenge of sharing across weights with mismatched shapes, we further propose **three shape transformation methods** (SS, GT, KE), which allow the model to flexibly align and share information across diverse modules. This design increases the **parameter efficiency** and enhances the model’s ability to capture shared features across tasks. Our experimental results show that BSLoRA not only achieves a further reduction in parameter redundancy but also delivers **consistent performance improvements**, showing a better balance between efficiency and effectiveness than existing sharing-based methods.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No proof.", "experimental_designs_or_analyses": "Partially.\n\n**Concerns about Baseline Comparisons and Method Selection**\n\n1. Fairness of Comparison with Baselines:\nThe comparison between BS LoRA and baselines such as ShareLoRA and Tied LoRA may be unfair. BS LoRA applies LoRA to both attention and MLP modules, whereas ShareLoRA and Tied LoRA are limited to attention modules only. This difference means BS LoRA updates more parameters, potentially inflating its performance relative to the baselines. To evaluate the true effectiveness of BS LoRA’s design, how does applying LoRA to both attention and MLP modules compare to applying it to only one module type (e.g., attention-only or MLP-only)? Does the combined approach yield synergistic benefits from updating both module types, or is the performance improvement merely a result of having more trainable parameters? Including ablation studies (e.g., separate versus joint tuning of attention and MLP modules) would help clarify the contribution of this design choice.\n\n2. Rationale for Tied-LoRA Variant Selection:\nThe paper employs the TL5 variant of Tied-LoRA in its experiments, despite stating that the TL6 variant performs better. What motivated the choice of TL5 over TL6, and how does this decision influence the reported results? A clear explanation of this selection—or additional results using TL6—would provide greater confidence in the findings and ensure consistency between the claims and the experimental setup.\n\n**Inconsistent Optimal Methods Across Setups**\nTab 2 results show that no single config among SS, GT, KE consistently outperforms others across tasks, model sizes, or ranks.\n\n**The improvements compared with ShareLoRA and Tied LoRA is marginal.**", "relation_to_broader_scientific_literature": "The key contributions of BS LoRA are deeply tied to the broader scientific literature on parameter-efficient fine-tuning of LLMs. It builds on the foundational work of LoRA and extends the parameter-sharing ideas pioneered by ShareLoRA and Tied-LoRA. By introducing a block-wise sharing strategy, BS LoRA offers a new approach that seeks to improve upon prior findings—balancing the reduction of trainable parameters with the need for layer-specific adaptability. This work contributes to the evolving field of PEFT, where efficiency remains a critical concern for deploying LLMs in resource-constrained environments.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "While the paper has some weaknesses, as discussed above, it also demonstrates notable strengths that enhance its value. The writing is clear and accessible, effectively simplifying the complex topic of shape mismatching in machine learning models. Additionally, the paper proposes multiple innovative solutions to address shape mismatching, each accompanied by a balanced discussion of its advantages and disadvantages. This thorough approach highlights the authors' expertise and offers practical insights for addressing similar challenges.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please see the above questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8AohY2mZz", "reviewer": "Reviewer_roJb", "summary": "The paper introduces Bi-Share LoRA (BSLoRA), which improves memory efficiency and inference speed by sharing parameters both within a layer (intra-layer) and across layers (inter-layer). The approach also introduces three shape transformation techniques—Slice Sharing, Gate Transformation, and Kronecker Extension—to ensure flexible parameter sharing across different module structures. Experimental results on LLaMA models (7B, 8B, and 13B) show that BSLoRA reduces parameters while improving or maintaining model performance on Commonsense Reasoning and MMLU benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment:** \"*...  settings for Table 1 and Table 2 are not consistent. The ablation study evaluates the effectiveness of the three different modules by removing each component sequentially*.\"\n\n**Answer of C1**:  Thank you for your observation. The settings in Tab 1 and Tab 2 were designed to evaluate the **overall performance of BSLoRA in real-world scenarios**. Therefore, the rank settings in Tab 1 and Tab 2 were selected through extensive experiments to achieve **optimal performance** while maintaining parameter efficiency. In contrast, we chose rank=8 in the ablation study to clearly isolate each component’s contribution. These different configurations address different experimental goals.\n\n---\n\n**Comment:** \"*Could you validate the effectiveness of BSLoRA by comparing it with ShareLoRA at different ranks, while keeping the number of trainable parameters similar?*\"\n\n**Answer to C2**: Thank you for your suggestion. We conducted additional experiments comparing **ShareLoRA at ranks 4, 8, 16, and 32** with **BSLoRA configured to have approximately same trainable parameters**. The results (https://anonymous.4open.science/r/BSLoRA-1E72/) show that at **lower budgets**, only the **KE** outperforms ShareLoRA. This suggests that under tight parameter budgets, **SS and GT have limited capacity to extract useful features**. However, as the rank increases, **all three variants consistently outperform ShareLoRA**, indicating that BSLoRA is more effective at leveraging **redundant capacity** when more parameters are available. These findings further support that there exists **significant redundancy both within and across layers**, and our sharing strategies can lead to **more efficient parameter utilization**, especially as the parameter budget increases.\n\n---\n\n**Comment:** \"*For the hyperparameter setting, how is r selected for different reshaping methods? Why not choose a consistent r across all methods?*\"\n\n**Answer to C3**: Thank you for your insightful comment. SS, GT, and KE each have distinct structural characteristics, so they they naturally require different r configurations to achieve optimal performance. Using a single rank across all methods can be suboptimal, as each interacts with model layers differently. Instead, we determined **empirical best r values** through systematic experiments, balancing parameter efficiency and accuracy across benchmarks.\n\n---\n\n**Comment:** \"*... the performance of GT is comparable to the other two methods (i.e., SS and KE) in Table 1 and Table 2. How can this phenomenon be explained?*\"\n\n**Answer to C4**: Your question is very interesting. One possible explanation is that GT **provides an implicit regularization effect, preventing overfitting to specific layers and encouraging a more generalizable representation**.  Additionally, the learned input and output transformations may help refine the information flow, ensuring that key features are preserved while removing redundant ones.\n\n**Comment:** '*“... local component of LoRA performs best on HellaSwag, ARC-e, BoolQ, and SIQA, while the intra-layer shared component excels on PIQA and ARC-c.” Could you provide some insight into this phenomenon?*'\n\n**Answer to C5**: Thank you for your insight. The varying effectiveness likely arises from differing task requirements. Tasks like PIQA and ARC-c may \"share similar patterns\", so consistent features from shared parameters help. Meanwhile, tasks like HellaSwag and BoolQ may \"demand fine-grained contextual understanding\", benefiting more from local parameter usage. This underscores BSLoRA’s hybrid design in adapting to different task needs.\n\n---\n\n**Comment:** “*The results in Figure 4(c) show that modifying the constant improves both expressiveness and information content.” However, this conclusion is not obvious from Figure 4(c). How was it derived?*”\n\n**Answer to C6**: Thank you for your comment. The current radar plot in Figure 4(c) does not clearly show the improvement. We inferred it by comparing **average performance** across methods. The results used in this analysis are shown in the link above. We will clarify this in the revised paper and improve either the figure or the accompanying explanation.\n\n---\n\n**Comment:** \"*In Table 10, different rank settings achieve similar performance, and the claim that 'Assigning a lower rank to the local component and a higher rank to the shared parameters yielded better performance, further illustrating the redundancy in the standard LoRA parameters' is not obvious.*\"\n\n**Answer to C7**: Thank you for your observation. Adjusting only the local component's rank merely brings minor performance improvement or worsen, indicating low-rank local component can achieve comparable performance. Besides, keeping a low-rank local component and adjusting shared components consistently results in comparable performance. Therefore, we can assign a lower rank to the local component and high ranks to shared components to achieve better parameter efficiency.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The paper claims reduced memory overhead, but does not provide empirical results on: Inference latency (Does BSLoRA speed up inference compared to LoRA?)", "theoretical_claims": "No explicit mathematical proofs are presented, so there are no incorrect derivations to check.", "experimental_designs_or_analyses": "- Even though the authors have conducted a contribution analysis by setting the rank of one sub-LoRA matrix to 8, the settings for Table 1 and Table 2 are not consistent. The ablation study evaluates the effectiveness of the three different modules by removing each component sequentially.\n\n- Could you validate the effectiveness of BSLoRA by comparing it with ShareLoRA at different ranks, while keeping the number of trainable parameters similar?", "relation_to_broader_scientific_literature": "The key contributions of the paper relate to prior work in parameter-efficient fine-tuning (PEFT), particularly in LoRA-based architectures. The paper builds on existing research in LoRA parameter sharing, addressing its limitations through a new bi-sharing approach that combines intra-layer and inter-layer weight sharing. While the paper contributes to efficiency, it would benefit from a more explicit connection to pruning or sparsity techniques in neural networks, as well as a deeper discussion on its applicability to broader PEFT techniques beyond LoRA .", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**  \n\n- The Bi-Share LoRA (BSLoRA) approach is interesting in how it combines intra-layer and inter-layer parameter sharing, addressing key limitations in LoRA-based fine-tuning.  \n- The method improves upon existing approaches such as VeRA, Tied-LoRA, and ShareLoRA, making it a significant contribution to parameter-efficient fine-tuning (PEFT).  \n- The paper is well-structured, with clear explanations of the methodology and mathematical formulations.  \n\n**Weaknesses:**  \n\n- The ablation study is relatively insufficient. See the section above, *\"Experimental Designs or Analyses,\"* and the section below, *\"Questions for Authors,\"* for details.", "other_comments_or_suggestions": "- Could you valid the effectiveness of BSLoRA by comparing with ShareLoRA with different ranks, under the case of similar trainable parameter?", "questions_for_authors": "- For the hyperparameter setting, how is r selected for different reshaping methods? Why not choose a consistent r across all methods?\n\n- The Gate Transformation (GT) method yields a rank value equivalent to the local rank plus 2, likely due to one-rank gates causing some information loss. However, the performance of GT is comparable to the other two methods (i.e., SS and KE) in Table 1 and Table 2. How can this phenomenon be explained?\n\n- “Specifically, the local component of LoRA performs best on HellaSwag, ARC-e, BoolQ, and SIQA, while the intra-layer shared component excels on PIQA and ARC-c.” Could you provide some insight into this phenomenon?\n\n- “The results in Figure 4(c) show that modifying the constant improves both expressiveness and information content.” However, this conclusion is not obvious from Figure 4(c). How was it derived?\n\n- In Table 10, different rank settings achieve similar performance, and the claim that 'Assigning a lower rank to the local component and a higher rank to the shared parameters yielded better performance, further illustrating the redundancy in the standard LoRA parameters' is not obvious.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRhZTumest", "reviewer": "Reviewer_qgmi", "summary": "This paper proposes Bi-Share LoRA (BSLoRA), a parameter-efficient fine-tuning approach for large language models (LLMs). The key idea is to improve upon standard Low-Rank Adaptation (LoRA) by introducing intra-layer and inter-layer parameter sharing to reduce the number of trainable parameters while maintaining or improving model performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment:** \"*Limited empirical improvements: The performance gain is marginal, making it unclear whether the complexity trade-off is justified.*\"\n\n**Answer to C1**: Thank you for your comment. Please refer to R2C4. \n\n---\n\n**Comment:** \"*No throughput/memory efficiency evaluation: The paper claims efficiency improvements, but no evidence of speedup or memory savings is presented in the main paper.*\"\n\n**Answer to C2**: Thank you for your comment. The goal of our method is to reduce parameter redundancy through effective parameter sharing. Our experimental results demonstrate that **BSLoRA achieves better performance with fewer trainable parameters**, indicating a significant improvement in **parameter efficiency**. In the appendix, we provide a detailed comparison of the **memory usage** between standard LoRA and BSLoRA under varying numbers of concurrently deployed downstream tasks. As shown in Figure 6, BSLoRA consumes **substantially less GPU memory during inference**, enabling **more downstream tasks to be deployed simultaneously**. This highlights the practical value of our method in **multi-task inference scenarios**, especially under memory-constrained conditions.\n\nWe also test the inference speed comparison on the dataset of wikitext2 and ptb, the results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ . Results indicate that both GT and KE can achieve inference acceleration. \n\n---\n\n**Comment:** \"*Not highly novel: The method mainly repackages known techniques (parameter sharing, Kronecker product, low-rank adaptation) rather than introducing fundamentally new ideas.*\"\n\n **Answer to C3**: Thank you for the thoughtful question. As pre-trained language models continue to scale up, even parameter-efficient fine-tuning methods like LoRA introduce a substantial number of trainable parameters. Moreover, in real-world inference settings, LoRA adapters are typically kept separate from the backbone model to support simultaneous deployment of multiple downstream tasks. This makes reducing the memory footprint of LoRA adapters a critical challenge. Existing methods such as ShareLoRA focus only on **inter-layer parameter sharing**, neglecting the **intra-layer redundancy** that also contributes significantly to parameter inefficiency. **BSLoRA addresses this gap by introducing both intra-layer and inter-layer sharing mechanisms**, aiming to further reduce redundancy and achieve comparable or even improved performance with fewer trainable parameters. To overcome the challenge of sharing parameters with incompatible shapes, we introduce **three lightweight but effective shape transformation techniques** (SS, GT, KE), enabling flexible parameter sharing across diverse model components. While each individual technique may not be novel, their **systematic combination and targeted application to large language model fine-tuning** constitutes a meaningful contribution. Looking ahead, we plan to explore even more advanced sharing strategies to further enhance parameter efficiency.\n\n---\n\n**Comment:** \"*Limited practical advantage: The main benefit only appears in large-scale multi-task settings (>200 tasks), which isn't a common real-world scenario.*\"\n\n**Answer to C4:** Reducing the size of LoRA adapters is practically important not only in large-scale multi-task settings (e.g., >200 tasks), but also increasingly so as model sizes continue to grow. For instance, applying LoRA with rank 64 to all linear layers in LLaMA 70B consumes approximately 1.4 GB of memory, and this rises to 4.5 GB for a 405B model. When deploying more than 10 tasks concurrently—common in real-world cloud services or multi-user systems—the cumulative memory usage of LoRA adapters alone can reach 45 GB, posing a significant burden for inference servers and edge devices alike. While smaller models may require hundreds of tasks to reach memory bottlenecks, larger models hit these limits even with far fewer concurrently deployed adapters. This makes parameter reduction a critical bottleneck for scaling both in terms of deployment cost and system responsiveness. BSLoRA addresses this challenge by reducing memory usage by over 50% compared to standard LoRA, enabling significantly more efficient multi-task deployment without sacrificing performance.", "claims_and_evidence": "The paper makes the following key claims, and most of them are supported by evidence in the experiments:\n- Claim 1: BSLoRA reduces trainable parameters.   \n  - The results show that BSLoRA achieves around 50% parameter reduction while maintaining comparable or slightly improved performance over standard LoRA.\n  - Limitation: The improvement is quite marginal, which raises concerns about whether the added complexity justifies the gains.\n- Claim 2: The proposed shape transformation techniques enable flexible sharing.\n  - The introduction of Kronecker Extension, Gate Transformation, and Slice Sharing provides different strategies for handling parameter shape mismatches, making parameter sharing feasible across different model components.", "methods_and_evaluation_criteria": "The proposed BSLoRA method aligns with the general objective of improving parameter-efficient fine-tuning (PEFT) in LLMs by reducing trainable parameters while maintaining model performance.", "theoretical_claims": "There are no deep theoretical contributions or proofs in this paper.", "experimental_designs_or_analyses": "- The benchmarks (commonsense reasoning and MMLU) are widely used in LLM research and provide a fair basis for evaluating performance.\n- The baselines (LoRA, VeRA, VB-LoRA, ShareLoRA, Tied-LoRA) are appropriate and competitive, ensuring that comparisons are meaningful.\n- The absence of throughput, training speed, and memory usage comparisons in the main paper weakens the argument for parameter efficiency.", "relation_to_broader_scientific_literature": "The paper is related to prior work on parameter-efficient fine-tuning (PEFT):  \n- LoRA (Hu et al., 2022): The foundational method that BSLoRA extends.  \n- VeRA (Kopiczko et al., 2024) and VB-LoRA (Li et al., 2024): Parameter-sharing approaches that BSLoRA compares against.  \n- MultiLoRA (Wang et al., 2023) and HydraLoRA (Tian et al., 2024): Related work on adapting LoRA for multi-task learning.  \n\nthe core techniques used (low-rank factorization, Kronecker product, and parameter sharing) are not new.", "essential_references_not_discussed": "The paper covers the most relevant prior work.", "other_strengths_and_weaknesses": "## Strengths\n- The proposed multi-layer parameter-sharing strategy is well-motivated and provides clear parameter savings.\n- The experiments are thorough, with evaluations of multiple models and tasks.\n- Shape transformation techniques (Kronecker Extension, Gate Transformation) effectively solve parameter sharing constraints.\n\n## Weaknesses\n- Limited empirical improvements: The performance gain is marginal, making it unclear whether the complexity trade-off is justified.\n- No throughput/memory efficiency evaluation: The paper claims efficiency improvements, but no evidence of speedup or memory savings is presented in the main paper.\n- Not highly novel: The method mainly repackages known techniques (parameter sharing, Kronecker product, low-rank adaptation) rather than introducing fundamentally new ideas.\n- Limited practical advantage: The main benefit only appears in large-scale multi-task settings (>200 tasks), which isn't a common real-world scenario.", "other_comments_or_suggestions": "No other suggestion.", "questions_for_authors": "- How does BSLoRA impact training and inference speed?\n- How does BSLoRA compare to LoRA in memory consumption during training?\n- The performance gains are modest, questioning whether the increased architectural complexity is worth the trade-off.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces BSLoRA, a new variant of LoRA fine-tuning aimed at reducing the overall number of adapter parameters while preserving/improving performance.  The approach of weight tying to reduce LoRA parameters is not new, yet BSLoRA innovates on previous work through two major contributions: i) a decomposition of adapters into (Local, Intra, Inter)-layer tied parameters--Local is at the vanilla LoRA level, Intra is at the Transformer-layer level, and Inter is at the global level--and (ii) three shape transformation techniques to deal with mismatched tensor dimensions in the Intra- and Inter-layer adapters.  By reducing the dimension of Local-layer adapters while retaining larger dimensions for the Intra- and Inter-layer adapters, BSLoRA largely outperforms related parameter-tying LoRA variants (VeRA, VB-LoRA, ShareLoRA, and Tied-LoRA) for Llama-3 and Llama-1 models across the widely used common sense reasoning and MMLU (0- and 5-shot) benchmarks.  The authors perform additional experiments demonstrating the effective rank of BSLoRA, the practicality memory savings using BSLoRA for massive-scale LoRA adapter service (e.g., Multi-LoRA), and ablations for the various shape transformation methods (evaluated on GSM8K and MMLU).

The paper is well written, the method is an intuitive improvement over existing methods, and the experiments extensively demonstrate the soundness of the approach.  The majority of reviewers agree that the work is worthy of acceptance.  Reviewers BXVq and qgmi both raised concerns of the need for reducing LoRA memory overhead, but the authors addressed these concerns, resulting in qgmi raising their score.  While I would have liked to see other model families evaluated to explore the effectiveness of shape transformations and other fine-tuning datasets than the Alpaca dataset utilized, the overall experiments are comprehensive and convincing.  Reviewer oFSX was in favor of weak reject, but did not participate in either the AC-reviewer or author discussions throughout the review process.

For the camera ready, I ask the authors to include a discussion in how their method can be integrated with S-LoRA in future work and extensively highlight the MultiLoRA results in the introduction to motivate the practical need for BSLoRA.
Decision: Accept (poster)