BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
Authors: Shiwei Li
Xiandi Luo
Haozhao Wang
Xing Tang
Ziqiang Cui
Dugang Liu
Yuhua Li
Yichen Li
Xiuqiang He
Ruixuan Li
Venue: ICLR 2026 Poster (2026)
Keywords: LoRA
Block Matrix Multiplication
Higher Matrix Rank
Primary area: optimization
TL;DR: This paper proposes BoRA to enhance the rank of LoRA weights from the perspective of block matrix multiplication.

Abstract: Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). 
It approximates the update of a pretrained weight matrix $W\in\mathbb{R}^{m\times n}$ by the product of two low-rank matrices, $BA$, where $A \in\mathbb{R}^{r\times n}$ and $B\in\mathbb{R}^{m\times r} (r\ll\min\{m,n\})$.
Increasing the dimension $r$ can raise the rank of LoRA weights (i.e., $BA$), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters.
In this paper, we propose **Block Diversified Low-Rank Adaptation (BoRA)**, which improves the rank of LoRA weights with a small number of additional parameters.
Specifically, BoRA treats the product $BA$ as a block matrix multiplication, where $A$ and $B$ are partitioned into $b$ blocks along the columns and rows, respectively (i.e., $A=[A_1,\dots,A_b]$ and $B=[B_1,\dots,B_b]^\top$).
Consequently, the product $BA$ becomes the concatenation of the block products $B_iA_j$ for $i,j\in[b]$.
To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix $\Sigma_{i,j} \in \mathbb{R}^{r\times r}$ for each block multiplication, resulting in $B_i \Sigma_{i,j} A_j$.
By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by a factor of $b$ while only requiring $b^2r$ additional parameters. 
Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability.

Published as a conference paper at ICLR 2026
BoRA: TOWARDS MORE EXPRESSIVE
LOW-RANK ADAPTATION WITH BLOCK DIVERSITY
Shiwei L Xiandi Luo', Haozhao W Xing Tang", Ziqiang Cui Dugang Liu Yuhua Li', Yichen Li', Xiuqiang He².*, Ruixuan Li
Huazhong University Science and Technology Shennzen Technology University
CCiyy University of Hong Kong SShenzhen University (lishiwei, ha_ man_, rxli @hust. wwwust.euu.....m. edu. en, hexiugiaaggezzu.eduu
ABSTRACT
Low-rank adaptation (LoRA) is parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix W E mR by the product of two low-rank matrices, BA, where A E gr*= and B E [r can raise the rank of LoRA weights (i.c., BA), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters. In this paper, we propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with small number of additional parameters. Specifically, BoRA treats the product B as a block matrix multiplication, where A and B partitioned into b blocks along the columns and rows, respectively (i.c., A = [A and = [B.... B.] T. Consequently, the product BA becomes the concatenation of the block products B,A, for i, E [6]. To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix Σjjj Rrxx for each block multiplication, resulting in , Ay. By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by factor of while only requiring 6 additional parameters. Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability. The code available https: / gi thub. com http//eooollloo (11eopold122/bbooo iclr26
(r < min m., n }) Increasing the dimension r
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks (DeepSeek-Al 2024 OpenAl 2024). However, the state-of-the-art models usually contain vast number of parameters, making full fine-tuning (FFT) for downstream tasks extremely expensive in terms of both training time and memory usage, thereby limiting their practical deployment (Hu ct al. 2022 Lester et al., 202I). To mitigate this issue, purameter-efficinne fine-tuning (PEFT) methods have been developed reduce the number of trainable parameters and decrease fine-tuning costs (Xu et al. 2023). These methods include techniques such as prompt tuning (Lester et al.) 2021). parallel adapters (He al. (022)) and sequential adapters (Houlsby al., (019). Among these, low-rank adaptation (LoRA) (Hu et al. 2022) has gained popularity due to its ability to avoid additional inference latency. Specifically, LoRA freezes the pretrained weight matrix W E mR and learns two smaller low-rank matrices to approximate the weight update as AW = B/~BA, where A E " B IR' mxx, is the LoRA rank & min(m, n n)), and an adjustable scaling factor. For simplicity, we omit the coefficient o/r in the subsequent description.
Despite its effectiveness, LoRA generally exhibits a performance gap compared FFT, which is typically attributed to the limited rank of LoRA weights (Hu et al.) 2022; Ren et al. 2024 Huang et al. 2025). Specifically, the rank of LoRA weights is constrained as rank A AW S
minkrank([A BB] < r. Numerous studies have also demonstrated that increasing the rank of
LoRA weights typically improves fine-tuning performance (Ren et al. 2024 Liu et al.) 2024 Suag et al. 2025). This trend is further corroborated by experimental results, shown in Tables
and Recently, Zeng and Lee (2024) further explored LoRA's expressive power, using the LoRA
rank to quantify the approximation error between the LoRA weights (i.e., AW B.A) and the

---

Published as a conference paper at ICLR 2026
B
B
B B
0
2 A
B 12 1334
B
B242
B2A3
Bz42
2,
B2
B Ag
B
Bz42
B3
0
0
3343
3 A
B 3,2 3,3
(a) LoRA
(b) MELoRA
(c) BoRA
Figure 1: An illustration of BoRA in comparison to LoRA (Hu al. 2022 and MELoRA (Ren
et al. 2024). (a) LoRA weights (i.e., B A) can be represented by block matrix multiplication, where the off-diagonal blocks to break the correlation between different blocks and enhance the rank of LoRA weights. (c) BoRA introduces diagonal matrix for each block multiplication to enhance the diversity among block products. Notably, LoRA and MELoRA are essentially specific instances of BoRA. BoRA reduces to LoRA when all Σ,j I, and MELoRA when Σ.,- = I for and Σ., for * j. where denotes the identity matrix.
A1, A2, Ag. and B [Bi B2, Ba] T. (b) MELORA zeros
assumed optimal weight update. Their findings suggest that higher rank of LoRA weights leads to smaller approximation error. Consequently, efficiently increasing the rank of LoRA weights has become a consensus strategy for improving fine-tuni seeetunnigg performance.
In this paper, we analyze the rank of LoRA weights through the lens of block matrix multiplication. Let matrices A and be evenly partitioned into three blocks along the columns and rows, respectively, ie., A [A1, Ag. As] and B [B,, B2, Bs] The product BA then be represented as the concatenation of the block products B,Ay. as shown in Figure I(a) This structure limits the independence of different block products, thereby constraining the rank of BA. Specifically, the rank of BA depends on the number of linearly independent row vectors; however, the difference between block rows lies solely in the of different B, This implies that the blocks in row be obtained from those in another row using the same transformation. For instance, left left-multiplying the blocks of the first row by B, (assuming B, exists) generates the second row, indicating that the second row does not contribute to the rank. A similar issue arises with the columns.
To address this issue, we propose Block-Diversifier Low-Rank Adaptation (BoRA), which ef- fectively breaks the correlation between the block products B,A, in different rows or columns. As illustrated Figure [(c) BoRA introduces a unique diagonal matrix, Ez.j. for each block multipli- cation, resulting in increase the rank of LoRA weights by factor of b, while requiring only 62r additional parameters. In contrast, MELoRA (Ren et al., 2024) achieves similar effect by disrupting the correlation of different blocks through zeroing out the off off-diagonal block products, illustrated in Figure [[b) Although this increases the rank of LoRA weights, may limit LoRA's expressive power due the presence of numerous zero entries. Instead, BoRA A ensures block diversity by utilizing biock-wise diagonal matrices, without compromising LoRA's expressiveness. Importantly, both LoRA and specific instances of BoRA. Specifically, BoRA reduces to LoRA when
Aj. Assuming matrices A and B are divided into b blocks, BoRA will
MELoRA can be all D, = I, and to MELoRA when Σi = for = and E, = 0 for &f j. where I denotes the identity matrix.
Our contributions can be summarized as follows:
We analyze LoRA from the perspective of block matrix multiplication, revealing that the rank of LoRA weights is constrained due to correlations between different block products.
We propose BoRA, which break the correlation among different block products with block- wise diagonal matrices. By dividing matrices A and B into blocks, BoRA increases the rank of LoRA weights by factor of b, requiring only 1 additional parameters.
2

---

Published as a conference paper at ICLR 2026
We conduct extensive experiments multiple models and datasets, demonstrating that BoRA consistently outperforms LoRA and its variants. Using a similar number of trainable parameters, BoRA can achieve 2-4% accuracy improvement over LoRA.
2 RELATED WORK
To reduce fine-tuning overhead, LoRA (Hu et al. 2022) decomposes the weight update, AW E R into two low-rank matrices, E Rr** and B E |mxrr where the LoRA rank r determines the number of trainable parameters. LoRA can be integrated into a model without altering its architecture or increasing inference overhead. contrast, DoRA (Liu et al. 2024) decomposes pretrained weights into direction and magnitude components, learning the magnitude via a trainable vector and updating the direction with LoRA. Recent studies have explored various aspects of LoRA to improve its performance, such as the scaling factor (Kalajdzievski 2023), initialization methods (Meng et al. 2024, Wang et al. 2024), learning rates (Hayou et al., 2024), and dynamic parameter allocation Li et al. 2024). For example, LoLA (Kalajdzievski 2023) modifies the scaling factor to o/vF for more stable fine-tuning. PíSSA (Meng et al. 2024) and LoRA-GA (Wang et al.
(Zhang al. 2023
2024) conduct singular value decomposition (SVD) pretrained weights and sampled gradients
initialize the matrices and of LoRA. LoRA+ (Hayou et al. 2024) suggests that using a higher learning rate for matrix B can improve fine-tuning performance. AdaLoRA (Zhang et al.] (023) adaptively adjusts the LoRA rank for different layers during fine-tuning, allocating more parameters important layers within fixed parameter budget. VB-LoRA (Li al. 2024) composites all the low-rank matrices of different LoRA layers from a shared vector bank.
In this paper, we focus on LoRA variants that efficiently enhance the rank of LoRA weights. For example. HiRA (Huang et al. (2025) and KronA (Edalati et al.) (2023) use the Hadamard and Kronecker products, respectively, y. improve the rank of LoRA weights. ReLoRA (Lialin et al. (024) periodically merges learned LoRA adapters into the pretrained weights to increase the rank of weight updates. MELORA (Ren et al. 2024) achieves a higher rank by stacking low-rank matrices along the diagonal. However, MELORA introduces many zero values, which can significantly reduce the expressiveness of LoRA, as shown in Figure [(b] In contrast to previous approaches, we analyze the rank of LoRA weights through the lens of block matrix multiplication. Our proposed BoRA increases the diversity of block products by introducing unique diagonal vectors for each block multiplication. Notably, both LoRA and MEL.RA A special cases of BoRA, illustrated in Figure [ Furthermore, MoFLoRA (Luo et al.] 2024) trains multiple LoRA adapters as distinct experts and combines their knowledge via routing network. HydraLoRA (Tian et al., (2024) improves on this by sharing the matrix A across the MoELoRA framework for more efficient adaptation. Essentially, MoELoRA can also be viewed as a form of block matrix multiplication. The main difference between BoRA and MoELoRA lies in the way matrices A and B are partitioned, which will be discussed in detail in Section 3.5 Additional discussions of other PEFT-related methods are deferred to Appendix
METHODOLOGY
3.1
BLOCK MATRIX MULTIPLICATION
Block matrix multiplication is a fundamental operation that partitions matrices into smaller sub- matrices for efficient multiplication. Let M E Raxr and N Brxn, where M and N are evenly divided into bm x b, and b, x bn sub-matrices, respectively. Then, the product P = MN can be computed block by block, with each block Peς (i E [bm].j [b,.]) calculated as follows:
by
Prj =
Ne.j-
k=1
In this paper, the matrices A "x and B E R of LoRA are evenly partitioned into b blocks along the columns and rows, respectively (i.c., A = ......: Ab] and = [BI....,Bi Bb]T). Consequently, each block of the LoRA weights AW;, (i.j E [b]) can be expressed as:
AW., = B.Ay.
(2)
Previous studies have shown that the rank of LoRA weights significantly affects the fine-tunin fine-tuning
performance (Ren et al. 2024 Huang al. 2025 Lialin et al. 2024). However, the rank of LoRA
3

---

Published as a conference paper at ICLR 2026
weights is generally limited by the dimension (i.e., the LoRA rank), regardless of the dimensions m and n, as follows:
rank (A)) = rank(BA) & min(rank(A)) rank r.
(3)
From the perspective of block matrix multiplication, this issue primarily stems from the lack of independence between block products (B;4j) across different rows columns. As shown in Figure [(a] the block products share the same B; for each row and the same A, for each column. Taking rows as an example, the rank of AW depends on the number of linearly independent row vectors. However, the difference between block rows lies only in the use of different B, This implies that blocks in one row can be derived from blocks in another row using the same transformation, For example, in Figure (a). if rank B1) = r, then B exists. meft-luuiplyiing the first row block by Bg B2 generates the second row, indicating that the second row does not contribute to the rank. A similar issue applies the columns. Breaking the correlation between these blocks can increase the rank of the weight matrix, thereby improving the expressiveness of LoRA.
3.2 BLOCK- DIVEERIIVESDIIED LOw-RANK ADAPTATION
To break the correlation between different block products, propose Block-Diversified Low- Rank Adaptation (BoRA). Specifically, BoRA introduces a unique diagonal matrix for each block multiplication to enhance block diversity, shown in Figure Assuming A and B are divided into blocks along columns and rows, BoRA will additionally learn set of diagonal matrices {Eig R j.j E [b]], such that the multiplication of each block pair is computed as follows:
AW Jj-
These block-diagonal matrices, E, amplify the differences in block products across rows or columns, thus enhancing the expressiveness of LoRA. Therefore, the core concept of BoRA lies in learning block-wise diagonal matrices (E..) E T2" i,j E [b]], where the corresponding parameters are represented by a dineesiimniii tensor Raxbbrr To ensure a can be effectively optimized with the same learning rate as the matrices A and B, we initialize a using the same Kaiming initialization (He et al. 2016) applied to the matrix Furthermore, to facilitate the learning of Σ, we normalize o by its mean absolute value and apply the exponential function to generate Σ as follows:
[|T Σ, = Diag(Expi Mav(a) )),
(5)
where Mav(o) ||ol: denotes the mean absolute value of a, Exp(-) denotes the exponential function, and Diag(.) denotes the diagonalization function, which converts a vector into a diagonal matrix. Since or [[][[]] is initialized with small variance, the differences between o [i]][]] across blocks minimal, which limits the diversity between blocks. Normalizing by the mean absolute value of can reduce the impact of its small initialization value on the distribution and optimization of Σ. Additionally, because Norm(a ()]]]) = Mav(>)
is still zero-centered, zero or near-zero values could nullify the values in B; and Ay- To prevent information loss associated with a zero value in Σ, we further apply the exponential function o, ensuring that contains only positive values and preventing any loss of information due to zero or near-zero values.
3.3 RANK UPPER BOUND OF BoRA
In the previous discussion, we presented the motivation and formulation of BoRA by expressing its weight as the concatenation of block products (B,A,) In fact, the weight BoRA can also be represented the product of three matrices, shown below:
AW = A'Y'A',
(6)
where A' E R2brxn and B' E Rmxbr are diagonal block matrices formed from the blocks
{Ai...., Ao} and (Bi...., BB), respectively. all Egy for i,j [b]. It clear that the rank of each of these three matrices is bounded above by br Using the properties rank in matrix multiplication (i.e., rank( BA) I min (aankran((A) (A), rank [B) }.
hhr is the matrix obtained by concatenating
can derive upper bound for the rank of the weights in BoRA, as stated in Proposition] 1
4

---

Published as a conference paper at ICLR 2026
Proposition (The Rank Upper Bound of BoRA). Using the low-rank matrices A [A
R" and B = [Bi..... Bu] E Rmxx, along with a set of diagonal matrices {Et,5 E Rexr i.j [6]], the weight update generated by BoRA, denoted AW, satisfies
rank(AW) S min(m (,, br).
(7)
where the rank upper bound can be achieved when all three mulrices are full rank (i.e., rank( aank(B)) min(m, br rank( = min(n, br), and rank(>") = br).
The detailed theoretical justification for Proposition [ is provided in Appendix F Note that, by using the same matrices and B, the rank of the LoRA weights is constrained by r. Proposition demonstrates that BoRA requires only 6²r additional parameters to increase the weight rank by factor of b. In contrast, LoRA needs (m + n n) parameters to achieve the same bound.
On the other hand, to achieve a target rank It br, the number of parameters required by BoRA is N -m + 7 + b²)r = R( ( + b), and this quantity is minimized when = v/m + n. From the perspective of parameter efficiency, this suggests that setting R ym + n is good practical choice. This optimal value arises from natural trade-off: the achievable rank of BoRA (i.e., br) increases linearly with b, while the additional parameter cost (i.e., 6²r) grows quadratically. Once b exceeds vm + n, further increasing b obtain higher rank becomes less efficient than simply increasing the base rank r directly. Therefore, in practice, one can simply set b = |vnl for BoRA.
3.4 EFFICIENT FORWARD PROPAGATION OF BoRA
In this section, we discuss the forward propa- gation process of BoRA. Assuming the input token is E IR", the forward propagation of LoRA is defined as Y WX + BAX, where BAX represents the LoRA output. Differently, BoRA divides the input token into several seg- ments to efficiently perform block matrix mul- tiplication. As shown in Figure if the ma- trices A and B are partitioned into blocks, BoRA also evenly divides the input token X into segments. These segments, denoted as ( ].). are processed produce b out- put segments, {Y1...., ,Y),, which are then con- catenated to form the final BoRA output. Each output segment Y, is of the following form:
Y RM
Y
B
B2
B
Pretroined weight
W E R m
A1
A
A3
Figure 2: BoRA divides the input token X into sev- eral segments to efficiently perform block matrix multiplication. The dotted line connecting Aj and B, represents the trainable diagonal matrix Σi,j-
5
Y, = B Σ Xx
(8)
Notably, compared to LORA, BoRA only re- quires the additional multiplication of the diagonal matrix Σj, B with the vector A RT. Since E,, is a diagonal matrix, this operation can be efficiently performed through element-wise multiplication. Formally, the floating-point operations (FLOPs) per token in the forward propagation of LoRA is given by mn + (m+ n)r, where mn and (m + n)r represent the computational overheads of the pretrained model and the LoRA module, respectively. Taking this additional overhead into account, the FLOPs b2r represents the extra computation introduced by the block-wis: diagonal matrix. Notably, the values of man + (m + n)r and mm + (m + n)r + b2r are also the number of trainable parameters in LoRA and BoRA, respectively. This indicates that the computational density of BoRA is equivalent to that of LoRA. It is important to note that, since < min(m, n)., the additional memory and computational costs of BoRA typically negligible.
token in the forward propagation of BoRA is mn - (m + n)r + b²r, where
3.5 COMPARISON WITH LORA AND MELORA
In this section, we analyze the relationships between BoRA. LoRA (Hu et al. (022), ), and MELoRA (Ren et al. 2024). As shown in Figure [ multiplications.
all three methods can be represented as block matrix
5

---

Published as a conference paper at ICLR 2026
LoRA is actually a special case of BoRA, where all Σ., matrices set to the identity matrix I. In contrast, the Ei, matrices in BoRA follow different distributions across various i and j. introducing diversity among the block products and enhancing the expressiveness of BoRA compared to LoRA. MELORA enhances LoRA by setting the off-diagonal blocks to zero, thereby breaking correlations between different blocks and increasing the rank of the LoRA weights. Similarly, MELoRA is also special case of BoRA, where M = I for i = j and Eigj = 0 for i * j. In contrast to MELoRA, which zeros out the off-diagonal blocks, BoRA employs block-diagonal matrices increase block diversity, thereby improving the rank of LoRA weights without introducing any zero values.
On the other hand, MoELoRA (Luo et al. 2024) is another form of block matrix multiplication for LoRA. In this paper, we partition the matrices A = AA.....,Ar As] and B [BI.... .bll into blocks along columns and rows, respectively. Conversely, if A [A1,..., A&] and B = [B....., B₂) are partitioned along the rows and columns, we can derive the LoRA weights as follows:
AW
AR
where each Ag can be viewed as an expert. Further, MoELoRA integrates LoRA with the Mixture- of-Experts architecture, using routing network to adaptively combine the knowledge of different experts AB)) In the next section, we compare BoRA's performance with these methods in detail.
4 EXPERIMENTS
In this section, we evaluate BoRA on three benchmarks using various model architectures, and then perform ablation studies to assess its scalability and visualize its singular values.
4.1 EXPERIMENTAL SETTINGS
Models and Datasets. First, assess BoRA's natural language understanding (NLU) capability on the GLUE benchmark (Wang et al. 2019), using RoBERTa Base (Liu et al. (019)) and RoBERTa- Large (Liu et al. 2019). The GLUE benchmark consists of eight sub-tasks, each with its training and test sets. For each sub-task, we fine-tune the model on the training set and evaluate its accuracy on the corresponding test set. Next, we evaluate BoRA's natural language generation (NLG) capabilities on mathematical reasoning (Math (0K) (Hu et al. 2023) and commonsense reasoning (Commonsensel 1 17KK) benchmarks (Hu et al., 2023). Both benchmarks include training corpus and multiple test sub-tasks. For each benchmark, we fine-tune the models on the training data and then assess their performance across all sub-tasks. To demonstrate the versatility of BoRA, we perform experiments across various model architectures and scales, including Gemma-7B (Mesnard et al. 2024), LLaMA-3-8B (Dubey et al. 2024), and Qwen2.5-144 (Yang et al. (024).
Baseline Methods. BoRA is compared with several baseline methods demonstrate its effective- ness, including LoRA (Hu et al. 2022), along with several LoRA variants: DoRA (Liu et al. 2024). MELoRA (Ren al. (024) and HydraLoRA (Tian et 2024). DoRA decomposes pretrained weights into magnitude and direction components. It learns the magnitude by training a learnable vec- tor and uses LoRA update the direction, enhancing learning capacity while maintaining parameter efficiency. MELoRA introduces mini-ensemble low-rank adapters that collectively achieve high-rank expressive power while requiring far fewer trainable parameters than standard LoRA. HydraLoRA improves upon LoRA by employing asymmetric architecture that increases parameter efficiency. This architecture uses one A matrix and multiple B matrices, which are combined through a router. These three methods represent distinct approaches to improving LoRA: better optimization, higher rank, and a mixture-of-experts architecture. A comparison with these methods clearly highlights the superiority of BoRA.
Implementation Details. All experiments were conducted using NVIDIA H20 GPUs. The general settings included the AdamW optimizer (Loshchilov and Hutter 2019), linear learning rate decay, a LoRA dropout rate of 0.05, and weight decay. For the GLUE benchmark, ratio of 0.03. The learning rates for RoBERTa- Base and RoBERTa- Large were set to 3e-4 and 1c-4, respectively. Additionally, the number of training epochs varies across different sub-tasks; further
applied warm-up

---

Published as a conference paper at ICLR 2026
details are provided in Appendix B By default, BoRA and the baseline methods were applied to the query and value weights. For mathematical and commonsense reasoning tasks, we employed a learning rate of le-4 with 100 warm-up steps and trained for epoch. In these tasks, BoRA and baseline methods were applied to the query, key, and value weights. Our implementation builds upon the code from (Hu et al. 2023). In all experiments,
evaluate LoRA's aceuracy across ranks 8, 16, and 32 to analyze the trade-off between parameter count and model performance. For BoRA, we the LoRA rank 8 and test with 8 and 16 blocks, respectively. For the three LoRA variants, we adjust the number of trainable parameters to align with LoRA at rank 8. Specifically, the rank of DoRA was set to 8, while the rank of HydraLoRA was the rank of mini LoRAs was set to 8, with four mini LoRA groups. Each experiment was repeated three times, and the average results were reported. Further details, including standard deviations, are provided in Appendices and
4 with three B matrices. For MELoRA,
4.2 OVERALL PERFORMANCE
Results on NLU Tasks. As shown in Ta BoRA consistently outperforms other methods on the GLUE benchmark. Specifically, BoRA achieves 2% improvement in average accuracy over LoRA at the same rank (r = 8). worth noting that the additional parameters introduced by BoRA with 8 or 16 blocks are minimal compared to the original LoRA parameters. Even when the rank of LoRA increased to 32, BoRA still maintains a comparable or even superior average accuracy. Regarding other LoRA variants, while some show modest accuracy improvements, they still lag significantly behind BoRA. Using a similar number of parameters, BoRA outperforms the best baseline by up to 1 on MaBERTa-Base and 2% on RoBERTa- enhances BoRA's performance. The impact of block numbers will be examined in Section [4.4, where is shown that further increasing the number of blocks continues to yield performance gains.
Moreover, doubling the number of blocks
Table 1: The accuracy on General Language Understanding tasks with various pretrained models.
#Params
RTF
MRPC
STS-B
CoI.A
SST-2
QNLI
MNI.
QQP
Avg
LoRA(r = 8)
0.29M
72.56
87.25
87.12
56.10
93.46
91.58
84.89
87.46
82,55
RoBERTa- Ta-BBsee
LoRA(r = 16)
0.59M
72.92
87.99
87.46
55.21
93.81
91.89
85.52
87.79
82.82
LoRA(r = 32)
1.18M
75.09
89.22
88.01
58.58
93.58
90.12
85.84
88.37
83.60
DoRA(r 8)
0.31M
72.92
87.75
87.58
55.14
93.12
91.26
85.02
87.41
82.53
MELORA(T 8)
0.29M
71.48
87.50
87.54
$2.12
92.78
90.96
84.68
86.83
81.74
HydraLoRA(r 4)
0.35M
71.84
89.46
88.07
55.32
93.81
91.62
85.27
87.25
82,83
BoRA(r 8,6
8)
0.31M
75.45
88.73
88.17
58.12
93.92
91.82
85.09
87.93
83.65
BoRA(r 8,6 16)
0.34M
76.90
88.24
89.31
57.35
93.12
91.78
85.69
87.86
83.78
LoRA(r =)
0.79M
71.96
88.40
89.88
59.76
95.41
93.07
88.67
87.89
84.38
ROBERTa- Large
LoRA(r 16)
1.57M
77.74
88.48
90.60
61.23
95.57
93.72
89.29
88.25
85.61
LoRA(r 32)
3.15M
81.35
89.64
91.45
60,90
95.60
93.76
89.52
88.61
86.35
DoRA(r 8)
0.84M
75.21
87.83
89.94
59.47
95.41
92.99
88.58
87.84
84.66
MELORA(r = 8)
0.79M
72.36
87.32
86.85
59.37
95.30
92.70
$8.37
87.33
83.70
HydraLoRA(r-
0.93M
74.01
89.22
89.47
59.90
95.41
92.88
88.87
87.98
84.72
BoRA(" =,0=8)
0.81M
78.34
89.95
91.18
60.34
95.18
93.48
89.66
88.52
85.83
BoRA(" 8,6 16)
0.88M
83.75
88.97
91.33
60.57
95.30
93.76
89.81
88.60
86.51
Results Mathematical Reasoning Tasks. As shown in Table BoRA demonstrates significant
improvements in mathematical reasoning tasks. At the same rank (r = 8), BoRA's accuracy across the three models is, on average, 2.4% higher than that of LoRA. Notably, even when LoRA's rank is increased by four times, the average improvement is only 1.9%. This highlights BoRA's superiority in increasing matrix rank and its enhanced expressiveness compared to LoRA. The improvements from DoRA and HydraLoRA are similarly modest. Although MELoRA A also increases the matrix rank, it introduces many zero elements, resulting in information loss and thus diminishing performance.
Results on Commonsense Reasoning Tasks. As shown in Table 3 BoRA also achieves the best accuracy in commonsense reasoning tasks. The experimental conclusions are highly consistent with those mathematical reasoning tasks. Note that the accuracy of commonsense reasoning tasks is generally higher than that of mathematical reasoning tasks, and the relative improvement is not so obvious. Nevertheless, at the same rank (r = 8). BoRA's accuracy across the three models is, on average, 0.95% higher than that of LoRA. In comparison, the average improvement from fourfold
7

---

Published as a conference paper at ICLR 2026
Table 2: The accuracy on Mathematical Reasoning tasks with various pretrained models.
#Params
AddSub
MultiArith
SingleEq
GSM8K
AQuA
SVAMP
Avg
LoRA(r 8)
4.82M
87.59
90.33
89.76
56.10
29.13
75.70
71.44
LoR AoRA(r 16)
9.63M
86.84
92.83
89.57
58. 15
30.71
74.90
72.17
LoRA(r 32)
19.3M
86.58
91.50
91.93
58.45
32.28
75.50
72.71
Gemma-
DoRA(r
5.16M
85.06
93.00
89.57
57.16
27.17
71.39
MELORA(r
4.82M
86.84
90.56
90.35
58.86
30.18
74.53
71.89
HydraLoRA(r
5.94M
85.82
91.06
91.27
58.68
29.13
74.47
71.74
BoRA(r 8,6
4.86M
87.93
93.22
90.35
58.91
29.66
75.37
72.57
(r 8,6 16)
4.99M
87.85
92.50
90.94
59.72
31.36
76.20
73.10
LoRA(r
4.72M
82.28
87.06
91.60
24.02
68.53
68.19
LoRA(r 16)
9.44M
84.56
91.22
92.26
57.22
25.72
70.17
70.19
LLama-3-8B
LoRA(r 32)
18.9M
87.17
93.39
93.50
57.87
26.25
71.83
71.67
DoRA(r 8)
4.92M
81.39
89.09
92.42
55.77
23.63
68.15
68.41
MELORA(r
4.72M
85.32
85.67
91.34
54.74
20.87
70.90
68.14
HydraLoRAG-
5.11M
77.22
89.17
91.73
56.63
24.41
66.30
67.58
BoRA(r = 8,
4.77M
87.85
93.67
92.72
58.45
26.38
70.10
71.53
BoRA(r 8, 16)
4.92M
88.35
93.00
92.72
58.83
27.17
73.40
72.24
LoRA(r 8)
8.65M
93.16
96.67
92.32
75.66
31.10
85.60
79.09
LoRA(r 16)
17.3M
91.90
96.33
92.91
74.37
34.65
86.40
79.43
Qwen2.5-144
LoRA(r 32)
34.6M
92.24
97.39
92.98
76.37
34.78
87.13
80.15
DoRA(r = 8)
8.99M
92.91
96.72
91.86
75.26
33.73
86.13
79.44
MELoRA(" 8)
8.65M
91.65
97.17
92.32
75.66
33.46
84.90
79.19
HydraLoRA(r 4)
9.29M
92.74
96.78
91.60
75.76
33.33
86.23
79.41
BoRA(r =8 8)
8.72M
91.65
96.83
92.78
75.41
86.93
79.73
BoRA(r 8, b=
8.95M
91.90
98.00
92.72
75.82
38.19
87.00
80.60
increase LoRA's rank is only 0.77%. This suggests that BoRA can achieve similar fine-tuning performance while requiring more than four times fewer parameters.
Table 3: The accuracy on Commonsense Reasoning tasks with various pretrained models.
#Param
BoolQ
PIQA
SIQA
HollaSwag
WinoGrande
ARC-c
ARC-e
OBQA
Avg
LoRA(r 8)
4.82M
70.15
88.96
78.05
94,08
89.82
83.96
94.02
88.60
85.95
LoRA(r 16)
9.63M
75.17
88.
77. 58
95.21
89.11
84.73
92.93
88.00
86.43
Gemman-T
LoRA(r
19.3M
74.34
89.72
78.10
95.77
88.95
84.73
87.20
86.61
DoRA(r = 8)
5.16M
74.86
89.55
78.76
93.66
89.74
83.45
92.68
86.20
86.11
MELoRA(r
4.82M
78.97
87.20
oRA
5.44M
71.77
87.92
80.76
95.00
88.24
84.47
94.44
87.20
86.23
BoRA(r 8.8
4.86M
72.66
90.26
78.97
94.77
90.77
84.73
89.00
BoRA(r
16)
4.99M
73.08
90.42
80.04
95.10
89.82
84,95
94.36
88.53
87.04
LoRA(r =
4.72M
73.17
89.34
80.64
93.22
87.42
80.20
92.51
87.13
85.45
LoRA(r
9.44M
73.54
88.53
Qwen2.5-114 1wen2.5-14BB LLam--3-88
LoRA(r 32)
18.9M
73.87
90.01
82.09
94.95
88.37
82,20
93.57
$8.73
86.72
DoRA(r 8)
4.92M
72.72
89.17
80.76
93.51
92.
S8.
MELoRA(1 8)
4.72M
72.66
89.28
81.53
94.19
86.74
80.89
93.10
89.00
85 92
Hydml
5.1
72.66
87.
BoRA(r = 8.0
4.77M
74.10
89.61
81.68
93.81
88.24
80.97
93.01
88.40
86.23
BoRA(r 8, 16)
4.92M
74.22
89.66
81.93
94.27
88.48
82.08
93.10
89,20
86.62
LoRA(r 8)
LoRA(r 16)
17.3M
76.15
93.74
84.44
96.87
92.50
94,20
98.36
95,80
91.51
LoRA(r 32)
34.6M
76.70
93.91
84.90
96.91
92.42
94.62
98.23
95.80
91.69
DoRA(r 8)
8.99M
76.35
93.49
84.32
96.56
92.66
94,34
98.13
95.73
91
MELoRA(Y
8.65M
76.41
93.67
84.87
96.78
91.92
94.25
98.01
95.93
91.48
HydraLoRAir
9.29M
76.02
93.91
84.24
96.50
92.58
94.11
97.98
95.20
91.32
BoRA(r 8,6 8)
8.72M
77.09
93.85
84.75
96.95
92.27
94.62
98.32
96.00
91.73
BoRA(r 8,6 16)
8.95M
77.06
93.91
85.57
96.96
92.66
93.52
96.80
91.82
4.3 ABLATION STUDIES
To optimize the block-wise diagonal matrices Σ effectively, BoRA applies both exponential and normalization function the learnalle parameters a, as shown 4). In this section, we conduct ablation experiments mathematical reasoning tasks to assess the importance of these two functions. As shown in Figure ((a) omitting either the exponential or the normalization function leads to a significant decrease accuracy across various models. Notably, the absence of normalization has
8

---

Published as a conference paper at ICLR 2026
BoRA
LoRA BoRA
BullA Norm
BollA w.n. Exp
17
7224
73.01
12.49
10.23
12.88
70.0
1930
10.03
71.00
71.65
70
E000
12-
12.23
66.11
4613
61
1915
03.45
65-
Lend Pretrained
Uwerd
1488
6md
Number Blocks
k-4
0010
(a) Ablation Results of BoRA
(b) Various Ranks and Block Numbers
(c) Differnt Tuning Granularity
Figure 3: Ablation studies and scalability analysis on mathematical reasoning tasks. (a) Ablation
results for the exponential function (Exp) and normalization function (Norm) in BoRA. (b) Accuracy of BoRA at varying ranks and block numbers. (c) Accuracy of BoRA at different tuning granularity. The figures present only the average accuracy, and the detailed results are available in Appendix
more sigmificant effect on performance. This can be attributed to the small initial values of a; without normalization, most entries Σ remain close to which restricts the expressiveness of BoRA.
4.4 SCALABILITY ANALYSIS
In this section, we evaluate the scalability BoRA on mathematical reasoning tasks using LLama-3- 8B, from three perspectives: the LoRA rank, the number of blocks, and the tuning granularity.
Results of Different LoRA Ranks and Number of Blocks. In Section 4.2 we set the rank BoRA to and evaluated its performance with 8 and 16 blocks. In this section, we investigate the effects of varying the LoRA rank (r) and the number of blocks (b) from the set (2, 4, 8, 16, 32, 64). Notably, the number of additional parameters introduced by BoRA is b2r, which increases quadratically with For comparison, when = 64, the number trainable parameters in BoRA is approximately 1.6 times greater than that in LoRA the same rank. As shown in Figure 3(b BoRA consistently outperforms LoRA across various ranks, even when = 2. As b increases, the performance of BoRA improves. However, when increases and b surpasses certain threshold, accuracy begins to decline, which is attributed to potential overfitting caused by the higher rank of the weight matrix.
Results of Different Tuning Granularity. Finally, we evaluate the scalability of BoRA under different tuning granularity. In Section 4.2 we apply LoRA and BoRA only to the query, key, and value weights. In this section, we introduce four additional strengths: Q. QV, QKVUD, and QKVOGUD, where O, G, U, and D represent the output, gate, up, and down projection weights, respectively. The rank is within any given tuning granularity, the number of parameters and computational costs in BoRA are approximately the same as LoRA. The goal of this experiment is to compare BoRA's performance relative to LoRA at each granularity. As shown in Figure 3(c) BoRA consistently outperforms LoRA across different tuning granularity, demonstrating that BoRA's performance improvement over LoRA consistent across various layers of the model (e.g.. k __rroj and v_proj).
8, and the number of blocks is set 16 for BoRA. Note that,
4.5 SINGUL.AR VALUE ANALYSIS
BoRA aims to enhance the rank of LoRA weights. To illustrate this more clearly, we analyze the singular values of both LoRA and BoRA. Specifically, we present the sum of the squared singular values and the count of singular values exceeding 0.005 for the weight of each query layer. The threshold of 0.005 is chosen to calculate the effective rank, following the settings used in HiRA Huang et al. (2025). As shown in Figure 4 the number of singular values greater than 0.005 in LoRA corresponds to its rank. As the rank increases, the sum of squared singular values also increases. In contrast LoRA, BoRA exhibits significantly more singular values greater than 0.005, and the sum

---

Published as a conference paper at ICLR 2026
Mathematican Reasoning Tasks
Commonsense Reasoning Tasks
suy of Sogun Strguhe Valae
29
LoRA (r=8)
LofA 16-
LoRA (m-32)
BuRA (
BoRA (r=6,b=16))
Figure 4: The sum of the squared singular values and the number of singular values greater than 0.005 across the query layers in LoRA and BoRA, with different ranks and block numbers.
of squared singular values is substantially larger, effectively demonstrating BoRA's role in enhancing rank.
4.6 EFFICIENCY ANALYSIS
As discussed in Section 3.4 we highlighted the minimal computational and memory overhead introduced by the block-wist diagonal matrices in BoRA. To validate this point more rigorously. we measured the wall-clock time and memory usage of various methods. One epoch of training on the mathematical reasoning task was performed using the LLama-3-8B model with an Nvidia A6000 GPU and a batch size of
Table 4: Computational cost and memory usage of different methods using the LLama-3-8B model.
# Param (M)
Memory (GB)
Time (Min)
LoRA(r = 8)
4.72
23.57
18.5
BoRA(r 8, 16)
4.92
23.63
18.6
MELORA(r 8)
4.72
23.57
18.6
HydraLoRA(r== 4)
5.11
24.98
19.6
DoRA( -8)
4.92
24.36
22.3
As shown Table 4 BoRA and MELoRA exhibit nearly identical training time and memory footprint compared to LoRA. HydraL oRA introduces additional gating layers, which add minor computational cost and result in a small increase runtime. DoRA, due to its reparameterization of directions and magnitudes, requires more complex computation and therefore introduces some overhead as well. Overall, the empirical evidence supports our argument that the end-to-end training overhead of BoRA essentially on par with LoRA, and comparable lower than several other state-of-the methods.
CONCLUSION
In this paper, we analyze the rank of LoRA weights from the perspective of block matrix multiplication. Our analysis revealed that standard LoRA suffers from rank limitations due to correlations among Low-Rank Adaptation
different block products. To solve this limitation, we propose Block-Diversinn (BoRA), a simple yet powerful extension to LoRA that enhances the rank LoRA weights. BoRA effectively breaks the correlations among different block products by introducing unique diagonal matrix for each block multiplication, thereby increasing the rank LoRA weights by a factor b, where b denotes the number of blocks. Extensive experiments demonstrate that BoRA outperforms LoRA and several LoRA variants across various tasks and models. Notably, BoRA achieves a 2-4% accuracy improvement over LoRA, while maintaining similar parameter and computational costs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China under grants 62376103, 62302184 184, 62436003 and 62206102: Major Science and Technology Project of Hubei Province under grant 2025BAB011 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DIC078; and Ant Group through CCF-An Research Fund. The computation completed in the HPC Platform of Huazhong University of Science and Technology.
ETHICS STATEMENT
The authors confirm that the submitted work does not raise any concerns with respect to the ICLR Code of Ethics. It does not involve human subjects, sensitive or private data, applications with potential ethical risks. All resources used are publicly available and properly licensed, and the research conducted in full compliance with ethical and legal standards.
REPRODUCIBILITY STATEMENT
This paper includes detailed descriptions of the training setups, hyperparameter choices, and evalua- tion protocols, enabling full verification of the methodology. To further support reproducibility, make our code publicly available at http:/hhhhhbcmcmeoopooooaa//aaaa ic1r26
REFERENCES
DeepSeek-AL. Deepseek-v3 technical report. CoRR, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama herd of models. CoRR, 2024.
Ali Edalati, Marzieh S. Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J. Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. In Advances Neural Information Processing Systems (NeurIPS) Workshop, 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation large models. In International Conference Machine Learning (ICML), 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficien transfer learning. In International Conference on Learning Representations (ICLR), 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee- Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm Llm-adapters: An adapter family for eframeter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-re rank adaptation for large language models. In International Conference on Learning Representations (ICIR), 2025.
11

---

Published as a conference paper at ICLR 2026
Damjan Kalajdzievski. A rank stabilization scaling factor for tne-ttnnig with lora. CoRR, abs/2312.03732, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, (ACI/IJCNLP), 2021.
Yang Li, Shaobo Han, and Jonathan Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. In Advances in Neural Information Processing Systems (NeurIPS). 2024.
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations (ICLR), 2024.
Shih Yang Liu, Chien Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Tongxu Luo, Jiahe Lei. Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts parameter-efficient fine-tuning for large language models. CoRR, abs/2402. 12851, 2024.
Fanxu Meng, Zhaohui Wang. and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08225, 2024.
OpenAI. Gpt-4 technical report. CoRR, 2024.
Anastasia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparameterization. In Findings of the Association for Computational Linguistics (ACL), 2023.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Menggi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhone Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations (ICLR), 2019.
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
12

---

Published as a conference paper at ICLR 2026
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-or Eexture-of-adaatationn for parameter-efficien model tuning. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: critical review and assessment. CoRR. 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report. CoRR, abs/2412.15115,, 2024.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In International Conference Learning Representations (ICLR), 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffcient fine-tuning, In International Conference on Learning Representations (ICLR), 2023.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE
Large language models (LLMs) were only used as writing assistants to polish the language, improve clarity, and check grammar. They were not involved in the generation of research ideas, the design or implementation of methods, data analysis, or the production of results. The authors take full responsibility for all content of the paper.
B DETAILED EXPERIMENTAL SETTINGS
Datasets and Models. The GLUE benchmark Wang et al. (0019) includes two single-sentence classification tasks (CoLA, SST-2), five pairwise text classification tasks (MNLI, RTE, QQP, MRPC, and QNLI), and one text similarity prediction task (STS-B). This paper reports the overall matched and mismatched accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, -B, and accuracy for the remaining tasks. Due to the differing dataset sizes, the number of epochs varies: 10 epochs for RTE and MRPC, 5 epochs for STS-B and CoLA, 2 epochs for SST-2 and QNLI, and 1 epoch for MNLI and QQP. The models used are RoBERTa- BBsse and RoBERTa-Large Liu et al. (2019).
LoRA Hyperparameters. The scaling factor is set to a = 2r, where r is the LoRA rank. LoRA is applied to the query and value weights with dropout rate of 0.05, using full precision (FP32).
Training Hyperparameters. AdamW Loshchilov and Hutter (019)) used with 81 = 0.9, 1e-8, and no weight decay. The learning rate is selected from the (3e-5, le-4, 3e-4, le-3]. with optimal values of 3e-4 4 for RoBERTa-Base and le-4 for RoBERTa- Large. A warm-up ratio 0.03 is applied, and the batch size is set 32. The maximum sequence length is 512.
82 0.999,
B. EXPERIMENTS ON MATHEMATICAL AND COMMONSENSE REASONING TASKS
Datasets. Mathematical and commonsense reasoning tasks contain 10K and 170K training samples, respectively, along with several test tasks. Note that we directly utilize the data from Hu al. (2023 for our experiments. The training process consists single epoch. Three models are employed: Gemma-7B Mesnard et al. 22024), LLama-3-8B Dubey al. (2024). and Qwen2.5-14B Yang al.
(2024).
LoRA Hyperparameters. The scaling factor set to a 2r, where is the LoRA rank. LoRA is applied to the query, key, and value weights with a dropout rate of 0.05, using half precision (BF16).
Training Hyperparameters. AdamW is employed with the same settings as previously mentioned. The learning rate is chosen from the set [3e-5, 1e-4, 3e-4, 1e-3} and is set to 1e-4. A warm-up of 100 steps is applied, and the batch size
16. The maximum sequence length 256.
C ADDITIONAL EXPERIMENTAL RESULTS
C.1 STANDARD DEVIATIONS
As discussed in Section each experiment repeated three times, and the average results reported. For conciseness, standard deviations are provided in the Appendix. Table shows the standard deviations for each GLUE dataset, where separate model was trained for each. Table 6 presents the standard deviation of the average accuracy for the commonsense and mathematical reasoning tasks, where single model was used across these sub-tasks. Notably, the standard deviation remains stable and is much smaller than the accuracy improvement achieved by BoRA.
C.2 DETAILED ABLATION RESULTS
In Section 4.3 ablation experiments were conducted on mathematical reasoning tasks to evaluate the contributions of exponential and normalization functions. Figure ([a) shows the average accuracy of various methods across multiple sub-tasks. Full experimental results are provided Table for a more detailed comparison.
14

---

Published as a conference paper at ICLR 2026
Table The standard deviation of different methods the GLUE benchmark.
RTE
MRPC
STS B
CoLA
SST-2
QNLI
MNLI
QQP
LoR A(r = 8)
0.51
0,46
0.27
0.57
0.11
0.15
0.11
0.10
LoR A(r = 16)
0.84
0.40
0.34
0.64
0.29
0.11
0.04
0.08
LoRA(r = 32)
0.78
0.35
0.39
0.56
0.14
1.00
0.11
0.03
DaRA(r = 8)
0.51
0.80
0.41
0.49
0.41
0.09
0.12
0.10
RoBERTa-Basc
MELoRA(r 8)
0.85
0.72
0.76
0.68
0.09
0.30
0.30
0.09
Hydral LoR((r = 4)
0.18
0.53
0.51
0.55
0.24
0.15
0.27
0.05
BoRA(r r = 8,6 8)
0.86
0.12
0.61
0.52
0.22
0.02
0.27
0.04
BoRA(1 = 8,6= 16)
0.64
0.61
0.88
0.64
0.19
0.20
0.14
0.12
LoRA(r 8)
0.55
0.81
0.26
0.64
0.50
0.22
0.23
0.05
LoRA(r
0.51
0.20
0.59
0.98
0.19
0.16
0.27
0.04
LoRA(r 32)
0.40
0.66
0.13
0.65
0.33
0.44
0.32
0.02
DoRA(r 8)
0.79
0.42
0.03
0.78
0.19
0.30
0.25
0.02
RoBERTa-Large
MELoRA(r 8)
0.59
0.35
0.89
0.24
0.19
0.25
0.38
0.04
Hydral .RR 4)
0.25
0.60
0.63
0.59
0.09
0.04
0.29
0.09
BoRA(r =8,6 8)
0.36
0.60
0.15
0.43
0.09
0.13
0.48
0.03
BoRA(r 8,6 16)
0.59
0.23
0.17
0.53
0.19
0.17
0.12
0.03
Table 6: The standard deviation of the average accuracy on the mathematical and commonsense reasoning tasks with various pretrained models.
Mathematical Reasoning
Commonsense Reasoning
Gemma-7B
T.I 3-8a-3-8B
Qwen2.5-14B
Gemma-7B
I.Lama-3-8
Qwen2.5-14PP
LoRA(r 8)
0.19
0.88
0.60
0.63
0.13
0.13
LoRA(r 16)
0.61
0.62
0.50
0.22
0.19
0.10
LoRA(r 32)
0.48
0.80
0.78
0.18
0,29
0.06
DoRA(r 8)
0.47
0.47
0.48
0.35
MELORA(r 8)
0.34
0.58
0.23
0.22
0.48
0.02
HydraLoRA(r=
0.21
0,36
0.39
0.43
0.19
0.23
BoRA(r =8,6=
0.42
0.61
0.38
0.12
0,26
0.06
BoRA(r 8.5 16)
0.44
0,38
0.32
0.38
0,28
0.01
C.3 DETAILED RESULTS OF THE SCALABILITY ANALYSIS
In Section 4.4 the scalability of BoRA was evaluated using LLama-3-8B on mathematical reasoning
tasks from three perspectives: LoRA rank, number of blocks, and tuning granularity. Figure presents the average accuracy of different settings across sub-tasks. Complete experimental results available Tables 8 and 9 for further comparison.
D BROADER IMPACTS
This paper introduces BoRA, a method for increasing the rank LoRA weights. BoRA achieves the same fine-tuning performance LoRA while using fewer parameters and reducing computational overhead, thus contributing to energy savings. Our work builds upon LoRA, and we assert that our approach does not introduce any negative social implications requiring further discussion.
E ADDITIONAL RELATED WORK
In addition to LoRA, there are two other commonly used PEFT methods: adapter- based and soft prompt-based methods. The adapter-based method Houlsby et al. (2019); [He et al. al. (2022); Wang ct al. (022)) inserts new layers into the model and fine-tunes only these layers, significantly reducing resource consumption. However, the additional layers introduce increased latency. The soft prompt- based method Lester Lester et al. (2021): Li and Liang 2021 Razdaibiedina et al. (2023) adds learnable soft tokens (prompts) the input, enabling the model to adapt to specific tasks. This method leverages the pretrained model's inherent capabilities and requires only appropriate prompts for downstream task adaptation. However, it also adds computational overhead and increases inference latency. In
15

---

Published as a conference paper at ICLR 2026
Table 7: The accuracy of BoRA without exponential or normalization functions on mathematical reasoning tasks using LLama-3-8B.
AddSub
MultiArith
SingleEq
GSM8K
AQuA
SVAMP
Avg
BoRA
87.85
92.50
90.94
59.72
31.36
76.20
73.10
Gemma-
BoRA w.o. Exp
86.33
90.67
89.37
58.45
30.71
75.10
71.77
BoRA w.o. Norm
91.83
88.78
58.23
32.28
75.20
72.15
BoRA
93.00
92.72
58.83
27.17
73,40
72.24
LLama-3-8B
BoRA w.o. Exp
87.00
90.94
55.19
26.77
68.10
68.51
BoRA w.o. Norm
81.77
88.17
91.34
56.10
25.20
68.30
68.48
BoRA
91.90
98.00
92.72
75.82
38.19
87.00)
80.60
Qwen2.5-14B
BoRA w.o. Exp
92.66
96.67
92.52
77.63
33.86
85,40
79.79
BoRA w.o. Norm
93.16
96.17
92.13
76.42
32.28
85.40
79.26
Table The accuracy of LoRA and BoRA with varying target modules on mathematical reasoning tasks using LLama-3-8B.
Target Modules
Method
#Params
AddSub
MultiArith
SingleEq
GSM8K
AQuA
SVAMP
Avg
LoRA(r =
2.10M
74.43
86.39
88.45
52.26
26.64
61.67
64.97
BoRA(r 8.6 16)
2.16M
81.18
90.42
54.54
23.23
67.10
67.87
LoRA(r
3.41M
81.27
90.00
91.54
56,25
22.83
66.40
68.05
QV
BoRA(r 8.6 16)
3,54M
86.50
91,22
93.18
58.18
25.98
71.03
71.01
QKV
LoRA(r = =8)
4.72M
82.28
87.06
91.60
$5.65
24.02
68.53
68.19
BoRA(r 16)
4.92M
88.35
93.00
92.72
58.83
27.17
73.40
LoRA(r
14.2M
88.69
92.11
93.83
59.79
24,93
73.50
72.14
QKVLD
BoRA(r = =8 16)
14.5M
89.11
95.25
93.60
61.79
24.41
76.20
73.40
LoRA(r =8)
21.0M
87.59
93.28
93.90
59.41
24.15
75.03
72.23
QKVOGUD
BoRA(r =8,b=10 21.4M
94.00
contrast, both LoRA and the proposed BoRA allow for manual integration weight updates into pretrained weights after fine-tuning, avoiding additional inference latency.
F THEORETICAL JUSTIFICATION FOR PROPOSITION
briefly discussed the theoretical justification for Proposition 1 Here, provide more detailed expansion and explain under which assumptions BoRA's rank can achieve the upper bound presented in Proposition |
In Section 3.3
Without loss of generality, for a pre-trained weight Wo E RNxn LoRA maintains two low-rank
matrices B E IR mxxr and A TR"x,, Let matrix B Rmxr [B1, B2,..., Bb] be divided into b blocks, where B, E R : xr Similarly, matrix A E I² x = A2,, A..... Ad] is divided into blocks, where A, prx The LoRA weights generated by the proposed BoRA can be denoted as follows:
I1 12
B1E,, Be
BiE B2
AW
1
Bg
B,E,,
where Ei E grxe j E [6]) are diagonal matrices. Note that AW can also be represented as the product three matrices (B' E moxbr E' E Rebxbr and A' E Rrxx)), as shown below:
B1 0
0
Σ1,1
E1
E1
A
0
B2
0
Σ21
Σ2
Σ2b
0
A2
AW = B'EA-
B,
Σ,,1
E.
Do.b
As
Based the shapes of the matrices, we can derive the upper bound for the rank the three matrices: rank( (B') min(m,br)) rank(A') min(n, br), rank(E')) s br
16

---

Published as a conference paper at ICLR 2026
Table The accuracy of LoRA and BoRA with varying ranks and block numbers on mathematical reasoning tasks using LLama-3-8B.
Rank
Method
#Params
AddSub
MultiArith
SingleEq
GSM8K
AQuA
SVAMP
Avg.
LoRA
1.18M
77.64
85.56
89.24
52.87
23.36
65.53
65.70
BoRA(b
1.18M
74.94
89.17
88.19
56.03
26.77
61.90
66.
BoRA(b
1.18M
80.64
88.75
90.36
54.93
24.61
66.65
67.65
BoRA(b
1.19M
79.37
88.67
90.06
56.94
25.59
67.10
67.96
BoRA(t 16)
1.23M
81.86
87.33
91.73
55.34
24.67
67.47
68.07
BoRA(b 32)
1.38M
80.76
88.67
91.14
55.42
26.77
67.70
68.41
BoRA(b 64)
1.97M
84.30
87.83
90.75
56.33
22.83
69.10
68.53
LoRA
2.36M
79.49
86.83
88,85
55.12
23.10
66.03
66.57
BoRA(E 2)
2.36M
79.24
88.67
90.16
55,88
22.05
66.73
BoRA(E
2.37M
79.24
89.50
93,90
58.30
22.05
63.90
67.81
BoRA(
2.38M
83.29
90.17
91.53
55,55
24.28
67.73
68.76
BoRA(E 16)
2.46M
82.95
90.56
92.92
57.70
24.54
69.17
69.64
BoRA( 2.75M
81.60
91.39
92.78
58.20
25.98
68.10
69.68
BoRA(b 64)
3,93M
85.06
92.33
92.91
56.68
27.04
69.23
70.54
LoRA
4,72M
82.28
87.06
91,60
55.65
24.02
68.53
68.19
BoRA(b 2)
4.72M
81.01
92.33
91.34
59.36
24.41
66.10
69.09
BoRA(b 4)
4.73M
84.30
91.39
92.78
58.10
24.67
69.67
70.15
==8
BoRA(E 8)
4.77M
93.67
92.72
58.45
26.38
70.10
71.53
BoRA( 16)
4,92M
88.35
93.00
92.72
58.83
27.17
73.40
72.24
BoRA( 32)
5,51M
87.34
95.17
92,91
58.53
27.56
73.60
72.52
BoRA(E 64)
7.86M
87.34
94.33
94.09
60.35
27.95
72.60
72.78
LoRA
9.44M
84.56
91.22
92.26
$7.22
25.72
70.17
70.19
BoRA(l 2)
9.44M
83.04
91.67
94,69
59.59
26.38
68.00
70.56
BoRA(t 4)
9.46M
86.16
91.33
93.83
59.26
26.51
70.70
71.30
r= 16
BnRA(b 8)
9.54M
85.82
93.61
93.31
59.61
25.46
72.10
71.65
BoRA(t 16)
9.83M
86.75
93.67
93,90
58.94
26.51
72.27
72.01
BoRA(b 32)
11.0M
89.37
94.83
94.29
58.91
25.20
74.10
72.78
BoRA(b 64)
87.34
95.58
94.29
59.40
30.52
73.20
73.39
LoRA BoRA(b 2)
18.9M
87.17
93.39
93.50
57.87
26.25
71.83
71.67
18.9M
87.85
91.83
94.29
59.06
25.98
72.30
71.89
BoRA(b 4)
18.9M
90.13
93.56
93.24
59.59
25.20
74.57
72.71
r- 32
BoRA(t 8)
19.1M
89.11
93.67
93.70
60.42
24.02
75.00
72.65
BoRA(t 16)
19.7M
88.61
95.83
93.50
59.59
27.56
73.50
73.10
BoRA(b 32)
22.0M
89.50
95.42
93.60
59.93
27.36
75.30
73.52
BoRA(t 64)
31,5M
89.62
96.67
92.91
60.65
23.62
74.00
72.91
LoRA
37.7M
88.10
94.33
93.70
60.05
25,20
73.10
72.41
BoRA(t 2)
37.8M
89.62
91.17
94.29
60.73
25,59
76.30
72.95
BoRA(E 4)
37.8M
88.77
94.72
93.70
60.78
26.51
74.53
73.17
r= 64
BoRA@t 8)
38. MM
90.13
94,50
94,49
62.70
25.20
75.40
73.74
BoRA(b 16)
39.3M
89.87
94.83
94.29
60.50
26.77
76.00
73.71
BoRA(t 32)
44.0M
89.62
96.17
95.08
61.56
27.95
75.90
74.38
BoRA(b = -64)
62.9M
90.13
95.67
93,90
61.26
25.59
76.90
73.91
According to the properties of matrix multiplication:
rank(AW) < min (rink((((A)) (an,
-in br).
important to note rank(AW) can only have clearly defined upper bound, with minimum value being 0. For example, in BoRA, the matrix B is initialized to zero, so at the beginning of training, rank(B') = 0, which results in rank(AW) = 0. However, when all three matrices are min(n, br), and rank(E') br), equality
full rank (i.c., rank(B') min br), rank(AA) holds, and the upper bound for (AW) can be achieved. This condition is typically satisfied because E' and A' initialized randomly, which usually ensures that rank(A') min (n, br) and rank(E') = br. As training progresses, B' is continually updated, and rank B3) can gradually approach full rank. This is evident from Figure in the paper, where the effective rank BoRA (the number of singular values greater than 0.005) approaches br.
17


Reviews:
[{"review_id": "3crm2dTdYQ", "reviewer": "Reviewer_dURX", "summary": "This paper proposes BoRA, which addresses the rank-constrained problem of LoRA by enhancing diversity through matrix partitioning and the introduction of a block diagonal matrix, thereby increasing the weight rank of LoRA by a factor of b with only a few additional parameters. Multi-model, multi-task experiments show that it outperforms LoRA and its variants, achieving a 2-4% accuracy improvement with similar parameters.", "questions": "- Why must the diagonal matrix $\\Sigma $ be non-negative?\n\n- Is it feasible to freeze the existing trained LoRA structure and train only the diagonal matrix $\\Sigma $?\n\n- Will the matrix rank change with different layers and training periods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The methodology is clearly and comprehensively described, and Figure 1 clearly explains BoRA and its differences from other methods.\n\n- The experiments in the paper are comprehensive, validating the effectiveness of the method on different tasks and models.", "weaknesses": "- The \"Results of Different Tuning Granularity\" section in Section 4.4 does not guarantee that the number of parameters or computational costs will be consistent (or as consistent as possible).\n\n- Although the authors analyzed the efficiency of BoRA, experiments comparing the efficiency of different methods are lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqvVVAm9BZ", "reviewer": "Reviewer_YooA", "summary": "This paper proposes a parameter-efficient fine-tuning method called **Block Diversified Low-Rank Adaptation (BoRA)**, which improves the rank of LoRA weights through matrix block  diversification. The core idea of BoRA is to divide the LoRA weight matrices $A$ and $B$ into $b$ blocks the columns and rows. To break the correlation between the block products $B_iA_j$ in different rows or columns, BoRA introduces a unique diagonal matrix,  $\\Sigma_{i,j}$ , for each block multiplication, resulting in $B_i\\Sigma_{i,j}A_j$. This formulation increases the upper bound of the LoRA weight rank by $b$ times while adding only $b^2r$ additional parameters. To demonstrate the effectiveness of BoRA, the authors conduct extensive experiments on natural language understanding, mathematical reasoning, and commonsense reasoning tasks, covering both RoBERTa and various mainstream LLMs. The results show that, compared with LoRA and three of its variants, BoRA consistently achieves stable performance improvements. Under comparable parameter settings, BoRA improves accuracy by about 2-4\\% over LoRA.", "questions": "1. Could you provide efficiency comparisons during training (e.g., convergence time, memory usage, training latency)?\n2. When dividing the matrices into $b$ blocks, on matrices of what size is this partitioning performed?\n3. Why are the ranks of the other baselines set to 8, while HydraLoRA is set to 4? Could you also provide results for HydraLoRA with rank 8?\n4. Why did the authors choose relatively small numbers of training epochs?\n5. In Fig. 3(b), standard LoRA improves as $r$ increases, whereas BoRA with $b=16$ or $64$ shows performance drops as $r$ increases (e.g., for $b=16, r=8 \\ vs. r=16$). Could you explain the reason for this behavior?\n6. For Different Tuning Granularity, why are comparisons made only between BoRA and LoRA, without including other baselines?\n7. Is the notation $i, j \\in [r]$ in Section 3.2 a typo by the authors? Should $r$ be $b$ here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper analyzes the rank limitation of standard LoRA from the perspective of block matrix multiplication, showing that the correlation between block matrices constrains the expressiveness of LoRA and thus provides a strong theoretical foundation for the proposed solution. \n- It introduces a LoRA variant called BoRA, which eliminates the correlation between block matrices by introducing diagonal matrices, achieving substantial performance improvement with only a small parameter overhead of $b^2r$. \n- The authors also prove that LoRA and MELoRA are special cases of BoRA, thereby establishing a clear connection with previous work. \n- Comprehensive evaluations on models with different architectures and across various task types demonstrate the broad applicability of the BoRA.", "weaknesses": "- Although the authors mention inference latency in the appendix, they lack efficiency comparisons during training (e.g., convergence time, memory usage, training latency), and adding such comparisons would make the work more convincing.\n- The authors claim that BoRA raises the rank upper bound, and in Related Work they also mention HiRA and KronA as methods that increase the rank of LoRA weights, yet these are not included as baselines for comparison.\n- In the original MELoRA paper, the number of epochs for GLUE tasks is much larger (e.g., 60 for SST-2), whereas this paper uses relatively small epoch counts (e.g., 2 for SST-2). This discrepancy may affect the resulting performance.\n- When discussing different tuning granularity, the authors compare BoRA only with LoRA and do not compare it with other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l6OFoBlCfh", "reviewer": "Reviewer_9WMS", "summary": "The paper presents Block-Diversified Low-Rank Adaptation (BoRA) to enhance rank by introducing block-wise diagonal matrices. \nThis increases rank with only a few additional parameters. Theoretical analysis confirms provable rank enhancement with an upper bound.\nEmpirical results show that BoRA achieves 2-4% accuracy improvements over LoRA across GLUE, mathematical, and commonsense reasoning tasks with similar parameter counts and involves various network architectures.", "questions": "1. Could you provide a theoretical justification that the rank of BoRA  can achieve $br$ under some assumptions, rather than merely being bounded by it?\n2. In Section 4.5, why is $0.005$ chosen as the threshold?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel perspective on LoRA by analyzing it through block matrix multiplication, revealing how correlations between block products constrain rank.\n2. The paper demonstrates that both standard LoRA and MELoRA are special cases of BoRA, creating a unified theoretical framework.\n3. Empirical results are solid, and BoRA is compared with various base models and tasks while achieving strong performance over a range of baselines.\n4. The paper provides thorough ablation studies that validate the importance of both the exponential function and normalization function in BoRA. Results also show the consistently superior performance with varying ranks and block numbers.", "weaknesses": "1. In Section 3.2, the authors states\n> Assuming $A$ and $B$ are divided into $b$ blocks along columns and rows, respectively, BoRA will additionally learn a set of diagonal matrices $\\\\{\\Sigma_{i, j} \\in \\mathbb{R}^{r \\times r} \\mid i, j \\in[r]\\\\}$.\n\n​\tThis notation shows $i, j \\in [r]$ for the indices of $\\Sigma_{i,j}$, but this is inconsistent with the block count $b$. Since $A$ and $B$ are divided into $b$ blocks, the indices should be $i, j \\in [b]$, not $[r]$.\n\n2. Proposition 1 claims that BoRA requires only $b^{2} r$ additional parameters to increase the weight rank by a factor of $b$. In contrast, LoRA needs $(m+n)(b^{2} r)$ parameters to achieve the same bound, which, however, is incorrect. To achieve a rank of $br$, standard LoRA would need to set its rank parameter to $br$, which would require $r'(m+n)$ parameters where $r'=br$, thus $br(m+n)$ parameters - not $(m+n)(b^{2}r)$. A correct comparison is thus  LoRA (rank $br$) with $(m+n)br$ parameters vs. BoRA (rank bound $br$) with $(m+n)r + b^{2}r$ parameters. The authors should compare these two terms in practice.\n\n3. Proposition 1 claims that the rank of BoRA's weight update $\\Delta W$ satisfies $\\operatorname{rank}(\\Delta W) \\leq \\min \\\\{m, n, b r\\\\}$, but this bound is not properly justified and is too loose. Proposition 1 only gives $\\operatorname{rank}(\\Delta W) \\leq b r$. The key claim that BoRA increases the rank by a factor of $b$ (i.e., up to $b r$) requires a tighter argument that BoRA can actually achieve this bound, which is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTON8pAg8G", "reviewer": "Reviewer_rMns", "summary": "This paper proposes BoRA, a parameter-efficient fine-tuning method designed to enhance the expressiveness of standard LoRA. The core idea is to partition the LoRA matrices A and B into multiple blocks and introduce a unique learnable diagonal matrix $\\Sigma$ for each block pair. Through extensive experiments, the authors demonstrate that BoRA consistently outperforms standard LoRA and its variants under a similar number of parameters.", "questions": "1. In practical fine-tuning, what is the percentage increase in time per training step for BoRA compared to standard LoRA or other baselines?\n2. Can the authors provide more insights on how to decide an appropriate $b$ when using BoRA in practice, where both model scales and targeted tasks can vary?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea to improve LoRA's power of expressiveness by incorporating blockwise learnable diagonal matrices is novel and makes sense to me.\n2. The experimental results consistently show the advantage of the proposed method.\n3. The presentation is clear, and the paper is easy to follow.", "weaknesses": "1. I think the comparison with other LoRA-like baseline algorithms is not enough. Beyond LoRA, DoRA, MELoRA and HydraLoRA, there are other similar methods that also aim to improve LoRA's expressive power yet not compared or discussed in this paper. For example, 1) ReLoRA is discussed but not compared with; 2) SLTrain [https://arxiv.org/abs/2406.02214] is neither compared nor discussed, it can also largely improve the model performance; 3) LoLDU [https://arxiv.org/abs/2410.13618] is neither compared nor discussed, I believe its idea has some similarity with this paper.\n2. The paper analyzes the FLOPs for the forward pass in Section 3.4, but does not discuss the actual overhead of backward propagation and end-to-end training time. Even if FLOPs are similar, introducing numerous small block operations may lead to increased kernel launch of CUDA, which could potentially result in a large amount of overhead. It is recommended to have a further discussion on the computation overhead.\n3. A minor question: I notice that a prior work [https://arxiv.org/abs/2407.15857] in this literature has already used the name *BoRA* for their method. I recommend using another name to avoid ambiguity, or the authors can include some discussions about that BoRA algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)