Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Authors: Shiwei Li
Xiandi Luo
Haozhao Wang
Xing Tang
Ziqiang Cui
Dugang Liu
Yuhua Li
xiuqiang He
Ruixuan Li
Venue: NeurIPS 2025 poster ()
Keywords: LoRA
Token-wise Projection
Primary area: deep_learning
TL;DR: This paper introduces TopLoRA, which learns token-wise LoRA weights (i.e., token-wise input-output projections).

Abstract: Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). 
LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank.
In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection.
This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens.
To address this limitation, we propose **Token-wise Projected Low-Rank Adaptation (TopLoRA)**, which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner.
Formally, the weights of TopLoRA can be expressed as $B\Sigma_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $\Sigma_X$ is a diagonal matrix generated from each input token $X$.
Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections).
Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. 
The code is available at https://github.com/Leopold1423/toplora-neurips25.

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Shiwei Li Xiandi Luo' Haozhao Wang' Xing Tang²
Ziqiang Cui Dugang Liu' Yuhua Li' Xiugiang He" Ruixuan Li"
Huazhhnn University of Science and Technology "Shenzhen Technology University *City University of Hong Kong "Shenzhen University (lishiwei, togawa_ sakiko, hz hanw, idcliyuhua, rxli) @hust.edu.cn xing. tang @ hotmail.com, ziqiang cui @ my.cityu, selyyu..du.hh edu.hk dugang. @ com, hexiuqiang @ sztu. edu.cn
Abstract
Low-rank adaptation (LoRA) is parameter-efficien fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimen- sionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA be expressed as BEx A, where A and B are low-rank matrices (as in standard diagonal matrix generated from each input token X. Notably,
LoRA), and X TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demon- strate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https: 4ttp://hww..o..oooo....//////ppa http://ppoora-uurip22222.12 neurips25
1 Introduction
Recent advancements in pretrained large language models (LLMs) have led to significant improve-
Traditionally, these models require full fine-tuning (FFT) to update all their parameters for specific downstream tasks. However, due to the large size of pretrained models, FFT can be computationally expensive, particularly in resource-constrainee fine-tuning (PEFT) methods have been introduced reduce the number of trainable parameters and decrease fine-tuning costs 14 38 41. Among these methods, low-rank adaptation (LoRA) [IO] has emerged as one of the most widely used techniques. As shown in Figure [(a) LoRA freezes the pretrained weight matrix W E pm and learns two smaller low-rank matrices to approximate the weight update AW BA, where A E gr B R and the LoRA rank & min (m, n).
ments in variety of natural language processing and vision tasks 47 40 2]
environments. To address this challenge, parameter-efficient
Despite its effectiveness, LoRA typically exhibits a performance gap when compared to FFT, often attributed to the limited number of trainable parameters [10 (7]. Previous studies have also shown that increasing the LoRA rank generally improves fine-tuning performance [35] 26). Recently,
Cooresppnding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pretrained weight
Pretrained weight
Diag((-)
Xx
Exp(RMS(Norm()
W E R mR
W R R
6
X
(a) Low-Rank Adaptation (LoRA)
(b) Token-wise Projected Low-Rank Adaptation (TopLoRA)
Figure 1: An illustration of TopLoRA in comparison to LoRA [10]. TopLoRA additionally learns projector e to generate a diagonal matrix Σx based on the input token X, which is then used to adjust the LoRA weights (i.e., the input-output projections) for each token. The operators Exp(.) and RMSNorm(-) refer to the exponential function and root mean square normalization
49
respectively,
Zeng and Lee 48 explored the expressive power of LoRA, using the LoRA rank to quantify the approximation errors between the LoRA weights (AW = BA) and the assumed optimal weight update. Their results suggest that smaller LoRA ranks lead to larger approximation errors. However, more intuitive explanation of how the LoRA rank affects fine-tuning performance remains lacking. This raises the question whether there is a more intuitive way to understand the role of LoRA rank and whether other factors, beyond LoRA rank, also influence fine-tuning performance.
To answer this question, we explore the impact of LoRA rank from the perspective of input-output projections. Specifically, the matrix B can be expressed using QR decomposition as B = Q E Rmxr,, where QB [Q...... Qal E Im RB E R is right triangular matrix. Similarly, the matrix A can be expressed using LQ decomposition as A LAQA E Re**, where QA = [Q),.... QA1 RR also unitary matrix (i.e., Q T) and I E TR" a left triangular matrix. this point, the LoRA weights can be represented follows:
semi-unitary matrix (i.e., Q = I) and
AW = BA = 0 1...0A=100....011IE.. ..
Q%1 (I)
As shown, the LoRA weights consist three components: the input space QA = 1 1 2%1 the output space QB Q B...... Qg. and the input-output projection P = RBLA = [PI P. Rr Notably, LoRA captures only the projection of the input token onto the column vectors of Q A. with all other projections transformed zero. Additionally, the LoRA output is constrained to linear combination of the row vectors of Qn- Thus, increasing the LoRA rank r raises the dimensionality of both the input and output spaces, thereby improving LoRA's expressive power.
Based on the above analysis, we reveal another factor, beyond increasing the LoRA rank r, that can enhance its expressiveness. Specifically, different tokens exhibit distinct semantics and distributions, and the shared input-output projection (P = Rg LA) is insufficient for capturing token-specific information. To address this limitation, introduce Tosen-wise Projected Low-Rank Adaptation (TopLoRA). which dynamically adjusts LoRA weights for each token, as shown in Figure [(b) TopLoRA utilizes projector, e, to generate diagonal matrix. Σx. based on the input token X. This matrix, in turn, adjusts the value P = RB LA to Px = Ex LAA- Importantly, the goal of LoRA is not increase the rank of LoRA weights (i.e., the dimensionality of the input and output spaces). Instead, it focuses on capturing distinct key information from different tokens through token-wise input-output projections, enabling finer-grained adaptation even with limited rank.
The main contributions can be summarized as follows:
We analyze LoRA from the perspective of input and output projection and identify another factor, beyond the LoRA rank, that limits the expressiveness of LoRA: the shared input- output projection of LoRA is insufficient for capturing token-specific information.

---

We propose TopLoRA, which enables LoRA to apply distinct oututt projections for different input tokens. By adjusting the LoRA weights with token-wise diagonal matrices, Topl.RRA achieves finer-grained adaptation even with a limited rank.
We conduct extensive experiments on various models and datasets, demonstrating that TopLoRR consistently outperforms LoRA and its variants, At the same rank, TopLoRA achieves 2-3% accuracy improvement over LoRA.
2 Motivations
2.1 Low-Rank Adaptation
Building on the hypothesis that weight updates during fine-tuning exhibit low intrinsic dimensionality. Hu
[T0] proposed Low-Rank Adaptation (LoRA), which updates pretrained weights using the product of two low-rank matrices. Given pretrained weight matrix W E Rm=x,, the forward propagation of LoRA can be formulated as:
Y (W + AW)X (W + %/
(2)
where B E R A E RTx", the LoRA rank r < min(m, n), and a is an adjustable scaling factor. For simplicity, we omit the coefficient a/r in the following discussion. Dur- ing fine-tuning, the pretrained weights remain frozen, and only the low-rank matrices A and B are updated. Note that the LoRA rank r determines the rank of the weight update matrix as rank(AW]) = rank (BA) min/rank(( (A)nk(A), rankknk(BB)))) rank(B)) S A higher LoRA rank generally leads to better fine-tuning performance, but also increases the number trainable parameters.
2.2 Input-Output Projections of Low-Rank Adaptation
Most existing studies acknowledge that the LoRA rank significantly affects fine-tuning performance
but lack an intuitive explanation 335 26 12 4 In this paper, we provide clearer interpretation of the role of LoRA rank from the perspective of input-output projections and explore additional factors that may influence LoRA's fine-tuning performance. As discussed in the previous section, applying LQ and QR decompositions to the matrices A R and B E RRmx yields:
= LAQA, B= RBQB,
(3)
where Q, = iQ....., QA] TT E R and QB iQ.....: 2i1 E mxx are unitary matrices (i.e. Q7 0- and Qg Q7'), ) LA Rxx and Rn E Rrxr are left- and respectively. Therefore, the LoRA weights can then be expressed as:
matrices.
AW = BA = Qn
QaPQa 1Q
(4)
where = R -P1,=-PP,.... Pr] E RR represents the interaction between the matrices QA and QB. Any input token X E Ra can be decomposed into X = x E a:Q's. where X denotes the component orthogonal all Q4a satisfying QA O. Thus, the LoRA output can be expressed as:
AWX QBPQAX Σ EaQOPQAA' Σ
a, Σ P.o
Pn
(5)
i=1
which indicates that LoRA captures only the component of the input token along directions 2x) and maps them to r corresponding output directions
-zhh where Pa denotes the scaling factor from Q' to Q//- Accordingly, the LoRA weights, as shown in Eq. 4. can
be divided into three components: the input space Q [Q) - ..... -2A). which represents the input range that LoRA can effectively capture; the output space QB = @ 17), which represents the output range that LoRA can cover; and the input-output projection P = RBLA = PP..... Pr), which models the scaling of each Q3 it transforms into each Q'B-
From this perspective, the LoRA rank determines the dimensionality of the input and output spaces. A lower rank restricts the model's 's ability to extract and process information from the input tokens, which intuitively explains why LoRA often performs worse than FFT and why increasing the rank typically enhances performance. Previous studies have mainly focused on increasing the rank of
3

---

LoRA weights to expand the dimensionality of both the imput space (QA) and the output space (QB). For example. HiRA [2] and KronA 4 increase the rank of LoRA weights using the Hadamard and Kronecker products, respectively. MELORA (35] on the other hand, achieves a higher rank by stacking low-rank matrices along the diagonal.
In contrast, we identify another important factor that affects the expressiveness of LoRA: the input- output projection (i.e., P). In standard LoRA, all tokens share the same input-output projection. However, since the semantics of different tokens vary significantly, even the same projection (e.g., Q'x) may represent different information for different tokens, requiring distinct processing. Existing methods that apply uniform input-ouipu projection across all tokens fail to capture token-specific information. This motivates us explore method that provides adaptive input-output projections tailored to individual tokens. In this way, we can increase the diversity of input-output projections, thereby enhancing the expressiveness of LoRA, even with smaller LoRA ranks.
3
Methodology
In this section, we introduce the Token-wise Projected Low-Rank Adaptation (TopLoRA), a LoRA architecture designed to optimize the input-output projections at the token level.
3.1 Token-wise Projected Low-Rank Adaptation
Based on the above analysis, the optimization of LoRA seeks to identify a pair of input and output spaces along with the corresponding input-output projections. However, different tokens share the same LoRA weights and thus undergo identical input-output projections, which limits the ability to capture spkeif-pecific information. To overcome this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts the LoRA weights for each token. To achieve this parameter-efficient manner, TopLoRA maintains an additional network, r, with parameters O, which generates a diagonal matrix Ex based on the input token X. This matrix is then used to modify the LoRA weight (i.e., the onput-output projection) as follows:
AWX BT (X) A.
(6)
is important to note that the goal of TopLoRA not to increase the rank of the LoRA weights, but rather to find the optimal input-output projection for each token, even within limited rank.
3.2 Optimization of Token-wise Diagonal Matrix
The effectiveness of TopLoRA depends on the careful design of its token-wise diagonal matrix, Σx, which generated by specialized network, Te. Notably, the parameters employ Kaiming initialization ( consistent with the initialization of matrix A, ensuring stable training when using a uniform learning rate for all parameters. Further, to facilitate effective learning of Σx. we apply
root mean square normalization (RMSNorm) 49 followed by an exponential transformation on the network output, OX,, as shown below:
Σx = Diag(Exx(RMMSNrml (X))).
(7)
RMSNorm ensures that the value of x is not influenced by the magnitude of token or the projec- tion parameter e. It essentially broadens the distribution range of Σx, thus increasing the differences between the diagonal matrices of different tokens. The exponential transformation converts the zero-centered normalized values into strictly positive scaling factors. It prevents information loss from near-zero values of eX and ensures that even subtle but important variations in the normalized projections can be effectively captured.
3.3 Comparison with Low-Rank Adaptation
TopLoRA introduces a dynamic term, Σx. which enhances standard LoRA by enabling the explicit modeling of token-spe -specffcc information. Specifically, the output of TopLoRA can be decomposed into two components follows:
AWxX BEXAX BAX + B(Ex 1)AX,
(8)

---

where I denotes the identity matrix. The first term, BAX, represents the standard LoRA output, capturing global patterns across all input tokens. The second term, B 1)XX, introduces dynamic, input-dependent adaptation mechanism that models fine-grained variations unique to each token. Here, Σx serves as learned gating mechanism that adjusts the importance of different feature dimensions for each token. Note that when Σx I, TopLoRR reverts to standard LoRA. This architecture retains the advantages low-rank parameterization while significantly enhancing LoRA's expressive power, particularly for tasks where tokens require distinct projections.
It is worth noting that MoELoRA 30 adopts the mixture-of experts mechanism and achieves similar token adaptive weights. Specifically, MoELoRA trains multiple LoRA adapters {(A,, B,) > 1 as experts, each specializing in distinct knowledge. also maintains a routing network that dynamically combines the values of these adapters to generate adaptive weights. However, the motivation of this paper differs from that of MoELoRA. MoELoRA utilizes the mixtureoof-experts architecture to cope with the knowledge diversity in training data and downstream tasks. Unlike MoELoRA, our analysis reveals that LoRA'S expressiveness may be constrained by the shared input-output projection. TopLoRA overcomes this limitation more efficiently and directly. Moreover, the TopLoRA structure can be incorporated into each expert in MoELoRA to further enhance its expressiveness. In the following section, we will experimentally demonstrate the superior performance of TopLoRA compared to an improved MoELoRA method.
Experiments
4.1 General Settings
Datasets and Models. In this section, we evaluate TopLoRA on three benchmarks using different model architectures. First, we assess TopLoRA's natural language understanding (NLU) capabilities on the GLUE benchmark (42), which includes eight sub-tasks. The models used are RoBERTa- Base and RoBERTa-Large 28). Next, we examine TopLoRA's natural language generation (NLG) capabilities on two reasoning benchmarks compiled by Hu et al. [[]: mathematical reasoning and commonsense reasoning. which consist of 10k and 170k training samples, respectively. The mathematical reasoning benchmark comprises six sub-tasks, whereas the commonsense reasoning benchmark includes eight. To demonstrate TopLoRA's versatility, we evaluate it across multiple architectures and scales, including Gemma-7B [36], LLaMA-3-8B
[5] and Qwen2.5-14B 14B (7
Baseline Methods. The effectiveness of TopLoRA is demonstrated by comparison with several baseline methods, including LoRA [10] and its variants: DoRA (27), MELoRA (35). and HydraLoRA [37] Each of these methods introduces distinct modifications to the original LoRA framework. For example, DoRA improves LoRA by decomposing pretrained weights into magnitude and direction components. The magnitude is learned through trainable vector, while the direction is updated using LoRA. MELoRA enhances LoRA by incorporating mini-ensemble low-rank adapters, achieving a higher rank at reduced parameter cost compared to the standard LoRA approach. HydraLoRA refines LoRA further with an asymmetric architecture that increases parameter efficiency. It employs shared A matrix and multiple B matrices, which are dynamically routed to optimize performance. Essentially, HydraLoRA improves upon MoELoRA by facilitating better knowledge sharing and differentiation among the experts. These methods represent different strategies for improving LoRA, including better optimization. higher rank, and the mixture-of-expprtt. architecture. By comparing with these techniques,
clearly demonstrate the superiority of TopLoRA
General Hyperparammeters In the main experiments, we evaluate the accuracy of LoRA across ranks 8, 16, and 32 to analyze the trade-off between the number of parameters and model performance. For TopLoRA, we test ranks 8 and 16. For the three LoRA variants, we adjust the number of trainable parameters to match those of LoRA with a rank of 16. Specifically, we set the rank of DoRA to 16, while HydraLoRA A uses a rank of with three B matrices. For MELORA, the Mini-LoRA rank is set to 16, with four Mini-LoRA A groups The general settings include the Adam W optimizer (29, LoRA dropout rate 0.05, and no weight decay. Each experiment is repeated three times, and the average results are reported. Further details can be found in Appendix A and
5

---

Table 1: The accuracy of different methods on General Language Understanding tasks with various pretrained models. The highest average precision bolded, and the hiihhd- underlined.
#Params
RTF
MRPC
STS-B
CoLA
SST-2
QNLI
MNLI
QQP
Avg.
LoRA(r 8)
0.29M
72.56
87.25
87.12
56.10
93.46
91.58
84,89
87
82.55
RoBERTa Base
LoRA(r 16)
0.59M
72.92
87.99
87.46
55.21
93.81
91.89
85.52
87.79
82.82
LoRA(r 32)
1.1MM
75.09
89.22
88.01
58.58
93.58
90.12
85.84
88.37
83.60
DoRA(r 16)
0.61M
74.85
87.83
88.48
56.46
93.39
91.86
85.25
87.89
83.25
MFLoRA(r 16)
0.59M
75.45
88.73
87.27
54.43
93.00
91.51
84.93
87.53
82.86
HydraLoRA(r" = 8)
0.65M
73.65
89.46
88.53
57.03
93.23
91.89
85.52
87.57
83.36
TopLoRA((== = 8)
0.44M
78.34
87.99
90.05
58.55
92.43
92.02
85.60
88,12
84,14
TopLoRA(r= 16)
0.89M
78.34
88.73
88.90
60.34
93.35
92.11
85.65
88.43
84.48
LoRA(r 8)
0.79M
71.96
88.40
89.88
59.76
95.41
93.07
88.67
87.89
84.38
RoBERTa- Large
LoRA(r 16)
1.57M
77.74
88.48
90.60
61.23
95.57
93.72
89.29
88.25
85.61
LoRA(r 32)
3.15M
81.35
89.64
91.45
60.90
95.60
93.76
89.52
88.61
86.35
DoRA(r = 16)
1.62M
80.14
88.73
90.94
61.73
95.68
93.45
89.30
88.29
86.03
MELoRA(r 16)
1.57M
79.48
87.75
90.17
60.59
95.87
93.21
88.86
87.89
85.48
HydraLoRAGr
1.72M
79.42
89.46
90.63
61.07
95.76
93.39
89,25
88.26
85. $5.91
TopLoRA(r
1.8M
80.51
89.30
91.54
61.75
95.64
93.89
89.66
88.83
86,39
TopLoRA(r- 16)
2.36M
85.20
90.44
91.50
64.56
95.64
94.20
89.94
88.94
87.55
4.2 Natural Language Understanding Tasks
Implementation Details. The GLUE benchmark comprises eight sub-tasks, each with distinct training and test sets. For each sub-task, fine-tuned the model on the training set and assessed its accuracy on the corresponding test set. The learning rates for RoBERTa-Base and RoBERTa-Larg were set to 3e-4 and le-4, respectively. A warm-up ratio of 0.03 and linear learning rate decay were used. The number of training epochs varied across sub-tasks; further details are provided in Appendix A TopLoRA and the baseline methods were only applied to the query and value weights.
Results on the GLUE Tasks. As shown in Table increasing the LoRA rank improves fine-tuning performance. Specifically, raising the rank from 8 to 32 increases the average accuracy of LoRA by 1.05% and 1.97% on RoBERTa-Bass and RoBERTa-Large respectively. Compared to LoRA (r = 8). TopLoRA (r = 8) achieves accuracy improvement of 1.5% and 2.01 %, surpassing LoRA (r = 32). Notably, TopLoRA's parameter count is only about 1.5 times that of LoRA at the same rank. The improvements of other baselines are relatively modest. DoRA (r 16) shows an improvement of approximately 0.4% over LoRA (r = 16), while HydraLoRA (r = 8) demonstrates a 0.5% improvement. MELORA (r 16) shows no significant improvement. In contrast, TopLoRA (r 8) uses fewer parameters while yielding significantly higher accuracy, highlighting the effectiveness of token-wise input-output projections. Moreover, increasing the rank of TopLoRA further enhances accuracy. This scalability will be discussed in more detail in Section 4.5
4.3 Natural Language Generation Tasks
Implementation Details. Both mathematical and commonsense reasoning benchmarks include training corpus and multiple test sub-tasks. For each benchmark, we fine-tune the models the training data and assess their performance across all sub-tasks. The learning rate is set to le-4 with 100 warm-up steps and linear decay, and the model is trained for one epoch. Both TopLoRA and baseline methods are applied to the query, key, and value weights. The implementation of these two reasoning tasks strictly adheres the code provided in
Results on the Mathematical Reasoning Tasks. As shown in Table 2 TopLoRA achieves the highest accuracy in mathematical reasoning tasks. Compared to LoRA with the same rank (r 8), TopLoRA improves the average accuracy of the three models by 1.67%, 3.65%, and 1.43%, respectively. Notably, these gains surpass those achieved by increasing LoRA's rank by factor of four, demonstrating that TopLoRA's adaptive input-output projection significantly enhances LoRA's expressiveness. Regarding the three LoRA variants, MELoRA's performance is comparable to even exceeds that of DoRA and Hydral unlike its results on the GLUE dataset. However, the overall improvements provided by these methods remain considerably smaller than those of TopLoRA

---

Table 2:
The accuracy of different methods on Mathematical Reasoning tasks with various pretrained models. The highest average precision is bolded, and the second-highest underlined.
Paaaams
AddSub
MultiArith
SingleEg
GSM8K
AQuA
SVAMP
Avg
LoRA(r 8)
4.82M
87.59
90.33
89.76
56.10
29.13
75.70
71.44
LoRA(r = 16)
9.63M
86.84
92.83
89.57
58.15
30.71
74.90
72.17
Gemma- 7B
LoRA(r = 32)
19.3M
86.58
91.50
91.93
58.45
32.28
75,50
72.71
DoRA(r 16)
9.98M
87.59
94.17
91.34
58.68
27.95
75.80
72.59
MELORA(T
9.63M
87.26
92.22
91.01
59.49
32.15
74.97
72.85
HydraLoRA(r 8)
11.1M
87.34
92.67
90.16
58.83
27.95
75.50
72.08
TopLoRA(r== 8)
6.88M
87.85
94.78
91.80
58.43
30.97
74.87
73.11
TopLoRA(r= 16)
13.8M
86.33
94.83
92.52
59.29
31.10
75.60
73.28
LoRA(r 8)
4.72M
82.28
87.06
91.60
55.65
24.02
68.53
68.19
LoRA(r = 16)
9,44M
84.56
91.22
92.26
57.22
25.72
70.17
70.19
LLama-3-88 88
LoRA(r 32)
18.9M
87.17
93.39
93.50
57.87
26.25
71.83
71.67
DoRA(r 16)
9.63M
85.95
89.67
92.62
56.52
26.19
70.40
70.23
MELoRA(r = 16)
9.44M
85.82
87.83
91.54
55.34
24.41
71,20
69.36
HydraLoRA(r 8)
9.04M
86.08
91.00
91.14
55,50
25.98
68. 10
69.63
TopLoRA(r
7.87M
87.34
92.83
92.91
59.21
24.02
74.70
71.84
TopLoRA(r- 16)
15.7M
88.86
92.17
93.31
61.11
28.74
73.50
72.95
LoRA(r 8)
8.65M
93.16
96.67
92.32
75.66
31.10
85.60
79.09
LoRA(r 16)
17.3M
91.90
96.33
92.91
74.37
34.65
86.40
79.43
Qwen2. 14B
LoRA(r 32)
92.24
97.39
92.98
76.37
34.78
80.15
DoRA(r 16)
17.6M
92.41
96.39
92.39
75.92
36.22
86.80
80.02
MELORAG - 16)
17.3M
92.91
97.33
92.13
75.89
33,86
85.60
79.62
HydraLoRA(1
16.4M
92.41
96.22
92.45
76.32
36.61
86.97
80.16
TopLoRA(r
97.67
93.44
77.31
35.96
87.43
80.52
TopL.oRA(r 16) 29.1M
91.65
98.50
93.90
75.74
37.40
87.40
80.76
Results the Commonsense Reasoning Tasks. As shown in Table TopLoRA also achieves the highest accuracy in commonsense reasoning tasks, and the experimental conclusions were consistent with those mathematical reasoning tasks. At the same rank 8). TopLoRA outperforms LoRA by 2.02%. 1.14%, and 0.61 261% 1% in the average accuracy across the three models, respectively. In comparison, increasing LoRA's rank by four times yields improvements of 0.66%, 1.27%, and 0.45%. This demonstrates that TopLoRA can achieve comparable fine-tuning performance while reducing parameter requirements by nearly fourfold or more. Note that Qwen2.5 1-1 achieves very high accuracy in this task, making the improvements from various methods less noticeable. Additionally, TopLoRA outperforms the other three LoRA variants in accuracy while using fewer parameters.
4.4 Ablation Studies
To optimize the loken-wise diagonal matrix Σx effectively, TopLoRA incorporates RMSNorm and exponential functions into the projector I'g. as shown in formula (7). To assess the impact of these components, we conducted an ablation study on mathematical reasoning tasks. fixing the rank The results in Table inndicate that removing either the exponential function or the normalization step significantly reduces model accuracy. However, omitting the normalization step results in a more substantial performance decline. This is due to the small values of OX; without normalization, the elements Σx remain close to limiting TopLoRA's ability to capture complex patterns. is worth noting that LoRL reverts standard LoRA when Σx
4.5 Scalability Analysis
In this section, we evaluate the scalability of TopLoRA A by varying the LoRA rank and tuning granularity. Our experiments focus on mathematical reasoning tasks using the LLama-3-88 model. To investigate the effect of different LoRA ranks, we tune the rank from the set {2, 8, 16, 32, 64, 128). The results, shown in Figure 2()), demonstrate that TopLoRA consistently surpasses LoRA across all tested ranks. Next, we examine how TopLoRA scales when different tuning granularity are applied. By default, LoRA and LoRA are only applied the query, key, value weights. Here, extend the evaluation include four additional configurations: Q. QV, QKVUD, and
7

---

Table 3: The accuracy of different methods on Commonsense Reasoning tasks with various pre- trained models. The highest average precision bolded, and the second hecond-highest is underlined.
#Param
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-c
ARC-e
OBQA
Avg
LoRA(r 8)
4.82M
70.15
88.96
78.05
94,08
89.82
83.96
94,02
88.60
LoRA(r 16)
9.63M
75.17
88.74
77.58
84.73
88.00
86.43
Gemma 7B
LoRA(r 32)
19.3M
74.34
89.72
78.10
95.77
88.95
84.73
94,11
87.20
86.61
DoRAOr 16)
9.98M
73.49
90.44
79.19
95.03
90.00
84.73
93.79
88.67
86.92
MELoRA(r 16)
9.63M
73.49
89.50
79.99
94.60
89.90
84,30
93.18
89.40
86,79
HydraLoRA(r 8)
1LIM
72.14
89.21
81.30
95.11
89.45
85.32
94,56
89. 00
87.01
TopLoRA(r
6.88M
75.20
90.15
82.80
95.94
90.92
85.67
95.12
88.00
87.97
TopLoRA(r 16)
83.01
LoRA(r
4.72M
73.17
89.34
80.64
87.42
80,20
92.51
87.13
85.45
LoRA(r
9.4MM
73.54
89.50
81.18
94.18
88.00
81.11
93.15
88.53
86.15
Laama: 3.88
LoRA(r 32)
18.9M
73.87
90.01
82.09
94.95
88.37
82.20
93.57
88.73
86.72
DoRA(r 16)
9.63M
73.85
88.79
81.83
94.35
89.11
81.45 14.48
93355
88,
86.40
MELoRA(r 16)
9.44M
73.59
89.77
81.85
94,84
88.29
82,00
93.06
88.93
86.54
HydraLoRAfr = 8)
9.04M
72.66
89.39
80.96
93.84
87.53
81.14
92.85
88.20
85.82
TopLoRA(r
7,87M
73.52
89.50
94,42
87.92
81.74
93.73
86.59
TopLoRA(r 16)
15.7M
74.28
90.10
83.42
94.47
88.00
82.00
94,02
88.40
86.84
LoRA(
8.65M
75.99
93.80
84.34
96.39
92.34
94,03
98.06
95.00
LoRA(r 16)
17.3M
76.15
93.74
84.44
96.87
92.50
94,20
98.36
95.80
91.55
Qwen2.5-1145
LoRA(r 32)
34.6M
76.70
93.91
84,90
96.91
92.42
94,62
98.23
95.80
91.69
DoRA(r = 16)
17.6M
76.68
93.76
84.72
96.97
92.45
94.48
98.20
95.67
91.62
MELoRA(r 16)
17.3M
76.97
93.89
84,90
97.12
91.95
94,62
98.06
96.33
91.73
HydraLoRA(r = 8)
16.4M
76.70
93.42
84.54
96.74
92.74
94,11
98.11
95.80
91.52
TopLoRA(r 5)
14.6M
77.09
84.80
93.29
93.94
98.36
96.60
TopLoRA(r 16) 29.1M
77.28
93.53
85.11
97.17
93.29
94.62
98.44
96.00
91.93
Table The accuracy of LopLoRA on mathematical reasoning tasks using I.Lama3-8BB, without employing the exponential function or the RMSNorm operation.
AddSub
MultiArith
SingleEq
GSM8K
AQuA
SVAMP
Avg
TopLoRA
87.85
94.78
91.80
58.43
30.97
74.87
73.11
Gemma-7B
TopLoRA w.o. Exp
88.10
93.67
90.94
58.15
31.10
75.10
72.84
TopLoRA w.o. RMSNom
88.10
92.83
90.94
58.23
29.53
76.50
72.69
Topl
87.34
92.83
92.91
59.21
24.02
74.70
71.84
L-8
TopLoRA w.o. Exp
83.80
93.83
92.32
56.86
26.77
69.80
70.56
TopLoRA RMSNorm
88.35
94.33
92.52
58.23
25.20
72.00
71.77
TopLoRA
91.31
97.67
93.44
77.31
35.96
87.43
80.52
Qwen2.5-14B
TopLoRA w.o. Exp
92.24
96.61
92.85
76.22
35.83
86.13
79.98
Topl .RRA w.o. RMSNorm
90.38
96.17
92.52
75.74
38.98
87.60
80.23
QKVOGUD, where the symbols O, G, U, and D represent output, gate, up. and down projection weights, respectively. Under the same rank of 8, the results presented in Figure 2(b) further confirm that TopLoRA outperforms LoRA under various tuning granularity.
Related Work
To reduce fine-tuning overhead, LoRA [10] decomposes the weight update AW Rmxn into two low-rank matrices, E IR and B E Rmxx, where the LoRA rank r determines the number of trainable parameters. LoRA be integrated into model without altering its architecture or increasing inference overhead. Recent studies have investigated various aspects of LoRA A to enhance its performance, including better optimization
227, initialization methods [3 45 [8] learning rates
[6). and dynamic parameter allocation (50) 22 as well as the use of higher ranks 33
12 4. [9].
For instance, DoRA [27] decomposes pretrained weights into magnitude and direction components. The magnitude is learned via trainable vector, while the direction is updated using LoRA. PiSSA BI and LoRA-GA [45] perform singular value decomposition (SVD) on pretrained weights and sampled gradients to initialize the matrices A and B. Li et al. [8] present comprehensive analysis of LoRA initialization, showing that non-zero initialization improves the robustness of fine-tuning to variations in learning rate. LoRA+ 6 uses larger learning rate for the matrix B to enhance fine- tuning performance. AdaL
50 dynamically adjusts the LoRA rank for different layers during
fine-tuning. VB-LoRA
22 combines low-rank matrices from different LoRA layers using shared
8

---

LoRA
76
LoRA TopLoRA
75.0
TopLoRA
13.35
Averssee Accurag
72.5-
70/0
72
72:34
90.22
Avcraoe
9343
60.13
$7.5
$4.00
65.73
85.0
64
62.5-
8000
F72
F-10
m-54
1-0
50
3V
DKV
DRNUB
GAVOGUD
Target
Different LoRA Ranks
(b) Different Tuning Granularity
Figure 2: The scalability analysis mathematical reasoning tasks using LLama-3-888 (a) Accuracy of TopLoRA at varying ranks. (b) Accuracy of TopLoRA at different tuning granularity. The figures present only the average accuracy, and the detailed results are available in Appendix
vector bank. HiRA 12 and KronA employ the Hadamard and Kronecker products, respectively, to enhance the rank of LoRA weights. Similar low-rank decompositions based Hadamard and Kronecker products have also been widely used for efficient communication distributed training
13] MELORA (35) achieves higher rank by stacking low-rank matrices along the diagonal. ReL.RA (26) periodically merges learned LoRA adapters into the pretrained weights to increase block matrix multiplication, in which the low-rank factors are partitioned into multiple blocks and each block pair is modulated by distinct diagonal matrix. This design enhances the diversity and rank of LoRA weights, substantially improving expressivity with minimal parameter overhead. More related work on LoRA variants and extensions is provided in Appendix C
20 46
the rank of weight updates. BoRA [99] reformulates LoRA
In this paper, we analyze LoRA from the perspective of input-output projections and express the LoRA weights BA QPQA. where QA and Q denote the input and output spaces, and P denotes the ouput- projection. LoRA-XS [ and LoRA-SB (32 demonstrate a similar structure, where the LoRA weight is decomposed BRA, with A E Rexn and B mmxx being frozen, while only Rex* is trained. Essentially, these methods fix the input and output spaces, learning only the input-output projection. Different from existing studies, TopLoR learns distinct input-output projections for different tokens to further improve the expressiveness of LoRA. Although the MoELoRA [30] and HydraLoRA (37] architectures
different weights for different tokens, they do not adequately address the limitations of shared input-output projections. Moreover, experimental results show that their performance does not match that of TopLoRA.
Conclusion
In this paper, we investigate LoRA from the perspective of input-output projections, decomposing the LoRA weights into three components: the input space, output space, and the input-output projections. Beyond the LoRA rank, we identify another factor that limits its expressiveness: the shared input-output projection in LoRA is insufficient for capturing token-specific information. To address this limitation, propose TopLoRA, which learns token-wise input-output projections in an end-to-end manner. Specifically, by adjusting the LoRA weights with boken-wise diagonal matrices, TopLoRA enables finer-grained adaptation, even with a limited rank. Extensive experiments show that TopLoRA outperforms LoRA and its variants across a range of tasks and model scales. Notably, TopLoRA achieves 2-4% accuracy improvement over LoRA the same rank.
Acknowledgements
This work is supported by the National Natural Science Foundation of China under grants 62376108 62302184 62436003 and 62206102: Major Science and Technology Project of Hubei Province under grant 2025BAB011 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DIC078: and Ant Group through CCF-Ant Research Fund. The computation completed in the HPC Platform of Huazhong University of Science and Technology.

---

References
[1] Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604 2024.
[2] Ziqiang Cui, Yunpeng Weng, Xing Tang, Peiyang Liu, Shiwei Li, Bowei He, Jiamin Chen, Yansen Zhang, Xiuqiang He, and Chen Ma. Core-rag: Lossless compression for retrieval- augmented Ilms via reinforcement learning. arXiv preprint arXiv:2508. 19282, 2025.
[3] Ziqiang Cui, Yunpeng Weng, Xing Tang, Fuyuan Lyu, Dugang Liu, Xiuqiang He, and Chen Ma. Comprehending knowledge graphs with large language models for recommender systems. In International Conference on Research and Development in Information Retrieval (SIGIR), 2025.
[4] Ali Edalati, Marzieh S. Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J. Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2023.
[5] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[6] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024.
[7] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor wards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR), 2022.
and Graham Neubig. To-
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[9] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for NLP. In International Conference Machine Learning (ICMI.), 2019.
[10] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[11] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Conference on Empirical Methods in Natural anguage Processing (EMNLP), 2023.
[12] Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adaptation for large language models. In International Conference Learning Representations (ICLR), 2025.
[13] Nam Hyeon- Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for cewpuuuunniaiioe-efffieeee federated learning. International Conference on Learning Representations (ICI.R), 2022.
[14] Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059, 2021.
[15] Shiwei Li, Huifeng Guo, Lu Hou, Wei Zhang, Xing Tang, Ruiming Tang, Rui Zhang, and Ruixuan Li. Adaptive low- cow-piccision training for embeddings in click-through rate prediction. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
[16] Shiwei Li, Yingyi Cheng, Haozhao Wang, Xing Tang, Shijie Xu, Weihong L Luo, Yuhua Li, Dugang Liu, Xiuqiang He, and Ruixuan Li. Masked random noise for communication-effiiies federated learning, In International Conference on Multimedia (ACM MM), 2024.
10

---

[17] Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang. Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Fedbat: communication--fficiee federated learning via learnable binarization. In International Conference on Machine Learning (ICML.,, 2024.
[18] Shiwei Li, Xiandi Luo, Xing Tang, Haozhao Wang, Hao Chen, Weihong Luo, Yuhua Li, Xiugiang He, and Ruixuan Li. Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics. In International Conference on Machine Learning (ICML), 2025.
[19] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang. Ziqiang Cui, Dugang Liu, Yuhua Li, Xiugiang He, and Ruixuan Li. Bora: Towards more expressive low-rank adaptation with block diversity. arXiv preprint arXiv:2508.000 06953, 2025.
[20] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang. Shijie Xu, Weihong Luo, Yuhua Li, Xiugiang He, and Ruixuan Li. The panaceas for improving low-rank decomposition in communication- efficient federated learning, In International Conference Machine Learning (ICML), 2025.
[21] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL). 2021.
[22] Yang Li, Shaobo Han, and Jonathan Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[23] Yichen Li, Wenchao Xu, Yining Qi, Haozhao Wang, Ruixuan Li, and Song Guo. Sr-fdil: Synergistic replay for federated domain-incremental learning. IEEE Transactions on Parallel and Distributed Systems (TPDS), 2024.
[24] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang. Yi Wang, and Ruixuan Li. Per- sonalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference ( (WWW), 2025.
[25] Yichen Li, Haozhao Wang, Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang, Xin Yang, Rui Zhang, Shui Yu, et al. Unleashing the power of continual learning on non- centralized devices: A survey, IEEE Communications Surveys Tutorials, 2025.
[26] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In International Conference on Learning Repre- sentations (ICLR), 2024.
[27] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024.
[28] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1077:1190. 2019.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[30] Tongxu Luo, Jiahe Lei, Fangyu Lei. Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on serramtter-efffciie fine-tuning for large language models. arXiv preprint arXiv:2402. 12851, 2024.
[31] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[32] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation a silver bullet for extremely efficient low-rank fine-tuning arXiv preprint arxiv:2411. 2024.
[33] Yang Qiu, Yixiong Zou, Jun Wang, Wei Liu, Xiangyu Fu, and Ruixuan Li. Quantifying distributional invariance in causal subgraph for irm-free graph generalization, arXiv preprint 2rXiv:2510.20295. 2025.
11

---

[34] Anastasia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparameter- ization. In Findings of the Association for Computational Linguisties (ACI.), 2023.
[35] Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Melora: Mini-ensemble low-rank adapters for parameter- efficient fine-tuning. In Annual Meeling of the Association for Computational Linguistics (ACL), 2024.
[36] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829, 2024.
[37] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[38] Jintao Tong, Haichen Zhou, Yicong Liu, Yiman Hu, and Yixiong Zou. Dynamic knowledge adapter with probabilistic calibration for generalized few-shot semantic segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[39] Jintao Tong, Yixiong Zou, Yuhua Li, and Ruixuan Li. Lightweight frequency masker for cross-domain few-shot semantic segmentation. In Advances Neural Information Processing Systems (NeurIPS), 2024.
[40] Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li, Yixiong Zou, Yuhong Li, Yuhua Li, and Ruixuan Li. Flowcut: Rethinking redundancy via information flow for efficient vision- laigunlanggaag models. arXiv preprint 1.XXi::2505115336 2025.
[41] Jintao Tong, Ran Ma. Yixiong Zou, Guangyao Chen, Yuhua Li, and Ruixuan Li. Adapter naturally serves as decoupler for cross-domain few-shot semantic segmentation. arXiv preprint arXiv:2506.07376 2025.
[42] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019.
[43] Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: aware federated knowledge distillation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[44] Haozhao Wang, Haoran Xu, Yichen Li, Yuan Xu, Ruixuan Li, and Tianwei Zhang. Fed- CDA: Federated learning with cross-rounds divergence-awe aggregation. In International Conference on Learning Representations (ICLR), 2024.
[45] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[46] Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, Guogang Zhu, and Hao Su. Decoupling general and personalized knowledge in federated lcarning via additive and low-rank decomposition. In ACM International Conference on Multimedia (ACM MM), 2024.
[47] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 2024.
[48] Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In International Conference on Learning Representations (ICLR), 2024.
[49] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances Neural Information Processing Systems (NeurIPS), pages 12360-12371, 2019.
[50] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference Learning Representations (ICLR), 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and scope of this paper: A simple yet powerful extension to LoRA is proposed to improve the fine-tuning performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: potential limitation of TopL discussed Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully discloses all necessary information to reproduce the main experimental results, including datasets, model architectures, and evaluation metrics, which can be found in Section and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper provides open access to the data and code, along with sufficient instructions for reproduction. Relevant information is available in the provided anonymous repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
sooololiccii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all training and test details in Section 4 and Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The statistical significance is discussed in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information on the compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This research conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses both potential societal impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models) used in the paper, properly credited, and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in the paper are properly credited, and their licenses and terms of use are explicitly mentioned and respected. Details can be found in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
The core method development in this research does not involve Large Language Models (LLMs) as any important, original, or non-standard components. LLMs were only used to assist with grammar checking during the paper writing process and did not impact the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy fbttps://neuriae http:///eeeniepecccc//neeeeeeeee cct/ooofeeeencce///2////// for what should or should not be described.
19

---

A Detailed Experimental Settings
Datasets and Models. The GLUE benchmark 42 includes single-sentence classification tasks (CoLA, SST-2), five pairwise text classification tasks (MNLI, RTE, QQP, MRPC, and QNLI), and one text similarity prediction task (STS-B). This paper reports the overall matched and mismatched accuracy for MNLI, Matthew's 's correlation for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks. Due the differing dataset sizes, the number of epochs varies: 10 epochs for RTE and MRPC, 5 epochs for STS-B and CoLA, 2 epochs for SST-2 and QNLI, and 1 epoch for MNLI and QQP. The models used are RoBERTa- Base and RoBERTa-Larg (28].
LoRA Hyperparameters. The scaling factor is set to a = 2r, where r is the LoRA rank. LoRA is applied to the query and value weights with dropout rate of 0.05, using full precision (FP32).
Training Hyperparameters. AdamW [29 used with 31 0.9, 32 = 0.999, < 1e-8, and no weight decay. The learning rate is selected from the set {3e-5, 1e 1e-4, 3e-4, 1e-3), with optimal values of 3 for RoBERTa-Bas and le-4 for RoBERTa-Large.
warm-up ratio of 0.03 is applied, and the batch size is set to 32. The maximum sequence length is 512.
A.1 Experiments on Mathematical and Commonsense Reasoning Tasks
Datasets. Mathematical and commonsense reasoning tasks contain 10K and 170K training samples, respectively, along with several test tasks. Note that we directly utilize the data from [II] for our experiments. The training process consists of single epoch. Three models are employed: Gemma-7B [5] and Qwen2.5-14B 47
36.. LLama-3-88
LoRA Hyperparameters. The scaling factor is set to 2r, where is the LoRA rank. LoRA is applied to the query, key, and value weights with a dropout rate 0.05, using half precision (BF16).
Training Hyperparameters. AdamW is employed with the same settings as previously mentioned. The learning rate is chosen from the set (3e-5, 5, 1e-4, 3e-4, 1e-3) and set to 1.-4. A warm-up of 100 steps is applied, and the batch size is set to 16. The maximum sequence length is 256.
B Additional Experimental Results
B.1 Standard Deviations
In the main experiments, each setting was repeated three times, and the average results are reported. For conciseness, standard deviations are provided in the Appendix. Table shows the standard deviations for each GLUE dataset, where separate model was trained for each. T the standard deviation of the average accuracy for the commonsense and mathematical reasoning tasks, where single model was used across these sub-tasks. Notably, the standard deviation remains stable and is much smaller than the accuracy improvement achieved by TopLoRA
Table 5: The standard deviation of different methods on the GLUE benchmark.
RTE
MRPC
STS-B
CoLA
SST-2
QNLI
MNLI
QQP
LoRA(r 8)
0.51
0.46
0.27
0.57
0.11
0.15
0.11
0.10
LoRA(r 16)
0.84
0.40
0.34
0.64
0.29
0.11
0.04
0,08
LoRA(r 32)
0.78
0.35
0.39
0.56
0.14
1.00
0.11
0.03
DoRA(r 16)
0.95
0.46
0.31
0.57
0.14
0.02
0.24
0.11
RoBERTa-Base
MELORA(r 16)
0.96
0.64
0.62
0.52
0.29
0.14
0.14
0.03
HydraLoRA(r 8)
0.59
0.87
0.75
0.61
0.34
0.17
0.21
0,08
TopLoRA(r 8)
0.89
0.31
0.39
0.38
0.61
0.14
0.11
0,03
TopLoRA(r 16)
0.72
0.58
1.01
0.47
0.29
0.03
0.23
0.16
LoRA(r
0.55
0.81
0.26
0.64
0.50
0.22
0.23
0.05
LoRA(r 16)
0.51
0.20
0.59
0.98
0.19
0.16
0.27
0.04
LoRA(: 32)
0.40
0.66
0.13
0.65
0.33
0.44
0.32
0.02
DoRA(r = 16)
0.48
0.20
0.13
0.56
0.14
0.33
0.13
0.07
RoBERTa-Large
MELoRA(r 16)
0.45
0.35
0.55
0.64
0.34
0.13
0.25
0.04
HydraLoRA(r = 8)
0.67
0.72
0.21
0.29
0.09
0.09
0.40
0.09
TopLoRA(r 8)
0.90
0.31
0.07
0.16
0.09
0.35
0.15
0.04
TopLoRA(r 16)
0.34
0.58
0.14
0.78
0.14
0.12
0.25
0.10
20

---

Table 6: The standard deviation of the average accuracy on mathematical and commonsense reasoning tasks.
Mathematicul Reasoning
Commonsense Reasoning
Gemma-7B
LLama-3-8B
Qwen2.5-14B
Gemma-7B
LLama-3-88
Qwen2.5-114
LoRA(r
0.19
0.88
0.63
0.13
LoRA(r 16)
0.61
0.62
0.50
0.22
0.19
0.10
LoRA(r 32)
0.48
0.80
0.78
0.18
0.29
0,06
DoRA(r 16)
0.58
0,25
0.31
0,46
0.05
0.07
MELoRA(r 16)
0.31
0.67
0.32
0.56
0.16
0.18
BydrrLLRA(r =
0.63
0.39
0.12
0.33
0.24
0.09
TopLoRA(r
0.33
0.30
0.28
0.79
0.26
0.09
TopLoRA(r= 16)
0.52
0.46
0.38
0.43
0.37
0.04
B.2 Detailed Results of the Scalability Analysis
the scalability of TopLoRA evaluated using LLama-3-8B mathematical reasoning tasks from two perspectives: the LoRA rank and tuning granularity. Figure Figure[]rrese presents the average accuracy of different settings across sub-tasks. Complete experimental results are available in Tables and ( for further comparison.
In Section
4.5
Table 7: The accuracy of LoRA and TopLoRA with varying target modules on mathematical reasoning tasks using LLama-3-8B.
Target Modules
Method
#Params
AddSuh
MultiArith
SingleFq
GSMBK
AQuA
SVAMP
Avg
LoRA
2.10M
74.43
86.39
88.45
$2.26
26.64
61.67
64.95
TopLoRA
3.15M
77.81
90.33
89.24
55.60
25.46
62.73
66.86
LoRA
3.41M
81.27
90.00
91.54
56.25
22.83
66.40
68.05
QK
TopLoRA
5.51M
86.58
92.28
93.31
57.57
25.33
71.80
71.14
LoRA
4.72M
82.28
87.06
91.60
55.65
24.02
68.53
68. 19
QKY
TopLoRA
7.87M
87.34
92.83
92.91
59.21
24.02
74.70
71.84
LoRA
14.2M
88.69
92.
93.83
59.79
24.93
73.50
72.14
QKVUD
TopLoRA
22.UM
90.38
93.00
94.29
60.50
26.38
76.10
73.44
LoRA
21.0M
87.59
93.28
93.90
59.41
24.15
75.03
72.23
QKVOGUD
TopLoRA
30.9M
89.28
96.33
94.69
61.41
27.17
75.87
74.12
Table The accuracy of LoRA and TopLoRA with varying ranks on mathematical reasoning tasks using L.ama-3-8B..
Rank
Method
#Params
AddSub
MultiArith
SingleFg
GSM8K
AQuA
SVAMP
Avg
LoRA
1.18M
77.64
85.56
89.24
52.87
23.36
65.53
65.70
TopLoRA
1.97M
81.77
88.39
90.55
55.60
25.59
68.20
68.35
LoRA
2.36M
79.49
86.83
88.85
55.12
23.10
66.03
66.57
r
TopLoRA
3.93M
82.11
89.61
91.27
56.94
25.20
66.57
68.62
LoRA
4.72M
82.28
87.06
91.60
55.65
24.02
68.53
68.1 19
TopLoRA
7.87M
87.34
92.83
92.91
59.21
24.02
74.70
71.84
LoRA
9.44M
84.56
91.22
92.26
$7.22
25.72
70.17
70.19
r= 16
TopLoRA
15.7M
88.86
92.17
93.31
61.11
28.74
73.50
72.95
LoRA
18.9M
87.17
93.39
93.50
57.87
26.25
71.83
71.67
32
TopLoRA
31.5M
89.62
93.67
94.29
60.42
27.17
75.20
73.39
LoRA
37.7M
88.10
94.33
93.70
60.05
25.2()
73.10
72.41
== 64
TopLoRA
62.9M
90.89
97.17
94.88
60.42
24.41
75.10
73.81
LoRA
75.5M
89.87
94.28
93.04
61.13
24.54
75.40
73.04
128
TopLoRA
125.8M
90.38
96.00
94.88
61.56
28.74
74.30
74.31
21

---

C More Related Work
In Section 5 we discuss various LoRA variants. In addition LoRA, two other widely used PEFT methods are the adapter-based and soft prompt-based approaches. The adapter-basel method 9 7 39 introduces new layers into the model and fine-tunes only these layers, significantly reducing resource consumption. The soft prompt-based method [14] [442) adds learnable soft tokens (prompts) to the input, enabling the model to adapt to specific tasks. Through effectiveness, these methods typically introduce computational overhead during inference and thus increase inference latency. In contrast, LoRA applies weight updates directly to the pretrained weights after fine-tuning, avoiding additional inference latency.
More broadly, LoRA can be regarded as a compression technique. A variety of model compression methods have been proposed to reduce computational and storage costs, such as quantization [13]
pruning [66, and knowledge distillation [4]]. We expect to see tighter integration between LoRA
and these complementary approaches to further enhance efficiency. Beyond efficiency, it is also
valuable to explore the deployment of LoRA and TopLoRA in distributed environments [24 44 I7
and continual learning scenarios [25] 23 as well as in addressing outoo--distribution generalization
under distributional shifts (33, which represent promising directions for future research.
D Limitations
A potential limitation of TopLoRA is the inference latency. TopLoRA shows improved accuracy standard LoRA; however, it relies on boken-wise LoRA weights, which cannot be directly integrated into pretrained weights after fine-tuning. This requires additional computations during inference and thus increases inference latency. Similar trade-off's between dynamic adaptation and computational overhead are observed in related approaches, such as MoELoRA (0) and HydraLoRA (37. Future work could explore optimizations to reduce inference costs while preserving TopLoRA's superiority. In addition, this paper does not address certain aspects of TopLoRA: (1) Beyond language models, can TopLoRA be applied to other (c.g., visual) tasks? (2) Can more theoretical or convincing explanations be provided to justify the advantages of TopLoRA? We actively investigating these questions. Nonetheless, we are encouraged by the promising results of TopLoRA in our current experiments and look forward further tests and feedback from the community.
E Broader Impacts
This paper presents TopLoRA, a method that enhances LoRA by learning coken-wise input-output projections. TopLoRA achieves the same fine-tuning performance as LoRA while utilizing fewer parameters, thereby reducing computational overhead and promoting energy efficiency. Additionally, TopLoRA paves the way for new research directions in dynamic weight LoRA. As an extension of the established LoRA framework, TopLoRA does not introduce any significant social concerns that would require further discussion.
22


Reviews:
[{"review_id": "6740JSk6fu", "reviewer": "Reviewer_2KRw", "summary": "This paper aims to improve the performance of Low-Rank Adaptation (LoRA) for fine-tuning large language models. The key insight is that the limitations of standard LoRA stem not only from its rank (which determines input/output space dimensionality) but also from its shared input-output projection across all tokens. TopLoRA addresses this by dynamically adjusting the LoRA weights ($B\\Sigma_X A$) for each input token $X$ through learned diagonal matrices $\\Sigma_X$. It enables token-wise projections while maintaining the original rank. TopLoRA is evaluated on a range of tasks and models, clearly demonstrating its advantages.", "questions": "The paper mentions applying Exp and RMSNorm to diagonal matrices, but their role needs further explanation. Also, does the order in which they are applied matter?\n\nHow does TopLoRA perform on vision tasks?", "limitations": "None", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- This paper is well written and easy to understand. It provides a clear framework to understand LoRA through QR/LQ decomposition, showing how to view LoRA weights from the perspective of input space, output space, and input-output projection.\n- TopLoRA does not follow the common method of increasing rank, but focuses on enhancing projection diversity. Its token-wise diagonal matrix $\\Sigma_X$ is simple and powerful, and can improve expressiveness without increasing rank. This is very inspiring for the optimization of LoRA.\n- The experiments cover three model sizes (7 B, 8 B, 14 B), multiple architectures (RoBERTa, Gemma, LLaMA-3, Qwen 2.5), multiple tasks (GLUE, mathematical reasoning, common sense reasoning), multiple ranks and strong baselines (DoRA, MELoRA, HydraLoRA). Compared with LoRA, the accuracy continues to increase by 2-4% at the same rank, clearly proving the effectiveness of TopLoRA.\n\nWeakness:\n\nAlthough MoELoRA is mentioned (in Section 3.3), only its variant HydraLoRA is compared. A detailed comparison with MoELoRA is needed as the two are very similar.", "was_revised": true, "final_justification": "Thank the authors for their detailed response. My concerns have been well addressed through the additional clarifications and new experimental results.\n\nSeveral reviewers, including myself, suggested adding more baselines (e.g., LoRA+, KronA, HiRA, AdaLoRA, and MoELoRA). I appreciate the authors’ significant efforts in this regard. The method’s orthogonality to existing approaches is now clearly demonstrated, and the inclusion of these baselines provides stronger empirical validation, showing that TopLoRA consistently outperforms these competitive methods. The experimental results are now more comprehensive and convincing.\n\nAccordingly, I raise my score to 5. This paper makes a valuable contribution to the field, and I recommend acceptance.", "rebuttal": "**Hi, Reviewer 2KRw:**\n\nThank you for your positive assessment of the **clarity** and **novelty** of our paper. In response to the concerns regarding comparisons on **vision tasks** and evaluations against **MoELoRA**, we have conducted substantial supplementary experiments and analyses to strengthen our findings. Please find our point-by-point responses below.\n\n***`Q1: \"Comparison with MoELoRA (and other methods).\"`***\n\n**R1:**\nAs suggested, we conducted a more in-depth investigation of MoELoRA. Specifically, we added experiments using MoELoRA with four fully activated experts, each implemented as a rank-4 LoRA module. This configuration yields a total parameter count comparable to that of a rank-16 TopLoRA.\n\nAs shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column reflects the performance improvement over LoRA ($r=16$).\nThe results show that TopLoRA significantly outperforms MoELoRA under a similar parameter budget.\n\nFurthermore, we replaced each expert in MoELoRA with a rank-4 TopLoRA module, creating a variant we refer to as MoETopLoRA. Our findings indicate that the performance gains from MoELoRA and TopLoRA are largely additive, resulting in further improvements.\n\nIn addition, we have included more relevant baseline comparisons, such as the rank-increasing methods HiRA and KronA, the dynamic rank adjustment method AdaLoRA, and the optimization-enhanced method LoRA+. For further details, please refer to **Q1** in our response to Reviewer **SWr9** and **Q5** in our response to Reviewer **2FJn**. Thank you again for your thoughtful feedback.\n\n|  |  | #Params | Mathematical | Accuracy Gain | Commonsense | Accuracy Gain  |\n|---|---|---|---|---|---|---|\n| Gemma-7B | LoRA ($r=16$) | 9.63M | 72.17 | - | 86.43 | -  |\n|  | MoELoRA ($r=4,n=4$) | 10.7M | 72.3 | **0.13** | 86.80 | **0.37**  |\n|  | TopLoRA ($r=16$) | 13.8M | 73.28 | **1.11** | 87.45 | **1.02**  |\n|  | MoETopLoRA ($r=4,n=4$) | 14.9M | 73.49 | **1.32** | 87.62 | **1.19**  |\n|  | LoRA ($r=32$) | 19.3M | 72.71 | **0.54** | 86.61 | **0.18**  |\n| Llama-3-8B | LoRA ($r=16$) | 9.44M | 70.19 | - | 86.15 | -  |\n|  | MoELoRA ($r=4,n=4$) | 11.0M | 71.36 | **1.17** | 86.42 | **0.27**  |\n|  | TopLoRA ($r=16$) | 15.7M | 72.95 | **2.76** | 86.84 | **0.69**  |\n|  | MoETopLoRA ($r=4,n=4$) | 17.2M | 73.33 | **3.14** | 87.28 | **1.13**  |\n|  | LoRA ($r=32$) | 18.9M | 71.67 | **1.48** | 86.72 | **0.57**  |\n| Qwen2.5-14B | LoRA ($r=16$) | 17.3M | 79.43 | - | 91.51 | -  |\n|  | MoELoRA ($r=4,n=4$) | 20.3M | 79.83 | **0.40** | 91.44 | **-0.07**  |\n|  | TopLoRA ($r=16$) | 29.1M | 80.76 | **1.33** | 91.71 | **0.20**  |\n|  | MoETopLoRA ($r=4,n=4$) | 32.1M | 80.65 | **1.22** | 91.73 | **0.22**  |\n|  | LoRA ($r=32$) | 34.6M | 80.15 | **0.72** | 91.69 | **0.18**  |\n\n***`Q2: \"The role (of Exp and RMSNorm) needs further explanation. Also, does the order in which they are applied matter?\"`***\n\n**R2:** \nTo clarify this, we first highlight three key points:\n\n1. TopLoRA extends LoRA by introducing a diagonal matrix $\\Sigma_X$ in the form of $B\\Sigma_XA$. \n$\\Sigma_X$ is generated by a learnable projector based on the input token $X$. Notably, LoRA is a special case where $\\Sigma_X$ is the identity matrix.\n1. When the standard deviation of the parameters in $\\Sigma_X$ is small (i.e., the values are tightly concentrated), $\\Sigma_X$ becomes nearly identical across different tokens, limiting its ability to modulate token-specific weights.\n2. If an element in $\\Sigma_X$ is zero or close to zero, it is equivalent to setting the column of B or the row of A to zero. If there are a large number of values distributed near zero, it may reduce the expressive power of TopLoRA.\n\nGiven that the standard deviation of the projector output is typically small (around 0.05), we first apply RMSNorm to scale $\\Sigma_X$, removing the effects of both input token variance and projector weight variance.\n\nAdditionally, since the projector output tends to have a mean close to zero, we use an exponential (Exp) transformation to shift the distribution of $\\Sigma_X$ from around 0 to around 1.\n\nNote that RMSNorm does not include a mean subtraction step (or implicitly assumes zero mean). Therefore, if we apply the Exp transformation before normalization, we must use LayerNorm instead of RMSNorm to account for the shift in mean. Continuing to use RMSNorm after Exp would fail to properly normalize the standard deviation of $\\Sigma_X$.\n\nWe also added experimental results to compare the effects of different orders of applying the exponential function (Exp) and normalization methods (RMSNorm and LayerNorm). Based on Table 4 in the paper, we introduced two additional settings where Exp is applied first, followed by either RMSNorm or LayerNorm. The results show that there is no significant difference between RMSNorm(Exp($\\cdot$)) and simply applying Exp($\\cdot$) alone. In contrast, LayerNorm(Exp($\\cdot$)) achieves performance close to the default setting, Exp(RMSNorm($\\cdot$)).\n\n| TopLoRA ($r=8$) / Mathematical  | Gemma-7B | Llama-3-8B | Qwen2.5-14B  |\n|---|---|---|---|\n| Exp(RMSNorm($\\cdot$))  (Default Setting) | 73.11 | 71.84 | 80.52  |\n| RMSNorm($\\cdot$) | 72.84 | 70.56 | 79.98  |\n| Exp($\\cdot$) | 72.69 | 71.77 | 80.23  |\n| RMSNorm(Exp($\\cdot$)) | 72.73 | 71.36 | 80.07  |\n| LayerNorm(Exp($\\cdot$)) | 72.97 | 71.63 | 80.58  |\n\n***`Q3: \"How does TopLoRA perform on vision tasks?\"`***\n\n**R3:** \nAs suggested, we have included fine-tuning results of TopLoRA on an image classification task. Specifically, we fine-tuned a pretrained ViT model (google/vit-base-patch16-224-in21k) on the CIFAR-100 dataset for 60 epochs. The learning rate was tuned over {1e-4, 3e-4, 1e-3, 3e-3}, with the final value set to 3e-4. The rank was varied among 8, 16, 32, and 64. Both LoRA and TopLoRA were applied to the query, value, and classifier weights.\nAs shown below, TopLoRA achieves comparable accuracy while reducing the number of trainable parameters by approximately 6×.\n\n|  | LoRA #Params | LoRA Accuracy | TopLoRA #Params | TopLoRA Accuracy | Accuracy Gain  |\n|---|---|---|---|---|---|\n| rank=8 | 0.30M | 88.95 | 0.45M | 90.20 | **1.25**   |\n| rank=16 | 0.60M | 89.96 | 0.91M | 91.16 | **1.20**   |\n| rank=32 | 1.20M | 90.12 | 1.82M | 91.17 | **1.05**   |\n| rank=64 | 2.41M | 90.26 | 3.64M | 91.60 | **1.34**   |"}, {"review_id": "q9uQSdjTWd", "reviewer": "Reviewer_6dXU", "summary": "This paper proposes a PEFT method called Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token. Specifically, it learns a mapping that projects the input token to a scaling matrix that is further multiplied by the LoRA matrix. The authors conducted extensive experiments on various models and datasets, demonstrating that TopLoRA consistently outperforms LoRA and its variants. At the same rank, TopLoRA achieves a 2-3% accuracy improvement over LoRA.", "questions": "Are there any evidence that support the authors' claim \"Existing methods fail to capture token-specific information\"?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n(1) This paper is well motivated by considering token-wise differences, and the proposed method TopLoRA is clear and makes sense to me.\n\n(2) The proposed method is comprehensively evaluated on multiple tasks (General Language Understanding, Mathematical Reasoning, Commonsense Reasoning) on multiple LLMs, and has performance improvement compared to baseline methods. \n\n(3) Some theoretical analysis is shown to make the motivation more convincing.  \n\n**Weaknesses**\n\n(1) The authors claim that their method achieves 2-3% accuracy improvement over LoRA. Firstly, if this comparison is between TopLoRA and LoRA with the same rank, then it will be unfair to LoRA as TopLoRA leads to ~50% trainable parameter increase. Under a fair setting, the improvement of TopLoRA will shrink. In addition, if we compare TopLoRA to stronger LoRA variants, the improvement is smaller or even marginal. \n\n(2) In L113, the authors claim that “Existing methods that apply a uniform input-output projection across all tokens fail to capture token-specific information.” But in my opinion, the original Q,K,V mapping is capturing token-wise information, so the attached updated LoRA layers are capable of capturing token-wise information as well. It seems to me that TopLoRA is actually providing an additional channel that strengthens the token-wise information. In such cases, the methodological contribution of TopLoRA is limited to me.", "was_revised": true, "final_justification": "The author response provides extensive experiments which generally address my concerns. Therefore I updated my score to 4.", "rebuttal": "**Hi Reviewer 6dXU:**\n\nThank you for your recognition of our **motivation**, **methodological clarity**, and **extensive experiments**. We notice that the concerns mainly relate to the **additional parameter cost of TopLoRA** and its **conceptual distinction from attention**. We have carefully addressed both points through further clarification and empirical evidence. Below, we provide responses to each point individually. \n\n***`Q1: \"Unfair comparison between LoRA and TopLoRA.\"`***\n\n**R1:**\nDue to the improved structure, TopLoRA introduces approximately 50% more parameters than LoRA at the same rank. \nHowever, the performance gains it achieves significantly surpass those obtained by merely increasing the rank of LoRA. \nAs noted in line 227, \"these gains surpass those achieved by increasing LoRA’s rank by a factor of four\".\nFor example, on Mathematical Reasoning tasks with Gemma-7B, LoRA (r=32) requires **19.3M** trainable parameters to achieve an accuracy of **72.71**, whereas TopLoRA (r=8) achieves a higher accuracy of **73.11** with only **6.88M** parameters.\n\nTo ensure a fair comparison, we evaluated TopLoRA (rank=5/10/20) and compared it with LoRA (rank=8/16/32). In these settings, they use a similar number of parameters. As shown below, we report the average accuracy on mathematical and commonsense reasoning tasks. The *Accuracy Gain* column indicates the performance gain of TopLoRA over LoRA under comparable parameter budgets.\n**TopLoRA achieves a 1–3% boost in accuracy with a similar number of parameters**. Notably, increasing the LoRA rank by a factor of four yields only a modest 1–2% improvement.\n\n|||#Params|Mathematical|Accuracy Gain|Commonsense|Accuracy Gain|\n|-|-|-|-|-|-|-|\n|Gemma-7B|LoRA($r=8$)|4.82M|71.44|-|85.95|-|\n||TopLoRA($r=5$)|4.30M|72.97|**1.53**|87.68|**1.73**|\n||LoRA($r=16$)|9.63M|72.17|-|86.43|-|\n||TopLoRA($r=10$)|8.60M|73.16|**0.99**|87.92|**1.49**|\n||LoRA($r=32$)|19.3M|72.71|-|86.61|-|\n||TopLoRA($r=20$)|17.2M|74.14|**1.43**|88.44|**1.83**|\n|Llama-3-8B|LoRA($r=8$)|4.72M|68.19|-|85.45|-|\n||TopLoRA($r=5$)|4.92M|70.94|**2.75**|86.37|**0.92**|\n||LoRA($r=16$)|9.44M|70.19|-|86.15|-|\n||TopLoRA($r=10$)|9.83M|72.37|**2.18**|86.68|**0.53**|\n||LoRA($r=32$)|18.9M|71.67|-|86.72|-|\n||TopLoRA($r=20$)|19.7M|73.52|**1.85**|87.92|**1.20**|\n|Qwen2.5-14B|LoRA($r=8$)|8.65M|79.09|-|91.24|-|\n||TopLoRA($r=5$)|9.09M|80.11|**1.02**|91.60|**0.36**|\n||LoRA($r=16$)|17.3M|79.43|-|91.51|-|\n||TopLoRA($r=10$)|18.2M|80.87|**1.44**|91.93|**0.42**|\n||LoRA($r=32$)|34.6M|80.15|-|91.69|-|\n||TopLoRA($r=20$)|36.4M|81.25|**1.10**|92.85|**1.16**|\n\nOn the other hand, inspired by Reviewer SWr9, we explored **sharing a common set of learnable projectors across different layers** for weight modulation. The results show that **a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA**. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy. Please kindly check **Q5** in our response to Reviewer **SWr9** for experimental results.\n\n***`Q2: \"If we compare TopLoRA to stronger LoRA variants, the improvement is smaller or even marginal.\"`***\n\n**R2:** \nCompared to stronger LoRA variants, our approach still achieves substantial improvements.\n\nOur paper benchmarks several state-of-the-art methods, including MELoRA, DoRA, and HydraLoRA. The results show that TopLoRA achieves higher accuracy while using fewer parameters. Please kindly check Tables 1/2/3 in the paper and compare TopLoRA (r=8) with DoRA (r= 16), MoELoRA (r=16), and HydraLoRA (r=8), respectively.\n\nWe also include comparisons with other strong baselines here, such as the higher rank methods HiRA and KronA, and the dynamic rank adjustment method AdaLoRA. Note that KronA does not have a rank hyperparameter, and its number of trainable parameters is fixed and non-adjustable. \nDue to their architectural modifications, HiRA and KronA require larger learning rates to perform well. Accordingly, we tuned the learning rate over {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and report the best results. AdaLoRA is configured with an initial rank of 24 and a target rank of 16. TopLoRA uses a rank of 10 to maintain similar parameter counts as the baselines.\n\nAs shown below, the *Accuracy Gain* column reflects performance gains relative to standard LoRA. \nKronA shows inferior accuracy compared to LoRA, likely due to its extremely small number of trainable parameters. Nevertheless, TopLoRA consistently outperforms all of these state-of-the-art methods under comparable parameter budgets.\n\nImportantly, as shown in the table in **R1**, doubling or even quadrupling the LoRA rank does not lead to accuracy gains comparable to those achieved by TopLoRA—gains that these baselines fail to match. The improvements delivered by TopLoRA are substantial, underscoring the effectiveness of its architectural design.\n\n\n|||#Params|Mathematical|Accuracy Gain|Commonsense|Accuracy Gain|\n|-|-|-|-|-|-|-|\n|Gemma-7B|LoRA($r=16$)|9.63M|72.17|-|86.43|-|\n||KronA|0.60M|69.46|**-2.71**|84.49|**-1.94**|\n||HiRA($r=16$)|9.63M|72.91|**0.74**|87.09|**0.66**|\n||AdaLoRA($r=16$)|9.63M|72.65|**0.48**|86.98|**0.55**|\n||TopLoRA($r=10$)|8.60M|73.16|**0.99**|87.92|**1.49**|\n|Llama-3-8B|LoRA($r=16$)|9.44M|70.19|-|86.15|-|\n||KronA|0.52M|67.41|**-2.78**|83.97|**-2.18**|\n||HiRA($r=16$)|9.44M|70.96|**0.77**|86.38|**0.23**|\n||AdaLoRA($r=16$)|9.44M|70.50|**0.31**|86.20|**0.05**|\n||TopLoRA($r=10$)|9.83M|72.37|**2.18**|86.68|**0.53**|\n|Qwen2.5-14B|LoRA($r=16$)|17.3M|79.43|-|91.51|-|\n||KronA|0.95M|78.73|**-0.70**|89.36|**-2.15**|\n||HiRA($r=16$)|17.3M|79.59|**0.16**|91.78|**0.27**|\n||AdaLoRA($r=16$)|17.3M|79.65|**0.22**|91.49|**-0.02**|\n||TopLoRA($r=10$)|18.2M|80.87|**1.44**|91.93|**0.42**|\n\n***`Q3: \"The original Q,K,V mapping is capturing token-wise information, so the attached updated LoRA layers are capable of capturing token-wise information as well ... In such cases, the methodological contribution of TopLoRA is limited to me.\"`***\n\n**R3:** \nThis is an interesting question​​, and we appreciate the opportunity to clarify the distinction between token interactions (handled by attention) and token-specific processing (our focus within LoRA layers), as well as why TopLoRA represents a methodological innovation.\n\n​**1. Roles of Attention vs. LoRA:**\nThe QKV mappings in attention are designed to capture inter-token dependencies (how tokens influence one another). In contrast, LoRA layers operate prior to attention, transforming individual token representations. The ability of LoRA to preserve token-specific information is therefore orthogonal to the function of attention.\n\n**2. ​​Limitation of Standard LoRA​​:**\nAs noted in Eq.(1) in our paper, LoRA weights $BA=Q_BPQ_A$​ projects inputs into a fixed subspace (spanned by $Q_A$). This means that token information orthogonal to $Q_A$ is irrecoverably lost in the transformation. Increasing rank (more $Q_A$ columns) expands this subspace but remains static across tokens, and it cannot adapt to token-specific features.\n\n**3. TopLoRA’s Innovation​​:**\nInstead of expanding rank, TopLoRA generates token-dependent diagonal matrices $\\Sigma_X$ to dynamically modulate the input-output mapping per token. This token-aware reweighting enriches the expressiveness of LoRA by allowing the transformation to adapt to each token’s unique features.\n\n**​4. Why This Matters for Attention​​:**\nBetter token-specific transformations (via TopLoRA) lead to higher-quality QKV representations. Attention can only model inter-token relationships effectively if the underlying QKV mappings already capture rich, discriminative token features.\n\n​​In summary​​, TopLoRA is not just \"another channel\" for token-wise information but a fundamental enhancement to LoRA’s capacity to preserve token-specific features—a prerequisite for attention to function optimally. This is a distinct contribution beyond static low-rank adaptations.\n\n\n***`Q4: \"Evidence that support the claim 'Existing methods fail to capture token-specific information'\"`***\n\n**R4:** \nAs noted in Eq.(1) of our paper, the standard LoRA formulation $BA=Q_BPQ_A$projects token representations into a fixed low-dimensional subspace spanned by $Q_A$. This inherently limits the model’s ability to capture token-specific variations: any input component orthogonal to this subspace is irrecoverably lost. Consequently, when the rank is small—as is often necessary for efficiency—the amount of discarded token-specific information increases.\n\nTo further quantify this limitation, we compute the **mutual information (MI)** between input tokens $X$ and output representations $Y$ using the InfoNCE lower bound [1], a standard technique in representation learning. Each input-output pair $(X_i,Y_i)$ is treated as positive, with others as negatives. A higher MI indicates better preservation of token-specific information. This method allows us to provide a tractable, quantitative comparison of how much token-level information is retained.\n\nDue to the image upload limitations during rebuttal, we summarize a subset of the results on LLaMA-3-8B (Math) in the table below and will include full visualizations and analysis in the revised paper.\n\n||Layer5|Layer10|Layer20|\n|-|-|-|-|\n|LoRA($r=8$)|1.13|0.62|1.96|\n|LoRA($r=16$)|1.56|0.76|2.12|\n|TopLoRA($r=8$)|2.31|1.02|2.97|\n\nThese results show that **TopLoRA retains significantly more mutual information than standard LoRA**, even at smaller ranks, confirming its stronger ability to capture token-specific features. Together with the consistent performance improvements shown in our experiments, this provides strong support for our central claim: TopLoRA’s dynamic modulation enhances the expressiveness of LoRA.\n\n[1] A. van den Oord et al., Representation Learning with Contrastive Predictive Coding, 2018."}, {"review_id": "WfwKvme5Lt", "reviewer": "Reviewer_SWr9", "summary": "This paper reformulates the low-rank adapter to show the limited accessibility of the current LoRA residuals to input-output subspaces. Coupled with the static behavior of conventional LoRA implementation when it comes to different tokens in varying contexts, they propose to modify this formulation to allow for token-dependent adjustments on the residual during both training and inference. This is done by adding an additional set of weights that transform the activations from the previous layer using normalization and exponential operators to a new vector. This vector is then added to a diagonal matrix within the new formulation of the adapter.", "questions": "- Regarding the statement “different tokens exhibit distinct semantics and distributions, and the shared input-output projection” in L51, given that both model weights and LoRA weights can learn to map different tokens differently, the only way for me to make sense of this statement is to relate it to the rank again. That way, I can interpret this in the sense that when the rank is low, there are fewer subspaces available to express every concept represented by the tokens in the fine-tuning dataset and additional token-dependent mechanisms are required. Is that what you mean by L51? The reason for the question is that you start the paragraph with “beyond the LoRA rank…” and that makes me think that you’re stating this is something completely disentangled with respect to the rank.", "limitations": "- An additional limitation would also be the dependence of the additional weights on the rank of the adapter. This also makes me wonder if it is possible to share the overhead between layers by attributing a single $\\Sigma_X$ operation for the whole model.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The writing is clear and fluid.\n- The experiments are comprehensive and tests are done on a variety of models.\n- The proposed reformulation is straightforward.\n\nWeaknesses:\n- The statement \"LoRA typically exhibits a performance gap when compared to FFT\" in L29 means that you'd need to compare to methods addressing this gap. You've already compared to DoRA but other methods that address the same problem are not compared against, e.g. LoRA+ [a].\n- Similar works like MoELoRA have already been mentioned by the authors but dismissed due to differences in goals. I understand the difference in goals and that MoELoRA should not be compared against as a SoTA for this work but it would have been complementary to see combinations of the two works as an additional experiment.\n- A parameter count on Figure 2 would complement the visualization.\n\n[a] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"LoRA+: Efficient Low Rank Adaptation of Large Models.\" Proceedings of the Forty-first International Conference on Machine Learning, 2024. https://openreview.net/forum?id=NEv8YqBROO.", "was_revised": true, "final_justification": "All my concerns in the initial review were addressed by the authors during rebuttal so I changed my score to 5 for my final review. I don't necessarily agree with the reasons of rejection by Reviewer 2FJn, especially considering the rebuttal answers (the answer to Q3 of this reviewer being the most important one).", "rebuttal": "**Hi, Reviewer SWr9:**\n\nThank you for your positive assessment of the **clarity** of our writing and the **comprehensiveness** of our experiments. \nRegarding your questions on **comparisons with LoRA+** and the **integration with MoELoRA**, we have conducted substantial additional analysis to strengthen our response. Please find our point-by-point responses below.\n\n***`Q1: \"Comparison with LoRA+.\"`***\n\n**R1:** \nFollowing your suggestion, we have added comparisons with LoRA+. \n\nThe core idea of LoRA+ is to apply a larger learning rate to the $B$ matrix, i.e., $\\beta = \\eta_B/\\eta_A >1$. \nWe tuned $\\beta$ over the range {2, 4, 8, 16}, and reported the best-performing result. The rank of both LoRA and LoRA+ is set to 16, while for TopLoRA, we used a rank of 10 to ensure comparable parameter counts across methods.\n\nAs shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column reflects the performance improvement over standard LoRA. With a similar number of parameters, **TopLoRA consistently achieves higher accuracy than LoRA+**.\n\nImportantly, **our method is orthogonal to LoRA+**. Specifically, we can also apply a larger learning rate to the $B$ matrix in TopLoRA—a variant we refer to as **TopLoRA+**, which further improves accuracy by approximately 1%. These results will be included in the revised version of the paper.\n\n|  |  | #Params | Mathematical | Accuracy Gain | Commonsense | Accuracy Gain  |\n|---|---|---|---|---|---|---|\n| Gemma-7B | LoRA ($r=16$) | 9.63M | 72.17 | - | 86.43 | -  |\n|  | LoRA+ ($r=16$) | 9.63M | 72.74 | **0.57**  | 87.48 | **1.05**   |\n|  | TopLoRA ($r=10$) | 8.60M | 73.16 | **0.99**  | 87.92 | **1.49**   |\n|  | TopLoRA+ ($r=10$) | 8.60M | 73.86 | **1.69**  | 88.57 | **2.14**   |\n| Llama-3-8B | LoRA ($r=16$) | 9.44M | 70.19 | - | 86.15 | -  |\n|  | LoRA+ ($r=16$) | 9.44M | 71.61 | **1.42**  | 86.5 | **0.35**   |\n|  | TopLoRA ($r=10$) | 9.83M | 72.37 | **2.18**  | 86.68 | **0.53**   |\n|  | TopLoRA+ ($r=10$) | 9.83M | 73.59 | **3.40**  | 87.33 | **1.18**   |\n| Qwen2.5-14B | LoRA ($r=16$) | 17.3M | 79.43 | - | 91.51 | -  |\n|  | LoRA+ ($r=16$) | 17.3M | 80.39 | **0.96**  | 91.62 | **0.11**   |\n|  | TopLoRA ($r=10$) | 18.2M | 80.87 | **1.44**  | 91.93 | **0.42**   |\n|  | TopLoRA+ ($r=10$) | 18.2M | 81.34 | **1.91**  | 92.06 | **0.55**   |\n\n\n***`Q2: \"Combination of MoELoRA and TopLoRA.\"`***\n\n**R2:** Based on your suggestions, we conducted a more in-depth investigation of MoELoRA. First, we added experiments using MoELoRA with four fully activated experts, each implemented as a rank-4 LoRA module. This configuration results in a parameter count comparable to that of a rank-16 TopLoRA. \n\nAs shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column reflects the performance improvement over LoRA ($r=16$).\nThe results show that TopLoRA significantly outperforms MoELoRA under a similar parameter budget.\n\nFurthermore, we replaced each expert in MoELoRA with a rank-4 TopLoRA module, creating a variant we refer to as **MoETopLoRA**. Our findings indicate that the performance gains from MoELoRA and TopLoRA are largely additive, resulting in further improvements. The corresponding content will be included in the revised version of the paper.\n\n|  |  | #Params | Mathematical | Accuracy Gain | Commonsense | Accuracy Gain  |\n|---|---|---|---|---|---|---|\n| Gemma-7B | LoRA ($r=16$) | 9.63M | 72.17 | - | 86.43 | -  |\n|  | MoELoRA ($r=4,n=4$) | 10.7M | 72.3 | **0.13** | 86.80 | **0.37**  |\n|  | TopLoRA ($r=16$) | 13.8M | 73.28 | **1.11** | 87.45 | **1.02**  |\n|  | MoETopLoRA ($r=4,n=4$) | 14.9M | 73.49 | **1.32** | 87.62 | **1.19**  |\n|  | LoRA ($r=32$) | 19.3M | 72.71 | **0.54** | 86.61 | **0.18**  |\n| Llama-3-8B | LoRA ($r=16$) | 9.44M | 70.19 | - | 86.15 | -  |\n|  | MoELoRA ($r=4,n=4$) | 11.0M | 71.36 | **1.17** | 86.42 | **0.27**  |\n|  | TopLoRA ($r=16$) | 15.7M | 72.95 | **2.76** | 86.84 | **0.69**  |\n|  | MoETopLoRA ($r=4,n=4$) | 17.2M | 73.33 | **3.14** | 87.28 | **1.13**  |\n|  | LoRA ($r=32$) | 18.9M | 71.67 | **1.48** | 86.72 | **0.57**  |\n| Qwen2.5-14B | LoRA ($r=16$) | 17.3M | 79.43 | - | 91.51 | -  |\n|  | MoELoRA ($r=4,n=4$) | 20.3M | 79.83 | **0.40** | 91.44 | **-0.07**  |\n|  | TopLoRA ($r=16$) | 29.1M | 80.76 | **1.33** | 91.71 | **0.20**  |\n|  | MoETopLoRA ($r=4,n=4$) | 32.1M | 80.65 | **1.22** | 91.73 | **0.22**  |\n|  | LoRA ($r=32$) | 34.6M | 80.15 | **0.72** | 91.69 | **0.18**  |\n\n\n***`Q3: \"Parameter count on Figure 2.\"`***\n\n**R3:** Thank you for the suggestion. We will add an additional vertical axis in Figure 2 to indicate the number of parameters. As image updates are not allowed during the rebuttal stage, we are currently unable to present the revised figure. However, this is a relatively straightforward modification, and we will incorporate it in the revised version.\n\n\n***`Q4: \"the only way for me to make sense of this statement (L51) is to relate it to the rank again.\"`***\n\n**R4:** Below is our response to the question raised by the reviewer in the Questions section.\n\nFirst, we believe that the reviewer’s understanding is correct, our proposed method is not completely rank-independent. As shown in Equation (1), the LoRA weights are expressed as $𝐵𝐴=Q_B P Q_A$. The rank determines the dimensions of the input space ($Q_A$) and output space ($Q_B$); increasing the rank expands both, thereby enhancing the expressive capacity of LoRA.\n\nIn this paper, we argue that beyond increasing the rank, LoRA’s expressiveness can also be improved by enriching the mapping from the input space to the output space. In standard LoRA, a fixed projection matrix $P$ is applied uniformly to all tokens $X$. We propose generating a diagonal matrix $\\Sigma_X$ based on the input token $X$ to modulate $P$, thereby introducing token-wise adaptation and improving expressive capacity of LoRA.\n\nWe realize that the reviewer’s question likely stems from the phrasing \"beyond the LoRA rank...\" in Line 51. A more precise statement would be \"beyond increasing the LoRA rank...\" which is consistent with our framing in the title: \"Beyond Higher Rank\".\n\n\n***`Q5: \"It is possible to share the overhead between layers by attributing a single operation for the whole model.\"`***\n\n**R5:** \nThank you for this insightful suggestion, it was truly inspiring for us. Indeed, the learnable projector in TopLoRA can be reconsidered from the perspective of sharing across layers.\n\nIn the original design of TopLoRA, each layer learns a distinct projector to modulate the LoRA weights based on the input token. However, if we temporarily ignore the layer-wise structure, input tokens from different layers can be grouped together and share a common set of projectors for weight modulation.\n\nMotivated by this idea, we conducted experiments on LLaMA-3-8B with a rank of 8, sharing a fixed number of learnable projectors $N\\in\\\\{1,2,4,8,16\\\\}$ across different layers. As shown below, we report the average accuracy on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column indicates the performance improvement relative to standard LoRA. \nThe results show that **a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA**. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy.\nThe revised version of the paper will include these discussions and results.\n\nOnce again, we sincerely thank the reviewer for this valuable comment, which has meaningfully improved our work. \n\n| LLaMA-3-8B ($r=8$) | #Params | Mathematical | Accuracy Gain | Commonsense | Accuracy Gain  |\n|---|---|---|---|---|---|\n| LoRA | 4.72M | 68.19 | - | 85.45 | -  |\n| TopLoRA | 7.87M | 71.84 | **3.65**  | 86.59 | **1.14**   |\n| TopLoRA ($N=1$) | 4.75M | 68.39 | **0.20**  | 85.56 | **0.11**   |\n| TopLoRA ($N=2$) | 4.78M | 69.06 | **0.87**  | 86.26 | **0.81**   |\n| TopLoRA ($N=4$) | 4.85M | 70.63 | **2.44**  | 86.17 | **0.72**   |\n| TopLoRA ($N=8$) | 4.98M | 71.98 | **3.79**  | 86.34 | **0.89**   |\n| TopLoRA ($N=16$) | 5.24M | 71.97 | **3.78**  | 86.55 | **1.10**  |"}, {"review_id": "pZt6Xpwb22", "reviewer": "Reviewer_2FJn", "summary": "The author proposes a token-wise projected LoRA method, which dynamically adjusts the LoRA weights for different tokens. It outperforms the standard LoRA method across multiple benchmarks.", "questions": "Q1: While the paper spends a lot of effort discussing its motivations, it lacks experimental support for its claimed advantages. Can the authors provide additional explanations or empirical evidence to better justify the motivations behind their proposed approach?\n\nQ2: In the abstract, the authors state that standard LoRA \"all input tokens share the same weight\", which seems to contradict the traditional LoRA framework. Can the authors explain further and how it differs from the TopLoRA design?\n\nQ3: The paper TopLoRA claims to improve expressiveness under low-rank constraints, but TopLoRA introduces about 50% more parameters than LoRA of the same rank. How do the authors distinguish between the performance improvement brought by the architectural improvements and the increased number of parameters?\n\nQ4: Since $\\Delta W$ becomes token-dependent, does this method give up the mergeability and inference efficiency advantages that LoRA usually has? Can the authors quantify the additional inference cost and comment on whether this trade-off is reasonable?\n\nQ5: Why are relevant baselines such as HiRA, KronA, and AdaLoRA omitted from the comparison? Inclusion of these approaches provides a clearer presentation of the relative strengths and weaknesses of TopLoRA.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe authors demonstrate superior results in multiple datasets.\n\nWeaknesses:\n\n1. The authors devote a substantial portion of the paper to describing the motivation, but lack experimental support.\n2. The authors mention in the abstract that \"all input tokens share the same weights.\" This appears to contradict the standard LoRA design, where each module has its own set of independent weights without weight sharing across tokens. A more detailed explanation or clarification would be appreciated.\n3. While the authors state in line 116 that TopLoRA improves the expressiveness of LoRA under low-rank settings, the experiments reveal that, for the same rank, TopLoRA introduces about 50% more parameters than standard LoRA. This raises the question of whether the observed performance improvements are due to the proposed design or merely the increased number of parameters. The current experiments lack sufficient evidence to disentangle these two factors.\n4. Since the proposed method learns a distinct $\\theta$ for each token, $\\Delta W$ becomes token-dependent. This raises a concern about whether the approach sacrifices the mergeability advantage of standard LoRA and incurs additional computational cost during inference. It would be helpful if the authors could clarify this trade-off and discuss its impact on inference efficiency.\n5. The authors have not compared their method with other rank enhancement techniques such as HiRA and KronA, or with rank adaptation methods like AdaLoRA. Including such comparisons would strengthen the experimental evaluation and better position the proposed approach within the existing literature.\n6. This paper lacks a related work section.\n7. There is an error in line 40: a unitary matrix must be square, but $Q_B$ and $Q_A$ do not satisfy this condition. This needs to be corrected or further clarified.", "was_revised": true, "final_justification": "The author provides a more in-depth motivation experiment and achieves better results compared with multiple baseline methods under fair conditions, which solves my concern, so I raise the score to 4.", "rebuttal": "**Hi, Reviewer 2FJn:**\n\nThank you for recognizing TopLoRA’s **superior accuracy**, and for your detailed and insightful feedback. It appears that your main concerns relate to the **empirical support for our motivation** and how our method **compares to other baseline methods**. Please find our point-by-point responses below. \n\n***`Q1: \"lack experimental support for the motivation.\"`***\n\n**R1:** \nAs noted in Eq.(1) of our paper, the standard LoRA formulation $BA=Q_BPQ_A$projects token representations into a fixed low-dimensional subspace spanned by $Q_A$. This inherently limits the model’s ability to capture token-specific variations: any input component orthogonal to this subspace is irrecoverably lost. Consequently, when the rank is small—as is often necessary for efficiency—the amount of discarded token-specific information increases.\n\nTo further quantify this limitation, we compute the **mutual information (MI)** between input tokens $X$ and output representations $Y$ using the InfoNCE lower bound [1], a standard technique in representation learning. Each input-output pair $(X_i,Y_i)$ is treated as positive, with others as negatives. A higher MI indicates better preservation of token-specific information. This method allows us to provide a tractable, quantitative comparison of how much token-level information is retained.\n\nDue to the image upload limitations during rebuttal, we summarize a subset of the results on LLaMA-3-8B (Math) in the table below and will include full visualizations and analysis in the revised paper.\n\n||Layer5|Layer10|Layer20|\n|-|-|-|-|\n|LoRA($r=8$)|1.13|0.62|1.96|\n|LoRA($r=16$)|1.56|0.76|2.12|\n|TopLoRA($r=8$)|2.31|1.02|2.97|\n\nThese results show that **TopLoRA retains significantly more mutual information than standard LoRA**, even at smaller ranks, confirming its stronger ability to capture token-specific features. Together with the consistent performance improvements shown in our experiments, this provides strong support for our central claim: TopLoRA’s dynamic modulation enhances the expressiveness of LoRA.\n\n[1] A. van den Oord et al., Representation Learning with Contrastive Predictive Coding, 2018.\n\n\n***`Q2: \"each (LoRA) module has its own set of independent weights without weight sharing across tokens.\"`***\n\n**R2:** \nWe believe there may be a misunderstanding. When we state that different tokens share the same LoRA weights, we mean that **within a given LoRA layer (or module), each input token $X$ (from a sequence of multiple tokens) is processed using the same set of LoRA weights**. This can be formally expressed as $Y=(W+BA)X$, where the same LoRA weights $BA$ are applied uniformly to all tokens in the sequence. \nTo introduce token-specific modulation, we propose generating a diagonal matrix $\\Sigma_X$ based on the input token $X$, which is used to adjust the LoRA weights for each token in the form $B\\Sigma_X A$.\n\n***`Q3: \"This raises the question of whether the performance improvements are due to the proposed design or merely the increased number of parameters.\"`***\n\n**R3:** \nDue to its improved structure, TopLoRA introduces approximately 50% more parameters than LoRA at the same rank. \nHowever, the performance gains it achieves significantly surpass those obtained by merely increasing the rank of LoRA. \nAs noted in line 227, \"these gains surpass those achieved by increasing LoRA’s rank by a factor of four\".\nFor example, on Mathematical Reasoning tasks with Gemma-7B, LoRA (r=32) requires **19.3M** trainable parameters to achieve an accuracy of **72.71**, whereas TopLoRA (r=8) achieves a higher accuracy of **73.11** with only **6.88M** parameters.\n\nTo ensure a fair comparison, we evaluated TopLoRA with ranks of 5, 10, and 20, and compared it with LoRA at ranks of 8, 16, and 32. In these settings, both methods use a similar number of parameters. As shown in the table below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column indicates the performance improvement of TopLoRA over LoRA under comparable parameter budgets.\n**TopLoRA achieves a 1–3% boost in accuracy with a similar number of parameters.** Notably, increasing the LoRA rank by a factor of four yields only a modest **1–2%** improvement.\nTherefore, we can confidently attribute TopLoRA’s performance gains to its structural design, rather than the approximately 50% increase in parameter count.\n\n|||#Params|Mathematical|Accuracy Gain|Commonsense|Accuracy Gain|\n|-|-|-|-|-|-|-|\n|Gemma-7B|LoRA($r=8$)|4.82M|71.44|-|85.95|-|\n||TopLoRA($r=5$)|4.30M|72.97|**1.53**|87.68|**1.73**|\n||LoRA($r=16$)|9.63M|72.17|-|86.43|-|\n||TopLoRA($r=10$)|8.60M|73.16|**0.99**|87.92|**1.49**|\n||LoRA($r=32$)|19.3M|72.71|-|86.61|-|\n||TopLoRA($r=20$)|17.2M|74.14|**1.43**|88.44|**1.83**|\n|Llama-3-8B|LoRA($r=8$)|4.72M|68.19|-|85.45|-|\n||TopLoRA($r=5$)|4.92M|70.94|**2.75**|86.37|**0.92**|\n||LoRA($r=16$)|9.44M|70.19|-|86.15|-|\n||TopLoRA($r=10$)|9.83M|72.37|**2.18**|86.68|**0.53**|\n||LoRA($r=32$)|18.9M|71.67|-|86.72|-|\n||TopLoRA($r=20$)|19.7M|73.52|**1.85**|87.92|**1.20**|\n|Qwen2.5-14B|LoRA($r=8$)|8.65M|79.09|-|91.24|-|\n||TopLoRA($r=5$)|9.09M|80.11|**1.02**|91.60|**0.36**|\n||LoRA($r=16$)|17.3M|79.43|-|91.51|-|\n||TopLoRA($r=10$)|18.2M|80.87|**1.44**|91.93|**0.42**|\n||LoRA($r=32$)|34.6M|80.15|-|91.69|-|\n||TopLoRA($r=20$)|36.4M|81.25|**1.10**|92.85|**1.16**|\n\nOn the other hand, inspired by Reviewer SWr9, we explored **sharing a common set of learnable projectors across different layers** for weight modulation. The results show that **a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA**. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy. Please kindly check **Q5** in our response to Reviewer **SWr9** for experimental results.\n\n\n***`Q4: \"Additional computational cost during inference.\"`***\n\n**R4:** \nAs described in Appendix C, we explicitly acknowledge this limitation: TopLoRA introduces token-dependent LoRA weights that cannot be merged back into the pretrained model after fine-tuning. As a result, this design incurs additional computational overhead during inference and may lead to increased latency.\n\n**Similar limitations are observed in methods such as MoELoRA and HydraLoRA** ([19,25] in our paper), which also adopt dynamic adaptation mechanisms to enhance expressiveness at the cost of higher inference complexity. Likewise, TopLoRA is best suited for scenarios where merging LoRA weights is not required; for example, in **multi-task settings** involving multiple LoRA modules.\n\n***`Q5: \"Comparison with HiRA, KronA and AdaLoRA.\"`***\n\n**R5:** \nAs suggested, we added comparisons with HiRA, KronA, and AdaLoRA.\nHiRA and KronA enhance LoRA by using the Hadamard and Kronecker products, respectively, to increase the rank of LoRA weights. AdaLoRA, on the other hand, dynamically adjusts the LoRA rank across layers during fine-tuning. \n\nIn the comparison, we primarily set the LoRA rank to 16. It is important to note that KronA does not involve a rank hyperparameter, and its number of trainable parameters is fixed and cannot be adjusted. \nDue to their structural modifications, HiRA and KronA require larger learning rates to perform well. Therefore, we tuned the learning rate among {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and report their best accuracy. AdaLoRA is configured with an initial rank of 24 and a target rank of 16.\nFor TopLoRA, we used a rank of 10 to ensure comparable parameter counts with the baselines.\n\nAs shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The *Accuracy Gain* column reflects the performance improvement over standard LoRA.\nKronA, likely due to its extremely low parameter count, shows worse performance than LoRA. HiRA and AdaLoRA provide moderate improvements, but the gains are less pronounced compared to TopLoRA.\n\nIt is also worth noting that TopLoRA's improvements vary across datasets and models. For instance, Qwen2.5-14B already achieves strong performance on commonsense reasoning tasks, so the additional gain from TopLoRA is relatively modest (0.42%). However, this still exceeds the improvement from doubling the LoRA rank, which only yields a 0.18% increase (as shown in the table in Q3).\n\n\n|||#Params|Mathematical|Accuracy Gain|Commonsense|Accuracy Gain|\n|-|-|-|-|-|-|-|\n|Gemma-7B|LoRA($r=16$)|9.63M|72.17|-|86.43|-|\n||KronA|0.60M|69.46|**-2.71**|84.49|**-1.94**|\n||HiRA($r=16$)|9.63M|72.91|**0.74**|87.09|**0.66**|\n||AdaLoRA($r=16$)|9.63M|72.65|**0.48**|86.98|**0.55**|\n||TopLoRA($r=10$)|8.60M|73.16|**0.99**|87.92|**1.49**|\n|Llama-3-8B|LoRA($r=16$)|9.44M|70.19|-|86.15|-|\n||KronA|0.52M|67.41|**-2.78**|83.97|**-2.18**|\n||HiRA($r=16$)|9.44M|70.96|**0.77**|86.38|**0.23**|\n||AdaLoRA($r=16$)|9.44M|70.50|**0.31**|86.20|**0.05**|\n||TopLoRA($r=10$)|9.83M|72.37|**2.18**|86.68|**0.53**|\n|Qwen2.5-14B|LoRA($r=16$)|17.3M|79.43|-|91.51|-|\n||KronA|0.95M|78.73|**-0.70**|89.36|**-2.15**|\n||HiRA($r=16$)|17.3M|79.59|**0.16**|91.78|**0.27**|\n||AdaLoRA($r=16$)|17.3M|79.65|**0.22**|91.49|**-0.02**|\n||TopLoRA($r=10$)|18.2M|80.87|**1.44**|91.93|**0.42**|\n\n\n\n***`Q6: \"This paper lacks a related work section.\"`***\n\n**R6:** The related work is currently presented in Section 5 to maintain a coherent narrative, allowing readers to first gain a complete understanding of our method before engaging with comparisons to prior approaches.\nWe will carefully consider relocating the related work to Section 2 in the revised version to improve clarity and alignment with standard conventions.\n\n\n***`Q7: \"There is an error in line 40: a unitary matrix must be square.\"`***\n\n**R7:** \nThank you for pointing out the error. The correct statement is that both $Q_A$ and $Q_B$ are **semi-unitary** matrices; that is, $Q_A Q_A^\\top=I$ and $Q_B^\\top Q_B=I$."}]

Author rebuttal:

Meta-review: This paper received all positive reviews. After discussion, all reviewers agreed to accept this paper.  The authors are encouraged to put the following new results and discussions into the camera-ready version.
1. Add more baselines (e.g., LoRA+, KronA, HiRA, AdaLoRA, and MoELoRA) in the experiments.
2. Add a more formal and thorough discussion of related works.
Decision: Accept (poster)