RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts
Authors: Tuan Truong
Chau Nguyen
Huy Nguyen
Minh Le
Trung Le
Nhat Ho
Venue: ICML 2025 poster (2025)
Keywords: Parameter-efficient fine-tuning
Low-rank adaptation
Reparameterization
Mixture of experts
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Low-rank Adaptation (LoRA) has emerged as a powerful and efficient method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remained underexplored. In this paper, we present a theoretical analysis of LoRA by examining its connection to the Mixture of Experts models. Under this framework, we show that a simple technique, reparameterizing LoRA matrices, can notably accelerate the low-rank matrix estimation process. In particular, we prove that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential to a polynomial scale. Motivated by this insight, we propose **Rep**arameterized **Lo**w-**R**ank **A**daptation (RepLoRA), incorporating a lightweight MLP to reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. With limited data, RepLoRA surpasses LoRA by a substantial margin of up to **40.0%** and achieves LoRA's performance using only **30.0%** of the training data, highlighting the theoretical and empirical robustness of our PEFT method.

RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
Tuan Truong"
Chau Nguyen
12
Huy Nguyen *3 Minh Le
Trung Le
Nhat Ho
Abstract
Prettained Weights
Frowen
Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foun- dation models. Despite its popularity, the theoret- ical understanding of LoRA has remained limited. This paper presents theoretical analysis of LoRA by examining its connection to the Mixture of Ex- perts models. Under this framework, we show that simple reparameterizations of the LoRA ma- trices can notably accelerate the low-rank matrix estimation process. In particular, prove that reparameterization can reduce the data needed to achieve desired estimation error from an ex- ponential to polynomial scale. Motivated by this insight, we propose Reparameterized Low- Rank Adaptation (RepLoRA), which incorporates lightweight MLPs reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. Notably, with limited data, RepLoRA surpasses LoRA by margin of up to 40.0% % and achieves LoRA's performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness our PEFT method.
Trainable
ho
Bg
go,
ghn
Av
By
d-
Pretrained Weights
+
by
Wr E Rdddd
Figure I. Overview of our proposed method RepLoRA, which reparameterizes the low-rank matrices as the output of a lightweight ML.P, whose inputs are two diagonal matrices.
models for specific downstream tasks can be prohibitively expensive. To address this challenge, several parameter- efficient fine-tuning (PEFT) methods (Houlsby et al., 2019; Lester al., 2021: Jia et al., 2022) have emerged, facili- tating effective adaptation of large pre-trained models by adjusting minimal set of parameters while keeping most of the backbone frozen. Among these methods, Low-Rank Adaptation (LoRA) (Hu et al., 2022) stands out for its sim- plicity and effectiveness and has been successfully applied across diverse domains (Li et al., 2022: Qin et al., 2023; Taori al., 2023; Liu et al., 2024a). Despite its successes, theoretical understanding of LoRA has remained limited, hindering ability optimize its performance further.
1. Introduction
With the rapid growth in data availability and computational resources, large-scale models trained extensive datasets have demonstrated remarkable generalization capabilities, enabling successful applications across language, vision, and multi-modal tasks (Dosovitskiy, 2021; Radford et al., 2021; Touvron et al., 2023). However, fully fine-tuning such
Building on the recent finding (Le et al., 2024) about the con- nection between attention mechanism (Vaswani. 2017) and the mixture of experts (MoE) (Jacobs et al., 1991; Jordan & Jacobs, 1994) models, we present rigorous theoretical study demonstrating how LoRA can be interpreted within this new framework. Leveraging this perspective, we show that straightforward reparameterization technique (Li & Liang, 2021; Le et al., 2025), which represents low-rank matrices as the output of an MLP, can theoretically enhance the performance LoRA. Specifically, our analysis re- veals that this reparameterization can reduce the data needed to achieve a desired estimation error from an exponential
'Equal contribution Qualcomm AI Research, Qualcomm Viet- nam Company Limited "Wrrk was completed while an employee Qualcomm 'The University of Texas at Austin, USA 'Monash University, Australia. Correspondence Tuan Truong <tu- antruo@qtii wawill@@iiillllommcoom qualcomm.com"
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
sights, Section 6 presents our method, RepLoRA. To demon- strate the effectiveness of RepLoRRA, Section 7 presents the experimental results. Finally, Section 8 concludes the paper.
%
17.0%
16
Notation. For any n E N. let [n] = {1,2.... n.). For any set S, |S| denotes its cardinality. Given a vector u == (i, ..,, Ua) E IR4 and a ;= (0,,, ai,,............ aa) E N we define u² ui u2 ... ud |u] == U1 + u2 ...... ud and a! := ai aylaz!..... agl, while ||u]| stands for its Euclidean norm. For positive sequences (an)n21 and (b,)n21, we write On = (bm,) or on S b, if there exists constant C > 0 such that an I Cb, for all n E N. The notation ag = (b,), indicates ab, is stochastically bounded.
Accuracy
66
40,8%
36
30
LottA RepLoRA
10
Training Fraction f (%)
2. Related Works
Figure 2. Sample Efficiency on FGVC Datasets. RepLoRA not only outperforms LoRA consistently but also achieves LoRA per- formance on full dataset with only f = 30% training fraction.
Parameter-Efficien Fine-tuning (PEFT). With the re- cent rise of large models, PEFT methods are growing in popularity for their ability to fine-tune large-scale models by training a relatively small number of parameters for adapt- ing to specific downstream tasks. Existing PEFT methods can be divided into three categories. The first category is referred to as adapter-based methods, which introduce ad- ditional trainable parameters to the frozen backbone. For example, Series Adapter (Houlsby et al., 2019) proposes adding linear modules in sequence to the existing layer, or Parallel Adapter (He et al., 2022) proposes integrating these modules in parallel. The second category of PEFT meth- ods is Prompt- methods that add extra trainable soft tokens, referred to as prompts, to the input (Lester et al., 2021; Razdaibiedina al., 2023; Wang al., 2023). A weakness of these methods is that they increase inference latency compared to the original model.
scale to a polynomial scale, thereby substantially improv- ing sample efficiency. Based on these insights, we intro- duce Reparameterized Low-Rank Adaptation (RepLoRA) a novel PEFT method reparameterizes low-rank matrices through a lightweight MI.P.
We conducted extensive experiments across multiple do- mains, including image, video, language, and multi-modal tasks. Our results indicate that Repl.oRA consistently demonstrates better performance than vanilla LoRA. When only a tiny fraction of the training data is subsampled, Re- PLoRA improves up to 40% % over LoRA. This highlights the robustness and effectiveness of our method, both the- oretically and empirically. Moreover, the MLP used for be discarded after training, ensur- ing that our approach remains efficient as the standard counterparts at inference time.
reparameterization
Low-Rank Adaptation (Hu et al., 2022). LoRA and its variants are among the third category of the PEFT method, which is well-known for its simplicity and for not adding extra inference burden. To fine-tune the linear layers of large model, LoRA applies low-rank matrices to approxi- mate the weight changes and then merges them with the pre- pre-trained weights for inference. A recent variant of LoRA is DoRA (Liu al., 2024b), which proposes to decompose the weight change into learnable magnitude and direc- tional component. Another example is AdaLoRA (Zhang et al., 2023), which parameterizes the incremental updates in the form of singular value decomposition and prunes less significant singular values for more efficient updates. Or- thogonal Fine-tuning (OFT) (Qiu et al., 2023) exploits the orthogonal factorization to fine-tune diffusion models. Re- cently, VeRA (Kopiczko al., 2024) significantly reduces the number of trainable parameters compared to LoRA by using learnable scaling vectors to adjust shared pair of frozen random matrices across layers. Notably, our method, which will be proposed in the following few sections, also
Contributions. In summary, our contributions are: (i) We provide a rigorous theoretical analysis of LoRA from the perspective of a mixture of experts. (ii) Our results show that reparameterization can substantially improve sample efficiency, transitioning from exponential rate poly- nomial rate. (iii) Building on these theoretical insights, we introduce RepLoRA, novel PEFT approach that integrates reparameterization into LoRA. (iv) Extensive experiments across diverse domains demonstrate that RepLoRA consis- tently outperforms vanilla LoRA by a significant margin, thereby underscoring its effectiveness and robustness from theoretical and empirical perspectives.
Organization. The paper is organized as follows: Section 3 provides the background on LoRA and MoE. Section 4 establishes the connection between LoRA and MoE. Section 5 presents our theoretical analysis, including the statistical benefits of reparameterizing LoRA. Building on these in-
2

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
falls within this category, and we validate its efficacy along- side LoRA and its variants through theoretical analysis and comprehensive experimentation.
d2 = -d, - m Low-Rank Adaptation. LoRA (Hu et al., 2022) has emerged as an efficient method for adapting large pre- trained transformer models to downstream tasks. Building upon the hypothesis that the updates during fine-tuning ex- hibit a low "intrinsic rank", LoRA proposes to fine-tune the transformer architectures' linear layers by incrementally updating the pre-trained weights with the product of two low-rank matrices. For a given pre pre-trained weight matrix Wo E IRmx", LoRA represents its update as AW = BA, where B E Rm and A E "= with r < min (m, n). Consequently, the output of the fine-tuned model is:
Expanding upon the foundational principles of mixture models (Jacobs et al., 1991; Jordan & Jacobs, 1994), earlier research (Eigen et al., 2014; Shazeer et al., 2017) established the Mixture of Experts (MoE) layer as a crucial mechanism for effectively scaling model ca- pacity. Over time, MoE models have gained significant recognition due to their versatility across multiple domains, including large language models (Du et al., 2022; Zhou et al., 2023), computer vision (Riquelme et al., 2021; Puigcerver et al., 2024), multimodal learning (Han et al., 2024), and multi-task learning (Ma et al., 2018). Recent studies have focused on analyzing the convergence rates of expert es- timation in MoE models, exploring different assumptions and configurations related to gating mechanisms and expert functions. For instance, in the context of softmax gating, Nguyen et al. (2023; 2024a;b) revealed that the expert es- timation rates are shaped by the solvability of polynomial systems arising from interactions between gating and ex- pert parameters. More recently, Nguyen et al. (2024d;c; 2025) employed least squares estimation to establish an identifiability condition for expert functions, particularly in feedforward networks with nonlinear activation functions. These findings indicate that under these conditions, estima- tion rates improve significantly compared to models relying on polynomial experts.
Mixture of Experts.
ÿ = W' x = Wax + BAx.
(2)
During training. Wo remains fixed, while A and B are updated. Typically, LoRA adjusts the linear layers that generate transformer models' queries and values (or keys, queries, and values). In line with prior work (Hu et al., 2022; Liu et al., 2024b; Xin et al., 2024), here we tne-tune the query and value projection matrices, leading to the following output expression:
SLORA (X; A, B) = -hin ... ...m)) (3)
where for each i E [m]. h, Attention XW, + X : XW.K .WV + -xBvvAAv.)) Here, we denote A = [Aq, Avl. and B = [Bo, Bv], where Aq= = bb......,A...... Likewise, BQ = (B..)...,BB.)). and Bv (Bv......, .i.....,BB.m).. For each head i [m], the dimen- sions of these matrices are AQ.5 Rexde, Bo. E Rdxr, Av, E prxdd and By, E gaxr.
and Av = (A......,,AA..
3. Preliminaries
This section briefly reviews the background for multi-head self-attention in transformers, low-rank adaptation, and a mixture of expert models.
Mixture of Experts. A mixture of experts (MoE) (Jacobs et al., 1991; Jordan & Jacobs, 1994) model consists of N expert networks, fs IRd - Rdd. for E [N], and a gating function G: Rd + N that allocates contributions of each expert based on the input x to the model. The output of the MoE model is given by:
Multi-head Self-attention. We begin by revisiting the architecture of the multi-head self-attention (MSA) layer in Transformer ( (Vaswani, 2017; Dosovitskiy, 2021). Let X = - xN [ E IR xd denote an input sequence of embeddings, where N is the sequence length and d denotes the embedding dimension. The MSA layer processes this sequence as follows:
1 Σ G(m); f((x). 1=1
MSA (SS,XQ,a Xx, Xv) = (hi, ".....,,))" W' (1)
where G(xr) = softmax((1 (zz) (x)), and 8; Rd - R is score function. In the subsequent sections, we discuss how MoE relates to LoRA and provide a theoretical analysis of the proposed method.
where each attention head is defined by h; Allention(X X x XW,K, .WVV for E [m]. (XW? ...., X X) (XW* ...... xW*),, and Ky = (XW) ...,) are the query, key, and value matrices, respec- tively. Furthermore, m is the number of heads, and Wo E Rdd xd is the output projection ma- trix. Each attention head h; is parameterized by ? E Rdxds, W.K E RRdxds and W.Y E dddx, with
Here, XQ
Xx
4. LoRA from the perspective of MoE
Prior work (Le et al., 2024; 2025) has shown that each attention head in the MSA layer can be viewed as a spe- cialized architecture of multiple MoE models. Specifically, from Eq. (1), consider the output of the I-th head by
3

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
[y.1...... .u.. E NNd, Let X [.]....,, E Nd denote the concatenated input embeddings. We then define N experts ʃa RR Rd. encoded within the MSA layer as follows:
Problem setup. Let (X1, Y1), (X2,, Y2)................... ...,... (xn, Yn)
Rd x Rd be i.i.d. samples of size n generated from the following regression model:
Y = fo. (X.) + Es. i= 1,2, ................................ n.
(10)
L(X) - W. B - W.Y Zj
(4)
Above, we assume that 11 x Xn are i.i.d. samples from some probability distribution μ with bounded support. Meanwhile, E2, :,, are independent Gaussian noise variables such that [E|s = 0 and Var(es[X::) = 1²1a for all E [n]] Next, the regression function fa. (.) takes the form of an MoE model with L unknown experts, that is,
for j E [M], where the matrix Ej E Nd is such that E x = Z.j- Next, we introduce N x N score functions
$1,5 R Nd + IR associated with these experts:
X E W WK E a. W. WP Bj
(X)
vd.
dd
L exp (X (M² + BB A
(x)
=
D2
for i e [N] and j E [M]. Consequently, each output vec- tor h1. can be formulated as the result of MoE model, utilizing the experts and score functions defined above:
j=1
(M Big AV,)
(1))
where we denote
N
(Xp($
Σ j=1 EK= exp($s.(X)))
(X)..
(6) D,(X) = Σ exp(x (TM00 + A2x ct),
k=1
Within a pre-trained MSA layer, all parameters of these experts and score functions w W.K, and W/V remain fixed. When LoRA is applicd, it refines these experts and score functions with low-rank updates:
while G. == E exp(c).) A.. By..
rep- resents for mixing measure, that is, a combination of Dirac measures 8, associated with unknown parameters "G.BBo' B Bi Af eter space C IR x Hd g x ı x grxd In addition, we assume that the matrices MQ E Rdxd M x Rddd and M E dxd are given to align with the formulation in Eq. (9).
in the compact param-
fxxx) (WY + BvAAvA) E,X,
(x) X ETET(WE (W8 + oo. E,X Vl
for E [N] and j E [N]. From these definitions and Eq. (3), the modified output of the I-th head h
With versus Without Reparametrization. Sub- sequently, we establish the convergence rates of estimating the unknown low-rank matrices BB Ai. ( B 1f under two scenar- ios, namely without shared structures among these low-rank matrices (equivalently, without reparametrization) in Section 5.1 and with shared structures among these low-rank matrices (cquivalently, with reparametrization) in Section 5.2. Our ultimate goal is to demonstrate that sample efficiency of estimating these low-rank matrices under the shared structures setting is much better than that under the non-shared structures setting with a given error e > 0. The theory sheds light on our design of Reparameterized LoRA (RepL.RRA oRR) A) in Section 6.
[h.1.-.... (......,,,,.] E Nxd, can be expressed as:
N
exp(3r,
F2).
(9)
k= x
From this perspective, LoRA effectively fine-tunes the pre- trained MoE models contained within each MSA head by incorporating low-rank modifications to both the expert and the score functions. Next section will leverage this MoE viewpoint analyze the theoretical properties of LoRA.
5. Theoretical Analysis of LoRA: With and Without Reparameterization
5.1. Without Reparametrization: Suboptimal Sample Complexity
This section presents the theoretical benefits of applying the reparameterization technique in LoRA via its connection to MoE as formulated in Section 4. For simplicity, we will take into account only the first row of the first attention head h specified in Eq. (9). Under this simplified setting, we will investigate the convergence behavior of low-rank matrices within the following MoE- regression framework:
We begin our convergence analysis for low-rank matrices with the scenario where the LoRA reparametrization is ab- sent. It is worth noting that we can estimate those unknown matrices via estimating the ground truth mixing measure G., For that sake, we utilize the least square method (van de

---

RepLoRR: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Geer, 2000) to obtain the following estimator:
5.2. With Reparamectization: Optimal Sample Complexity
GR E arg min (e)
Σ
Y (x- (o-ff((x))
(12)
In this section, we consider the scenario where the low- rank matrices share their structures with each other. In particular, we consider the case where the low-rank matrices are reparameterized as:
i=1
where
we denote by (z) = (G =
Σy= Ba Byg Byr I', Bey' .Q.. , Av,r) E θ) the set of all mixing measures with at most I'' atoms. As the number of ground-truth experts L is typically unknown in practice, we assume that the number of fitted experts L' is large enough such that I! > L. Then, to determine the convergence rates of the estimator Gn. we use loss function built upon the concept of Voronoi cells (Manole & Ho, 2022).
S e S
AQ = Av = (A)
BQ = By = =22(B),
where (01 Rmxm Rxxd (P2 : Rmxm + Rxxd are some functions, A Rmxm B E Rmm xm' are learnable matrices with given dimensions m. m' 2 1. We specifically note that, for the simplicity of the theoretical development, this formulation is simplified compared to what was used in practice because we set the low-rank matrices of queries to be equal to that of values. As we will show in this section, even with this simplified formulation, reparameterization gives superior sample complexity compared to vanilla LoRA without reparameterization.
Voronoi loss. Given a mixing measure G with L' > L
atoms, its Voronoi cell set (V/ = Vy( j [0]] is generated by the atoms of G. as follows:
V = (i [Z] IH. - Hill S H.
+
where H == AQ: Bv, Av). Then, the Voronoi loss function used for this section is defined as
After training, the reparameterization can be discarded, and only the low-rank matrices AQ, Av. Bq, By need to be stored. We observe that the reparameterization strategy implicitly encodes a shared structure between the query and value low-rank matrices. The primary difference compared to the original LoRA is that instead of learning the low-rank matrices separately, we reparameterizee those matrices as the output of the two shared structures ęI, P2. To study the theoretical advantages of the reparametrization technique, we focus on the following two settings of the functions (P and y22
G.) =
Σ
exp(c) - exp(c
j==1
VEVY
exp(c) Bu Il" + ||
j'=liEVy
+ +1IABv I1 |1A I!',
for r E N, where we denote AHl := H. Hj. for any i,j". Now, we are ready to study the sample efficiency of LoRA without reparametrization in Theorem 5.1 whose proof is deferred to Appendix A.I.
(i) Simple linear reparametrization: (A) A and ( = W.B.
Theorem 5.1. The following bound holds for any r E N:
(ii) Non-linear reparametrization d (4) = 01 (W, A) and
92 (B = (),, where Ø1 and 02 are two non-linear activation functions applied element-wise to the matrices A and W₂B.
1
sup Di. (
G)L Vn
where Efe denotes the expectation taken with respect to the product measure fg-
It should be noted that the above reparametrization settings can totally be generalized to the scenarios where the matri- ces Ao and Av share only the learnable matrix A. In par- ticular, we can reformulate those matrices as AQ = WQ.1 A and Av = Wv.1 for the simple linear reparametrization and as Aq = a1 (WQ,14) and Av = σ1 (Wv11 A) for the non-linear reparametrization. In order to tailor to these settings, is necessary to include several terms involving parameters Wº.1 and Wv,1 rather than merely parameters W1 as in the above settings. However, realize that these terms not only provide no additional information on our convergence analysis but also make it unnecessarily compli- cated. Therefore, we assume without loss of generalization that AQ = Av = W1A or Aq = Av = o1 (W1A) to simplify the analysis, making more accessible.
Let us denote Gn E/m exp((")) (BB Ag By... Then, it follows from the result of Theorem 5.1 and the formulation of the loss D1,- that the convergence rates of the low-rank matrix estimators B2 A BV Ab. are slower than any polynomial rates Op(n 1/2r) for r e N. Thus, these rates could become as slow as Op(1/ log" (n)) for some constant 0 (due to the inequality log(n) < n). As consequence, we need an exponential number of data ((exp(=-//1) to obtain the approximations of the low- rank matrices with an eтror c. This observation reflects the suboptimality of the sample complexity of the LoRA without applying the reparametrization technique.
5

---

RepLoRA:
Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
5.2.1. SIMPLE LINEAR REPARAMETTEIZZTTION
5.2.2. NON-LINEAR REPARAMETERIZATTIN
We first take into account the simple linear reparametriza- tion where AQ Ay W A and Bº = By W.B. Under this setting, the ground truth regression function in Eq. (11), which we denote now as fc. (X) to avoid confu- sion, takes the following form:
Next, we draw our attention to the LoRA with non- linear reparametrization where the low-rank matrices parametrized as AQ Av an (W1A) and BQ By = d² W2B). Then, the ground-truth regression func- tion in (11() denoted as fā. (X) in this section, admits the following form:
L exp(X X + W2, W 4
(x)
exp T (MS + (W2,
+g))
(M + W B, Aj)X, (13)
(X)
(MY, + ( Bj Jo, (Wi, (15)
where we denote (x) = E exp(X T (Ma +
W2, W 1,k M k X + G)), while the mixing measure is of the form G, := プ exp(e).)ow, / By Bi. Ay Similar to Section 5.1, we estimate the unknown low-rank matrices via estimating the ground-truth mixing measure G. using the least square method:
where we denote D == Σx= exp(X (MQ +
02 (2 (o (Wi, 1,k A M + (t) and the mixing
measure G. = Σ exp( (xp(cg))( ( B B associated
unknown (c,,, W By, WT, 45 4,,
parameters in the parameter space O OCR x Rdxr x Rx The least-square estimator of the ground-truth mixing measure G, is now defined as
with
Σ (v. Σ(r: ( ,o(x)))
G₂ E arg
min
(14)
(8)
where G2 = IG= Ex W...A. 1 SSSS (c, W2 W. E θ} stands for the mixing measure set. To capture the convergence behavior of the estimator, we use the following Voronoi loss tailored to the simple linear reparameterization setting given by
Gi E arg min (EG
Σ X:
(xxx fa(xi)) 2
(16)
where (0) = (G ( (xpeec))) < f S L', (c,, B. W1, E θ]. For the sake of capturing the convergence rate of the estimator G., the Voronoi loss function is tailored to the non-linear reparametrization setting as
D2(G, = ==1
Σ
exp(ci) - exp(cr)
VV
E
Σ
exp(c)) Zi - Zj
Σ exp(c) exx(e))))( Dg y'=1 JEV
rE(MIVV 1 iH
Σ DELLEEYIDSLEEE
Σ
exp(c) 12 13
Σ exp(a)(IA(WBB))))) +IA(W ()g j'E[N] |y-1=
where we denote Z = W2BW, A. Given the above loss function, we study the sample efficiency of the LoRA with simple linear reparametrization in Theorem 5.2.
TEV,
Σ exp()\(IA(.BB),,11 (/ J'ELLI: 11,1> KEV,
Theorem 5.2. The estimator G., converges to the true mix- ing measure G. at the following rate:
Dz(G G.) = Op( (/))
where we denote A (W2B) ky := W2, - W B and A (( := W1,44 Wi, Ai Aj. for any i.j'. Before presenting the main result of this section in Theorem 5.3, it is necessary to impose some mild assumptions on the activations 0i and σ2. Due the space limit, we defer those assumptions to the proof of Theorem 5.3 in Appendix A.3.
Proof of Theorem 5.2 is in Appendix A.2. The above bound together with the construction of the loss D2 indi- cates that the convergence rates of estimating low-rank ma- trices By. B.. for j' [L], range from order Op(loo(n)//)) /n|² order Op (loo(())//]] Thus, it costs at most a polynomial number of data O(E (( to approximate those low-rank matrices with the error e. Compared to the exponential number of data required in the LoRA without reparametrization in Section 5.1, we observe that the LoRA with linear reparametrization is much more sample efficient.
Theorem 5.3. Assume that the activation functions 0n and d2 meet the assumptions specified in Appendix A.3. Then, the estimator Gn converges to the true mixing measure G. at the following rate:
Dz(Gnn, G.). = -O(( Vlog(n//n)
6

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Theorem 5.3 suggests that the convergence rates of esti- mating low-rank matrices Wi and WT,, A] are either or Op((lo((n)/n)) (p(()o(()//m))) cardinalities of their associated Voronoi cells, or equiv- alently, the number of their fitted parameters. In other words, we need a polynomial number of data, 3) or O(e "), to achieve the approximations of the low-rank ma- trices with the error when employing the LoRA with non- linear reparametrization. Compared with the LoRA with- out reparametrization, which requires to an exponential amount of data for the same task, the LoRA with non-linear reparametrization is more sample efficient.
ReptatkA (Lines)
ReelaoRAA (Nonelnew))
Imper Improworens (9)
depending on the
long Clamsitation
lungn Closdiettin (ROVC)
Video Artan Beoggnt
Commanmenn (MAMAA 717
iLLM Eab))
Lango Tiat
Viducted teal
Figure 3. Performance improvements over LoRA. RepLoRA out- performs LoRA across all domains, with non-linear reparameteri- zation substantially surpassing its linear counterpart.
6. Reparameterized Low-rank Adaptation
In the previous section, demonstrated that vanilla LoRA without reparameterization establishes suboptimal rate for low-rank matrix estimation while introducing shared struc- tural reparameterization to achieve the optimal rate. Build- ing on this theoretical insight, we introduce our method: Reparameterized Low- Rank Adaptation (RepLoRA). This method is tailored explicitly for fine-tuning transformer ar- chitectures by refining the linear layers that generate queries and values (or keys, queries, and values). This paper focuses on fine-tuning the queries and values for simplicity and clar- ity. Recall in vanilla LoRA, the matrices that generate the queries and values are given as:
7. Experiments
Experimental Settings. We conduct extensive experi- ments across multiple domains to demonstrate the versatil- ity and effectiveness of RepLoRA in a wide range of tasks. Our evaluation spans four distinct settings: language (com- monsense reasoning), image (classification), video (video action recognition), and multi-modal (image/video-text un- derstanding). To provide a comprehensive evaluation, we compare RepLoRA against several PEFT methods, such as Full Fine-tuning, Prefix Tuning (Li & Liang. 2021), LoRA (Hu et al., 2022), DoRA (Liu et al., 2024b), and Series Adapter (Houlsby et al., 2019). As summarized in Fig- ure 3, Repl.oRA consistently outperforms LoRA across all settings, highlighting its robust adaptability and superior performance in various tasks.
W/ Wi BoAQ WE Wv Bv (17)
where Bo, By E Rmxr and Aq, Av Rexa are learn- able low-rank matrices. Inspired by our theoretical results, RepLoRA innovatively reparameterizes A and B, modeling them as outputs of two MLPs. With non-linear reparameter- ization, the low-rank matrices are given by:
Eyaluation Metrics. In Parameter-Efficien Fine-Tuning (PEFT), evaluations typically focus on performance and the number of trainable parameters. The goal is to maximize performance while minimizing the parameters required. To assess this trade-off, in addition reporting performance, we adopt the Iiformance-Parranee metric, proposed by Xin et al. (2024). Specifically, for a PETL algorithm M, the PPT metric incorporates its task performance M,, the number of trainable parameters PM. and a normalization constant C. Formally, we have:
[Aq, Avl = ge (
(A)
[Bq: Bv] = g0 (B), (18)
Trade-off (PPT)
where A, B are learnable matrices, and A I0 are two- layer MIPs with shared part and distinct output heads. In this approach, Aq and Av are derived from a shared under- lying input A, with distinct outputs Aq and Av produced by the separated heads of ge Similarly, Bo and By fol- low the same structure, leveraging a shared B input. While we focus on fine-tuning the queries and values to streamline the analysis, this formulation can naturally be extended to fine-tune the keys. We implement A and as diagonal matrices ensure parameter efficiency. After training, the reparameterization go and g0g can be discarded, and only the fine-tuned matrices Aq. Av. Bq. and By need to be retained for inference. Hence, this approach does not incur any additional computational overhead for inference. An illustration of this method is provided in Figure
PM C +1))
PPT M, x exp((llogn
Commonsense Reasoning. In our first experiment, we compare the performance RepLoRA against LoRA and other PEFT methods using LLAMA-7B/13B (Touvron et al., 2023) on the Commonsense Reasoning task. We also include the accuracy of ChatGPT, measured with the GPT-3.5-ttubb API with a zero-shot Chain of Thought ap- proach (Wei et al., 2022). The commonsense reasoning benchmark comprises eight sub-tasks with predefined train-

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Table Top-1
Accuracy and PPT commonsense datasets.
The accuracies reported with I.LaMA- 77 and I.haMA- 13H.
Model
Method
#Params (%)
BoolQ
PIÇA
SIgA
He HellaSwag
WinoGrance
ARC-e
ARC-c
OBQA
AVG
PPT
ChalGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74,8
77.0
Prefxx
0.11
64.3
76.8
73.9
42.1
72.1
72.9
54.0
60.6
64.6
0.83
LoRA
0.83
67.2
79.4
76.6
78.3
78.4
77.1
61.5
74.2
74.1
1.70
LLaMA-7B
Adapter
0.99
63.0
79.2
76.3
67.9
75.7
74.5
57.1
72.4
70.8
1.74
DoRA
0.98
69.7
83.4
78.6
87.2
81.0
81.9
66.2
79.2
78.4
1.81
RepLoRA
1.01
71.8
84.1
79.3
85.2
83.3
82.4
66.2
81.2
79.1
1.96
Prefix
0.03
65.3
75.4
72.1
55.2
68.6
79.5
62.9
68.0
68.4
0.79
LoRA
0.67
71.7
82.4
79.6
90.4
83.6
83.1
68.5
82.1
80.2
2.15
LLaMA-13B
Adapter
0.80
71.8
83.0
79.2
88.1
82.4
82.5
67.3
81.8
79.5
1.80
DoRA
0.68
72.4
84.9
81.5
92.4
84.2
84.2
69.6
82.8
81.5
2.19
RepLoRA
0.99
73.1
85,2
84.7
91.1
85.9
84.7
73.4
85.6
82,9
2.60
ing and testing datasets. Following the settings outlined in (Hu al., 2023), we combine the training datasets from all eight sub-tasks into a single training dataset and evaluate performance on the individual testing datasets for each task. To ensure a fair comparison, we fine-tuned the models with RepLoRA A using the same configuration as LoRA, keeping the rank fixed. As presented in Table 1, RepL.RR LoRA achieves significantly better results than LoRA across all settings, de- livering substantial improvements not only in accuracy but also in the PPT score, emphasizing its parameter efficiency.
On average, RepLoRRA achieves a notable improvement of over 3% compared to LoRA, with notable gains exceeding 6% on datasets like dSprites ceriitees-ocatiinn RepLoRA outperforms all baselines on the FGVC datasets, with the sole exception of Prefix Tuning on the Stanford Cars dataset. The performance gap with LoRA par- ticularly significant, which was > 6% on average. On Stanford Cars, the improvement reaches a remarkable 10%. RepLoRA remains highly parameter -efficcieeet despite these substantial gains, reflected in its PPT scores.
looaationn Similarly,
Image Classification. We extend our evaluation of RepLoRA to the image domain and fine-tune the ViT- B/16 architecture (Dosovitskiy, 2021), pre-trained on the ImageNet: -2k dataset (Deng et al., 2009), on two chal- lenging benchmarks: the VTAB- 1K dataset suite (Zhai et al., 2019) and the FGVC dataset collection (Jia et al., 2022).
Video Action Recognition. Given RepLoRA's strong per- formance in the image domain, we expand our experiments to the video domain. We evaluate our method against base- line approaches using the Video Swin Transformer on two datasets: S8v2 (Goyal al., 2017a), which offers rich dataset with abundant data, and HMDB51 (Kuehne et al., 2011), which presents more challenging scenario with limited data and fewer categories. Despite the contrast- ing characteristics of these datasets, Table 4 indicates that RepLoRA remarkably outperforms all basclines while main- taining parameter efficiency, underscoring its adaptability and robustness across diverse data settings.
The VTAB-1K K benchmark is diverse suite of 19 datasets spanning various domains designed to test image classifica- tion and prediction capabilitics. These datasets cover a wide range of tasks involving distinct semantics and object cate- gories, organized into Natural. Specialized, and Structured domains. Each dataset includes 1,000 training examples, with an official 80/20 wrainvvalidation split, making it a rigor- ous test for generalization across different domains. On the other hand, the Fine-Gradd Gree-Gr Visual Classification (FGVC) suite, which consists of five datasets tailored for fine-grained recognition, focuses on tasks requiring subtle visual discrim- ination between closely related categories within specific do- mains. These datasets challenge models to identify nuanced differences, robustly evaluating RepLoRA's capabilities in fine-grained classification.
Image/Video-Tee Text understanding. Having demonstrated that RepLoRA outperforms the baselines on the language and vision tasks, we attempt to see if RepLoRA remains competitive on multi-modality tasks. This experiment compares RepLoRA with LoRA and full fine-tuning (FT) on the VL-BART (Lewis al., 2020). The experiments were conducted on four image-text tasks: VQA" (Goyal et al., 2017b), GQA (Hudson & Manning, 2019) for vision question-answeriin
NLVR² (Suhr et al., 2019) for visual reasoning. MSCOCO (Chen al., 2015) for image caption- ing, and four video-text tasks from the VALUE benchmark (Li et al., 2021): TVCA (Lei et al., 2018) and HowZOA (Li et al., 2020) for video question answering, TVC (Lei et al., 2020) and YC2C (Zhou et al., 2018) for video captioning. We follow Sung et al. (2022) and adopt the same setup of LoRA when applying RepLoRA. it evident that RepLoRA consistently surpasses both FT and LoRA in accuracy and PPT in both Tables 5 and Table 6. In particular, RepLoRA
Table 2. Classification performance on FGVC datasets.
Method
NABirds
71aowrx
Stanford
Dogs
Carx
AYG
JPT
87.3
82.7
98.8
89.4
84.5
88.5
LoRA
64.6
76.2
98.9
85.1
77.1
8488
0.2
Adapter
57.1
98.5
Prefix
87.5
82.0
98.0
742
90.1
863
0.5
D-RA
87.3
80.0
99.1
87.6
81.9
87.2
0.8
RepLoRA
89.1
86.1
99.3
91.2
87.6
90.7
0.90
The results, summarized in Table and Table 2, highlight
the superior performance of RepLoRA across most settings.
8

---

RepLoRR: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Table Performance VTAB- K with ViT-B/16 pre-trained on ImageNet- 221
Natural
Specialized
Structured
Retinopetny
Cleve Count
CIFARI
Caltechioz
Flower102
Camm yon
ROS Ros15000 EarosAr
Cleve-Dise
dBpr 100
SMORS A2-1
SNORB-FIL MORB--BI
Pets
SVEN
3eLab
XITTI
d5pp
Method
AVG
FFT
68.9
87.7
64.3
97.2
86,9
87.4
38.8
79.7
95.7
84.2
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
65.5
LoRA
67.1
91.4
69.4
98.2
90.4
85,3
54
84.9
95.3
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
31
44.0
72.2
0.72
Adapter
69.2
90.1
68
98.8
89.9
82.8
54,3
84
94.9
81.9
75.5
80.9
65.3
48.6
78.3
74.8
48.5
29.9
41.6
71.4
0.71
Pretxx
75.5
90.7
65.4
96.6
78.5
46.7
79.5
95.1
80.6
74.0
69.9
58.2
40.9
69.5
72.4
46.8
23,9
34.4
67.6
0.73
RepLoRA
99.3
97.5
72.6
55.7
35.7
47.3
75.9
Table 4. Performance on Video Action Recognition task.
Table 6. Performance on video-text tasks with VL-BART.
6EW2
ENDRS]
Method
#Params (%)
TVDA
How20A
YC2G
AVG
PPT
Method
Model
Pretraining
PParams IMU
Aec
FFT
Atctr
PPT
Video
Kinctie4ti0
50..9
68.07
FT
100
76.3
73.9
45.7
154.0
87,5
LoRA
Video Swin-
Kinetics4U1
38.34
0.37
62.12
Adaptor
Video Swin-R
Kincties4n0
1.56
39.00
0.36
67.52
0.63
LoRA
75.5
140.9
RepLoRA
Pielix
Video Swin-8
Video Swin
Kincties4U00
Kineties400
6,37
46.12
39.46
0.31
68.23
56.13
0.45
DoRA
76.3
145.4
RepLooRA
151.6
exceeds LoRA's performance by nearly 2% in image-text understanding tasks and roughly 4% video-text under- standing tasks, reaching the performance of FT.
conducted empirical experiments, with the results presented in Figure 3. These results demonstrate that non-linear repa- rameterization outperforms the linear setting by substantial margins, underscoring its effectiveness. For a more detailed comparison of linear versus non-linear reparameterization performance, please refer to Appendix C.2.
Table Performance image-text tasks with VL-BART.
Method
#Params (%)
VOx"
NVL3?
COCO
Cap
AVG
PPT
FT
100
66.9
56.7
73.7
112.0
77.3
LORA
5.93
65.2
53.6
71,9
115.3
76.5
0.99
DoRA
3.96
65.8
54.7
73.1
115.9
77.4
1.00
RepLoRA
6.02
66.5
55.4
74.2
116.2
78.1
1.02
8. Conclusion
We introduced a theoretical framework that bridges LoRA with MoF, offering new insights into the benefits of reparam- eterizing LoRA for achieving optimal sampling efficiency. Building on this theoretical foundation, we proposed Re- pLoRA, effective and efficient approach to PEFT. To evaluate RepLoRA, we conducted extensive experiments across four diverse domains: image, video, text, and multi- modal tasks. RepLoRA substantially outperformed LoRA and other PEFT methods in all settings, demonstrating its adaptability and effectiveness. These results highlight the potential of reparameterized structures in enhancing effi- ciency and effectiveness for fine-tuning large-scale models.
Enhancing Sampling Efficiency. Our theoretical analysis has demonstrated that reparameterization achieves superior rates of sample efficiency compared vanilla LoRA. To validate this claim, we evaluate the sample efficiency of RepLoRA FGVC datasets. Following the approach of d' Ascoli et al. (2021), we subsample each class at fractions f = (1%, 10%, 30%. 50%, 100%} and scale the number of training epochs by 1/ʃ, ensuring the total number of data seen by the model remains constant. Figure shows that Re- PLoRA consistently outperforms LoRA across all sampling fractions. The improvements significant smaller frac- tions, with RepLoRA achieving remarkable 40.4% gap at f=1%. 1%. More importantly, emphasize that RepLoRA matches LoRA's performance with only 30 % training frac- tion, therefore underscoring RepLoRA's superior sample efficiency, as predicted by our theoretical analysis. We refer to Appendix C.1 for a breakdown of these results.
Impact Statement
This paper presents work that aims to advance the field of Machine Learning. Our work has potential societal conse- quences, none of which we feel must be specifically high- lighted here.
Linear vs. Non-linear Reparameterization. Another conclusion from the theoretical analysis is that even sim- ple linear reparameterization with shared structure offers significant efficiency gains compared to vanilla LoRA. Fur- thermore, as shown in Theorems 5.2 and 5.3, incorporating non- non-linear reparameterization further improves the rate of low-rank matrix estimation. To validate this hypothesis, we
Acknowledgement
Trung Le was partly supported by ARC DP23 grant DP230101176 and by the Air Force Office of Scientific Research under award number -A9550-23-S-0001.

---

RepLoRR:
Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
References
Gelly, S. Parameter-cfficien transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitmick, C. L. Microsoft coco captions: Data collection and evaluation server, 2015.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models, In ICLR. OpenReview.net, 2022. URL http://dblp: uni-triiiest de/bb/ conf/iclr/icl1206
Deng, J., Dong, W. Socher, R., Li, L.-J., Li, K., and Fei-Fei, I.. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 R.LL0//CVVRR.2209520000
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P.. Bing, L., Xu, X., Poria, S., and Lee, R. I.I M-adaptters: An adapter fam- ily for parameter-efficien fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023, Association for Computa- tional Linguistics. doi: 10.186533 319. URL emn ammlp-maiin in. 319/.
Dosovitskiy, A.
An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of- experts. In International Conference Machine Learn- ing. pp. 5547-5569, PMLR, 2022.
Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., and Sagun, L. Convit: Improving vision trans- formers with soft convolutional inductive biases. In Inter- national conference on machine learning, pp. 2286-2296. PMLR, 2021.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3, 1991.
Eigen, D., Ranzato, M., and Sutskever, Learning factored representations in a deep mixture of experts. In ICLR Workshops, 2014.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C.,, Bax, I., and Memisevic, R. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017a.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Kopiczko, J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. In International Conference on Learning Representations (ICLR), 2024.
Goyal, Y., Khot, T., Summers-Stay D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question an- swering, 2017b. URL /nttts::/arxtiie org/ abs/
Kuchne, H., Jhuang, H., Garrote, E., Poggio, T. A., and Serre, T. Hmdb: A large video database for human mo- tion recognition. In 2011 International Conference Computer Vision, pp. 2556-2563, 2011. ICCV.2011.6126443
Han, X., Nguyen, H., Harris, C., Ho, N., and Saria, S. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In Advances in Neural Information Processing Systems, 2024.
doi: 10.1 109/
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
He, J., Zhou, C., Ma, X., Kerg-Kirkpatrick, T., and Neubig, G. Towards a unified view of parameter-efffcciett trans- fer learning. In International Conference Learning Representations (ICLR), 2022.
Le, M., Nguyen, A., Nguyen, H., Nguyen, T., Pham, T., Van Ngo, L., and Ho, N. Mixture of experts meets prompt- based continual learning. Advances in Neural Information Processing Systems, 38, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and
10

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Le, M., Nguyen, C., Nguyen, H., Tran, Q.. Le, T., and Ho, N. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth International Conference on Learning Representations, 2025.
Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653/w/2202eemmpp... 161. URL emn emnlp-main.16/.
Li, L., Lei, J., Gan, Z., Yu, L., Chen, Y.-C., Pillai, R. Cheng, Y., Zhou, L.., Wang, X., Wang, W. Y., Wang, W. Y., Berg, T. L.., Bansal, M., Liu, J., Wang, L., and Liu, Z. Value: A multi-task benchmark for video-and- language understanding evaluation. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 2021.
Lei, J., Yu, L., Bansal, M., and Berg, T. TVQA: Localized, compositional video question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Pro- ceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1369-1379, Brus- sels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 18.18653/v ww///////////----./ /D18-1167.
Lei, J., Yu, L., Berg, T. L., and Bansal, M. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Pro- ceedings. Part XXI, pp. 447-463, Berlin, Heidelberg, 2020. Peringer-Verlag, ISBN 978-3-030-58588-4, doi: 10. 10.10077978833-030-585588 27. URL https://doi. org/10.1007/
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Zong. C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Com- putational Linguistics. doi: 10. 10.186653/11/22221.clllll 353. URL https: http:///aaaaaathcccccc www.eoooo..oooo.go... 2021. acl- 1aa-longg3550
Lester, B., AI-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Moens, M.-F,, Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguistics. doi: 10. 10.18653//120021.mmmlpp 243. URL https: //raacaaanthhoooo htt////annnnoolorrr////2/22 emn cmmlp-maing 1.rmain..200 243/.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-. F., Cheng, K.-T., and Chen, M.-H. Dora: weight- decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024b. 20244..
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, pre-training for natural language generation, translation, and com- prehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Compu- tational Linguistics. doi: 10.18665/v1/22220.c--m 703. URL
L. BART: Denoising
Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H. Modeling task relationships in multi-task learning with multi-gate mixture- of-uurr-s--epprtss.. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 1930-1939, 2018
Manole, T. and Ho, N. Refined convergence rates for max- imum likelihood estimation under finite mixture mod- cls. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14979-15006. PMLR, 17-23 Jul 2022.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping Lunguage-image pre-training for unified vision- langugl understanding and generation, 2022.
Nguyen, H., Nguyen, T., and Ho, N. Demystifying softmax gating function in Gaussian mixture of experts. In Ad- vances in Neural Information Processing Systems, 2023.
Li, L., Chen, Y.C.,, Cheng, Y., Gan, Z., Yu, L., and Liu,
J. HERO: Hierarchical encoder for Video+Language renissepresentttion pre-training.
In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pp. 2046-2065,
Nguyen, H., Akbarian, P., and Ho, N. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML. 2024a.

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Nguyen, H., Akbarian, P., Nguyen, T., and Ho, N. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICML, 2024b.
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. In Ranzato, M.. Beygelzimer, A.. Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8583-8595. Curran Associates, Inc., 2021.
Nguyen, H., Ho, N., and Rinaldo, A. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In Advances in Neural Information Processing Systems, 2024c.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.
Nguyen, H., Ho, N., and Rinaldo, A. On least square estima- tion in softmax gating mixture of experts. In Proceedings of the ICML., 2024d.
Nguyen, H., Akbarian, P., Pham, T., Nguyen, T., Zhang, S., and Ho, N. Statistical advantages of perturbing cosine router in mixture of experts. In International Conference Learning Representations, 2025.
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H.,, and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Anmual Meeting of the Association for Computational Linguis- tics, pp. 6418-6422 Florence, Italy, July 2019. Associ- ation for Computational Linguisties. doi: 10. 18653/v 1/ 1694. URL https: htwp://ttaaaaaaaaaooaaoo P19 1644/.
Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. In International Conference on Learning Representations (ICLR), 2024.
http///aaaanttttogoooe org/
Qin, C., Zhang, A., Zhang, Z. Chen, J., Yasunaga, M., and Yang. D. Is ChatGPT a general-purpose natu- ral language processing task solver? In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1339-1384, Singapore, December 2023. Association for Computational Linguistics. doi: URL https://
Sung, Y.-L., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 5227- 5237, June 2022.
10. 18665//11/0233.emmmpmmiiin emn.p-mainnn..mmmmmmmmmmmmmmmmmm mo.main... B5/.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, B. Stanford alpaca: An ceeructtoo-follwiing llama model, 2023.
ac sclanthology.orge
Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B. Controlling text-to- image diffusion by orthogonal finetuning, In Proceedings of the 37th International Conference on Neural Informa- tion Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023.
van de Geer, S. Empirical processes in M-estimation. Cam- bridge University Press, 2000.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., ct al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
Vaswani, A. Attention all you need. Advances in Neural Information Processing Systems, 2017.
PMLR, 2021.
Wang. Y., Wu, J., Dabral, T., Zhang, J., Brown, G., Lu, C.-T., Liu, F., Liang, Y., Pang, B., Bendersky, M., and Soricut, R. Non-intrusive adaptation: Input-centric parameter-efficien fine-tuning for versatile multimodal modeling, 2023. URL https: www.ii.oorr....... /abs/ 2310 12100.
Razdaibiedina, A., Mao, Y., Khabsa, M., Lewis, M., Hou, R. Ba, J., and Almahairi, A. Residual prompt tuning: improving prompt tuning with residual repa- rameterization. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 6740-6757, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023 acl.. 421. URL https: ////ccaannt /alaaaatlaoooyyyoooo httpt/oolgg..oggg//o2eo org/2023. fi Eindings-ace 421/.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F. Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS
12

---

RepL.RR:: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
'22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Xin, Y., Luo, S., Liu, X., Du Y., Zhou, H.., Cheng, X., Lee, C. L., Du, J., Wang, H., Chen, M., al. V-petl bench: A unified visual parameter-efficiet transfer learning bench- mark. In The Thirty-eight Conference Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024.
Yu, B. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam, pp. 423-435, 1997.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. The visual task adaptation bench- mark. ArXiv, abs/1910.04867, 2019. URL https: //arxiv. org/ www.aa////0 1910. 04867.
Zhang, Q. Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficiet fine-tuning. In International Conference on Learning Representations (ICLR), 2023.
Zhou, L., Xu, C., and Corso, Towards automatic learn- ing of procedures from web instructional videos. Pro- ceedings of the AAAI Conference on Artificial Intel- ligence, 32(1), Apr. 2018. doi: 10.1609/aaii v3221. 12342. URL battps://ojas aaai. orq index. php/AAAT AAATAA////AA htt///////tttteeeieeeeeeeeeeeeee 12342.
Zhou, Y., Du N.. Huang. Y., Peng, D., Lan, C., Huang, D., Shakeri, S., So, D., Dai, A. M., Lu, Y., et al. Brainformers: Trading simplicity for efficiency. International Con- ference on Machine Learning, pp. 42531-42541 42542. PMLR, 2023.
13

---

Supplement to "RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts"
In this supplementary material, we provide proofs of the main results in Appendix A. Details of experiments are provided in Appendix B, while additional experiments are presented in Appendix C.
A. Proofs of Theoretical Results
This appendix provides proofs for key results in the main text.
A.1. Proof of Theorem 5.1
The proof is divided into two steps follows:
Step 1. To begin with, we demonstrate that the following limit holds for any r 2 1:
lim inf - fa. (L)
D1, 0.
(19)
Note that it is sufficient construct mixing measure sequence (G,)n21 that satisfies both (Gn, G.) - 0 and
llfa. fa. (DllLe/())/i,rCCCC G.) + 0,
For that purpose, we take into account the sequence G2 -(=
I. exp(c, 0 . AB, where
exp(c?) exp(c²) exp(ct) + 2er+1 and exp(c() = exp(pc for any 3 s L L+1;
B B² B2g and Bº, BB.s- for any sii L +1
AQ.1 1Q. A and A2,- AQ for any < is 1+
B By
L n 11,1, BV Bi, e and B B for any 3 sii
AV A = Av, and AV, = A for any 3 < < 1;
where we denote 61 as the matrix of size x d such that its (1, 1)-th element is one while all other elements are zero.
Then, we can compute the loss function D1, (Gn; G.) as
1 1 n n"
(G, G.) n1+I + exp(cct
= O(n
(20)
It can be seen that (G,, G.) + 0 as 0c.
Subsequently, we illustrate that llfa. fc. fft.clllla///...lelco G.) + 0. In particular, let consider the quantity
exp (MQ + BQ ci3] -2) (X)),
Qx =
14

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
which can be decomposed follows:
L.
(x) =
exp(c") exp(X (Må + BQ, A2,))) + AV,)x
j=liev 1eV
- eexp(x (M8 + AQ, + A
exp(c") (xp(c")(exp(e) exp(x (MQ. + BQ, - X (Må + AQ))) (oxx)sonn0 (x) (x)
j=liE,,
Σ
exp(c") exp(e)) exp((X((MX)) + Aox))) (M&
(M + - (x.(x)
/=1
= An (X) (X) (X).
It follows from the choices of B
A AQu BV AV and of that
2
1
(x)
2 exp(ci)+ + n-+1 exp(x (M²- + Bº, (B 48, By
1=1
1
exp(br,1) + nttl] exp(X (M8 + B A xx)
A 1 4 + (B)244;2 - Avr))
2
0,
where the last equality occurs as B AV. By, Av, e and B A - BB Ai; = -1,,,1 Av.
Moreover, we can also verify that (X) 0, and C, (X) On (r+1)). Thus, we deduce that Qx) (rr G.) - 0 as n + oo for almost every X.
Ekn (X X (M) + BQ + <t) is bounded, we have (X) fc, ( G=) 0
As the term for almost every X. This limit suggests that
lls.. fc.
-
as 7 + c0. Thus, we obtain the claim in equation (19).
Step 2. We will establish the desired result in this step, that is,
inf G. (e) GES (O)G) [I,, G)] Z n 112
sup
(21)
Since the noise variables follow from the Gaussian distribution, we get that Y; X; NNc.. (Zi), for all E [m]. Additionally, for sufficiently small > 0 and a fixed constant C₁ > 0 which we will select later, we can find mixing measure G., E (e) such that (G,, G.) = 2: and Ilfar - fa. (1))))) I E thanks to the result in equation (19).
According to the Le Cam's 's lemma (Yu, 1997), as the Voronoi loss function D1,- satisfies the weak triangle inequality, follows that
inf G (e) G9)
sup
E (In, G)]
Di,r (C', G.)
exp( n - CCC (X)
8
Z 8 - exp(-ni]]] - fa. Sc. 1(2)(w)
Z e exp(-Cinee
(22)
where the second inequality follows from the equality
(fo: (X) fc. (x))
20²
15

---

RepLoRR: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Let 11/2, then get that exp(-Cine") -n11/2ex(--C) Consequently, we achieve the desired minimax lower bound in equation (21). Related works on parameter-effcient fine-tuning techniques, low-rank adaptation, and mixture of experts are in Appendix
A.2. Proof for Theorem 5.2
We first start with the following result regarding the convergence rate of the regression function estimation fa., to the true regression function fc.:
Proposition A.1. Given the least square estimator Gn in equation (14), the convergence rate of the regression function
estimation fc. (:) to the true regression function fa. under the (p) norm is parametric on the sample size, that is,
llf fa. fa.ll12(0) Op( Op(vlog(n)/n).
(23)
Proof of Proposition A.1 is given in Appendix A.4. Given rate fa. in Proposition A.I, our goal is to demonstrate the following inequality:
inf |lfc ( 0.
(e)
We divide the proof of the above inequality into local and global parts.
A.2.1. LOCAL PART
For the local part, we prove that
lim -0+ (e..
inf
llf - fa. (ec-fe.llew///llG.... 0.
Assume that above claim does not hold.
indicates that we can find a sequence of mixing measures G, =
E exp(Cn.3' )d B Any in 2z (0) such that
{
D2n = D2(Gnn G.) 0, 11, Sc. Jo.ll D2n 0.
as n + cc. We denote V := V,,,,, Voronoi cell of Gn generated by the j-th components of G,, Without loss of generality, we may assume that those Voronoi cells do not depend on the sample size, i.e., V, = Vj. Therefore, the Voronoi loss D2n can be rewritten as follows:
D2n =
Σ VV
exp(cn.)) - expp(c))
Σ
Σ
exp(c., AW 2, Bräs W Wn, An.g' ||
JELL)IV E
Σ
Σ
exp(c... AW n,y B Baay W Wiy An.ig
yELLH DEEE,
where W W, A ABnsy Wi An,yy' W Wn.5 Bay Wn.1y Ang's
W WWW, for Vy.
To simplify the ensuing presentation, throughout the proof we denote Z W,BW, A for all the matrices Wi, W2, 4. and B. Given the notation, the Voronoi loss Dan becomes
Dan Σ F=1
Σ
exp(cn.)) - -xxpec)))xp((et)
"'ELL SEV, exp(co.:))AAnn.
Σ
Σ
KEV,
+
Σ
Σ
(lp(e,.))lAZn.... 11.
I' JELLLA
Since Dan + 0. have Eie,, exp(cn,)) exp(cr.), Zn,i Zj for any i V, and j E [4]. Throughout this proof, we assume without loss of generality that M&,, = Ia with a note that our techniques can be extended to the general setting of that matrix. Now, the proof of the local part is divided into three steps as follows:
16

---

RepLoRR: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Step 1 Taylor expansion.
First, we define
(x)
exp X (M² Z + +x] Ifa, (X)).
Then, we can decompose the function (x) as follows:
I.
(x)
exp(cn.)) exp(( (MO + (M + Zn,)X exp(x (Mă ()) +2j)x
J= iEV,
exp(co.t) :) [exp(xX + - -xx))) (M¹ + 1o)x)]1cn.88 (X) (X)
j= 1
exp(Cn,i) exp(cx,))=exp(()) exp((c)) exp(x (M²- 2j)x) (MC + 2ÿ) - Sc., (x)
1=1
EEV,
: An (X) (x) + (x).
(24)
Decomposition of the function (X). We denote (X: Z) == exp(X' (M² + Z) and V(X; 2) (M) + Z)X, and FX:; Z) (x; (X; Z). Based on the number of elements in each Voronoi cells, we decompose the function An (X) as follows:
Ax(x)= = Σ Σ
Zn.t) = 2
ivV,,
Σ
exp(cn.) F(X; - F(X: 2
1
== An, (X) + (X)
An application of the first-order Taylor expansion leads to
0
U(X; =0x;
Σ
AA m
(x; Zj) + Rx,,1(X), (Z
aa =1
glai
V(X; Zm.) v(X; + Σ
3lZ)a (%; Zj) + (x),((X),
|a|
for any and j such that i E Vj and |Vyl = 1. Here, the functions (x),(()) and (x) denote the Taylor remainders. Collecting the above results leads to
exp(Cn.))
Σ (AZ,, aa Bla U
(x)
a(Z)a (X: Vx; 21)
a!!
J:V3
IEV,
|a/=1
glal V
(X; Zj (x) 2j) } + R
+ (A Z )5)
a
0(Z)a (X: 2j) VX; Zj) + Mus.a a(Z (x: 2,)((x |3)} + (X)
alalū
ΣΣ
Masia
where the function Rm,, (X) satisfies that Rn, 1 D2n + O. It is due to the uniform Lipschitz property of the function F. In the above display, the formulations of the coefficients Mn are given by:
exp((n.i) NEV; a! (A )n )a
Mn.ja
17

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
for any |o|| 1.
Moving to the function (X), an application of the Taylor expansion up to the second order leads to
all Mn.j.a
alalV
Ax,((x)=
E
Σ
0 (x:
2j) Mn.ja 0(z)a (x: (x:
0(Z)a (x; 4j) V
alalU
Σ
( (x; 2j) + Rx)
11
where the remainder (X) satisfies that 2 D2n -> In this equation, the coefficients Mn..a and Magaa take the following forms:
Σ exp(Cn.)) a!
Mn.ja
(AZn,i))
LEV,
for any |a| and
exp(cn.s)
M
( (ZZn,s)
a!B!
iEV;
for any |a|- = |,8| 1. Simple algebra leads to the following formulations of the partial derivatives of Ú(;; Z) and V(X; Z):
a0
(X; Z) = (( exp(X (M8 + Z)X),
0 Z)) (m)m)
U
(X; Z) = x(1)) ( ((( exp(x (MQ + 2)X),
(Z) (M1v1) 1 Z (a2e2)
av
(X; Z) ((11) eay:
O Z ((131)
X: Z) O2-
0 (1 1 Z (1gbg)
Plugging these formulations into the functions An, (X) and (X), obtain that
d
d
(X) = Σ
exp(X (M& + 2,)x]
Σ
Ma ((1))("))(M + 2,)
Σ
Mnjjeu x(v1) Eus
J:1,
#1.-1=1
uI.1=1
+ Rn,1(%)
2x2 Σ
exp(X (M& Z2)x)
Σ
M Majeeu x(u))x(1) 2j) + Σ
Mujicu g(er)
>
11..1
d
d
Σ
Σ
M Mnjjev tengue ((1))(())((22)(( exp(X (Mg + 2) + 2;)
01, 11 W2,0a=1
d
d
M Mn Saggn g(ui)glenig(es) exp(x (M8 + Zj)X)ens + (%).
122P2
In these equations, e, is denoted as the vector in Id such that its w-th element while its other elements are for any 1 S u S Furthermore, eag is denoted as matrix in ddxd with its wi-th entry while other entries are zero.
18

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Decomposition of the function B,,(X),
Moving to the function (x), we can decompose this function as follows:
(x) = Σ
exp(cn.))
[0(X; Zn.i) UX: ( Sa.,
(x) (X)
J: HHVy/=1se 1EV;
+ Σ
Σ exp(ca.t) Zn.i) 0(X; 2 fc.
ハ 1iE,;
= Br,1 (x) + (X).
An application of the Taylor expansions up to the first order for (x) and the second order for (x) leads to
al
(x) = E
Σ
M Ma
(x: fo. (X) + (%),
/:1//==1 |1
U
Bx = Σ
Σ
Majia (Z)a
(x; 42) Mca (X) + (x)
where the Taylor remainders Rx,a(%) (X) satisfy that (x)(XX) + 0 and (X) + 0. Direct calculation leads to
d
(x) = Σ
exp(X (M8 + 2,)x8
Σ
M MnJienjo (()) fa.. (X) + (x),(X),
3:12,1=1
01,01=1
d
Bx,(())
Σ
exp(X (M8 + Zj)x)
Σ
Mu ((1))((1))
3//31>1
01."1=
d
d
Σ
Σ
M Mujjenj
(()))()))(()))( (x)
(8) + (%),
M1.
U2.
Putting all the above results together, we can represent the function Qx(%) as follows:
d
Qx(X)= = Σ
exp(XT(MM)) Z))
Σ
Mnjeen,o x(m))(())(M + 22)x) +
Σ
M Majicp
x x(vr)
JiV,=1
=1
uy =1
Σ
exp(X X (M8 + Zj)
M Mn.ieu, (iliiieee((M 2;)x+ +
Majeeu x(e
11,ii=1
7
d
Σ
M Mn.j.cujy teugez
((1))(()))(())) exp (M + (M + Zj)x
01.01=1 12
d
d
Σ
Mn by Sugny
(()))(() cxp (M) + Jem
=1
19

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
(O0(X((M)) exp(X (MQ 2;) #1EL=1
Σ
Σ
Mn.i.uie X(m))x(1)) (X) (X)
01,01=1
Σ
exp(X ( + 2j)x)
Mudieuio x(m1)x((1))
AM
M1,
Σ
Σ
Mn Majeern
f0,, (X)
01:01
12,12=1
1.
I.
Σ exp(X (MM Zj)X)fG. (X) + Σ
Nn. exp(x (M8 + ()) + 22)
3=1
j=1
Rm,1 (X) + (x) - P (X) - P
(25)
where Nu. = Eiev, exp(cn.i) exp(c)) for any j E [L].
Step 2 Non-vanishing coefficients. As indicated in equation (25), the ratio (x)/ can be expressed as linear combination of the following independent functions:
Ú(x; 2)
+ 2j)X, U(X Zj ) (11) EuS,
Z)):
((1)) (M) + Zj)X, U(X; fa. (X), 0( Zj ) (1))(())(())))(((() fc.. (X).
engs
Ú(X; fa,. (X), UX Zj ( (M) + 2j)X,
for any indices 1 $ j S L and 1 S u1, 01 u2 U/ d.
We demonstrate that at least one of the coefficients of these independent functions does not go to 0 as -> oc. Assume by contrary that all these coefficients of these linear independent functions to O. From equation (25), we obtain that Mn.j.a/ aanij.a//2nn DDn Mnj.a,,/ D2n. and //. //D..
DD2. go to for all the coefficients a. B E Nddxd satisfying that 1 I |a + [3] < 2.
Since Na./Da + O, we find that for any j (T.)
Ever, exp(cn.)) -exp(c)))))((()))
|Nv..l + 0.
D2n
D2n
Taking the summation of these limits leads
E exp(en.)) exx((c) 0.
(26)
Dza
Evev, exp(e...)) Ain.isl - 0. That
Now, for any indices j [L] such that [Vj== 1, the limits Ma D2n - lead to
Dan
result directly implies that
En Eiev, explen.))lAA.....
+ 0.
(27)
Den
Moving indices j E [L] such that their corresponding Voronoi cells satisfy that [V,] > 1. The limits j. DDn 0 lead to
Eiev, exp(c..))lAZ... n.ij 122 - 0.
(28)
Dzn
By putting the results in equations (26), (27), and (28) together, we arrive at 1 = Dan + 0 as n - 00, which is a contradiction. As a consequence, at least one of the coefficients of the linear independent functions in (X)/DD2 does not go to n + co.
Dzn
20

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Step 3 Application of the Fatou's lemma. We denote m, as the maximum of the absolute values of the coefficients of the linear independent functions in (.x))/Dzn, As at least one of these coefficients does not go to 0, it indicates that
1/m, A oo + Since ll fc. (()) Pnn + 0 as n co, we obtain [| fa. (w) (m,,) D2n) + 0. An
application of the Fatou's lemma leads to:
[a.. Sc. 1C
|J,. Sc. (x)
0 = lim
2
lim inf 12-00
dj(xx) 2 0.
+0+00
myn D2n
mn, 22n
f5a, (X) - Sa. (x)
That inequality demonstrates that lim inf, 1-000
O for almost surely X. As n - 00, we denote
my Dn
Mn..ca MnDD2
Mns.a. + Eha
Nn. m, D2n
+ 23.00
- TJy
Mn
for any indices j E [L] and any coefficients 8 E Ndxd such that 1 S |a| + |8| S 2. Here, we have that at least one coefficient from {Z $5,0,3 F3 j E [L] a, 3 E Naxd 115 |a| + [3| S 2] is different from (indeed, it should be equal
fc., (X) fc. (x)
to 1). Given the above notations, the limit lim inf
0 implies that
mn D2n
a Σ
exp(X (M& 22)
Σ
Xy X (M8 + 2j)x)
+
Mn x
/:13/==1
01,91
#1,91=1
d
d
+
Σ
exp(X T (M8 + 23)x)
A Siennrs
(())) (M + 22}X+ +
Mn.j.eury x(vrle eus
;//;1>>
14 17
d
d
Σ
X oy teugos x (10))x(01))(11))) exp(X (MQ) + x + Z2)x
01,01=1 ug.
d
d
Σ
exp(( (M1 + Zí eews
Shitgy Rugez
01.01
12
Σ 3//y
exp(X (Må + Z2)x)
X Ea x(1)x((1)
Sca (X)
11,b==
Σ
exp((T + Z1)x3
X Xyseuyy ((1))
Ms =1,01=1
d
Σ
Σ
E X ((()))())))())))))))) (X)
01.01 12,12=1 L
L Σ 7 exp (M&- + fc) (X) + Σ 7, exp(x (Må + ()) + Zj)x =0 ==1
(29)
for almost surely X. However, that equation implies that all the coefficients {Ay Es, Jy ijE E [L], B e Naxd |a| + |8] < 2) are O. is a contradiction. As a consequence, we obtain that
15
lim s 8-0 (0).
inf
f/ Sc. /G
A.2.2. GLOBAL PART
The result of the local part implies that we can find positive constant e' such that
inf |la - Sa. ( 0.
21

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Therefore obtain the conclusion of the theorem, we only need to prove that
inf (G:...GG))
Ilfa (aa.lllaw//P/(G,GG
We assume by contradiction that the above claim does not hold. It indicates that there exists a sequence of measures
Gn E/=1exp(es exp(cn.) BB,, Ang in (9) such that
{
[lfcc fa. ( G.) +
as oo, which implies that Ilfo; - fc. (|eP()) +
Given that ④ is a compact set, there exists mixing measure G' in Gr such that one of the G 's subsequences converges to G". Since D2 D2(G", n: G.) > E', we obtain that D2(G', G.) > E'. An application of the Fatou's 's lemma leads to
0 = lim llfe, - fG.ll1a(w 2
lim inf fc; (X) - fa. (X) (x) 120000
n0000
The above inequality indicates that for fc. for almost surely X. From the identifiability property, deduce that
G' = G.. It follows that D2 G.) 0, contradicting the fact that D2(G', G.) e' > Hence, the proof is completed.
Proof for the identifiability property. We will prove that if Sa (X) = fa. (x) for almost surely X, then G = G.. To ease the presentation, for any mixing measure G E/= exp(c))0B,A A, E Gr (E), denote
exp(u)
softmaxa(u)
Σ/ exp(X (Mg + 2,)X +g)
:jjj ; [2]}. The equation fa(X) fa. (X) indicates that where E {x (Ma + 2,)X c
i
softmax(X (M& + Zj)X c + Zj)X Σ
softmax (M& + +,) +,)x.
(30)
That equation implies that L = L. As consequence, we find that
(softmax(XX X (M) + +z) :j E [L]} = (softmax(X (M² + Z,)X+ + j [4]}
for almost surely X. By relabelling the indices, we can assume without loss of generality that for any j [4]
softmax (M& + 2j) <)) sofumax(X (M² + x
for almost surely X. Given the invariance to translation of the softmax function, the equation (30) leads to
exp (cf) exp(( (M8 + (M) + Zz)x
Σ exp (c;) exp ex (Mă) + (M) + 2,)X,
(31)
for almost surely X.
Now, the index set [L] can be partitioned into m subsets K1, K2...., Km where m L, such that exp (cj) exp (cz.) for any indices j.j E K and E [m]] Thus, equation (31) can be rewritten as follows:
m
exp (cf) exp(( (M& + (M + 2
i=1 JER,
exp (xp (M& + 2,) + +2,
1=1 jEK,
22

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
for almost surely X. The above equation implies that
{ + Zj) K.) = [(M + Zz) K.},
for any i [m]] and for almost surely Hence, we obtain that
Σ
exp(e)))
Σ
Σ i=1 jEK,
exp (cz)8zz
jEk;
As consequence, G G. and the proof is completed.
A.3. Proof of Theorem 5.3
Firstly, we can reduce to the case where W1, W2 are identity matrices. In particular, we may denote of ( 01 ( (WiX) for input X, and consider of in the place of o We first start with the following result regarding the convergence rate of the regression function estimation fü. the true regression function fä.
Proposition A.2. Given the least square estimator G. in equation (16), the convergence rate of the regression function estimation fe. (-) to the true regression function fe_ under the L [p) norm is parametrie on the sample size, that is,
llfa. - fa. fa. (1 = Opl Op))log())/n).
(32)
Given rate of fa. in Proposition A.2, our goal is to demonstrate the following inequality:
inf Ma - Sc. De,llua G.) > 0. (e)
We divide the proof of the above inequality into local and global parts. Before going into the proof details, let us introduce some essential assumptions on the activation function a.
Assumptions. We impose the following assumptions on the activation functions 01 and d2:
(A.1) (Algebraic independence) If there exist parameters A) and (B', A') such that (2(B)o, (A) oz(B')o, then we obtain that (B, A) (B',A')
(A.2) (Uniform Lipschitz. Let F(X: ; B, A) == exp(x (M² (2(()) +oz T E 2), we have
Then, for any
all F
glal F
Σ la|=r
B,
s SCI( A) (B',A')] (X: B'.
DA a
dA
for any vector y E 2dr and for some positive constants and C which are independent of X and (B, A), (B',A)). Here, = (0n, E Nexd x Ndxr.
(A.3) (Strong identifiaiilit) For any natural number and distinct parameters (B),:,): :j [4], the functions in the set
(A,) X 02 B, (A,
(2)
02 (1) (1) 1o (A))
x
02 B, 00 (A,) X j (8. u [d]
linearly independent for almost surely X.
A.3. LOCAL PART
For the local part, we prove that
lim + (0..
inf
llfe - / 0.
23

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Assume that the above claim does not hold. It indicates that we find sequence of mixing measures G, Σnm exp(Co.j )o And in &1/ (0) such that
{ Dan == DD(G C.) 0, U -G. (L)) /Dan Dan O.
as n - oc. We denote V := (G as Voronoi cell of CN generated by the j-th components of ,. Without loss of generality, may assume that those Voronoi cells do not depend the sample size, i.e., Vj Vy. Therefore, the Voronoi loss D3n can be rewritten follows:
Dzn
Σ
exp(cn.)) exp(c))
E
Σ
exp(c..))lAB.....
|| | im
f==1 VEVy
Σ
Σ exp(c..) (I (I 11 + |I AA..)y1)?)
1 yEL:ivylssiev,
where ABn,y == Bud Bj. and Ann == An.i - Aj. for all i Vj and j' E [L].
Since D3n O. we have Eiev, exp(en,) + exp(c)), B,, - Bj. and An,: + A; for any Vy and j [L].
Throughout this proof, we assume without loss of generality that Mkg; = La with a note that our techniques can be extended to the general setting of that matrix. Now, the proof of the local part is divided into three steps as follows:
Step 1 - Taylor expansion.
First, we define
(x)
exp(X' (M& + (A)
+c]] Ifa, (X)- fa. (X)).
Then, we can decompose the function (x) as follows:
(x)
exp(cn.t) exp((X (M) (B (4.,.)))))) (o..
(B
exxp(X (M (M@ (420) (B)
exp(cns) exp(( (M2. + (o exp(x (M + (B)))) Sa. (x)
exp(cn.)) exp(c))) exp(X (Mā + (43(3) (M) (2)())) -Ga (x)
1=1
V
= A, (X) B, + (x.)..
(33)
Decomposition the function A, (X). We denote Ú(X; B, A): == exp(X (M) (B)oo (A))x, and F(X; B, A) Úx: B, V (X; B, A). Based on the number of elements in each Voronoi cells, we decompose the function A, (X) as follows:
T (a)) and V(X; B, :=
(x(
Σ
Σ
exp(co.)) F(X; Bn,i, An.) - A Bj.
A})) 4;]]
H& =1 1iE,,
+
Σ
exp(en.) F(X; An,i) F(X: -
= Am, Ax,(%x)
24

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
An application of the first-order Taylor expansion leads to
ala
U(X: Bu,i An,) U
Σ [n
(Anni) (ABan)) 02
(x; Bj, Aj) + Rij, (X),
0A01 B
alal V
V(X: A + Σ (AAn.t)) (a (ABn.t)) m
(X; Bj Aj))) + R
0A B
|=|=1
for any andd) j such that i V and [V,] 1. Here, the functions 1x and (X) denote the Taylor remainders. Collecting the above results leads to
1x,
E
exp(caa)
Σ
0 8 (x, AN,
ala
ad
glal
+ (AAn, (ABnn.))) 02
0 (X; Bj, A))ū Bj, A3) + (X)
alo
Σ Σ
Mn.ja
(X: Bjj A)) B3 Aj)
0
+1V, |a|=1
alali
+ Mn.j.
Bj. X Bj. A AJ) A3)
+ 1x))
where the function 1 (X) satisfies that Rr,, 1 (3n O. It due to the uniform petsschitzppropeeee of the function F. In the above display, the formulations of the coefficients Mn.j.a are given by:
exp(cn.)) ad!
Majjon
iEV,
for any |a| = 1.
Moving to the function (X), an application of the Taylor expansion to the second order leads to
ala Ū
(x, = Σ Σ
Ma.jan
XX; B A3)(
0A
alali
X 2: (X; Bj }
+ M Majjanco 01 Bo
@ (X: Aj) + (X))
(X: B Aj) DA AB
0A a
10|=1,18111
where the remainder (x) satisfies that 2x)2(X)/Dsn + 0. In this equation, the coefficients M Mn.jan and Mn Br 32 take the following forms:
exp(cnn)
Majaano Σ
a!! (A
KEV,
for any |a|| 2 and
exp(Cna)
Σ
(AAn,y) (ABn.j)) jo2t0s
a!!!!
25

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
for any |a| = |8| 1. Simple algebra leads to the following formulations of the partial derivatives of ((; B, A) and
VXX; B, A):
aÜ
XX; B, A) = X (x))) exx(x (M@+ (2())
0A(n) (1)
aU
XX; B, A) - x (0
(M) tog(B)o1(A))))
(B()) (8)
U-Ú
Xx B, A)
o2 o (B
x exxp( (M (0
(X: A)
+ | BB
O (D)
x exxp(x (M8 + (B)
X B, A) = ((A
(2))
+ x (d) X 02 (B))
exp(x T (M8 (B2(
aV
(X; B, A) = (
0A (a)
OV
(%; B, A) = (A) B yew.
B
V.
B, 1 (a- (A (1)) )o2((),
(A (v)
8
X B, A) 1 (m=a) 01 (A)Xa B (1) jew
O
VV
(X; A) = (A Je..
Plugging these formulations into the functions (x) and (X), we obtain that
(B3) Ax)( = Σ exp(x eM (2(3)))))444 (L (LT,,xx)o0((B3)) +
Xo, (Aj)x)
(MD + (2 0n (4))) + +0,
+ RR,,(())
An,((%) = Σ exp(X (M& +o
(LT,1, xx 02 (B;) + LT (A;)x
+ x Im.3,3% x 02 (
n, xx (B,) + In X (0)
+ ITT.o Xo, (Aj)x
+ x Ln..
+ X L 7, xx X o2 x (M + 0z By |1 (Aj)) + Xo, (Bj)
+ 01 01(4)) + (B)) +
+ (
-,,
R,,,(X),,
26

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
where the formulations of Ln,15; 2n22.1 ............ 1,, are given by:
In = of (A(1)))
L,,2 = (Masusee o2 (B(1)) )d u==1
In,3, = (Mnseuter. 0r 0f (A (1) )oi (A (v)) )2 u,
Ln = (M 0 [A (w) )
Ln,5, = (MaJ.aa to, 02 (1) (o)))) m,
Ln,6, (Mng. 20, 0/6 ( (B( )==1:
Ln, = (Mn.semne of (A (1)) Jo² (B()))) 1
In these equations, e, is denoted as the vector in I& such that its u-th element while its other elements are for any I < u S d. Furthermore, Ear is denoted as matrix in Rdxd with its wi-th entry is while other entries are zero
Decomposition of the function Bx(%): Moving to the function (x),, we can decompose this function as follows:
Bx(
Σ
Σ exp(cr.s)
- -U Jā. (X) [KK; Ba,i, An,)
A-Vy/=LiEV,
Σ
Σ exp(cn.i)
Ü Bugi An.)- Bj B3A (c. (X)
= Br, (%) + (%).
An application of the Taylor expansions up to the first order for Bx, and the second order for (x) leads to
Olal
(x) Σ
Σ
Mn.j.o 0z
(%: B3: (X) + Rx,,(x),
5:113 11
|a|=1
(X)
Σ
Σ
Mn
(X: Bj. 4j)fs, (X) + (
0
111
where the Taylor remainders (X), (XX satisfy that R + 0 and (X) -> 0. Direct calculation leads to
(X) Σ
exp(X (MM (M8 (2 LT,1 (x)
(B,) + L2,,, (Aj(x Sa. (X) Rx,,(X),
=
(x, = Σ exp(X (MQ (2(33)04433))
L XX o2 + L,,,,, Ko Koi
>
X X 02 (B + -T XX (B,) + x ((o + L LT, Xoo
Xo. (Aj)x
x Ln.,,% + X Ln, Xx To (B))) fa. (X) (%),
27

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Putting all the above results together, we can represent the function Qn (x) as follows:
(x(x)= = Σ exp(X (Mg + ox(B))o (Lm1, XX o2(B)) + (0,2))) (M)
(34)
5:13/==1
+ 02 (2 )o
4,1, Xo. (3, + +0
+ Σ exp(X (MQ 0 (o (4 (Ln,1, XX X oy(B)) + Xo,
X (A))
(35)
(B)) + X Ln.5, (Aj)x)² 2,, X (Aj)x + X T Ln,7, x + X Ln, Xx 02 (3)
+ X Ln.3, o2 (Bj
x (M + 02 (Bj)o [ + L,,, 1, Xo (B;)
+ (A + L x B3) + 01
| 6 + _
Σ exp(X (023)))
xx (B3) + xxa
2 Xo, (Aj) fa. (X)
Σ cxp(x (M (22 I,,,, XX o (B)) + _T Xo, (Aj)x
(36)
+ X 3 02 (2,))) |,A. 02 (B) + L 3n,5, X (Aj)x)²
+ L % (Aj)% X Ln, + X Lnn 02 (B) |0 (A))x 12 (x)
1
+ Σ Nns exp(x T (M8- + (B)()))) (M + BJAJ)X (3..)) (x)
j=1
+ (X) + (%) - Rxa(X)- - (%),
(37)
where Nu.= = EiEv, exp(cn.)) exp(e)) for any j E [L].
Step 2- Non-vanishing coefficients. As indicated in equation (37), the ratio (x) can be expressed as a linear
combination of the following independent functions:
", (X Aj), A (B)) [ AJ), (
(4) "
(x; A,),
(x (X), (X (X),
( oz (B)) (x), ((x; 02 B3 (x),
( X4 (8),
XXx,, fa, (X), T1 (8).
Ü(x; B; Bj. Aj), Ú(X: Bj Aj)oa, (X),
for any indices 1 j < L and < u1 U1. u2, 0/2 < d.
We demonstrate that at least one of the coefficients of these independent functions does not go to as n -> oc. Assume by contrary that all these coefficients of these linear independent functions go O. From equation (37), we obtain that
Mn.,oa /Dan, Mn.jan.8, 00.02 DDn and Nn.3 Dan go to 0 for all the coefficients 01, 01,02, a2, B2 E Naxd satisfying that
1 S Jail + |B1| + Jazl + 182] s 2.
28

---

RepLoRA: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Since / O, we find that for any [Z]]
EiEn, exp(en.)) -xxp(()
|Nn.sl
Din
D3n
Taking the summation of these limits leads
Ej=1 Eiev, exp(cn.)) -xx(ex))xxp(())) - 0.
(38)
D
Now, for any index [L] such that [V)] = 1, the limits MaJ.e..on 0s DDn + lead to Evev, explex, Alnn.nl - 0 as n + oo. Due to the equivalence between the fnomr and the fn-norm, this result directly implies that
Dan
EaN Eiev, ()
- 0.
D3n
Similarly, since Mnj.aa../ /Da ~ O, we also get that Eni Eiev, app(e.,))AAB... -> Thus, we obtain that
Dan
E exp(c,.)
0
(39)
Dmn
Moving to indices j E [L] such that their corresponding Voronoi cells satisfy that [V,] > 1. The limits Mn 0s/ Dan and Ma..00..2e. /Dn + induces that
0
ΣmiAA Liev, ((I( An.jl + | (Bn,i 122) - 0
(40)
D3n
By putting the results in equations (38), (39). and (40) together, we arrive at Dan. as n - 0C, which is Dzn contradiction. As consequence, least one of the coefficients of the linear independent functions in (x) does go to + 00.
Step 3 Application of the Fatou's lemma. We denote m, as the maximum of the absolute values of the coefficients of the linear independent functions in Qn (X)/ As at least one these coefficients does not go to 0, it indicates that 1 1/m, A oo as - co. Since llfe. - fa. (1)) DDn - as n co, we obtain llJa.. -Ja Saalli (m, Dsn) application of the Fatou's lemma leads
0 = lim lff. (e.11aam)
fa. (X) fa. (X) dl((%) 2 0.
liminf
my Don
min Dan
fā, (X) fā. (X)
That inequality demonstrates that lim i 400
for almost surely X. As n - co, we denote
Min Den
Nn.
Landd + Arjr
+ No.
Dam
my Dan
29

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
for any indices j E [L] and E [7]. Here, least one element of the set (do,s Ans = e [L], [7]] is different from 0.
fā. (X) Sa. (X)
Given the above notations, the limit lim i
implies that
My 3m
Σ exp(X (M&+ + 022()))) (A
XX o2(B)) + (A) (M + (B)()))0)))))
J
T X Bj) + 1 +
exp(X (M2
( (, x Xx (0,x(((X'BBB)) (B,))
02 -,
A7, x
X7, XX 02 (B))o
(M 02 o0
(A)))
/ X02 (Bj)
A |XAo, + A75x
E
exp
( M 02 (Bj) + X, Xo (4))
fa. (X)
=
Σ (M 023 X o2
2, X (A))
XX 02 (B)) X X,, (T, X \7 + X X7 XX (A))) (X)
A exp(X' (M) + o2 o2(B)) (M + ( B; fā, (x) =0
(41)
for almost surely X. However, that equation implies that all the coefficients {Ão.s Ang j E [L], E [7]] are 0. is contradiction. As a consequence, we obtain that
lim 100
inf Me fa. Lt 0.
A.3.2. GLOBAL PART
The result the local part implies that we can find positive constant such that
inf llfa fä_ / 0.
Therefore to obtain the conclusion of the theorem, we only need to prove that
inf GEū. (0:2
lifa - faa H/sn//D((G,G. > 0.
We assume by contradiction that the above claim does not hold. It indicates that there exists sequence of measures G" ((i=exp((nn,)) (,.,,,, in (0) such that
{
Ds G.) 111a.
(D G.) +
as n oo, which implies that llfā; fa. (1<9()) + as + x..
Given that a compact set, there exists a mixing measure G¹ in T. (9) such that one of the G 's subsequences converges to C Since (G, G.)) > e', we obtain that (3 G.) > s'. An application of the Fatou's 's lemma leads to
0 = lim [|5a; - S3. // 2 / lim inf +2
Ja: (x) - fā. (x) dx(
200
The above incquality indicates that fōr = fa. for almost surely X. From the identifiability property, we deduce that
T' = G.. It follows that (3 "G, = 0. contradicting the fact that Dz G.) e' > Hence, the proof is completed.
30

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Proof for the identifiabilitty property. We will prove that if (x(X) fa. (X) for almost surely X, then G = G.. To ease the presentation, for any mixing measure G
Σラ= Eー=eep((c) exp(e () E (O), we denote
exp(u)
softmax (u)
2 exp(x (MQ + o2 Bj | (4)))
where u E fx (MQ) (22+o((B)))(4))))) cg: :j [Z]]. The equation fā(X) = fa. (X) indicates that
1
( Σ softmax [x (Mg + 02
+, (2,(() + + (12()))0(AA)
softmax(X (Må + +=((B))o((44)) +z2(())) (42)
That equation implies that L L. As consequence, we find that
(softmax(X (M& σ. (z(B))o (4,)) e [ [softmax X (MQ + (o + oz) : e [4]
for almost surely X. By relabelling the indices, we can assume without loss of generality that for any j [4]
softmax(X (MQ (43)) softmax(X (Mā + 02 (B))or cy),
for almost surely X. Given the invariance translation of the softmax function, the equation (42) leads
exp (cj) eexp(xX(MM))) + + (B) (2)3))01)))
exp exx)) exp (M& + +0) +02(33001443)4 (43)
for almost surely X.
Now, the index set [L] can be partitioned into m subsets K2...,, Kn where m L, such that exp (cz) exp (ez.) for any indices j.j' E K, and E [m]] Thus, equation (43) can be rewritten as follows:
exp (c,) exp(X (M² + (2
(B,
JER,
exp (c)) exp) exp(X' (M&
jeK;
for almost surely The above equation implies that
(M + (2,))) j E R.) = ((M)) + (2 K.),
for any i E [m] and for almost surely X. Since the activation functions and 02 are algebraically independent, the above result indicates that
exp (c,))
Σ
Σ exp (c))
JER
jEk,
As consequence, G. and the proof completed.
31

---

RepLoRA: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
A.4. Proof of Proposition A.1
Recall from the setting that (X1, (X2, 12)..... (Xn, Yn) 12 Rd are ii.. samples from the following regression model:
Y, = fa. (X;) t i= 1,2......7
where the Gaussian noises E,,...,En Furthermore, (.) admits the following form:
En are i.i.d. and satisfy that E(e X 0 and Var(e;(X,) = o² for all i E [m].
1 exp(X (MQ + W2, A + (M² + W B,W,,,1))))
fe.
(x
where we denote D,(X) = Ek exp(X (MQ + W i cR). Finally, the least-square estimator G, takes the following form:
G == arg min (
Σ yr (x.)))
From the Gaussianity assumption of E, X; for all i E [n]. we have Y, ~ Nfc.. (X,),o²1a) for all E [n]. Therefore, the least square estimator G, indeed maximum likelihood estimator with respect to the data Y1 XI,..., Y, 7/,,, which takes the following form:
G E arg max (E)
of
Here, (XX)) stands for multivariate Gaussian distribution with mean fa(X) and covariance matrix o² An application of Theorem 7.4 from (van de Geer, 2000) leads
Mp(Y)Ja (X), p(V (x. Or( (Vlo))))))))
where h stands for the Hellinger distance. As the Hellinger distance between multivariate Gaussian distributions has closed-form expression, direct calculation yields that
1 1l.... Il. (X) fa. (x)
(X), 02)) (x) ((),,
exp
Therefore, for sufficiently large n, for some universal constant C' the above inequality leads to
1
flfa (X) fx.(x) < S0 log
[- C Clog(n)/n)
11+ C'log(n)/n
8o² log
C-Cllog(n))
C Clog(n)/n
S 802.
1 Clooggnn)/
< 16. 1logC\oo(n)/n.
That inequality is equivalent to As consequence, we find that
lff.
- fo, (X) (X)
Op( O /log(n)/n).
|lf.. - Sa.lla = Or Or((Voo(m)m/))
The proof of the proposition is completed.
32

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
B. Experimental Details
B.1. Hyperparameters
For the vision tasks, we use grid search to tune the learning rate in the range of {0.001, 0.005, 0.01, 0.05, 0.1}. and the weight decay in the range of {0.0001, 0.0005, 0.001, 2.01,,013, Other hyperparameters are reported in the tables below:
Table 7. Hyperparameter configurations RepLoRA for ViT-B/16 on the vision tasks.
Hy Hyperparameters (RepLoRA)
Classification Video-Action Recognition
Rank
8
Dropout
0
Base Optimizer
Adam W Cosine
Lr Scheduler
Batch size
64
512
Warmup steps
100
Epochs
100
90
Table 8. Hyperparameter configurations of RepLoRA for LLaMB-7B/13B the commonsense reasoning tasks.
Hyperparameters (RepLoRA)
LLaMA-7B
LLaMA- 133
Rank r
16
32
16
32
Q
32
64
32
64
Dropout
0.05
Base Optimizer
Adam
LR
2.00E 04
1.00Æ
04
2.00 04
1.00E 04
Lr Scheduler
Linear
Batch size Warmup steps
32
100
Epochs
Table 9. Hyperparameter configurations RepLoRA for VL-BART on the Image/Vidco- Text Understanding tasks.
Hyperparameters (RepLoRA)
Image- Text Hideo-Text
Rank
128
128
Dropout
0
Base Optimizer
AdamW
LR
1.00Æ
03
3.00E - 04
Lr Scheduler
Linear
Batch size
300
40
Warmup ratio
0.1
Epochs
20
7
C. Additional Experiments
C.1. Sample Efficiency on the FGVC Datasets
C.2. Linear vs Non-linear Reparameterization
Recall that in our practical method, the low-rank matrices are given by:
33

---

RepLoRA: Reparamcterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Table 10. Detail statistic of RepLoRA and LoRA sample efficiencies five PGVC datasets.
CUB 200 2011
NABirds
OxfordF lower
StanfordDogs
Stan
AVG
0.01
10.1
2.2
15.1
12.1
13.3
10.56
0.1
70.6
50.3
71.6
65.3
60.2
63.6
LoRA
0.3
75.1
70.8
85.3
80.2
73.3
76.94
0.5
80.1
76.9
96.2
85
75.5
82.74
84.6
78.2
98.9
85.1
77.1
84.78
0.01
50.2
55.3
51.2
59.8
51.32
0.1
80.2
79.6
85.9
80.8
79.1
81.12
RepLoRA
0.3
85.3
85.9
93.1
79.1
81.6
85.0
0.5
87.9
86
98.9
86.3
84.9
88.8
89.1
86.1
99.3
91.2
87.6
90.66
CUB-200-2011
NABirds
OxwordFlowers
20.3%
Accunnty (h4l
Accrsoo
40.15
37.9%
Lo RcoLoRA
Lota
RecLoRA
RepLoRA
Teatiing Fraction 153
Training Trction
Traniing Frection
StanfordCars
StanfordDogs
LoRA Laplalt
18.9%
AcCuozy 150
5.55%
1%
LoRA HapLiHA
Trining
Figure 4. Visualization of sample efficiency LoRA and RepLoRA on five FGVC Dalasets.
AQ (A), BQ = (B)
Av of (A), By /. (B)
For the linear reparameterization, of 0f of 02 implemented with linear layers without activation. For the nonlinear reparameterization setting, these functions implemented with two-layer neural network with hidden dimension of 64 and sigmoid activation functions on all settings.
Detail results. The tables below report all results for linear vs. non-linear reparameterization
34

---

RepLoRR: Reparameterizing Low-rank Adaptation from the Perspective of Mixture of Experts
Table 11. Image classification accuracy of Linear vs. Non-linear Reparameterization VTAB-1K.
ClevE Dist
SNORB-AZ -iim
Flowes10z
Came yon
Re919945
Ret.noopar
dSpr-Ori
2NORB-E14
STFARIN
Sun392
BUrOSAr
D15
Tete
SVIN
DMLab
KITZI
d5px
Method
AVG
LoRA
G7.1
91.4
69.4
98.2
90.4
83.3
54
84.9
95.3
84.4
73.6
82.9
69.2
49.8
78.3
75.7
47.1
31
44
72.2
Repl (Linear)
71.7
86.3
75.6
70.6
76.7
RepLoRA (Nom Nin-lineur)
73.2
94.1
73.3
99.3
94.4
89.1
58.9
89.2
97.5
87.9
77.8
85.1
72.6
55.7
81.2
81.7
49.2
35.7
47.3
75.9
Table 12. Image classification accuracy of Linear Non-linear Reparameterizattion FGVC datasets.
Method
CUB.200.2011
NABirds OxfordFlower StanfordDogs StanfordCars
AVG
LoRA
84.6
78.2
98.9
85.1
77.1
84.7
RepLoRA (Lincar) 88.6
85.2
98.1
89.9
83.3
89.0
RepLoRA (Non-linear)
89.1
86.1
99.3
91.2
87.6
90.7
Table 13. Video action recognition performance of Linear Non-linear Reparameterizatioon SSv2 and EMDB51 datasets.
SSv2
HMDB51
Method
Model
Pretraining
#Params (M)
Acc
PPT
Ace@ 1
PPT
LoRA
Video Swin-B
Kinetics400
0.75
38.34
0.37
62.12
0.60
RepLoRA (Linear)
Video Swin-B
Kinetics400
0.91
41.89
0.40
66.01
0.63
RepLoRA (Non-linear)
Video Swin-B
Kinetics400
1.45
43.12
0.41
68.23
0.64
Table 14. Performance of performance of Linear vs. Non-linear Reparameterization on the Commonsense Reasoning task.
PEFT Method
BoolQ
PIQA
STQA
Hell Hallaswag
Wi WinoGrande
ARC-e
ARC-c
OBQA
AVG
LoRA
67.2
79.4
76.6
78.3
78.4
77.1
61.5
74.2
74.0875
I.aMA-7B
RepLoRA (Linear)
67.1
81.7
79.3
77.9
79.6
78.4
64.1
77.4
75.6875
RepLoRA (Non-linear)
71.8
84.1
78.9
85.2
83.3
82.4
66.2
81.2
79.1375
LoRA
71.7
82.4
79.6
90.4
83.6
83.1
68.5
82.1
80.175
LLAMA-13B
RepLoRA (Linear)
72.6
82.2
82.3
90.4
84.1
82.9
67.9
83.9
80.7875
RepLoRR (Non-linear)
73.1
85.2
84.7
85,9
84.7
73.4
85.6
82.9625
Table 15. Performance of Linear vs Non-linear reparameterization on the image-text understanding task on VL-BART.
Method
VQA
GQA
NVLR
COCO
Cap
Avg.
LoRA
65.2
53.6
71.9
115.3
76.5
RepLoRA (Linear)
65.5
55
72.3
115.9
77.2
RepLoRA (Non-linear)
66.5
55.4
74.2
116.2
78.1
Table 16. Performance of Linear Non-linear reparameterization the video-text understanding task VL-BART.
Method
TVQA
How2OA
TVC
YC2C
Avg.
LoRA
75.5
72.9
44.6
140.9
83.5
RepLoRA (Linear)
76.3
73.4
44.9
143.2
84.5
RepLoRA (Non-linear)
77.8
75.1
46.6
151.6
87.8
35


Reviews:
[{"review_id": "NZ02KSP6d9", "reviewer": "Reviewer_zZoN", "summary": "The apper combines LoRA into multi-head parts of MSA and treats different heads as experts to build a mixtural of experts. In addition, authors use lightweight MLP to conduct reparameter opertations, which improves sampling efficiency while reducing data requirements compared to the original LoRA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful feedback. In response to their concern, we’ve expanded our analysis to include additional comparisons with three LoRA adapters: VeRA [1], DoRA [2], and MoR [3], as suggested by the reviewer. These experiments were carried out on the image classification task using the FGVC datasets. For MoR, we specifically report results with 8 experts. The detailed results are presented below:\n\n| Method | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | AVG  | PPT |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LoRA     | 84.6     | 78.2     |98.9     | 85.1     | 77.1     |84.8     |0.82 |\n| DoRA     | 87.3     | 80.0     |99.1     | 87.6     | 81.9     |87.2     |0.88 |\n| VeRA     | 85.1     | 79.2     |97.4     | 87.3     | 76.3     |85.1     |0.88 |\n| MoR     |  87.6    |  82.5    | 99.3    |  89.7    |   84.7   |  88.8    | 0.89 |\n| RepLoRA     | **89.1**     | **86.1**     | **99.3**    | **91.2**     | **87.6**     | **90.7**     |**0.90** |\n\nFollowing the reviewer’s suggestion, we will incorporate these results into the final revision. \n\nWhen it comes to training, RepLoRA introduces only a minimal increase in parameters compared to LoRA, as seen in the number of parameters and PPT. As a result, RepLoRA does not introduce any significant additional training time, FLOPs, or memory usage when compared to LoRA. Additionally, as highlighted in the main text, the reparameterization matrices can be discarded during inference, making the inference process identical to that of LoRA.\n\n**References**\n\n[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024\n\n[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024\n\n[3] MoR: Mixture of Ranks for Low-Rank Adaptation Tuning. Arxiv. 2024", "claims_and_evidence": "The claims in the submission are generally well-supported.", "methods_and_evaluation_criteria": "The proposed method offers a novel insight, reducing  the data needed to achieve a desired estimation error from an exponential\nscale to a polynomial scale. \n\n+ Authors measure classification accuracy trends at different training scales.\n+ Linear and nonlinear RepLoRA modules are ablated on 7 application scenarios.\n+ Based on the baseline LLama7B/13B, the proposed methods are verified to be effective on multiple datasets", "theoretical_claims": "Theoretical proofs are provided. Based on it, authors discuss LoRA with MoE for achieving optimal sampling. Furthermore, RepLoRA was proposed with an effective and efficient approach to PEFT.", "experimental_designs_or_analyses": "The experimental designs are comprehensive, with LLama7B/13B baselines and extensive datasets. By experimental analysis, the proposed methods are verified in both aspects in performance and parameters.", "relation_to_broader_scientific_literature": "This paper is closely related to the relevant research in fields of LoRA, MOE, and PEFT. If possible, authors can further illustrate the related mathematical basis/thinking in scientific literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "+ Strengths \n1) The proposed framework serves as a theoretical foundation that underpins the various methodologies and processes within LoRA scope. \n2) Authors measure classification accuracy trends at different training scales.\n3) Authors perform experiments on multiple scenarios for linear and non-linear RepLoRA modules.\n4) LLama7B/13B are classical LLMs. The proposed methods utilize  LLama7B/13B to verified to be effective on multiple datasets\n\n\n- Weaknesses\n1) The experimental situation with more LoRA adapters is unknown.\n2) Should compare LoRA with MoE works [1, 2 , 3]\n\n[1] Mixture-of-loras: An efficient multitask tuning for large language models. COLING. 2024.\n\n[2] Mixture-of-subspaces in low-rank adaptation. EMNLP. 2024.\n\n[3] MoR: Mixture of Ranks for Low-Rank Adaptation Tuning. Arxiv. 2024", "other_comments_or_suggestions": "The proposed method is a general method, but the current work focuses on verifying it in combination with LoRA on LLama. If possible, more experiments are better.", "questions_for_authors": "1) Please see weaknesses.\n2) The cost of training or inference is unclear, such as GPU memory usage, training/inference time, and flops.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBJBcPkYv0", "reviewer": "Reviewer_yfWw", "summary": "This paper proposes RepLoRA, a method that reparameterizes the low-rank matrices of LoRA using a lightweight MLP. RepLoRA surpasses baseline LoRA by up to 40.0% and achieves similar results with baseline with only 30.0% of the training data. Additionally, this work provides a theoretical analysis of LoRA from the perspective of a mixture of experts, demonstrating that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential scale to a polynomial scale. Experiments across various tasks, including language (commonsense reasoning), image (classification), video (video action recognition), and multi-modal (image/video-text understanding), demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and would like to address your concerns as follows:\n\n**Regarding the analysis of the theoretical results:** Our theoretical analysis demonstrates that LoRA with reparameterization offers superior *sample efficiency* compared to LoRA without reparameterization. To empirically analyze this theoretical claim, we dedicated the final experiment in the experimental section to validate these theoretical findings. The results show that reparameterization in LoRA significantly improves *sample efficiency*, as illustrated in Figure 2.\n\n**Regarding the comparison with related works:** Following the reviewer’s suggestion, we have included additional comparisons with VeRA [1] and DoRA [2] on the image classification task using the FGVC dataset, as detailed below:\n\n\n| Methods | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | AVG | PPT |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LoRA     | 84.6     | 78.2     |98.9     | 85.1     | 77.1     |84.8     | 0.82 |\n| DoRA     | 87.3     | 80.0     |99.1     | 87.6     | 81.9     |87.2     |0.88 |\n| VeRA     | 85.1     | 79.2     |97.4     | 87.3     | 76.3     |85.1     |0.88 |\n| RepLoRA     | **89.1**     | **86.1**     | **99.3**    | **91.2**     | **87.6**     | **90.7**     |**0.90** |\n\nThe results demonstrate that RepLoRA outperforms both LoRA variants by large margins, emphasizing its practical advantages. In response to the reviewer’s suggestion, we will include this finding in the final revision.\n\n**References**\n\n[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024\n\n[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024", "claims_and_evidence": "Most of the claims are supported by cited works.", "methods_and_evaluation_criteria": "The proposed method is effective and the evaluation criteria are valid.", "theoretical_claims": "I cannot verify the correctness of the proofs due to my non-mathematical background. Please refer to other reviewers for the correctness check.", "experimental_designs_or_analyses": "The experiments robustly demonstrate the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "Low-rank Adaptation (LoRA) has gained significant traction as a method for fine-tuning large-scale foundation models, yet its theoretical underpinnings have remained relatively unexplored. This paper contributes to the broader scientific literature by providing a theoretical analysis of LoRA through its connection to Mixture of Experts models. By situating LoRA within this framework, they demonstrate that simple reparameterizations of LoRA matrices can significantly expedite the low-rank matrix estimation process. Specifically, the findings show that reparameterization can reduce the data required to achieve a desired estimation error from an exponential to a polynomial scale, thereby enhancing sample efficiency.", "essential_references_not_discussed": "Most of the relevant related works have already been cited.", "other_strengths_and_weaknesses": "Strengths\n1.This work provides an insightful analysis of the impact of LoRA on multi-head self-attention layers from the perspective of a mixture of experts (MoE), offering valuable inspiration.\n2.The proposed RepLoRA achieves outstanding results across various tasks, including commonsense reasoning, image classification, video action recognition, and image/video-text understanding.\n\nWeaknesses\n1.It would be better to provide experimental analysis for the theoritical proofs, such as the convergence curve w./w.o reparametrization.\n2.Lack of comparison with similar works, such as [a,b,c]\n[a] DoRA: Weight-Decomposed Low-Rank Adaptation\n[b] VERA: VECTOR-BASED RANDOM MATRIX ADAPTATION.\n[c] LORA-FA: MEMORY-EFFICIENT LOW-RANK ADAPTATION FOR LARGE LANGUAGE MODELS FINE-TUNING", "other_comments_or_suggestions": "see Strengths And Weaknesses", "questions_for_authors": "see Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kRPeIEkDde", "reviewer": "Reviewer_8a5d", "summary": "The authors proposed two reparametrizations of LoRA under which the convergence rate improves from $\\mathcal{O}_P(\\frac{1}{\\log^{\\tau}(n)})$ of vanilla LoRA to $\\mathcal{O}_P(\\sqrt{\\frac{\\log (n)}{n}})$. Empirical results demonstrate that both reparametrizations outperform vanilla LoRA on real datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and would like to address the concerns raised as follows:\n\n+ **Regarding the discrepancy between the theoretical results and the experiment setup**: Thanks for this feedback. We would like to clarify that the assumption in Section 4.2 that $A_Q=A_V$ is for simplicity and can totally be generalized to the setting where they share only the learnable matrix $A$. In particular, we can reformulate those matrices as $A_Q=W_{Q,1}A, A_V=W_{V,1}A$ for the simple linear reparametrization and as $A_Q=\\sigma_1(W_{Q,1}A), A_V=\\sigma_1(W_{V,1}A)$ for the non-linear reparametrization. Tailored to that setting, we need to add several additional terms involving parameters $W_{Q,1}$ and $W_{V,1}$ rather than merely parameters $W_{1}$ as in the current setting, making the convergence analysis unnecessarily complicated. Therefore, we assume without loss of generalization that $A_{Q}=A_{V}=W_{1}A$ or $A_{Q}=A_{V}=\\sigma_1(W_{1}A)$ in Section 4.2 to simplify the analysis, making it more accessible.\n\n+ **Regarding the experiment tying $A_Q$ and $A_V$:** To further support our theoretical findings, we conduct experiments on FGVC datasets by tying $A_Q$ with $A_V$ and $B_Q$ with $B_V$ in RepLoRA. The results, summarized in the table below, show that tying $A_Q$ and $A_V$ slightly reduces the model's expressiveness, resulting in a modest drop in performance compared to the original RepLoRA with untied matrices. However, tied RepLoRA still significantly outperforms vanilla LoRA, which reinforces the practical value of our approach, even under constrained parameterization.\n\n\n| Methods | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | AVG |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | \n| LoRA     | 84.6     | 78.2     |98.9     | 85.1     | 77.1     |84.8     |\n| RepLoRA (tied) | 87.2 | 83.8 |99.0     | 85.6     | 85.4     |88.9     |\n| RepLoRA (untied) | **89.1** | **86.1** | **99.3** | **91.2** | **87.6** | **90.7**  |\n\n+ **Regarding the implications of Theorem 4.1:** Thanks for your comment. We would like to emphasize that the results in lines 187-196 (right column) are consequences of Theorem 4.1 in our paper. In particular, by combining the result of Theorem 4.1 and the formulation of the Voronoi loss $\\mathcal{D}_{1,r}$ in lines 167-173 (right column), we deduce the convergence rates of low-rank matrices estimation are slower than polynomial rates $O(n^{1-/2r})$ for all $r\\geq 1$, where $n$ is the sample size. Due to the inequality $\\log(n)<n$, these rates are even slower than the order $O(1/\\log^{\\tau}(n))$ for some positive constant $\\tau$. As a result, to achieve a given approximation error $\\epsilon=O(1/\\log^{\\tau}(n))$ of estimating the low-rank matrices, we need exponentially many data points $O(\\exp(\\epsilon^{-1/\\tau}))$. We will consider formulating those implications into a corollary following Theorem 4.1 in the revision of our manuscript.\n\n\n+ **Regarding the activation function**: The nonlinearity was implemented using the sigmoid function. We appreciate the reviewer’s suggestion and will include this detail in the final revision.", "claims_and_evidence": "The claim that the proposed reparametrizations are more sample efficient are backed up by formal theorem statements and their proofs. However, there is a discrepancy between the theoretical results and the experimental setup: while Theorems 4.1-3 assume $A_Q$ and $A_V$ are tied, the experiments only assume that they result from the same low rank matrices. Therefore the theoretical results cannot fully explain the empirical successes.", "methods_and_evaluation_criteria": "The experiment setup seems correct. However, I would love to see experiments that validate the theoretical results: namely experiments that tie $A_Q$ and $A_V$ together.", "theoretical_claims": "I checked proof for Theorem 4.1 which seems correct. However the implications in L187-196 (right column) should be formalized into a proposition and proven. I did not carefully check proofs for Theorems 4.2-3 which also appear in the appendix.", "experimental_designs_or_analyses": "A main message of this paper is that parameter estimation of the proposed reparameterizations is much more sample efficient. Besides the discrepancy between what is being proven and what is being implemented, another issue is that the authors did not explicitly evaluate whether the reparametrization hurts expressiveness in real world tasks when data is abundant. One way to help answer this question is to evaluate on pretraining tasks where there is much more training data.", "relation_to_broader_scientific_literature": "This work adds on the ongoing discussion on how to best train LoRA models ([Yen et al., 2025](https://openreview.net/forum?id=VpWki1v2P8), [Hayou et al., 2024](https://arxiv.org/abs/2402.12354)).", "essential_references_not_discussed": "Discussion of previous empirical observations on LoRA's performance (_e.g._, [Biderman et al., 2024](https://arxiv.org/abs/2405.09673)) can further improve the paper.", "other_strengths_and_weaknesses": "This paper explores the sample efficiency aspects of LoRA which is a novel and significant contribution.", "other_comments_or_suggestions": "- Minor editing errors on L218, L175 (right column), and L716.\n- The paper is both dense and packed; proof sketches in the main text can provide useful guidance for the reader.\n- Choice of nonlinearity function used is missing in the manuscript.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jLM6G6vUAJ", "reviewer": "Reviewer_ykAM", "summary": "This work studies a new variant of LoRA. First, the authors show that under certain settings, LoRA requires exponential sample complexity. Then, they introduce a simple reparameterization strategy, which builds a single generator for Q, V layers. The generator can be a single layer with or without activations. Using this reparameterization, the authors show that the sample complexity can be reduced to polynomial scale, which reveals the advantage of the new method. Experiments are conducted on multiple domains including LLMs, images/videos and multi-modal datasets. The proposed method consistently outperforms LoRA.\n\n## Update after rebuttal\n\nAs the authors' address most concerns of all reviewers, I would keep the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and would like to address the concerns as follows:\n\n+ **Regarding the comparison with other variants of LoRA:** Following the reviewer’s suggestion, we conducted an additional experiment on the image classification task using the FGVC dataset to compare RepLoRA with VeRA [1] and DoRA [2]. The results are presented below:\n\n| Methods | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | AVG | PPT |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LoRA     | 84.6     | 78.2     |98.9     | 85.1     | 77.1     |84.8     | 0.82 |\n| DoRA     | 87.3     | 80.0     |99.1     | 87.6     | 81.9     |87.2     |0.88 |\n| VeRA     | 85.1     | 79.2     |97.4     | 87.3     | 76.3     |85.1     |0.88 |\n| RepLoRA     | **89.1**     | **86.1**     | **99.3**    | **91.2**     | **87.6**     | **90.7**     |**0.90** |\n\nFurthermore, we would like to highlight that RepLoRA has already been compared to other methods that utilize hypernetworks to generate adapters in the main paper. Specifically, in the main text, we included a comparison with prefix tuning [3], which leverages an MLP to generate the adapters, specifically the prepended prompts. Building on the reviewer’s helpful suggestion, we will also incorporate comparisons with VeRA and DoRA in the revised version to provide a more comprehensive evaluation.\n\n+ **Regarding the comparisons with comparable parameter sizes:**  In all experiments, we reparameterized the low-rank adapters using MLPs with a hidden dimension of $h=64$, which we identified as the optimal setting through extensive tuning in terms of both PPT and accuracy. To address this concern, we also conducted experiments with a reduced hidden dimension of $h=8$, resulting in a comparable parameter size. The results, presented below, show a slight overall drop; however, RepLoRA still significantly outperforms vanilla LoRA, demonstrating its robustness and practical benefits under tighter parameter constraints:\n\n| Methods | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | AVG | PPT |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LoRA     | 84.6     | 78.2     |98.9     | 85.1     | 77.1     |84.8     | 0.82 |\n| RepLoRA $(h=8)$     | 87.5     | 85.8     |98.4     | **91.6**     | 83.0     |   89.2  |0.90 |\n| RepLoRA $(h=64)$    | **89.1**     | **86.1**     | **99.3**    | 91.2     | **87.6**     | **90.7**     |**0.90** |\n\nFollowing the reviewer's suggestion, we will include this finding in the appendix of the final manuscript.\n\n**References**\n\n[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024\n\n[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024\n\n[3] Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL. 2021", "claims_and_evidence": "Yes, the claims are supported by theoretical and empirical analysis.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I did not check the proof.", "experimental_designs_or_analyses": "The experimental design is sound, including datasets and pre-trained models of different domains.", "relation_to_broader_scientific_literature": "The key contribution is to improve LoRA by reparameterization. It may contribute to the field of parameter-efficient fine-tuning, including many other variants of LoRA.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n\n1. This work studies LoRA from the view of MoE, and theoretical analysis about the sample complexity. These results may be useful for future research in this field.\n\n2. The proposed method is simple and theoretically sound. It may be also applied to other variants of LoRA.\n\n3. Experiments are also conducted on diverse domains. The proposed method seems good.\n\n### Weakness\n\n1. While the method is simple and advantageous over LoRA, it would be better to compare with some more advanced variants of LoRA, especially some work that applies hypernetworks to generate adapters, which is very similar to this one. For example the following.\n\n    https://openreview.net/forum?id=iP8ig954Uz\n\n2. The parameters of RepLoRA are higher than LoRA. Did the authors try experiments with comparable parameter sizes?", "other_comments_or_suggestions": "In Line 349, PETL is not introduced.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a compelling perspective on rethinking LoRA as a specific mechanism of Mixture of Experts (MoE) with a reparameterization scheme. All the reviewers have provided positive feedback regarding this paper. During the rebuttal phase, the authors offered additional justifications and experiments to support their methodology. Overall, I recommend accepting this paper. However, I strongly encourage the authors to incorporate their responses and any relevant discussions into the final manuscript.
Decision: Accept (poster)