A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Authors: Mengyang Sun
Yihao Wang
Tao Feng
Dan Zhang
Yifan Zhu
Jie Tang
Venue: ICML 2025 poster (2025)
Keywords: Parameter-Efficient Fine-Tuning
Low-Rank Adaptation
Mixture of Experts
Foundation Models
Gradient Optimization
Riemannian Preconditioner
Primary area: deep_learning->large_language_models
TL;DR: We propose a stronger training strategy for Mixture of LoRAs based on improved Riemannian preconditioners, which boosts learning procedures and downstream performances.

Abstract: In order to streamline the fine-tuning of foundation models, Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation. The underlying concept of LoRA involves decomposing a full-rank matrix into the product of two lower-rank matrices, which reduces storage consumption and accelerates the training process. Furthermore, to address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters. The integration of LoRA experts leads to a visible improvement across several downstream scenes. However, the mixture of LoRAs (MoE-LoRA) still exhibits its low robustness during tuning and inferring. Inspired by the Riemannian Preconditioners which train LoRA as a sub-space projector, we propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning by gate-rescaled multi-space projections. We provide both a theoretical solution as well as an alternative engineering strategy. Examinations on SGD and AdamW optimizers demonstrate the effectiveness of our methodology. Source code is available at https://github.com/THUDM/MoELoRA_Riemannian.

A Stronger Mixture of Low-Rank Experts for Fine- Tunnng Foundation Models
Mengyang Sun
Yihao Wang?
Tao Feng'
Dan Zhang 1 Yifan Zhu Jie Tang'
to its versatility and simplicity. In detail, LoRA introduces trainable low-rank matrices A and B update the internal modules of FMs, which is given by X W + BA, where X represents the overall weight matrix after integrating pretrained weights W and LoRA modules A and B. In a sense, the product of A and B serves as an approximation of the full-rank update (in other words, Fully Fine-Tuning or FFT) for the pre- rrentraind weights. While LoRA significantly reduces the number of trainable parameters, it also imposes two limitations: One is limited representation, and the other one is gradient sub-optimality.
Abstract
In order to streamline the fine-tuning of founda- tion models, Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adapta- tion. The underlying concept of LoRA involves decomposing full-rank matrix into the prod- uct of two lower-rank matrices, which reduces storage consumption and accelerates the training process. Furthermore, to address the limited ex- pressive capacity of LoRA, the Mixture-of- -E-eer (MoE) has been introduced for incorporating mul- tiple LoRA adapters. The integration of LoRA experts leads a visible improvement across sev- eral downstream scenes. However, the mixture of LoRAs (MoE-LoRA) still exhibits its low robust- ness during tuning and inferring. Inspired by the Riemannian Preconditioners which train LoRA as sub-space projector, we propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning by gate-rescaled multi-space projections. We provide both a theoretical solu- tion as well as an alternative engineering strategy. Examinations on SGD and AdamW optimizers demonstrate the effectiveness of our methodology. Source code available at https: //tl::///tttttt thub. cOm/THUDM//oBLLC. Bttp://////.LLL..AARRAAeEEE.... www.eeanaaaeaaaaiia..inia
Limitation I: Limited representation. A natural problem of low-rank matrices lies in less powerful representation, espe- cially in complex tasks. To tackle this, one straightforward solution is the integration of multiple LoRA modules into the centure-of-expers framework, known as MoE-LoRA. Figure (left) illustrates plain MoE-LoRA framework. These efforts tangibly improved the performance of LoRA in many scenarios, like vision-language tasks, multi-task learning, continual learning, etc. In nutshell, the route of MoF-LLRR can be roughly categorized into two lines: (i) Designing dedicated MoE-LoRA frameworks for spe- cific domains, such as MOELoRA (Liu et al., 2023) and MoCLE (Gou et al., 2023). (ii Technically improving MoE- LoRA via architectural, updating, and loss constraints, such as MoLA (Gao et al., 2024) and HydraLoRA A (Tian et al., 2024). Nevertheless, most of these efforts fail to consider the instability and inefficiency of training MoE-LoRA.
Limitation 2: Gradient Sub-optimality Another concern that plagues LoRA is gradient sub-optimality. This occurs since the low-rank matrices A and B together form quo- tient manifold space with a certain curvature, leading to an inconsistency between the inner-manifold optimal and the full-rank optimal gradient. This further leads to a sub- optimal training process for LoRA. To alleviate, Zhang et al. (Zhang & Pilanci, 2024) enhances LoRA gradients by Riemannian gradient preconditioner, given by VAL ( V and VgC- V V=((AA')- These pre- conditioners contribute to constructing two gradient projec- tors after a mathematical derivation, ensuring the update is done in accord with the full-rank gradient projection onto the row space of A and the column space of B, that is Xnew X - (Vx +Projrom(A)) (Vx Here the notation Projv represents a projection func-
1. Introduction
Parameter-Fifficientt Fine-Tuning (PEFT) techniques offer a cost-effective solution for fine-tuning foundation models (FMs) (Zhang et al., 2025a; Fu et al., 2024). Among these, Low-Rank Adaptation (LoRA) is a prevalent technology due
1 Work was done while these authors interned at Zhipu AI. 'Department of Computer Science and Technology. Tsinghua University, Beijing, China: "Computer School, Beijing Informa- tion Science and Technology University, Beijing, China; 'School of Computer Science, Beijing University of Posts and Telecom- munications. Beijing, China. Correspondence to: Jie Tang <ji- etang @ were@@tssighuuuu..oomm edu, Tao Feng <fengtao.hi hi ceii@@mail.com .om...com>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
MoE-LoRA Framework
Add & Norm
Opiiaal Desced
Oytinal Douss Packetio
Expent Optanal Deseet
A
B
B,A, Manifold
X
81
Expert,
Expen, Full Matrix Space
B, Manifold
FFN
Ronter
ei A
200
Add Norm
Multi-Head Attention
A2
B2
B. Manifold
X
82
of M
Manifold
Embedding
Expert;
Expert: Full Matris Space
MoB Full Space
Figure 1. The whole MoE-LoRA architecture and an insight into its gradient updating process. The left part of this figure shows a pipeline of mixture of LoRAs, which fixes the pretrained weights of Feed- Forward Network (FFN), and trains series of LoRA adapters together with a routering gate. The right part exhibits how MoE-LoRA is updated. Specifically, we plot an example of a 2-Expert MoE- LoRA in condition that 91 92. which results in further distorted manifold gn Bi Ai. Here we simply omit the fixed pretrained weights and suppose X = g1 F1 + 92 Foz for convenient display. Since that, for a random step we plot a state point 1 x which equals to 21 (t) B, (9) (+) tox (1) (*) and so that serves as the center point of the manifold states at This figure illustrates that gi B1 A1 has a higher curvature so that its local optimal descent and its global optimal descent projection are more distinct. That indicates a requirement for gate-related preconditioners.
tion which projects given matrix M onto a subspace con- structed by all vectors in set V.
gate value g. Our improved gradient updating process for MoB-LoRA is given by:
Through comprehensive analysis of Limitation I and Lim-
N
Nappert
itation 2, natural question arises:
Xnew = X -
Σ i=l
How can LoRA-based structure further approximate the full fine-tuning with the guaranteed Limitations and 2?
Nnagert
Σ
gi rA
Inspired by MoE-LoRA and the gradient preconditioning methods, straightforward answer this question is to integrate both approaches to simultancously overcome the
We summarize our contributions as follows:
We integrate the mixture of LoRAs structure with the Riemannian preconditioners to alleviate both limited representation and sub-optimality issues of LoRA.
representative and sub-optimal limitations. Specifically, the gradients of each LoRA expert can be refined by a respec- tive Riemannian preconditioner. However, we claim that the process weighed summing experts MoE-LoRA intro- duces a gate-based scaling for each LoRA expert's 's manifold, thereby altering their curvatures with regard to their respec- tive gate value 9:s. We illustrate this phenomenon in the right part of Figure 1, which plots an example of 2-Expert MoE-LoRA in a condition that < 92- Specially, in their respective spaces of Expert 1 and 2, Manifolds constructed by B, A, and A2 initially share the same curvature since their low-rank matrices are the same rank. However, after being multiplied by gate values, Manifold gi Bi more rescaled that it provides larger curvature than 92 B2A2 in the MoE full space. As a result, Expert 1 exhibits higher distinction between global optimal and inner-manifold opti- mal descents. This phenomenon indicates that the precondi- tioners for each expert shall be further refined, to take the impact of gate values into consideration. In this paper, we propose a simple but effective solution further rescale the gradients of each expert in a lightweight way by respective
We emphasize a distortion issue behind per-expert pre- conditioning, and respectively propose a theoretical and an engineering solution for satee-vllle--eccalle gradient preconditioning of MoE-LoRA.
We implement and examine our rescaling approach for MoE-LoRA under a series of foundation models, illustrating our effectiveness across various tasks.
2. Related Works
2.1. LoRA and LoRA Variants
LoRA (Hu al., 2022) decomposes full-rank matrix into product of two low-rank matrices, which has been widely considered an effective solution for parameter-effiiient fine- tuning. Studies have proposed several variants to reform LoRA: For initialization, PISSA (Meng et al., 2024) lever- ages singular value decomposition (SVD) to obtain the prin- cipal singular components of W, while MiLLRA (Wang
2

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
al., 2025a) utilizes secondary singular values and vectors. LoRA-Pro (Wang et al., 2025b) and LoRA-GA (Wang et al., 2024b) approximate the direction of initial gradients to align them with that of the fully fine-tuning. LoRA+ (Hayou et al., 2024) introduces a learning rate separating strat- egy with BBB > TA.- ResLoRA (Shi et al., 2024b) and SIBO (Wen et al., 2024) accelerate convergence and mitigate over-smoothing by introducing residual paths. DoRA (Liu et al., 2024b) decomposes the weight vector into direc- tion and magnitude and only uses its direction component. LsLoRA (Kalajdzievski, 2023) proposes a rank-stabilizeed scaling factor \₁ = r2/² to ensure stable gradient updates. To prevent overfitting. BiLoRA (Qiang et al., 2024) adopts bi-level optimizing strategy, while others implement dropout mechanisms (Wang et al., 2024a; Lin et al., 2024).
ters by calculating gradient-based updates. To accelerate the optimizing process, the concept of gradient preconditioning has been introduced. Advanced techniques such as Ada- grad (Duchi et al., 2011) dynamically adjust the learning rate by an accumulated squared gradients G. = Σ o and update model by A6: -n 9t. Adam (Kingma & Ba, 2015) extends this approach by incorporating mo- mentum and bias correction, scaling gradients through diagonal preconditioner, and resulting in updates in the Mi. form of 20, = -17 Vic+"" where = 52 + (1 - -2)99
AdamW (Loshchilov & Hutter, 2019) further introduces weight decay to Adam.
Recent studies have provided theoretical support for scaled gradient descent methods under different preconditioning strategies. The core idea is to adjust both the direction and magnitude of updates by introducing a scaling matrix to gradients. Tong et al. (Tong et al., 2021) demonstrate the local convergence of scaled gradient descent methods. Jia et al. (Jia et al., 2024) extend this work by proving global convergence of scaled gradient descent for the leaat-squares matrix decomposition problem || - /2 showing that this approach achieves global convergence under dif- ferent condition numbers. Other variants of scaled gradient descent have also emerged, such as Zhang al. who pro- posed two regularization strategies (Zhang et al., 2023; 2024c). In higher-dimensional settings, scaled gradient descent has been further extended to tensor optimization (Tong et al., 2022; Ma et al., 2024). Mishra et al. (Mishra et al., 2013; Mishra & Sepulchre, 2016) also applied the principles of Riemannian to the optimization involving low- rank matrices. Considering the data's ' manifold geometry, a Riemannian metric 9p w) is introduced to guide gradient updates along the manifold. Recently, Zhang et al. (Zhang & Pilanci, 2024) introduced the idea of Riemannian precon- ditioners to LoRA by attaching an r X preconditioner to the gradients of low-rank matrices. As a result, they provide improved fine-tuning performance of LoRA, compared with conventional gradient optimizers such as SGD and AdamW.
2.2. Mixture of LoRAs
MoE has emerged as a critical framework for addressing complex tasks. By incorporating multiple expert modules, it dynamically selects appropriate experts based on specific inputs (Jacobs et al., 1991). Early studies, such as Lo- RAMoE (Dou et al., 2024) and MixLoRA (Li et al., 2024), have pioneered the introduction of the MoE-LoRA architec- ture by integrating LoRA experts for both global and down- stream tasks. Afterward, MoE-LoRA has demonstrated its effectiveness across a range of fields such as continual learn- ing (Dou et al., 2024; Yang et al., 2024), vision-language multi-model tasks (Gou et al., 2023; Chen et al., 2024), and multi-task applications (Liu et al., 2023).
Recent studies have focused on enhancing MoE-LoRA through architectural advancements and improved train- ing strategies. For instance, MoLA (Gao et al., 2024) allocates a varying number of experts at different lay- ers, and MixDA (Diao et al., 2023) introduces multiple domain-adaptive modules to support multi-domain knowl- edge. Other methods such as (Wu et al., 2024a; Liu et al., 2023; Wu et al., 2024b; Gou et al., 2023: Wang & Agar- wal, 2022) have also been proposed for strengthening MoB- LoRA. To boost the training of MoE-LoRA, Luo et al. (Luo et al., 2024) address the random routing issue by introduc- ing a contrastive loss (Shi et al., 2024a). At the same time, MoV (Zadouri et al., 2024) chooses to combine lightweight vectors with a sparse selection mechanism for efficient ex- pert allocation. Other approaches, including (Dou et al., 2024; Li et al., 2024; Zhu al., 2023), focus on load balanc- ing among experts. However, to the best of our knowledge, there is still a lack of work on gradient optimizing (Bian et al., 2024) specifically for MoE-LoRA models.
3. Method
We elaborate on our motivations and detail the modification we have made to the Riemannian preconditioning method specifically for MoE-LoRA. Our theoretical foundations and engineering solutions are also presented.
3.1. Riemannian Preconditioner in LoRA Expert
As a preliminary, we first briefly introduce the Rieman- nian preconditioner (Zhang & Pilanci, 2024). Suppose the pretrained model weight is W and its additive low-rank com- ponents B and A, let X W + BA denote the whole weight matrix and let L and 17 denote the loss function and
2.3. Gradient Preconditioners
In most deep learning cases (Zhang et al., 2024a; Luo et al., 2025a), gradient descent algorithms update model parame-
3

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
the learning rate, respectively. For the plain gradient de- scent method, the gradient updating process is described through Equation (1) to (4), in which the derivation from (2) to (3) relies on ignoring the second-order term of learn- ing rate. Obviously, BV aL + VBCA in (4) serves as an approximation of the ideal FFT gradient of X.
it not only brings gate value di for each expert into Equation (1)-(4), but also introduces extra gate value gi for each expert into (5), since the derivation chain rule V V. C = 9i V x A), T further clarify, we formally derive the whole result. Note that gate values are computed through a softmax with com- plex non-linear operations, thus we just treat them as con- (I) stants for an casier deriving approximation. Following the conventional Riemannian preconditioners in (6), we have:
and V VA. = g ".B,"(Vx (VxL). To
Xnew = W + Bnew Anew
+ (B - V (A-
(2)
W + -BA BA-1 BB BVA
(3)
N Nappert
X +
(4)
Xnew WW
N(B.-nVVB,Cy)) A.C.
Subsequently, according to the derivation chain rule and the simple fact that X W BA, we directly obtain that V V = (VxX)V
NEspert
2 X-1
Σ
g. A, 4 VB,AA,
B² (). and likewise BL = (VxC)AT. Thus, (4) can be transformed to:
NEspert
X
B,B,(BB, (B), A.C
Xnew= X - 1nnnnBBB(( (Vx (VxC)A"A (5)
+ (VB,C(A,A A.]
(9)
which actually updates the model in different direction compared to the FFT update formula Xnew = X - x.. This phenomenon occurs since the distorted sub-space of X constructed by BA brings inconsistency between the optimal gradient descent within its manifold and that of the full matrix X. To address this inconsistency, Zhang et al. (Zhang& & Pilanci, 2024) scale the gradients of A and B by:
N Magprrt
X Σ
B. BB (VxL)
+ + (V
N Naspart
X-n
Σ
g Brojjot(B.)((x)))
VAC = (BT
Naspert
V -+
(6)
9i 9 " (1)) (V Xx
(10)
so that (5) is expressed as:
in which the derivation step (9) denotes the conventional Riemannian preconditioner scaling. It should be interpreted that (10) consists of an ensemble of projections of the full matrix gradient onto the row spaces of A experts and the column spaces of B experts.
Xnew = X - B
+ (AV=))
(7)
= X 1/ (ProjcoL(B) (Vx2)T
+ (roj)oo(A))(X44)
3.2. Rescaling Preconditioners
where the update inside the manifold is performed according to the full matrix gradient projection onto the row space of A and the column space of B. Therefore, it better approxi- mates fully fine-tuning than the unscaled descent step.
Equation (10) presents a squared-value weighted sum of an ensemble of gradient projections. Generally, more acti- vated experts lead smaller per-expert gate values and lead to more reduced assembled gradient; On the other hand, more balanced experts also lead to a more re- duced assembled gradient since the basic inequality theorem
Inspired by this work, a straightforward way to expand their solution to MoE-LoRA is to individually scale the gradient of each LoRA expert by (6). However, equation X = W + BA lays out in a different form in MoE-LoRA:
Σe x;² > =;)² 1 satisfies its equality condition when T;s are equal. As a result, the gradient of the full matrix X will be underestimated due those squared gate values. From the perspective of manifolds and curvature, we explain that by considering di in (8) as manifold scaler, which reduces the size of B,A, so that would probably in- crease its curvature. However, the conventional Riemannian preconditioner failed to take the manifold scaler g, into con- sideration, since is designed for a single LoRA adapter.
n
Narpert
==W Σ B.
(8)
where Nxppert denotes the number of activated experts and gi denotes the gate value of specific expert i. As a result,
4

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
To alleviate this squared issue, we assume a further rescaling step for the Riemannian preconditioners:
its square root v during model forwarding. Consequently, Equation (12) can be achieved only under the precondition- ers of (6), because the quadratic terms of gate values g in Equation (10) are now naturally become linear terms 9s.
VA.C= -B."B.)-IVA.
91
(11) Replacing 9i by vge simultaneously introduces destruction to forwarding, as the sum of square roots does not equal 1. One possible solution is to re-normalize those square roots to be summed up as 1. However, brings inconsistency between the assigned weights of experts during forwarding and backwarding. Therefore, we propose another strategy to accommodate both aspects, which is manually assigning optimizable and unoptimizable components of Equation (8), to satisfy the requirements of both forwarding in (8) and backwarding in (12). During the forwarding process, the proposed strategy simply expressed by:
VB C V (BAA()
gi
which is introduced replace (6) in the derivation of Equa- tion (9), to eliminate the variable 9; and keeps only first power of g. in the final equation (10). Throughout this trans- formation, the final ensemble of multi-expert projections shares an equivalent scale with the projection of single LoRA adapter, shown in Equation (12). Therefore, training of an MoE-LoRA will be alleviated from under-estimation.
Nasport
Xnew = -n
n
g (.PrOjeatB.)(Xx("
X = W +
NRapart
Vg. (9: Vo.) (13)
N Naspert
9i (VxL).
(12)
where we define the hat symbol as an operation of gra- dient detaching. Specifically, a á denotes that the variable p does not require gradient, which also means p should be detached from gradient tracking along the whole neural network. By decomposing optimizable and unoptimizable components like this, low-rank matrices A and B are able to be optimized following (12) (see Appendix E for detailed derivations), while the equivalent behavior and the same result of conventional module forwarding are still preserved. Moreover, by maintaining the optimizable gi terms in for- warding and treating all the vg. as constants that are not subject optimization, (13) also keeps the conventional training behaviors of gates. Additionally, this modification introduces only minimal overhead to the original forward computation process.
We claim that (12) further approaches global fully finetun- ing because of two basic reasons: Firstly, it is derived by implementing Riemannian preconditioners to calibrate each LoRA expert's gradient (given by (6)), thus ensuring each LoRA expert can get close to their respective local full-rank training behavior (i.e., per-expert fully- finetuiign equiva- lency), according to Zhang al. (Zhang& & Pilanci, 2024); Secondly, since there exists a further distortion of each ex- pert space introduced by their respective gate value, leading to an inconsistency between per-expert local optimals and the global optimal. Therefore, (11) further introduces a re- spective gate value Ii as a re-scaler to each expert's Rieman- nian preconditioner to relieve the expert distortion resulting from the multiplication of gate value during forwarding. De- rived by (11) and (9), (12) is achieved. As result, (12) can further approach global fully-finetuning equivalency. For example, larger gate values introduce less distortion. Thus, through (12) experts with larger gate values are re-scaled less than those with smaller ones.
4. Experiments
We present a series of comparative experiments to evalu- ate the performances of MoE-LoRA across various down- stream tasks (Zhang al., 2024b; Luo al., 2025b) in- cluding Question Answering, the GLUE Benchmark, and the Vision-Languee task. Specifically, two types of experi- mental candidates are mainly involved in our experiments: (1) MoE-LoRA with experts updated independently using Riemannian scaled optimizer; and (2) MoE-LoRA updated using Riemannian scaled optimizer, plus incorporating our proposed rescaling technique (the engineering approxima- tion). We implement both of them SGD and AdamW W optimizers respectively. As further reference, we also ex- hibit our comparisons and possibility of integrations with previous MoE-LORA baselines, such as MoLA (Gao et al., 2024). Finally, to lend support to our theoretical foundation, we conduct an ablation study by assessing our forwarding
3.3. Engineering Approximation
Although Equation (11) provides an approach to eliminate under-estimation for MoE-LoRA, it is unrealizable since each LoRA module exists a respective gi for every single token of every batch sample. Actually, during the training, backpropagation always runs after averaging all the losses of each single token of each sample in batch. Thus, it is impossible to reconstruct and rescale the respective gradient contributed by each single token when we optimize a LoRA module. Alternatively, we design an engineering approxima- tion to (1 and (12), by replacing each gate value 9: with
5

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Table Question answering evaluations across four QA datasets with IJama-3.2-30 the foundation model. Our gate-based rescaling methodology outperforms conventional Riemannian preconditioned optimizers, in terms of both SGD and AdamW. Each pair of comparing candidates is trained through the same steps until they both achieve good stable performances.
ScienceQA
CommonsenseQQ
OpenBookQA
SIQA
avg.
RSGD20,10.
62.8
52.4
53.2
65.7
58.5
g 20,
70.1
55.4
59.8
68.5
63.5
R RAdamW20,10.4
82.6
67.7
70.4
81.5
75.6
gRAdamm 22,,10,4 10,4
83.8
68.2
72.4
82.3
76.7
R50D Lass
PASE Ve wwiiiie Loi
4F500 viki 1896
ROOD and Actuni
(8000 lnd Aoceney
CommansarGeoA
020
SIGA
-00
8.5
415
84
04
24
803
-05
202
562
200
Figure 2. Converging Performances of RSSDD20.100 and R2.10.. MoE-LoRA with Llama-3.2-3B as the foundation model. The x-axis represents training steps, the left y-axis in each figure represents the training or validation losses, while the right y-axis in each figure represents the accuracy metrics of test sets. Before implementing our gate based rescaling method, the training and validation losses RSGD optimizer across four tasks are significantly reduced around 100-200 steps, while after implementing our method, they are significantly reduced earlier around 0-100 steps.
revisions only under classic optimizer without Riemannian preconditioners support.
those candidates integrated with our gate- gate-based rescaling ap- proach are denoted as gRSGGD.,,, and y Adamm Wa,eer respectively, in which the front g represents that we rescale the gradient by gate values.
4.1. Experimental Setup
For most experiments, unless otherwise specified, we con- struct mixture of LoRAs modules with a total of 20 experts, a rank of 4 for each expert, and a selection of top-10 experts activated each time. Furthermore, a range of other architec- tural MoE settings are also discussed in the ablation section. We perform experiments based on Llama-3.2-3B (Touvron et al., 2023), GLM-4-9B (Zeng et al.. 2024), and LLaVA- v1.5-7B (Liu et al., 2024a) the foundation models. During training, we follow linear decay learning-rate scheduler. We assign a relatively smaller learning rate to gate module compared to other trainable components, to achieve a stable training behavior. The reduced learning rate for gate helps to prevent model from experiencing abrupt and erratic routing changes. For further stabilization, we also cap its maximum gradient norm at 1.0. We carefully assign different initial learning rates for various tasks, trying to ensure all models achieve their best performances in a capable running time.
4.2. Question Answering Evaluations
We evaluate our proposed method on several question- answering benchmarks, including ScienceQA (Lu et al., 2022), CommonsenseQA (Talmor et al., 2019), Open- BookQA (Mihaylov et al., 2018) and SIQA (Sap et al., 2019). These question-answering datasets encompass a di- verse range of domains and types, such as science, social interactions, common sense, and open-book exams, etc. We implement all the experimental candidates based on Llama- 3.2-3B as their foundation model. For the SGD optimizer, we set an initial learning rate to x 10 5 for every LoRA ex- pert; For the Adam optimizer, we utilize an initial learning rate of 1 x 10-5, We run through all the experiments until they are stabilized at stable performance, and especially ensure that each pair of comparing candidates (i.c., inde- pendently Riemannian preconditioned MoE-LoRA, and that with our proposed rescaling approach) are trained through the same steps to make sure they are fairly comparable. Specifically, depending on the complexity of datasets, we choose from two settings, 800 or 1,400 steps, for all the QA evaluations, except that of R RAdamW and AodggRAdde RAAdaw
We denote the number of experts, top-k, and the per-expert rank as n, k respectively; For experimental candidates using conventional Riemannian preconditioned optimiz- ers, we denote them as RSGD,,krr and RAdamW,, Wr.k.r in which the front R represents the word Riemannian; While
6

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Table GL.UR Benchmark evaluations across nine tasks with LJama-3.2-31 and GI-M-499 as the foundation models. Our gate-based
rescaling method contributes an overall improvement over GLUE Benchmark, in terms of Riemannian preconditioned SGD and Adam W.
CoLA
SST-2
MRPC STS-B
QQP
MNLI
QNLI
RTE
WNLI
ayg.
R2,,.Do0,1.444
48.25
93.58
73.57
48.15
80.22
62.42
78.39
62.45
39.44
65.16
Llama 3.2
gRSGD2o.100
55.86
94.95
76.29
56.70
83.48
80.37
83.42
72.92
52.11
72.90
(3B)
R.AdamW 20,100
65.94
96.56
70.03
60.12
85,36
81.36
91.55
53,95
45.07
72.22
g 2.AAdam22.000
67.38
96.67
81,57
61.08
86.35
81.59
91.32
55.75
47.89
74.40
20
62.33
91.74
82.32
63.06
85.36
81.67
90.27
62.59
76.05
77.27
GLM 4
g gRSGD20.10.
62.97
95.18
81.68
63.68
86.84
86.27
91.17
80.21
77.46
80.61
(9B)
R.Adam220. 20,10.4
62.54
45.53
79.30
64.15
88.72
88.03
91.58
89.56
84.51
77.10
gR.dam
62.25
46.22
79.59
64.36
88.82
88.19
91.66
91.37
85.92
77.60
on CommonsenscQA, which we train up to 2,000 steps to achieve a more clear distinction between two comparable candidates. We present our evaluated performances in Ta- ble 1. It is observed that: (1) Riemannian preconditioned Optimizers incorporating
Table 3. Visual7W and VMCBench performances after trained for 1,000 steps, with VALv VMCBench, we use 100 samples to evaluate, thus the accuracy will be at most two-digit decimal. That's why we list all numbers in percentage here for more comfortable present.)
the foundation model. (For
approach achieve better per- formances for every QA benchmark, albeit with varying degrees of improvement: (2) Overall, we exhibit more con- tribution to Riemannian preconditioned SGD than to that of AdamW: We improve the performance of RSGD by around 8.5%, while we improve R.AdaWW by around 1.5%.
Visual7W
VMCBench
RSG
68%
59%
gRSGD20.10
%
70% %
R.Adam
77%
75%
gRAdamW20.10 78%
75%
Besides our improvements in final performances, we also witness boost in terms of converging speed under our op- timization. To clearly display this, we plot desreascgg curves and metric variations of the four question-answering datasets under SGD optimizer in Figure 2. It is clearly shown that gRSGD converges faster than RSGD, in terms of training and evaluating losses as well accuracy metrics.
Subsequently, we extend experiments a larger foundation model, GLM-4-9B, Since the 9B model is more powerful in few-shot learning, for some datasets such as SST-2 etc., set lower learning rates such as 3 10-6 and 1 x 10-6 re- spectively for SGDs and Adam to make sure clear loss decreasing period can be witnessed. We train for the same number of steps for each pair of competitive candidates. Table also illustrates the performances of training MoE- LoRA through different optimizing strategies with GLM- 4-9B. Results still witness our overall outperformance. In particular, we improve the average performance of RSGD by around 4.3%, and that of RAdamW by around 0.7%.
4.3. Performance on GLUE Benchmark
To comprehensively examine our effectiveness, we perform series of downstream evaluations on the benchmark of GLUE (Wang al., 2019), which is collection of resources for evaluating model performances natural language un- derstanding. We first run through all the evaluations in GLUE with L.ama-3.2-3B as the foundation model and present the benchmark results in Table 2. For most SGD ex- periments we set an initial learning rate for LoRA experts as 3 x 10 except WNLI for which we set its initial learning rate 3 x 10-6, For AdamW experiments we choose initial learning rate from {3 10-3, 3, x 10-5). For most datasets, we train for 2,000 steps, excluding some AdamW experiments in which we perform an early stop at around 1,000 since they appear to be converged or even overfit- ting- Table illustrates our effectiveness across various downstream applications as well as the overall assessment under Llama-3.2-3 3.B..B. In terms of overall performances, our approach improves RSGD and R.damW by 11.9% and 3.0% respectively.
4.4. Performance on LLaVA
Beyond pure textual tasks (Bi et al., 2025), vision-language cross-modal tasks have garnered increasing attention in re- cent years, witnessing the emergence of notable achieve- ments such as LLaVA, CLgVLM, etc. (Chen et al., 2024; Wang et al., 2024c; Ge et al., 2021; Jin et al., 2024) Thus, we further evaluate our gate-based rescaling approach in computer vision (Feng et al., 2022: 2025). Specifically, implement an MoE-LoRA architecture for the well- known vasion-language foundation model, LLaVA-vI. 5- 7B (Chen et al., 2024). We introduce trainable MoE-LoRA adapters into both visual and textual modules of LLaVA- vl.5-7B. For evaluation, Visual7W (Zhu et al., 2016) and VMCBench (Zhang et al., 2025b) datasets are employed,

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
which both consist of multimodal samples each containing multiple-choice question paired with related image. The question can be answered through understanding the pro- vided image. Visual7W is a subset of Visual Genome (Kr- ishna et al., 2017) dataset, while VMCBench is a benchmark created from 20 existing VQA datasets. For VMCBench, we only use their dev set since their test set not labeled.
5.0
SGD Training Loss
4.9
g Training Loss
RSGD Training Loss
4.8
gRSGD Training Loss
4.7
We take 900 of all the 1,000 labeled samples as training sam- ples, while the rest 100 are for evaluation. Table 3 exhibits the results of all experimental candidates. Our approach consistently demonstrates visible improvements, especially for SGD.
Loss
4.6
4.5
4.4
4.3
4.5. Compare and Integrate with MoF-LLRA Baselines
4.2
We then compare and integrate our method with existing MoE-LoRA baselines. We provide our comparisons with two baselines: (1) The pure mixture of LoRAs (Liu et al., 2023), which we denote as MoELoRA and use token-level routing: (2) MoLA (Gao et al., 2024), which is MoE-LoRA variant specifically focusing assigning different numbers of experts to different layers, and proving that higher lay- ers need more LoRA experts. It should be noted that our proposed gate-based rescaling approach can be integrated with most MoE-LoRA variants since they are not in conflict. Take MoLA as an example, we can integrate our method with MoLAA by implementing model with more experts in its higher layers and trained through Riemannian precondi- tioners and gate-based rescaling approach. We reproduce MoELoRA A and Mol.A, implement the integrations, and il- lustrate their performances in Table 4. We use Llama-3.2-33 as the foundation model and follow MoLA's configurations here, which means we set the per-expert rank to 4, top-k to 2, and the total number of experts of all layers to 140. this way, MoELoRA and our method assign experts to each layer, while MoL.A assigns 2, 4, 6, and 8 experts respectively to the bottom, lower middle, higher middle, and top layers. In Table 4 we denote this special assignment strategy as (2,4,6.8), while the average assignment is (5,5,555) where each digit covers seven layers under L.ama-3.2-3B. We still provide enhancement in the context of MoLA architecture.
0
40
60
100
Step
Figure 3. Curves of ScienceQA training losses under the optimiza- tion of conventional and Riemannian preconditioned SGDs, and also both integrated with the gate-based rescaling approach. Llama- 3.2-3B serves as the foundation model.
antee a projection of full matrix gradient in low-rank space, it should be trivial for them to normalize the sum of expert gradients by replacing 9. with vgi. To confirm this, we conduct an ablation study by integrating our gate-based revi- sion with conventional un- pr-cooonndiio SGD optimizer. The loss loss-decreasing curves shown in Figure 3 illustrate that applying our approach directly on a pure SGD optimizer does not provide help, which oppositely demonstrates our refinement is highly coupled with the Riemannian precondi- tioning algorithm.
Various MoE architectures. To demonstrate that our pro- posed approach can be generalized to various settings of LoRA mixtures, we construct different MoE-LORA archi- tectures for further exploration, including the variations in the numbers of experts, per-expert ranks, and the number of top-k. Specifically, we test seven structural conditions on the ScienceQA dataset, and all candidates are trained for 800 steps using the same initial learning rate. Table 5 ex- hibits the results, showing we are able outperform across most circumstances in terms of MoE structure. Moreover, it is also observed from SGD performances that, variations in expert numbers or per-expert ranks introduce limited im- pacts on our effectiveness, while larger top-k roughly exhibit higher boosts. This observation aligns with our theoretical analysis. which suggests a larger number of activated ex- perts results in more reduced per-expert gate values, thereby leaving a larger margin for our revision to take effect.
4.6. Ablation Study
Theoretical Dependence. Although our proposed approach grounded in the context of Riemannian preconditioners, is important to note that our engineering implementation does not inherently require coexistence with Riemannian preconditioners. The reason is that our modifications are solely focused on altering the forward propagation con- ventions of MoE-LoRA. This consequently raises a vital question about the standalone efficacy of our modifications in enhancing MoE-LoRA's performance, without depending on the Riemannian preconditioning context. Ideally, since the conventional un-preconditioned optimizer does not guar-
5. Conclusion
We introduce the Riemannian gradient preconditioners to train a mixture of Low-rank Experts (MoE-LoRA), Instead
8

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Table Baselines Comparison and Integration. The first three lines provide comparisons between pure MoF--LoRA (MoFLoRA-SGH) MoLA (MoLA-SGD). and our sate-rescaled Riemannian preconditioning method (MoELoRA-ggAtGGD) The last two lines provide MoLA integrated with conventional (MoLA- -)GGD) and gate-rescaled preconditioning methods (MoLA- respectively. candidates are trained using SGD optimizers for to 2,000 steps.
Experts
ScienceQA
CommonsenseeQA
OpenBookQA
SIQA
avg.
MOELORA-SGD
(5,5,5,5)
54.68
48.90
48.40
57.92
52.47
MOELoRA-gRRSGD (Ours)
(5,5,5,5)
70.01
54.80
63.60
64.45
63.22
MoLA-SGD
(2,4,6,8)
54.99
49.20
52.80
58.94
53.98
MoLA-RSGG
(2,4,6,8)
68.03
53.90
59.80
64.08
61.45
goLG- (Ours)
(2,4,6,8
70.46
56.30
64.00
64.90
63.92
Table 5. Accuracies and boosts of ScienceQA for conventional and gate-rescaled Riemannian optimizers under various MoE architectures. Llama-3.2-3B serves as the foundation model.
n/ n/k/r
RSGD
gRSGD
Boost
RAdamW
gRAdamW
Boost
5/5/4
65.78
71.31
8.41%
76.67
78.19
1.98%
8/5/4
65.11
69.96
7.45%
78.69
79.45
0.97%
10/5/4
64.34
69.33
7.76%
79.05
80.31
1.59%
10/5/2
72.03
78.06
8.37%
79.63
80.13
0.63%
10/5/1
79.95
87.28
9.17%f
80.71
81.11
0.50%
10/10/2
68.62
77.16
12.45%
77.47
77.65
0.23%
10/2/2
79.68
81.16
1.86%f
83.63
83.45
-0.22%
of directly attaching Riemannian preconditioners to each expert's gradient for pursuing local optimality, we claim that multiplying expert A; by its respective gate value şi dur- ing forwarding leads to further rescaling of the manifold constructed by expert i. To alleviate this, Riemannian pre- conditioners designed for MoE-LoRA shall be revised to in- corporate gate values. To approximate this concept, we pro- pose an engineering solution that decomposes forwarding variables into optimizable and un-opiimizable components. Experiments across various downstream tasks demonstrate performance improvement over conventional Rieman- nian preconditioners. Ablation studies further demonstrate our theoretical foundation and universality. We claim that, our work can be applied to the fields like efficient and low- resource model training, continual or multi-task learning, stablized training and modular task adaptation, etc.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
References
Bi, W., Kou, F., Shi, L., Li, H., Chen, J., and Xu, M. Leveraging the dual capabilities of llm: LIm-enhanced text mapping model for personality detection. In AAAI, 2025.
Bian, A., Li, W., Yuan, H., Wang, M., Zhao, Z., Lu, A., Ji, Feng. T., al. Make continual learning stronger via c-flat. NeurIPS, 2024.
Chen, S., Jie, Z., and Ma, L. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv e-prints, pp. arXiv-2401, 2024.
Acknowledgments
This work has been supported by the National Natural Sci- ence Foundation of China (NSFC) for Distinguished Young Scholar under Grant 62425601, the NSFC under Grant 62406036, the National Key Research and Development Program China under Grant 2024YFC3308500, the State Key Laboratory of Networking and Switching Technology under Grant NST202501 10, the New Cornerstone Science Foundation through the XPLORER PRIZE, and also spon- sored by SMP-Zhipu.AI Large Model Cross-Disciplinary Fund under Grant ZPCG20241029322..
Diao, S., Xu, T., Xu, R., Wang, J., and Zhang, T. Mixture- of-domain-aue -ddaaii-aaadpters: Decoupling and injecting domain knowledge to pre-trained language models' memories. In 6/st Annual Meeting of the Association for Computa- tional Linguistics, 2C23 pp. 5113-5129. Association for Computational Linguistics (ACL), 2023.
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., al. Loramoe:
9

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1932-1945, 2024.
Jin, H., Zhang, Y., Shi, L.., Zhang, S., Kou, F., Yang. J., Zhu, C., and Luo, An end-to-end graph attention network hashing for cross-modal retrieval. Advances in Neural Information Processing Systems, 1112106-2122 2024.
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. arXiv e-prints, pp. arXiv-2312, 2023.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7). 2011.
Kingma, D. and Ba, Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Confer- ence Track Proceedings, 2015.
Feng, T., Wang, M., and Yuan, H. Overcoming catastrophic forgetting in incremental object detection via elastic re- sponse distillation. In CVPR, 2022.
Feng, T., Li, W., Zhu, D., Yuan, H., Zheng, W., Zhang, D., and Tang, Zeroflow: Overcoming catastrophic forgetting is easier than you think. ICML, 2025.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.
Ju, J., Ge, X., Xin, X., Karatzoglou, A., Arapakis, I., Wang, J., and Jose, J. M. lisan: Efficiently adapting multimodal representation for sequential recommendation with de- coupled peft. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 6877697,40 2024.
Li, Ma, Y., Wang, N., Cheng, Z., Duan, L., Zuo, J., Yang, C., and Tang, M. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. CoRR, 2024.
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang, J., and Subrahmanian, V. Higher layers need more lora experts. arXiv e-prints, pp. 2402, 2024.
Lin, Y.. Ma, X., Chu, X., Jin, Y., Yang. Z., Wang, Y., and Mei, H. Lora dropout sparsity regularizer for overfit- ting control. CoRR, 2024.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024a.
Ge, X., Chen, F., Jose, J. M., Ji, Z., Wu, Z., and Liu, X. Structured multi-modal feature embedding and alignment for image-sentence retrieval. In Proceedings of the 29th ACM international conference on multimedia, pp. 5185- 5193, 2021.
Liu, Q., Wu, X., Thao, X., Zhu, Y., Xu, D., Tian, F,, and Zheng, Y. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. CoRR, 2023.
Gou, Y., Liu, Z., Chen, K., Hong. L., Xu, H., Li, A., Yeung, D.-Y., Kwok, J. T., and Zhang, Y. Mixture of cluster- conditional lora experts for vision- languale instruction tuning. arXiv e-prints, pp. arXiv-2312, 2023.
Liu, S., Wang, C., Yin, H., Molchanov, P., Wang, Y. F. Cheng, K., and Chen, M. Dora: Weight-dccomposed low- rank adaptation.
Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning, pp. 17783-17806. PMLR, 2024.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.- Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Jia, X., Wang, H., Peng, J., Feng, X., and Meng, D. Precon- ditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent. Advances in Neural Information Processing Systems, 36, 2024.
Luo, H., Chen, G., Zheng, Y., Wu, X., Guo, Y., Lin, Q., Feng, Y., Kuang, Z., Song, M., Zhu, Y., et al.
10

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Hypergraphrag: Retrieval-augmented generation with hypergraph--truucurred knowledge representation. arXiv, 2025a.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4153 2019.
Luo, H., Guo, Y., Lin, Q. Wu, X., Mu, X., Liu, W., Song, M., Zhu, Y., Tuan, L. A., et al. Kbqa-ol: Agentic knowledge base question answering with monte carlo tree search. arXiv preprint arXiv, 2025b.
Tian, C., Shi, Z.., Guo, Z., Li, L.., and Xu, C.-Z. Hydralora: An asymmetric lora architecture for efficient fine-tuning, Advances in Neural Information Processing Systems, 37: 9565 9584, 2024.
Luo, T., Lei, J., Lei, F., Liu, W., He, S., Zhao, J., and Liu, K. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv e-prints, pp. arXiv-2402, 2024.
Tong, T., Ma, C., and Chi, Y. Low-rank matrix recovery with scaled subgradient methods: Fast and robust convergence without the condition number. IEEE Transactions on Signal Processing, 69:2396-2409, 2021.
Ma, C., Xu, X., Tong, T., and Chi, Y. Provably accelerating ill-conditioned low-rank estimation via scaled gradient descent, even with overparameterization. Explorations in the Mathematics of Data Science: The Inaugural Volume of the Center for Approximation and Mathematical Data Analytics, pp. 133-165, 2024.
Tong, T., Ma, C., Prater-Bennette Bennette. A., Tripp. E., and Chi, Y. Scaling and scalability: Provable nonconvex low- rank tensor estimation from incomplete measurements. Journal of Machine Learning Research, 23(163):1-77, 2022.
Meng, F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-12107 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv e-prints, pp. 23077 2023.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391 2018.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Mishra, B. and Sepulchre, R. Riemannian preconditioning.
SIAM Journal on Optimization, 26(1):635-660, 2016
Mishra, B., Meyer, G., Bach, F., and Sepulchre, R. Low- rank optimization with trace norm penalty. SIAM Journal on Optimization, 23(4):2124-2149) 2013.
Wang, H., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter- efficient LLM finetuning. In Chiruzzo, L., Ritter, A.. and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL 2025 Volume 1: Long Papers, Al- buquerque, New Mexico, USA, April 29- May 4, 2025, pp. 4823 4836. Association for Computational Linguisties, 2025a.
Qiang, R., Zhang, R., and Xie, P. Bilora: A bi-level op- timization framework for reerfittigg-resllentt low-rank adaptation of large pre-trained models. CORR, 2024.
Sap, M.. Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiga: Commonsense reasoning about social interac- tions. In Conference on Empirical Methods in Natural Language Processing, 2019.
Shi, L., Yang, J., Lv, P., Yuan, I.., Kou, F., Luo, J., and Xu, M. Self-derived knowledge graph contrastive learning for recommendation. In ACM MM, 2024a.
Wang, S., Chen, L.,, Jiang, J., Xue, B., Kong, I.,, and Wu, C. Lora meets dropout under a unified framework. In Findings of the Association for Computational Linguistics ACL 2024. pp. 1995-2008, 2024a.
Shi, S., Huang. S., Song, M., Li, 7., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., and Zhang, Q. Reslora: Iden- tity residual mapping in low-rank adaption. In Findings of the Association for Computational Linguistics ACL 2024b.
Wang, S., Yu, L., and Li, J. Lora-ga: Low-rank adaptation with gradient approximation. Advances in Neural Infor- mation Processing Systems, 77:54905-54993 2024b.
2024, pp.

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Wang, W. Lv, Q. Yu, W., Hong. W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, I.., XiXuan, S., al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems. 37:121475- 121499, 2024c.
Zhang, D., Zhoubian, S., Hu, Z.., Yue, Y., Dong, Y., and Tang, Rest-mets*: Llm self-training via process re- ward guided tree search. In The Thirty-eight Conference on Neural Information Processing Systems, pp. 64735- 64772, 2024b.
Wang, Y. and Agarwal, S. Adamix: dixture-of-adappaaie for parameter-efficient model tuning. In The 2022 Con- ference Empirical Methods in Natural Language Pro- cessing, 2022.
Zhang, D., Feng, T., Xue, L.., Wang, Y., and Tang, J. Purameter-efficient fine-tuning for foundation models. arXiv e-prints, pp. arXiv-2501, 2025a.
Zhang, F. and Pilanci, M. Riemannian preconditioned lora for fine-tuning foundation models. In Proceedings of the 41st International Conference on Machine Learning, pp- 1596699, 2024.
Wang, Z., Liang, J., He, R., Wang, Z., and Tan, Lora- pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Rep- resentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025b.
Zhang. G., Fattahi, S., and Zhang, R. Y. Preconditioned gradient descent for overparameterized nonconvex burer- monteiro factorization with global optimality certification. Journal of Machine Learning Research, 24(163):1-55 2023.
Wen, Z., Zhang, J., and Fang, Y. SIBO: A simple booster for parameter-efficien fine-tuning. Ku, L.,, Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thai- land and virtual meeting, August 11 11-6, 2024, pp. 1241- 1257. Association for Computational Linguistics, 2024. doi: 10. 1865/////22AFNNNNGSSASIII
Zhang. J., Zhang, R. Y., and Chiu, H.-M. Fast and accu- rate estimation of low-rank matrices from noisy measure- ments via preconditioned non-convex gradient descent. In International Conference on Artificial Intelligence and Statistics, pp. 3772-3780. PMLR, 2024c.
Wu, T., Wang. J., Zhao, Z., and Wong, N. Mixture-of- subspaces in low-rank adaptation. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 7880-7899, 2024a.
Zhang, Y., Su, Y., Liu, Y., Wang, X., Burgess, J., Sui, E., Wang, C., Aklilu, J., Lozano, A., Wei, A., et al. Au- tomated generation of challenging multiple-choice ques- tions for vision language model evaluation. arXiv e-prints, pp. 2i0111551,, 2025b.
Wu, X., Huang, S., and Wei, Mixture of lora experts. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b.
Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5000 2016.
Yang, S., Ali, M. A., Wang, C.-L., Hu, L., and Wang, D. Moral: Moe augmented lora for Ilms' lifelong learning. CORR, 2024.
Zhu, Y., Wichers, N., Lin, C.-C., Wang, X., Chen, T., Shu, L., Lu, H., Liu, C., Luo, L., Chen, J., et al. Sira: Sparse mixture of low rank adaptation. CoRR, 2023.
Zadouri, T., Üstün, A., Ahmadian, A., Ermis, B., Locatelli, A., and Hooker, S. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G.. Zhao, H., Lai, H., Yu, H., et al. Chatglm: A family of large language models from glm-1300 130b to glm-4 all tools. CoRR, 2024.
Zhang, D., Hu, Z., Zhoubian, S. Du, Z., Yang. K., Wang, Z., Yue, Y., Dong. Y., and Tang, J. Sciinstruct: a self- reflective instruction annotated dataset for training scien- tific language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
12

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
A. Various Model Sizes and MoE Architectures
To evaluate our method, we already conducted experiments on Llama-3.2-3B, GLM-4-9B, and LLaVA-vI.5-7B. We also presented our n/k/r analysis in Table 5 in Section 4.6, which consists of seven different candidates tested under SGD and AdamW optimizers with Llama-3.2-3B serving their foundation model. To make our investigation further sufficient, here add Llama-3.2-1B n/k/r configurations (16/8/4 and 10/5/4).
foundation model, and add two experiments under LLaVA-v1.5-7B with two different
Specifically, we conduct further experiments on JJama-3.2-IB for four QA benchmarks, each trained for 2,000 steps. To speed up the training, we set a relatively smaller MoE configuration which is 10/5/1. Results are illustrated in Table 6. For those two new n/ n/k/r configurations
LLaVA-v1.5-7B, we conduct their experiments on both Visual7W and VMCBench benchmarks under SGD and A optimizers. An overall enhancement of our method can still be witnessed under different configurations (especially for SGD), illustrated by Table 7.
Table 6. Question answering evaluations across four QA datasets with Llama-3.2-1 as the foundation model.
ScienceQA CommonsenseQA
OpenBookQA
SIQA
avg.
10.5
47.71
49,47
48.80
50.41
49.10
g ARSGD10.5.
49.87
59.30
54.00
57.06
55.06
RAdamW10,5.1
46.18
42.92
41.60
44.11
43.70
JRAdamW10.5.
46.58
43.82
43.40
45.50
44.83
Table 7. Visual7W and VMCBench Performances of -LaVA-v1.5-70 across various MoF. architectures.
n/k/r
Task
RSGD
gRSGD
R.AdamW
gRAdamW
Visual7W
71%
74%
76%
76%
10/5/4
VMCBench
63%
73%
76%
77%
Visual7W
72%
74%
76%
77%
16/8/4
VMCBench
59%
69%
71%
71%
B. Covergence Efficiency
In the main body of our paper, we illustrate the converging speed enhancements of our proposed approach, gRSGD, over the conventional RSGD through series of loss-decreasing plots. To further exhibit the comprehensive comparisons of convergence efficiency, we provide more results on GLUE benchmarks. In particular, for experiments conducted under Llama-3.2-3B as well as GLM-4-9B, we record metrics after the initial 100 training steps for each of the GLUE evaluations, as detailed in Table 8 and Table 9 respectively.
Table and 9 clearly demonstrate the superior convergence speed of our solutions over the conventional Riemannian preconditioned SGD optimizers. Nevertheless, they simultaneously illustrate an overall equivalent performance with trivial differences between our gate-based approach and the conventional Riemannian preconditioning method under AdamW optimizers. This indicates that our proposed approach is more valuable for SGD optimization. AdamW optimizers already present robust converging performances due to their adaptive gradient and learning rate mechanisms. As result, our global optimal approximation under AdamW W optimizing mainly contributes to the final optimality rather than significantly accelerating the initial gradient descending.
Table GLUE Benchmark evaluations after the initial 100 training steps conducted under llama-3.2-3B Our propesed gate-hased rescaling method contributes an overall converging speed enhancement over conventional Riemannian preconditioned SGD optimizers, while for Adam W optimizers, we provide a similar converging speed compared with the conventional Riemannian ones.
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
WNLI
avg.
RSGD20.10.4
22.52
91.63
49.97
0.00
62.22
36.27
58.90
56.68
1.41
42.18
g gRSGD2..10.
39.30
92.66
66.55
24.49
66.17
43.40
65.33
63.18
26.76
54.20
1.AdomW2o.10.
52.05
92.66
68.99
54.04
72.11
60.81
84.14
53.95
32.39
63.46
g @RAdamWao.10.
50.63
93.23
67.01
54.72
71.41
61.96
83.80
55.75
40.85
64.37
13

---

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Table GLUE Benchmark evaluations after the initial 100 training steps conducted under GL-M-499B. Our proposed gate-hased rescaling method still contributes an overall converging speed enhancement over conventional Riemannian preconditioned SGD optimizers, while for AdamW optimizers, we still provide a similar converging speed compared with the conventional Riemannian ones. (Note that for CoLA, SST-2, and MRPC, we utilize lower initial learning rate such as 3 10 and 1 x 10 while the others are 3 10 and
1 10-5. Therefore CoLA, SST-2, and MRPC converge much slower than others.)
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
WNLI
avg.
10
0.00
0.00
0.41
61.15
84.57
62.35
87.49
41.73
63.38
44.56
20.GD20.10.4
7.22
48.97
62.26
63.06
85.95
85.66
89.28
75.18
74.65
65.80
A4ddam220010..
17.37
0.00
0.00
63.34
86.15
0,00
89.05
89.21
80.28
47.27
g 2RAdamW20.10.1
16.75
0.00
0.00
63.54
86.05
0.61
89.34
91.37
78.87
47.39
C. Adam W Weight Decay Analysis
AdamW implements strategy called weight decay, which decays the trainable weights after each gradient update by 0, = 0, atO, Instead of the original Adam algorithm, AdamW separates the weight decay from the gradient update, which leads to better performance in some cases. To comprehensively prove the effectiveness of our gate-based rescaling method over Riemannian preconditioned Adam we evaluate our boosts across various weight decay factors x. Results are exhibited Table 10.
Table 10. ScienceQA boosting performances under I.ama-3.2-3B, across different Adam W weight decay.
0
le-5
1e-4
1e-3
RAdamWa2.nn
82.60
83.50
83.23
83.99
g R.A.dmmWa NaatmmWoo.rr 10,4
83.80
84.58
84.98
84.67
Boost
1.45%f
1.30%f
2.10%f
0.81%f
D. Multi-Task Performance
One of the most valuable features of MoE architectures is their capability of modeling multiple tasks. Through gating mechanism, the MoE system adeptly delegates specific tasks individual experts, thereby facilitating more focused and efficient learning process within each expert module. As result, one question arises regarding proposed sate-based rescaling approach: Can it still effectively augment the performance of MoE architectures in multi-task scenarios?
To illustrate this, we manually construct two mixed datasets, each consisting of three irrelevant natural language tasks from the GLUE Benchmark. The first mixture consists of CoLA, SST-2 and MRPC tasks, which serves as multi-task scenario involving both grammar checking, sentiment classification, and equivalent sentences judging; The second mixture consists of STS-B, QQP and QNLI tasks, which serves another multi-task scenario involving both sentence similarity scoring, equivalent questions judging, and inrestion-answering NLL. For evaluation, we test candidates each of the tasks individually and then average per-task performances within the mixture the overall evaluation for that mixture.
For sufficiently assessing the multi-task performance of our proposed gate-based rescaling method, we conduct experiments under two different MoE configurations, i.e., 20/10 /1// and 10/5/4. We train each candidate for 2,000 steps under R.Adam and g gRAddmmW (RAdamW with our proposed gate-based rescaling method) optimizers. Results are exhibited in Table 11. Our proposed method still effective to boost feature learning under multi-task scenarios
Table 11. Conventional and gate gate-rescaled optimizers performed on two mixed datasets consisting of tasks from the GLUE Benchmark. All candidates are trained 2,000 steps. Our gate-hased rescaling method contrihutes enhancements.
Mixture
n/k/r
R.damW
gRAddmmW
20/10/4
70.15
71.39
CoLA + SST-2+ MRPC
10/5/4
71.64
72.13
20/10/4
74.61
75.74
STS-B + QQP + QNLI
10/5/4
74.81
75.36
14

---

Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
E. Backward Derivation of the Engineering Strategy
We further elaborate why we implement (13), which our engineering approximation for achieving (11) and (12). Since by forwarding as (13), the gradient updating process of X can be derived as the following (Similar as previous, we treat gate value gis as constants when focusing on the gradients of A,s and B(s):
Espprri
X =W +
:V - (g: B;A,
==1
Maapart
NRapert
=(W +
VonB.A + (g. Vo)).AA) - Σ
(V
NExprrt
=X q
V
Naspart
N Naapent
=X 1
E 1=1
(Vo))² Projeot(B) (VxL)* - E
(Vii)² Projrou(AA) (Vx
(same as the derivation of (10))
1=1
NExpert
NExpert
=X 7
(roroooot(B)) (Vx -
Σ (VPr
so that (12) can be achieved.
The advantages of implementing (13) can be elaborated from two aspects: Firstly, it achieves (12) when conducting gradient updating of X while still keeps the original behavior of training gates, because it holds the same gate gradient, v = A;" B; as normal forwarding; Secondly, it provides equivalent behavior and the same result of normal module forwarding X- W + Σ Li=l
Nexpert
gi B.A, and only requires relatively low overhead.
F. Method Implementation
The engineering alternative solution of the gate-based rescaling approach is to manually separate the forwarding into optimizable and unoptimizable components. Here we provide our implementation in Python-like pseudocode. We only update two lines of the original MoE-LoRA code.
Algorithm Engineering Alternatiee Solution of Gate-based Rescaling Method
def (eear,, x, ...):
compute gate values gvs
execute each activated expert
for exp_id id aetivated_experts::
A = As
B - He gv g
exp_cut (x)
sqrt_ gv (g update
w e_exp_ord cut = gyveeppouts (gv-sttggtr xexp_ out. () # update 2
result result + x_exp_out
G. Experimental Details
We present our experimental details in Table 12. All experiments this paper follow this configuration unless they specify their partieular settings. For training steps, some of the experiments may converge earlier, therefore we perform an early
15

---

Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
stop for those experiments. We constrain the maximum of training steps by 2,000, considering it relatively fair setup for various downstream tasks, especially those with different scales of training corpora but in the same level of complexity.
Table 12. Default experimental details implemented throughout this paper. All experiments follow this configuration unless they specify their particular settings, like MoF structural experiments, baselines comparing experiments, and experiments of Adam weight decay.
SGD
Adam W
Train batch size (logical)
80 for textual tasks, 40 for dision-language tasks
Max training steps
<= 2, 000
QA, GLUE: 3e-5
QA: le-5
WNLI: 3e-6
MRPC, CoLA, QNLI, STS-B: 3e-5 SST-2, QQP, MNLI, WNLI: le-5
Initial for 3B (expert)
Multi-Taske 3e-4
RTE: le-6
Multi-Task: le-4
CoLA, SST-2, MRPC: 3c-6 STS-B, QQP. MNLI, QNLI, RTE, WNLI: 3e-5
CoLA, SST-2, MRPC: le-6
Initial Ir for 9B (expert)
STS-B, QQP, QNLI, RTE, WNLI: le-5
MNLI: 3e-6
Initial Ir for 7B (expert)
3e-5
le-5
Initial (gate)
3e-8
Lr scheduler (expert)
Linear
Warmup steps
Max gradient norm (gate)
1.0
Default LoRA expert rank
Default number of experts
20
Default activated Top-K
10
LoRA
16
LoRA dropout
0.05
Default weight decay
91,82.6
0.9, 0.999, 1e-6
16


Reviews:
[{"review_id": "Jlsm3Jm5sG", "reviewer": "Reviewer_76YT", "summary": "This work proposes an improved training strategy for MoE-LoRA, aiming to address the limited representation and suboptimal gradient issues when fine-tuning foundation models with plain MoE-LoRA. They first analyze the limitations of LoRA, including the insufficient representation capacity of low-rank matrices and gradient optimization problems. To enhance the representation power of LoRA, they introduce the MoE framework and then incorporate Riemannian Preconditioners to optimize the gradient update process. Through theoretical analysis and experimental validation, they demonstrate the effectiveness of the improved method in various downstream tasks, including question answering, language understanding, and vision-language tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your **acknowledgment on our innovations and theoretical value**. We’ve checked our paper again carefully to address any issues you mentioned. Here are our responses to your valuable concerns:\n\n**Response to W1 and W2** about the notations and abbreviations issues\n\nWe have carefully reviewed our paper again to address those undeclared or unclear notations and abbreviations. For example, as you mentioned, the symbol $X$ represents the overall weight matrix after integrating pretrained weights $W$ and LoRA modules $A$s and $B$s; $Proj_V(M)$ represents a projection function which projects a given matrix $M$ onto a sub space constructed by all vectors in set $V$. And when we treat all vectors in $V$ as the rows of another matrix, such as $P$, then the projecting action can be calculated as $Proj_{row(P)}(M) = MP^T(PP^T)^{-1}P$; FFN and FFT represent different concepts. FFN is the abbreviation for Feed-Forward Network; and FFT is the abbreviation for Fully Fine-Tuning. In our revision, we have addressed all the above notation issues, as well as some other unclear notation issues we found during our re-check.\n\n**Response to W3 and Q1** about more explanations of Eq.12 on its fully-finetuning equivalency\n\nEq.12 is the refined Riemannian-preconditioned backpropagating equation in MoE cases, after applying our proposed gate-based rescaling method. It further approaches global fully finetuning because of two basic reasons: Firstly, it is derived by implementing Riemannian preconditioners to calibrate each LoRA expert's gradient (given by Eq.6), thus ensuring each LoRA expert can get close to their respective full-rank training behavior locally (i.e., per-expert fully-finetuning equivalency), according to Zhang et al.[1]; Secondly, we notice a further distortion of each expert space introduced by their respective gate values, leading to an inconsistency between per-expert local optimals and the global optimal. Therefore, we further introduce a respective gate value $g_i$ as a re-scaler to each expert's Riemannian preconditioner (given by Eq.11), to relieve the expert distortion resulting from the multiplication of gate value during forwarding. As a result, Eq.12 can further approach global fully-finetuning equivalency (e.g., larger gate values introduce less distortion. Thus, through Eq.12, experts with larger gate values are re-scaled less than those with smaller ones). We have already integrated above discussions about fully-finetuning equivalency in our revision.\n\n**Response to W4** about insufficient $n/k/r$ analysis\n\nWe already presented our $n/k/r$ analysis in Table 5 in Section 4.6, which consists of seven different candidates tested under SGD and AdamW optimizers. Llama-3.2-3B serves as their foundation model. Table 5 already demonstrates our overall effectiveness across various $n/k/r$ configurations. To make the investigation further sufficient, we added two new experiments under LLaVA-v1.5-7B with two different $n/k/r$ configurations (16/8/4 and 10/5/4). Please refer to the results in our rebuttal to Reviewer 1wAK. An overall enhancement of our method can still be witnessed. \n\nWe have already emphasized our preliminary conclusion in Section 4.6 that, the value of $k$ is more important to performance boosting of our method under SGD optimizers. Now we provide a further analysis about both our performance boosting and the final overall performance: Firstly, for performance boosting, we calculate its correlation with $n$, $k$ and $r$ from Table 5, which are $0.357$, $0.912$ and $0.093$ respectively under SGD optimizers, demonstrating our preliminary conclusion is correct. While under AdamW optimizers they are $0.001$, $0.197$ and $0.806$, indicating that $r$ might be more important for boosting AdamW; Secondly, for final performance, we claim that it results from both our boosting effectiveness and the MoE fundamental features (such as a too-large MoE structure may lead to underfitting, while a too-small one may lead to overfitting). After the mixed influence of both aspects, the optimal configuration for LLama-3.2-3B in our ScienceQA experiments is the $10/5/1$ with our re-scaled SGD optimization; while for LLaVA-v1.5-7B in VMCBench experiments, for example, it is $10/5/4$ with our re-scaled AdamW optimization.\n\n**Response to your suggestion** of reviewing each derivation step\n\nThank you for your suggestion. We have reviewed all the derivation steps in our paper again to make sure each step is theoretically correct. \n\n[1] Zhang, Fangzhao, and Mert Pilanci. \"Riemannian preconditioned lora for fine-tuning foundation models.\" arXiv preprint arXiv:2402.02347 (2024).", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "Yes, I carefully reviewed the theoretical proofs presented in the manuscript, particularly focusing on the core contributions related to the improved MoE-LoRA training strategy.", "experimental_designs_or_analyses": "Yes, I have reviewed the soundness and validity of the experimental designs and analyses.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-grounded and significantly advance the broader scientific literature, particularly in LoRA, MoE, and optimization techniques for foundation models.", "essential_references_not_discussed": "All key related works are discussed in the paper, to the best of my knowledge.", "other_strengths_and_weaknesses": "***Strengths,***   \n**Innovation and Theoretical Value**: The introduction of the Riemannian Preconditioner into the MoE-LoRA is an innovative approach. This work addressed the instability issues encountered during the training of plain MoE-LoRA. Moreover, the manuscript provides a detailed theoretical analysis of the gradient update process in MoE-LoRA, revealing the underlying problems. And they further propose a gradient rescaling method based on gating values, which offers a solid theoretical foundation for their method.   \n**Experiments and Performance**: The method has been extensively tested across a variety of downstream tasks, including question answering, GLUE benchmark tests, and vision-language tasks, using different base models such as Llama, GLM, and LLaVA. The results demonstrate the effectiveness and generalizability of the proposed approach. The improved MoE-LoRA achieves significant performance improvements when using base optimizers.   \n**Practicality**: The manuscript introduces an engineering approximation method, which decomposes the optimized and non-optimized parts in the forward propagation.  This approach effectively resolves the difficulties associated with directly implementing the theoretical method, making it practical. For me, I'm intrigued by this section.   \n**Flexibility**: This work can be seamlessly integrated into existing MoE-LoRA baselines, such as MoLA. It can serve as a theoretical complement to current MoE-LoRA training strategies.   \n\n***Weaknesses,***   \n**Notations issues**: Some necessary notations and operators should be declared before use, even if they are commonly used conventions. For instance, symbols like X and Proj. should be defined.   \n**Unclear Abbreviations**: Abbreviations should be fully explained, especially those that may not be universally understood. For instance, FFN should be clearly defined. Additionally, what is the meaning of FFT? Might this be a small typo, I guess?   \n**More explicit conclusion** (Equation 12): As far as I understand, Equation 12 appears to be the core conclusion of this work. Therefore, it is crucial to provide a clear and detailed explanation of how and why Equation 12 can achieve full fine-tuning equivalency.  This will help readers better understand the core contribution of this work.   \n**Analytical glitches**: n/k/r seem to be of significant importance. However, the analysis of these parameters appears to be insufficient. A more thorough investigation is needed, and the optimal candidates should be emphasized.", "other_comments_or_suggestions": "It is commendable that this work offers a nice theoretical depth. They provided rigorous derivations to demonstrate that this work can achieve full fine-tuning equivalency in mathematical way. Additionally, they provided robust engineering implementations and alternative approximations, making this method both practical and scalable. However, given that the manuscript involves a substantial number of formulas and derivations, it is strongly recommended that the authors carefully review each step of the derivations to ensure their rigor and accuracy.", "questions_for_authors": "After carefully reviewing the theoretical section, although I understand how the proposed training strategy achieves full fine-tuning equivalency, it would be highly beneficial for the manuscript if the authors could provide a clearer explanation of how and why Equation 12 enables full fine-tuning equivalency. This would greatly enhance the readability and comprehension of the core contribution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTwsBzhFBW", "reviewer": "Reviewer_utbG", "summary": "This paper introduces a training strategy for Mixture-of-Experts (MoE) models with LoRA. It uses Riemannian preconditioning and gate-value scaling to address gradient sub-optimality and representation limitations. The proposed method modifies traditional preconditioners to stabilize gradient updates and improve training robustness. Experiments on NLP and VQA tasks, including QA datasets, the GLUE benchmark, and VG/VMCBench, demonstrate faster convergence and enhanced performance.\n\n## update after rebuttal\nThe authors have addressed my concerns, and l  would like to retain my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing **postive feedbacks on our presentations, derivations, and also experiments**. For the weaknesses and suggestions you pointed out, we highly value them and provide responses below: \n\n**Response to W1** about further explaning the convergence and fixing the issues in Figure 2\n\nWe have added in our revision further explanations for the converging figures in Figure 2. For example, the x-axis represents training steps, the left y-axis in each figure represents the training or validation losses, while the right y-axis in each figure represents the accuracy metrics of test sets; Before implementing our gate-based rescaling method, the training and validation losses of RSGD optimizer across four tasks are significantly reduced around 100-200 steps, while after implementing our method, they are significantly reduced earlier around 0-100 steps; In addition to converging speed, we also notice an out-performance of our method in terms of converged loss and QA accuracy; Finally, to address the axis overlapping issue, we re-arranged each subplot in the figure to be less tight with each other and re-drew the Figure 2 in our revision.\n\n**Response to W2 and W3** about further elaborating Eq.13\n\nWe further elaborate why we implement Eq.13 ($X=\\hat{W}+\\sum_{i=1}^{N_{Expert}}\\hat{\\sqrt{g_i}}B_iA_i+(g_i-\\hat{\\sqrt{g_i}})\\hat{B_i}\\hat{A_i}$), which is our engineering approximation for achieving Eq.11 and Eq.12. Since by forwarding as Eq.13, the gradient updating process of $X$ can be derived as the following (Similar as Eq.9 and Eq.10, we treat gate value $g_i$s as constants when focusing on the gradients of $A_i$s and $B_i$s):\n$$\n\\begin{aligned}\nX_{new}=&\\hat{W}+\\sum_{i=1}^{N_{Expert}}[\\hat{\\sqrt{g_i}}(B_i-\\eta\\nabla_{B_i}\\mathcal{L})(A_i-\\eta\\nabla_{A_i}\\mathcal{L})+(g_i-\\hat{\\sqrt{g_i}})\\hat{B_i}\\hat{A_i}] \\\\\\\\\n=& (\\hat{W}+\\sum_{i=1}^{N_{Expert}}\\hat{\\sqrt{g_i}}B_iA_i+(g_i-\\hat{\\sqrt{g_i}})\\hat{B_i}\\hat{A_i})-\\eta\\sum_{i=1}^{N_{Expert}}\\hat{\\sqrt{g_i}}[B_i(\\nabla_{A_i}\\mathcal{L})+(\\nabla_{B_i}\\mathcal{L})A_i] \\\\\\\\\n=& X-\\eta\\sum_{i=1}^{N_{Expert}}\\hat{\\sqrt{g_i}}(B_i\\nabla_{A_i}\\mathcal{L}+\\nabla_{B_i}\\mathcal{L}A_i) \\\\\\\\\n=& X-\\eta\\sum_{i=1}^{N_{Expert}}(\\hat{\\sqrt{g_i}})^2Proj_{col(B_i)}(\\nabla_{X}\\mathcal{L})^T-\\eta\\sum_{i=1}^{N_{Expert}}(\\hat{\\sqrt{g_i}})^2Proj_{row(A_i)}(\\nabla_{X}\\mathcal{L}) \\text{  (same as the derivation of Eq.10)} \\\\\\\\\n=& X-\\eta\\sum_{i=1}^{N_{Expert}}g_iProj_{col(B_i)}(\\nabla_{X}\\mathcal{L})^T-\\eta\\sum_{i=1}^{N_{Expert}}g_iProj_{row(A_i)}(\\nabla_{X}\\mathcal{L}), \n\\end{aligned}\n$$\nso that Eq.12 can be achieved. \n\nThe advantages of implementing Eq.13 can be elaborated from two aspects: Firstly, it achieves Eq.12 when doing gradient updating of $X$ while still keeps the original behavior of training gates, because it holds the same gate gradient, $\\nabla_{g_i}X={A_i}^T{B_i}^T$, as normal forwarding; Secondly, it provides equivalent behavior and the same result of normal module forwarding $X=\\hat{W}+\\sum_{i=1}^{N_{Expert}}g_iB_iA_i$, and only requires a relatively low overhead.\n\n**Response to W4** about the presentation issue of MoLA experiments\n\nWe found that the candidates order presented in our baseline comparison experiments (Table 4) may not appropriate. As you suggested, we have moved the $MoLA-SGD (2,4,6,8)$ candidate to the lower part of the table, alongside with other MoLA candidates. \n\n**Response to W5** about the legend of Figure 3 \n\nAs you mentioned, we notice that the legend of Figure 3 should be streamlined since it has a chance to cover part of the blue line (actually it does not cover, but that still leads to unclarity). As a result, we shortened the names of lines in the legend by deleting the \"Loss\" word and drew the figure again. \n\n**Response to your suggestion** of discussing practical implications\n\nThank you for your suggestions. The practical implications of this work are mainly focused on boosting the training of MoE-LoRA, which may be applied to the fields like efficient and low-resource model training, continual or multi-task learning, stablized training and modular task adaptation, etc. For example, some current works use MoE-LoRA structure to distillate knowledge from a much larger dense model, such as Xu et al.[1] Our proposed method may enhance their distillation process. We have added these practical implications to our paper, as a new section before the conclusion section. \n\n**Response to your suggestion** about open-sourcing\n\nYes, we will open-source our implementation on Github after this paper is published.  \n\n[1] Xu, Haiyang, et al. \"Sparse Mixture of Experts Language Models Excel in Knowledge Distillation.\" CCF International Conference on Natural Language Processing and Chinese Computing. Singapore: Springer Nature Singapore, 2024.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. The authors provided a solid theoretical foundation for their method, integrating Riemannian preconditioner and gate-value scaling to address key challenges.", "methods_and_evaluation_criteria": "Methods and evaluation criteria are highly appropriate and well-suited for addressing the problem and application at hand, including benchmarks such as QA datasets, the GLUE benchmark, and VG/VMCBench.", "theoretical_claims": "I reviewed the theoretical claims and their corresponding proofs in the submission. The authors provided detailed derivations and mathematical justifications for their methods, mainly focusing on the integration of Riemannian Preconditioner, such as Limitation 1/2 (Limited representation and Gradient Sub-optimality), Riemannian Preconditioner in LoRA Expert and Rescaling Preconditioners (Section 3.1/3.2).", "experimental_designs_or_analyses": "I checked them. The experimental setup and analyses appear to be well-structured and appropriate for assessing the claims made, such as multiple benchmarks (Table 1/2/3), convergence analysis (Figure 2), and ablation study (Table 4.5).", "relation_to_broader_scientific_literature": "The paper is well-aligned with recent literature, like LoRA and LoRA Variants, MoELoRA, and Gradient Preconditioners. The authors discuss literature related to these concepts, including works such as MiLoRA, LoRA+, DoRA, MoLA, MoV, etc. And more, the authors have done a nice job of using theory to build connections between these concepts.", "essential_references_not_discussed": "The literature discussed by the authors is indeed comprehensive and closely related to the core topics addressed in the paper. They highlighted key advancements in LoRA, MoELoRA, and Riemannian preconditioning.", "other_strengths_and_weaknesses": "Strengths:\n\n* Nice presentation, Reasonable motivation, and Interesting theoretical contribution. The authors introduce a simple yet powerful idea inspired by mathematical principles.  \n\n* The authors propose a method based on gate scaling theory to enhance the performance of MoE-LoRA, which takes into account the influence of manifold curvature.   \n\n* The authors derive a rescaling method based on Riemannian preconditioning and provide a complete theoretical derivation process.\n   \n* This method effectively balances the gradient updates among experts, addressing challenges such as curvature distortion in the MoE.   \n\n* The engineering approximation seems to provide computational efficiency.   \n\nWeaknesses:\n\n* The convergence is crucial for understanding this method. Regrettably, the authors did not carefully address this in Figure 2. A detailed explanation is necessary, such as the meaning of the dual-axis, and the significance of training loss and validation loss. Then, overlapping the axes in the middle of the subplots should be noted.\n\n* Equation 13 appears highly valuable, yet its explanation could be more transparent. The authors should provide a clearer and more detailed interpretation of this equation.\n\n* In Subsection 3.3, the authors propose a more flexible engineering approximation, which is an interesting contribution, this approach seems to achieve low computational overhead. However, more elaboration on the advantages of this approach would be helpful. \n\n* A significant advantage of this method that enhances MoE-LoRA as a training strategy. However, the experiments comparing the proposed method with the MoE- LoRA baseline should be clearer. For example, MoLA-SGD (2, 4, 6, 8) should ideally be presented alongside MoLA in the lower part of the Table.\n\n* The legend of Figure 3 should be streamlined for clarity.", "other_comments_or_suggestions": "While the theoretical contributions and experimental results are well-presented, it would be beneficial to include a more detailed discussion of the practical implications of this method.\nAnd I highly recommend that the authors consider open-sourcing their implementation.", "questions_for_authors": "Please see Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcPTCSI2XQ", "reviewer": "Reviewer_1wAK", "summary": "This paper introduces a new approach to enhance the performance of MoE-LoRA for fine-tuning foundation models by incorporating Riemannian Preconditioners. This approach ensures that the gradient updates align more closely with the full-rank optimization, thereby stabilizing and accelerating the training process. Moreover, they identify a previously overlooked issue: the gate values in MoE-LoRA introduce additional scaling that distorts the gradient updates and undermines the effectiveness of Riemannian Preconditioners. To resolve this, the authors propose a novel gate-value-based rescaling method that adjusts the gradients of each expert to account for the impact of gate values. The results show substantial improvements in performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable reviews and your **agreements on our proposed method and our efforts on literature review**. We have conducted several new experiments and provided responses to all your concerns: \n\n**Response to W1** about the limitation of LLaVA experiments\n\nIn our revision, we conducted more LLaVA experiments on both Visual7W and VMCBench using LLaVA-v1.5-7B. Specifically, we implemented candidates under two new MoE configurations with different expert numbers: $16/8/4$ and $10/5/4$. The following table illustrates the results. An overall enhancement of our method can still be witnessed under different configurations, especially for SGD.\n\n|  **Candidates**  |   **Visual7W**   |   **VMCBench**  |\n|  :-----   |   -----:  |   -----:  |\n| $RSGD_{16,8,4}$ |     0.72     |    0.59    |\n|$gRSGD_{16,8,4}$ |   **0.74**   |  **0.69**  |\n| $RSGD_{10,5,4}$ |     0.71     |    0.63    |\n|$gRSGD_{10,5,4}$ |   **0.74**   |  **0.73**  |\n| $RAdamW_{16,8,4}$ |   0.76     |   0.71   |\n|$gRAdamW_{16,8,4}$ |  **0.77**  |   0.71   |\n| $RAdamW_{10,5,4}$ |   0.76   |   0.76   |\n|$gRAdamW_{10,5,4}$ |   0.76   | **0.77** |\n\nOur performance boosts for LLaVA under AdamW might not be that remarkable. Therefore, to conduct a further significance analysis of our AdamW boosting, we also implemented more candidates and trained them by AdamW. Please refer to the following table. Similar phenomenon can be invariably witnessed. \n\n|  **Candidates**  |   **Visual7W**   |   **VMCBench**  |\n|  :------   |   ------:  |   ------:  |\n| $RAdamW_{5,5,4}$   |    0.77    |    0.75    |\n| $gRAdamW_{5,5,4}$  |    0.77    |  **0.76**  |\n| $RAdamW_{5,2,4}$   |    0.73    |    0.75    |\n| $gRAdamW_{5,2,4}$  |  **0.77**  |  **0.78**  |\n| $RAdamW_{3,2,4}$   |    0.75    |    0.75    |\n| $gRAdamW_{3,2,4}$  |  **0.76**  |    0.75    |\n\n**Response to W2** about performances on different LLaMA models\n\nWe conducted more experiments on LLaMA models besides Llama-3.2-3B. Specifically, among all the LLaMA 3.2 models there are only 1B and 3B models that are purely textual. Therefore, we decided to include further experiments on Llama-3.2-1B. Four QA benchmarks have been tested, each trained for 2000 steps. Due to the limited resources and time, we set a relatively smaller MoE configuration to speed up training, which is $10/5/1$. Results are illustrated in the following table. We also included this table in the revision of our paper. \n\n|  **Candidates**  |  **ScienceQA**  |  **CommonsenseQA**  |  **OpenBookQA**  |  **SIQA**  |  **Avg.**  |\n|  :-----  |  -----:  |  ----:  |  ----:  |  -----:  |  -----:  |\n|  $RSGD_{10,5,1}$     |    47.71    |    49.47    |    48.80    |    50.41    |    49.10    |\n|  $gRSGD_{10,5,1}$    |  **49.87**  |  **59.30**  |  **54.00**  |  **57.06**  |  **55.06**  |\n|  $RAdamW_{10,5,1}$   |    46.18    |    42.92    |    41.60    |    44.11    |    43.70    |\n|  $gRAdamW_{10,5,1}$  |  **46.58**  |  **43.82**  |  **43.40**  |  **45.50**  |  **44.83**  |\n\nBesides LLaMA 3.2 models, we also tested Llama-3.1-8B. Until now we only conducted four ScienceQA evaluations with each training for only 600-800 steps. Results are below:\n\n|  **Candidates**  |  $RSGD$  |  $gRSGD$  |  $RAdamW$  |  $gRAdamW$  |\n|  :-------------  |  -----:  |  ------:  |  -------:  |  --------:  |\n|   **ScienceQA**  |  71.49   |  **76.35**  |  87.50  |  **87.68**  |\n\n**Response to W3** about the sum-to-1 constraint in MoE\n\nIn general MoE, constraining the sum of all gate values to 1 contributes to the model stability and probabilistic interpretation. Firstly, it normalizes gate outputs to get rid of some uncontrolled issues during training and inferring, such as gradient explosion or vanishing, overflow of variables from accumulated forwarding, etc. During training, by fixing the sum to 1, the gating network focuses solely on allocating relative importance among experts, rather than learning absolute weight magnitudes. This also reduces the complexity of the optimization problem and stabilizes training; Secondly, sum-to-1 constraint can be interpreted as the probabilistic distribution of selecting each expert, aligning with the requirements of probabilistic models and enabling clearer theoretical foundation of MoE.\n\n**Response to W4 and your suggestion** of prominently highlighting our contributions\n\nThank you for your suggestions. Our contributions include integrating mixture of LoRAs with Riemannian preconditioners to alleviate both limited representation and sub-optimality issues; emphasizing the distortion issue behind per-expert preconditioning; and proposing a gate-based rescaling method and its engineering approximation to further boost MoE-LoRA training. We have already revised our abstract, introduction and conclusion sections, to explicitly involve these statements.  \n\n**Response to W5** about small errors\n\nWe revised the mentioned \"table 4\" to \"Table 4\" in Section 4.5 in our revision, together with checking and fixing some other small notation errors.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria, including the use of benchmark datasets, are appropriate and well-suited for the problem or application at hand.", "theoretical_claims": "Yes, I checked the correctness of the derivation for the theoretical claims.", "experimental_designs_or_analyses": "Yes, I checked the soundness and validity of the experimental designs and analyses presented in the submission. There are no issues with the experimental design.", "relation_to_broader_scientific_literature": "The manuscript provides a thorough discussion of the relevant literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths are as follows,**   \n1-This manuscript is written with a theoretical style.   \n\n2-This manuscript features comprehensive and in-depth related work, such as conducting theoretical analyses of the most relevant work.   \n\n3- This manuscript presents experiments conducted on both QA datasets, the GLUE benchmark, and Multimodal benchmarks. The results consistently show performance improvements compared to RAdamW and RSGD.   \n\n4-The experimental results demonstrate that the authors have effectively addressed the two limitations they declared, show that the lift of the method is remarkable.   \n\n**Weaknesses are as follows,**   \n1-They effectively showcased the scalability of their method on Multimodal Large Model LLaVA, which speaks to its broader utility. However, I noticed that they only tested a single configuration of expert numbers. This limits the contribution of this section compared to the rest of the manuscript.   \n\n2- Despite testing the performance on multiple foundation models, I am curious to see how the method performs on LLaMA models of varying sizes.    \n\n3- In the MoE, the sum of all gating values (g) is constrained to 1. In my experience，I’m not entirely clear on the rationale behind this choice, and it would benefit from a clearer explanation.   \n\n4- While the authors have done a thorough job reviewing related work, they should more clearly to highlight the contributions.   \n\n5- In Subsection 4.5, it should be “Table 4” instead of “table 4.”", "other_comments_or_suggestions": "Please see weaknesses above, and I recommend that the authors more prominently highlight the contributions of this work in the abstract, introduction, and conclusion.", "questions_for_authors": "Please see the weaknesses 1 and 3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YG9oYWfBP3", "reviewer": "Reviewer_cEEk", "summary": "The authors propose to apply the Riemannian Preconditioner introduced in the previous work to improve the Mixture of LoRA framework. The Riemannian Preconditioner enhances LoRA training by projecting the full matrix gradient to the subspace of LoRA matrixes, which better approximates full fine-tuning compared to the unscaled gradient descent. However, applying the preconditioner with Mixture of LoRA is coupled with a further rescaling of the manifold constructed for each expert, leading to underestimated gradients. The authors incorporate a new scaling mechanism and develop an engineering approximation to address the issue. Extensive experiments across various downstream tasks including Question Answering, the GLUE Benchmark, and the Vision-Language task are conducted to validate the approach's efficacy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your reviews and thank you for **acknowledging our efforts on theoretical and experimental analysis**. For your concerns mentioned in the review, we provide corresponding response below: \n\n**Response to your concern** of AdamW performances under Multi-Task scenarios\n\nOur supplementary material does not indicate an out-performance of our method for AdamW optimizer under multi-task scenarios. We considered it may due to lacking a sufficient exploring across different multi-task scenarios, since we only conducted the multi-task experiments under a single mixture of two tasks (ScienceQA and MRPC). Consequently, we conducted more experiments on multi-task scenarios in our revision, including two mixtures of tasks and two various configurations of MoE structure.\n\nIn our revision, we grouped six tasks from the GLUE Benchmark into two mixtures. The first mixture consists of CoLA, SST-2 and MRPC tasks, which serves as a multi-task scenario involving both grammar checking, sentiment classification, and equivalent sentences judging; The second mixture consists of STS-B, QQP and QNLI tasks, which serves as another multi-task scenario involving both sentence similarity scoring, equivalent questions judging, and question-answering NLI. For evaluation, we tested candidates on each of the tasks individually and then averaged per-task performances within the mixture as the overall evaluation for that mixture. \n\nFor sufficiently assessing the multi-task performance of our proposed gate-based rescaling method, we conducted experiments under two different MoE configurations, i.e., $20/10/4$ and $10/5/4$. We trained each candidate for 2000 steps under RAdamW and gRAdamW (RAdamW with our proposed gate-based rescaling method) optimizers. The following two tables illustrate our performances under the first and the second mixtures respectively. \n\n**Mixture 1: CoLA + SST-2 + MRPC**\n|  **Configuration**  |  **$RAdamW$**  |  **$gRAdamW$**  |\n|  -----------------  |  ------------  |  -------------  |\n|      $20/10/4$      |      70.15     |    **71.39**    |\n|      $10/5/4$       |      71.64     |    **72.13**    |\n\n**Mixture 2: STS-B + QQP + QNLI**\n|  **Configuration**  |  **$RAdamW$**  |  **$gRAdamW$**  |\n|  -----------------  |  ------------  |  -------------  |\n|      $20/10/4$      |      74.61     |    **75.74**    |\n|      $10/5/4$       |      74.81     |    **75.36**    |\n\nAs a result, we concluded that overall our proposed method is still effective to boost AdamW optimization under multi-task scenarios. We have revised our supplementary material to integrate these experiments and conclusions.", "claims_and_evidence": "The overall claims are clearly stated and well supported by extensive experiments across different benchmarks:\na) The authors claim that incorporating a Riemannian Preconditioner into the Mixture of LoRA framework yields superior performance.\nb) They claim that the scaling mechanism introduced in the preconditioner helps address the issue of underestimated gradients.\nc) They claim that the engineering approximation solution improves training dynamics and model performance.", "methods_and_evaluation_criteria": "Yes. The proposed method builds upon the Riemannian Preconditioner approach from prior work to develop modifications aligned with the Mixture of the LoRA framework. Extensive experiments across various downstream tasks including Question Answering, the GLUE Benchmark, and the Vision-Language task are conducted to validate the approach's efficacy.", "theoretical_claims": "The theoretical claims in this paper is generally correct and sound. The Riemannian Preconditioner is built upon the previous work. They formally derived the preconditioner's form under the Mixture of LoRA scenario. They further propose a rescaling mechanism to address the underestimation issue—although this component lacks a fully rigorous theoretical proof, it is validated by experimental results, which demonstrate improved performance.", "experimental_designs_or_analyses": "The experimental methodology is valid. The authors evaluate their approach across different tasks, from language to vision understanding. I did not detect any significant issues with their design or analysis.", "relation_to_broader_scientific_literature": "This work combines two established research directions: Mixture of LoRA (Low-Rank Adaptation) and Riemannian Preconditioning. Mixture of LoRA extends the low-rank fine-tuning paradigm by introducing multiple “expert” components. These experts are activated selectively for each token via the gating mechanism. Riemannian Preconditioners are proposed to ensure the update is done in accordance with the full rank gradient projection onto the subspace of LoRA matrices, stabilizing the training process. By merging these ideas and introducing a rescaling mechanism to address gradient underestimation, the paper demonstrates improved performance on various benchmarks. This approach introduces moderate innovation supported by extensive experiment results.", "essential_references_not_discussed": "The references cover the related works about Mixture of LoRA and the gradient preconditioners. Works about LoRA and LoRA variants are also covered.", "other_strengths_and_weaknesses": "This paper combines Mixture of LoRA and Riemannian preconditioning with a rescaling mechanism to address gradient underestimation. While this integration is useful and supported by thorough experiments, the approach remains incremental.The theoretical justification for rescaling appears partially heuristic. Overall, the work provides a moderate advance that could benefit practitioners interested in more effective low-rank fine-tuning.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem of fine-tuning foundation models using low-rank adapters (LoRAs). In particular, the authors identify that the mixture of LoRAs (MoE-LoRA) exhibits low robustness during both tuning and inference. To address this issue, they propose a new training strategy for MoE-LoRA that stabilizes and enhances its feature learning process through multi-space projections. Experiments on several datasets demonstrate the effectiveness of the proposed model.


Strengths:
1. The introduction of the Riemannian Preconditioner into MoE-LoRA is an innovative approach, and the authors further propose a gradient rescaling method based on gating values, which provides a solid theoretical foundation for their method.
2. The experiments are sufficient to demonstrate the effectiveness of the proposed model, with strong performance results.
3. The paper is well-written and easy to follow.


Weaknesses:
1. Some notations and terms are not clearly defined or consistently used, which may confuse readers.
2. Certain parts of the paper require further clarification, such as the convergence analysis in the figure and the usage of Equation 13.

Overall, this paper makes a valuable contribution to the field of training mixtures of LoRAs. I recommend the authors address the aforementioned issues as well as those pointed out by the reviewers, particularly in the camera-ready version, to improve clarity and ensure a more thorough explanation of key concepts.
Decision: Accept (poster)