Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
Authors: Chenghao Fan
Zhenyi Lu
Sichen Liu
Chengfeng Gu
Xiaoye Qu
Wei Wei
Yu Cheng
Venue: ICML 2025 poster (2025)
Keywords: Parameter-Efficient Fine-Tuning
Large Language Model
Mixture of Experts
Low-Rank Adaptation
Primary area: deep_learning->foundation_models
TL;DR: In this work, we propose GOAT, a novel framework that enhances LoRA fine-tuning by adaptively integrating SVD-structured priors and aligning low-rank gradients with full fine-tuned MoE through theoretical scaling.

Abstract: While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for Large Language Models (LLMs), its performance often falls short of Full Fine-Tuning (Full FT). 
Current methods optimize LoRA by initializing with static singular
value decomposition (SVD) subsets, leading to suboptimal leveraging of pre-trained knowledge. 
{Another path for improving LoRA is incorporating a Mixture-of-Experts (MoE) architecture.}
{However, weight misalignment and complex gradient dynamics make it challenging to adopt SVD prior to the LoRA MoE architecture.} 
To mitigate these issues, we propose \underline{G}reat L\underline{o}R\underline{A} Mixture-of-Exper\underline{t} (GOAT), a framework that (1) adaptively integrates relevant priors using an SVD-structured MoE, and (2) 
aligns optimization with full fine-tuned MoE by deriving a theoretical scaling factor. 
We demonstrate that proper scaling, without modifying the architecture or training algorithms, boosts LoRA MoE’s efficiency and performance. Experiments across 25 datasets, including natural language understanding, commonsense reasoning, image classification, and natural language generation, demonstrate GOAT’s state-of-the-art performance, closing the gap with Full FT. Our code is available at:  https://github.com/Facico/GOAT-PEFT

Boosting LoRA with Adaptive Singular Values and Mixture-of-Experre Optimization Alignment
Make LoRA Great Again:
Chenghao Fan 1 Zhenyi Lu 1 Sichen Liu
Chengfeng Gu
Xiaoye Qu'
Wei Wei |
Yu Cheng?
Abstract
et al., 2021; Zhao al., 2024). To reduce costs, parameter- efficient fine-tuning (PEFT) techniques (Hu et al., 2021; Pfeiffer et al., 2021; Houlsby et al., 2019; Tian et al., 2024;
While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for Large Lan- guage Models (LLMs), its performance often falls short of Full Fine-Tuning (Full FT). Current meth- ods optimize LoRA by initializing with static sin- gular value decomposition (SVD) subsets, leading to suboptimal leveraging of pre-traine pre-trained knowl- edge. Another path for improving LoRA is incor- porating coaturre-f-Expertt (MoE) architecture. However, weight misalignment and complex gra- dient dynamics make it challenging to adopt SVD prior to the LoRA MoE architecture. To mitigate these issues, we propose Great LoRA Mixture-of- Expert (GOAT), framework that (1) adaptively integrates relevant priors using SVD-structured MoE, and (2) aligns optimization with full fine- tuned MoE by deriving a theoretical scaling fac- tor. We demonstrate that proper scaling, without modifying the architecture or training algorithms, boosts LoRA MoE's efficiency and performance. Experiments across 25 datasets, including natural language understanding, commonsense reason- ing, image classification, and natural language generation, demonstrate GOAT's s sfate-of-the-art performance, closing the gap with Full FT. Our code is available at: https: htt///ttubb...o com/ Facico/GOAT-PEFI.
Fan et al., 2024b) have been proposed. Among them, LoRA (Hu et al., 2021) is popular for its simplicity and effective- ness. It reparameterizes the weight matrix W R=** into W Wo BA, where Wo RR is frozen full-rank matrix, and B E Rm ,A E R are low-rank adapters to be learned. Since the rank < min(m, nt), LoRA only updates a small fraction of the parameters, greatly reducing memory usage.
Despite its computational efficiency, LoRA often underper- forms full fine-tuning (Full FT) (Wang et al., 2024d;c; Fan et al., 2024a), even with Mixture-of-Expert (MoE) archi- tectures (Zadouri et al., 2024; Liu & Luo, 2024; Tian et al., 2024). Our rigorous analysis identifies two key factors lim- iting LoRA's performance: (1) Suboptimal Initialization: The isotropic random initialization for matrix A and zero initialization for matrix B provide a non-informative prior, resulting in unguided optimization subspaces. While Wang et al. (2024b); Meng et al. (2024) applied singular value decomposition (SVD) for better initialization, their reliance on static, predefined subset of pre-trained weights limits the capture of the full range of pre-trained knowledge. It raises the question: Can we adaptively integrate relevant priors of pre-trained knowledge based on input? (2) Un- aligned Optimization: Furthermore, the intrinsic low-rank property of LoRA leads to large gradient gaps and slow con- vergence in optimization, therefore underperforming Full FT. In LoRA MoE scenarios, the total rank is split among experts, resulting in lower ranks and further increasing this challenge. Existing strategies (Wang et al., 2024c;d) focus only on single LoRA architectures and ignore the added complexity of random top-k routing and multiple expert weights within MoE architecture. When SVD-based initial- ization is applied to LoRA MoE, weight alignment becomes challenge, which has never been considered in previous methods that used zero initialization. This further raises the question: How do we mitigate the optimization gap in LoRA MoE initialized with prior information?
1. Introduction
Recent large language models (L.LMs) have shown impres- sive capabilities (Dai et al., 2024; Touvron et al., 2023; Yang et al., 2024; OpenAl et al., 2024), but fine-tuning them for downstream tasks is computationally expensive (Hu
'Equal contribution 'School of Computer Science & Technol- ogy, Huazhong University of Science and Technology *Zhejiang University 'The Chinese University of Hong Kong. Correspon- dence to: Wei Wei <weiw hust.edu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address these challenges, we propose GOAT (Great LoRA Eexture-of-Experty which employs an SVD-

---

Make LoRA Great Again
structured MoE with theoretical scaling match full fine- tuning performance. Our method highlights two impor- tant innovations. (I) Initialization: We demonstrate that
DTD RESISCAS GTSRB EuroSAT
different segments of pre-trained knowledge in the SVD structure are crucial depending on the input. To capture this adaptively, we propose initializing LoRA MoE experts with distinct singular value segments, with the router se- lecting the appropriate prior information. (2) Optimization:
Rather than directly targeting the gap with full fine-tuning, we focus upcycled MoE with full-rank fine-tuning. We show that when each low-rank expert plus pre-trained weight approximates its full-rank counterpart, the router's s behavior remains consistent, enabling effective optimization of expert weights. Through simple scaling, without altering architecture or algorithms, we significantly improve both convergence speed and performance. We also derive the optimal weight alignment strategy and theoretical scaling scheme for better gradient alignment.
20 Different SVD segments
Figure 1. The effect of initializations from different SVD segments (us, di, Vi ) for rank 32 and 128. The performance normalized by min-max scaling.
In summary, the contributions of our method are as follows:
RTxm, E, = Ejrn(( Drr,))) R and Vi ((r:())))) E Rx Fine-tuming methods are represented as:
Adaptive Priors Initialization: We propose a novel SVD- structured MoE framework that adaptively integrates pre- trained knowledge, addressing the limitations of non- informative or static priors.
Full FT: VOE.VO + +UE + +U.."
Theoretical Optimization Alignment: We reveal key connection between LoRA and full fine-tuning upcycled MoE, deriving optimal weight alignment strategy and scaling scheme to close the performance gap.
PISSA: VOEVVO + (U>22 V2 + +.... )
MiLoRA: (UnEoVV + Un V U.EIV
KaSA: (UoIoVy +... UI EL-EE-IVVL VIL," V'V'
State-of-the Art Performance: Extensive experiments 25 tasks demonstrate that our method achieves superior performance while maintaining scalability.
(2)
Here, (-)* denotes frozen components, while non-frozen components initialize LoRA:
B = U-UN E Ra A 1= S//V E ER
(3)
2. Background and Motivation
We observe PiSSA freezes minor singular values and fine- tunes only the components Lo with the largest norms, achieving the optimal approximation to Wo.3 In contrast, MiLoRA and KaSA retain segment 0 (I-1) as preserved pretrained knowledge, but KaSA treats the minor U.E.VV noise and replaces it with practice, PiSSA converges faster by focusing principal singular values, while MiLoRA and KaSA preserve more pre-trained knowledge for better final performance.
2.1. Rethinking Singular-Valus Initialization
Singular-value initialization is widely used in LoRA pre- serve pre-trained weight characteristics (Zhao et al., 2024; Meng al., 2024; Wang et al., 2024a; Lu al., 2024). PiSSA (Meng et al., 2024) only updates the largest singular values, while MiLoRA (Wang et al., 2024b) adjusts minor singular values for strong performance.
random U'ETTT In
To unify SVD-based methods with full fine-tuning, let Wo E m be the pre-trained weight with SVD, Wo = U Assuming h = min (m, n) and LoRA rank r, we decompose Wo into rank-r blocks:
This raises the question: Is it reasonable to use only the principal or minor part as a fine-tuning prior? Figure I illustrates the performance of fine-tuning from different segments (U,, E,, V,T), E (...... .I. where each segment is used for initialization while others remain frozen. The x-axis represents segment indices (e.g.. I = 0 for PiSSA, x = I for MiLoRA), and the y-axis shows min-max normal- ized performance. We can identify two notable observations: (1) The same initialization exhibits varying trends for differ- ent datasets. For example, x achieves better results on
Wo Σ U.E.V.
where 7 = and i denotes the segment [i.r: (i 1). r]. The submatrices are defined as U, = U)) E
UUpyccled MoE initializes all experts with the same pre-trained weights, which we adopt for simplicity.
PPooo Appendix
2

---

Make LoRA Great Again
Loss (8
Grac-Norm
The formula for SVD-based initialization is:
PISSA -
W a sBA= (!U.E.V.. = =UE,V?
PSSSA =
40
P558A 1
(7)
4-
PSSA s 10
30-
7
3-
PISSA
PISSA 0.5
20-
g 2²
(U.U.. g+
(8)
1-
Though the equivalent weight is independent of 5, equivalent gradient is proportional to s. As shown in Figure 2, s = 2 is too small. Increasing the scaling factor in SVD-based methods boosts the gradient, Ieading to faster convergence.
200
200
Step
Figure 2. SVD initialization vs. scaling and rank
the EuroSAT dataset, while = 0 performs better the GTSRB dataset. (2) Middle segments play a crucial role. e.g., when r = 128, the highest performance is typically observed the middle segments. These findings suggest that each singular value segment contains task-specific infor- mation, motivating us to allow the model to automatically select segments during optimization, leveraging all singu- lar values while preserving the original pre-trained matrix characteristics.
Next, we examine the effect of different ranks, as shown in Figure 2. With low rank (e.g., r = 1), the gradient norm is small and deviates from the trend of 64, creating per- formance gap (95.77 vs. 98.55). However, applying proper scaling (s = 16) increases the gradient norm, reducing the performance gap (from 95.77 to 97.70). This is especially beneficial in MoE scenarios, where the total rank is split among experts, resulting in lower ranks. Increased scaling can compensate for this, as supported by Tian et al. (2024).
2.2. Rethinking Scaling Factor
3. Method
In LoRA, it is common practice to use the scaled variant W = Wo + sBA, yet the effects of scaling factor have not been fully explored. Biderman et al. (2024) consider s should typically set to 2. The SVD-based method (Meng et al., 2024) empirically makes s BA independent of s by dividing B and by V [ while Tian et al. (2024) use larger scaling for LoRA MoE to achieve better performance.
3.1. LoRA MoE Architecture
Hixture-of-Experte (MoE) An MoE layer (Qu al., 2024; Zhu et al., 2024a;b; Zhang et al., 2024) comprises E linear modules {W1..... We and router W, m F that assigns input x to experts based on routing scores:
ex)(="(x))
(x()
(9)
Σj= exp(((xxxx
To investigate it, as illustrated on the left of Figure 2, we first adjust s in the SVD-based LoRA with a fixed rank, revealing that still impacts the convergence speed. To study the effect, we introduce the equivalent weight and gradient to quantify the gap between LoRA and Full FT.
where 2(x) = Wax and p' (x) is the score for expert i.
Let (x) denote the indices of the top-k scores, ensuring
[4g (x) = and =x(x > 2x(x) for all E 94 (x)
and
$ (x). Define the weights as:
Definition 2.1 (Equivalent Weight and Gradient). For LoRA optimization, we define the equivalent weight as:
w w=(xx= Eyen, (2) (x]) if lx(x)),
exp(s'(x))
(10)
00
otherwise.
W A W + sBA,
(4)
The MoE layer output is the weighted sum of the top-k experts' outputs:
The equivalent gradient of Wi is defined as:
MoF(x) = En n0 (x()))
(11)
OL
a
(5)
aW
LoRA MoE. We integrate LoRA into the MoE framework, retaining the router (Equation (10) and using the balance loss from vanilla MoE4. Each expert W is replaced by low-rank matrices B mxd and A [2dx where d = =:
where s is the scaling factor, and A and GB are gradients with respect to A and B, respectively.
Lemma 2.2. Let gi be the gradient in fuall- and B, A be the fow-rank weights. At the t-th optimization step, the equivalent gradient can be expressed as:
E
MOELORA W(x)
Σ w (x(xx (B A'xx) 1m1
(12)
9. s² =(BB.B. 9: + g.A. A1)
(6)
*See Appendix C
3

---

Make LoRA Great Again
Single Low-Rank Adaptation
(l) Adaptive Priors Initialization
g
g
Ee g
v g
W3
We
W
We
d=
Organt codt
in Ugentul-tne 19 a guiee
c 2
MoE Fine-tuning
(II Theoretical Optimization Aligument
ab Sipl
quialaa graee
W
W
W
W
spinkar
(1) Weggi Aligunenc.
ang min
|
wlll
Grudien Alianment
ang min 8 5/l
2
Multish
Tuxable
Frozzi
Freward
Gradical
Rooter
No Forwand
Figure 3. Illustration of Our Method. Single Low-Rank Adaptation: LoRA reduces trainable parameters by reparameterizing W W = Wo + sBA, with B and A as low-rank matrices. MoE Fine-tuning: Full MoE fine-tuning, where experts W and W are selected by the router in this moment. Subfigure (I): Our method replaces the single pair B, A with multiple pairs ( A'') initialized from different segments of the SVD of Wo and adaptively selected by the router. Subfigure (II): We align optimization with SVD-structured MoE by separately aligning each expert. Wies ensures the equivalent weight equals Wo before optimization, and scale each expert's equivalent gradient to closely approximate full MoE fine-tuning.
where Win is the pre- pre-trained weight matrix and is the LoRA scaling factor. Since k < F,, LoRA MoE uses fewer active parameters than dense MoE.
to mitigate this and show how scaling resolves the issue.
By ensuring equivalent weight Wo ze Wo at initialization and maintaining equivalent gradient 9: R 9t. throughout optimization, we can align LoRA with Full FT. (See Definition 2.1 for equivalent weight and gradient.)
Theorem 3.1.
3.2. Adaptive Priors Initialization
According to Section 2.1, the utilization of different SVD segments depends on the input. We propose initializing each expert in LoRA MoE with different SVD segments, leveraging the MoE architecture to dynamically activate experts associated with different singular values. Specially, we init expert evenly by define the set & as:
Theorem 3.1 mitigates the performance gap in single LoRA architectures (Wang et al., 2024c;d). For MoE architectures, however, routers and top-k selection complicate direct align- ment. Thus, we focus on Full FT MoE and establish:
Theorem 3.2. For all E [1,..., E), by ensuring equíva- lent weight Wi = Wi at initialization and gradient gi ze gi for each expert, we can align LoRA MoE with an Upcycled MoE with full-rank fine-tuning.
E, = (U de Vik v(())4a)))
E
(13)
where min(m,.)) hs (j 1 the starting index from E
segment for 5-th expert, d = is each expert rank. Then we can construct each expert by V'T) Ex:
Theorem 3.2 reveals a key connection between LoRA and full fine fine-tuning in MoE, simplifying the problem to optimiz- ing each expert separately. We outline the steps below.
Bi v 1 E mxd Ab = -V e Rxn
(14) The B, A divide v make sure that s B.A is independent of s (Meng et al., 2024). This allows the model to adapt flexibly to various fine-tuning scenarios.
Initialization Alignment. At initialization, align the equivalent weight at initialization with an upcycled MoE, where each expert weight E /r=1 is derived from the pre-trained model's 's weight Wo (He et al., 2024). This B),A with
equivalent to aligning Wi, = Wo + Σ the original weight Wg. As B3 AB are initialized with prior information, We need additionally subtracting a constant Wies ensuring the weight alignment:
3.3. Theoretical Optimization Alignment
Directly applying SVD priors in MoF. architectures causes weight misalignment and complex gradient dynamics, a challenge not encountered with previous zero initialization methods. Moreover, the gap in bbsed architectures remains under-explored. We derive the following theorems
E
W. = Wo W
Σ
w (x(x Rz Wo
(15)

---

Make LoRA Great Again
Lemma 3.3. For all i,j E [I..... E] (i + j):
1. Full-Finetuning: Full FT fine-tunes all parameters. while Full FT MoE is Upcycled MoE with full-rank fine-tuning and 2 active experts out of 8 total experts.
Ex(w' [x]] =
(16)
E²
Ks
2. Single-LoRA baselines: LoRA (Hu et al., 2021);
Var(w' (x)) kF
(17)
DoRA (Liu al., 2024); PiSSA (Meng et al., 2024); MiLoRA (Wang et al., 2024b); rsLoRA (Kala- jdzievski, 2023); LoRA-Dash (Si et al., 2024); NEAT
Theorem 3.4. Consider the optimization problem:
W arg min Ex Was
Wms 8
Σ w (x) Bi Ab
(18)
(Zhong et al., 2024); KaSA (Wang et al., 2024a)
3. LoRA MoE baselines: MoLoRA (Zadouri et al., 2024); AdaMoL.E (Liuu & Luo, 2024); Hydral.oRA (Tian et al., 2024).
The closed-form solution is W :
Σ E i=1 B
Theorem 3.4 provides an appropriate initialization scheme for MoE scenarios. Note that the original LoRA-MoE (Zadouri et al., 2024; Tian et al., 2024) uses a zero initializa- tion scheme thus With = 0, a special case of Theorem 3.4.
For a fair comparison, we closely follow the configurations from prior studies (Hu et al., 2021; Meng et al., 2024; Wang et al., 2024d). Details on the baselines are in Appendix G.
Obviously, the variance of res - - w² Bx) is proportional to E Aj. Additionally, from Lemma 3.3, when l is small (e.g. 2k- < F),, std(w"(x) > E[w"(x)] To preserve the informative SVD prior while reducing initial- ization instability, we scale BiA4 by 1. straightforward method to decrease variance and make more accurate approximation in Equation (15):
4.2. Datasets
We evaluate GOAT across 25 tasks, spanning domains:
1. Image Classification (IC): We fine-tune and evaluate VIT-B/32 (Radford et al., 2021) on 7 image classifica- tion datasets (Ilharco et al., 2023).
2. Natural Language Generation (NLG): We fine-tune LLaMA2-7B (Touvron et al., 2023) on subset of Wiz- ardLM (Xu et al., 2023), MetaMathQA (Yu et al.) and
B U. .A8
(19)
sp
(Zheng et al., 2024). We eval- uate its performance on dialogue (Zheng et al., 2023), math (Cobbe et al., 2021) and coding (Chen al.,
Gradient Alignment First, provide the optimal scal- ing for zero-initialized LoRA MoE:
2021) following Wang et al. (2024d)
Theorem 3.5. For By = 0,Ag ~ U (-V
6 3 =
3. Commonsense Reasoning (CR): We fine-tune LLaMA2-7B on Commonsensel 70K and evaluate on
3² <BiDi" st + 4/ Ai). and learning rate ratio be-
8 commonsense reasoning datasets (Hu et al., 2023) (multi-domain setting).
tween full taning vs. LoRA 17.
sill. E
arg mii
96
(20)
4. Natural Language Understanding (NLU): We RoBERTa-large (Liu et al., 2020) on 7 GLUE tasks (Wang et al., 2019) following (Hu et al., 2021).
The cloxed-form solution of optimal scaling is 8
3nz
A >> r, it is typically the case that s > 2, which explains why standard scaling insufficient and why simple scal- ing enhances effectiieness, demonstrated in Section 2.2. While it is tricky to directly analyze complex gradient dy- namics with SVD priors, an alternative approach is recog- nizing that larger scaling and ensure B and A become small and approach zero (Equation (19)), aligning with the settings in Theorem 3.5. Thus, we adopt this scaling factor in GOAT, and in practice, this approximation performs well (see Section 4.3). For scenarios with proper scaling, we extend the method to "GOAT-s", as detailed in Appendix E.
Due to the huge memory requirements of Full FT MoE, we only evaluate it on IC and NLU tasks. Detailed of the datasets can be found in Appendix G.I.
4.3. Main Results
Tables 1, 3 and 4 present results on 4 domain benchmarks:
IC (Table 1): GOAT achieves 99.07% of full FT perfor- mance and surpasses LoRA with quadruple the parameters (rank 32). It improves 6.0% over PiSSA and 2.4% over HydraLoRA, outperforming all LoRA variants.
NLG (Table 2): Our method shows the smallest perfor- mance gap with Full FT, outperforming MoLoRA by 0.25 on MTBench, 6.30% on GSM8K, and 3. 3.14% on HumanEval, highlighting GOAT's superiority.
4. Experiment
4.1. Baselines
CR (Table 3): GOAT consistently outperforms all estab- lished baselines, exceeding the best single LoRA method,
We compare GOAT with Full FT, single-LoRA, and LoRA
MoE methods to substantiate its efficacy and robustness:
5

---

Make LoRA Great Again
Table We evaluate CLIP ViT-B/32 with full fine-tuning and LoRA variants with total rank 8 across StanfordCars, DTD, FuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets, Bold indicates the highest results.
Method
# Params (%)
Cars
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Average
Full
100
60.33
73.88
98.96
98.30
93.65
53.84
96.78
82.25
MoE
75.53
97.09
Single LoRA Methods
LoRA
41.02
70.15
47.51
95.39
77.09
LoRA (1ank16)
2.99
46.51
72.07
98.74
98.04
92,08
51.63
96.00
79.30
LoRA (rank32)
5.98
50.13
72.87
98.88
98.13
92.87
53.65
96.55
80.44
DoRA
1.49
40.75
71.91
98.89
97.71
90.19
47.54
95.46
77.49
PISSA
MiLoRA
1.49
39.77
70.48
98.
97.52
89.92
45.38
95,49
76.68
LoRA MoE Methods
MoLoRA
2.24
50.83
73.51
98.63
97.72
92.58
52.55
966.00
80.26
AdaMoLE
2.33
49.47
71.65
98.52
97.73
91.95
52.29
95.82
79.63
HydraLoRA
1.58
48.42
72.18
98.40
97.28
92.93
51.80
96.06
79.58
GOAT
Table evaluate Jlama-2-7B on MT-Bench, GSM8K, and
HumanEval for dialogue. math, and coding.
PISSA
5
Method
MT-Bench
GSM8K
HumanEval
Full
GOAT (Ours)
Full FT
5.56
59.36
35.31
Loss
AdaMoLE
Single LoRA Methods
3-
McLoRA
HydraLoRA
LoRA
5.61
52.84
21.34
DoRA
5.97
54.59
19.75
1
PISSA
5.30
55.42
19.52
MiLoRA
5.23
54.44
19.51
300
LoRA MoF. Methods
Steps
MoLoRA
5.84
56.63
24.83
Figure 4. Training loss curves Different LoRA methods and Full Fine-tuning MoE on Cars. The balance loss is excluded in the MoE baselines for a fair comparison with single LoRA baselines.
HodraLoRA
5.82
57.39
24.21
GOAT
6.01
60.20
25.61
original zero initialization, yielding 81 006/80.26). Without MoE, initializing single LoRA with our SVD fragments achieves performance of 77.62. In contrast, our MoE ar- chitecture achieves 80.35, demonstrating its clear advantage in effectively integrating expert functionalities.
KASA, by 1.47%, the best LoRA-MoE method, Hy- AraL by 1.98%, and ChatGPT by 7.42%.
NLU (Table 4): performing Single LoRA Method, MiLoRA, by 0.28%, surpasses the best-performing LoRA MoE Method, MoLoRA, by 0.27%, and achieves a 1.98% improvement over Hod..LoRA. Furthermore, our method surpasses the Full FT (89.47 vs. 89.76) and reduces the gap with Full FT MoE just 0.1%.
method outperforms the best-
4.5. Convergence Speed
As shown in Figure 4, we compare the training loss curves of PiSSA, various LoRA MoE baselines, our proposed GOAT, and Full FT MoE on the Cars and MetaMathQA datasets. GOAT demonstrates faster convergence compared to the LoRA MoE baselines and achieves performance closest Full FT MoE. Notably, method achieves lower final loss, balancing performance and efficiency. In contrast, methods like PiSSA converge quickly initially but yield suboptimal final performance, as discussed in Section 2.1.
In summary, GOAT outperforms across all benchmarks, achieving superior results in nearly every sub-task, and closes or surpasses the performance gap with Full FT, demonstrating the superior effectiveness of our approach.
4.4. Ablation Study
We conduct ablation experiments to evaluate the impact of our adaptive priors initialization and gradient scaling, as summarized in Table 5. Our initialization, with or with- MoE scaling, consistently outperforms other m (note that SVD-based initialization corresponds to the
4.6. Scaling Property
Scaling across Different Rank. To evaluate the scalabil- ity of our method, we increase the rank GOAT from to 128 on CV benchmarks, as shown in Figure 5. As the rank increases, the performance gap between GOAT and
Details provided Appendix G.3
6

---

Make LoRA Great Again
Table Performance comparison of LLaMA2 7B with different methods on eight commonsense reasoning datasets. The symbol f
indicates that the results are taken from (Wang et al., 2024a; Zhong et al., 2024; Si et al., 2024).
Method
Paramsi %)
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
ChatGPT
77.01
Single LoRA Methods
LoRAt
0.84
69.80
79.50
83.60
82.60
79.80
64.70
81.00
77.61
DoRAF
0.84
71.80
83.10
79.90
89.10
83.00
84.50
71.00
81.20
80.45
PISSA
67.60
78.10
78.40
76.60
78.00
75.80
60.20
75.60
73.78
MiLoRAf
0.84
67.60
83.80
88.20
82.00
82.80
68.80
80.60
79.24
LoRA-Dashi
0.84
71.00
75.70
79.30
91.10
78.60
84.20
69.80
78.80
78.56
NEAT!
71.70
83,90
80.20
84,30
86.30
71.40
83,00
81.21
KaSAi
84.40
91.50
84.50
84.70
72.10
81.20
81.53
LoRA MoE Merhods
MoLoRA
0.96
73.15
83,68
80.09
74.57
85,95
87.33
72.53
86,20
80.43
ORA
0.84
72.78
84.06
79.68
80.3 34
86.66
87.12
72.35
86.00
81.12
GOAT
87.79
76.88
82.73
Table 4, Performance comparison of RoBERTa la-lage with different methods on 7 GLUE tasks. Total rank is set to 32.
Method
# Params (%)
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
Average
Full FT Full FT MoE
94.49
84.84
698
86.02
96.22
85.05
92.20
90.20
95.10
84.48
89.90
Single LoRA Methods
LoRA
4,00
83.41
95.64
83.33
90.06
89.00
93.28
84,47
88.46
DoRA
4,00
85,33
95. 99
84.07
91. 91.24
89.52
93.54
84.48
89.17
PISSA
4,00
69.12
95.98
82.84
91.24
93.59
73.29
MiLoRA
4,00
84.65
96.10
86.02
91.33
89.51
94.12
84.83
89.51
rsLoRA
4,00
83.51
95.98
86.02
90.75
88.97
93.84
84.
89.03
LoRA MoE Methods
MoLoRA
4.50
83.94
96.10
87.75
91.45
89.36
93,90
84.11
89.52
AdaMoLE
4.56
83,99
95.76
86.03
91.48
89.21
93.64
83.75
89.12
HydraLoRA
2.75
83.89
95.52
85.04
91.02
89.34
93.87
81.22
88.56
GOAT
4.50
86.86
96.21
84,55
91.40
89.55
94.19
85.56
89.76
Table Ablation study of GOAT. "MoE" denotes using the MoE architecture instead of single LoRA. "MS" refers to using MoE scaling. "O", "P", "M", and "R" represent initializations from seg- ments that selected by ours, with the principal singular value, with the minor singular value, and are randomly selected, respectively.
/ HydraLoRA
MoLoRA
GOAT
Full Mol (84.40)
84
Porf.
82
SVD Initialization
80
MoE
Avg.
Avg. (w/o MS)
78
81.49
80.35
16
32
64
128
81.11
80.02
81.14
80.03
Figure 5, Performance of different methods across ranks.
81.22
80.07
81.06
80.26
77.62
ing total rank 32 to verify the scalability of our method with different expert numbers and activation ratios, as shown in Figure 6. Key findings include: (1) With 8 experts, the 2in8 configuration achieves strong performance. Activat- ing more experts may yields lower performance, showing that sparse expert activation is important.
full fine-tuning MoE narrows significantly. Notably, GOAT consistently outperforms both MoLoRA and HydraL across all ranks. At rank 32, GOAT achieves 83.04, surpass- ing MoLoRA (82.15) by 1.08% and HydraLoRA (82. 12) by 1.12%. While higher ranks improve performance, gains diminish as ranks inerease. For instance, GOAT improves by just 0.38% from rank 64 to 128, highlighting diminishing returns with higher computational costs.
(2) Increasing the total number of experts may improves performance, as seen in 2in8 4in16 / 8in32 but makes routers harder to train, increases memory consumption, and reduces runtime efficiency. (3) GOAT consistently outperforms MoLoRA, especially when activate only one expert, consistent with discussion in Section 2.2. In practice, 2in8 offers a balanced trade-off between performance and storage efficiency.
Scaling across Different Expert Number and Activated ratios. We also conduct experiments on CV datasets fix-

---

Make LoRA Great Again
82.5
Table 7. Comparison of LoRA- and Full FT MoE in memory cost, training time, and GSM8K performance. Memory cost was measured and training time was recorded on the MetaMath dataset using one A GPU with identical batch sizes.
2im8
80.0
Per.
77.5
GOAT MoLoRA
75.0
Method
Memory Cost
Epoch Time
Performance
lin4
lin8
2in8
3in8
4in8
5in8
6in8
7im8
8in8
4in16
8in32
Full FT MoE
2 GB
=106h 03min
2 59.36
MoLoRA
34.85 GB
36h56min
56.63
HydraLoRA
34.81 GB
36h56min
57.39
Figure 6. Performance vs. number of experts and activation ratio (total rank=32). "2 in 8" means activating 2 out of 8 experts.
GOAT
34.85 GB
36h59min
60.20
4.7. Routing Analysis
O(He), significantly smaller than Full FT's O(1f2) and
Full FT MoE's O(HH2). Since r, < H. GOAT is much more efficient. Detailed analysis is in Appendix H.1.
1,00
Token Densiyy Pee Expert
FLOPs Analysis To compare with Full FT MoE, we esti- mate the memory usage, runtime, and performance of FT MoE based on the single GPU runtime of Full FT. As shown in Table 7, the LoRA-MoE series trains much faster than Full FT MoE. Among LoRA-MoE variants, our method achieves the best performance with identical memory and time costs. FLOPs analysis (see Appendix H.2) reveals that Full FT MoE scales as O(kHH?), while LoRA MoE sim- plifies to ((HH²) since k < e and r < H. Thus, LoRA MoE's FLOPs remain nearly constant, independent of k, unlike Full FT MoE, which scales linearly with k.
C.00
6:05
Cofs
DTD RESISC4S
SUN397
MAPC
RTE
GISMS
Huumen vall
Figure 7. Expert Load Distribution across different tasks. We illus- trate the fraction tokens assigned to each expert {e We visualize the expert load distribution of models trained on 9 tasks in Figure 7. With 8 experts (2 activated), the expected token density is 125. The visualization highlights several key observations: (1) The load is evenly distributed, with no inactive experts and fluctuations remaining within 0.125, varying by no more than 15% (0.02). (2) CV and GLUE tasks show balanced expert usage, while generation tasks (GSM8K and HumanEval) favor the bottom-2 experts (ei and e2) with a load around 0.14. (3) This validates the effectiveness of each SVD chunk, as experts are initialized with distinct singular value regions.
5. Related Work
Since the introduction of LoRA (Hu et al., 2021), various variants have emerged, focusing on three key areas: (1)
Architecture Improvements: DoRA (Liu et al., 2024) decom- poses updates into magnitude and direction, while NEAT (Zhong et al., 2024) introduces nonlinear adaptations. (2)
Adaptive Rank/Scale, AdaLoRA (Zhang et al., 2023) of- fers dynamic rank allocation, LsLoRA (Kalajdzievski, 2023) adjusts scaling factors and LoRA+ (Hayou et al., 2024) im- proves learning rate. (3) Initializationm/ppimmatttt (Meng et al., 2024), MiLoRA (Wang et al., 2024b), and KaSA (Wang et al., 2024a) utilize SVD-based strategies to preserve knowledges. LoRA-Dash (Si et al., 2024) au- tomates optimal direction discovery, whereas LoRA-GA (Wang al., 2024c) and LoRA-Pro (Wang et al., 2024d) align updates with full fine-tuning gradients. However, they still exhibit performance gap between full fine-tuning.
4.8. Different Learning Rate
PiSSA
Table Performance comparison of different learning rates.
Learning rate MoLoRA
HydraLoRA
GOAT
le-5
56.18
55,19
58.74
2e
56.63
57.39
60.20
56
60.19
60.96
62.05
To evaluate GOAT's sensitivity to learning rates, we tested its performance on GSM8K using rates ranging from 1 10-5 to x 10-3, comparing it against MoLoRA and HydraLoRA. As shown in Table 6, GOAT consistently out- performs the other methods, showcasing its robustness and the effectiveness of our initialization and scaling strategies in accelerating convergence and enhancing performance.
Multi-LoRA architectures further boost performance: Lo- RAHub (Huang et al., 2024) combines task-specific LoRA modules, MoLoRA (Zadouri et al., et al., 2023) and LoRAMoE (Dou et al., 2023) integrate MoE structures with LoRA. MultiLoRA (Wang et al., 2023) introduces learnable scaling for each expert, while AdaMoLEE (Liu & Luo, 2024) introduces learnable thresh- olds for dynamic experts selection. HydraLoRA (Tian et al., 2024) adopts an asymmetric MoE architecture. Unlike these methods, GOAT introduces a novel SVD-structured MoE framework that adaptively integrates relevant priors while
(Liu
4.9. Computation Analysis
Parameter Size. The "# Params (%)" column in Tables I,
2, 3, and 4 compares the parameter ratios of LoRA baselines and GOAT to full fine-tuning MoE. GOAT achieves state- th-the-rrt performance with a parameter size of O(Hr) +
8

---

Make LoRA Great Again
addressing weight misalignment and gradient dynamies through theoretical scaling.
G., al. Evaluating large language models trained on
code. arXiv preprint arXiv:2107.03374, 2021.
6. Conclusion
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10):1865-188) 2017.
In this work, we propose GOAT, a novel framework that enhances LoRA fine-tuning by adaptively integrating SVD- structured priors and aligning low-rank gradients with full fine-tuned MoE through theoretical scaling. Without al- tering the architecture or training algorithms, GOAT sig- nificantly improves efficiency and performance, achieving state-of-the-are results across 25 diverse datasets. Our ap- proach effectively bridges the performance gap between LoRA-based methods and Full Fine- Tuning.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613. 2014.
Clark, C.,, Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M. and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Do- ran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ N19-1300. URL https: hta///aaaaaatttooooooooo org/ N1 N19-1300/.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110, No. 62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). We thank the Shanghai AI Laboratory for supporting GPU resources. The authors would also like to thank the anony- mous reviewers for their comments on improving the quality of this paper.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https: //tp:///aavvvvviiiv .owvvoorg//oo /bs/ 1803. 05457.
Impact Statement
GOAT enhances the efficiency and performance of fine- tuning large models, significantly reducing computational and memory costs. This makes advanced AI technologies more accessible researchers and practitioners with limited resources, fostering innovation across diverse fields such as NLP, CV, and multi-mode applications. By leveraging adaptive priors and robust gradient handling, GOAT can drive breakthroughs in solving real-world challenges, en- abling more efficient and scalable Al solutions for a wide range of industries. Our work focuses on improving model efficiency and adaptability and does not introduce any direct ethical concerns risks.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.11104 14168, 2021.
Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F,, Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. https::: //arxiv. barxivv.ogg/bb... / 2401.06066.
Dolan, B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005.
References
Biderman, D., Portes, J., Ortiz, J. J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J.. et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024.
Dou, S., Zhou, E., Liu, Y., Gao, S., Zhao, J., Shen, W., Zhou, Y., Xi, Z., Wang, X., Fan, X., al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint urXiv:2312.0999 4(7), 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Fan, C., Lu, Z., Wei, W., Tian, J., Qu, X., Chen, D., and Cheng, Y. On giant's 's shoulders: Effortless weak to strong by dynamic logits fusion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: httpttpss//pppeeeeeeeeeeeee /oppnrevieiieeeeeeooeeeeeee net/ forum? á dd=RMF i fWANg.
Chen, M., Tworek. J., Jun, H., Yuan, Q., Pinto, H. O.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman,

---

Make LoRA Great Again
Fan, C., Wei, W., Qu, X., Lu, Z., Xie, W., Cheng, Y., and Chen, D. Enhancing low-resource relation representa- tions through multi-view decoupling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16): 17968-17976, Mar. 2024h. doi: 10. 10.1609/aai..v38i1
loRA composition. In First Conference on Language Modeling, 2024. URthttps: openreview net/ forum? Corum?id=Ten -ammiiddd-roxx....
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic, 2023. URL https:// arxiv. rexiv.org/ee orggorggaaeooo htgg/bb///// 2212.04089.
29752. URL https: aaw.: org/ index. php/
29752..
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Kalajdzievski, D. A rank stabilization sealing factor for fine-tuning with Iora, 2023, URL https: /arxiv. org/ abs/2b12 03732.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine- fineggraieed categorization. In Proceed- ings of the IEEE international conference computer vision workshops, pp. 554-561, 2013. 20133.
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, W. B. The third pascal recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Liu, Q., Wu, X., Zhao, X., Zhu, Y., Xu, D., Tian, F., and Zheng, Y. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. CoRR, 2023.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models, 2024. URL https: //a //arxivvco org/ abs / 2222 12354.
He, E., Khattar, A., Prenger, R., Korthikanti, V., Yan, Z., Liu, T., Fan, S., Aithal, A., Shoeybi, M., and Catanzaro, B. Upcycling large language models into mixture of experts. arXiv preprint arXiv:2410.07524 2024.
Liu, S.-y., Wang. C.-Y., Yin, H., Molchanov, P., Wang, Y-C. F., Cheng, K.-T, and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2223 2019.
Liu, Y. Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Ro(bert)a: a: A robustly optimized (bert) pretraining ap- proach, 2020. URL https: http///pppeeeeeeeeeeeeeeeeeeeeee net/ forum idrum?id==yxSST4tvv
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly. S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Liu, Z. and Luo, J. AdamoLE: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts. In First Conference on Language Modeling, 2024. URL https:/ /oppnrevieeeeeeeeeeeeeeeeeeeeeee net/ forum? id=ndy fFFa.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2021.
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-merging:
Dynamic integration of modular ex- pertise in model merging. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https:////// /pwnreviewwweeeeeeee net/ forum? id- S1-81II633TT.
Hu, 7., Wang, I., Lan, Y., Xu, W., Lim, E.PP., Bing, I.., Xu, X., Poria, S., and Lee, R. LLM-adapters: An adapter fam- ily for parameter- efffciient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5254-5276, Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10. 10.18653/w1/0223.mmmll 319. URL https: ha//aclanntt /ttpa//aanhhooly.ooo////0000 emn 1 pp-maine 3196
Meng, F., Wang, Z., and Zhang, M. PISSA: Princi- pal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https:: htt.tp:///ppeeeeiitiit wwwpreeeeeeeeeeeeeeeeeeeeeeeeewe net/ forum? id=62BHIE:dP4.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, (eds.), Proceedings of
Huang, C., Liu, Q., Lin, B. Y., Pang. T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic
10

---

Make LoRA Great Again
the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18- 1260. URL https: //acl anchon httg:/rrg/11-1111
Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J.. Shoker, S., Shyam, P.. Sidor, S., Sigler, F.., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang. J., Tezak, N., Thompson, M. B., Tillet, P.. Tootoonchian, A.. Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. Wang. A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wi- ethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S.. Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024. URL https::/p//........ / a wwwaaavvvvvoo. abs 2203. 08774.
Netzer, Y., Wang, T.,, Coates, A., Bissacco, A., Wu, B., Ng. A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and wnsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
OpenAl, Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, L., Bal- aji. S., Balcom. V. Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shappe Shapiro,, G., Berner, C., Bogdonoff,, L.,, Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G.. Brooks, T., Brundage, M., Button, K. Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B. Cho. C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry. T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Floundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S.S Guo, Y., Hallacy, C., Han, J. Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K. Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J. Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, W. Kim. C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., t.kasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li. C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T. Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y.. Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mchta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., MU, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang. L., O' O'eeffe. C., Pachocki, J. Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G. Parish, J., Parparita, F.,, Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong. V. H., Powell, T., Power, A., Power, B., Prochl, E.,
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pp. 487-503, 2021.
Qu, X., Dong, D., Hu, X., Zhu, T., Sun, W., and Cheng, Y. Llama-man mone v2: Exploring sparsity of Ilama from per- spective of mixture-of-eeppert with post-training. arXiv preprint arXiv22411.15708 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL https: / arxiv www.i.wwwg.................ooe.e wtrrgg/bb//2000 00000000000000.....
Rajpurkar, P., Zhang. J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Su, J., Duh, K., and Carreras, X. (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 10.18653/v1/D16-1122 1264. URL https: //aclanthology prg/
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C.,

---

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, www.enrwweieweeeeeeeeeeeeeeeeeee Make LoRA Great Again
Processing (EMNLP-IJCNLP) pp. 4463-4473, Hong
Wang, H., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter- efficient Ilm finetuning, 2024b. httpttps:/ //rxiv.. org/ abs /bbs/22 2200 09044.
Kong, China, November 2019. Association for Compu- tational Linguistics. doi: 10.18653/1//D19-144 URL https: https///aaaaaaahhaeee
Si, C., Shi, Z., Zhang, S., Yang, X., Pfister, H., and Shen, W. Unleashing the power of task-specific directions in parameter efficient tinettuning 2024. URL https:// arxiv. org/ abs / 2409. 01035.
Wang. S., Yu, L.,, and J. Lora-ga: Low-rank adaptation with gradient approximation. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024c.
Socher, R., Perelygin, A.. Wu, J., Chuang, J., Manning, C. D., Ng. A. Y., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Wang. Y., Lin, Y., Zeng, X., and Zhang, G. Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXi::2311.1150) 2023.
Wang. Z., Hamza, W., and Florian, R. Bilateral multi- perspective matching for natural language sentences. In International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence, 2017.
Stallkamp. J., Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, pp. 1453-1460. IEEF, 2011.
Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. Lora-pro: Are low-rank adapters properly optimized?, 2024d. URL https: /ttp://aaivvvv//v ort. abs 182.2
Tian, C., Shi, Z., Guo, Z., Li, L., and Xu, C. Hy- dralora: An asymmetric lora architecture for efficient fine-tuning, 2024. WRL.httss://arxi..o 2404. 19245.
Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019, doi: 10. 1162/tacl URL https: httppp:://acaaathooooooo
abs/
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale. S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Moly- bog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subra- manian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / arxiv. org/ abs/ 2307 09288.
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers), pp. 1112-1122, 2018.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
M. Empirical evaluation Xu, B., Wang, N., Chen, T., and Li, M. of rectified activations in convolutional network, 2015. URL https:://///
hatp///aa/155.... 00853.
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng. J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 0at.:2304.122444 12244, 2023.
2019. URL https: id=r km2R5t77.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C.. Li, C., Li, C., Liu, D.,, Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang. J., Tu, J., Zhang, J., Ma. J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue,
Wang, F., Jiang, J., Park, C., Kim, S., and Tang. J. Kasa: Knowledge-award singular-value adaptation of large lan- guage models, 2024a. URL https: : /aaaaii.ooorr org/ abs/2412. 06071.
M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T.
12

---

Make LoRA Great Again
Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang,
Zhu, T., Dong, D., Qu, X., Ruan, J., Chen, W., and Cheng, Y. Dynamic data mixing maximizes instruction tuning for mixture-of-experts arXiv preprint arXiv:2406.11255 2024a.
X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y. Wan, Y. Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z. and Fan, Z. Qwen2 technical report, 2024. URL https: / arxiv... www.i.wrgg/.a....aooo.t...oowtre abs//ab///// 2407. 2467.10671.
10671.
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-moe: Building mixture-of-experts from Ilama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 15913-15923, 2024b.
Yu, L.. Jiang, W., Shi, H., Jincheng. Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large lan- guage models. In The Twelfth International Conference on Learning Representations.
Zadouri, T., Üstün, A., Ahmadian, A., Ermis, B., Locatelli, A., and Hooker, Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference Learning Representations, 2024. URL http:::://////eve/eeeeeeeeeeee/e net/for fortmoorm?id EvDeiLv7ge.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D., and Marquez, L. (eds.), Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.186533/1//199-444 URL https: httpss://acaaatthhhhacc
Zhang, J. Qu, X., Zhu, T., and Cheng, Y. Clip-moe: To- wards building mixture of experts for clip with diversified multiplet upcycling. arXiv preprint arXiv:2409. 19291, 2024.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parameter-efficient fine-tuning. In The Fleventh Interna- tional Conference on Learning Representations, 2023.
Zhao, J., Zhang, Z.,, Chen, B., Wang, 7., Anandkumar, A., and Tian, Y. Galore: Memory-efficien I1m training by gradient low-rank projection, 2024. URL https: //axiiiiv. org/ abs 03507.
Zheng, L., Chiang, W.-L. Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z. Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter:: Integrating code generation with execution and refinement. arXiv preprint ar Xiv: 0 /4658 2024.
Zhong, Y., Jiang, H. Li, L., Nakada, R., Liu, T., Zhang, L., Yao, H., and Wang, H. Neat: Nonlinear parameter- efficient adaptation of pre-trained models, 2024. URL https: //trxivi..vv//////////eee org/ wwggabs/////eeooooooo.ooo.. abs 01870.
13

---

Make LoRA Great Again
A. Pseudocode
Algorithm GOAT
Require: Input vector input dimension n, hyperparameters 7) P. number of experts E Output (o(x) + Ei= w'xx) A))A))
Ensure:
1: function Initialization
2: Scaling factor: $ t V3nn/p
3: SVD decomposition: Wo EVT
4: for to E do
5: B v1/(sp) 1'
6:
Ab f VI/(sp) //////T
7:
end for
8: W Li=1 Bb
9: Wo + Wo W
10: return: Wo, (Bi Ab)
end function
1: function Forward(z)
2:
Compute gating weight (x) (1 sis E)
3:
return: (o(x) Σi (1)) B (x)
4: end function
B. Proof related with PiSSA Select Segment
Lemma B.1. Let Wy E mxx be the pretrained weight matrix with SVD Wo = UEV Assuming m S n and LoRA rank we decompose Wo into rank-r blocks:
Wo Σ U.V.V.
(21)
1=
where 1 = m2 - are block numbers, U, Uprr(t11) R E, ((m(tt)))))))(() E R and
Vi = (1 grxn submatrices of U, U.
We demonstrate that EoV
VJJ has the largest norm and is the best rank-r approximation of Wo-
Proof. By the singular value decomposition (SVD), Wo mi) descending order (a1 2 2 .). For each block V, V,T T, the Frobenius norm can be written as:
o,niy, where o; are singular values sorted in 11=1
((-1).r
| Σ
o
(22)
Since the Frobenius norm satisfies the property of orthogonal invariance, we can simplify this expression:
(6+1)r
JU IF
Σ
of
(23)
j=ir
This result shows that the norm each block V,E,V, depends solely on the singular values o, within the block. As the singular values are sorted descending order (a, 2 02 2 ...), the block UgEn which contains the largest singular values (oI...., J,), has the largest Frobenius norm:
1I...Vo
(24)
14

---

Make LoRA Great Again
By the Eckart-Young-Miske theorem, the best rank-r approximation of Wo minimizes the reconstruction emror:
IWa w ||
min
[
X||F,
(25)
(r)
where V/ Vo Therefore, VV Vo not only has the largest norm but also preserves the most significant information in Wo. making it the optimal rank-r approximation.
C. Load Balance Loss
In Vanilla MoE methods (Fedus et al., 2022; Dai et al., 2024), balance loss Ls mitigates routing collapse by ensuring even token distribution among experts:
C6
R.P
(26)
E
fi
(Token assigned to expert i)
(27)
ET
1 P: T
softmax(2" (xx))
(28)
where T is the number of tokens and 1(-) is the indicator function. Here, fa is the fraction of tokens assigned to expert i, and P: is the average routing probability for expert i. This loss promotes an even distribution of tokens across experts.
D. Proof of Theoretical Results
D.1. Proof of Lemma 2.2
Lemma (2.2). Let 9t be the full-tuning gradient, and B, be low-rank weights. the t-th optimization step, the equivalent gradient can be expressed as:
ge 5² (B.B, at + Tc A.)
(29)
Proof. According to the assumption, W.- W,. Let LoRA sBA where E R A E Rrxa s R, the loss L, the th update of SGD optimizer. We denote W, Wint + sB,A,, we can write the gradient of B, as:
G aL aw, ÖL ow.
aw. AB oW, aB s
(30)
G 0L oW.
aL ow, sB gr
(31)
aW, 0A aw, AA
In the gradient descend algorithm (SVD), the updates for B, and A₃ are
dB, - -ng A, dd,, m01 -87 ge
(32)
The change in the equivalent weight W can be expressed as:
dw oW. dA,
WW: dB
(33)
0A, 0B,
8 B,dA, + .. dB..
(34)
(Bi(-nsBl ge) (- A. )A)
(35)
= =n BB.B. g: + AT A.)
(36)
Therefore, the equivalent gradient 9. is given by:
g. 5 BBB, ge + A A.)
(37)
This concludes the proof.
15

---

Make LoRA Great Again
D.2. Proof of Theorem 3.1
Theorem (3.1). Let the learning rate in Full FT and LoRA be 1JFT, "LoRA- By ensuring equivalent weight Wo s Wo at initialization and maintaining equivalent gradient LoRAR s MeFT9 throughout each optimization step, we can effectively align LoRA with Full FT. (Equivalent weight and gradient are defined in Definition 2.1.)
Proof. We verify this alignment using induction. The equivalent weight is defined as W, Wint - sB, and the aL
equivalent gradient is It- aw- Using the gradient descent algorithm (considering only the SGD optimizer), we have:
W++1 W, T/FFT
(38)
W4+1 W. 1L.RAA
(39)
Base Case (t. = 0): We have ensured Wo Wo- Inductive Step: Assume W. W, and g: 9t. Then:
We+1 W, -TLLRAe
(40)
W, - -IFFTg
(41)
W1+1-
(42)
By induction, W, W, for all 4, ensuring the alignment between LoRA and Full FT.
D.3. Proof of Theorem 3.2
Theorem (3.2). Let the learning rate in Full FT MoE and LoRA MoE be T/FT, TLORA For all [1, [l...., E}. by ensuring the equivalent weight of the i-th expert Wi W initialization and maintaining the equivalent gradient of the i-th expert MORRL NEFTT throughout each optimization step, can effectively align LoRA MoE with Full FT MoE.
Proof. We aim to show that under the given conditions, the LoRA MoE aligns with the Full FT MoE by effectively making the MoE routers behave identically in both models.
Base Case (t = 0): At initialization, by assumption, the equivalent weights each expert satisfy Wi Wi because our Full FT MoE is an upcycling MoE which makes all Wi = Wo. Additionally, since both models use the same random seed, the routers are initialized identically, ensuring that the routing decisions are the same for both Full FT MoE and LoRA MoE.
Inductive Step: Assume that at step t, the equivalent weights satisfy W W for all i, and the routers in both models are identical. During the t-th optimization step, the gradients are scaled such that MoRA NFFTS/ This ensures that the weight updates for each expert in both models are equivalent:
W441 = W -TORAA Wt merig W
(43)
16

---

Make LoRA Great Again
First, the routers identical, the router weight w the same, the layer output is the same:
MoB(x)
w (x)
(44)
Σ w (x)
(45)
R
Σ (x)(W + +B''A')
(46)
((x) Σ (x(xx A(x))
(47)
(XFLORA(x)
(48)
Since the weight updates are equivalent and the routers are optimized from the output induced by these weights, the routers remain identical at step t + Therefore, by induction, the routers are identical for all
With identical routers, the routing decisions do not differentiate between Full FT MoE and LoRA MoE layers. Consequently, the alignment of individual experts (as established by Theorem 3.1) ensures that the overall behavior of both MoE variants effectively aligned.
D.4. Proof of Lemma 3.3
Lemma (3.3). Let (x) be the set of indices corresponding to the top-k largest values of 2 2x(x), and xx(x) are independent and identically distributed (i.idd),, and S E. w is defined as:
exp(=*(x)) w (x) = E tse (x))
if lu(x),
(49)
if $ lx(x),
We demonstrate the following properties for all i,j E [1...., El (i + j):
Bx(ww(xx) =
(50)
E.
Varx (w''(w"e (xx)) E k
(51)
kE
Proof. Because the (x) are i.idd random variables, any permutation of the indices {I...., E}} leaves the joint distribution of {2 (x)...... 2 <x()) unchanged. The Top-K operation (pick the indices of the largest K logits) is also symmetric with respect to permutations: permuting (=+,..., = accordingly permutes the set Pu (x(x) of selected indices. Because of this symmetry, each wxx) is distributed in the same way as w/ (x) for any j. By definition of w (xx) we have Vx, D w²(xi = 1, so:
Elw Elw [[w] (x) = E
Σ
w
(x))
B[1] =1
(52)
1=1
[x([x((x)] Ex [u (x)]
1
Vi E [1...., E
(53)
E.
The variance of (x()) is given by:
Var (w' (x)) = Ex
- Ex ww (xx)
(54)
17

---

Make LoRA Great Again
Since Ex |oo (x)] we have:
(arx(w" Ex
(x))
(55)
E
We aim to compute Ex (x)) but it's tricky to directly obtain this expectation. Given that Σ Wz 1, we can expand this expression. Omitting the x for simplicity, we get:
Σ
= E
Σ
W2
=
E
+ Σ
Efwawy,
(56)
1 E Elu E(E 1) Eizs [uruuy]-
(57)
where [[wwey is the expectation we need to compute. This expression is derived based on the rotational symmetry of wo, wy. which means the cross-term B[w,w,] is the same for all distinct * j.
To compute E[w;w,], rewrite the weights w; follows:
exp 24
W1
Eyen, exp
Eyen, Us
(58)
where
exp
if E (xx),,, if & (x))
(59)
0
Thus, the product wytty becomes:
MKWj
wiyy
(60)
Eyen, M)
Now, due rotational symmetry of the terms ys. My we can compute:
E Elu, wy]
31-35
k(k -1) 1)
(61)
E(E 1) 12
E(E 1)
Substituting this back into Equation (57) for E(we]]
E Elui]
E(E- 1) E(E 1)k
(62)
we get:
Biwl] 1
(63)
Ek
Thus, the variance of w' in Equation (55) is:
Var(w")
(64)
Ek E kEa
18

---

Make LoRA Great Again
D.5. Proof of Theorem 3.4
Theorem (3.4). Consider the optimization problem:
W arg min Ex Wis
W nos Σ w i=
(x)
(65)
The closed-fore forem-fommmmoo solution is W res
Ee= BB
Proof. W denotes the optimal value of W The solution this optimization problem, Wress can be derived the expected value over all possible
W = sBx
w (b.
(66)
Ex Ex/w' (x)) BB
(67)
A
(68)
E
where Equation (66) use the linear property of expectation and Equation (67) utilize Lemma 3.3.
D.6. Proof of Theorem 3.5
Theorem (3.5). Consider the optimization problem where Bo 0 and Ao ~ U 5² (BiBi" si + gi At). the ratio between full tuning learning rate vs. LoRA learning rate 1).
6 (--V3
6
Øt
Vi [1, E min g6 si|
(69)
The closed-form solution is 3mm
Proof. By analyzing the first step gradient,
Go =BBodd + G Ao) = BBo go + Ao Ao)
(70)
arg min
Bo go + 90 Ao) 1/90
(71)
mank
As LoRA init Bg 0 and Ao UU-V-_ n 1). The above equation becomes
arg min
Ag Ao
790
(72)
22
19

---

Make LoRA Great Again
First We notice that the matrix Ag Ag can express the entries in the following way
Ao Abli, = Σ Aoli, Ao [k:
(73)
For the diagonal entries (i j). the formula simplifies to:
(Ag Ao).. Σ
Abaaa aa
(74)
k=1
This is because the entries of Ap are i.idd with mean O and variance AA, we can compute:
Ab) Σ B[A0.IAl FOA TOA
(75)
For the non-diagonal entries (i * f), the formula is:
(Ag Ao)as
Ao [.. AJAo 0
(76)
Since [i
[i, k] and j] are independent random variables (for i j), their product has an expected value of zero.
RA, [A] Ao] rax
(77)
Given that EA, [AD Ao] = r
Lxx (use Leaky ReLU (Xu et al., 2015) with negative slope v5, that Var(A) = 3n ). we
3m
3mn get V
2r 3m
3nz
go
I nI
=
(78)
Though derived by the first step gradient, as in practice, the weight change low-rank update hypnosis in Hu et al. (2021)), we can consider the following steps.
WW is typically small (thus has the B is small, so the above s can extended to
and
||
d.
A
E. Extend Our Method to Scenarios with Proper Scaling
GOAT assumes a scenario where LoRA MoE has not been properly scaled. Here, we supplement it with an extended approach for scenarios where proper scaling has been applied.
Here, we assume that the routing strategy of the fully fine-tuned MoE aligns with our method. Since the router is non- differentiable, we ignore its impact and focus solely the gradient of each expert. Our goal is to align the gradient of each expert method with that of the fully fine-tuned MoE. Thus, for the i-th expert, aim to solve:
arg min
?
Bi BBB' u gb + giAt Ab)
gú
(79)
meke22
When using the balanced initialization strategy, the above equation can be rewritten as:
arg min
2? (nn 0, + 960931. mo
90
(80)
2r
20

---

Make LoRA Great Again
If each expert has rank 1, the equation be further simplified to:
arg min of (unu gô g6oe ws 90
(81)
From this, we observe that acts scaling factor for the gradient, stretching compressing the direction represented by the current expert during optimization. Here, we assume that the hyperparameters have already been correctly scaled for the first expert (which corresponds to the optimal low-rank approximation of the original matrix), aligning it with the first expert of the fully fine-tuned MoE. Since the stretching strategy for the direction represented by each expert should remain consistent during MoE fine-tuning, we need to align the scaling factors s; the other experts to reduce the gap between our method and full MoE fine-tuning. Specifically, 8; must satisfy the following condition:
0:00 = =600 6.
(82)
Thus, we transform each 8; as follows:
V du
(83)
Vo:
When the rank each expert is greater than 1, we approximate the solution by using the sum the singular values within the segment.
Here, we modify the scaling of all experts except the first one, while keeping other initialization methods consistent with 19,
We refer to this extended method as GOAT-s, and its performance across all benchmarks is presented in Table 8. While designed for different scenarios, demonstrates performance comparable to GOAT.
Table Performance comparison our method extended to properly scaled scenarios.
Method NLG(Avg.)
NLU(Avg.)
IC(Avg.)
CR(Avg.)
Avg.
GOAT 30.60
89.76
81.49
82.64
71.12
GOAT-s
30.54
89.61
81.54
82.41
71.02
F. Further Discussion
F.1. Discussion on the Practical Applications and Real-world Impact of LoRA MoE
MoE is popular for managing large parameters while activating only a sparse subset during inference, making it ideal for large-scale models. However, in Section 4.9 and Table 7, without optimization. fully fine-tuning an MoE model significantly increases trainable parameters and FLOPss compared to Full FT.
LoRA MoE addresses these challenges by replacing experts with low-rank matrices, reducing computation, preserving MoE benefits, and enabling faster training, lower memory usage, and reduced energy consumption crucial for resource-limited or real-time applications.
For instance, in NLP, where large-scale models are common, LoRA MoE achieves SOTA performance with lower computa- tional cost. This efficiency benefits industries like autonomous driving, healthcare (Tian et al., 2024), where lower latency and costs enhance performance and scalability.
Overall, LoRA MoE balances MoE's model capacity with cost-effective deployment, making adaptable to various real-world applications.
21

---

Make LoRA Great Again
F.2. Analysis of the Phenomenon Where PiSSA and MiLoRA May Perform Worse Than LoRA in Experiments
Previous works (Wang et al., 2024b;a) show that PiSSA and MiLoRA don't always outperform LoRA, KaSA found that PiSSA accelerates convergence but uses limited pre-trained knowledge at lower ranks, limiting performance. Similarly, MiLoRA's minimal adjustments to pre-trained weights often fail to improve over LoRA. In Table 1, adopt the same rank settings KaSA and reach the same conclusion.
In contrast, our method consistently achieves superior performance across both low and high ranks by effectively balancing convergence speed and final performance, as demonstrated in Tables 1,2,3,4 and Figure 5.
E33 Compared to Other Routing Techniques
As shown in Table 9, we extend our experiments to include alternative routing strategies such as top-p routing and a top-k variant with shared experts. We find that, compared to other approaches, setting it = 2 achieves the best performance. We will incorporate these our revised version of the paper.
Table 9. Comparison of Routing Strategies on Average Accuracy
Routing Strategy
Avg. ACC (%)
Ours (top-k = 2)
81.49
Top-p (p = 0.25)
79.40
Top-k Shared Expert
78.68
F.4. Analysis of the coefficient for the balance loss
We use top-k routing with k=2 and set the coefficient for the balance loss to 1e-3. As shown in Table 10, we attach the load-baldd ballnbnanning loss coefficient experiment by activating 2 out 8 experts on Cars. We can observe that setting the coefficient
Table 10. Performance Comparison under Different Coefficient Values
Coefficient
GOAT
MoLoRA
HydraLoRA
1 10 49.09
49.02
48.45
1 x 10
50.52
49.33
49.45
x 1
53,50
50.83
48.42
10-4
51.53
49.03
48.52
0
49.85
48.02
49.06
too low (e.g., 0 or Ie-4) leads to expert imbalances, which in turn degrades performance. Conversely, excessively high coefficients (e.g.. 0.01 or 0.1) can disrupt the normal learning process. Our results show that coefficient of le-3 achieves the best tradeoff in GOAT/MoLoRA between balancing expert load and maintaining stable learning.
Notably, GOAT consistently outperforms across all tested coefficients, demonstrating its robustness in these settings.
F.5. Is expert activation balanced without the coefficient for the balance loss?
Table 11. Expert Activation Distribution of GOAT without Load Balancing
Method
fl
f2
f5
f6
GOAT w/o Load Balance
0. 1043
0.1379
0.1275
0.90
O. 0.02 O. 0.1405
0.1259
0.1338
We further ablate the load-balee balancing component GOAT (2in8, Cars task) and observe that all experts remain active (see Table 11), indicating that each SVD chunk contributes meaningfully to the final representation.
22

---

Make LoRA Great Again
G. Experiment Details
G.I. Dataset details
Natural Language Understanding Tasks, We evaluate our model on the following natural language understanding tasks from the GLUE benchmark (Wang et al., 2019):
1. CoLA (Warstadt al., 2019): A binary classification task that requires determining whether a given sentence is grammatically acceptable.
2. SST-2 (Socher al., 2013): A sentiment analysis task where the goal is classify sentences expressing positive or negative sentiment.
3. MRPC (Dolan & Brockett, 2005): A binary classification task focused on identifying whether two sentences in a pair are semantically equivalent.
4. QQP (Wang et al., 2017): A binary classification task to determine whether two questions from Quora have the same meaning.
5. MNLI (Williams et al., 2018): A textual entailment task that involves predicting whether a hypothesis entailed, contradicted, or neutral with respect a given premise.
6. QNLI (Rajpurkar et al., 2016): A binary classification task to determine whether question is answerable based on a given context.
7. RTE (Giampiccolo et al., 2007): A textual entailment task where the goal to predict whether a hypothesis logically follows from a given premise.
We report the overall accuracy (including matched and mismatched) for MNLI, Matthew's correlation coefficient for CoLA, and accuracy for all other tasks.
Natural Language Generation Tasks. We evaluate our model on the following natural language generation tasks:
1. MT-Bench (Zheng et al., 2023): A benchmark for evaluating dialogue generation capabilities, focusing on multi-turn conversational quality and coherence.
2. GSM8K (Cobbe et al., 2021): A mathematical reasoning task designed to assess the model's ability to solve grade school-level math problems.
3. HumanEval (Chen et al., 2021): A code generation benchmark that measures the model's ability to write functional code snippets based on natural language problem descriptions.
Following previous work (Wang et al., 2024c), we evaluate three natural language generation laks-dialogue mathematics, and code using the following three datasets for training:
1. Dialogue: WizardIM (Xu et al., 2023): WizardL.M leverages an AL-driven approach called Evol-Instruct. Starting with a small set of initial instructions, Evol-Instruct uses an LLM rewrite and evolve these instructions step by step into more complex and diverse ones. This method allows the creation of large-scale instruction data with varying levels of complexity, bypassing the need for human-generatee data. We use a 52k subset of WizardLM to train our model for dialogue task (MTT-encch).
2. Math:
MetaMathQA (Yu et al.): MetaMathQA is a created dataset designed specifically improve the mathematical reasoning capabilities of large language models. We use 100k subset of MetaMathQA to train our model for math task (GSM8K).
3. Code: Code-Feedback (Zheng et al., 2024): This dataset includes examples of dynamic code generation, execution, and refinement guided by human feedback, enabling the model to learn how to improve its outputs iteratively. We use a 100k subset of Gode-Feedback to train our model for code task (HumanEval).
We evaluate performance on GSM8K using Exact Match, HumanEyal using Pass@1 1, and MT- using the First-Turn Score assessed by GPT-4.
Image Classification Tasks. We evaluate our model on the following image classification tasks:
1. SUN397 (Xiao et al., 2016): large-scale scene classification dataset containing 108,754 images across 397 categories, with each category having least 100 images.
2. Cars (Stanford Cars) (Krause et al., 2013): car classification dataset featuring 16, 185 images across 196 classes,
23

---

Make LoRA Great Again
evenly split between training and testing sets.
3. RESISC45 (Cheng et al., 2017): remote sensing scene classification dataset with 31,500 images distributed across
45 categories, averaging 700 images per category.
4. EuroSAT (Helber et al., 2019): A satellite image classification dataset comprising 27,000 geo-referenced images
labeled into 10 distinct classes.
5. SVHN (Netzer et al., 2011): A real-world digit classification dataset derived from Google Street View images, including
10 classes with 73,257 training samples, 26,032 test samples, and 531,131 additional easy samples.
6. GTSRB (Stallkamp et al., 2011): A traffic sign classification dataset containing over 50,000 images spanning 43 traffic
sign categories.
7. DTD (Cimpoi et al., 2014): A texture classification dataset with 5,640 images across 47 classes, averaging approximately 120 images per class.
We report the accuracy in all tasks.
Commonsense Reasoning Tasks We evaluate our model on the following commonsense reasoning tasks:
1. BoolQ (Clark et al., 2019): A binary question-answering task where the goal is to determine whether the answer to question about given passage is "yes" or "no."
2. PIQA (Physical Interaction Question Answering) (Bisk et al., 2020): Focuses reasoning about physical commonsense to select the most plausible solution to given problem.
3. SIQA (Social IQa) (Sap et al., 2019): Tests social commonsense reasoning by asking questions about motivations, reactions, outcomes in social contexts.
4. HellaSwag (Zellers et al., 2019): A task designed test contextual commonsense reasoning by selecting the most plausible continuation of given scenario.
5. WinoGrande (Sakaguchi al., 2021): pronoun coreference resolution task that requires reasoning over ambiguous pronouns in complex sentences.
6. ARC-e (AI2 Reasoning Challenge Easy) (Clark et al., 2018): A multiple-choice question-answering task focused on elementary-leved science questions.
7. ARC-c (AI2 Reasoning Challenge Challenge) (Clark et al., 2018): more difficult subset of ARC, containing questions that require advanced reasoning and knowledge.
8. OBQA (OpenBookQA) (Mihaylov al., 2018): question-answering task requiring reasoning and knowledge from small "open book" of science facts.
We report the exact match accuracy in all tasks.
G.2. Baseline details
Full-Finetune
1. Full FT refers to fine-tuning the model with all parameters.
2. Full FT MoE refers to fine-tuning all parameters within Mixture of Experts (MoE) architecture.
Single-LoRA baselines
1. LoRA (Hu et al., 2021) introduces trainable low-rank matrices for efficient fine-tuning.
2. DoRA (Liu et al., 2024) enhances LoRA by decomposing pre-trained weights into magnitude and direction, fine-tuning the directional component to improve learning capacity and stability.
3. PiSSA (Meng al., 2024) initializes LoRA's adapter matrices with the principal components of the pre-trained weights, enabling faster convergence, and better performance.
4. MiLoRA (Wang al., 2024b) fine-tunes LLMs by updating only the minor singular components of weight matrices, preserving the principal components to retain pre-trained knowledge.
scaling factor make the scale of the output invariant rank 6. LoRA-Dash (Si al., 2024) enhances PEFT by leveraging task-specific directions (TSDs) to optimize fine-tuning efficiency and improve performance on downstream tasks.
5. rsLoRA (Kalajdzievski, 2023) introduces
7. NEAT (Zhong et al., 2024) introduces nonlinear parameter-efficient adaptation method to address the limitations of
24

---

Make LoRA Great Again
existing PEFT techniques like LoRA.
8. KaSA (Wang al., 2024a) leverages singular value decomposition with knowledge-aware singular values to dynami- cally activate knowledge that is most relevant the specific task.
LoRA MoE baseliness
1. MoLoRA (Zadouri et al., 2024) combines the Mixture of Experts (MoE) architecture with lightweight experts, enabling
extremely parameter-efficient fine-tuning by updating less than 1% of model parameters.
2. AdaMoLE (Liu & Luo, 2024) introducing adaptive mechanisms to optimize the selection of experts.
3. HydraLoRA (Tian et al., 2024) introduces an asymmetric LoRA framework that improves parameter efficiency and performance addressing training inefficiencies.
G.3. Abaltion details
Here, we provide a detailed explanation of the construction of each initialization method. Suppose h min(m, n2),
1. Ours (0): &, (UpAAt++d) ( 2( (ise-da)) le 1)t, 1....,..................... E}
2. Principal (P): & (U ddskkt V ( k = - (j-i)d,j = 1, 1,...,
3. Minor (M))&, (U (U 2. V (kotddd)
k= ---hjj jd,j 1,.... .F}
4. Random (R):E, = (U..et-d) 2e V lock
tjjt t- = = random((, 2/2 1).j = ..... E 6-11 1}
G.4. Implementation Details
Image classification and natural language understanding experiments conducted on 8 Nvidia 4090 GPUs with 24GB of RAM each. Commonsense reasoning and natural language generation experiments are conducted on single Nvidia A100 GPU with 80GB of RAM. For training and evaluating all models, we enabled 6f16 precision.
Table 12, Hyperparameters the commonsense reasoning task GOAT.
Hyperparameter Commonsense Reasoning
Batch Size
16
Rank
32
Alpha
64
Optimizer
Adam W
Warmup Steps
100
Dropout
0.05
Learning Rate
le-4
Epochs
3
Table 13. Hyperparameters of the image classification task for GOAT.
Hyperparameter
Cars
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Batch Size
512
Rank
8
Alpha
16
Optimizer
Adam W
Warmup Steps
100
Dropout
0.05
Learning Rate
lc-4
Epochs
35
76
11
15
14
4
G.S. Hyperparameters
We fine-tune our model on each task using carefully selected hyperparameters to ensure optimal performance. Specific details for each task, including learning rate, batch size, number of epochs, and other configurations, are provided to ensure
25

---

Make LoRA Great Again
Table 14. Hyperparameters of the natural language understanding tasks for GOAT.
Hyperparameter
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
Batch Size
256
Rank
8
Alpha
16
Optimizer
Adam W
Warmup Steps
100
Dropout
0.05
Learning Rate
le-4
Epochs
10
10
10
10
10
10
50
Table 15. Hyperparameters of the natural language generation task for GOAT.
Hyperparameter
Natural Language Generation
Batch Size
32
Rank
8
Alpha
16
Optimizer
Adam W
Warmup Steps
100
Dropout
0.05
Learning Rate
2c-5
Epochs
5
reproducibility and consistency across experiments. These details are summarized in Table 12, Table 13, Table 14 and Table 15. We p 10. The ratio between the full fine-tuning learning and the LoRA learning rate v is empirically for ViT/RoBERTa. In the LLaMA experiments, when using learning rate the 1e-4 level, for learning rate at the 1e-5 level, we set v 1. This configuration aligns with common practice, where LoRA tuning typically uses learning rate around 1e-4, while FFT- based methods operate at a lower learning rate near 1e-5. We set the coefficient for the balance loss to le-3 our LoRA-MoF experiments. In our LoRA-MoE setup, we use a top-k routing strategy with 2, which as shown in Table and Figure 6, outperforms other strategies. The same routing strategy also adopted by all LoRA-MoB baselines.
1j 0.1;
H. Parameter and FLOPs Analysis
H.1. Parameter Analysis
Here, provide parameter analysis for each baseline and our method based on different backbones. We assume H represents the model dimension, denotes the rank, indicates the number of experts, L indicates the number of layers, V indicates the vocabulary size, P indicates the patch size in ViT and indicates the number of channels in VIT. The analysis for RoBERTa-large, ViT-base, and LLAMA2 7B is as follows:
RoBeRTa-large: H = 1024,r r 32, = 2, 24, V = 50265. The activation parameters are dense from all
attention and MLP layer.
1. FFT (Full Fine-Tuning):
Total Parameters: (12HF+ VVH
Breakdown:
Embedding layer: VH
Attention mechanism: 4H2 4H
MLP layer: 8/22 + 5/l
LayerNorm (2 layers): 4H
Total per layer: 12H² 13H
2. Full FT MoE:
Total Parameters: (12eH2 + 2H +He)
26

---

Make LoRA Great Again
Proportion: 698%
3. LORA/ISSANMILORRVSLOORAA
Total Parameters: 18HrL
Proportion: 4.00%
4. DoRA:
Total Parameters: (18Hr + 6)L
Proportion: 4.00%
5. MoL.RA/GOAT:
Total Parameters: (18/fr + 9H1e)
Proportion: 4.50%
Breakdown:
Attentioo mechanism: 8Hr + 4He
MLP layer: 10Hr 5He
Total per layer: 18/r + 9He
6. HydraLoRA:
Total Parameters: (9Hr + 9He + 9Hr/e)L
Proportion: 2.75%
7. AdaMoLE:
Total Parameters: (18Hr + 9He 9H)
Proportion: 4.56%
ViT-base:
= 768, 8, -,, P 32, C 3. The activation parameters include a, k, v, o, fel,
fc2.
1. FFT:
Total Parameters: (C+1)) + (12H1 2H) - 3H + PH + 1
Breakdown:
Embedding layer: PH + H + (C+ 1
encoder (L layers): (12H² 2H)L
LayerNorm (1 layers): 2H
Pooler: H²
2. Full FT MoE:
Total Parameters: (C+1)PPHH+ + (12el1² + 2H + 9He 3H + PH H?
Proportion: 770%
3. LoRA/PiSSSAMMLLORA::
Total Parameters: 18HrL
Proportion: 1.49%
4. LoRA (rank=16):
Total Parameters: 18HrL
Propprtion: 2.99%
5. LoRA (rank=32):
Total Parameters: 18HrL
Proportion: 5.98%
6. DoRA:
Total Parameters: (18Hr L)
Prroportion: 1.49%
7. MoLORA/GOAT:
Total Parameters: +
27

---

Make LoRA Great Again
Breakdown:
Attention mechanism: 8Hr + 4Hf
MLP layer: 10Hr 5He
Total per layer: 18//r + 9/f
Proportion: 2.24%
8. HydraLoRAA
Total Parameters: (9Hr +
Proportion: 1.58%
9. AdaMoLE:
Total Parameters: [18ff+ + 9
Proportion: 2.33%
LLAMA2-7B: H 4096, r 32, 2, 32, V 322000 The activation parameters g, V. up, down.
1. FFT:
Total Parameters: (10.25H7 2HH H + 2VH
Embedding layer and LM head: 2V H
Attention mechanism: 2.25H²
MLP layer: 8H²
RMSNorm (2 layers): 2H
Additional RMSNorm (last layer): H
Total layer: 10.25H + 2H
2.
Total Parameters: 11.58HrL
Proportion: 0.84%
3. DoRA:
Total Parameters: (11.58Hr + 5
Proportion: 0.84%
4. NEAT:
Total Parameters: (11.58/r+ 1
Prroportion 0.84%
5. MoL.ORA/GOAT:
Total Parameters: (11.58Hr + L.
Attention mechanism: 4.25Hr 3He
MLP layer: 7.33 + 3.66/f
Total per layer: 11.58/r 6.66He
Proportion: 0.96%
6. HydraLoRA:
Total Parameters: (4.91 -
Proportion: 0.84%
7. AdaMoLE:
Total Parameters: (11.58Hr 6.66/6666666.. 6.6 (.
Proportion: 0.97%
H.2. FLOPs Analysis
Here, we mainly analyze the forward FLOPs. Since LLaMA 2 7B uses GQA (Grouped Query Attention) and SwiGLU FFN, the calculation of FLOPs differs from that of standard Transformers. Here, we assume that all linear layers in the Transformer block are extended with MoE (Mixture of Experts). We assume H represents the model dimension, s denotes sequence lengths, denotes each expert rank, indicates the number experts, total rank r = ed.L indicates the number of layers, V indicates the vocabulary size. Notice each MAC (Multiply-Accumill Operations) counts as two FLOPs.
28

---

Make LoRA Great Again
FLOPs for FT MoE:
1. MoB linear for and The FLOPs calculated 2 (2Bs/le + k. 2BsH(²).
2. MoE linear for k and o: Since LLaMA 2 7B's GQA reduces the number of heads for & and to 1/8 of q's heads, the FLOPs are: 2- (2 k. 2BsHHH/)
The FLOPs for q and score remain independent k; as we only upcycle the linear projection to copies. The FLOPs for these operations are 2Bs² H 22 H.
4. MoE linear for down and gate: Since LLaMA 2 7B uses SwiGLU FFN, the FLOPs 2 (2BsHe++ k- 2BsH 8/3H).
5. MoE linear for up: The FLOPSs are: 2Bs 8/3He + k 2Bs 8/3HH.
Across L layers, including the vocabulary embedding transformation, the total FLOPs are:
FLOP FT BL esH + 41 ksHH + 4) + 2BBHV
52
(84)
FLOPs for GOAA//NoLORA/HyyaaLLORR:
MoE linear for and o: The FLOPPs are calculated as 2B. (2sH1² + 2e8H + 2k(sHd + sHd)).
MoE linear for & and u: Consider the effect of LLaMA 7B's GQA on and v: 2B (2sH2//8 + 2esHl + 2k(s/dd + sHd/8)).
3. FLOPs for q k and score The FLOPs for these operations 2Bs H 2B
4. MoE linear for down and gate: Since LLaMA 7B uses SwiGLU FFN, the FLOPs are: 2B 12-H2228/3H 2esH + 2k (sHd + sd8/3H))
MoE linear for up: The FLOPs are: 2BsH8/3F + 2Bs8/3He + 2 (Bs8/3Hd + BsrH).
Across L layers, including the vocabulary embedding transformation, the total FLOPs are:
FLOPS Mot
BL
EsH
sH 4
ksHd
2BsHV
(85)
52 sH + 4s
69 k
BL esH +
HHr + +B&HV
(86)
2 e
29


Reviews:
[{"review_id": "29xSC9gUbv", "reviewer": "Reviewer_ETXs", "summary": "The paper proposes a novel fine-tuning framework for LoRA (Low-Rank Adaptation) MoE (Mixture-of-Experts). Two challenges identified in the paper: 1) how to design an effective initialization for the matrices  A  and  B across different experts. 2)  unaligned optimization leads to large gradient gap and slow convergence rate. \nAccordingly, the paper first proposes initializing LoRA MoE experts with distinct singular value segments, allowing the router to select the appropriate prior information. It then derives an optimal weight alignment strategy and a theoretical scaling scheme to improve gradient alignment.\nExtensive experiments on 25 tasks demonstrate that the method's superiority while maintaining scalability. Compared with Full fine-tuning, the proposed method shows comparable or even better performance.\n\n### update after rebuttal\nMy concerns about experiment  evaluations are mostly addressed. Thus, I remain positive about this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `ETXs`\n\n> Q1: The performance evaluation metrics are unclear.\n> \n\nSorry for the confusion. Here is a more detailed explanation of our performance metrics:\n\n- NLU & CV: Accuracy, except for CoLA (Matthew’s correlation). See Appendix E.1 for details.\n- CommonSense: Exact match.\n- NLG: GSM8k(Exact match), Human-Eval(Pass@) and mtbench: (First-turn score (GPT-4))\n\nWe will incorporate these clarifications into our revised version of the paper.\n\n> Q2: Clearer description and analysis of the MoE router’s strategy and behavior, especially regarding latency and potential biases, would be valuable.\n> \n\nTo clarify, we use a top-k routing strategy (Eq(10) and (12)), where each token selects the top k experts based on the highest router logits.\n\nWe offer insights about topk hyperparameter, routing biases and latency in Figure 6, 7, and Table 7:\n\n- **Top‑k Hyperparameter:** Figure 6 shows performance vs. the k/E (E is total number of experts) activation ratio in top-k routing. Activating 2 out of 8 experts balances sparsity and performance, so we use this setting in Tables 1–4.\n- **Routing Biases and Load Balance**: Figure 7 shows token distribution across experts. CV and NLU tasks exhibit balanced expert usage, while NLG tasks favor the first two experts, suggesting larger SVD chunks play a key role in complex generation, aligning with PiSSA’s insights.\n- **Latency:** Section 4.9 (Computation Analysis) and Appendix F.1 provide a detailed breakdown of latency and computational efficiency.\n\n> Q3: The paper lacks an adequate introduction to the practical applications and real-world impact of LoRA MoE.\n> \n\nThanks for your advice. We will incorporate additional discussion on the practical applications and real-world impact of LoRA MoE into the revised version of the paper.\n\nMoE is popular for managing large parameters while activating only a sparse subset during inference, making it ideal for large-scale models. However, in Section 4.9 and Table 7, without optimization, fully fine-tuning an MoE model significantly increases trainable parameters and FLOPs compared to Full FT.\n\nLoRA MoE addresses these challenges by replacing experts with low-rank matrices, reducing computation, preserving MoE benefits, and enabling faster training, lower memory usage, and reduced energy consumption—crucial for resource-limited or real-time applications.\n\nFor instance, in NLP, where large-scale models are common, LoRA MoE achieves SOTA performance with lower computational cost. This efficiency benefits industries like autonomous driving, healthcare[1], where lower latency and costs enhance performance and scalability.\n\nOverall, LoRA MoE balances MoE's model capacity with cost-effective deployment, making it adaptable to various real-world applications.\n\n[1]Hydralora: An asymmetric lora architecture for efficient fine-tuning(NeurIPS2024)\n\n> Q4:This paper doesn’t report results on multi-domain datasets.\n> \n\nWe actually conducted experiments on Commonsense using a multi-domain setting. In Table 3, our evaluation method for commonsense reasoning follows a classic multi-domain setting, following prior work[1]. We train on a 170K multi-task mixed dataset and evaluate on **8 datasets**. Our approach outperforms the single LoRA method by at least 1.2 points and the LoRA MoE method by at least 1.6 points.\n\n[1] DoRA: Weight-Decomposed Low-Rank Adaptation(ICML2024)\n\n> Q5:An analysis of why PiSSA and MiLoRA perform worse than LoRA in Table 1.\n> \n\nTo clarify, previous works[1,2] show that PiSSA and MiLoRA don't always outperform LoRA. KaSA found that PiSSA accelerates convergence but uses limited pre-trained knowledge at lower ranks, limiting performance. Similarly, MiLoRA’s minimal adjustments to pre-trained weights often fail to improve over LoRA.  In Table 1, we adopt the same rank settings as KaSA and reach the same conclusion.\n\nIn contrast, our method consistently achieves superior performance across both low and high ranks by effectively balancing convergence speed and final performance, as demonstrated in Tables 1–4 and Figure 5.\n\n[1]MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (NAACL2025)\n[2]KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR2025)\n\n> Q6. Do you explore alternative routing techniques in LoRA MoE, and how do they affect performance?\n> \n\nThanks for your suggestion. In our paper, we use the commonly adopted top-k routing and set k=2 based on the analysis in Figure 6. we primarily adopt top‑k routing. We extend our experiments to include alternative routing strategies such as top-p routing and a top‑k variant with shared experts below.\n\n| Routing Strategy | avg.  ACC |\n| --- | --- |\n| **Ours(top-k=2)** | **81.30** |\n| top-p(0.25) | 79.40 |\n| top-k + share expert | 78.68 |\n\nWe find that, compared to other approaches, setting k=2 achieves the best performance.We will incorporate these into our revised version of the paper.", "claims_and_evidence": "The claims are mostly supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-aligned with the problem. A better LoRA initialization method is crucial for narrowing the performance gap between parameter-efficient tuning and full fine-tuning. It is commendable that the proposed approach is validated on extensive CV and NLP tasks.", "theoretical_claims": "Yes, I have checked the theorems in Section 3.3. Theoretical Optimization Alignment, both the initialization and gradient alignment. The claims seem reasonable.", "experimental_designs_or_analyses": "I have checked the experiment part. The performance evluation metrics are not clear in Table 1-4.", "relation_to_broader_scientific_literature": "The intialization of AB matrices in LoRA is a rising topic and has been studied in previous methods, including PISSA [1] (tuning the principal components) and MiLoRA [2] (tuning minor singular components). For LoRA-MoE, the initialization has not been well studied, the proposed method divides the SVD of W into different segments and allocate segments to different experts. From this perspective, the proposed method is a timely study.\n\n[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models\n[2] Milora (MiLoRA): Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths：\n\n1.\tThe motivation is clear. Relying solely on either the principal or minor singular values is not ideal and does not guarantee optimal performance across different datasets. In LoRA MoE, different experts can be responsible for different parts of the task. Moreover, the effects of the scaling factor are also crucial for optimization\n\n2. Evaluations are quite comprehensive. Significant Improvement over othe LoRA MoE methods, and in NLP benchmarks the proposed method is even better than Full FT. The proposed method also shows good scalability across different rank and different expert numbers.\n\nWeakness:\n\n1. The routing strategy in the proposed method is not clear. In Formula (12), it introduces a soft routing stratey, while in Figure 6, it shows the results of different activation ratio. A more clear description and analysis of the MoE router’s behavior would provide valuable insights, particularly in terms of its latency, and potential routing biases.\n2. The paper does not adequately introduce the practical applications of LoRA MoE, nor does it sufficiently demonstrate the real-world impact of the proposed method.\n3. To my knowledge, existing approaches typically train multiple LoRA experts for different tasks. However, this paper does not report results on multi-domain datasets, especially in Image classification and NLU tasks, which limits its practical relevance.", "other_comments_or_suggestions": "It would be better to incoporate the practical importance of LoRA MoE in the introduction by highlighting its real-world applications, like multi-task or multi-domain scenarios.", "questions_for_authors": "1. In table 1, the single LoRA method comparison part, methods like PiSSA and MiLoRA achieves worse performance than LoRA. Are there any analysis on this phenomenon?\n2.  Do you experiment with alternative routing techniques in LoRA MoE? Could you discuss how these different strategies impact performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KX3NPdonAt", "reviewer": "Reviewer_9dgB", "summary": "This paper presents GOAT (Great LoRA Mixture-of-Experts), a novel framework to enhance the LoRA MoE structure for fine-tuning LLMs. GOAT (1) adaptively initializes each expert using different SVD segments to integrate relevant priors from pre-trained models, and (2) derives a theoretical scaling factor that aligns LoRA MoE optimization with Full FT by minimizing gradient misalignment. Experiments across 4 multi-task benchmarks demonstrate GOAT’s superior performance than existing LoRA MoE-based methods, closing the gap with Full FT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `9dgB`\n\n> Q1: Why does gradient alignment with Full FT improve performance theoretically, given that Full FT's updates aren't always optimal due to data/learning rate dependencies?\n> \nThanks for your insightful question. First, Full FT outperforms LoRA in most cases, making it a natural alignment target in previous works [1]. \nSecond, For scenarios where Full FT perform poorly, is because that its unregularized fitting capacity in low-data regimes lead to overfitting on intricate patterns and noise in smaller datasets. (e.g., CoLA, MRPC with 8.5K and 3.6K samples only in Table 4)\nIn contrast, our method combines Full FT strengths with three key regularizers to mitigate overfitting issues:\n- Low-Rank Updates: low-rank updates enforce robust feature learning and reduce sensitivity to noise [2].\n- MoE Architecture: Experts specialize in distinct patterns, avoiding over-adaptation to spurious variations.\n- SVD initalization:  Experts are initialized by different pretrained SVD-segmented features, enhancing specialization and mitigating overfitting.\nThus, our method approximates Full FT’s strong fitting ability while avoiding its pitfalls, achieving comparable or superior performance (e.g., CoLA in Tables 4).\n\n[1] LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS2024)\n\n[2] LoRA Learns Less and Forgets Less (TMLR2024)\n\n> Q2: Does the slightly worse performance of GOAT+ in Appendix D, despite its more precise alignment with Full FT, suggest that strict alignment may not be beneficial in practice?\n> \nSorry for the confusion. GOAT+ is not intended as an improvement over GOAT (not a more precise alignment), but rather as a variant that explores a different assumption.\n\nIn GOAT, we assign the same scaling factor to each expert, even though each expert is initialized with a different singular value, leading to varying norms. In contrast, GOAT+ adjusts each expert’s scaling factor in proportion to its singular value, ensuring that the product of the scaling factor and singular value is consistent across all experts. While this adjustment doesn't always improve performance, we found the underlying assumption interesting enough to include it in our appendix.\n\nIn the ablation study (Table 5), removing the module responsible for aligning with Full FT degrades performance, demonstrating the effectiveness of strict alignment. We will revise the naming in the paper to avoid any confusion.\n\n> Q3:Could you provide detailed settings and selection strategies for the coefficient of the load balancing loss used in GOAT and other MoE-based methods?\n> \nThanks for your suggestion. We use top-k routing with k=2 and set the coefficient for the balance loss to 1e-3. \nWe attach the load-balancing loss coefficient experiment by activating 2 out of 8 experts on Cars. \n\n|coefficient|GOAT|MoLoRA|HydraLoRA|\n|-|-|-|-|\n|1e-1|49.09|49.02|48.45|\n|1e-2|50.52|49.33|**49.45**|\n|1e-3|**53.50**|**50.83**|48.42|\n|1e-4|51.53|49.03|48.52|\n|0|49.85|48.02|49.06|\n\nWe can observe that setting the coefficient too low (e.g., 0 or 1e-4) leads to expert imbalances, which in turn degrades performance. Conversely, excessively high coefficients (e.g., 0.01 or 0.1) can disrupt the normal learning process. Our results show that a coefficient of 1e-3 achieves the best tradeoff in GOAT/MoLoRA between balancing expert load and maintaining stable learning. \n\nNotably, GOAT consistently outperforms across all tested coefficients, demonstrating its robustness in these settings.\n\n> Q4: Given GOAT's faster convergence, is it fair to train all methods for the same number of epochs (Appendix E.5)? Should we compare the best performance of each method instead?\n> \nTo clarify, our evaluation strategy indeed follows your suggestion by comparing the best performance achieved by each method. Specifically, we train each model for a sufficiently large number of epochs so that the loss converges to a stable plateau, then evaluate the model at every epoch and select the best-performing result for all baselines.\n\nFor the epoch number: \n\n- NLG/NLU: We use more epochs than previous studies to ensure convergence. For example, while prior work often uses just one epoch for NLG tasks [1,2], we employ five epochs to guarantee convergence.\n- Commonsense Reasoning: We strictly follow prior work [3] by using a large dataset (approximately 170K samples) to ensure thorough convergence. We then directly compare our results with the best-reported values from earlier studies, where our method still achieves superior performance.\n- CV: we retain the original epoch settings from previous work [4], as they ensure proper convergence for each task.\n\n[1] KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR2025) (ICLR2025)\n\n[2] LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS2024)\n\n[3] DoRA: Weight-Decomposed Low-Rank Adaptation (ICML2024)\n\n[4] Localizing Task Information for Improved Model Merging and Compression (ICML 2024)", "claims_and_evidence": "Why does the scaling scheme for gradient alignment with Full FT theoretically improve performance? In other words, the gradient update of Full FT may not always be optimal across all scenarios, as it is influenced by factors such as training data and learning rate. Therefore, aligning with Full FT does not necessarily guarantee the best results.\n\nMoreover, the GOAT+ method in Appendix D, which achieves a more precise alignment with Full FT, appears to perform slightly worse than the GOAT method. This raises concerns about whether strict alignment is indeed beneficial in practice.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, the paper provides correct theoretical derivations to support the claims.", "experimental_designs_or_analyses": "1. Could you provide detailed settings and selection strategies for the coefficient of the load balancing loss used in GOAT and other MoE-based methods? Since this coefficient can significantly impact the final performance, a clearer explanation would be beneficial.\n2. Given that GOAT improves convergence speed, is it fair to train all methods for the same carefully selected number of epochs detailed in Appendix E.5? Would it be more appropriate to compare the best performance achieved by each method instead?", "relation_to_broader_scientific_literature": "Somewhat.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSJU1fYA2y", "reviewer": "Reviewer_5G79", "summary": "This paper proposed a PEFT method with SVD-structured MoE and theoretical scaling. It initializes LoRA MoE experts with distinct singular value segments, and derives an optimal weight alignment strategy and scaling scheme to improve both convergence speed and performance. Extensive experiments on 25 tasks validates the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `5G79`\n\n> Q1:In Theorem 3.1, the authors claim that ‘we can align LoRA with Full FT’, ‘addresses the performance gap in single LoRA architectures’. Actually, the proposed method is still worse than full finetuning in most of the tasks, as shown in the experiments. In my opinion, this proposed method can only reduce the gap between LoRA and full FT, so the contribution here should be clarified.\n> \n\nThanks for your suggestion. Due to practical limitations (e.g., low-rank approximation, error accumulation), which may impact theoretical alignment, we will address this in the revision.\n\n> Q2: \nIn Theorem 3.5, the authors derive the optimal scaling factor from an assumption: there is a fixed learning rate ratio between full tuning vs. LoRA. However, as the learning rate (LR) is actually a hyperparameter, we cannot know the optimal LR for finetuning before we really do multiple runs of full FT, which is not applicable in PEFT setting. If the LR for full FT is arbitrarily selected in practice, the derived scaling value looks less useful. \nMoreover, in real experiments, we usually use a LR scheduler, and the LR ratio between LoRA and FT are even changing along the training trajectory, posing additional challenges for this assumption\n> \n\nThanks for your insightful question. To clarify, the learning rate ratio in our framework serves as a tunable hyperparameter, not a fixed constant. This means in practice, rather than arbitrarily selecting a full FT learning rate by multiple exhaustive full FT runs, one can first identify an optimal LR specifically for LoRA (which is computationally more feasible), then tune the ratio hyperparameter to implicitly define the corresponding optimal LR for full FT. In fact, similar assumptions and approaches have been validated and utilized effectively in existing literature [1]\n\nRegarding dynamic LR scheduling, it does not impact the alignment. This is because our theoretical framework is grounded on aligning the weight updates: $\\eta_{\\text{FFT}} \\cdot g_{\\text{FFT}} = \\eta_{\\text{LoRA}} \\cdot g_{\\text{LoRA}}$. i.e. $\\frac{\\eta_{\\text{LoRA}}}{\\eta_{\\text{FFT}}} g_{\\text{LoRA}} \\approx g_{\\text{FFT}}$, as long as LoRA and full FT share identical learning rate scheduling patterns at each training iteration, the relative LR ratio $\\frac{\\eta_{\\text{LoRA}}}{\\eta_{\\text{FFT}}}$ remains consistent. \n\nTo empirically substantiate this theoretical robustness, we refer readers to **Table 6**, where we report results across various LR settings. Our approach consistently maintains superior performance, exceeding baseline methods by a clear margin of 1.09-2.56 points, demonstrating its practical resilience and broad applicability.\n\n[1] LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS2024)\n\n> Q3:The paper could be improved for better clarity. For example, there is no algorithm description for the whole method, so the whole algorithm is splitted into different sections which hinders the readability.\n> \n\nThank you for your reminder. We incorporate the following pseudocode in the revised paper: \n\nAlgorithm: GOAT\nInput: $x$ (input),$n$ (input dim), $\\eta, \\rho$ (hyperparameter), $E$ (num experts)\n- Set Scaling Factor:  $s = \\sqrt{\\frac{3n\\eta}{r}}$\n- SVD Decomposition: $W_0 = U \\Sigma V^\\top$\n- Initialization ($\\forall i \\in [1,E]$): \n    - trainable component: $B_0^i = \\sqrt{\\frac{1}{s\\rho}} U' \\Sigma'^{1/2}, \\quad A_0^i = \\sqrt{\\frac{1}{s\\rho}} \\Sigma'^{1/2} V'^\\top$\n    - residual component: $W_{\\text{res}}^+ = \\frac{s}{E} \\sum_{i=1}^E B^i_0 A^i_0, \\quad \\tilde{W_0} = W_0 - W_{\\text{res}}^+$\n- Forward ($\\forall i \\in [1,E]$): \n    - Compute gating weights: $w^i(x)$\n    - Output: $\\tilde{W_0}(x) + \\sum_{i=1}^E w^i(x) s B^i_0 A^i_0(x)$\n\n> Q4:Typo: L195 ‘segement’, Figure 3 ‘Graident Alignment’.\n> \n\nThanks for your valuable advice. We will carefully revise our paper based on your suggestions.", "claims_and_evidence": "Most of the claims are supported, but there are still a few claims that are not convincing to me: \n\n(1) In Theorem 3.1, the authors claim that ‘we can align LoRA with Full FT’, ‘addresses the performance gap in single LoRA architectures’. Actually, the proposed method is still worse than full finetuning in most of the tasks, as shown in the experiments. In my opinion, this proposed method can only reduce the gap between LoRA and full FT, so the contribution here should be clarified. \n\n(2) In Theorem 3.5, the authors derive the optimal scaling factor from an assumption: there is a fixed learning rate ratio between full tuning vs. LoRA. However, as the learning rate (LR) is actually a hyperparameter, we cannot know the optimal LR for finetuning before we really do multiple runs of full FT, which is not applicable in PEFT setting. If the LR for full FT is arbitrarily selected in practice, the derived scaling value looks less useful. Moreover, in real experiments, we usually use a LR scheduler, and the LR ratio between LoRA and FT are even changing along the training trajectory, posing additional challenges for this assumption.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes, all of those in the experiments section.", "relation_to_broader_scientific_literature": "The initialization of weights and scaling methods are novel.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper could be improved for better clarity. For example, there is no algorithm description for the whole method, so the whole algorithm is splitted into different sections which hinders the readability.", "other_comments_or_suggestions": "Typo: L195 ‘segement’, Figure 3 ‘Graident Alignment’.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EHKkmrNsNV", "reviewer": "Reviewer_4qHz", "summary": "The paper proposes the GOAT, an SVD-derived LoRA-MoE finetuning framework. Through SVD decomposition, the authors found that existing LoRA finetuning schemes are insufficient due to restrictive training on only specific pre-selected SVD segments. Based on this finding, the authors proposed employing a LoRA-MoE architecture for finetuning, initializing each LoRA expert with different SVD segments and dynamically selecting segments via MoE during training. To enforce alignment of GOAT with full finetuning, specialized optimization for weight and gradient alignments are designed. Based on experiment results against various baselines on different datasets (tasks), GOAT is demonstrated to yield superior performance in almost all cases.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `4qHz`\n\n> Q1: Lemma 3.5 requires (non-intuitive) knowledge about Leaky-ReLU with negative slope of sqrt(5) resulting in Var(A)=1/(3n). Please cite the source or provide additional proof for this information.\n> \nThanks for your suggestion. We follow the derivation of the commonly used Kaiming initialization[1] (assuming activation function is Leaky ReLU[2]):\n\nProof: Leaky ReLU is defined as $f(x)=x \\text{ if } x \\geq 0, \\text{else } f(x) = a·x (a = \\sqrt{5})$\nFollowing Kaiming initialization[1], assume input $x$ is zero-mean with variance $\\sigma^2$, symmetrically distributed ($P(x \\geq 0) = P(x < 0) = \\frac{1}{2}$),\nwe can obtain output variance as: \n$$Var(f(x)) = \\frac{1}{2}(1 + a^2)\\sigma^2=3\\sigma^2 \\quad (a^2 = 5)$$\nTo ensure unit variance per layer ($Var(y)=1$), because $Var(y) = n Var(x)$, where $n$ is the layer width (fan-in).\n$n Var(f(x)) = 1 \\implies 3n\\sigma^2 = 1 \\implies \\sigma^2 = \\frac{1}{3n}$,\nThus, weights $A$ should be initialized with $Var(A) = \\frac{1}{3n}$.\n\n[1] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (ICCV2015) \n\n[2] Empirical Evaluation of Rectified Activations in Convolution Network (2015)\n\n> Q2:For Fig. 7, is load balancing used during training? While load balancing is necessary for most MoE training, it also weakens the conclusion of \"validates on the effectiveness of each SVD chunk\". This is because balanced workload distribution is enforced by the load balancing loss, rather than implicitly achieved through SVD-based initialization alone.\n> \nThank you for raising this insightful point. In Fig.7, we use load balancing, but the conclusion holds because: \n- Section 2.1 shows single-LoRA initialization via distinct SVD segments (no load balancing), the Fig.1 revealing their dataset-dependent roles.\n- We further ablate the load-balancing in GOAT (2in8, Cars task), which confirms all experts remain active (see table below), proving each SVD chunk contributes meaningfully.\n|GOAT|f1|f2|f3|f4|f5|f6|f7|f8|\n|-|-|-|-|-|-|-|-|-|\n|w/o loadbalance|0.1043|0.1379|0.1275|0.1094|0.1207|0.1405|0.1259|0.1338|\n\n> Q3: The goal of Section 2.2 (Rethinking Scaling Factor) is not understood. What is the final verdict from the exploration? how does this exploration relate to alignment wrt full-finetuning?\n> \nThe goal of Section 2.2 is to establish two key insights: (1) Weight initalization alignment alone is insufficient - gradient alignment is equally crucial  (2) The scaling factor s fundamentally controls gradient dynamics.\n\nSection 2.2 experiments (Figure 2) reveal that even with perfect initalization alignment, common choices (s=2) produce small gradient norms and slow convergence.  Increasing s - particularly in low-rank settings - boosts gradient magnitudes and accelerates training. This leads to Lemma 2.2: scaling factor s directly governs gradient behavior, meaning poor s choices degrade optimization dynamics regardless of weight initialization.\n\nThis motivates our core contribution in Theorem 3.2 and Theorem 3.5. Theorem 3.2 establishes that both weight initialization and gradient updates must align for $W_{LoRA}$ to track $W_{FFT}$ based on the first insight. In Theorem 3.5, we chose to derive the optimal s to ensure gradient updates alignment, as the second insight and experiment tells us control s is a practical way to adjust the gradient dynamics. Together, these theorems provide a complete framework where proper scaling factor selection enables LoRA to match full fine-tuning performance through principled optimization dynamics.\n\n> Q4: The algorithm pseudocode should be presented.\n> \nThank you for your reminder. We incorporate the following pseudocode in the revised paper: \n\nAlgorithm: GOAT\nInput: $x$ (input),$n$ (input dim), $\\eta, \\rho$ (hyperparameter), $E$ (num experts)\n- Set Scaling Factor:  $s = \\sqrt{\\frac{3n\\eta}{r}}$\n- SVD Decomposition: $W_0 = U \\Sigma V^\\top$\n- Initialization ($\\forall i \\in [1,E]$): \n    - trainable component: $B_0^i = \\sqrt{\\frac{1}{s\\rho}} U' \\Sigma'^{1/2}, \\quad A_0^i = \\sqrt{\\frac{1}{s\\rho}} \\Sigma'^{1/2} V'^\\top$\n    - residual component: $W_{\\text{res}}^+ = \\frac{s}{E} \\sum_{i=1}^E B^i_0 A^i_0, \\quad \\tilde{W_0} = W_0 - W_{\\text{res}}^+$\n- Forward ($\\forall i \\in [1,E]$): \n    - Compute gating weights: $w^i(x)$\n    - Output: $\\tilde{W_0}(x) + \\sum_{i=1}^E w^i(x) s B^i_0 A^i_0(x)$\n\n> (Q5) On Figure 3.II, please emphasize that the goal is to find W_res and s. Currently, this is not intuitive without scrutiny over the figure (font too small for W_res and s) and explicit rereading of Section 3.3 (please set the closed-form solutions for W_res and s as labelled equations).\n> \nThanks for your valuable advice. We will revise our figure to make it clearer.", "claims_and_evidence": "Yes. The paper claims that:\n\n1) the integration of SVD segments via dynamic selection is necessary to achieve good initialization for LoRA(-MoE) finetuning;\n2) specialized optimization for alignment to full finetuning is necessary to improve model performance.\n\nThese claims are supported by:\n\n- good preliminary empirical analysis in Fig. 1, which shows the effects of initializing different SVD segments for different scenarios;\n- good method design demonstrating the integration of SVD segments into LoRA-MoE architecture and the computation of W_res and s for weight and gradient alignment;\n- good experiment performance showing SOTA performance.\n\nThus, the claims made within the paper are well supported.", "methods_and_evaluation_criteria": "Yes. \n\nThe proposed method is sensible.\n\n- Employing MoE to facilitate dynamic selection of LoRA related to specific SVD segment is a reasonable design. \n- The mathematics for deriving W_res and s for alignment appears sensible.\n\nThe benchmark datasets are sensible.\n\n- Since GOAT is task-agnostic, both vision and language tasks are used in the experiment.\n- The baseline (compared) algorithms are recent, with most published in 2024.\n\nThus the overall methods and evaluation are sensible.", "theoretical_claims": "Yes. The critical mathematical proofs in paper are predominantly focused on finetuning alignment (Section 3.3). They are found in Appendix C. Specifically:\n\n- Lemma 3.1-3.4 appears correct, and are generally intuitive.\n- (Minor Weakness) Lemma 3.5 requires (non-intuitive) knowledge about Leaky-ReLU with negative slope of sqrt(5) resulting in Var(A)=1/(3n). Please cite the source or provide additional proof for this information.\n\nAside from the minor clarification required at Lemma 3.5, the critical proofs appear correct.", "experimental_designs_or_analyses": "Yes. The experiment designs are considered valid.\n\n- A good variety of image and language tasks are considered, with GOAT outperforming baseline methods in most cases.\n- The ablation study, as shown in Table 5, compares GOAT with alternative schemes of initialization on only the principal, minor, and random to validate the claim that SVD-based initialization is necessary.\n- Table 7, when taken together with Table 1 and Table 3, shows that GOAT does not introduce excessive parameters, and utilizes similar GPU RAM and training time compared to baseline methods. This demonstrates that the experiment evaluates GOAT against baseline methods fairly.\n- (Minor Weakness) For Fig. 7, is load balancing used during training? While load balancing is necessary for most MoE training, it also weakens the conclusion of \"validates on the effectiveness of each SVD chunk\". This is because balanced workload distribution is enforced by the load balancing loss, rather than implicitly achieved through SVD-based initialization alone.\n\nAside from minor clarification required from Fig. 7, the experiment and analysis design are valid.", "relation_to_broader_scientific_literature": "- The research advances ongoing research on improving the performance of LoRA-finetuned models. The paper focuses particularly on addressing deficiencies in parameter initialization. GOAT outperforms existing initialization techniques, such as PiSSA and KaSA. GOAT also introduces another perspective on weight alignment wrt full finetuning orthogonal to preceding works, such as DoRA and analysis by (Shuttleworth, 2024).\n- The integration of LoRA with Mixture-of-Experts expands on an emergent trend of Mixture-of-LoRA frameworks, such as MoLE (Wu, 2024) and MixLoRA (Li, 2024). Moreover, the author provides SVD-based analyses to justify the use of Mixture-of-LoRA frameworks to ameliorate low-rank-related deficiencies found in conventional LoRA finetuning.\n\nReferences:\n\nShuttleworth, R., Andreas, J., Torralba, A., & Sharma, P. (2024). Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228.\n\nWu, X., Huang, S., & Wei, F. (2024). Mixture of lora experts. arXiv preprint arXiv:2404.13628.\n\nLi, D., Ma, Y., Wang, N., Ye, Z., Cheng, Z., Tang, Y., ... & Tang, M. (2024). Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404.15159.\n\nPiSSA, KaSA, and DoRA are cited within the reviewed paper.", "essential_references_not_discussed": "No. To the knowledge of this reviewer, no essential references are missing.", "other_strengths_and_weaknesses": "The authors have investigated an important problem (improving LoRA-finetuning) and proposed an interesting design with solid analyses and good experiment results. However, some additional weaknesses are noted below.\n\nWeakness (not previously addressed):\n- (Weakness) The goal of Section 2.2 (Rethinking Scaling Factor) is not understood. What is the final verdict from the exploration? Specifically, how does this exploration relate to alignment wrt full-finetuning?\n- (Minor Weakness) The pseudocode of your algorithm should be presented (if necessary, in the appendix).\n- (Minor Weakness) On Figure 3.II, please emphasize that the goal is to find W_res and s. Currently, this is not intuitive without scrutiny over the figure (font too small for W_res and s) and explicit rereading of Section 3.3 (please set the closed-form solutions for W_res and s as labelled equations).", "other_comments_or_suggestions": "None.", "questions_for_authors": "For ease of reference, these comments/questions regarding weaknesses in the paper are repeated below. Upon providing satisfactory response, the overall recommendation will be changed to Accept (from Weak Accept).\n1) (Minor Weakness, More Information) Lemma 3.5 requires (non-intuitive) knowledge about Leaky-ReLU with negative slope of sqrt(5) resulting in Var(A)=1/(3n). Please cite the source or provide additional proof for this information.\n2) (Minor Weakness, Clarify) For Fig. 7, is load balancing used during training? While load balancing is necessary for most MoE training, it also weakens the conclusion of \"validates on the effectiveness of each SVD chunk\". This is because balanced workload distribution is enforced by the load balancing loss, rather than implicitly achieved through SVD-based initialization alone.\n3) (Weakness, Revision) The goal of Section 2.2 (Rethinking Scaling Factor) is not understood. What is the final verdict from the exploration? Specifically, how does this exploration relate to alignment wrt full-finetuning?\n4) (Minor Weakness, More Information) The pseudocode of your algorithm should be presented (if necessary, in the appendix).\n5) (Minor Weakness, Revision) On Figure 3.II, please emphasize that the goal is to find W_res and s. Currently, this is not intuitive without scrutiny over the figure (font too small for W_res and s) and explicit rereading of Section 3.3 (please set the closed-form solutions for W_res and s as labelled equations).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents GOAT, a fine-tuning framework based on LoRA for MoE architectures. The key idea is to initialize the experts' weights using different pieces of an SVD and allowing a router to dynamically select the right expert during training. There's also some theoretical justification showing an equivalence to full fine-tuning of the MoE. The paper is generally well written with solid experimental evidence. Concerns were raised regarding the presentation (particularly, the fact that the method is not clearly written down in terms of pseudocode) and some ambiguities around the choice of scaling factors, but the authors gave a satisfactory response. Overall, I recommend an accept.
Decision: Accept (poster)