MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
Authors: Shen Yuan
Yin Zheng
Taifeng Wang
BINBINLIU
Hongteng Xu
Venue: NeurIPS 2025 poster ()
Keywords: LLM
MoE-ization
SVD
Orthogonality
PEFT
Multi-Task Adaptation
Primary area: deep_learning
TL;DR: We propose a novel model ``MoE-ization'' strategy using SVD, which leads to a conflict- and oblivion-resistant multi-task adaptation method.

Abstract: Adapting large-scale foundation models in multi-task scenarios often suffers from task conflict and oblivion.
To mitigate such issues, we propose a novel "model MoE-ization" strategy that leads to a conflict- and oblivion-resistant multi-task adaptation method. 
Given a weight matrix of a pre-trained model, our method applies SVD to it and introduces a learnable router to adjust its singular values based on tasks and samples.
Accordingly, the weight matrix becomes a Mixture of Orthogonal Rank-one Experts (MoORE), in which each expert corresponds to the outer product of a left singular vector and the corresponding right one.
We can improve the model capacity by imposing a learnable orthogonal transform on the right singular vectors.
Unlike low-rank adaptation (LoRA) and its MoE-driven variants, MoORE guarantees the experts' orthogonality and maintains the column space of the original weight matrix.
These two properties make the adapted model resistant to the conflicts among the new tasks and the oblivion of its original tasks, respectively. 
Experiments on various datasets demonstrate that MoORE outperforms existing multi-task adaptation methods consistently, showing its superiority in terms of conflict- and oblivion-resistance.
The code is available at https://github.com/DaShenZi721/MoORE.

MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
Shen Yuan 1.2
Yin Zheng²
Taifeng Wang²
Binbin Liu?
Hongteng Xu 13,43
Gaoling School of Artificial Intelligence, Renmin University of China +ByteDance SBeijing Key Laboratory of Research on Large Models and Intelligent Governance Engineering Research Center of Sext-Generation Intelligent Search and Recommendation, MOE
Abstract
Adapting large-scale foundation models in multi-task scenarios often suffers from task conflict and oblivion. To mitigate such issues, propose novel "model MoE- ization" " strategy that leads to a conflict- and oblivion-resistan multi-task adaptation method. Given weight matrix of pre-trained model, method applies SVD to and introduces a learnable router to adjust its singular values based on tasks and samples. Accordingly, the weight matrix becomes Mixture of Orthogonal Rank-one Experts (MoORE), in which each expert corresponds to the outer product of left singular vector and the corresponding right one. We improve the model capacity by imposing learnable orthogonal transform on the right singular vectors. Unlike low-rank adaptation (LoRA) and its MoE-driven variants, MoORE guarantees the experts' orthogonality and maintains the column space of the original weight matrix. These two properties make the adapted model resistant the conflicts among the new tasks and the oblivion of its original tasks, respectively. Experiments on various datasets demonstrate that MoORE outperforms existing multi-task adaptation methods consistently, showing its superiority in terms of conflict- and oblivion-resistance, The code is available at hht.ps:: // Ecom/DaShen22722//Mo0
Introduction
crrameter-efficiinn eitapration/fine-tniiii [71] plays central role in the practical deployment of
large-scale pre-training models, especially multi-task learning (MTL) scenarios [80 61
[T] 62 42]
Take large language models (LLMs)) [82 in natural language processing (NLP) tasks as an example.
To increase intrinsic knowledge and maintain generalization power, a pre-trained LIM often needs to
learn multiple downstream tasks in different domains simultaneously sequentially. To achieve multi- task adaptation, parameter- efficieet fine-tuning (PEFT) methods like low-rank adaptation (LoRA) (25)
and its variants [68] 2 60 65 31 have been proposed. However, the practical applications of these methods are often limited because they often suffer from task conflict and oblivion: The model adapted for one task often leads to the performance degradation in other tasks (also known negative transfer 770 79 or destructive interference [T1). ). ti) Learning new tasks may result in catastrophic forgetting (62 the model performance deteriorates severely in previously learned tasks.
Essentially, task conflict and oblivion arise because the diversity of different tasks requires the models to adapt their parameters in different directions 772
74
13
26
15). Some recent methods B1
30 60) combine the Mixture-of- -fpeetef--xtssssseee (MoF) architecture
55 29 with PEFT, mitigating the
interferences across different tasks by activating task-specific parameters. Given layer of a pre- trained model, these methods learn multiple adapters associated with router multi-task scenarios.
CCorresponding author. Email: hongtengxu3 13 .om
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A Mixture of Orthogonal and Rank-one Experts (MoORE)
Singular Vectors
Singular Values
Right Singularr Vectors
Learnable
Router
Export
X
Weight Matrix
Weights
X
MoE-ize
Learnable
X
W
Orthogonal Adapter
Projectod
Myrb
Input
(a) illustration our SVD-based model MoE-ization strategy and MoORE architecture
92
MuSLD
MTL LERA
62
LoRAMoE
Jerfamaanc 190
S8
Perfomannce
MOGRE
MisLaRA
56
Tinelmns
54
52
86
50
MediD D
MicieRA
84
48
LoRANGE HydrrLoRA
LLAAA 3.1 38
46
2
4
5
6
5
The Number Tasks
The Number of Tasks
(b) Resistance to task conflict
(c) Resistance to task oblivion
(d) Inference efficiency
Figure I: (a) An illustration of our model MoE-ization strategy and the corresponding MoORE architecture. (b) The comparison for various multi-task adaptation methods on fine-tuning LLaMA-3.1 8B (0) on the CSR-MTL constructed by nine tasks 9 g 45
5 [9] MoORE consistently works better than the baselines when the number of tasks is larger than one. (c) Before adaptation, LLaMA-3.J 8B achieves encouraging overall
S
54
77
51 7 4 [0]. MoORE mitigates the performance (i.e., the gray dashed line) in seven tasks [24] 23 83 58
performance degradation and outperforms the most competitive baseline LoRAMoE 14] (d) MoORE's s runtime
is comparable to that of its competitors. Compared to the original aMM33. 1 8B (i.c., the gray dashed line),
MoORE increases the inference time moderately.
Each adapter may inherit task-specific knowledge, and the router selects/fuses these adapters based on the input data and tasks.
In principle, we call the above strategy "Model MoE-ization" because it converts the original linear layer to an MoE architecture. In this study, we propose a novel model MoE-ization strategy, with the help of the singular value decomposition (SVD), leading to a conflict- and oblivion-resistam multi- task adaptation method. As illustrated in Figure [(a) given a weight matrix a pre-trained model, method applies SVD [I] to it and introduces learnable router to adjust its singular values based on tasks and samples. Accordingly, the weight matrix becomes a Mixture of Orthogonal Rank-one Experts (MoORE), where each expert is constructed by the outer product of left singular vector and the corresponding right one. To further improve the capacity of MoORE, we impose a learnable orthogonal adapter, which is implemented by a Householder reflection adaptation module 77).
Unlike existing methods, which learn some additional low-rank experts without any constraints on their relations, MoORE extracts many rank-one experts intrinsically from the SVD of the original weight matrix. Such a design imposes the orthogonality of the experts, avoiding their information redundancy and undesired interferences. Moreover, similar existing orthogonal fine-tuning strate- gies [49 75), MoORE maintains the column space of the original weight matrix and thus inherit the pre-training ability better. Thanks to the above two properties, MoORE consistently outperforms SOTA methods various multi-task adaptation scenarios. The results in Figure T show the superiority of MoORE in mitigating task conflict and oblivion and its competitive inference efficiency.
Related Work and Preliminaries
2.1 Adapter-based Methods for Multi-Task Adaptation
Multi-task adaptation aims to fine-tune a pre-trained foundation model simultaneously or sequentially
in multiple downstream tasks 34
32 Focusing on this problem, many adapter-based methods have
been proposed [37] 50
57
65
40
In particular, denote the data of K tasks as [D Given

---

pre-trained foundation model, whose parameters are denoted as 0, multi-task adaptation can often be formulated in the framework of maximum likelihood estimation:
max Σ
k=1 PoLao(DD);
(I)
where PoLAe (Di) denotes the likelihood of the k-th task's data, which is parameterized by the original model parameters 0 and the adapters' parameters 26.
Unlike learning and ensembling different models, udapter-based methods improve the overall model performance by sharing parameters and domain knowledge across various tasks, leading to moderate increases in parameters and complexity. For instance, Hyperformer (44) utilizes a shared hyper- network to generate specific adapters, reducing the number of learnable parameters. Recently, some methods extend LoRA (25) multi-task adaptation, e.g., MultiLoRA [68), MTL-LoRA [73], HydraLoRA [60], and so on, which learns multiple low-rank adapters to handle diverse tasks.
2.2 Connections to MoE Architectures
The ox--eerrss (MoE) architecture was initially introduced by the work in [27 which is constructed by multiple specialized networks (called experts) and a router. Given an input data a E X. the MoE derives the output y y as y = Emm Im fm) where fm denotes the m-th expert, which achieves mapping from the sample space X' to the output space V. 9 : x+ + IR denotes the router, and Im denotes its m-th output. The router adjusts the experts' significance based on input data. When applying sparse routing strategy. i.e., activating only few experts for each input [55] 29 17 3 16, 46). the MoE architecture supports building large-scale models while maintaining computational efficiency. Due to its advantages, many large language models, e.g., DeepSeek [1). Grok3, and Qwen are built based on MoE, and many efforts have
+ y
been made convert well-trained dense models into MoE architectures 28 22
52
47 78).
As mentioned before, most existing adapter- based multi-task adaptation methods actually "MoE-ize" pre-trained models. Given a weight matrix W and its input æ, these methods apply multiple low-rank and add them to the pre-trained models, i.e.,
adapters as experts 776
35 41
M
y Wx+
9m (x) B Am
(2)
where Am and Bm are low-rank matrices constructing the m-th expert fm- For the router g(x).
some attempts have been made to develop advanced routing strategies, e-g., the dynamic routing in AdaMoLE [38] and the token-task hybrid routing in HMoRA [II. Recently, some new MoE
architectures have been proposed, including the asymmetric "Hydra" structure in HydraLoRRA (60), the I.EMoEE [66] for lifelong model editing, and the OMoE [18] for orthogonal output. However, the above methods rely purely on data-driven strategies to determine the experts' functionality and domain knowledge. Without necessary regularization on the relations across different experts, the experts often suffer from the load imbalance issue, i.c., a limited number of experts are over-trained and applied for multiple tasks, while many experts are seldom used. This issue harms the capacity and generalizability of the models, increasing the risk of task conflict and oblivion.
Proposed Method
In this study, we propose new model MoE-ization method for multi-task adaptation. In principle, our method imposes orthogonality on the experts and further regularizes their output spaces, which helps mitigate task conflict and oblivion.
3.1 SVDebased Model MoE-ization
Consider a pre-trained weight matrix W e Do Without loss of generality, we assume Dant D and Rank W) D. The SVD of the matrix is denoted as
W = Udiag(o)V Σ
(pan))
(3)
d=1
weight expert
3

---

where U [u1.-.. up] E R RDoxx contains left singular vectors, V [01.-..,00 E R DxD is a vector of singular values.
contains right singular vectors, and o = [01.-.. (0 _D]
As shown in 3 there exists intrinsic but static MoE architecture hidden in the SVD of the weight matrix - each W corresponds to the mixture of D orthogonal and rank-one experts, in which the d-th expert is the outer product of F and va and its weight is fixed as a. Motivated by this intrinsic MoE, our method derives the proposed MoORE model for multi-task adaptation, which reuses the experts while introducing the following modifications:
A hybrid routing strategy: Inspired by HMoRA [3T,, we adjust the experts' weights according to input data and tasks, leading to a hybrid routing strategy. Given an input of the k-th task, denoted as x (k) E Rd, we determines the weight of the d-th expert as
(xx
Pd tg +
qd Ta
task-level
sample-lovel
where & E Dr denotes the embedding of the k-th task, and the vector Pa E RD projects the task embedding to task-level weight. Similarly, applying the matrix E RD,x and the vector qu E ",,,, we project the input (6) to a sample-level weight. In practice, we set D,, Dr < D to reduce the router's parameters and computational cost. The final weight (a():(k) is the summation of the task- and sample-level weights.
To further increase the model capacity, we can apply learnable orthogonal transform to the input, i.e., Hæ, where the learnable orthogonal transform H can be implemented efficiently by the butterfly orthogonal fine-tuning (BOFT) module in (36), the Givens rotation adapter [43], or the Householder reflection adapter (HRA) 775 In this study, we implement R by HRA, which corresponds to the product L Householder reflections, i.e.
An orthogonal adapter of input:
H
II
(1
Pere
|rel
whose learnable parameters are R [r1,... .T.. E DXL Note that applying orthogonal adapters maintains the angles between neurons (i.e., the rows the weight matrix W). which helps preserve the knowledge of the pre- rreitrined model when enhancing model capacity 49 36 75).
Applying the above SVD>-based model MoF-ization strategy, obtain the proposed MoORE model, which introduces D orthogonal and rank-one experts into the model and encodes the input data as
y = WHa (R)
Σ
d=1 (k(
(n H) (k) (x: + +oV Hx)
(6)
rouler
expert
where (*) = P tu + T (k) R.D
The learnable parameters of MoORE include T E P [pa R P= iqal E DD. r E DRD, and R E DxL As shown in (6), given the SVD of W (which can be computed in advance), can implement MoORE with low complexity via the following steps:
1) == (k) 2) (z) O(D(L+D)
-P + TT
3) y) = (
O(DD.)
O(DD.)
O(DD...
As shown in (7), the overall complexity of MoORE is O(D(Dour + D, D. L)). In practice, we set L, D., D. < min(D, Doa) to reduce the complexity. Moreover, we can merge the orthogonal adapter into each expert in the inference phase, i... obtaining V' = V H, and the complexity further reduces to (DD(D. D Dr + D,)). Figure ((d) shows that the inference efficiency of MoORE is comparable to its competitors.
3.2 Comparisons with Existing MoE-based Multi-Task Adaptation Methods
Our SVD-baser model MoE-ization method provides new technical route for multi-task adaptation: Instead of learning an MoE with few strong extrinsic experts, our method constructs an MoE with many simple but structured experts intrinsically based on the pre-trained weight matrix. Tables [ and compare different methods on their MoE architectures, theoretical properties, and computational efficiency, respectively.

---

Table 1: Comparisons for various MoE-based multi-task adaptation methods their implementations and properties, where M (D in our method) the number experts, & the task index, GS(+) denotes the Gram- Schmidt orthogonalization
Orthogonality Maintaining of Experts of Experts
Rank
Method
Router
Experts
#Experts
(W))
LoRAMoF 14
Softmax (Sx)
Bm Am
r
No
No
MixLoRA 30
(Soffmax(aaa
Bm A
M
M
No
No
m=
MoSLD SI
Softmax (Sx)
[BAm
m==
r
No
No
HydraLoRA 60)
Softmax (Sx)
{B A J
M
No
No
MTL-LoRA 73
Softmax (bx)
m=
M.K
MK
V
No
No
OMoE 18
Softmax (Sx)
GS (B Am)m=
r
Yes
No
MoORE (Ours)
P L.
Q Tw
(u H
D
1
Yes
Yes
The design of router. Given an input æ E D. most existing methods leverage a sample-level routing strategy. Typically, they apply a linear projection S E RMD to it and pass the projection result through a softmax operator, leading
nonnegative and normalized weight vector for M experts. Among them, MixLoRA [30] further applies a sparse routing mechanism - for each input, it only activates the two experts that correspond the top-2 weights. Instead, MTL- LoRA (73) leverages task-level routing strategy, which determines the experts' weights by passing task-specific embedding (i.e., Φx E R2,, through a softmax operator. Unlike existing methods, our MoORE considers the task- and sample- level information jointly. The advantage of this hybrid routing strategy is that when the same sample serves for different tasks, MoORE can assign task sses-ssecffic weights to the experts and thus leverage different domain knowledge accordingly.
{1,... K} indicates the task index)
The design of experts. Most existing methods apply M low-rank adapters as experts, i.e., { (Bm M m=1 where B R. Duuxr and A Rrxb are two rank-r matrices. To reduce the number of learnable parameters, some methods, e.g., MoSLD [8I), HydraLoRAA [60], and MTL-LoRA 773), reuse the same A or B for all experts. MTL-LoRA (73 further introduces task-specific matrix A₃ for each expert, where = I,..., K indicates the task index. As a result, it creates MK low-rank experts and activates M experts per task. Our MoORE contains D orthogonal rank-one experts aarn H in which the orthogonal adapter is shared by all experts. Unlike existing methods, MoORE applies many simple but structured experts. Such design has several advantages:
Imposing orthogonality for mitigating task conflict: The experts of MoORE are orthog- onal to each other because u, u, for all d * d'. The orthogonality ensures that the experts have different functionalities and domain knowledge, without redundant information. In particular, by activating different experts for different tasks, MoORE suppresses the interferences across the tasks in the training phase, thus mitigating the task conffict issue. is worth noting that O-LoRA (67] also introduces orthogonality constraints within its LoRA adapters. However, there are two notable differences between O-LoRA and MoORE: i) O-LoRA uses regularization term to pursue orthogonality, which can not strictly make its LoRA adapters orthogonal to each other. ii) O-LoRA does not adopt a MoE architecture. To deal with different tasks, O-LoRA adds new LoRA adapters for each sequentially incoming task, rather than designing and learning a flexible routing mechanism for given and fixed ex- perts. As a result, the model complexity of O-LoRA is linear to the number of tasks. Among existing MoE-based methods, OMoE (8) is the only one imposing orthogonality on experts. However, it applies Gram -chhmdt orthogonalization algorithm 66] to the concatenation of M experts' output, i.c., GS ( Am, ... BM A., æ)). As a result, imposing orthogonality requires additional ODD.m 2 operations per sample, which is less efficient than ours.
2. Maintaining Range( (W) ) for mitigating task oblivion: The column space of each expert, i.c., Range o H), ), is the same with Range (ua). Accordingly, the output space of MoORE is the direct sum of (Range( uu)) )z which is the same as W's column space, i.e.,
Range(u.v) H)
Range(ua) = Range(U) (U) Range W).
(8)
d=1
5

---

Table 2: Comparisons for various MoE-based multi-task adaptation methods on their computational efficiency.
#Learnable Parameters
Method
Computational Complexity
Router
Experts
LoRAMoB 14
MD
M Dou + D)r
O(Dau D M(D + Dout + Dr))
MixLoRA 30
MD
Mon + D)r
O( 1 + MD + (Dant + Dr)
MoSLD 81
MD
(Don MDr
O + Daur M D))
HydraLoRA 660
MD
(M + D)r
O + Dr + MDDuu D))
MTL-LoRA
73
MK
(Mu + Kr + D)r
O(Dan + Dr +- + r²)
OMoE 18
MD
M(Don + D)r
O(Daa M2) M(D + Dour + Dr))
MoORE (Ours) D,K (Dz + 2D.)D
LD
O((Dan mm D D. + D. + L)D)
Table 3: Comparisons for various SVD-based fine-tuning methods on their implementations.
Method
Model in Training
Learnable Parameters
Model in Inference
SVF (66
V)
Nonparametric a
Original architecture: W new
SVDiff 21
Udiag(Rel.U(r + V))
Nonparametric 5
Original architecture:
new
SVFT 33
(ddiag(a) V
Nonparametric M
Original architecture: W now
KaSA
64
Wtr + AUdiag( AV
Nonparametric AU, Ao, AVT Nonparametric a7
Original architecture: W Woww
SVFCI.
69
Udiag(o -.=0%.)V
Original architecture: W
MoORE
WH + + (e(ax (arn H)
Parametric 9(-) and
An MoE, where W H and Va H are pre-computed
(Ours)
Householder reflections H
In single task adaptation scenarios, the work in [75] has shown that maintaining the column space of the weight matrix makes the adapted model inherit the ability of the pre-trained model better, mitigating the oblivion of the previously pre-trained task. In our work, we find that such maintenance is helpful in multi-task adaptation scenarios as well.
Computational efficiency. Table shows each method's learnable parameters and computational complexity. For the methods using the sample-based routing strategy, their routers contain MD learnable parameters. Given a sample, the complexity of the router is (M For the method 773 using the task-based routing strategy, its router is lightweight, containing MK learnable parameters and determining the weights of its experts with complexity ((M). For the experts, using M rank-r experts leads to the complexity O O(M
D)r). Reusing A B (c.g., MoSLD [8] and HydraLoRA (60) and applying sparse routing (c.g.. MixLoRA (30]) can reduce the computational complexity significantly. In contrast, introducing additional parameters (e.g., the Ae in MTL-LoRA (33) or operations (e.g., the Gram-Schmin orthogonalization in OMoE (18)) leads to higher complexity. Existing methods construct MoE with M rank-r experts, while our MoORE is an MoE with D rank-one experts, whose number of experts is determined by the input dimension and thus much larger than M. To improve the computational efficiency of MoORE, we set D, and D, comparable to the rank and set L comparable to M, respectively. As a result, the number of learnable parameters and the complexity of MoORE become comparable to most existing methods.
3.3 Comparisons with Existing SVD-based Fine- Fine-Tuning Methods
Our method employs SVD to perform model MoE-ization, constructing a complete MoE architecture.
This fundamentally differs from existing SVD-based fine-tuning methods [48 [56] 21
33 64 69).
To more clearly illustrate the differences between MoORE and these methods, we select five recent ones for comparison, including SVF [56], SVDiff (21), SVFT [3), KaSA [64], and SVFCL (69]
Tables 3 and compare different methods in terms of implementation and application, respectively.
In the aspect of architecture, MoORE is the only method that learns a parametric router to adjust singular values, leading to an MoE architecture for inference. In MoORE, each expert is constructed by the outer product of singular vectors and the learnable Houscholder reflections.
In the aspect of application, only MoORE and SVFCI. consider multi-task adaptation. MoORE is applicable for both simultancous and sequential multi-task adaptation, while SVFCL only

---

Table 4: Comparisons for various SVD-based fine-tuning methods their applications.
Method
Multi- Task Learning
Consider Task Conflict
Consider Task Oblivion
SVF [56
No
SVDiff
21
No
No
No
SVFT 33
No
No
No
KaSA 64
No
No
No
SVFCL
Yes (incremental learning)
Partially discussed
MoORE (Ours)
Yes
Yes, by orthogonal experts Yes, by maintaining Range W)
considers incremental learning. Moreover, MoORE makes the first attempt to connect task conflict and oblivion to the orthogonality of experts and the column space of the parameter matrix.
4 Experiments
To demonstrate the effectiveness of MoORE in multi-task adaptation, we apply three MTL datasets and conduct comprehensive experiments them. Representative results are shown in Figure and the following content. More experimental details, e.g., the basic information of datasets, hyperparameter settings, ablation studies. routing weight analysis, and numerical results associated with figures, are shown in the Appendix.
4.1 Implementation Details
Base model and baselines. In the following experiments, we utilize LLaMA-3.1 88 20 as the base model and adapt it by various multi-task adaptation methods. particular, we compare MoORE with LoRA (25) and the methods incorporating low-rank adapters as MoEs, including LoRAMoE (14), MoSLD SU, MTL-LoRA (73). HydraLoRA [60], and MixLoRA [30]. We implement the MoE architectures of the baselines mainly based on their default settings. For a fair comparison, we modify some baselines' hyperparameters to make the number of learnable parameters comparable for each method. For MoORE, we MoE-ize all linear layers of LLaMA-3.1 8B, including the "QKVO" O" modules attention layers and the weight matrices of FFNs.
Two datasets for evaluating conffict-resistance. We consider two MTL datasets for commonsense reasoning (CSR) and natural language understanding (NLU), respectively. The CSR-MTL dataset is constructed by nine tasks, including ARC-Challenge (ARC-C), ARC-Easy (ARC-E) (. Open-
BookQA (OBQA) [45), PIQA (5), SocialIQA (SIQA) (54), BoolQ (8]. Hellaswag (HellaS) (77). Winogrande (WinoG) [53) and CommonsenseQA (CSQA) (59. These tasks are widely used to evaluate LLMs on various commonsense reasoning challenges, ranging from genuine grade-school level science questions to physical commonsense reasoning. The NLU-MTL dataset consists of seven tasks from GLUE 63 including CoLA, SST-2, MRPC, QQP, MNLI, QNLI, and RTE. These tasks are applied to evaluate the natural language understanding capabilities of LLMs, including natural language inference, textual entailment, sentiment analysis, semantic similarity, and so on.
One dataset for evaluating OR-MTL, for evaluating the tasks, including MMLU 24 23). IFEval
In addition, we construct one more dataset, called different methods. The dataset includes seven [83] BIG-Bench Hard (BBH) (58), GPQA [51], Hu- manEval 7 MBPP 44, and GSM-8K [T]. The base model, LLaMA-3.1 8B, achieves encouraging performance in these tasks. After adapting on CSR-MTL, we record the performance of the adapted models in OR-MTL and assess the ability of different methods to mitigate task oblivion.
Experimental settings. When adapting the pre-trained model on CSR-MTL and NLU-MTL, we set the training epoch 2 and 5, respectively. The learning rate set to 10-4 4, with AdamW [3) as the optimizer. For CSR-MTL, we set the batch size to 8, whereas for NLU-MTL, we set the batch size 64. Both training and testing are conducted on one NVIDIA 1100 GPU.
Instruct

---

Table 5: Results (%) of various methods on CSR-MTL. The best results on each dataset are shown bold, and the second best results are shown in underline.
Method
#Params
ARC-C
ARC-E BoolQ
OBQA
PIQA
SIQA
HellaS
WinoG
CSQA
Overall
Full Fine-tuning
100.00%
80.12
87.92
74.56
87.60
88.96
80.55
95.91
81.69
81.16
84.27
LoRA 25
2.09%
77.56
85.77
70.43
81.60
82.97
76.00
93.00
71.11
77.40
79.54
MixLoRA
30
3.00%
79.18
87.50
72.02
86.60
87.38
78.86
93.65
77.35
80.43
82.55
MoSLD [8]]
1.49%
80.29
86.87
74.16
86.40
88.58
79.73
95.03
80.58
80.34
83.55
HydraLoRA 60
2.72%
79.27
88.09
74.34
85.60
88.96
79.84
95.36
83.03
80. 10
83.84
MTL-LoRA
73
2.69%
80,55
88.38
75.26
85.80
88.41
80.45
95.57
81.61
80.75
84.09
LoRAMoE [4
2.19%
80.97
88.51
74.37
86.20
89.45
80.91
95.26
81.29
82.06
84.34
MoORE 1=0
2.72%
82,51
89.35
74.59
87.80
89.39
79.89
95.52
81.53
84.22
84.98
MoORE L=2
2.75%
81.91
89.35
74.56
87.60
89.83
80.45
95.46
81.53
84.52
85.02
MoORE L-4
2.78%
82,25
89.23
74.74
88.60
89.83
80.04
95.41
80.98
84.
85.02
MoORE
2.84%
82,25
89.31
74.74
87.80
89.55
79.89
95.48
82.40
84.60
85.11
Table 6: Results (%) of various methods on NLU-MTL. The best results on each dataset are shown in bold, and the second best results are shown in underline. We report the matched accuracy for MNLI, Matthew's 's correlation for CoLA, and average conrlaation for STS-B.
Method
#Params
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
Overall
LORA
22
2.09%
39.69
86.20
80.88
88.16
87.80
80.14
90.60
79.07
MixLoRA
30
3.00%
61.71
89.98
82.84
94.47
90.11
85.56
95.53
85.74
MoSLD 81
1.49%
58,55
90.34
85.29
94.95
90.08
89.53
96.33
86.44
MTL-LoRA
73
2.69%
60,65
90.26
87.01
95.26
91.95
90.98
96.10
87.46
HydraLoRA
660
2.72%
64,92
89.97
86.76
95.31
91.62
87.73
96.44
87.54
LoRAMoF
2.19%
62.84
91.01
88.97
95.55
91.01
90.61
96.56
88.08
MoORE r==0
2.72%
69.09
89.86
91.91
94.44
90.73
91.34
96.22
89.08
MoORE 7.=2
2.75%
69.23
90.35
89.70
95.37
90.75
91.34
96.56
89.04
4.2
Performance in Coaflict-Resistann
Tables 5 and compare MoORE with its competitors on CSR-MTL and NLU-MTL, respectively. Using a comparable number of learnable parameters for each dataset, MoORE achieves the best or comparable performance across all tasks and thus obtains the best average performance. These results demonstrate the superiority of MoORE in mitigating task conflict.
The impact of orthogonal adapter. In the experiment on CSR-MTL, we increase the number of Householder reflections in H (i.e.. L) from ments in performance. In the experiment on NLU-MTL, however, applying the orthogonal adapter may not improve performance. In our opinion, this phenomenon indicates that the commonsense reasoning tasks in CSR-MTL require more domain knowledge not covered by the pre-trained model. As a result, introducing the orthogonal adapter increases the number of learnable parameters and enhances the model capacity accordingly. In contrast, the text classification tasks in NLU-MTL rely more on the non-specific natural language knowledge captured by the pre-traine pre-trained model. Therefore, without introducing more learnable parameters, adjusting the singular values of the pre-trained weight matrix is sufficient to achieve encouraging performance.
8 and find that MoORE exhibits consistent improve-
Conffict-resistance regarding task number and difficulty. To further compare and analyze the conffict-resistacce capabilities of different methods, we conduct comparative experiments on CSR- MTL by varying the number and difficulty of the tasks. In particular, for each task in CSR-MTL, we first calculate the average of all the methods' performance based on the results in Table The average performance of the methods in a task measures the difficulty of the task for the base model - the lower the average performance is, the more difficult the task is. Then, we sort the tasks in ascending order based on their difficulty. Finally, we adapt the base model for the K tasks, K = 2, 2.... 9, and show the performance of different methods in Figure l(b) With the increase of task number and difficulty, all the methods suffer performance degradation because i) task conflict becomes severe as the number of tasks increases, and ii) difficult tasks are more likely to have conflicts with other tasks. Notably, MoORE outperforms all other basclines across all settings.
8

---

MoSLD
MTL-LoRA
LoRAMoE
HydraLoRA
MiXLoRA
MoORE
60
86
68
84
67
33
Performance
66
Parformancc
80
65
78
32
64
76
50
34
72
32
62
70
30
(a) MMLU
(b) IFEval
(c) BBH
(d) GPQA
68
60
59
64
50
Peformancai
57
62
56
60
55
79
58
56
54
(e) HvmanEval
(I) MBPP
GSM-8K
(h) Overall degradation
Figure 2: The loss of performance in OR-MTL. (a-g) The gray dashed line represents the performance of
LLaMA-3.1 8B before adaptation. (h) The overall performance degradation across all tasks.
4.3 Performance in
To demonstrate the superiority of MoORE in mitigating task oblivion, we compare various methods in the Language Model Evaluation Harness framework [19]. In particular, we first adapt the base model on CSR-MTI using different methods. Then, we evaluate the adapted models on OR-MTL, comparing them with the base model. Figure[]shows that MoORE consistently mitigates task oblivion across all seven tasks of OR-MTL, with an average performance drop of only 1.31% compared to the base model. It significantly outperforms the other adaptation methods. On HumanEval, MoORE achieves performance exceeding the original model, with an improvement LoRAMoR and MTLL-LRA also obtain slight improvements of 4.27% and 0.16%, respectively. This intriguing phenomenon may imply that the datasets in CSR-MTI. have some relevance to HvmanEval, providing information and capabilities beneficial for solving this task.
9.75%. Similarly,
Oblivion-resistanee regarding task number and difficulty. Furthermore, we conduct experiments to investigate how the ability of oblivion-resistance changes with increased task number and difficulty, The results in Figure [(c] show that after adapting to the top-3 tasks of CSR-MTL, MoORE and its strongest competitor, LoRAMME [14), achieve comparable performance on OR-MTL. However, with the increase of task number and difficulty, the performance of LoRAMoE changes slightly while the performance of MoORE increases and becomes superior to that of LoRAMoE: This result demonstrates that MoORE has stronger oblivion-resistance than its competitors.
The impact of task correlation. We investigate the reason for MoORE's empirically. In particular, consider the samples of several sub-tasks in MMLU and those six tasks in CSR-MTL. For each task/sub-task, we compute the average weights of MoORE's experts over its samples, i.e., 9k TD Enmep, (k)) Given a sub-task of MMLU and a task of CSR-MTL, denoted D3 and De,, respectively, measure their correlation by ||g|- g ||2. For each sub-task of MMLU, we record the performance degradation of MoORE compared to the base model. Figure shows normalized performance degradation and task correlation. This visualization result indicates that the Dolivion--esistaan arises from the correlation of tasks - for correlated tasks, the model can learn some common domain knowledge during adaptation and thus avoid catastrophic forgetting.
The generalization to unseen task Ds. The computation task-level weights requires task IDs input, but the ID of new task is unavailable during inference. Therefore, use the average of

---

$9
23
80
50
dfference
1.0
2.28
00
1000
Winegaande
02
ecpped poor "upuno ponce
Allonen
GGoyeoos
Auc
Aocrsy bunn" poosa pooues ydo
48 Aboyee offego
Aucionu' "o jnuoserouu
Aacusyy "aanii"eeeiia "eestuuo "ccups" wdy
augans Aurces
montflp joun
Ast Annwaus jongpe 14y
Besocuttio
couiing jooees odhy aunyeou
Bixtt besamn
aoooegt pondheses
acuees isinhes 1oindueo "Seepo
oddy TorTouee
acunuuu poaee udu
communaten
Lonusno adiecco aoe scotur
cotttd" ediagoo
pooge cettin cemuune uumu solayao may josolf
0.0
"oouss" poouss "fft
"adeeoo adiego
Figure 3: The visualization of normalized performance degradation and task correlation. The "difference" shown in the first row is the normalized performance degradation, i.e. (AcCrie AcCmmoeE) (OMmoR))/110%% /100%. The following matrix records the normalized task correlation. The element in the j-th row and the i-th column is llg. 2|2 ///xx, llg 96l12.
the existing task ID embeddings the embedding of the new task. Specifically, after adapting the base model on CSR-MTL, MoORE only learns the task IDs of the nine tasks CSR-MTL. During evaluation on OR-MTL, we use the average of the nine task ID embeddings the task ID embedding of each new task. The results in Figure [[c] and 2
validate the effectiveness this approach.
In addition, this experiment also explains the result in Figure [[c] In particular, the more tasks considered in the adaptation phase, the more likely some tasks correlate with those covered in the pre-training phase. As result, increasing the number of tasks in the adaptation phase helps enhance the obsisicn of MoORE.
4.4 Inference Efficiency
To compare inference efficiency, we adapt the base model using MoORE and other baselines, and evaluate their inference time on GPQA. For fair comparison, all experiments conducted single NVIDIA A100 GPU with the same batch size. The results in Figure [(d) the average inference time per batch for each method, which indicates that MoORE achieves superior inference efficiency compared to most baselines and is comparable to MixLoRA. Compared to the base model (i.e., the gray dashed line), MoORE moderately increases inference time.
5 Conclusion and Future Work
In this paper, we propose simple but effective model MoE-ization strategy for multi-task adaptation, leading novel MoE architecture, called MoORE. Experiments on various datasets show the effectiveness and superiority of MoORE, demonstrating that imposing orthogonality on experts and maintaining the column space of pre-trained weight matrix help improve the resistance of adapted model to task conflict and oblivion.
Limitations and future work. As shown in Figure [(d) the inference efficiency of MoORE is comparable to that of baselines. However, further reducing its complexity for large-scale applications challenging. particular, MoORE applies many more experts than classic MoE models, and the analytic experiments in Appendix C.4 show that it learns dense weights for the experts. In addition, besides imposing orthogonal adapters, we need to find a more effective solution enhance model capacity in broader scenarios. In theory, the results in Figures ((b) and [(c) show that increasing the number of tasks leads to higher risk of task conflict while helping mitigate task oblivion. How to understand this "trade-off""? How can the limitation in multi-task adaptation be quantified regarding the number of tasks? These are still open problems. Based on the above analysis, we plan to explore efficient implementations of MoORE and study its theoretical properties in the future.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation (92270110), Beijing Natural Science Foundation (L2333088) the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China. We also acknowledge the support provided by the fund for building world-class universities (disciplines) of Renmin University of China and by the funds from Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, and from Intelligent Social Governance Interdisciplinary Platform, Major Innovation Planning Interdisciplinary Platform for the "Double-Fire Class" Initiative, Renmin University of China.
References
[1] Hervé Abdi. Singular value decomposition (svd) and generalized singular value decomposition. Encyclopedia of measurement and statistics, 907(912):44, 2007.
[2] Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16196-16205, 2024.
[3] Mikel Artetxe. Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan lyer, Ramakanth Pasunuru, et Efficient large scale language modeling with mixtures of experts. In Proceedingy of the 2022 Conference Empirical Methods Natural Language Processing, pages 11699-11733 11732, 2022.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint (eXX:::210.0077:: 2021.
[5] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. In Thirty-Fourrt AAAI Conference on Artificial Intelligence, 2020.
[6] Åke Björck. Numerics of gram-schmidt orthogonalization. Linear Algebra and Its Applications, 197:297-316, 1994.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[8] Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michaelan Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
2018.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 20.v:2110.14166 2021.
[11] Michael Crawshaw. Multi-task learning with deep neural networks: survey. arXiv preprint arXiv:2009.0777 097969 2020.
[12] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization mixture-of-experts language models. arXiv preprint arXiv:2401.066 2024.
[13] Pala Tej Deep, Rishabh Bhardwaj. and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-based sampling, arXiv preprint 16111:2406.11677, 2024.

---

[14] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 1932-1945, 2024.
[15] Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang. Daojing He, et al. Parameter competition balancing for model merging. arXiv preprint 1rXiv:2410.0233% 2024.
[16] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Efficient scaling of language models with mixture-of-eeperts. In International conference on machine learning. pages 5547-5569. PMLR, 2022.
[17] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research, 23(120):1-39, 2022.
[18] Jinyuan Feng, Zhiqiang Pu, Tianyi Hu, Dongmin Li, Xiaolin Ai, and Huimu Wang. Omoe: Diversifying mixture of low-rank adaptation by orthogonal finetuning. arXiv preprint arXiv:2501.10062, 2025.
[19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[20] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407:21788, 2024.
[21] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[22] Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. arXiv preprint ar arXivv2410.07524, 2024.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations ( (ICLR), 2021.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[25] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models, In International Conference on Learning Representations, 2021.
[26] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging Tuning-free free perf-pceformance model merging. Advances in Neural Information Processing Systems, 37: 37:122741-122769 2024.
[27] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation. 3(1):79-87, 87, 1991.
[28] Aran Komatsuzaki, Joan Puigcerver, James Lee-Th Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[29] Dmitry Lepikhin, HyoukJoong Lee. Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations, 2021.
[30] Dengchun Li, Yingzi Ma, Naizheng Wang. Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404. 21.e224444..5555 15159, 2024.
[31] Mengqi Liao, Wei Chen, Junfeng Shen, Shengman Guo, and Huaiyu Wan. Hmora: Making Iims more effective with hierarchical mixture of lora experts. In The Thirteenth International Conference on Learning Representations, 2025.
[32] Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, and Jianling Sun. Pemt: Multi-task correlation guided mixture-of Association for Computational Linguistics ACL 2024, pages 1869-68832 2024.
enables parameter-efficient transfer learning. In Findings of the
[33] Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Paraaeeter-fffcieent fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 37:41425-41444 2024.
[34] Bingehang Liu, Chaoyu Chen, Zi Gong, Cong Liao, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, et al. Mftcoder: Boosting code Ilms with multitask fine-tuning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5440, 2024.
[35] Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Ilms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1 1104-1114 1144, 2024.
[36] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. In International Conference on Learning Representations, 2024.
[37] Yen-Cheng Liu, Chih- Yao Ma, Junjiao Tian, Zijian He, and 7solt Kira. Polyhistor: parameter- efficient multi-task adaptation for dense vision tasks. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 36889-36901, 2022.
[38] Zefang Liu and Jiahua Luo. Adamole: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts. In First Conference on Language Modeling, 2024.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05100, 2017.
[40] Yuxiang Lu, Shalayiding Sirejiding, Yue Ding, Chunlin Wang, and Hongtao Lu. Prompt guided transformer for multi-task dense prediction. IEEE Transactions on Multimedia, 26:6375-6385, 2024.
[41] Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efffcieet fine-tuning for large language models. arXiv preprint 2xXiv:2402.12851, 2024.
[42] Fan Lyu, Shuai Wang, Wei Feng, Zihan Ye, Fuyuan Hu, and Song Wang. Multi-domain multi-task rehearsal for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8819-8827 2021.
[43] Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, and Junfeng Zhao. Parameter efficient quasi-orthogonal fine-tuning via givens rotation. International Conference on Machine PMLR, 2024.
Learning, pages
13

---

[44] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghami, and James Henderson. Parameter-efficienn multi-task fine-tuning for transformers via shared hypernetwrrks. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 565-576, 2021.
[45] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[46] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-imase mixture of experts. Advances in Neural Information Processing Systems, 35:9564 9576, 2022.
[47] Taishi Nakamura, Takuya Akiba, Kazuki Fujüi, Yusuke Oda, Rio Yokota, and Jun Suzuki. Drop- upcycling: Training sparse mixture of experts with partial re-initialization. In The Thirteenth International Conference on Learning Representations, 2025.
[48] Zelin Peng, Zhengqin Xu, Zhilin Zeng, Xiaokang Yang, and Wei Shen. Sam-parser: Fine-tuning sam efficiently by parameter space reconstruction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4515-4523, 2024.
[49] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362, 2023.
[50] Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy-Chowdhury and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10955-10964, 2022.
[51] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-pros proof q&a benchmark. In First Conference on Language Modeling, 2024.
[52] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[54] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: 1904. 09728, 2019.
[55] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-experrs layer. In International Conference on Learning Representations, 2017.
[56] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning. Advances in neural information processing systems, 35:37484-37496. 2022.
[57] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision-and-languagg tasks. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 5227-5237, 2022.
[58] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 11X1v.2210.09261, 2022.
14

---

[59] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQ question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies, Volume I (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[60] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565- 2024.
[61] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey, IEEE transactions on pattern analysis and machine intelligence, 44(7):3614-366 2021.
[62] Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics, 9(9):1363, 2020.
[63] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, 2019.
[64] Fan Wang, Juyong Jiang, Chansung Park, Sunghun Kim, and Jing Tang. Kasa: Knowledge- aware singular-van value adaptation of large language models. arXiv preprint 1rXiv:2412.06071, 2024.
[65] Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai, Jingdong Wang, and Yong Liu. A multimodal, multi-task adapting framework for video action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5517-5525, 2024.
[66] Renzhi Wang and Piji Li. Lemoe: Advanced mixture of experts adaptor for lifelong model editing of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2551-2575, 2024.
[67] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310. 14152, 2023.
[68] Yiming Wang, Yu Lin, Xiaodong Zeng, and Guamnan Zhang. Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.1 1.1 11501, 2023.
[69] Zhiwu Wang. Yichen Wu, Renzhen Wang. Haokun Lin, Quanziang Wang, Qian Zhao, and Deyu Meng. Singular value fine-tuning for few-shot class-incremental learning. arXiv preprint arXiv:2503.10214 2025.
[70] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1293-11302 2019.
[71] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv: 2023.
[72] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
[73] Yaming Yang, Dilxat Muhtar, Yelong Shen, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang, et al. Mtl-lora: Low-rank adaptation for multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22010-2201 2025.
15

---

[74] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In Forty-first International Conference Machine Learning. 2024.
[75] Shen Yuan, Haotian Liu, and Hongteng Xu. Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[76] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
[77] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
[78] Qizhen Irene Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar Talupuru, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, Ahmet Üstün, et al. Bam! just like that: Simple and efficient parameter upcycling for mixture of experts. Advances in Neural Information Processing Systems, 37:56304-5653 2024.
[79] Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey negative transfer. IEEE/CAA Journal of Automatica Sinica, 10(2):305-323 -22900 2022.
[80] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12):5586-5609 2021.
[81] Lulu Zhao, Weihao Zeng, Shi Xiaofeng, and Hua Zhou. Mosld: An extremely parameter- efficient ceixture-of-sharee loras for multi-task learning. In Proceedings of the 31 International Conference on Computational Linguistics, pages 1647-1659, 2025.
[82] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint 2r03v 18223, 1(2), 2023.
[83] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Inslloition-folloiinng evaluation for large language models, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims made in the abstract and introduction accurately reflect the contri- butions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the proposed method in Limitations, such as its inability to fully leverage the potential of orthogonal fine-tuning.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification:
We provide the complete proof process in both the main text and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The detailed hyperparameter configurations well experimental setups thoroughly documented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released the code anonymous GitHub repository. The datasets used in the experiments are all publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips. cc/
public/ http://uuieeeeeeeeeneeee
ss oon ollcy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (https
/ //nips. httccc/pbbb. wtcp/pbblcc/gee wttp//uiieeeeeee/ SCoee ss onPol icyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The detailed hyperparameter configurations as well as experimental setups are thoroughly documented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not have sufficient computational resources to run multiple rounds of experiments to obtain error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided detailed information about the computational resources used in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics o http:///wwwsssscoooo/iiicicdttdd
Answer: [Yes]
Justification: Every aspect of the paper complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work performed in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators original owners of assets including code, data, and models used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapesswi haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the comments in the released code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The proposed method is specifically designed for multi-tass adaptation of LLMs, making LLMs essential component of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httt::/s//eurrpp.cc///oaefeeeeee
LLM) for what should or should not be described.
23

---

A Hyperparameters and Implementation Details
The detailed hyperparameter setups presented in Table 7 on one NVIDIA A100 GPU.
Both training and testing conducted
Table 7: Hyperparameter configurations of MoORA on CSR-MTL and NLU-MTL.
Hyperparameters
MoORA
Cutoff' Length
512
Batch Size
/ 64
Epochs
2 //5
Learning Rate
3E-04
LR scheduler
Warmup-Stable Stable--abeeey
Warmup Ratio
5%
Decay Ratio
5%
Optimizer
AdamW W
Dropout Rate
0.0
Target Modules
Q K, O, Up, Down, Gate 128
Dr
D
64
L
B Tasks and Datasets
Detailed information about the CSR-MTL, the NLU-MTL and the OR-MTL presented in Tables X Table Jablegandd and Table I0 respectively. These tables include the sizes of the training and test sets, as well as the task types.
Table 8: The basic information of CSR-MTL
Task Name
#Train
#Test
Task Type
ARC-Challenge
1,119
1,172
Question Answering
ARC-Easy
2,250
2,380
Question Answering
BoolQ
9,427
3,270
Text Classification
OpenBookQA
4,957
500
Question Answering
PIQA
16,100
1,840
Question Answering
SocialIQA
33,410
1,954
Question Answering
HellaSwar
39,905
10.042
Sentence Completion
WinoGrande
9,248
1,267
Fill in the Blank
CommonsenseQAA
9,741
1,140
Question Answering
Table 9: The basic information of NLU-MTL
Task Name
#Train
#Validation
Task Type
CoLA
8,551
1,043
Text Classification
MNLI
392,702
9,815
MRPC
3,668
408
QNLI
104,743
5,463
QQP
363,846
40,430
RTE
67,350
873
SST-2
2,491
277

---

Table 10: The basic information of OR-MTL.
Task Name
#Test
Task Type
MMLU
14,042
Question Answering
1FEval
541
Text Generation
BBH
6,511
Question Answering
GPQA
448
Question Answering
HumanEval
164
Text Generation
MBPP
500
Text Generation
GSM-8K
1,319
Text Generation
C More Experimental Results
C.1 Performance on other Model Architectures in Conflict-Resistacce
To evaluate the cross-model performance of MoORE, we take Qwen-2.5 7B as the backbone model and apply various MoE-based multi-task adaptation methods to it. Experimental results on CSR-MTL shown in Table I1 We can find that MoORE outperforms baselines consistently under different hyperparameter settings, and the performance of MoORE is robust to the change of L. These results verify the robustness of our method and its compatibility with various model architectures.
Table 11: Results (%) of Qwen-2.5 7B adapted hy various methods on CSR-MTI. The best results on cach dataset are shown in bold, and the second best results are shown in underline.
Method
#Params
ARC-C ARC-E BoolQ OBQA PIQA SIQA HellaS WinoG CSQA
Overall
LORA 25
2.12%
87.29
91.88
70.37
90.60
87.92 79.02 92.73
74.98
84.36
84.35
MixLoRA
30
3.32%
86.86
91.75
71.38
91.40
89.28 80.45 95.06
78.45
85.09
85.53
MoSLD 81
1.43%
86.69
92.13
71.96
90.40
89.66 80.40
95.17
78.30
85.50
85.58
HydraLoRA
600
2.83%
88.23
91.54
72.57
91.20
85.80
80.09 95.95
82.24
85.26
85.87
MTL-LoRA
73
2.80%
87.80
92.09
72.72
92.20
86.07
79.79
95.92
80.66
84.36
85.73
LoRAMoE
14
2.21%
87.20
91.84
73.61
90.40
90.37
81.27
95.99
80.27
84.60
86. 17
MoORE L=0
2,29%
88.06
92.17
73.49
92.20
90.37
79.58
95.40
81.53
84.03
86.31
MoORE L=2
2.32%
88.23
92.34
73.09
91.60
90.32
80.60
95.50
81.45
84.19
86.37
MoORE /-44
2.35%
88.40
92.21
73.18
91.40
90.21
81.27
95.25
80.58
84.93
86.38
MoORE
2.53%
88.23
92.59
73.30
92.40
90.26
81.12
95.50
81.37
84.93
86.63
C.2 Numerical Performance in Oblivion-Resistance
Following LLaMA-3.1 8B, we select MMLU 24 23), IFEval 83 BIG-Bench Hard (BBH) (58), GPQA [51), HumanEval (7]. MBPP (4) and GSM-8K I0 as evaluation tasks and adopt the same experiments are presented in
few-shot settings. More detailed results of the
Table 12 corresponding to the results shown in Figures Tíc) and
C.3 Ablation Study
To analyze the impact of the task-level routing, sample-level routing, and orthogonal adapter quan- titatively, we conduct ablation study on the CSR-MTL dataset. In Table L3 we record the overall performance achieved when only one of these three modules is applied. In particular, without the orthogonal adapter, our method degrades to the MoE with fixed experts. When merely applying the orthogonal adapter without any routing, our method is simplified as the single-task adaptation method HRA 75). The results show that when only one of these three modules is applied, even with wider network and more trainable parameters, the model's 's performance declines significantly across the entire dataset. These results demonstrate that all these three modules contribute to the overall effectiveness of MoORE.
25

---

Table 12: The loss of performance in the original tasks. Each method is represented by of data: the row indicates performance on the current task, while second row shows difference between post-fine-tuning score and the pre-fine-tuning score.
Category
General
Reasoning
Code
Math
Task
MMLU
IFEval
BBH
GPQA
HumanFval
MBPP
GSM-8K
Overall
#Shots
5
0
0
0
3
8
Llama-3.1 1-8B-Instruct
68. 18.15
84.53
46.09
34.15
56.71
59.00
85.52
62.02
32.40
29.14
13.65
27.23
0.00
0.00
2.12
14.93
LoRA
-35.75
-55.39
-32.44
-6.92
-56.71
-59.00
-83.40
-47.09
63.74
78.06
34.16
31,47
60.98
57,40
78.54
57.76
LoRAMoE
-4.41
-6.47
-11.93
-2.68
4.27
-1.60
-6.98
-4.26
61.77
75.06
34.74
28.57
56.10
53.60
70.05
54.27
MoSLD
-6.38
-9.47
-11.35
-5,58
-0.61
-5.40
-15.47
-7.75
61.55
72.30
31.59
31.92
57.32
51.60
66.57
53.26
MTL-LoRA
-6.60
-12.23
-14.50
-2.23
0.61
-7.40
-18.95
-8.76
62.34
69.78
30.27
30.36
54.27
54.80
66.41
52.60
HydraLoRA
-5.81
-14.75
-15.82
-3.79
-2.44
-4.20
-19.11
-9.42
64.47
77.82
38.21
30.13
54.27
56.20
74.83
56.56
MixLoRA
-3.68
-6.71
-7.88
-4.02
-2.44
-2.80
-10.69
-5.46
65.32
82.49
40.21
31.92
66.46
58.20
80.36
60.71
MoORE
-2.83
-2.04
-5,88
-2.23
9.75
-0.80
-5.16
-1.31
Table 13: Ablation studies of MoORE on CSR-MTL. indicates the absence that component.
Method
D,
D,
L
Overall #Params
16
x
x
0.14%
79.55
Merely Apply
32
0.29%
81.80
Task-level Routing
0.58%
83.06
128
1.16%
83.53
0.39%
82.30
Merely Apply x
32
0.78%
82.91
Sample-level Routing
x
64
X
1.57%
83.57
x
128
3.13% X
83.80
Merely Apply x
0.03%
80.76
0.06%
82.15
Orthogonal Adapter
X
0.12%
82.98
X
x
16
0.25%
83.90
MoORE
128
64
2.84%
85.11
C.4 Analysis of Routing Weights
To investigate the router's preferences for the experts MoORE, we analyze the distribution of routing weights across different tasks in the CSR-MTL dataset. Take the Q module from the first attention layer of the adapted model an example. For each task, we compute and show the mean (red point) and variance (blue stem) of each expert's weight over all samples of the task in Figures 4 and 6 In each figure, the mean weight of an expert shown red point, and the blue stem associated with the point shows the standard deviation. These visualization results provide us with some insights in the mechanism behind MoORE:
Preference on the originally important experts. Compared to other components, the routing weights corresponding to the principal components of W (i.e., the experts corresponding to large singular values) are adjusted more significantly. The routing weights corresponding to the significant experts are distributed relatively symmetrically around the zero scale, whereas the routing weights for the minor components more concentrated above the zero scale. This
26

---

0.75
Average Weights of Experts
0.50
0.25
0.00
-0.25
-0.50-
-0.75
500
1000
1500
2000
2500
3000
3500
4006
Expert Index
Figure 4: The mean and variance of routing weights obtained in HellaS.
indicates that the model tends to focus on adjusting the routing weights of the experts that are important in the pre-training phase. In other words, these originally important experts often maintained their significance when adapting for new task.
Based on the mean and variance of the routing weights, the nine tasks in CSR-MTL can be grouped into three categories:
Large Mean, Large Variance: Tasks such as Hellaswag fall into this group. MoORE performs well on the Hellaswag. A large mean indicates that the model significantly enhances or discards well-learned pretraining knowledge, while large variance means that the model can capture substantial differences between samples and assign different experts them with high flexibility.
Small Mean, Large Variance:
Tasks such OBQA, PIQA. SIQA, and CSQA belong to this group. This result indicates that for each of these tasks, its samples have significant diversity, and MoORE needs to activate different experts to handle different samples.
Small Mean and Variance: Tasks such as ARC-C, ARC-E, BoolQ, and Winogrande (WinoG) are in this group. Except for ARC-E, MoORE performs poorly on these tasks. A small mean indicates minimal adjustment to pretraining knowledge, and small variance suggests minor differences between samples. For the relatively simple ARC-E task, the model may have already mastered the ability solve it, requiring little adjustment. However, for the more challenging tasks in this group, the model may struggle to learn the ability to solve them within the column space spanned by the singular vectors.
D Broader Impacts
MoORE can effectively enhance the performance LLMs multi-task adaptation and facilitate the application of LLMs in real-world scenarios. Similar to other multi-task adaptation methods, such as MixLoRA and HydraLoRA, MoORE may result in some potential negative social impacts, such as the risk of abuse for adapting LLMs in illegal activities. However, these issues not unique to MoORE - other adaptation methods are also susceptible similar risks. Addressing these issues thoroughly will require advancements in technology, the establishment of relevant legal frameworks, and shifts in societal perspectives.
27

---

0.75
Average Weights of Experts
0.50
0.25
0.00
-0.50
-0.7%
500
1000
1500
2000
2500
3000
3500
4000
Expert Index
(a) OBQA
Average Weights of Experts
0.50
0.25
0.00
-0.25
0.50
-0.75
4000
Expert
(b) PIQA
0.75
Average Weights of Experts
0.25
0.00
-0.50
-0.75
500
1000
1500
2000
2500
3000
3500
4006
Expert Index
(c) SIQA
0.75
Average Weights of Experts
0.50
0.00
-0.25
-0.75
500
2500
4000
Expert Index
(d) CSQA
Figure 5: The mean and variance of routing weights obtained in four QA tasks.
28

---

0.75
Average Weights Experts
0.50
0.25
0.00
-0.50
-0.7%
500
1000
1500
2000
2500
3000
3500
4000
Expert Index
(a) ARC-C
Average Weights of Experts
0.50
0.25
0.00
-0.25
0.50
-0.75
4000
Expert
(b) ARC-E
0.75
Average Weights of Experts
0.25
0.00
-0.50
-0.75
500
1000
1500
2000
2500
3000
3500
4006
Expert Index
(c) BoolQ
0.75
Average Weights of Experts
0.50
0.25
0.00
-0.25
-0.75
500
2500
4000
Expert Index
(d) WinoG
Figure 6: The mean and variance of routing weights obtained in the remaining four tasks of CSR-MTL.
29


Reviews:
[{"review_id": "5v2mE3S1y3", "reviewer": "Reviewer_P3D7", "summary": "Mixture of Experts is now a popular paradigm for multi-task adaptation of transformers. In this approach an adapter is assigned/learned for each task and a learnable router selects the appropriate adapter for the given task. This achieves parameter efficient fine-tuning. An important drawback of this approach is destructive interference and catastrophic forgetting. A number of solutions has been suggested for to solve this problem. Many of them lack elegance and a theoretical foundation.\n\nThe present paper proposes an simple yet effective approach to solve this problem. It constructs a stack of \"weak\"experts using orthogonal factorization of the foundation weight matrix. A router is learned for choosing among these simple experts. The method is found to be effective on benchmark tasks.", "questions": "1. Is it possible to provide a set of formulations in terms of the underlying linear algebra operations of the proposed method vis-a-vis other MoE-zation techniques for PEFT?\n\n2. How does the proposed method compare with other state-of-art PEFT techniques in terms of efficiency on multi-task benchmarks?", "limitations": "The proposed approach has a great deal of similarity with the eigenstructure assignment problem in linear algebra. It would be beneficial to study this relation in order to have a better theoretical understanding of the interference problem.Often \"regularization functionals\"are used in eigenstructure assignment. Such functionals may be useful for PEFT as well.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents an excellent application of linear algebra tools for mixture of experts formulation of the parameter efficient fine-tuning process. The results demonstrate the effectiveness of this relatively simple approach.\n\nA number of very similar approaches to MoE-ization is existent in literature. Many of them are compared in the experimental results section. They are also qualitatively described. A more mathematical formulation of the \"weak\" MoE approach as opposed to exisitng ones using routers in the light of linear algebra concepts might be possible and will definitely add more value to the topic/ It will definitely affirm the novelty of the proposed technique.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your appreciation of our work. For your concerns, we provide our answer below.\n\n\n**Q1: The formulations of other MoE-zation methods for PEFT and the comparisons for various methods mathematically.**\n\n**A1:** In our submission, **the formulation of most existing MoE-zation methods has been shown in Eq. (2), and the formulation of our MoORE is given in Eq. (6).** We paste the equations below for a quick review.\n\nGiven a pre-trained parameter matrix $\\mathbf{W}$, whose SVD is $\\mathbf{U}\\text{diag}(\\boldsymbol{\\sigma})\\mathbf{V}^{\\top}$ and an input of the $k$-th task $\\mathbf{x}^{(k)}$, the formulations of MoORE and baselines are shown below:\n\nMoORE: \n\n$\\mathbf{y} =\\mathbf{U}(g(\\mathbf{x}^{k})+\\boldsymbol{\\sigma})\\mathbf{V}^{\\top}\\mathbf{WHx}^{(k)}=\\mathbf{WHx}^{(k)}+\\sum_{d=1}^{D}g_d(\\mathbf{x}^{(k)})(\\mathbf{u}_d\\mathbf{v}_d^{\\top}\\mathbf{H})\\mathbf{x}^{(k)}$, \n\nwhere $\\mathbf{H}$ is a learnable Householder reflection matrix, and $g$ is the router determining the weight of each expert $\\mathbf{u}_d\\mathbf{v}_d^{\\top}\\mathbf{H}$.\n\nBaselines: \n\n$\\mathbf{y} =\\mathbf{Wx}^{(k)}+\\sum_{m}g_m(\\mathbf{x}^{(k)})\\mathbf{B}_m\\mathbf{A}_m\\mathbf{x}^{(k)}$, \n\nwhere $\\mathbf{B}_m$ and $\\mathbf{A}_m$ are low-rank matrices of the $m$-th expert, and $g_m$ is the weight of the expert determined by a router.\n\nIn Table 1, we have shown the implementation details of the methods, corresponding to the specific configurations of the routers and experts shown in Eqs. (2) and (6).\n\nIn Lines 161-176, we have explained the differences between MoORE and the existing methods from the perspective of linear algebra. Following your suggestion, we emphasize the differences below for a quick review.\n\n**1) The maintenance of orthogonality mitigates task conflict:** MoORE leverages $D$ rank-1 and orthogonal experts extracted from the given parameter matrix. The orthogonality eliminates the information redundancy across our experts. In contrast, the baselines learns $M$ LoRA adapters as experts ($M\\ll D$), without any orthogonality constraints. As a result, for each baseline method, the output spaces of its experts overlap, and accordingly, different experts may output similar latent representations with redundant information. Eliminating the information redundancy helps mitigate task conflict, i.e., the orthogonal experts of our model can serve for different tasks.\n\n**2) The maintenance of output space mitigates task oblivion:** Because MoORE leverages experts extracted from the given parameter matrix, it does not change the output space of the model, as shown in Eq. (8). In PEFT-based single-task adaptation, the work like HRA [a] has shown that maintaining the range of parameter matrix (the output space of model) is important for avoiding losing the model capability obtained in the pre-training phase. Our work further extends this claim to both simultaneous and sequential multi-task adaptation scenarios.\n\n[a] Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation. NeurIPS 2024.\n\n**Q2: The efficiency of MoORE**\n\n**A2:** In our submission, **Figure 1d shows a comparison of various methods on inference efficiency.** Among them, MoORE is only slightly less efficient than MixLoRA, but outperforms all other baselines. This demonstrates the efficiency of MoORE in practice. Besides, we have analyzed the computational complexity of various methods in Table 2 and Lines 177-191, which demonstrates in theory that the complexity of MoORE is comparable to the baselines.\n\n**Q3: The connection to the eigenstructure assignment problem**\n\n**A3:** Thanks for your insightful suggestion! We think connecting MoORE to the eigenstructure assignment problem is interesting. Although it is out of the scope of this work, we will definitely explore their connections in the near future. In the final version, we will discuss the potential connection of our work with eigenstructure assignment in the conclusion section and outline it as a future direction.\n\nNote that, in Appendix C.3, we have provided a detailed empirical analysis on the singular spectrum of the parameter matrix (i.e., the weight of experts determined by the route) for different tasks. Some empirical patterns have been found and discussed. In our opinion, this preliminary analysis makes the first step towards exploring the theory behind MoORE.\n\nIn summary, we hope the above responses can resolve your concerns thoroughly and further enhance your confidence to support our work in the following decision phase."}, {"review_id": "GvrKwBJ4ZS", "reviewer": "Reviewer_eS7y", "summary": "This paper presents MoORE, which repackages task-aware Singular Value Fine-tuning within a MoE story. \n\nThe authors decompose model weights via SVD, term each pair of left/right singular vectors as \"rank-1 experts\", and refer to the natural summation in SVD reconstruction as a \"mixture\" of experts. Projected singular values are rebranded as \"routing weights\" (router).\n\nMethodologically, model weights first undergo SVD decomposition. Subsequently, a task_id is input to train two projections: one on singular values and another between inputs and model weights. Household adaptors are utilized to ensure orthogonality.\n\nThe core contribution lies in applying Singular Value Fine-tuning to address key multi-task adaptation challenges: task conflicts and catastrophic forgetting.\n\nTheoretical underpinnings regarding orthogonality and column space preservation provide justification for the approach's efficacy.\n\nEmpirical results demonstrate modest improvements.", "questions": "- The paper's central methodology appears to be functionally equivalent to established Singular Value Decomposition (SVD) based fine-tuning techniques. Could the authors precisely articulate the fundamental technical novelty of \"MoORE\" that distinguishes it from prior work on using SVD for model adaptation? Specifically, what aspects of the proposed method constitute a genuine innovation beyond a rebranding of existing concepts?\n- The paper frames its approach as \"Model MoE-ization,\" yet the theoretical motivation and empirical evaluation appear disconnected from the established principles of Mixture-of-Experts (MoE) architectures. Given that the method operates on pre-trained models via SVD and traditional MoEs focus on sparse activation during training, what is the theoretical justification for this MoE analogy? Why decomposing pre-trained model weights should be considered \"MoE-ization\" beyond superficial structural similarities? Consequently, why were direct comparisons with genuine MoE models omitted from the experiments, which instead focused on LoRA-based methods? \n- How does the proposed framework address scenarios where task identifiers are unavailable, such as in continual learning with unseen tasks or contexts where tasks are defined implicitly? This dependency seems to severely constrain the method's practical applicability in real-world settings.", "limitations": "the authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths:**\n- Engineering-Friendly: The proposed method is straightforward to implement, requiring only SVD decomposition and lightweight projection layers, making it practically accessible for deployment.\n- The paper exhibits clear exposition with logical flow, making the story easy to follow despite the conceptual reframing.\n\n**Weaknesses:**\n- Fundamental Lack of Novelty and Conceptual Misrepresentation: The core contribution appears to be a rebranding of established SVD fine-tuning techniques rather than genuine innovation. The authors introduce \"Model MoE-ization\" as a novel concept, but the proposed method shows no essential difference from existing singular value fine-tuning approaches ( e.g. https://proceedings.neurips.cc/paper_files/paper/2022/file/f3bfbd65743e60c685a3845bd61ce15f-Paper-Conference.pdf ). The MoE terminology creates conceptual confusion without adding substantive technical value. This MoORE is fundamentally singular value adaptation disguised with MoE nomenclature.\n- Questionable Theoretical Motivation for \"Model MoE-ization\" and Misleading Framing: The rationale behind applying MoE-ization to already pre-trained language models remains unclear. Traditional MoE architectures primarily sparsify model parameters when processing vast training data to prevent overfitting, while the proposed SVD decomposition serves entirely different purposes. What meaningful purpose does \"MoE-izing\" an already-trained language model serve beyond SVD decomposition with alternative terminology? The authors fail to articulate why decomposing pre-trained model weights should be considered \"MoE-ization\" beyond superficial structural similarities.\n- Despite the MoE framing, the experimental evaluation completely omits comparisons with actual MoE methods, instead focusing on LoRA-based approaches. This fundamental disconnect between the conceptual narrative and empirical validation raises serious concerns about the paper's positioning and undermines the paper's central narrative.\n- Limited Generalizability due to Task_id Dependency: The method's reliance on explicit task_ids restricts its applicability to scenarios with tasks without a task_id. This fundamental limitation contradicts claims of general-purpose multi-task adaptation, as the approach cannot handle scenarios where task_id is unavailable or unseen.\n- Inconsistent Performance Claims: The authors claim MoORE \"consistently outperforms existing multi-task adaptation methods\", yet the experimental results reveal performance dependence on hyperparameter L selection. The paper lacks comprehensive hyperparameter sensitivity analysis.\n- Limited Model Scope: Despite that the MoORE theory seems to be general, all experiments are confined to one single language model, raising questions about cross-model transferability and robustness, limiting confidence in applicability on diverse model families and architectures.\n- Insufficient Mechanistic Validation: While the paper proposes that column space preservation promotes forgetting resistance, it lacks direct quantitative evidence supporting this causal relationship. The theoretical claim requires empirical validation through metrics such as activation space similarity or representation overlap analysis before and after fine-tuning.\n\n\n**Issues:**\n- Incomplete Analysis of SVD Components: The method consistently employs all singular vectors without analyzing performance variations across different rank selections, missing opportunities for efficiency optimization and theoretical insights.\n- Inadequate Table Documentation: Table captions are overly brief and fail to clearly explain notation and symbols (e.g., parameter L).\n- Missing Routing Analysis: The paper lacks analysis of routing strategy sparsity patterns or expert collapse phenomena, which would be essential for genuine MoE evaluation. Though given the fundamental differences from traditional MoE experts, this may be less critical.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. Below, we try to resolve your concerns one by one.\n\n\n**Q1: The novelty of the proposed method.** \n\n**A1:** We respectfully disagree with the comment that our work lacks novelty for the following reasons:\n\n**1. New methodology for constructing MoE:** To the best of our knowledge, our work makes the first attempt to convert an arbitrary model to an MoE architecture with the help of SVD decomposition. Different from existing MoE-based adaptation methods, which achieve MoE architectures by learning and routing external low-rank adapters, our MoORE construct orthogonal experts intrinsically from the SVD of a pretrained parameter matrix. \n\n**2. New methodology for achieving orthogonal experts efficiently:** Among existing MoE-based multi-task adaptation methods, OMoE is the only one considering the orthogonality of experts. However, it introduces orthogonality among experts’ outputs using the Gram-Schmidt orthogonalization algorithm, which requires high computational complexity. Our MoORE achieves orthogonal experts naturally by the SVD of the parameter matrix. The Housholder reflection imposed on each expert maintains their orthogonality while increasing their model capacity at the same time.\n\n**3. Considering the mitigation of task oblivion, which is seldom considered by existing methods:** Because of maintaining the column space of the original parameter matrix, MoORE has the ability to mitigate task oblivion. Existing multi-task adaptation methods mainly consider mitigating the conflicts among the downstream tasks. Few of them consider maintaining the original pre-training knowledge and mitigating the oblivion of pre-training tasks. Our work fills this blank.\n\n**4. Valuable routing analysis on MoE-based multi-task adaptation:** In the Appendix, we analyze MoORE’s routing weights across different tasks and provide valuable conclusions.\n\n**Q2: The differences between MoORE and existing SVD-based fine-tuning methods**\n\n**A2:** There are several key differences between MoORE and existing SVD-based fine-tuning methods like SVF [a]:\n\n**1. Methodology:** MoORE converts pre-trained models to MoE architectures in multi-task adaptation scenarios. On the contrary, without introducing any routing mechanism, existing SVD-based fine-tuning methods do not result in MoEs.\n\n**2. Application:** MoORE is designed to address multi-task learning problems of LLMs, mitigating task conflict and oblivion. SVF focuses on a few-shot image segmentation tasks. It does not consider learning a single model in simultaneous or sequential multi-task adaptation scenarios.\n\n[a] Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning. NeurIPS 2022.\n\n\n**Q3: The rationality and motivation for introducing the new concept \"Model MoE-ization\"**\n\n**A3:** Firstly, we must remind the reviewer that when MoE was first proposed in 1991, it was not sparsely activated. It was not until 2017 that sparsely activated MoE architectures were introduced, and the purpose of sparse activation was mainly to increase model capacity with high computational efficiency.\n\nWe introduce the concept \"Model MoE-ization\" for the following reasons.\n\n1) The community of multi-task adaptation, state-of-the-art methods introduce multiple LoRA adapters into pre-trained models and learn a router to adjust their significance based on input data or tasks, which actually converts the pre-trained model to an MoE architecture. In fact, the baselines like OMoE, LoRAMoE, MixLoRA, and MoSLD have explicitly claimed that they convert a pretrained model to an MoE architecture for multi-task adaptation. We introduce the concept \"Model MoE-ization\" to name this strategy, which matches well with its principle. \n\n2) In fact, the development of MoE has always been about increasing model capacity to solve more complex tasks, while at the same time minimizing the loss in computational efficiency. Our SVD-based “Model MoE-ization” strategy is fully aligned with this philosophy. Our MoORE method alleviates task conflict and task forgetting, achieves better multi-task adaptation compared to other MoE-based baselines, and also reduces inference time overhead.\n\nDespite the MoE framing, the experimental evaluation completely omits comparisons with actual MoE methods, instead focusing on LoRA-based approaches. This fundamental disconnect between the conceptual narrative and empirical validation raises serious concerns about the paper's positioning and undermines the paper's central narrative.\n\n**Q4: The lack of MoE baselines**\n\n**A4:** **We respectfully disagree with this comment because all the baselines in our study are based on MoE architectures.** As shown in **A3**, the baselines like OMoE, LoRAMoE, MixLoRA, and MoSLD have explicitly claimed that they achieve multi-task adaptation by learning a model with an MoE architecture. The remaining methods (MTL-LoRA and HydraLoRA) leverage MoE architectures as well, based on their implementations. \n\nNote that it is reasonable to implement experts as low-rank adapters --- if each expert is full rank, the computational complexity will be too high and the model size will be too large. For the LLMs using MoE architectures, their experts are always simple and lightweight in practice.\n\n\n**Q5: The risk of limited generalizability due to the usage of task ID**\n\n**A5:** In our oblivion-resistance multi-task learning experiment (OR-MTL), we have considered scenarios where task IDs are unavailable - when new tasks come, we use the average of existing task ID embeddings as their ID embeddings. Specifically, after adapting LLaMA with MoORE on CSR-MTL, MoORE only learns the task IDs of the nine tasks in CSR-MTL. During evaluation on OR-MTL, we use the average of the nine task ID embeddings as the task ID embedding of each new task in OR-MTL. The results in Figure 1(c) and Figure 2 validate the effectiveness of this approach. We will add the content above in the revised version.\n\n\n**Q6: The lack of sensitivity analysis on $L$**\n\n**A6:** Table 3 has shown the robustness of our method to $L$: 1) When $L\\in\\{0, 2, 4, 8\\}$, MoORE consistently outperforms all other baselines. 2) As $L$ varies, the results of MoORE do not change significantly—the variation is only 0.16%. These results demonstrate that the model is robust to $L$.\n\n\n**Q7: Add experiments on other model architectures.**\n\nA7: Following your suggestion, we take Qwen-2.5 7B as the backbone model and apply various MoE-based multi-task adaptation methods to it. Experimental results on CSR-MTL are shown below:\n\n|Qwen-2.5 7B|CSR-MTL| \n|-|-|\n|LoRA|84.35|\n|MixLoRA|85.53|\n|MoSLD|85.58|\n|HydraLoRA|85.87|\n|MTL-LoRA|85.73|\n|LoRAMoE |86.17|\n|MoORE L=0|86.30|\n|MoORE L=2|86.37|\n|MoORE L=4|86.38|\n|MoORE L=8|86.24|\n|MoORE L=16|**86.63**|\n\nWe can find that 1) MoORE outperforms baselines consistently under different hyperparameter settings, and 2) the performance of MoORE is robust to the change of $L$. These results verify the robustness of our method and its compatibility with various model architectures\n\n**Q8: Insufficient mechanistic validation on the necessity of preserving the column space of the parameter matrix**\n\n**A8:** We have already stated in line 173 of the paper that “the work in [a] has shown that maintaining the column space of the weight matrix makes the adapted model inherit the ability of the pre-trained model better, mitigating the oblivion of the previously pre-trained task.” Our work further verifies this discovery and extends it in multi-task adaptation scenarios.  \n\n[a] Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation. NeurIPS, 2024.\n\n\n**Q9: The impacts of SVD components on efficiency**\n\n**A9:** Firstly, MoORE focuses on addressing task conflict and oblivion in multi-task adaptation. We admit that the model efficiency is important for multi-task adaptation, and we can further accelerate MoORE by various methods. However, as shown in Figure 1(d) and the analysis shown in Table 2, the current MoORE has achieved encouraging efficiency with strong performance. Further improvements in efficiency are out of the scope of this work.\n\nSecondly, as analyzed in Table 2, the number of singular vectors (equivalent to the number of experts) only partially contributes to computational complexity. We can improve efficiency by other methods, e.g., simplifying the router.\n\nFinally, since singular vectors contain pretrained knowledge, discarding any singular vectors will inevitably lead to a drop in model performance. We conducted experiments on CSR-MTL by retaining different proportions of singular values. The results are shown below.\n\n|MoORE|CSR-MTL| \n|-|-|\n|Top 25%, L=0|33.61|\n|Top 25%, L=8|35.32|\n|Top 50%, L=0|61.09|\n|Top 50%, L=8|65.28|\n|Top 75%, L=0|77.55|\n|Top 75%, L=8|78.78|\n\nHere, \"Top 25%\" means only the largest 25% of singular values are retained, reducing the number of experts to 25% of the original. The results confirm our claims --- even when 75% of the singular values and corresponding singular vectors are retained, MoORE’s performance drops significantly. Therefore, all singular values and singular vectors should be preserved.  \n\n**Q10: Inadequate table documentation**\n\n**A10:** The notations and symbols appearing in each table have been defined/denoted in the related content, before the table is mentioned in the paper. For example, we have defined $L$ in Eq. (5). If possible, please indicate which table caption is too brief, and we will make improvements in the final version. \n\n\n**Q11: Missing routing analysis**\n\n**A11:** We have already provided an analysis of the distribution of routing weights across different tasks in the CSR-MTL dataset in Appendix C.3. Based on the mean and variance of the routing weights, we divided the nine tasks in CSR-MTL into three groups and presented our analysis and conclusions.\n\n**In summary, we hope the above responses can resolve your concerns and help re-evaluate our work.**"}, {"review_id": "Jbq68lMzWG", "reviewer": "Reviewer_KfX6", "summary": "This paper proposes converting the dense pre-trained model weights into low-rank expert models and dynamically combining these expert models to learn new tasks as much as possible while reducing the forgetting of general knowledge (or old knowledge).", "questions": "In SVD decomposition, directions with larger singular values are more important. Does this mean that most samples/tasks will choose the rank-one experts associated with the larger singular values, thus exacerbating the expert routing imbalance?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- This paper proposes splitting a dense pre-trained model into a series of rank-one experts and dynamically combining these experts through a gating network.\n- The proposed method has been validated on a large number of commonsense reasoning (CSR) and natural language understanding (NLU) tasks.\n\n**Weaknesses:**\n- The meaning of adding orthogonal transformations to the input is unclear. Simply interpreting it as increasing representational capacity is insufficient, and the results in Table 4 also show that increasing the orthogonal projection of the input is not always effective.\n- In the last column of Figure 2, it should be MoORE instead of SVDMoE? According to the results in Figure 2, the proposed MoORE still shows a significant performance drop on general tasks (even though relative to other methods, the drop is smaller), such as a 6% drop on the BBH task and about a 3% drop on MMLU.\n- It is unclear how the proposed method compares with full fine-tuning (upper bound) on CSR and NLU datasets in terms of performance differences.\n- Existing works have also discussed applying orthogonal constraints to LoRA [1], which needs to be discussed.\n- Existing works have also explored splitting dense models into MoE models [2,3], and these works need to be discussed and compared.\n\n[1]Wang, Xiao, et al. \"Orthogonal subspace learning for language model continual learning.\" EMNLP  (2023).\n\n[2]Komatsuzaki, Aran, et al. \"Sparse upcycling: Training mixture-of-experts from dense checkpoints.\" ICLR (2023).\n\n[3]Wu, Haoyuan, et al. \"Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.", "was_revised": true, "final_justification": "I am not particularly familiar with the content of this work, so I tend to give positive scores and listen to the opinions of other experts.", "rebuttal": "Thanks for your appreciation of our work and constructive comments. Below, we try to resolve your concerns one by one.\n \n\n**Q1: The reason for applying orthogonal transformations**\n\n**A1:** Firstly, as stated in Line 161, we aim to maintain the orthogonality among experts during training because this can reduce redundant knowledge and help mitigate task conflict. In MoORE, each expert is constructed by the outer product of singular vectors. Updating the experts directly by stochastic gradient descent may lead to the loss of orthogonality. Therefore, we introduce $L$ learnable Householder reflections (i.e., the proposed orthogonal transformations) into the experts, making them learnable and maintaining the orthogonality throughout the training process.\n \nIn Lines 235-242, we have analyzed the results of NLU-MTL shown in Table 4. In particular, the natural language understanding tasks in NLU-MTL mainly rely on the pretraining knowledge, so adding additional learnable parameters is redundant to some degree. However, in the CSR-MTL experiment, applying the learnable orthogonal transformations helps improve model performance consistently, as shown in Table 3.\n  \n\n**Q2: The inconsistency of the method name, and the loss of performance in OR-MTL**\n\n**A2:** Firstly, thank you for pointing out the inconsistency of our method name. The last column in Figure 2 should be MoORE, and we will correct this in the revision.\n \nSecondly, the results in Figure 2 are not in conflict with our statement in Line 173. In particular, we claim that MoORE can **\"mitigate\"** task oblivion and achieve better multi-task adaptation results than existing solutions. It does not mean that our method can completely **\"eliminate\"** this issue. We admit that MoORE may still exhibit some degree of task oblivion after being trained on other tasks. However, the results in Figure 1(c), Figure 2, and Table 9 have demonstrated that MoORE has a stronger ability to resist task oblivion compared to the baselines. In summary, MoORE indeed outperforms the baselines consistently in various multi-task adaptation scenarios, showing better resistance to task conflict and oblivion.\n \n\n**Q3: The comparison with full fine-tuning.**\n\n**A3:** Firstly, we need to emphasize that in multi-task adaptation scenarios, full fine-tuning does **NOT** achieve the upper bound performance compared to the MoE-based methods (including MoORE and the MoE-based baselines).\n\nIn particular, fine-tuning the whole pre-training model on the data of $K$ tasks can outperform LoRA because it adjusts all model parameters rather than a low-rank adapter. However, full fine-tuning does not change the model architecture and does not introduce additional parameters or modules. In contrast, MoE-based methods introduce additional modules, e.g., the baselines introduce routers and external low-rank experts to the model, and our MoORE introduces a router and a learnable orthogonal transformation. As a result, the MoE-based methods often work better than full fine-tuning in multi-task adaptation scenarios.\n\nFollowing your suggestion, we compare full fine-tuning with the other methods in adapting LLaMA-3.1 8B on CSR. The results shown below verify our claim - full fine-tuning works better than LoRA but can be inferior to state-of-the-art MoE-based methods, e.g., LoRAMoE and our MoORE.\n\n|Method|CSR-MTL| \n|-|-|\n|Full Fine-tuning|84.27|\n|LoRA|79.54|\n|MixLoRA|82.55|\n|MoSLD|83.55|\n|HydraLoRA|83.84|\n|MTL-LoRA|84.09|\n|LoRAMoE|84.34|\n|MoORE L=0|84.98|\n|MoORE L=2|85.02|\n|MoORE L=4|85.02|\n|MoORE L=8|85.11|\n \n\n**Q4: Add a comparison with the LoRA method applying an orthogonal constraint, e.g., the O-LoRA in the reference [1] mentioned in the comment.**\n\n**A4:** Thank you for your suggestion. In the revised paper, we will add O-LoRA [1] to the Related Work section and discuss the similarities and differences between O-LoRA and MoORE accordingly. However, it should be noted that the O-LoRA and MoORE have the following differences:\n\n1. MoORE constructs strictly orthogonal experts based on SVD, while O-LoRA uses a regularization term to pursue orthogonality, which can not strictly make its LoRA adapters orthogonal to each other.\n\n2. O-LoRA does not adopt a MoE architecture. To deal with different tasks, O-LoRA adds new LoRA adapters in a continual learning scenario, rather than designing and learning a flexible routing mechanism for given and fixed experts. As a result, the model complexity of O-LoRA is linear to the number of tasks. From this viewpoint, the scalability of our MoORE is better.\n \n\n**Q5: Add a comparison with the references [2, 3] mentioned in the comment**\n\n**A5:** The work in [2] focuses on upcycling, i.e., initializing a sparsely activated MoE model from a dense model checkpoint to reduce the pre-training cost, rather than adapting a pre-trained dense model in multi-task learning scenarios. The scope of this work [2] does not overlap with ours, and the method cannot be applied directly to our task. \n\nIn our experiments, we find the performance of the work in [3] is similar to that of MixLoRA. Essentially, this is because this work is a simplified version of MixLoRA: \n\n1. Both of them perform MoE modifications in the FFN layer. For this method, each expert in an MoE module is a LoRA adapter, and the experts share the same FFN parameters.\n\n2. For the expert in the work [3], a LoRA with a residual connection is applied in a \"sequential\" manner. On the contrary, MixLoRA has more learnable parameters, and the width of each layer is larger than [3]. In particular, MixLoRA inserts additional LoRA adapters into attention layers, and within each expert, the FFN and LoRA are in a \"parallel\" structure. \n\nWe will add the work in [2, 3] to the Related Work section and discuss them accordingly. \n \n\n**Q6: The risk of expert routing imbalance**\n\n**A6:** The expert routing imbalance issue (or called load imbalance issue) only arises in sparsely activated MoE architectures (such as MixLoRA). In contrast, our proposed MoORE activates all experts (this design is also adopted by LoRAMoE, HydraLoRA, and MTL-LoRA), so there is no load imbalance problem.\n\n\nIn summary, we hope our responses can resolve your concerns successfully and increase your confidence in raising your rating score. It would be appreciated if you could further support our work in the following decision phase."}, {"review_id": "3YzTW8lsb5", "reviewer": "Reviewer_5xhJ", "summary": "This paper addresses the challenges of task conflict and forgetting in multi-task adaptation of large-scale foundation models by proposing MoORE. MoORE applies SVD to the pre-trained weights and introduces a learnable router to dynamically combine orthogonal rank-one experts. To ensure fair comparison, all the compared methods are built upon the same base model, and extensive experiments across multiple datasets are conducted to verify the effectiveness of MoORE.", "questions": "1. I'm a bit confused about the title — why is there a hyphen between “Conflict” and “and”, but no hyphen between “and” and “Oblivion”? That seems a bit odd.\n\n2. The manuscript claims to validate the superiority of the proposed method \"MoORE\" in mitigating task oblivion. However, Figure 2 presents results for \"SVDMoE\" rather than \"MoORE,\" leading to potential confusion about whether they are equivalent. The authors should clarify the relationship between MoORE and SVDMoE and use consistent terminology throughout the paper.\n\n3. The abstract claims MoORE demonstrates superiority in terms of efficiency. Could the authors clarify what aspects “efficiency” encompasses—is it a composite metric or specific factors? Also, please specify where in the paper this efficiency is demonstrated experimentally or analytically. Clearer definitions and supporting evidence are recommended.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The paper tackles an important problem in multi-task learning with large-scale foundation models.\n\n2.The paper conducts extensive comparisons and evaluations on diverse datasets to verify the effectiveness of the proposed method.\n\nWeaknesses：\n\n1.The proposed method in the paper does not achieve the best performance on all datasets.\n\n2.The paper claims superiority in efficiency in the abstract, but later notes that MoORE increases inference time compared to the original LLaMA-3.1 8B. This seems inconsistent.", "was_revised": true, "final_justification": "The authors have clarified my comments, so I keep my positive rating.", "rebuttal": "Thank you for your appreciation of our work. We provide the responses below to resolve your concerns one by one.\n\n**Q1: The proposed method in the paper does not achieve the best performance on all datasets.**\n\n**A1:** Instead of fine-tuning $K$ different models for $K$ tasks, multi-task adaptation aims to learn a single model to deal with $K$ tasks. **In multi-task adaptation scenarios, the average model performance over all tasks is the most critical evaluation metric, which reflects the resistance of the model to task conflict and oblivion.** In particular, given the datasets of $K$ tasks, the target model should obtain strong overall performance across all tasks, rather than performing strongly in some tasks while degrading severely in the other ones. \n\nBased on the above information, although MoORE does not achieve the best performance on all datasets, we have demonstrated its superiority in terms of overall performance and provided the following evidence in our submission.\n\n**1) Superiority on mitigating task conflict:** In Figure 1(b), the \"Overall\" columns in Tables 3 and 4, MoORE outperforms its competitors consistently in various multi-task adaptation scenarios, which means that it mitigates the conflicts across different tasks, achieving the best average performance over all the tasks.\n\n**2) Superiority on mitigating task oblivion:** In Figure 1(c), Figure 2, and Table 9 (in Appendix), MoORE mitigates the loss of model capability achieved in the pre-trained phase. In particular, when adapting LLaMA using the baseline methods, the adapted models tend to forget the capability of the original LLaMA, suffering severe performance degradation in the original tasks that LLaMA performs well. Our MoORE can mitigate this phenomenon significantly --- **in the original tasks that LLaMA performs well, the model obtained by MoORE achieves comparable overall performance.** \n\n**Q2: The definition and evidence for the efficiency of MoORE**\n\n**A2:** Thank you for your question. Due to space limitations, our explanation of \"efficiency\" in the paper was somewhat brief, and we will improve this aspect in the revision. The detailed explanation is shown below for a quick review.\n\n**In our paper, \"efficiency\" refers specifically to the inference time of MoE-based multi-task adaptation.** \n\n1) The results in Figure 1(d) show that MoORE is comparable to MixLoRA in inference time, but outperforms all other baselines. \n\n2) The results in Figure 1(b) show that MixLoRA significantly sacrifices model performance when pursuing computational efficiency, while MoORE maintains high performance with comparable complexity. \n\nThese results provide strong evidence demonstrating MoORE’s superiority in terms of efficiency compared to other MoE-based multi-task adaptation methods.\n\nNote that, because of the router in MoE architectures, all the MoE-based multi-task adaptation methods (including MoORE and its competitors) introduced additional trainable parameters that cannot be fully merged into the original parameter matrix. \nTherefore, these methods introduce additional computations, making the increase in inference time unavoidable compared to the original LLaMA model. **Therefore, the \"superiority in efficiency\" claimed in the abstract refers to MoORE’s performance compared to other MoE-based baselines rather than the original LLaMA model.**\n\n\n**Q3: The grammar correctness of the hyphen between “Conflict” and “and”** \n\n**A3:** The complete expression should be “Conflict-Resistant and Oblivion-Resistant”.  This hyphenation (“Conflict- and Oblivion-Resistant”) is a convention to avoid repeating the word “Resistant”, which is commonly used in English.\n\n\n**Q4: The inconsistency of the method name --- MoORE or SVDMoE**\n\n**A4:** Thank you for pointing out the mistake. The \"SVDMoE\" in Figure 2 should be \"MoORE\", and we will correct this in the revision.\n\nIn summary, we hope the responses above can resolve your concerns successfully and increase your confidence in raising the rating score of our submission. Feel free to contact us if you have additional comments in the following discussion phase."}]

Author rebuttal:

Meta-review: This paper proposes MoORE, a novel approach to convert pre-trained dense models into orthogonal rank-one “experts” combined through a learnable router for multi-task adaptation. Through SVD on the weight matrix of a pre-trained model, the proposed approach maintains the experts’ orthogonality and the column space of the original weight matrix which provides a simple yet effective solution to address task conflict and oblivion challenges in multi-task adaptation. At the same time, the method is computationally efficient with minimum learnable parameters. The authors provide a clear theoretical motivation and validate the approach using diverse commonsense reasoning and natural language understanding benchmarks.

While one reviewer raised concerns about novelty and framing, the rebuttal offers detailed comparisons to representative SVD-based competitors, and discussion of related MoE-style methods, clarifying MoORE’s unique positioning. Experimental results validate the strong performance compared to baselines, supporting the method’s practical value. Upon carefully reading the reviews and authors' responses to these concerns, it appears the concerns are more of different views of the approach, and I believe the comparison with competitive baselines are clear in terms of both methodology and empirical results. I'd suggest adding these discussion to the final paper. 

Overall, given its simplicity, theoretical grounding, and empirical effectiveness, I believe MoORE will be a valuable contribution to the efficient adaptation of large language models. I'd suggest accepting the paper.
Decision: Accept (poster)