LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
Authors: Wenbing Li
Zikai Song
Hang Zhou
Junqing Yu
Yunyao Zhang
Wei Yang
Venue: ICLR 2026 Poster (2026)
Keywords: MoE Optimization
LLM
Low resource adaptation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent attempts to combine low-rank adaptation (LoRA) with mixture-of-experts (MoE) for multi-task adaptation of Large Language Models (LLMs) often replace whole attention/FFN layers with switch experts or append parallel expert branches, undermining parameter efficiency and limiting task specialization.  We introduce **LoRA-Mixer**, a modular MoE framework that routes task-specific LoRA experts into the core projection matrices of the attention module (input/output linear layers), rather than primarily targeting FFN blocks. The design delivers fine-grained token-level specialization by fully exploiting the attention mechanism, while remaining drop-in compatible with Transformers and state-space models (SSMs) as the linear projection layers are ubiquitous. To train robust routers from limited data while promoting stable, selective decisions and high expert reuse, **LoRA-Mixer** employs an adaptive **Routing Specialization Loss (RSL)** that jointly enforces global load balance and input-aware specialization via an entropy-shaping objective. The framework supports two regimes: (i) joint optimization of adapters and router with a differentiable hard–soft top-k routing scheme, and (ii) plug-and-play routing over frozen, pre-trained LoRA modules sourced from public repositories.  Across 15 benchmarks—including MedQA, GSM8K, HumanEval, and GLUE—RSL-optimized LoRA-Mixer outperforms state-of-the-art routing and LoRA-MoE baselines while using 48% of their trainable parameters, with gains of +3.79%, +2.90%, and +3.95% on GSM8K, CoLA, and ARC-C, respectively. Cross-model transfer and adapter reuse experiments further demonstrate the approach’s versatility and data efficiency.

Published as a conference paper at ICLR 2026
LORA-MIXER: COORDINATE MODULAR LORA Ex- PERTS THROUGH SERIAL ATTENTION ROUTING
Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, Wei Yang Huazhong University of Science and Technology (d202581 843) @husthhueeeee .du.cn
ABSTRACT
Recent attempts to combine low-rank adaptation (LoRA) with mixture-of-expert (MoE) for multi-task adaptation of Large Language Models (LLMs) often place whole attention/FFN layers with switch experts or append parallel expert branches, undermining parameter efficiency and limiting task specialization. We introduce LoRA-Mixer, modular MoE framework that routes task-specific LoRA experts into the core projection matrices of the attention module (input/output linear layers), rather than primarily targeting FFN blocks. The design delivers fine-gres aineggrinedd token-level specialization by fully exploiting the attention mechanism, while remaining drop-in compatible with Transformers and state-space models (SSMs) as the linear projection layers are ubiquitous. To train robust routers from limited data while promoting stable, selective decisions and high expert reuse, LoRA-Mixer employs an adaptive Routing Specialization Loss (RSL) that jointly enforces global load balance and input-aware specialization via entropy- shaping objective. The framework supports two regimes: (i) joint optimization of adapters and router with a differentiable hard-soft top-k routing scheme, and (ii) plug-and and play routing over frozen, pre-trained LoRA modules sourced from public repositories. Across 15 cachamaarkk-icluddii MedQA, GSM8K, Hu- manEval, and routing and LoRA-MoE basclines while using 48% of their trainable parameters, with gains of +3.79%, +2.90%, and +3.95% on GSM8K, CoLA, and ARC-C, respectively. Cross-model transfer and adapter reuse experiments further demon- strate the approach's ' versatility and data efficiency, Our code can be obtained at https//ggghubb.oom/mssttewww//
LoRA-Mixes outperforms state-of-the-art
INTRODUCTION.
Large Language Models (LLMs) have achieved unprecedented proficiency in general-purpose reason- ing and generation, yet their adaptation to specialized downstream domains remains computationally
prohibitive for full-scale fine-tuning Brown al. (2020); Touvron et al. (2023). To mitigate these
resource demands, parameter-efficies fine-tuning (PEFT) methods Li & Liang 2021); Zaken et al.
(2021): Liu al. (2022); Houlsby al. (2019; Liu
(202) have emerged
scalable
paradigm. Among them, Low-Rank Adaptation (LoRA) Hu et al. (2022); Tian et al. (2024) has demonstrated particular efficacy, operating through low-rank decomposition of updates the pre-
trained weights weights-cnabling efficient tuning with minimal parameter overhead. Recent work has explored modularly composing independently trained LoRA modules promising strategy for multitask adaptation; however, naive composition can result in interference between task task-specific subspaces, limiting their synergistic potential HHuag et al. (2023); WW et al. (2024). This limitation has motivated exploration of mixture-of-exxper (MoE) architecture (Shazeer al. (2017); Fedus) ct al. (2022), which treat each task-specific LoRA as an expert and sparsely activate and fuse the experts. Recent studies demonstrate promising directions in hybrid LoRA-MoE frameworks Wu et al.)
(2024); Li et al. (2024)); Huang et al. (2023): Zhao et al.] (2024); Liao et al. (2025); : Gao et al. (2024); Chen et (2024) Dou al. (2023), aiming to enhance model performance complex tasks across multi-domain datasets while preserving the parameter efficiency of fine-tuning.
ddnotee the corresponding author.

---

Published as a conference paper at ICLR 2026
Vanilla
Fuse with Lora-MoF
LoRA-Mixer
Attention
Apply LORA-Miser
FEN
FEX
FEN
FFN
FFNO
Top-
Cancal
Bonter
Linear
Soated Dos-Prodacy
Attention
Atin O
Jm
Top-K
Router
Figure 1: LoRA and MoE integration methods. (I) and (2): replace the attention or FNNs with switch experts; (3) and (4): introduce LoRA experts branches in parallel with the attention feedforward layers, and fuse the output into the main branch. Our LoRA-Mixer (right) applies mixture of LoRA experts to the projection layers, which effectively leverages the attention mechanism.
The central challenge in composing pre-trained LoRAs is to realize synergy-improving performance across constituent Existing LoRA-MoE integrations largely follow two patterns: (i) replacing attention or FFN blocks
inflating training cost erasing task-specific inductive biases.
with LoRA-based switch experts in standard MoE setup Li et al. (2024a); Chen et al. (2024) Fedus
et al. (022);; Shazeer et al. (2017: or (ii) attaching parallel LoRA A branches whose outputs are fused
(2024): Huang al. (2023) (Fig. [ Despite encouraging results Wu et al. (2024): Liao et al. ): Krajewski et al. (2024): Zhang et al. (2019); Li et al. (2022);
back into the main path Wu
Gross et al. (2017). 7), switch-style designs typically require joint training of all experts, increasing data demands and hindering modular reuse and transfer of off-the-shelf LoRAs. Parallel-branch schemes sidestep native attention or state- tratstrranitionn pathways, yielding shallow output fusion and weak integration. Moreover, common auxiliary routing losses emphasize uniform load balancing, suppressing input- and task-aware specialization Li et al. (2024a). These issues motivate a plug-and- play, architecture-agnoste alternative compatible with both Transformers and state-space models (SSMs) Gu & Dao (2023) that learns discriminative routing with minimal compute and data while maximizing reuse of independently trained LoRA modules.
In response, we introduce LoRA-Mixer, a novel framework designed to efficiently synergize multiple pre-trained LoRA modules by treating them dynamic, pluggable memory cells. L-RA-Mixer equips the linear projection layers of the original model with mixed LoRA experts, enabling these experts to directly leverage the core attention or ttaeetransition mechanisms. To further enhance efficiency and maintain routing effectiveness, LoRA-Mixer adopts novel Router Specialization Balancing Loss (RSL) to align routing decisions with token-level expert usage, maintaining moderate entropy to encourage exploratory behavior. During inference, we employ sparse top-K fusion, effectively balancing computational cost and scalability without compromising expert selectivity, LoRA-Mixer supports LoRA modules sourced from external repositories. independently trained, or jointly trained through hard routing strategies, allowing seamless plug-and-pl usage across various tasks and domains. Importantly, method significantly reduces the necessity for training data or extensive re-adaptation, requiring only minimal additional data to effectively train the routing mechanism. Consequently, LoRA-Mixer is particularly suitable for constructing large-scale, modular language models characterized by task-specific memory, computational efficiency, and strong transferability. Extensive evaluation 15 benchmark datasets (MedQA, GLUE, GSMSK, ARC-E, E, ARC-C, HumanEval, PIQA, HellaSwag, and BoolQ) demonstrates that integrating LoRA- Mixer significantly improves model performance across all evaluated tasks. With only 48% of the parameters of existing methods, the RSL-optimized LoR A A-Mixer outperforms the state-of-tth-arr routing and LoRA-MoF baseline models, achieving improvements of +3.79%, +2.90% and +3.95% on GSM8K, CoLA, and ARC-C, respectively. Cross-model transfer and adaptive module reuse experiments further demonstrate the method's versatility and data efficiency.
2 RELATED WORK.
PEFT For LLMs Low-rank adaptation (LoRA) Hayou et al. (2024); Zhou et al. (2024): | Hu et al.
2022 Xuet al. 2204 (Sheng et al. (2023) Biderman al. (2024 al. 2223 Zeng et all (2225
2

---

Published as a conference paper at ICLR 2026
LLMs
LoRA experts SoUrCE
Trunsformers
SSMS
Liucar
our
Ainnar
SSAY
Routee
Selective
CRL
Fralning Sap
Nesteal Bust-
Hybrid Hard Rowning Training
Producr Attention
Cpee
Bi
B. (N(p(=))
Livaris
Sagee
Stage
K
Hard
Tank dowias
C00000
Appll LoRd--Miarr
Roster
Seft Ronter
The overall architecture of LoRA-Mixer. LoRA-Mixer applied to the linear projection layers in serial with the Attention and SSM modules and support all major LLM structures. LoRA- Mixer reueses the LoRA experts sourced from Internet, trained individually or jointly trained using hard routing. The routing training is guided by RSL loss for balancing experts loads and specificity.
Figure 2:
Lin ct al. (2025a b) effectively fine-tunes large models by learning a low-rank matrix and freezing
the original weights. While effective for single task, its task-specific nature limits generalization.
Recent work combines LoRA with mixture of experts (MoE) Li et al. (2024a) Feng et al. (2025a) Wu
al. (002)) Ouyang al. (2025); Feng et al. (20256); Huang et al, (2023): Zhao et al. (2024): Song
et al. 2022 ; Hu et al. (2025) to achieve dynamic adaptation. For example, MixLoRA Li et al.
(2024a) uses LoRA experts for top-k routing in FFN, improving multi-task performance but suffering
from gradient entanglement issues. MoLE Wu et al. (2024) combines LoRA layers via gating but
lacks sparse routing. LoraHub Huang et al. (2023) performs gradient-free few-shot combination of LoRA modules for unseen tasks but struggles with complex semantics due to lack of gradient optimization and dynamic routing. Other methods Li et al. (2025) ; Huang et al. (2025) ); Yang et al.) (2024); Chen al. (2023). Luo al. (2024); Li et al. (2024b); FFng et al. (2025c) explore flexible
: routing mechanisms to improve the model's 's adaptability. However, these methods usually introduce additional routing networks optimization targets, resulting in instability during training, limiting
their application in actual multi-task or low-resource scenarios.
Mixture of Experts In recent years, the mixture of experts (MoE) architecture has attracted much attention as promising LLM expansion paradigm. By selectively activating subset of expert modules for each input, MoE allows the model to scale capacity without linearly increasing the amount of computation. As a result, more and more large models have adopted MoE, including GLaMDu al. (2022), Switch Transformer&Fedus et al. (2022): Song et al. (2025): Zhang al.] (2025a by: Ye al. (2025), and the recent DeepSeek serieş al. (024)) Guo al. (2025a). These advances indicate that MoE becoming a mainstream architectural trend the development of next-generation base models. Among them, GLaMDu al. (2022) and Switch TransformerFedus cet al. (2022) build a mixture experts (MoE) model in the FFN module, and use a sparse activation mixed expert architecture to expand the model capacity and achieve better performance. LLaVA- Mol-CChen et al.] (2024) uses a top-1 strategy to route tokens to domain-specific expert models, the
thereby alleviating data conflicts and achieving continuous performance improvements ordinary LoRA baseline. LoR et al. (2023) uses routers to integrate LoRA experts while retaining general knowledge. HMORALiaa al. (2025) combines the layered fine-tuning methods of MoE and LoRA, and gradually switches the routing strategy as the number of layers increases.
MoLA'Gao al. (2024) assigns different numbers of experts different levels, proving that deeper layers often require more experts. Notably, we only utilize hybrid LoRA experts in the core projection layer, resulting in extremely high parameter efficiency. By applying RSI. to LoRA-Mixer, we can efficiently reuse LoRA modules and require minimal training data, saving computational resources while expanding model capacity and improving generalization.
METHOD.
In this section, we introduce LoRA-Mixer, a flexible and pluggable Mixture of Experts (MoE) framework for combining LoRA experts from multiple L.LMs.
3

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARIES
vixtureeoffExpert is a sparse neural architecture where each input token is processed by a small subset of expert networks. Given K experts and router that produces score vector G(x) IRK softmax is applied to obtain the routing distribution:
(xp(G.(())
(x Σ exp(G,
i= i K.
(1)
The top-k experts are selected based on (r)),, and the final output is computed as a weighted sum over their outputs:
K
MoB(x) = Σ Ie TopK(p(x))] px()) Expert;
This design allows MoE to reduce computational cost while enabling experts to specialize different input patterns.
Auxiliary loss. To encourage balanced expert utilization, MoE training introduces auxiliary loss. We define the expected routing probability and top-1 usage as
pj = BE-DDpp((())
==
T(i arg max (z)),
Lamxx
Σ P.
where p; denotes the average routing intention and f. the empirical usage. We demonstrate in Appendix A.17 why the auxiliary loss leads to over-averaging.
3.2 LORA-MIXER FOR COMPOSITING LoRAS
Combining independently trained LoRA modules for multi-task adaptation provides promising approach to provide LLMs with cross-domain composition capabilities. For example, we can fuse mathematics- and pedicine-specifi LoRA to enable LLMs have both stronger mathematical reasoning capabilities and medical-specific knowledge to solve complex cross-domain queries.
Our proposed LoRA-Mixer implements this mechanism by treating each pre-trained LoRA module as an expert and learning a routing function Frome that dynamically fuses these experts based on the input semanties. The routing mechanism is lightweight and data-efficient, and can achieve task awareness with only a small amount of additional training. LoRA-Mixer uses a set of E' low-rank experts and a router (xx) E R to enhance the pre-trained projection matrix W E Rd..x Xd,, Each expert is parameterized as A (e(e) ((r))(()) where A(r) Rdduxr and ((e) grxd, The output of LoRA-Mixer is:
y = Wx x + F route (fa.(x) (fa.(x)-AA()))
E
where, FFrone (-) represents the routing function output by the fusion expert. The output will be passed to the subsequent attention module or state-space module, enabling directly influence the representation learning path. This strategy ensures that LoRA-Mixer acts at the most expressive point of the model the projection layer- without disrupting the underlying architecture.
LoRA Experts Acquirement. Our proposed LoRA-Mixer framework is highly flexible and supports the integration of LoRA modules from diverse sources. In one common scenario, users may download pre-trained LoRA adapters from public repositories such as LoRAHub Huang et al. (002)). which currently hosts 196 high-quality LoRA modules across wide range of domains. These can be directly composed using LoRA-Mixer with minimal additional data. Alternatively, users may independently train domain-specific LoRA modules tailored to their own datasets. For scenarios requiring joint training of multiple LoRA modules on a heterogeneous, labeled dataset, LoRA-Mixer further supports a hard-routing strategy. Specifically, we fix the routing module and apply a deterministic routing scheme based on known domain labels. Given a domain ID d (I, K} associated with each training instance, all tokens within the sample are routed exclusively to expert d. This design enables efficient joint optimization while maintaining expert modularity. Collectively, these capabilities make LoRA-Mixer a versatile and scalable framework for composing heterogeneous LoRA modules. The overall architecture is illustrated in Figure?
4

---

Published as a conference paper at ICLR 2026
3.3 SPECIALIZATION BALANCE LOSS FOR ROUTING OPTIMIZATION
The next step is to optimize the expert routers. While previous research has introduced auxiliary loss functions align the average gating score with expert utilization, thereby promoting load balancing. we observed that this approach overemphasizes consistency, resulting in an overly balanced expert distribution. In this case, all experts are forced be used equally, regardless of the input semantics. This hinders effective routing and generally requires more training data.
We propose a new perspective: viewing the router as an information bottleneck that determines the degree to which semantic differences between tokens are preserved or compressed. From this perspective, the entropy of the routing distribution quantifies the uncertainty of the router. Therefore, load balancing and specialization selection appear to be in conflict. To achieve balance between the two, ensuring balanced expert load and input-aware routing, we propose an improved optimization objective, called Routing Specialization Balance Loss (RSL). RSL addresses this issue by unifying the global budget and local selectivity into a single objective. Specifically, RSL enhances the auxiliary loss with entropy regularization term, improving the accuracy of expert selection by suppressing overly flat distributions. Formally, let Pt denote the average soft routing score (across tokens) and fü denote the normalized score assigned to the token of expert i in the first & routes. The RSL loss function is defined as:
K
CRSL- = =a-
Pp.ht EEAD ( [H((()))},
(5)
1=1
Where controls the strength of the equilibrium consistency term, is a small positive coefficient of the entropy regularization term:
H(p(x)) = Σ log
(6)
The design principles behind RSL are: 1) From an information bottleneck perspective, minimizing H(((x))) reduces condiiccnniii uncertainty under a fixed global load, directly promoting special- ization without disrupting the balance. 2) The entropy term acts as curvature provider in the routing simplex, resulting strong convexity and well-conditioned optimization (see|A.I) ; this is precisely where traditional auxiliary loss functions are weak. 3) The coefficient À becomes interpretable knob that trades off global fairness and local specialization, rather than simply a regularization weight.
Specifically, the entropy term introduces token-specific entropy. We derive the entropy gradient with respect to the routing score pz()):
0H(((z)) logp;(z) subject @p((z)
K
Σ
(1(x)
-loogppp) 1+ μ,
(8)
where j is the Lagrange multiplier due to the simplex constraint. Therefore, the total gradient of the RSL loss becomes:
Op. Op((x) f. + A(log(((I)+ + 1- 1)).
p.(x) CRSt a
This demonstrates that RSL introduces token-level gradient signal via logp.(x)., unlike the auxiliary loss, which only propagates global gradients. Specifically, the log P:(x) term provides token-level signal that amplifies the input-awan aware variance in the expert's choices, thereby counteracting the bias towards consistency in global balance. To quantify input- routing, we define:
Varg~D(p(x)) Ex [l]((x)) -r13]]
(10)
We call routing token-aware if Var(p(z)) > and >>> 0. The auxiliary loss function tends to reduce this variance (driving uniform routing), while RSL encourages high variance and peaked distributions that are consistent with the input semantics. In addition to the above proof', we provide convergence analysis and generalization bound proofs in the Appendix A. A.2 The former introduces strong convexity on the product simplex and proves that the added entropy term can maintain stable optimization dynamics; the latter shows that entropy regularization reduces the assumed complexity of the router and improves robustness in low-data environments.
5

---

Published as a conference paper at ICLR 2026
Routing Optimization. After LoRA is ready, we soft-train the router. To prevent expert knowledge from being contaminated, we introduce a regularization term to penalize deviations from the previ- ously learned expert parameters. Let the first-stage parameters of expert be 0!!) and the current parameters be Og. We define the regularization term as:
Epeeserve = 3 > 0 0 = B. Σ |A0,11
(11)
RCC
EC
where C represents the set of constrained experts and | controls the regularization strength. This regularization term constrains sensitive experts to maintain their original knowledge while allowing other experts to flexibly adjust, supporting multi-expert learning for complex tasks.
To ensure effective gradients and stable optimization during joint training, we employ soft expert fusion: the routers output softmax score Рo.: E R.K, achieving differentiable LoRA hybridization. While this approach provides stable gradients, combining it with an auxiliary loss results in equal activations among experts, inhibiting specialization. To address this, we introduce RSL, which promotes expert differentiation while maintaining load balancing and enables efficient reuse LoRA with minimal data. The total loss during joint training is:
Σ 6, 1001 REC
Lotal
Lisk + LRSL + B.
(12)
where Lask denotes the standard task loss (e.g., enoss-yntropy loss). LRSL is the routing specializa- tion balance loss, which is used to promote consistency between routing scores and actual token assignments. a is weighting factor for RSL. Lereserve the expert regularization loss described above, scaled by B. We provide an exploration of how to choose hyperparameters Appendix A.8
EXPERIMENT.
In this section, conduct extensive experiments on 15 datasets across five domains (MedQA, commonsense reasoning, NI.,, mathematics, and coding).
4.1 EXPERIMENTAL SETUP
Datasets We tested RSL and LoRA-Mixer 15 public benchmarks: MedicalQA, ARC-E, ARC-C, GSM8K, GLUE, PIQA, BoolQ. HumanEval, and HellaSwag. These datasets cover five different domains. For detailed dataset information, please refer Appendix [1
Baseline We selected LLaMA3-8B, Mistral-7B, and Falcon-Mamba- as the basemodel. LLaMA3- 8B and Mistral-7B We compare LoRA-Mixer and RSL with other state-of the art methods, including MoLE Wu et al.
Transformer architectures, while Falcon-Mambo is pure SSM architecture.
(2024), MixLoRA Li et al. 2022a)), LoraHub Huang al. 2023), LoRA-LEGO Zhao et (2024) and PHATGOOSE Mugeee et al. (2024). We also select strong baselines that specifiicllly optimize auxiliary losses demonstrate the effectiveness of RSL, including GMoE Bai et al. (2024). AESL
ct al. (205b). and DsMoE/Pan et al. (2024). It is worth noting that MoLE and MixLoRA also optimize auxiliary losses. For parameter, training and inference analysis, please refer to A.7
For HumanEval, we use the Pass@1 metric. Considering the domain-specine freedom and rigor required by the Medical-QA dataset, we use DeepSeek-PP R1 for evaluation. For the remaining tasks, we use ACC as the metric. To reduce randomness, all experiments are run three times and the average reported.
Metrics
4.2 COMPARISONS
Table I: Performance of three base models, Falcon-Mamba-71,, Mistral-7B, and LLaMA3-8B, on seven benchmarks.
Base Model
Medical
CoLA
SST2
GSM8K
ARC-E
ARC-C
HumanEval
Falcon-Mamba-77
73.67
82.42
92.81
52.54
77.61
68.78
29.29
Mistral-7B
66.32
71.21
85.24
40.83
80.00
61.50
27.95
LLaMA3-8B
78.47
79.14
93.12
57.92
88.45
78.65
52.44
Table shows the performance indicators of the three basemodels on various tasks. We compare our approach with the oate--of-the-are methods seven datasets. The experimental results are shown in

---

Published as a conference paper at ICLR 2026
Table 2
Our approach outperforms the baselines most tasks. For the Falcon-Mann Mamba dataset, approach significantly outperforms the baselines on all tasks. For model details, see Appendix [A.11
We the LLaMA2-7B from the LoRA-LEGO paper the basemodel and conduct experiments the four tasks: CoLA, SST2, MRPC, and RTE. The LoRA configuration uses r = 6 and a 12. The experimental results shown in Table 4 three of the four tasks.
As can be seen, method outperforms LoRA-LEGO
Table 2: Comparison of our LoRA-Mixer with LoRAHub, MoLE, and Mix-LoRA across tasks (best scores bold). Note that MixLoRA is excluded from Ralcon-Mamba due to its Transformer- specific design.
Metho
Medical
CoLA
SST2
GSM8K
ARC-E
ARC-C
VvaanEval
(7B)
LoRAHub
70.14
81.11
93.35
51.64
81.16
72.37
30.68
MOLE
74.51
84.77
94.22
54.28
83.46
76.61
33.57
LoRA
77.26
85.62
95.07
56.27
85.68
76.51
33.54
LoRA-Mixer (ours)
78.01
85.91
95.76
57.87
86.87
77.19
35.37
Mistral (7B)
LoRAHHu
69.17
75.73
90.21
44.94
81.14
69.21
32,60
MOLE
71.07
78.51
94.17
45.31
85.68
68.77
35.37
MixLoRA
69.74
78.61
93.44
45.50
85.42
69.15
33.80
LoRA
70.33
79.19
93.58
46.67
86.66
70.53
35,31
LoRA-Mixer (ours)
71.25
82.17
95.16
46.48
87.87
71.22
36.76
LLMMA-3 (8B)
LoRAHub
78.11
79.84
92.77
59.10
87.13
80.14
52.83
MOLE
78.43
81.37
94.18
63.81
88.15
81.77
55.87
MixLoRA
79.87
80.67
94.22
64.44
88.70
82.90
55.49
LoRA
81.09
81.50
95.30
65.14
89.59
82.15
55.61
LoRA-Mixer(ours)
81.55
82.22
95.41
65.53
89.88
83.24
57.32
Table 3: Evaluation of LoRA-Mixer on LoRAs
Table 4: Comparison of LoRA-Mixer and LoRA-
sourced from Internet on five GLUE tasks, the base model is Flan-TSChung et al. (2024).
LEGO. Results for LORA-LEGO are from its pa- per.
Method
SST-2 CoLA MRPC RTE QQP
Method
CoLA SST-2
MRPC
RTE
Flan-T5
94.01
74.21
79.90
80.08
82.32
LoRA
61.63
75.74
68.00
52.22
LoRA
94.50
80.54
83.76
83.47
85,55
LEGO
55,48
73.22
66.00
71.85
LoRA-Mixer
95.07
82.14
85.15
85.31
84.75
LoRA-Mixer
64.60
80.31
72.24
61.47
Considering that Mistral-7B and LLaMA3-8B have the same architecture, we directly migrate the parameters trained on Mistral-7B to LLaMA3-8B without any fine-tuning and adaptation, and conduct experiments on three datasets: ARC-E, ARC-C, and GSM8K. The results are shown in Table It is worth noting that we use the Zero-Sho CoT method to test the basemodel in the GSM8K task, and the results under different Few-Shot settings are also shown in Table 5
Table 5: Evalution LoRA-Mixer parameter transferability from Mistral-7B to LLaMA3-8B. Values show absolute performance (relative to baseline in parentheses).
Method
GSM8K
ARC-E
ARC-C
0-shot
2-shot
5-shot
0-shot
0-shot
IJLMM3-8B
57.92 (1.00)
75.88 (1.00)
78.64 (1.00)
88.45 (1,00)
78.65 (1.00)
+ Mistral
59.13 (1.02)
76.26 (1.01)
81.43 (1.04)
85.89 (0.97)
79.14 (1.01)
7

---

Published as a conference paper at ICLR 2026
Table 6: OOD comparison across datasets for Table Results on BoolQ, HellaSwag, and PIQA. Base is LLaMA3-8B.
Phatgoose, and LoRA-Mixer.
Dataset
Base
Phatgoose
LoRA-Mixer
Dataset
Base
LoRA
LoRA-Mixer
Gap
QQP
68.84
69.28
69.47
BoolQ
71.25
74.46
79.37
+4,91
RTE
70.88
69.87
71.31
HcllaSwag
75.33
77.39
82,41
+5.02
MRPC
74.73
74.59
74.93
PIQA
78.47
80.71
84.94
+4.23
Interestingly, we observe that we outperform the LLaMA3-8B on two the three tasks. This cross- model transfer validates the design motivation of LoRA-Mixer and demonstrates that the routing learned via RSL extremely robust and transferable, making possible share experts between models with the same architecture.
To further demonstrate the performance LoRA-Mixer, we conducted experiments on BoolQ. HellaSwag, and PIQA using LLaMA3-8B, with r 32 for LoRA. The results shown in Table 7 can be seen that LoRA-Mixer can effectively coordinate the capabilities of LoRAs to enhance the basemodel. To test the generalization of LoRA-Mixer, conducted experiments with PHATGOOSE on 9 datasets, six which were in-distribution experiments and three were out-of -iito----ittiuttion experiments. Here only show the OOD experimental results because OOD tasks reflect the generalization ability of LoRA-Mixer. For the results of the in-distribution experiment, please refer to A..4. The results show that LoRA-Mixer has excellent generalization ability, which means that through RSL, routing can make professional decisions and achieve better performance.
4.3 TESTING ON LoRAS SOURCED FROM INTERNET
To verify that Loxe-Mixer can reuse LoRAs with minimal effort, we tested it LoRAs from the internet. We downloaded five different LoRAs from LoRAHub Huang et al. (2023) and trained them on SST2, CoLA, MRPC, RTE, and QQP (see Appendix A. for details). We used the Flan-Ty T5 model as the base model, froze the LoRAs parameters, and collected an additional 2k mixed data points for routing training, which independent of the LoRAs training data. The results are shown in Table 3 LoRA-Mixer achieved excellent performance on all four tasks, demonstrating its potential for production- grade multi-task applications.
4.4 COMPARISON WITH OTHER OPTIMIZED ROUTING LOSSES.
Although we have compared RSL with other optimized routing losses (MoLE, MixLoRA), to further demonstrate the effectiveness of RSL, we choose three strong baselines for comparison: GMoE, Ds-MoE, and AESL, all of which specifically optimized for routing losses. It is worth noting that all experiments are conducted with the same training data (2k), and the only difference the routing loss. As shown in Table & RSL significantly outperforms other strong baselines in low-data-resource scenarios.
Table 8: Comparison with GMoE, DS-MoF, and AESL under the same training data (2k)
Average load of different experts on 1K mixed data.
Export
Export
Export
and LoRAs parameters. Demonstrates the ad-
vantages of RSL under low data resources.
Percentage (%)
Task
GMoE
DS-MoE
AESL
RSL.
10
SST-2
91.38
92.45
92.64
95.41
CoLA
79.57
79,83
80.42
82.22
ARC-E
85.65
85.32
86.24
89,88
ARC-C
76.42
78.45
79.88
83.24
HumanEval
46.37
48.92
50.46
57.32
Expert O
Export
Expert 2
Export 3
Export 4
Export
Figure 3: Expert Assignment Overview.
4.5
ABLATION STUDY
Impact of LoRA Rank. To evaluate the impact of in LoRAs, we conducted experiments on r = 16. = 32, 64, and r 128, while keeping all other hyperparameters (c.g., dropout rate, learning rate) unchanged. The results of r = 64 can be found in Table 2 We place the remaining results in Appendix A.12
8

---

Published as a conference paper at ICLR 2026
Expert Load Analysis. To analyze the overall load, uniformly sampled 1K data from seven benchmarks (MedicalQA, CoLA, SST2, GSM8K, ARC-E, ARC-C, and Humaneval). We report the average load per expert on this 1K data, as shown in Figure The activation rates of different experts are relatively balanced, ranging from 15% to 18%. However, across tasks, expert load reflects "perception" ability, with experts certain tasks having higher loads than others, as shown in Figure 4 This demonstrates that our routing mechanism effectively avoids expert collapse and achieves balanced expert utilization across different tasks.
The impact of K and Cross-domain QA. We also analyzed the impact of K in Top-K. In general, increasing K within an appropriate range can improve model performance. For details, please refer to A.3 Furthermore, to evaluate LoRA-Mixer's generalization capabilitiees on cross-domain tasks, we created two cross-domain datasets: Mathematics-Coding and
For detailed experimental results, please refer to A.3 These results demonstrate that LoRA-Mixer maintains strong generalization and input-awareness capabilities on cross-domain datasets.
The impact of the RSL. We investigated the impact of our proposed RSL on the LoRA-Mixer. RSL offers three key advantages. First, it enables routing to achieve global load balancing while maintaining robust input awareness. Second, compared to auxiliary losses, RSL requires less training data. Finally, routing trained with RSL is completely independent of the original LoRAs training data.
To verify the first conclusion, we conducted experiments Medical, GSM8K, and HumanEval. As shown in Figure 4. when using RSL, the router consistently assigns higher activation weights to relevant experts, demonstrating its strong domain awareness and adaptive specialization capabilities. In contrast, when using only the auxiliary loss, the router disregards the semantics of the input and evenly distributes experts, resulting in suboptimal performance.
Table 9: Average performance across seven tasks under different routing training data sizes, with or without RSL. With RSL, LoRA-Mixer re- quires much less data while showing better per-
Expert Activation Comparison
formances.
Activation (%)
Data
w/ RSL
w/o RSL
Gap
76.80
75.47
+1.33
2K
79.26
77.29
+1.97
4K
78.77
79.14
-0.37
6K
79.41
79.37
-0.04
Medical
GSMBK
HumanEval
8K
79.75
79.48
+0.27
10K
79.94
79.51
+0.43
Figure 4: Expert Load Distribution across Tasks. To support the second conclusion, we analyze the impact of training data size on routing performance. Specifically, we construct training sets of different sizes by sampling from a multi-task dataset pool and evaluate the performance on seven benchmark tasks. For clarity, we report the average performance over all tasks.
As shown in Table RSL achieves comparable or even superior performance using only 51.62% of the training data with auxiliary loss. We explain the suboptimal RSL results at 4k in A.16 This exceptionally high data efficiency is demonstrated by generalization bound estimation. See the Appendix A.2 for detailed proof.
CONCLUSION.
This paper introduces RSL, designed to address the issue of excessively uniform auxiliary loss. achieves highly selective routing while achieving overall load balancing, and enables efficient routing training with minimal data, providing strong support for the reuse of LoRAs modules. To verify the effectiveness RSL,, we propose the LoRA-Mixer framework and integrate RSL. LoRA-Mixer flexible and architecture-agnosti MoE framework for combining LoRAs and adapting L.MSs to multitasking. Unlike other methods that indiscriminately insert MoEs or completely replace attention or FFN modules, LoRA-Mixer only adapts the core projection layer, improving the performance of Transformers and SSM models. Although LoRA-Mixer is effective, its fixed top- K routing may limit adaptability to ambiguous inputs. Uniformly applying it across all layers redundancy, as different layers capture different information. Future work will explore dynamic or differentiable routing and adaptive integration apply LoRA-Mixes only where most beneficial.
also introduce

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This research was supported by the National Natural Science Foundation of China (Nos. 62272184 and 62402189), the China Postdoctoral Science Foundation (Nos. 2024M751012, 2025T180429, and GZC20230894), and the Hubei Provincial Postdoctoral Research Funding Program (No. 2024HBB- HCXB014). The computational work was performed on the high-performance computing platform at Huazhong University of Science and Technology.
REFERENCES
Ting Bai, Yue Yu, Le Huang, Zenan Xu, Zhe Zhao, and Chuan Shi. Gmoe: Empowering 1lms fine-tuning via moe graph collaboration. arXiv preprint arXiv:2412.12216, 2024.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint urXiv:2405.09673. 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020 2020.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint arXiv:2401.16160, 2024.
Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang. Yu Qiao, and Jing Shao. Octavius: Mitigating task interference in milms via lora-moc. arXiv preprint 211 1.02684, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fieetueed language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint 112v:2312.09979 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with ofxture-of-expers pp. 5547-5569, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Leurning Reseurch, 23(120):1-39 2022.
Sicheng Feng, Gongfan Fang, Xinyin Ma. and Xinchao Wang. Efficient reasoning models: A survey, arXiv preprint arXiv:2504. 10903, 2025a.
Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. Rewardmap: Tackling sparse rewards in finennrained visual reasoning via multi-stage reinforcement learning. arXiv preprint arXiv:2510.02240 2025b.
Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang. and Xinchao Wang. Can milms guide me home? benchmark study on fine-grained visual reasoning from transit maps. arXiv preprint arXiv:2505.1800 2025c.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint St0.:::242208866666 2024.
Sam Gross, Marc' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6865-6873, 2017.
10

---

Published as a conference paper at ICLR 2026
Albert Gu and Tri Dao. Mamba: Sthear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini vrXiv:2501.12945 2025a.
Hongean Guo, Haolang Lu, Guoshun Nan, Bolun Chu, Jialin Zhuang, Yuan Yang, Wenhao Che, Sicong Leng, Qimei Cui, and Xudong Jiang, Advancing expert specialization for better moe. arXiv preprint arXiv:2505.2233 2025b.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint 20ti:::20222235544 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019.
URL BtRtss:s:///xxi org/ abs /ass/1992. 00751
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Aluen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Yangliu Hu, Zikai Song, Na Feng, Yawei Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Sf2t: Self-supervised fragment finetuning of video-Ilms for fine-grained understanding. arXiv preprint arXiv:2504.007443 2025.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023.
Zhanbo Huang, Xiaoming Liu, and Yu Kong. H-more: Learning human-centric motion representation for action analysis. arXiv preprint arXiv:2504.10676, 2025.
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóżdž. Piotr Sankowski, et al. Scaling laws for fine-gres tne-grained mixture of experts. arXiv preprint arXiv:2402.0788 2024.
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-exppers are domain generalizable learners. arXiv preprint arXiv:2206.04000 2022.
Dengehun Li, Yingzi Ma. Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404.15159, 2024a.
Dengehun Li. Naizheng Wang, Zihao Zhang, Haoyang Yin, Lei Duan, Meng Xiao, and Mingjic Tang. Dynmole: Boosting mixture of lora experts fine-tuning with hybrid routing mechanism. arXiv preprint arXiv:2504.0000 2025.
Wenbing Li, Hang Zhou, Junqing Yu, Zikai Song, and Wei Yang. Coupled mamba: Enhanced multi-modal fusion with coupled state space model. arXiv preprint arXiv:2405. 18014, 2024b.
Xiao Li and Percy Liang. Prefix-tunings tuning: Optimizing continuous prompts for generation. In Proceed- ings of the 59th Annual Meeting of the Association for Computatoonll Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 3458-3470, 2021.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: sora-finettunin--aae aware quantization for large language models. arXiv preprint arXiv:2310.08633 2023.
Menggi Liao, Wei Chen, Junfeng Shen, Shengnan Guo, and Huaiyu Wan. Hmora: Making ilms more effective with hicrarchical mixture of lora experts. In The Thirteenth International Conference on Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Junhong Lin, Xinyue Zeng, Jie Zhu, Song Wang, Julian Shun, Jun Wu, and Dawei Zhou. Plan and budget: Effective and efficient test-time scaling on large language model reasoning, 2025a. URL
https://arxiz: org/ abs 2503 16122
Junkai Lin, Hang Long, Huipeng Guo, Jielei Zhang, JiaYi Yang. Tianle Guo, Yang Yang, Jianwen Li, Wenxiao Zhang, Matthias Nießner, and Wei Yang. Meshripple: Structured autoregressive generation of artist-meshes, 2025b. URL wtt.::ttppp/..o / rxw...i..ommmm
www.iiv.or/aaaao org/abs 22522 C7514
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
Xiao Liu, Kaixuan Peng, Zheng Zhao, Ying Song, Xinyu Tan, Chen Wang. Ming Lyu. Weinan Zhou, Jin Yang, Jianlin Su, al. Gpt understands, too. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 1016-1024, 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning be comparable fine-tuning universally across scales and tasks, 2022. BtRLhttps:////
wwtp://orrgaa/aaaaaeaa/ooossee/ /ttbb////11...m 07602
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficien fine-tuning for large language models. arXiv preprint arXiv:2402.12851, 2024.
Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization. arXiv preprint arXiv:2402.05885 2024.
Ziheng Ouyang, Zhen Li, and Qibin Hou. K-lora: Unlocking trainingffree fusion of any subject and style loras. arXiv preprint arXiv:2502.1 18461, 2025.
Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture-of-experts language models. arXiv preprint arXiv:2404.05567, 2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated offture-of-eepperss layer. arXiv preprint arXi::
2017.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint 3xXv:2331.00385. 2023.
Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Transformer tracking with cyclic shifting window attention. In Proceedings of the IEEE/CVÉ conference on computer vision and pattemn recognition, pp. 3791-8800, 2022.
Zikai Song, Run Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Compact transformer tracker with correlative masked modeling. In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 2321-2329, 2023.
Zikai Song, Run Luo, Lintao Ma, Ying Tang, Yi-Ping Phoebe Chen, Junqing Yu, and Wei Yang. Temporal coherent object flow for multi-object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6978-6986. 2025.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, Advances in Neural Information Processing Systems, 37: 9565-9584 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
12

---

Published as a conference paper at ICLR 2026
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.13628. 2024.
Jingwei Xu, Junyu Lai, and Yunpeng Huang. Meteora: Multiple-tasks embedded lora for large language models. arXiv preprint arXiv:2405.13055 2024.
Shu Yang, Muhammad Asif Ali, Cheng-Long Wang. Lijie Hu, and Di Wang. Moral: Moe augmented lora for IIms' lifelong learning, arXiv preprint arXiv:2402.1 /260, 2024.
Liliang Ye, Yunyao Zhang, Yafeng Wu, Yi-Ping Phoebe Chen, Junqing Yu, Wei Yang. and Zikai Song. Mvp: Winning solution to smp challenge 2025 video track. arXiv preprint arXiv:2507.000 009000 2025.
Elad Zaken, Shauli Ravfogel, Yoav Lang, Ran El-Yaniv, and Naftali Tishby. Bitfit: Simple parameter- efficient fine-tuning for transformer- language models. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 1164-1174, 2021.
Xinyue Zeng. Haohui Wang, Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Unveil- ing fine-tuning dynamics for Ilm selection, 2025. URL
03793
Lianbo Zhang, Shaoli Huang, Wei Liu, and Dacheng Tao. Learning a mixture of granularity experts for ame-grained categorization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8331-8340, 2019.
Yunyao Zhang, Zikai Song, Hang Zhou, Wenfeng Ren, Yi-Ping Phoebe Chen, Junqing Yu, and Wei Yang. ga s²: Comprehensive social network simulation with group agents. Findings of the Association for Computational Linguistics: ACL 2025, pp. 8950-8970, 2025a.
Yunyao Zhang, Xinglang Zhang, Junxi Sheng, Wenbing Li, Junqing Yu, Wei Yang, and Zikai Song. From ambiguity to verdict: A semiotic-gron groutded multi-perspective agent for Ilm logical reasoning. arXiv preprint arXhv:2509.24765 2025b.
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. arXiv preprint arXiv:2409. 2024.
Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao and Muyun Yang. Lora-drop: Efficient lora parameter pruning based on output evaluation. arXiv preprint arX17:240..0772, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 OPTIMIZATION AND CONVERGENCE ANALYSIS.
We analyze the empirical counterpart of LRSI. dataset S {
(S: (p(x,)()))))))))>>))))){}))))(
H(p(z,)),
i=1
j=1
p
p. (ay),
7,
n Σ iii arg maxp)
(13)
n
Smooth surrogate for optimization. by standard smooth surrogate:
For optimization analysis we replace the hard top-1 frequency
Assumption 1 (Smoothed usage surrogate). Replace fr in equation 13 by 3, where ss (x)
softmax(G(()/T)) for some O and & L, 8 (xz)). We assume the composite E, P; 84 is
convex and L-smooth in {p(x,)} on the product simplex A
regatii-eettoo--rrgllaizzz objective.
Define
K
Fs((p(zz))) = 8
(x(1)) log (zz))
(14)
Since - -, logp; = H(p). minimizing P' is identical to minimizing ERST. with f, replaced by 3;; change of logarithm base absorbed into X. We optimize F's over {p(z,)} A using entropic mirror descent (exponentiated gradient)
pl p(az)* O exp (esp(-nV (is then normalize onto A.
(15)
Lemma 1 (Strong convexity). The function h(p) = EK Ps log is L-strongly convex on A w.r.t.
the & norm. Hence the regularizer h is X-strongly convex the product simplex
A Therefore, Fs in equation 14 x-strongly convex, since the usage surrogate term is convex by
Assumption 7
Lemma (Gradient bounds and interior smoothness). Under Assumption 7 there exists > 0 Ouucch that for all feasible {p(z,)}.
V
Vp(z,)
(a
> 1=1
ps 8:)
S..
Moreover,
V |p(z)) n
(((x,) log (gg)
<
+ lo,
so if we maintain Ps (x1) 2 during training (e.g. via temperatuue//clpping)) the right-hand side is uniformly bounded by a constant C(),e, K). On the interior {p p: 2 e), the entropy term Low S Trem + T'aux for some finite Timux ((x/n).
smooth (w.rt. (2), hence Fss T-smooth with ne
Theorem 1 (Convergence of EG/EMD for Fs). Under Assumption [ and Lemmas 7 Fs is
X-strongly convex on A (a) Sublinear rate. With steps Xt" the averaged iterate P Et= pt satisfies
Fs(PT) Fs((*) < O af
'This includes the commonly used choice 3, u giving the auxiliary term a Σ, which is convex since each linear in (p(x)))-
po
14

---

Published as a conference paper at ICLR 2026
where Geft aggregates the uniform gradient bounds in Lemma in addition, training maintains pz(zz) 2 e (hence Fs is L-smooth on the interior), then with any constant step r S L/L L the EG iterates enjoy geometric contraction in KL divergence:
(b) Linear rate on the interior. If.
DKL(P* p(+l) S (1- - (1-1AA) ( Pt),
and consequently (s(Pt) (PP) also decays geometrically.
The negative-entropy term supplies curvature in the routing simplex (Lemma D. turning the merely convex surrogate into a strongly convex objective. This stabilizes and accelerates routing optimization while preserving input-awareness through the token-level log (z) signals (cf. the gradient form in the main RSI., Eq. equation 9).
A.2 GENERALIZATION BOUND VIA ALGORITHMIC STABILITY.
We analyze generalization under the same smoothed usage surrogate as in optimization (Assump- tion i.e., replacing f, by & (in particular, the commonly used choice &; = ir yields convex quadratic auxiliary term). We also keep the empirical objective in sample-average form that the regularization strength does not scale with n.
Let S = {= and (t() be S with the i-th example replaced. Denote by P(S) a (unique) minimizer of the empirical objective
K Fs((p(a;))) = Pj
K
p:(zy)
and similarly P(((i)) for Fsu. For reference, the population counterpart is
K F(((()))) a 5 AE
Σ p:(x) logp,())))) P: Ex((( 5, Eg1sT (z)].
Assumption (Lipschitz loss & bounded differences). (i) The map {p(x,)} + a ΣK 8y is L- Lipschitz on the product simplex endowed with the averaged block norm |P Pllarg-1 := E, [I((;))lı, and replacing
sample in S changes this auxiliary term by most (bounded-difference property). (ii) The per-exana eer-example evaluation loss E(p; x)) is p-Lipschitz in (under || 111 on a single simplex) and bounded in [0, M).
Lemma (Strong convexity under the averaged product norm). Negative entropy h(p)
ΣK1 p; log pa 1-strongly convex on w.r.t. || 111. Therefore A ΣE= h(p(z,)) >-strongly convex on the product simplex endowed with lloog- Since the surrogate usage term is convex (Assumption 7). Fs is X-strongly convex.
Lemma 4 (Uniform stability of ERM for Fs). Under Assumptions (andd2 let P(S) and P(((i) be the ERM solutions. Then the solution mapping uniformly stable:
[|((S) - ((
An
Theorem 2 (Generalization bound for RSL-trained router). Under Assumptions and 2 generalization gap satisfies
the expected
=(D(L+BB) BB}B An
F [=((((())) - 1 Σ ((((S)))))
and, with probability least - 8,
Ex [=((((())):) 1 Σ (((())=))) S {(P(LBB)) + dV
log(1/8) 25
An
15

---

Published as a conference paper at ICLR 2026
Lemma gives A-strong convexity of Fss under llag-1 By standard argument for strongly convex ERM with bounded differences, replacing one sample perturbs the objective by at most O(LL B)/n) and yields uniform stability O(L + B) (An)) for the solution map (Lemma The Lipschitz evaluation loss then converts stability into generalization: Elgap) s p. and a standard McDiarmid/Hoefddin argument implies the high-probability term ((V/oo(1//)///
This matches the empirical robustness of RSL low-data regimes: entropy regularization both improves optimization conditioning and reduces the hypothesis sensitivity to single-sample perturba- tions.
A.3 ADDITIONAL EXPERIMENTAL RESULTS.
The impact of Top-K. To explore the impact of Top-K routing, conducted experiments SST-2 and CoLA using Falcon-Mamba. The results are shown in Figure 5 As K increases from 1 to observe improved accuracy, suggesting that multiple experts can gain complementary information. However, further increasing K may actually degrade performance. Therefore, how to or dynamically learn the optimal K value is direction worthy of further research.
Top-K K Routing Ablation on SST-2 & CoLA
96
(+2.01%)
SST-2
CoLA
94
Accuracy (%)
922
90
88
(4 (.70%)
86
84
82
Top-K K
3
5
6
Figure 5: Top-K Routing Impact.
Cross-domain QA To evaluate LoRA-Mixer's s cross-domain generalization capabilities, we con- structed two guestion-answering datasets:
Mathematics and SineemmatissCCddine Each dataset contains 200 examples generated by DeepSeek-B These questions are more challenging. We conduct DeepSeek-R1 evaluations, and the results shown in Table TO
Table 10: Cross-domain performance of LoRA-Mixer on LLaMA3-8B.
Task
Base LoRAHub MOLE MixLoRA LoRA-Mixer
Math-Medical 69.88
70.53
72.11
72.74 63.10
73.41
Math-Coding 59.37
61.08
62.24
63.46
A.4 PARAMETER ANALYSIS.
We tested the trainable parameters of LoRA-Mixer MoLE, MixLoRA, and LoRAHub under LLaMA33 8B. The trainable parameters LoxA-Mixer and MixLoRA are 3,88% and 8. 8.08%, respectively (six experts routing). parameters accounting for 0.04%. MoLE's training parameters are routing parameters. To ensure a fair comparison, adjusted MoLE's routing reduce the trainable parameters to 0.04%. LoRAHub a train-free free method with 0% trainable parameters. PHATGOOSE's trainable parameters are gating vectors, specifically 0.01 1.
total parameters account for 48% of MixLoRA's. with routing
16

---

Published as a conference paper at ICLR 2026
A.5
DATASETS DETAILS.
We present information about all datasets used here for reference.
Table 11: Statistics of datasets used in experiments.
Dataset
Train
Dev
Test
Task
ARC-E
2251
570
2376
Multiple-choice QA
ARC-C
1119
299
1172
Multiple-choice QA
GSM8K
7473
1319
Math reasoning
MBPP
779
195
Code generation
CoLA
8551
1043
1063
Acceptability judgment
SST-2
10000
872
1821
Sentiment classification
Medical
10030
4004
4004
Medical QA
MRPC
3668
408
1725
Paraphrase detection
MNLI
392702
9815
9832
Natural language inference
QNLI
104743
5463
5463
Question answering/NL.I
RTE
2490
277
3000
Natural language inference
QQP
363846
40430
390965
Paraphrase identification
PIQA
16113
1838
3277
Physical commonsense reasoning
HellaSwag
39905
10042
10042
Commonsense reasoning
BoolQ
9427
3270
3270
Boolean question answering
A.6 COMPUTATIONAL RESOURCES AND REPRODUCIBILITY GUARANTEE.
conducted Linux workstation equipped with single NVIDIA A800 80GB GPU and 32-core Intel Xeon CPU, using Py Torch version 2.1.0 and Transformers version 4.36.2. For the first stage of training, we used LoRA with parameters 64, alpha 128, dropout 0.1, batch size = 1, gradient accumulation = 4, and Ir = le-5. The task loss function was cross-entropy loss, and cosine scheduler with warmup ratio (ratio 0.1) was used. In stage 2, RSL was added with 0.1 and À = 0.5 in RSL. 8 of Epreserve was fixed to 5e-5, and top-3 routing strategy was used. the inference phase, we also adopt the top-3 routing strategy and test the same device environment.
Our experiments
A.7 EFFICIENCY AND RESOURCE OVERIEAD ANALYSIS.
We tested inference/training time, and memory usage, and the results are shown the table 12] These metrics highlight the advantages of LoRA-Mixer beyond task performance. Inference was performed using a single A800 80G GPU, with max_ max_length=512 and max_mexs maa__newttkkes=-le to test single-sample inference time. To climinate the effects of cold starts, all measurements were performed after warm-up run.
Table 12: Inference time comparison across different methods.
Model
Time (s)
LLaMA3-SB (Baseline)
0.441
LoRAHub
0,482
MoLE
0.563
MixLoRA
0.597
LoRA-Mixer (Ours)
0.574
LoRAHub uses training-free approach, resulting faster inference speed. MoLE is nearly identical to LoRA-Mixer. However, this coarse-grained approach often leads to poor performance. Overall, LoRA-Mixer achieves a better balance between inference speed and performance. Overall, LoRA- Mixer optimized with RSL achieves faster convergence and more stable training. For training information, LoRA-Mixer 's training is split into two stages: Stage 1 (expert training) and Stage (router training). The results shown in the table 13
17

---

Published as a conference paper at ICLR 2026
Table 13: Training stages with dataset size, time.
Stage
Data
Time (hours)
Stage 1
40k
8,86
Stage 2
2k
1.74
Stage 2
4k
3.38
LoRA-Mixer can skip stage and quickly complete the stage 2 of training using very little data when computing resources are limited, and provide strong performance. This is very important in resource-constrainedd scenarios. For memory usage, we measure memory usage for 6 experts, r=64 (A800-80G, float32, batchsize=1, gradienticcuullaiinn-))) results are shown in the table 14
Table 14: Memory usage comparison across different methods.
Method
MixLoRA
MoLE
LoRA-Mixer
Memory Usage (GB)
43.07
36.57
38.48
A.8 HYPERPARAMETER GRID SEARCH.
The hyperparameters include a, and À are not many, and we design simple yet effective strategy for tuning the hyperparameters. We start with heuristic values informed by prior work on multi-objective optimization: a 0.1 and À = 0.05 in LASL- We fix B = 5e - 5 in Greserve These values are chosen ensure that the task loss dominates the training the early stages of training. We perform grid search over a narrow range of and À values on validation split of all training data (10% of the total data).
Table 15: Ablation study and X. Best results are highlighted in bold.
Performance
Load Balance (Variance)
Routing Entropy
0.05
0.01
76.2%
0.
1.9
0.1
0.05
78.5%
0.08
1.5
0.2
0.1
75.8%
0.05
1.2
The values a = 0.1 and À = 0.05 achieve the best trade-off across all metrics. To prevent RSL from interfering too much with task loss in the early stages of training, we dynamically adjust parameters during training, for example, gradually increasing a and to their final values in the first 30% of epochs and then keeping them constant. This "warm-up" strategy stabilizes training and improves convergence.
A.9 ADDITIONAL RESULTS ON GLUE TASKS.
We present the results of ARRA-Mixes on the QNLI and MNLI tasks, as shown Table 16 The results show that LoRA-Mixer achieves of-te-of-the-ar results on all GLUE tasks, demonstrating the powerful combination of RSL and LoRA-Mixer. The experimental settings are LLaMA3-8B and 16, and the average performance over three runs is reported.
18

---

Published as a conference paper at ICLR 2026
Table 16: Results on QNLI and MNLI. Higher is better.
Method
QNLI
MNLI
Base
90.36
89.95
LoRAHub
90.44
90.56
MOLE
91.48
91.04
MixLoRA
91.73
91.27
LoRA
91.34
90.87
LoRA-Mixer (Ours)
92.04
91.45
We exclude STSB because it focuses on measuring semantic similarity, which is inconsistent with our core goal of "cross-task expert collaboration' (classification, reasoning, and generation). and the results are more likely to be random. Furthermore, we want to be consistent with previous studies such as GOAT, LoRA-LEGO, and MixLoRA, which did not conduct experiments on this dataset.
A.10 ADDITIONAL EMPIRICAL SUPPORT FOR LORA MIGRATION.
To better understand LoRA-Mixer's cross-model transfer experiments, analyzed the architectures of Mistral-7B and LLaMA3-8B. The Mistral-7B and LLaMA3-8B models share a core decoder- specific Transformer structure, and the key dimensions match, shown below:
Table 17: Comparison of architecture components between Mistral-7B and LLaMA3-SB.
Architecture components
Mistral-7B
LLaMA3-8B
Number of layers
32
Hidden dimension (dmodel)
4096
4096
Attention heads
32
32
Feedforward dimension (dn)
14336
14336
Attention projection layer
Q, K, V. output (all 4096 > 4096) SwiGLU
Q. K. V, output (all 4096 + 4096)
Activation function
SwiGLU
This structural alignment ensures that linear projection layers with the same input/output dimensions can achieve parameter transfer.
A. FALCON-MAMBA ARCHITECTRRE ANALYSIS.
Mamba builds upon the state space model. It processes input sequence sr(() RRL to produce output ytt E R² by employing a hidden state h(t) e RR, This relationship is initially defined by continuous system:
h((t) = Ah(t) + Br(t), (t) Ch(t).
(16)
Here, A E NxN is the state transition matrix, and B E Nx1, CE Nx1 are projection matrices.
To process discrete sequences, Mamba discretizes the continuous parameters and B using a time scale parameter A and the zero-order hold (ZOH) principle, resulting in discretized parameters A and B:
A exp (AA),
(17)
B (AA) (exp (AA) - I) AB.
The discrete state-space equation with a step size of A becomes:
Ah Bat:
(18)
Cha
19

---

Published as a conference paper at ICLR 2026
By iteratively expanding the hidden state h4-1, Mamba derives global convolution kernel K E R This kernel is then used to compute the output through a convolution operation with the input æ:
K CB, CAB CA CA)")) B), T K.
(19)
Falcon Mamba 7B adopts a pure Mamba architecture, departure from hybrid designs incorporating staggered attention. This deliberate choice aims to maintain the intrinsic linear scalability characteris- tie of Mamba models. To enhance adaptability, the model employs decoupled input embeddings and output weights.
At core, Falcon-Mambb features 64 layers the Falcon-Mamba Mixer. Each Mixer layer integrates an SSM (State Space Model) module alongside in-projection and out-projection layers, RMS Norm, and a convolutional layer.
Within the SSM module, the input is mapped to A, B, and C' through projection layer denoted as x-proj:
proj (A, B)C)
where x represents the input to the SSM module. Furthermore, another projection layer, dt dt-proj. discretizes A:
dt-proj A discretized
These discretized values A diccetizzdd A, B, C, and D are then fed into the selective scanning module for processing, This architectural design of Falcon-Mamba fully enables the application of LoRA-Mixer specifically tailored for the projection layer.
A.12 THE IMPACT OF IN LORAS
Table 18: Comparison LoRA-Mixer on Falcon-Mamba, Mistral, and LLaMA across seven tasks. LoRA denotes single-task fine-tuning with rank 16. Best results per model and task highlighted in bold.
Methods
Medical
CoLAA
SST2
GSMSK
ARC-E
ARC-C
HumanFyal
FalconMamba-LoRA
76.33
82.75
93.23
54.62
83.97
76.08
28.66
+LoRA-Mixer
77.03
83.80
93.41
55.15
84.17
76.51
29.48
Mistral-LoRA
67.87
75.55
89.14
45.96
84.37
69.51
34.76
+LoRA-Mixer
68.27
77.64
90.27
45.61
84.46
70.15
34.68
LLaMA-LoRA
79.35
77.65
94.15
61.79
88.64
79.47
51.78
+LoRA-Mixer
79.88
78.11
94.97
61.14
89.29
79.87
53.39
Table 19: Comparison LoRA-Mixer on Falcon-Mamba, Mistral, and LLaMA across seven tasks. 32. Best results per model and task are
LoRA denotes single-task fine-tuning with rank r highlighted in bold.
Methods
Medical
CoLA
SST2
GSM8K
ARC-E
ARC-C
HumanEval
Dalcon-Mamba-LLod
76.32
85.90
93.12
54.76
84.86
75.67
32.33
+LRRA-Mixer
76.67
86.00
95.35
55.41
85.55
76.81
34.15
Mistral-LoRA
68.57
76.89
93.87
46.29
84.87
68.83
32.93
-LoRA-Mixes
68.88
78.81
94.60
45.91
85.91
71.80
33.56
I.A.MA-LORA
79.15
81.11
95.30
61.38
88.76
79.31
52.34
LLoRA-Mixe
80.87
81.30
95.53
62.46
89.04
79.48
53.65
20

---

Published as a conference paper at ICLR 2026
Table 20: Comparison LoRA-Mixer on Falcon-Mamba, Mistral, and LLaMA across seven tasks. LoRA denotes single-task fine-tuning with rank 128. Best results per model and task are highlighted in bold.
Methods
Medical
CoLAA
SST2
GSM8K
ARC-E
ARC-C
HumanEval
Falcon-Mann Mamba-LoRA
76.42
85.25
92.88
55.25
83.91
75.72
32.41
+LRRA-Mixer
76.81
85.97
94.30
56.11
85.50
77.75
33.10
Mistral-LoRA
69.63
79.85
90.14
46.05
84.62
68.59
34.87
+LoRA-Mixer
69.82
80.75
91.55
44.55
85.85
71.74
35.17
LLaMA-LORA
79.38
81.48
95.27
62.04
89.21
80.28
55,40
+LoRA-Mixe
80.82
82.25
95.50
63.41
89.33
81.43
56.31
As shown in Table 18 Table 19 and Table 200 within a certain range, as r' increases, the performance of the model can be improved to certain extent. Our method is not only better than the basic model, even better than the fine-tuned basic model on most tasks. This shows that our method can effectively combine existing knowledge through dynamic expert combination to form a more "intelligent" model.
A. 13 BALANCE LOSS VISUALIZATION.
Training Balance Curve
1.00
0.95
0.90
0.85
0.80
0.75
0.70
500
1000
1500
2000
Step
Figure 6: Balance loss curve using RSL loss during training.
Figure shows the changing trend of the Balance Loss when the RSL loss function is used during training. As shown in the figure, the Balance Loss drops rapidly in the early stage of training, indicating that our model can quickly Iearn an effective expert routing strategy, thanks to the synergy of the global consistency term and the local entropy penalty term in the RSL loss function. In the middle stage of training, the Balance Loss continues to drop steadily with small fluctuation, which reflects the stability that the RSL loss function brings to the training process. In the late stage of training, the Balance Loss remains stable at low level, further demonstrating the balance and optimization effect of the model the use experts. Overall, this Balance Loss curve not only reflects the model's ability to converge quickly, but also demonstrates the robustness of the training process, verifying the significant advantages of the RSL loss function in improving model performance and training efficiency.
21

---

Published as a conference paper at ICLR 2026
A.14 IN-DISTRIBUTION EXPERIMENTAL RESULTS.
Here, we present experimental results for LoRA-Mixer and PHATGOOSE on five in-distribution experiments (GSM8K, ARC-E, ARC-C, CoLA, and SST2). The results are shown in Table 21 Both methods use the same LoRA adapter (Q/K/V/O) on LLaMA3-8B and are trained using the same data; their routing strategies differ: LoRA-Mixes uses RSL., while PHATGOOSE uses sigmoid gating based cosine similarity. It can be seen that LoRA-Mixer outperforms PHATGOOSE across the board.
Table 21: Comparison of Base, Phatgoose, and LoRA-Mixer across five datasets.
Dataset
Base
Phatgoose
LoRA-Mixer
SST2
93.12
91.35
94.83
CoLA
79.14
78.27
81.17
GSM8K
57.92
58.87
62.66
ARC-E
88.45
87.13
89.47
ARC-C
78.65
75.94
79.89
In order to verify the enhanced performances of individual expert after LoRA-Mixer optimization. We selected four tasks, GSM8K, SST2, CoLA and HumanEval, for experiments. The results are shown in Table ??. We can find that LoRAs optimized by LoRA- mixer exhibit improved performance, especially in the GSM8K task, where the performance improved by 5.36 % after adding mathematical experts. This result confirms the enhanced ability of each individual LoRA expert after LoRA-Mixer optimization.
A.15 DETAILS OF LORA MODULES OF FLAN-T5.
To better understand the Flan-T5 LoRAs experiment from the Internet, we introduce the corresponding LoRAs configuration information here, details as 22
Table 22: LoRA configuration details used in our experiments.
Parameter
Value
Dase_model_name_or__pe
google/flan-t5-1ee
bias
none
fan_in_fan iin_fan_out
false
inference_mene _mddeeeeeeddeeeeee
true
init_lora_weights
true
layers_pattern
null
Layers_to_transtorr
null
1ora_alpha
32
lora_dropout
0.1
modules_toosavv
null
peft_
LORA
r
16
revision
null
target_modules
[q. v]
task_type
SEQ_2_SEQ_In
A.16 EXPLANATION OF RSL'S SUBOPTIMAL PERFORMANCE WITH 4K DATA.
RSL's temporary performance fluctuations on 4k datasets can be explained by its design philosophy. RSL integrates entropy regularizer to balance expert specialization and load balancing. On small datasets (1k-3k instances), this mechanism helps the router quickly focus on the most relevant experts by avoiding overly uniform activation functions, thus outperforming common auxiliary loss functions that tend to enforee an indiscriminate balance. When the number of instances reaches 4k, RSL begins explore finer-grained expert tasks (consistent with its goal of input-aware routing). However,
22

---

Published as a conference paper at ICLR 2026
this phase can lead temporary instability, as the data volume is sufficient to trigger exploration but not yet rich enough to fully stabilize finer routing patterns. In contrast, the original auxiliary loss function maintains stable but suboptimal uniform activation function, showing temporary advantage. However, as data size increases further, RSL's advantage increases. This is because the auxiliary loss becomes overly averaged and loses input awareness, resulting in less accurate expert selection and suboptimal performance.
A.17 EXCESSIVE AVERAGING OF AUXILIARY LOSSES.
The standard auxiliary loss mixture-of-experts is designed to encourage load balancing by aligning the average routing probability p; with the average usage frequency ft:
Lax
P.
Σ
P: =1
(20)
Here p, = (x)) and f. B = arg max, Pý (.))}. Since f, depends on hard top-1 routing and is not differentiable, implementations usually adopt smooth surrogate e.g. softmax usage with temperature 7 > 0. With this surrogate, the alignment loss is often combined or replaced with a quadratic balancing form:
Lbal
Σ
1
(21)
which penalizes deviations from equal usage. Consider the Lagrangian
((p,
Pj
Σ P 1).
(22)
1=1
Setting OC/0p. yields 2p. 1 =0 P X/2 for all i. Summing over gives 2/ hence
p
Vi.
(23)
K
The unique minimizer of Loal is the uniform distribution, showing that traditional auxiliary losses inherently bias the router towards equal activation and thus hinder expert specialization.
A.18
USE OF LLMS.
We use LLMs as basemodel for our experiments, and we also use LLMs for syntax polishing and checking.
23


Reviews:
[{"review_id": "qrvM67Yr7t", "reviewer": "Reviewer_RivB", "summary": "To enhance the capability of expert specialization in Mixture-of-Experts (MoE) architecture models, a method called LoRA-Mixer is proposed. LoRA-Mixer incorporates multiple LoRA modules as experts into the linear projection layer, enabling adaptation to models with either Transformer or SSMs architectures. Additionally, to balance global load balancing in routing and expert specialization, an RSL loss is introduced. This loss adds a token-level entropy term (at the token level) to the existing auxiliary term for global load balancing.", "questions": "Can the authors provide further ablation experiments on adding LoRA-Mixer to the input projection layer and the output projection layer respectively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By treating multiple LoRA modules as experts and integrating them into the linear projection layer, the method can adapt to models with either Transformer or SSMs (State Space Models) architectures.\n2. The RSL loss is proposed to balance global load balancing in routing and expert specialization.\n3. Experiments conducted on models with different frameworks have validated the effectiveness of LoRA-Mixer.", "weaknesses": "1. The authors added LoRA-Mixer to both the input projection layer and the output projection layer, but they did not further analyze the respective contributions of these two parts to the experimental results in their experiments.\n2. The authors haven't provided the code, so it's impossible to determine whether the results can be reproduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0zYuENumH", "reviewer": "Reviewer_A4gH", "summary": "This paper introduces LoRA-Mixer, a modular mixture-of-experts (MoE) framework that coordinates multiple LoRA adapters within the linear projection layers of attention or state-space models.\nIt proposes a new Routing Specialization Balance Loss (RSL) to balance expert load while promoting input-aware specialization through entropy regularization.\nThe method supports plug-and-play reuse of pre-trained LoRA modules and demonstrates strong performance and data efficiency across 15 benchmarks such as GLUE, GSM8K, and HumanEval.\nResults show LoRA-Mixer achieves higher accuracy with 48% fewer trainable parameters compared to existing LoRA-MoE baselines and transfers well across architectures (e.g., from Mistral-7B to LLaMA3-8B).", "questions": "**Question on benchmark selection (no extra experiments required)**\n\nCould the authors clarify the rationale for the main-table task set (Medical, CoLA, SST-2, GSM8K, ARC-E, ARC-C, HumanEval)? In practice, we’ve observed high variance on some GLUE-style tasks (e.g., CoLA, SST-2, RTE), which can blur comparative conclusions. It would help to include a short analysis addressing:\n+ Why these tasks? Please explain how they map to the paper’s goals\n+ Variance and reliability. Do these tasks provide stable rankings under multiple seeds? If possible, comment on typical standard deviations or confidence intervals you observe (no new runs needed—prior experience/known stats are fine).\n+ Difficulty/calibration. Why not include more complex and widely used suites like MMLU, math benchmarks, or additional code benchmarks? If excluded, a brief justification (e.g., cost, mismatch with claimed capability, or lack of compatible LoRA experts) would make the scope clearer.\n\nIf the above concerns can be reasonably addressed or clarified in a revision, I would be inclined to raise my overall score", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Novel modular design: Routes LoRA experts directly through projection layers, improving attention specialization and maintaining compatibility with both Transformers and SSMs.\n\n+ Effective routing optimization: The proposed RSL loss theoretically and empirically stabilizes training while enhancing expert selectivity and data efficiency.\n\n+ Comprehensive evaluation: Strong empirical gains on diverse benchmarks and clear demonstrations of cross-model transferability and low-data robustness.", "weaknesses": "1. **Layer placement strategy is unclear and may be sub-optimal.**\nLoRA-Mixer is described as being “applied to the linear projection layers” and claimed to be architecture-agnostic, but there is no systematic ablation to determine which layers benefit most or whether per-layer routing depth should vary. The authors choose to integrate the router into the attention projection layers (Q/K/V/O) rather than the FFN projections, yet this design choice is not well justified.\nWhile the motivation seems to be leveraging attention’s semantic sensitivity and avoiding interference with task-specific FFN adaptation, most prior LoRA-MoE and MixLoRA works inject modular experts into the FFN for scalability and expressive power. A comparative analysis of applying LoRA-Mixer in attention vs. FFN projection layers—under equal compute and parameter budgets—would help validate whether the current design is optimal. Moreover, the paper itself notes that using a uniform Top-K across layers may introduce redundancy, suggesting this design space remains underexplored.\n\n2. **Lack of evidence for scalability to larger base models.**\nAll experiments are conducted on small-to-medium-scale LLMs such as Mistral-7B, LLaMA3-8B, and Falcon-Mamba-7B, with limited discussion of scalability to models exceeding 30B parameters. It remains unclear whether LoRA-Mixer maintains routing stability, expert load balance, and training efficiency at larger scales.\nWhen scaling up, the authors are encouraged to provide empirical results or analysis on larger LLMs, examining aspects such as training/inference latency, memory overhead, convergence speed, and RSL behavior. Demonstrating consistent benefits in larger architectures would significantly strengthen the paper’s generality and practical value.\n\n3. **Missing complexity and parameter-efficiency comparison with LoRA, MixLoRA, and MoLE.**\nWhile performance comparisons are extensive, the paper lacks a quantitative analysis of trainable parameters, FLOPs, memory usage, and inference latency across methods such as LoRA, MixLoRA, and MoLE. Without aligning these metrics, claims of data or parameter efficiency cannot be fairly assessed.\nIt would be valuable to include a table summarizing, for each baseline, the number of trainable parameters, computational cost (FLOPs), and peak memory footprint, as well as an efficiency–accuracy trade-off curve under comparable resource budgets. This would make the results more interpretable and confirm whether LoRA-Mixer’s gains arise from better design rather than higher capacity or computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0QyO0Mg8BD", "reviewer": "Reviewer_EiuV", "summary": "This paper introduces LoRA-Mixer, a modular mixture-of-experts (MoE) framework for combining multiple LoRA adapters through serial attention routing at the projection layers of Transformers or state-space models (SSMs). Instead of replacing full attention or FFN blocks, LoRA-Mixer injects multiple LoRA “experts” directly into the linear projections and learns a lightweight router optimized by a novel Routing Specialization Balance Loss (RSL). RSL integrates entropy-based specialization and global load balancing, encouraging both expert diversity and stability. The authors claim that this approach achieves fine-grained token-level routing with substantially fewer trainable parameters. Experiments show consistent gains over LoRA-MoE baselines such as MixLoRA, MoLE, and LoRAHub while using only ~48 % of their parameters.", "questions": "(1) Could the authors provide runtime measurements (training/inference latency, FLOPs, or GPU memory) under matched hardware?\n\n(2) Please clarify whether the router operates per-projection (Q/K/V/O), shared per layer, or selectively on certain layers. \n\n(3) Could the authors include quantitative ablations or trend plots showing the sensitivity of RSL coefficients (λ, α), preservation regularizer β, and Top-K to model performance? \n\n(4) Would it be possible to visualize token-level routing entropy or variance to directly support the claim of input-aware specialization?\n\n(5) In Table 5, performance improves on GSM8K but drops slightly on ARC-E when transferring from Mistral-7B to LLaMA3-8B. Could the authors analyze why transferability differs across tasks or layers?\n\n(6) Would the authors consider adding results for recent MoE-LoRA variants (e.g., DynMoLE 2025, MORAL 2024) under identical data and training setups to strengthen external comparisons?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) the paper introduces a simple yet effective way to route LoRA experts through attention projections, improving modularity without large architectural changes.\n\n(2) The proposed RSL loss is a reasonable refinement that balances specialization and load, supported by clear theoretical grounding.\n\n(3) Experiments across multiple models and tasks show consistent, reproducible gains with good parameter efficiency.", "weaknesses": "(1) The efficiency claim relies mainly on the “48% trainable parameters” figure, without corresponding latency, throughput, FLOPs, or memory comparisons under matched hardware conditions.\n\n(2) The paper does not clarify whether routers are defined per-projection (Q/K/V/O), shared per layer, or applied only to selected layers, which affects both computational cost and specialization behavior.\n\n(3) The influence of RSL coefficients (λ, α), preservation regularizer β, and Top-K settings is not systematically reported; the cited appendices lack quantitative curves or trend analyses.\n\n(4) Figures primarily show average expert load across tasks rather than per-token entropy or variance, leaving the “input-aware routing” claim insufficiently supported.\n\n(5) Transfer from Mistral-7B to LLaMA3-8B improves GSM8K but slightly degrades ARC-E, indicating partial rather than robust transferability.\n\n(6) While several routing-loss baselines are included, recent MoE-LoRA variants from 2024–2025 (e.g., DynMoLE, MORAL) under matched protocols are missing, limiting external comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)