LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
Authors: Yuan Zhuang
Yi Shen
Yuexin Bian
Qing Su
Shihao Ji
Yuanyuan Shi
Fei Miao
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture of Experts
Mixture of LoRA Experts
Dynamic routing
Fully differentiable
LoRA
MoE
Primary area: foundation or frontier models, including LLMs

Abstract: Recent studies have shown that combining parameter-efficient fine-tuning (PEFT) with mixture-of-experts (MoE) is an effective strategy for adapting large language models (LLMs) to the downstream tasks. However, most existing approaches rely on conventional TopK routing, which requires careful hyperparameter tuning and assigns a fixed number of experts to each token. In this work, we propose LD-MoLE, a Learnable Dynamic routing mechanism for Mixture of LoRA Experts that enables adaptive, token-dependent, and layer-wise expert allocation. Our method replaces the non-differentiable TopK selection with a differentiable routing function and a closed-form solution. Moreover, our design allows the model to adaptively determine the number of experts to activate for each token at different layers. In addition, we introduce an analytical sparsity control objective to regularize the number of activated experts. Extensive experiments on the Qwen3-1.7B and Llama-3.2-3B models show that LD-MoLE achieves the highest average scores compared to state-of-the-art baselines, across a diverse set of benchmarks. Our method not only achieves superior performance, but also demonstrates the ability to learn token-dependent and layer-wise expert allocation.

Published as a conference paper at ICLR 2026
LD-MOLE: LEARNABLE DYNAMIC ROUTING FOR MIXTURE OF LORA EXPERTS
Yuan Zhuang!; Yi Shen": Yuexin Bian', Shihao Ji', Yuanyuan Shi,,
Qing Sut,
Fei Miao
Uniiersityy of Connectieut, University of Pennsylvania, UUniversity of California San Diego (yuan. 2. zhuang, qing. su, shihao. ji, fei feiimmao/o aacaoeeuconnee. edu, eshen6959 6e16699999mm1111 com, {yubian, eyshieeucsd.edu
ABSTRACT
Recent studies have shown that combining parameter-efficient fine-tuning (PEFT) with mixture-of-experts (MoE) is an effective strategy for adapting large language models (LLMs) the downstream tasks. However, most existing approaches rely conventional TopK routing, which requires careful hyperparameter tuning and assigns a fixed number of experts each token. In this work, we propose LD- MoLE. a Learnable Dynamic routing mechanism for Mixture of LoRA Experts that enables adaptive, token-dependent, and layer-wise expert allocation. Our method replaces the non-differentiable TopK selection with differentiable rout- ing function and a closed- form solution. Moreover, our design allows the model adaptively determine the number of experts to activate for cach token at dif- ferent layers. addition, we introduce an analytical sparsity control objective to regularize the number of activated experts. Extensive experiments the Qwen3- 1.7B and LJama-3.2-3B models show that LD-MoLE achieves the highest average scores compared state-of-the-art baselines, across diverse of benchmarks. Our method not only achieves superior performance, but also demonstrates the ability learn token-dependent and layer-wise expert allocation. The code is available at: https: / github..hh....g.. com/ wttehh/tt--Mont
INTRODUCTION
Large language models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing (NLP) tasks. However, their growing size requires significant compu- tational resources for full-parameter fine-tuning. To address this, Parameter-Ffficcae (PEFT) methods, such as Adapter-tuning (Houlsby et al., 2019) and LoRA (Hu et al., 2021), have emerged as crucial techniques for reducing training costs.
Fine-tuning
Recently, the Mfxture-of-Experre (MoE) design (Jacobs al., 1991; Shazeer et al., 2017) has been successfully integrated into transformer feed-forward networks during LLMs pretraining (Dai et al., 2024: Yang et al., 2025), demonstrating that MoE can reduce computational cost while maintaining strong performance. This has inspired a promising direction for PEFT, leading to the Mixture of LoRA Experts (MoLE) framework (Wu et al., 2024: Dou al., 2024: Zadouri al., 2023). MoLE utilizes multiple LoRAs as experts, providing a scalable and efficient alternative to relying on single LoRA- where high-rank configurations risk overfitting and increased compute cost (Zhang et al., 2023), while low-rank ones often underperform (Liao et al., 2025; Gao al., 2024).
Despite substantial advances, many recent MoE variants remain constrained by rigid routing strate- gies. A prominent example is MoLA (Gao et al., 2024), which relies conventional TopK routing. This approach forces every token to consult fixed number experts, introducing a manually tuned hyperparameter that prevents adaptive allocation of resources based token complexity. In
'These authors contributed equally to this work.
1 Cooresponning author: Fei Miao

---

Published as a conference paper at ICLR 2026
Hidden
Learnalee Dynamic Routing
Experts Layer
Pretrained Weight
Mixture
LoRA Experts
Hidden Output
Figure 1:
The overview the LD-MoLE architecture, which enables Learnable Dynamic Routing (details in Section 3 and Fig 2 (c)) for LoRA adapters with the Mixture-of-Fxperrr paradigm.
addition, the discrete and non-differentiable nature of the Topk operator hinders end-to-end opti- mization, ultimately limiting both performance and scalability (Shazeer al., 2017; Zoph et al., 2022; Wang et al., 2025). Recent efforts such as ReMoE (Wang et al., 2025) attempt to bypass this bottleneck by replacing TopK with ReLU-based router, but this dynamic scheme can suffer from instability, as some tokens may be routed to no experts at all, degrading overall performance. Taken together, these limitations highlight central challenge: Can we design a routing mechanism that adaptively learns to allocate experts in stable and differentiable way?
In this work, we propose LD-MoLE (see Figure 1), a Learnable and Dynamic routing method to adaptively control LoRA experts allocation. We adopt Sparsegen (Laha et al., 2018) as the projec- tion onto the probability simplex and propose a dynamic routing mechanism and the corresponding training pipeline that has the following benefits: (1) the closed-form formulation with Sparsegen to decide routing probability ensures differentiability and guarantees that every token is assigned to at least expert; (2) the routing admits well-defined subgradient; (3) the derivative of the routing upper-bounded, facilitating stable optimization; and (4) the routing design supports sparse yet controllable allocations. Building this foundation, we introduce lightweight, shared multi-layer perceptron (MLP) that predicts a token-specific sparsity parameter X, governing expert selection. In addition, we formulate a sparsity control objective derived from Sparsegen's 's analytical solution, enabling direct regularization over the number of activated experts.
We conduct extensive experiments validate the effectiveness of LD-MoLE, Specifically, adopt Llama-3.2 3.2-BB and Qwen3-1.78 as base LLMs and fine-tune them on wide range of instruction- tuning and sequence classification benchmarks. LD-MoLE achieves the best performance across these benchmarks, outperforming prominent baselines that follow different routing strategies Mol.A (Gao et al., 2024) with conventional TopK routing and ReMoE (Wang al., 2025) with ReL.U-based dynamic routing. These results indicate that our learnable routing mechanism yields consistent improvements across tasks and architectures. Moreover, we show that our sparsity con- trol loss effectively reduces the number of activated experts without compromising performance.
Our contributions are threefold:
1. We propose LD-MoLE, novel MoLE framework with an end-to-end learnable dynamic routing mechanism that adaptively allocates experts to tokens across layers.
2. We introduce an analytical sparsity loss, derived from the closed-form solution of Sparsegen, to explicitly regulate the number of activated experts.
3. We conduct comprehensive experiments on Llama-3.2-3B. Llama-3.2-881 and Qwen3-1. 1.7n3 including ablation studies and detailed analyses, demonstrate the effectiveness of LD- MoLE and to elucidate the mechanisms behind its improvements over TopK and ReLU- based routing.
2 RELATED WORK
Mixture of Experts. MoE first introduced in the 1990s (Jacobs et al., 1991) and later applied to lcrge-scale neural networks (Shazeer et al., 2017) to efficiently scale up model capacity. Land- mark models like Google's GShard (Lepikhin et al., 2020) implement sparse MoE frameworks with
2

---

Published as a conference paper at ICLR 2026
Top2 expert routing, while Switch Transformer (Fedus et al., 2022) simplifies this to single expert per token to reduce overhead. Today, MoE has been widely adopted in several well-known large language models, including GLaM (Du al.. 2022), Mixtral-8x7B (Jiang et al., 2024), DeepSeek- MoE (Dai et al., 2024), Qwen3 (Yang et al., 2025) and LongCat-Flast (Team et al., 2025).
Routing Approaches in MoE. Various routing strategies have been proposed for expert selection. The most common is TopK routing (Shazeer et al., 2017), where each token selects a fixed number of experts. There are also several works that discuss variants of TopK. AdaMoE (Zeng et al., 2024) uses conventional Topk routing with k larger than vanilla MoE but achieves token-adaptive expert se- lection by incorporating null experts, which are defined as an empty operation. Ada-K K Routing (Yue et al., 2024) introduces an allocator and then obtains k* for customized expert resource allocation instead of fixed TopK through non-differentiable sampling operation with RL-based optimiza- tion framework. Alternative designs, such as expert-choice routing (Zhou et al., 2022) reverse this perspective by allowing experts select tokens. Beyond fixed-A: approaches, several methods aim to enable dynamic routing. For instance, TopP routing (Huang et al., 2024) selects experts until a cumulative probability threshold is reached, while DYNMOE (Guo et al., 2025) introduces Top-Any Gating to eliminate the need for tuning k. Soft MoE (Puigcerver et al., 2024) merges tokens and assigns them to experts as linear combinations, and Lory (Zhong et al., 2024) proposes a fully differ- entiable routing mechanism but underperforms TopK routing. Closest to our work, ReMoE (Wang et al., 2025) employs Rel.U-based routing for differentiable and dynamic selection.
Mixture of LoRA Experts. Combining multiple LoRA modules (Hu et al., 2021) with MoE struc- ture has led to the Mixture of LoRA Experts framework (Wu et al., 2024). Several variants have since been proposed: LoRAMoE (Dou et al., 2024) introduces MoE-style plugins to enhance downstream performance while mitigating knowledge forgetting: HMoRA (Liao al.. 2025) employs hybrid scheme that hierarchically integrates token-level and task-level routing. MixLoRA (Li et al., 2024) builds a resource-efficient sparse MoE from LoRA modules. Other works, such as MoLA (Gao et al., 2024) and Alphal LoRA (Qing et al., 2024), analyze expert allocation patterns across layers. In this work, we introduce LD-MoLE, which integrates the fully differentiable Sparsegen formulation with learned MLP to predict X, enabling end-to-end dynamic expert routing.
3 APPROACH
As illustrated in Figure 2, we introduce learnable dynamic routing mechanism that adaptively selects experts. Traditional MoE models (a) employ TopK routing, where each token is assigned to fixed number of experts according its top softmax scores. In contrast, our method (b) em- ploys a closed-form routing formulation involving a token-dependent sparsity factor X, predicted by lightweight shared MLP, that controls the projection function, and thereby regulates the number of activated experts. This design enables the model to allocate more experts to tokens that demand greater modeling capacity and fewer to those that are easier to represent, effectively balancing effi- ciency and expressivity.
3.1 THE LD-MOLE ARCHITECTURE
In this section, we first review the TopK routing, then present our proposed Sparsegen routing with dynamic expert allocation, and finally describe how is combined with LoRA to form the complete LD-MOLE architecture.
TopK Routing. The TopK router in a MoE layer determines the assignment of each token to the most suitable & experts. In general, TopK routing computes a softmax distribution over the experts and calculates a weighted sum of the largest & experts.
Formally, let E and d be the number of experts and input dimension respectively. We define the gate
score weight matrix W gate R and logits E E The conventional TopK routing method takes token embedding x as inputs to predict the scores assigned to each expert:
W D R
3

---

Published as a conference paper at ICLR 2026
Topk
(c) Learnable Dynamic Routing Structure
Shared
Hidden Input
MLP f(x)
Learnable Dynamic Routing
Router
gate score weight matrix W.-
Softnax Score
Token
(a) routing
Sparsegen Router
Sparsegen Projection
Sparsegen routing
Rouber
Router
Score
p
P
sparsity factor
(b) Sparsegen routing
LoRA Experts
Figure 2: (a) Standard TopK routing activates a fixed number (K) of experts using non-differentiable selection. (b) Sparsegen routing introduces differentiable projection onto the probability simplex, controlled by a sparsity parameter A, which enables adaptive expert selection. (c) In the Sparsegen routing module, for each token, lightweight shared MLP predicts the sparsity factor A. Together with the logits u, À determines the probability simplex p over LoRA experts, enabling dynamic, token- dependenn expert allocation across layers. The detailed mathematical formulation is provided in Section 3.1.
Define (u) the index set of the TopK largest entries of u. The routing output p R is then given by
exp(u:) Ejs exp(u,))
i E (u),),
p:
otherwise,
which yields a sparse probability vector with at most nonzero entries. Note that the selection operator Sx(.) introduces jump discontinuity at the k-th largest value. Consequently, an arbitrarily small perturbation the router scores can change the selected set and induce an abrupt change in the gradient, rendering the routing function non-differentiable at these boundaries.
Despite the success conventional TopK routing with softmax operation in improving training and inference efficiency, two limitations persist (Guo al., 2025; Wang et al., 2025): (1) TopK routing con-differentiable during the learning process.
(2) The value of requires carefully tuned to optimize model performance and would be fixed throughout the training process (Guo al., 2025).
In contrast, Sparsegen (Laha et al., 2018) produces sparse routing weights via a closed-form pro- jection, avoiding discrete TopK selection and yielding well-defined gradients that better align opti- mization with the routing behavior.
Learnable Dynamic Routing. To address the aforementioned limitations, propose learnable dynamic routing mechanism based on Sparsegen (Laha et al., 2018), which a projection onto the probability simplex that generates sparse outputs via closed-form and fully differentiable solution. Given the score vector u in Eq. 1, the routing function adaptively determines the effective number of activated experts by solving closed-form transformation where is a sparsity scalar:
p argmin |pp P
ull
s.t. p2p20. 0. 1 1 1,
In our work, we introduce a lightweight MLP to predict a token-wii wise sparsity factor to control the degree of sparsity in the expert allocation, where f denotes the shared MLP that produces scaling coefficient λ conditioned on æ:
A f(x) R.
The proposed routing function admits the following closed form.
Proposition 1 (Closed-foml form Sparsegen routing: Proposition 0.1 in (Laha et al., 2018)). Let u e E in Eq. be the expert scores associated with token æ. and let u(1) 2 u(E) be the
4

---

Published as a conference paper at ICLR 2026
sorted coordinates of Define the cumulative sums UR- -iz u(i) k 1...., ....,,E. E. Then the
Sparsegen routing distribution p R with sparsity parameter À E (-00, 1) is given by
u; T 1-
pj =
E
(4)
where [a]+ = max(x 0), and the threshold T is determined as
Uk 1+ k h max E { X Fu(()) > Uk}
(5)
such that p lies the probability simplex, i.e, EES Pi
Proof. This result follows from solving the Sparsegen projection problem, which minimizes a
strongly convex objective subject to simplex constraints (Laha et al., 2018).
As shown in Figure 2, the sparsity factor and the input logits u jointly determine the threshold T, which defines the change point of the differentiable routing function. Intuitively, À controls the tendency toward sparsity in the solution. As À + 1 it pushes the distribution toward the simplex corners (sparse), while as À - -00, it drives the solution toward uniform simplex. Furthermore, we establish a key property of Sparsegen relevant to our setting:
Lemma 1 (Sparsegen selects at least one expert.). Let u Rº and X < The sparsegen solu- tion (equation 2) always has nonempty support: ||llo 2 1.
We provide a full proof for this lemma in Appendix A. Overall, the closed-form formulation in Proposition 1 offers both theoretical and practical advantages. It enables efficient computation of routing distributions and introduces a tunable sparsity factor X, which allows the model to adaptively select dynamic number of experts. Importantly, the routing remains fully differentiable, ensuring compatibility with end-to-en training.
Model Layout. In our work, we incorporate carameter-efficient LoRA adaptation into the MoE architecture. Each expert network is a LoRA module, where instead of updating the full weight matrix W. E Rdea Xdi, low-rank update is introduced:
AW, = A, A, E R.uux B. E TRTx,..
with r < min(doa, dmn) To adaptively capture non-linear relationships between token-level fea- tures, employ a d (equation 3) predicted by shared MLP in Fig 2 with & 1, 1,.... T for sequence of I' tokens. The æ, is the token feature and would be the input for the shared MLP in Eq. 3. For each unique input size, dimensionality. The shared MLP structure greatly reduces the number of additional parameters re- quired while still allowing the router to predict A dynamically. This design decouples the parameter cost of predicting A, from both the number of layers and the number of experts, leaving it dependent solely on the set of unique input dimensions. Given A the proposed router generates the rout- ing weights p, (equation 4) for each token and determines the weighted aggregation of the output embedding h, from the LoRA-augmentt experts:
instantiate single MLP, shared among all layers with that
h W base Dt +
Σ Pr. (A,,,a.)
Our framework also remains flexible: alternative MLP structures can be adopted, and we investigate a local variant in Appendix C.
3.2 TRAINING Loss
In this work, we adopt the standard cross-entropy loss for the Language Model (LM) in both next- token prediction and sequence classification tasks (Xue et al., 2024; Liao et al., 2025; Wu et al., 2024; Dou et al., 2024). Formally, this could be expressed
LIM =
Σ
M, log PLM(I; xes),
(8)
5

---

Published as a conference paper at ICLR 2026
In this formulation, X = (2'1 (........ .... In+m) denotes the concatenation of the input sequence and target sequence with length n and m respectively. M, E {0. 1 is a binary mask that specifies whether the i-th token contributes to the loss. In particular, M, 0 for tokens belonging to the input sequencefignored during optimization), and M; = 1 for tokens in the target sequence. This ensures that the model is trained to predict only the target tokens conditioned both the input prompt and the previously generated target tokens, while not penalizing predictions over the input context.
To further stabilize the training process, we incorporate the conventional load-balancing loss (Fedus et al., 2022; Yang et al., 2025; Dai et al., 2024), which mitigates the risk of routing collapse (Shazeer al., 2017). Such collapse can also arise LoRA- aoRA-augmented expert settings during fine-tuning, where only a few experts dominate the token assignments. Additionally, we introduce sparsity loss that leverages the closed-form nature of our routing to directly regulate the sparsity level. In the following, we present the mathematical formulation of both the load-bat had--aaancing loss and the proposed sparsity loss detail.
3.2.1 LOAD BALANCING LOSS
Given E experts indexed by i E and batch B with T = m tokens, the auxiliary loss is computed as the scaled dot-product between vectors P and P,
E Li = Σ F. P,
(9)
==1
where F; the fraction of tokens dispatched to expert i, and P, the fraction of the router proba- bility allocated for expert i,
F
Σ .EB
1TTkenn t selects Expert i), P. (x)) #EB
(10)
This objective encourages both F F= == and P= -PI...... PE) to approach a uniform distribution. In the ideal case of perfect balance, each expert receives equal share, i.e., F, P 1/E for all i, which minimizes Eq. 9. By penalizing concentration of both token assignments (F.) and router probabilities (P,) small subset of experts, this simple yet effective loss plays crucial role in ensuring stable and efficient MoE training.
3.2.2 CONTROLLING SPARSITY WITH SPARSITY LOsS
The proposed routing mechanism enables explicit control over sparsity via the predicted factor X. To achieve desired number of activated experts, we introduce sparsity loss that regularizes À toward values corresponding to the target sparsity level.
Suppose we aim for exactly k experts to be activated for a given token. From Proposition 1, this requires that the k-th largest score satisfies u(k) > т while the (A: + 1)-th largest score satisfies U(k+1) < This condition uniquely determines the target value range of À that yields & activated experts. We formalize this in Proposition 2, which gives an analytical range of À that yields exactly l activated experts.
Proposition 2 (k expert activation). Let f(x) = u E R. and let u()) 2 ... 2 u(F) be the sorted coordinates of u, with Up defined as Proposition Then exactly is experts are activated, ie..
P(t) 0, < k, and P(t) 0,i k,
(11)
if and only if the sparsity factor X lies in the interval
X - 1Ux
k ((())1))
(Up- - -ku())) 1sksE-1.
(12)
For k E, the condition reduces
X (- 1 (UE E E)))
(13)
Proof. The result follows by characterizing the threshold J in Proposition and enforcing the con- ditions u(K) > 2 u(k+1)- We provide the detailed derivation in Appendix B.

---

Published as a conference paper at ICLR 2026
From Proposition 2. we define (k) as the lower bound of the interval in Eq. 12. When the goal is to maintain the number of selected experts less than or equal to k, motivate À to remain within this interval (with only lower bound) during training by introducing a sparsity loss of the form:
Capase ReLU(O)owwr (k) X)
(14)
This loss penalizes À whenever it falls below the lower bound, while leaving it unchanged when À lies inside the feasible region.
Finally, altogether optimize the following total loss objective with two coefficients and 8 that are hyperparameters to control the relative contribution of auxiliary losses:
Liotal = LLM + a Lib + 3 Spaaee
(15)
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
We evaluate our method by incorporating the MoE structure with Sparsegen routing in the Mixture of LoRA Experts setting to finetune the base model on various common benchmarks.
Benchmarks and Metrics: We evaluate the overall accuracy of our method against several base- lines across range of downstream tasks. Specifically. we test on standard NLP benchmarks, in- cluding cnstructionfinetunnie ARC-Easy (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), CommonsenseQA (Talmor al., 2019), SWAG (Zellers et al., 2018), HellaSWAG (Zellers al., 2019), as well as sequence classification tasks from GLUE: CoLA, and RTE (Wang et al., 2019). For all benchmarks, we use standard accuracy as the evaluation metric. Please refer to Appendix E for more detail of the dataset and setup.
datasets such as MMLU-Pro (Wang al., 2024). ARC-Challenge
Base Model and Baselines: We test baseline approaches on different open-source L.MMs, including Llama-3.2-30 alamma-3.2-8b (Dubey et al., 2024) and Qwen3-1.7B (Yang et al., 2025). We compare our method primarily against MoLA (Gao et al., 2024), a TopK routing strategy within the MoLE framework, denoted as MoLA(8888). We also evaluate its proposed variant MoLA(2468) which assigns fewer experts to lower layers and progressively increases the allocation toward higher layers, reportedly yielding consistently better performance. In addition, the ReLUUbased routing from ReMoB (Wang et al., 2025) to the LoRA experts setting. Simliar to L22-MMLEE, ReMoLE supports both dynamic and differentiable routing.
include ReMoLE, which adapts
Implementation: For our method, training is conducted on 4 NVIDIA H200 GPUs with a hatch size of 16 for 10 epochs, with the learning rate of 0.0001 decayed by factor of 0.1 epochs 6 and 8. For Llama-3.2-8b. we run the experiment for only run 3 epochs as we observe convergence at the early stage of training. We set the number of LoRA experts to 8, with rank 8 and scaling factor 16, and apply a dropout rate of 0.1. Across all methods, we pair the training with the load- mala-cnnnn loss and follow the settings described in the original baseline papers. For MoLA, we choose the top 2 expert selections follows the original settings (Gao et al., 2024). For ReMoLE, we employ the mad-balaneing objective function introduced in CeMoF((Wang et al., 2025) with exact the same coefficients. More details of the method and experiment training setting are provided in Appendix D.
4.2 OVERALL PERFORMANCE
The overall performance of our proposed LD- D-MMEE is summarized in Table 1. Across all tested con- figurations, L.DMMLLE E achieves the highest average scores on both the Llama-3.2-3B, Llama-3.2-bb and Qwen3-1.7B models, demonstrating the consistent benefits of its learned dynamic routing. For this comparison, set a 1.0 and disable the sparsity loss (i.e., 3 0), defined in Eq. 15. A detailed analysis of sparsity control is deferred to Section 4.4.
In particular, LD-MoLE outperforms both fixed and dynamic routing baselines. Compared to the fixed TopK routing of MoLA, our method excels on sessoonnn-hhayyy benchmarks, achieving average cross-model gains of over +3.5% ARC-E, SWAG, and HellaSWAG, For MMLU-Pro LD-MoLE outperform the baselines in all testing models. On OpenBookQA, it achieves an average improve- ment of about +1.2%, and CommonsenseQA, it surpasses MoL.A by more than +2.0%. While
7

---

Published as a conference paper at ICLR 2026
Method
Model
P
ARC-C
ARC-E
Open
Commn
SWAG
CoLA
RTE
Avg
MoLA(8888)
Llama3.1-88
2.17
48.21
76.79
86.32
86.40
84.11
89.15
94,06
87.73
89.53
82.48
MoLA(2468)
Llama3.1.80
1.26
50.37
77.90
87.37
87.60
83.87
86.97
94,03
87.25
90.25
82.85
RcMoLE
Llama3.1-88
2.17
54.98
82.00
87.60
83.17
92.22
85.27
88.43
84.47
L.D.Mol.E.
2.33
55.98
83.67
83.01
92.29
85.25
91.32
85.18
MoLA(S888)
Llama3.22 2:33
3.11
40.31
71.57
83.51
81.00
79.77
83.56
87:47
85.81
90.61
78.18
MoLA((268
LIama3.2.33
1.80
42.31
71.91
83.86
83.60
80.02
83.96
87.31
86.00
89.53
78.72
ReMoLE
1lama3.2-3B
3.11
48.01
75.25
89.30
83,40
79.52
90.45
93.44
83,95
89.46
81.42
LD.MoLE
Llama3.2 3.3
3.28
49.58
74.58
81.42
91.37
86.02
88.38
82.05
MoL8(8888)
Qwen3-1.7B
4.12 %
51.12
76.59
88.60
82,40
76.49
84.11
83.35
83.89
86.65
79.24
MoL.N(2688
Qwen3-1.7B
2.39
49.96
76.92
88.42
83.00
75.84
84.17
87.09
83.60
84.48
79.28
ReMoLEF
Qwen3-1.7B
4.12
53.40
79.60
91.75
84.80
79.44
86.37
88.00
82.12
83.74
81.02
LD-MoLE
Qwen3-1.7B
4.23
53.82
78.67
85.00
79.30
86.72
82.61
87.72
81.63
Table I: Comparison between methods across downstream tasks.
MoLA attains slightly better results on certain sequence classification tasks such as RTE, LD-MoLE consistently delivers higher overall averages, underscoring the effectiveness of learnable dynamic routing across diverse task types. Compared to ReMoLE, LD-MoLE achieves higher overall av- erages, including +0.5% on Llama3-2.3B and +0.6% on Qwen3-1.7B. Notably, ReMolE exhibits large performance drops CoLA with Llama3-2.3B and RTE with Qwen3-1.7B, whereas LD- MoLE maintains stable effectiveness across benchmarks.
We observe that dynamic routing methods generally perform better on instruction fine-tuning tasks, while fixed routing approaches show slight advantages in certain sequence classification tasks. A possible explanation that classification tasks often benefit from more uniform expert usage. where fixed routing ensures stable allocation. Interestingly, the pruned variant MoLA(2468) outperforms the standard MoLA(8888), suggesting that many experts in the fixed routing setup are underuti- lized, introducing redundancy as also noted in their work (Gao et al., 2024). In contrast, dynamic routing adapts expert selection on a token-by-token basis, which benefits complex reasoning and cestructioo-followinn tasks but may introduce variability that is less advantageous for shorter classi- fication settings. Overall, LD-MoLE provides a stronger balance between parameter efficiency and performance, adapting better effectiveness across diverse tasks.
4.3 PREDICTED À FOR DYNAMIC EXPERT ALLOCATION
In this section, we compare the performance of the predicted against fixed À values to demonstrate that the shared learnable MIP structure proposed for the prediction À achieves superior results in the setting of LoRA experts. We conduct experiments with Qwen3-1.77 as the base model and report results in Table 2. We evaluate a range of fixed À values against our predicted ones. Recall our routing formulation (equation 2) and Proposition 1, the parameter À directly controls the sparsity of the probability distribution over LoRA experts.
We provide the visualization of X value distribution with 25-75 quantile range in Figure 3 for K projection, gate projection and down projection module. We observe that the distribution of À varies substantially between layers. The value increases in magnitude and exhibits greater variance at deeper layers and that fixed À cannot capture this depth-wise variance. Motivated by this observa- tion, we tested our predicted À against fixed values ranging from 0.5 to 100.0. The results show that the predicted À consistently outperforms all fixed settings, demonstrating its ability to dynamically adapts across both layers and tokens. Naturally, predicted À could flexibly adjusts without per-task or per-layer hyperparameter tuning thus yields improved performance against fixed
4.4 SPARSITY CONTROL ANALYSIS
In this section, we evaluate how the proposed sparsity loss influences the expert pattern and impacts task performance on 5 datasets. To encourage sparsity, we set the target number of activated experts to < 2. Results show that the sparsity loss effectively reduces the overall number of activated ex- perts. Increasing the sparsity alignment coefficient enforces a stronger constraint on the admissible range of À in Eq. 12. Moreover, Figure 4 illustrates the effect of applying the sparsity loss. Normally, more experts are activated in the lower layers, with activations gradually decreasing toward higher layers. Stronger regularization further suppresses higher-layer activations, while lower layers
8

---

Published as a conference paper at ICLR 2026
Krrgaction
Without Sparity loss
With Sparshty Loss
25-75 carrde
Gste Projection
20
25
10
25
20
23
2.0
Gdt Yearee
15
Down Po ection
Number
25-75
Indea
Figure 3: Layer-wise values for Figure 4: Average number of LoRA experts selected per token K, gate, and down projections.
across layers.
ARC-C
Open
Avg.
Caeff
ARC-F
Open
Avg.
0.5
77.92
91.93
82.60
78.13
87.72
83.66
1,0
76.25
90.88
82.60
77.15
88,69
83.11
-1.0
77.26
91.93
83.80
78.38
86.17
83.50
76.92
92.28
82.80
79.03
87.30
83.66
-10.0
78.71
83.35
0.01
91.40
Predicted
Table 2: Quantative result on different À values for sparsity loss (Qwen3-1.7B).
Table 3: Quantitative results on different coeffi- cient values for sparsity loss (Qwen3-1.7B).
main relatively dense. As shown in Table the results emphasize clear trade-off between sparsity and task performance: disabling the sparsity loss achieves the best average score, yet certain tasks benefit from reduced expert usage, suggesting that the optimal sparsity level is task-dependent. Our main contribution in this aspect is not simply confirming the sparsity and performance trade-off, but demonstrating that LD-MoLE makes sparsity both controllable and learnable within a dynamic rout- ing framework. This enables the number of activated experts to be reduced in principled way, while maintaining competitive performance. We also provide an additional experiment of computational analysis for this loss with respect to different hyperparameter /3 in Appendix C.3.
At the same time, excessive sparsity can degrade performance, as we observe performance drops in the later stages of training. This suggests that enforcing too much sparsity beyond certain point restricts flexibility expert usage across layers, ultimately limiting performance. Overall, maintain- ing a balance dynamic system appears most effective for improving efficiency without undermining model capability.
4.5 HARDER TOKEN NEED MORE EXPERTS
Dynamic routing allows the model to flexi- bly dedicate more capacity to tokens that re- quire richer representations while conserving resources more frequent predictable ones.
6.5
Average Active Expurt
6.0
As shown in Figure 5, tokens that frequently ap- pear during training (e.g., prompt- and context- related tokens) tend to activate fewer experts, effectively compressing their representations. In contrast, rarer or less familiar tokens acti- vate larger and more diverse set of experts, suggesting that tokens requiring greater model- ing capacity benefit from richer expert combi- nations. This behavior is consistent with obser- vations reported in ReMoE (Wang et al., 2025) for MoE pretraining. Overall, such adaptive routing enables the model to balance computa- tion efficiently across tokens, allocating more resources to informative ones.
activated expert rumber
4.0
-3.0
-2.5
Tokan Prequency Dog scale)
Figure 5: Correlation between the frequency of the top 200 most common tokens and their av- erage number of activated experts. Each seatter point represents the average number of experts ac- tivated for a given token.

---

Published as a conference paper at ICLR 2026
5 MORE ABLATION STUDIES
Ablation study zero-activation problem: In Appendix C.I, we provide detailed analysis of the zero-activation issue in dynamic routing. In particular, we show that ReMoLE can assign zero experts to token, which leads to degenerate representations, whereas our method guarantees at least one expert is activated through the closed-form Sparsegen routing.
Ablation study on expert patterns during training: In Appendix C.2, show that expert activa- tion patterns are largely established early in training and remain fixed thereafter.
Additional exploration of LD-MoLE: In Appendix C.3, present further experiments and dis- cussions on our method. We provide an alternative local MLP design and analyze the impact of varying hidden dimensions in the shared MLP. In addition, present experimental results effi- ciency.
CONCLUSION
In this work, introduce LD-MoLLE, learnable dynamic routing method for Mixture of LoRA Experts. Building on Sparsegen, our approach leverages a shared MIP to learn the sparsity param- eter enabling adaptive expert allocation across layers and tokens
parameter-efficient manner. Comprehensive experiments show that LD-MoLE achieves the highest average scores on both the Llama-3.2-3B, Llama-3.2-88 and Qwen3- 1.7B models compared strong baselines, including TopK routing and ReLU- routing, across range of instruction-tuniin and sequence classifi- cation tasks. For future research, we want to see how LD-MoLE performs the pretraining stages of LLMss with its differentiability and controllable sparsity. Furthermore, integrating our dynamic routing framework with other PEFT methods or extending its applicability to new domains, such as multi-modal models, presents exciting opportunities for future exploration.
7 ACKNOWLEDGEMENTS
The work of Yuan Zhuang and Fei Miao is partly supported by the National Science Foundation under Grants CNS-2047354 (CAREER). We also gratefully acknowledge the generous support of Qualcomm gift fund and our collaborator Jonathan Petit.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https::////..m. /ttpss////avvv../// arxiv.ii.v. org/ 10t.:/abb//11...../ 05457.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan. Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards Ultimate Expert Specialization in Mfxture-of-Expprts Language Models, January 2024.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE- Style Plugin, March 2024.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experrs In International conference on machine learning, pp. 5547- 5569. PMLR, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. CoRR, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(10)) 2022.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and V. S. Subrahmanian. Higher Layers Need More LoRA Experts, February 2024.
Yongxin Guo, Zhenglin Cheng, Xiaoying Tang. Zhaopeng Tu, and Tao Lin. Dynamic Mixture of Experts: Auto-Tuning Approach for Efficient Transformer Models, March 2025.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efticien Transfer Learning for NLP, June 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021.
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models, 2024. URL https: /ttass///axiiii /taxxvaivvv//////////aa/a/aaaev/ org/ abs 22003. 07652.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux. Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https:
bottps://arxiss http://aaxiv.vei /rg/ abs /01./ 04088.
Anirban Laha, Saneem A. Chemmengath, Priyanka Agrawal, Mitesh M. Khapra, Karthik Sankara- narayanan, and Harish G. Ramaswamy, On Controllable Sparse Alternatives to Softmax, October 2018.
11

---

Published as a conference paper at ICLR 2026
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding, 2020. URL 16668.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts, July 2024.
Mengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo, and Huaiyu Wan. Hmora: Making Ilms more effective with hierarchical mixture of lora experts. 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018. URL https:/ /arxiv.. org/abs/ /109. 02789.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From Sparse to Soft Mixtures of Experts, May 2024.
Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality, October 2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely- Eexture-of-Experse Layer, January 2017.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega:: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv... abs/1811 00937.
Meituan LongCat Team, Bayan, Bei Li, Bingye Lei, Bo Wang. Bolin Rong, Chao Wang. Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, Chengeheng Han, Chenguang Xi, Chi Zhang, Chong Peng, Chuan Qin, Chuyu Zhang, Cong Chen, Congkui Wang, Dan Ma, Daoru Pan, Defei Bu, Dengchang Zhao, Deyang Kong, Dishan Liu, Feiye Huo, Fengcun Li, Fubao Zhang, Gan Dong, Gang Liu, Gang Xu, Ge Li, Guoqiang Tan, Guoyuan Lin, Haihang Jing, Haomin Fu, Haoman Yan, Haoxing Wen, Haozhe Zhao, Hong Liu, Hongmei Shi, Hongyan Hao, Hongyin Tang, Huantian Lv. Hui Su, Jiacheng Li, Jiahao Liu, Jiahuan Li. Jiajun Yang, Jiaming Wang, Jian Yang. Jianchao Tan, Jiaqi Sun, Jiaqi Zhang. Jiawei Fu, Jiawei Yang, Jiaxi Hu, Jiayu Qin, Jingang Wang. Jiyuan He, Jun Kuang, Junhui Mei, Kai Liang, Ke He, Kefeng Zhang, Keheng Wang, Keqing He, Liang Gao, Liang Shi, Lianhui Ma, Lin Qiu, Lingiin Kong, Lingtong Si, Linkun Lyu, Linsen Guo, Liqi Yang. Lizhi Yan, Mai Xia, Man Gao, Manyuan Zhang, Meng Zhou, Mengxia Shen, Mingxiang Tuo. Mingyang Zhu, Peiguang Li, Peng Pei. Peng Zhao Pengeheng Jia, Pingwei Sun, Qi Gu, Qianyun Li, Qingyuan Li. Qiong Huang. Qiyuan Duan, Ran Meng. Rongxiang Weng, Ruichen Shao, Rumei Li, Shizhe Wu, Shuai Liang, Shuo Wang, Suogui Dang, Tao Fang, Tao Li. Tefeng Chen, Tianhao Bai, Tianhao Zhou, Tingwen Xie, Wei He, Wei Huang, Wei Liu, Wei Shi, Wei Wang, Wei Wu, Weikang Zhao, Wen Zan, Wenjie Shi, Xi Nan, Xi Su, Xiang Li, Xiang Mei, Xiangyang Ji, Xiangyu Xi, Xiangzhou Huang, Xiampeng Li, Xiao Fu, Xiao Liu, Xiao Wei, Xi- aodong Cai, Xiaolong Chen, Xiaoqing Liu, Xiaotong Li, Xiaowei Shi, Xiaoyu Li, Xili Wang, Xin Chen, Xing Hu, Xingyu Miao, Xinyan He, Xuemiao Zhang. Xueyuan Hao, Xuezhi Cao, Xunliang Cai, Xurui Yang, Yan Feng, Yang Bai, Yang Chen. Yang Yang. Yaqi Huo, Yerui Sun, Yifan Lu, Yifan Zhang, Yipeng Zang, Yitao Zhai, Yiyang Li, Yongjing Yin, Yongkang Lv, Yongwei Zhou, Yu Yang, Yuchen Xie, Yueqing Sun, Yuewen Zheng, Yuhuai Wei, Yulei Qian, Yunfan Liang, Yunfang Tai, Yunke Zhao, Zeyang Yu, Zhao Zhang, Zhaohua Yang. Zhenchao Zhang, Zhikang Xia, Zhiye Zou, Zhizhao Zeng, Zhongda Su, Zhuofan Chen, Zijian Zhang, Ziwen Wang. Zixu Jiang, Zizhe Zhao, Zongyu Wang, and Zunhai Su. Longcat-flash technical report, 2025. URL https://arxix: org/ alss 2509.500.0 01322.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https: // aarxiv v. org/ abs 404. 07461.
12

---

Published as a conference paper at ICLR 2026
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan. Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task task language understanding benchmark, 2024. URL https:::/// rxi v. org/ abs/ 24004 01574.
Ziteng Wang, Jun Zhu, and Jianfei Chen. ReMoE: Fully Differentiable Mixture-of-Expert with
ReLU Routing, February 2025.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of LoRA Experts, April 2024.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. OpenMoE: An Early Effort Open Mfxture-of- ooaeee--ff-t-ptttt Language Models, March 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, May 2025.
Tongtian Yue, Longteng Guo, Jie Cheng. Xuange Gao, and Jing Liu. Ada-k routing: Boosting the efficiency of moe-based Ilms, 2024. URL https: //ae////rxiiii org//abse http//bb//21..111
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning, 2023. URL https: arxiv. org/abs//////eo..o.../..a./a.. /200//3309 05444.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversar- ial dataset for grounded commonsense inference, 2018. URL 18C8. 05326.
otg/ abs/
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence?, 2019. URL https: / //axxiivvv..///////////////avavm www.vvoorg/aaaa htabbs/1995..7..
Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token- adaptive routing with null experts for mixture-of-experts language models, 2024. https:
/ www.ii.org//ae..eee htrrg/bba//eee aaa.//4666.. 13233.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- 10512.
efficient fine-tuning, 2023. URL htt.ttsss////xiii http://awww..ogga/b/////aa//////
Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully Differentiable Mixture- of of-Experts for Autoregressive Language Model Pre-training, August 2024.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le. James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https: //wpiv.r.ivvv///////////aa....iv org/ abg/abs///
08906.
13

---

Published as a conference paper at ICLR 2026
A
PROOF FOR LEMMA 1
Proof. When 1, the closed form of sparsegen is
U4
1=
where 7 is chosen that E least one term must be strictly positive. Hence the support S(u) = {i pi > 03 is nonempty and Ipplo 2 1.
si= Pj 1. Since each term is nonnegative and their sum equals 1,
Equivalently, using the support-size characterization, let u(1) 2 u(2) 2 ... 2 ((F) be the sorted coordinates and UR E M(s)- From Eq.
max { k E [E] 1- A- kuk) > Uk}.
For k 1 the inequality reduces one index is selected.
0, which holds when À 1. Thus h 2 1, so least
For the edge case À 1, the quadratic term vanishes and the objective reduces linear program:
P u. s.t. p 2 0, 1 1.
max
PERE
Its maximizer is any one-hot vector supported on arg max, Ug Again, |/p|lo = In all cases with A S 1, optimizer p is feasible (p 0, 1 = 1). A feasible vector on simplex cannot be identically zero, hence its support is nonempty.
B PROOF FOR PROPOSITION
Proof. From Proposition 1, the routing probabilities are
p: ((3)
where т defined in (equation 5). For exactly & experts to activated, we require
((k) > and u(k+1) S..
Substituting (cquation 5), the first inequality gives
Un 1
><1- - ku()))
(k)
k
Similarly, the second inequality gives
- 1+A
u(K+1) S
A21 (Us -k(++1))-
k
Combining the two inequalities, we obtain
A (Us - -ku((41)) 1 (Ux ku(())) )
which establishes (equation 12) for 1 S k < E 1. For the case E, only the condition u U(E) applies. Substituting again yields
UE 1
x<1 - (Ug- - E.)(
((F)
E
14

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENT RESULTS
C.1 THE ZERO-ACTIVATION ISSUE IN DYNAMIC ROUTING
A key challenge in designing dynamic and differentiable routing mechanisms is the possibility of zero activation, where a token is not assigned any expert. This problem occurs when the routing function produces zero outputs, leaving the token without activated expert. Such cases not only waste model capacity but also hinder gradient flow, making it difficult for the affected experts to learn meaningful representations.
This issue arises in activation- gating mechanisms such as ReLU-based routing, where the router may output all non-positive values for certain tokens. In practice, this leads to suboptimal expert utilization: some tokens receive no expert processing, while others may be redundantly as- signed. Figure 6 compares the expert activation patterns of LD-MoLE and ReMoLE. From Figure 6, ReMoLE shows a similar trend to LD-MoLE, activates more experts in the lower layers and fewer in the higher layers. However, its higher layers often fall below average 1.0 activated experts. This indicates that, for some tokens, the ReLUUbased router fails to activate any experts in the upper layers.
In contrast, our proposed L2D-MoLE framework guarantees at least one routing coefficient remains strictly positive for every token. This ensures that all tokens are processed by at least one expert, while still enabling dynamic and sparse expert allocation across layers.
K. projection
Down projectin
Big Expert Eigorr. Actvosoon
qwerd
opentonkoge
qpenkookg
Ang Expert Actiettin
Ramaa
openbockss
openbookas
Layer Incex
Layer Incek
Figure 6: The average expert activation for ReMoLE on K and Down Projection modules. The green and orange line indicate the activation pattern on OpenbookQA and ARC-Chanllenge dataset respecitively.
C.2 EXPERT PATTERN DURING TRAINING
In Fig. 7, we compare the expert activation patterns at the first and final training epochs. The trend described in Sec 4.4 which more experts are activated in the lower layers, with a gradual decrease toward the higher layers has already established by the end of the first epoch. The distribution re- mains largely consistent throughout training, as shown by the similarity between the heatmaps of routing ratio for Epoch 1 and Epoch 10. This indicates that routing specialization emerges very early and stabilizes quickly, leaving little room for substantial redistribution across layers as train- ing progresses. Such stability underscores the importance of the early training phase: the model rapidly learns how to allocate experts, and subsequent optimization primarily ine-tunes within this established structure rather than reshaping it.
15

---

Published as a conference paper at ICLR 2026
Epoch
Epoch 10
0.8
L10
111
-.66
Rousing Rotio
L12
L13
114
L15
116
117
110
119
120
121
122
0.2
123
124
24 Expert ID
87
E2
85
Expert ID
Figure 7: Comparision the routing ratio heatmap of the expert activation pattern between the epoch and epoch 10.
Method
Model
Open
HellaSWAG
Avg
Local
Llamm3.2.3B
3.13 %
73.67
89.65
83.80
81.59
91.29
93.50
84.28
89.70
85.93
Shared
Llama3.2-38
3.28 %
74.58
89.47
84.00
81.42
91.37
93.60
86.02
88.38
86.10
Local
Owen3-1.7B
85.88
Shared
Qwen3-1.7R
4.23 %
78.26
92.11
85.00
79.30
86.72
88.71
82.61
87.72
85.05
Table 4: Comparison between shared and local MLP structure for LD-MoLE.
C.3 ADDITIONAL EXPLORATION ON LD-MOLE
Shared vs Local MLP
In Sec. 4, we presented results using the shared MLP design for learning the parameter X. Here, investigate an alternative architecture in which, instead of instantiating one expert MLP per unique input dimension (as described in Sec. 3.1), assign dedicated MLP to every target module (i.e., Q K, V, O, Up, Gate, and Down projections). This design allows each module to learn its specialized routing strategy, which is intuitively reasonable since different modules process distinct types of information. However, this approach significantly increases the number of tunable parameters, as modern LLMs contain hundreds of such modules. To mitigate this overhead, restrict each local MLP to a single linear layer, such that f in Eq. 3 reduces to a weight matrix Wmip E 4 But still, unlike the shared MLP design, the trainable parameters of the local MLP structure would be associated with the layer number of pretrained transformer models.
We report the comparison between shared and local structures on Qwen3-1.7B and Llama3-2.3B in Table 4. Results show that the local design adds fewer additional parameters than the shared counter- part, but overall achieves weaker performance. While the local MI.P occasionally outperforms the shared version certain datasets, the gains marginal. This suggests that although local MLPs can individually learn X, its limited capacity that using only a single linear transformation hinders their ability fully capture the complexity of routing decisions.
Hidden Dimension in Shared MLP
In Sec. 4, we only one MLP per unique input dimension. For example, Qwen3-1.7B contains seven modules but only two distinct input sizes (2048 and 6144), so only two MLPs are required. We set the hidden size of the ML.ss 256 for Qwen3-1.7 1.7B and 512 for Llama-3.2 3.2-BB. Here, we provide
16

---

Published as a conference paper at ICLR 2026
Dimension
Model
TP
ARC-C
ARC-E
Open
Comm
RTE
Avg
Llama3.2-3B
3.15 %
73.91
88.77
82.20
81.51
90.28
83.33
256
Llama3.2-33
3.20 %
72.24
89.65
82.80
81.18
88.23
82,82
512
Llama3.2-33
3.28 %
74.58
89.47
84,00
81.42
88.38
83.57
128
Qwen3-1.78
4.17 %
76.59
91.75
84.00
79.46
86.68
83.66
256
Qwen3-1.77 1.7B
4.23 1
78.67
92.11
85,00
79.30
87.72
84.56
512
Qweni--1.78 1.3m3--1.77
4.34 16
77.59
91.58
83.40
79.54
88.23
84.07
Table 5: Comparison between different hidden dimension (128, 256 and 512) used in LD-MoLE MLP.
comprchensive results across five datasets using various hidden dimensions in Table 5. The results show that performance peaks at 256 for Qwen3-1.7B and 512 for Llama3.2-33 2-3B,, suggesting that each base model has optimal hidden dimension. A plausible explanation is the difference in input size across models. For Llama3.2-3B. the module dimensions are larger, requiring higher-capacity MLP (larger hidden dimension) effectively capture the meaningful information and relationships needed for routing. Conversely, for Qwen3-1.7B. 1.7,B, a smaller hidden dimension is sufficient, as overly large MLPs may introduce redundancy and lead diminishing returns. Therefore, selecting the hidden dimension should balance representation capacity, parameter efficiency, and generalization ability.
Additional Experiment for the Sparsity Loss:
this section, we compare the computational cost (in FLOPs) under different hyperparameter settings for our analytical loss function (Eq. 15). As shown in Table 6, the FLOP analysis further highlights the efficiency of the sparsity loss introduced in Section 3.2.2. As sparsity-loss coeffi cient 3 increases, the number of activated LoRA experts decreases, leading overall FLOPs. However, compared with conventional TopK and ReLU routing, the primary source of additional computation in method arises from the shared MLP used to predict each token's sparsity factor X. To further mitigate this overhead, a promising direction is to augment an additional dimension into the gating projection for generating À and we leave it to the future exploration.
notable reduction in
Qwen3-1.7B MoLA-88S8
MoLA-2468
ReMoLE
Ours(9
1.0) Ours( 0.1) Ourst 0.01) Ours(
MFLOPs
43
40
74
83
90
100
106
Table 6: Effective FLOPS (router LoRA experts) across different B parameter for the sparsity loss and different routing baseline. Backhone FLOOPs are excluded since they are identical across methods.
D HYPERPARAMETER AND TRAINING SETUP
To ensure fairness, we adopt a consistent parameter-tuning pipeline and apply identical prompts across all datasets and methods. For instruction-tun tunnng tasks, mask out the prefix and context, training only on the final answer tokens. For sequence classification tasks, since LLMs lack dedicated classification separator token, we omit the former and replace the latter with the end-of-sentence token to mark sentence boundaries.
Training Setup:
Hyperparameters: Table summarizes the hyperparameter configurations used across different routing methods. To ensure fairness, we keep most training settings consistent, including optimizer (AdamW), batch size (16), and number of epochs (10). All methods are trained with LoRA rank r 8, scaling factor a 16, and experts. We also apply the method on all the target modules (i.e.. Q. K. V, O, Up, Gate, and Down projections). For optimization, we adopt different learning rate schedules to align with prior works. Both LD and ReMoLE use the MultiStepLR scheduler with an initial learning rate of 1 x 10-4, while MoLA follows its original implementation with cosine annealing and a slightly higher learning rate (3 10-4). This setup provides a balanced comparison by respecting the design choices of each baseline while maintaining comparable training stability. Dropout applied to mitigate overfitting. LD-MoL.E and ReMoL. EE dropout of 0.1,
17

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameters used for different methods.
Method
LD-MoLLE
ReMoLE
MoL.A
Cutoff Length
1024
1024
1024
Learning Rate
le-4
le-4
3e-4
scheduler
MultiStepL.
MultiStepL.
Cnnenl
Optimizer
AdamW
A
AdamW
Batch size
16
16
Dropout
0.1
0.1
0.05
Epochs
10
10
10
Target Modulcs
All
All
All
Routing type
Dynamic
Dynamic
Fixed
LoRA Rank
8
8
LoRA Alpha
16
Experts
8
8
TopK
whereas MoLA uses 0.05, again consistent with its reported configuration. The cutoff length for all experiments is fixed 1024 to ensure uniform input context across models.
E DATASET INFORMATION
In this section, we provide additional details about the datasets and experimental setup. Each dataset is divided into three splits: training, validation, and test. Our experiments are conducted by training on the training split and evaluating on the validation split, without using the test split.
ARC (AI2 Reasoning Challenge): ARC is a benchmark of grade-school level science ques- tions with 4 choices, divided into two subsets: ARC-Easy, which consists of relatively straightfor- ward questions, and ARC-Challenge which requires more complex reasoning and deeper scientific knowledge. For ARC-Easy, there are 2251 samples in train split, 570 samples in validation split and 2376 samples in test splits. For ARC-Challenge, there are 1119 samples in train split, 299 samples in validation split and 1172 samples in test splits.
CommonsenseQA is multiple-choice question answering dataset with 5 choices that evaluates a model's ability to apply various forms of commonsense knowledge. consists of 12,102 questions, each with one correct answer and four distractors. There are 9741 samples in train split, 1221 samples in validation split and 1140 samples test splits.
CommonsenseQA::
OpenBookQA is designed advance research in complex question answering with 4 choices by evaluating both scientific knowledge and language understanding. The dataset is mod- eled after open-book exams: it provides a collection of scientific facts that must be combined with broader commonsense knowledge to answer multiplecchoice questions. Unlike simple fact- retrieval tasks, OA emphasizes multi-step reasoning. integration of external knowledge, and deeper text comprehension. There are 4957 samples in train split, 500 samples in validation split and 500 samples in test splits.
OpenBookQA:
This benchmark evaluates commonsense reasoning by asking the model to predict the most plausible continuation of given scenario. Each instance formulated as 4-way multiple-choice question, with one correct answer and three adversarially generated distractors. There are 73546 samples in train split, 20006 samples in validation split and 20005 samples test splits.
SWAG:
It's designed to evaluate a model's ability to complete sentences in coherent and contextually appropriate way. Similar to SWAG, each examples has 4 options or candidate endings, where the task is to select the most plausible continuation. The challenge lies in the fact that suc- cess requires more than recognizing surface-level word patterns it demands an understanding of meaning, context, and commonsense reasoning. While this task is trivial for humans with extensive real-world and linguistic experience, remains a significant hurdle for machines. There are 39900 samples in train split, 10000 samples in validation split and 10000 test samples.
HellaSWAG:
18

---

Published as a conference paper at ICLR 2026
MMLU-Pro:
is designed to evaluate model's multi-domain knowledge understanding and com- plex reasoning abilities. Compared to MMLU, MMLU-Pro introduces more challenging, reasoning- focused questions and expands the answer choices from 4 to 10 options, while removing trivial and noisy items. Models using Chain-of- Thought reasoning perform better, reflecting the benchmark's emphasis deeper reasoning. The dataset contains 12,102 samples.
Corpu of Linguistic Acceptability): It's part of the General Language Understanding Evaluation benchmark and it consists of 10,657 sentences drawn from 23 linguisties pub- lications, each annotated for grammatical acceptability by the original authors. The public release includes 9,594 sentences for training and development, while 1,063 test sentences held out.
( Textual Entailment): It's part of the General Language Understanding Evalua- tion(GLUE) benchmark is consist of series of annual entailment challenges. Examples drawn from news and Wikipedia text. All datasets are converted into two-class classification: entailment vs. not entailment, containing 2,490 training, 277 validation and 3000 test samples.
19


Reviews:
[{"review_id": "HabOsB21Zd", "reviewer": "Reviewer_5J1o", "summary": "The paper proposes LD-MoLE, a learnable dynamic routing mechanism for Mixture of LoRA Experts. Built on Sparsegen, the method introduces a differentiable and token-wise adaptive expert allocation strategy, with a learnable sparsity parameter predicted by a shared MLP. The framework also includes a sparsity control objective to regulate the number of activated experts. Experiments across several downstream tasks show improved performance compared to TopK and ReLU-based routing baselines.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a clear motivation for improving expert allocation in MoE models and proposes a differentiable routing mechanism with controllable sparsity. The method is well-integrated into the LoRA-based MoE setting, and the overall presentation is clear. Experiments cover multiple tasks and include comparisons against relevant baselines, along with some ablation studies.", "weaknesses": "* **Applicability to Standard MoE Architectures**: While LD-MoLE is instantiated within the MoLE setting, it remains unclear whether the proposed routing mechanism can be seamlessly integrated into conventional MoE architectures with full FFN experts or applied during pretraining. Clarifying its plug-and-play compatibility with standard MoE training and inference pipelines would help better demonstrate the generality and practical advantages of the approach.\n\n* **Insufficient Efficiency Evidence**: The paper claims that LD-MoLE effectively reduces the number of activated experts, but the evaluation largely reports task accuracy and qualitative activation plots. To substantiate efficiency, more concrete system metrics (e.g., FLOPs, latency, throughput and peak memory) would better support the conclusion that reduced activation translates into real compute savings.\n\n* **Missing Discussion Regarding Sparse Routing.**: the paper lacks comparison with prior methods that target similar goals through dynamic routing or explicit sparsity control [1-3]. A clearer conceptual and empirical comparison with related works would strengthen the contribution.\n\n[1] Team M L C, Li B, Lei B, et al. Longcat-flash technical report[J]. arXiv preprint arXiv:2509.01322, 2025.\n\n[2] Zeng Z, Miao Y, Gao H, et al. Adamoe: Token-adaptive routing with null experts for mixture-of-experts language models[J]. arXiv \npreprint arXiv:2406.13233, 2024.\n\n[3] Yue T, Guo L, Cheng J, et al. Ada-k routing: Boosting the efficiency of moe-based llms[C]//The Thirteenth International Conference on Learning Representations. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v8KvTZ3Djm", "reviewer": "Reviewer_Ckcz", "summary": "This paper proposes LD-MoLE, a novel **Learnable Dynamic** routing mechanism for **M**ixture **o**f **L**oRA **E**xperts. The method addresses key limitations in existing MoLE approaches by replacing the non-differentiable Top-K routing with a fully differentiable routing function based on Sparsegen projection. LD-MoLE introduces a lightweight shared MLP that predicts a token- and layer-dependent sparsity parameter (λ), enabling adaptive expert allocation. An analytical sparsity loss is derived to explicitly control the number of activated experts. Extensive experiments on Llama-3.2-3B and Qwen3-1.7B demonstrate state-of-the-art performance across diverse benchmarks, outperforming strong baselines like MoLA (Top-K) and ReMoLE (ReLU-based routing).", "questions": "1. The shared MLP predicts λ for the same module type across all layers. Does this design potentially limit the model's ability to learn highly specialized, layer-specific routing strategies? What is the trade-off between parameter efficiency and layer-wise flexibility here?\n\n2. In the sparsity control analysis (Sec. 4.4), using the sparsity loss (β>0) reduces expert count at a performance cost. Do you see a pathway for the model to learn a task-optimal sparsity level automatically, rather than relying on a pre-defined k?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The idea of combining Sparsegen-based differentiable routing with a learned λ-predicting MLP is novel and impactful. It elegantly solves the non-differentiability of Top-K and the instability of ReLU routing, advancing the MoLE paradigm.\n- Theoretical grounding such as closed-form solutions, proofs is solid. Experiments are comprehensive, covering multiple models, tasks, and ablation studies including sparsity control, λ analysis and zero-activation issue.\n- The paper is well-structured, clearly written, and figures effectively illustrate the methodology and findings.\n- LD-MoLE provides a principled, efficient, and high-performing framework for dynamic expert routing, with potential influence on both PEFT and MoE research communities.", "weaknesses": "- **Computational Overhead**: While parameter-efficient, the computational cost (e.g., latency, FLOPs) of the dynamic routing mechanism—especially the Sparsegen projection—compared to simple Top-K is not quantified. A brief analysis would strengthen the efficiency claim.\n- **Baseline Comparison**: Including a fixed-λ Sparsegen baseline would more directly isolate the contribution of the learned λ (via the MLP) from the contribution of Sparsegen itself.\n- **Hyperparameter Sensitivity**: The impact of the new hyperparameters (α, β, and target sparsity k) on performance and stability is not discussed. A sensitivity analysis would improve practical usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fB8gOYhgGb", "reviewer": "Reviewer_GWxU", "summary": "This paper proposes LD-MoLE, a learnable dynamic routing framework for Mixture of LoRA Experts (MoLE) that replaces the traditional non-differentiable Top-K selection with a Sparsegen-based routing mechanism. The model adaptively determines the number of activated experts per token and per layer through a shared MLP predicting a sparsity factor λ, allowing token-dependent and layer-wise routing. The authors also introduce an analytical sparsity control loss to regularize the number of activated experts. Extensive experiments on Llama-3.2-3B and Qwen-3-1.7B show consistent improvements over MoLA (Top-K routing) and ReMoLE (ReLU-based routing) across multiple NLP benchmarks. The results demonstrate improved efficiency, stability, and interpretability of dynamic routing in parameter-efficient fine-tuning.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Novel Differentiable Routing Mechanism:** The paper presents a clean and theoretically grounded alternative to Top-K routing using Sparsegen, providing both differentiability and guaranteed non-zero expert activation, addressing stability issues observed in prior dynamic routing work (e.g., ReMoE).\n\n**Comprehensive Analysis:** The authors provide solid theoretical derivations, ablations on sparsity and λ prediction, and empirical validation on two base LLMs across 8 diverse benchmarks, clearly showing robustness and consistency.\n\n**Strong Practical Value:** The design achieves dynamic, token-aware expert selection without increasing parameter count significantly (thanks to the shared MLP), making it practical for real-world PEFT and instruction-tuning applications.", "weaknesses": "1. **Gains are modest and transferability isn’t established.**\nReported improvements over Top-K/RELU routers (e.g., MoLA, ReMoLE) are incremental on small–mid LLMs and a limited task suite; the paper doesn’t convincingly show zero-shot / cross-dataset transfer (e.g., to harder or different domains such as math/code/MMLU) or robustness under distribution shift. Including a brief analysis (no extra training) of cross-dataset generalization would better support the method’s claims. \n\n2. **No scale-up evidence (3B→8B→30B+) and limited backbone diversity.**\nExperiments focus on small models (e.g., 1.7B–3B), leaving open whether learnable sparsity remains stable/efficient as parameters and depth grow (routing entropy, expert utilization balance, convergence speed, memory/latency). By contrast, scalable MoE work (e.g., Switch Transformer, DeepSeek-V3) validates routing and load balancing at tens–hundreds of billions of parameters; LoRA-MoE baselines like MixLoRA also report on a broader set of bases (e.g., LLaMA/Gemma/Mistral), aiding claims of generality. Adding at least partial scaling curves and results on a more diverse set of backbones would strengthen the paper.\n\n3. **Why choose Sparsegen over other differentiable routers isn’t empirically settled.**\nTheoretical appeal aside (deterministic, truly sparse, λ-controllable), the paper lacks a side-by-side comparison under the same budget against other differentiable routing families widely used in practice:\n– Soft routing + Top-K variants from large-scale MoE (e.g., Switch/DeepSeek-V3 and modern aux-loss-free balancing);\n– Gumbel-Softmax/Concrete reparameterization for differentiable categorical choices;\n– ReLU-based differentiable routers.\nA controlled ablation (same base/rank/FLOPs) on accuracy, stability, load balance, latency, and activated-expert count would clarify whether Sparsegen offers practical advantages beyond its theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpFgEQiyBu", "reviewer": "Reviewer_oEkd", "summary": "This paper introduces LD-MoLE, a Mixture-of-LoRA-Experts framework with learnable dynamic routing based on a closed-form Sparsegen function. A shared MLP predicts a token-wise sparsity factor λ , allowing the model to dynamically adjust the number of activated experts per token in a differentiable and stable manner. The method also includes an analytical sparsity loss for expert control.", "questions": "See the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality - The paper proposes a Sparsegen-based dynamic routing method with a mathematically proven sparsity loss, showing novelty and clear motivation.\n\n2. Quality and Clarity – The methodology is well-designed and theoretically sound, with clear writing that makes the approach easy to follow.\n\n3. Significance – The visualization of LoRA experts selected per token clearly demonstrates the effectiveness of the proposed dynamic routing mechanism.", "weaknesses": "1. Insufficient Ablation Studies and Experiments\nThe paper lacks an ablation study on the load balancing loss weight and its impact on the model's performance. It would be beneficial to explore how varying this parameter affects the stability and performance of the model, especially since load balancing plays a crucial role in expert distribution.  \nThe paper also does not compare against important baselines like:  \n[1] Harder Task Needs More Experts: Dynamic Routing in MoE Models  \n[2] HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts Including these baselines would provide a more comprehensive evaluation and highlight the specific advantages of the proposed approach.\n\n2. Lack of Large-Scale Model Comparison\nThe experiments are limited to relatively smaller models (Llama-3.2-3B and Qwen-3-1.7B). A comparison against larger-scale models would help validate the scalability of the proposed method and its ability to handle more complex tasks. This would provide a clearer picture of how the method performs in more resource-intensive settings.\n\n3. Unconventional Benchmark Selection\nThe choice of experimental benchmarks seems somewhat unusual. The paper compares against a small subset of tasks, whereas other multi-task MoLE baselines (e.g., [3] KASA: Knowledge-Aware Singular-Value Adaptation of Large Language Models and [4] MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning) involve a broader range of tasks. The rationale behind selecting this specific subset is unclear, and a more diverse set of benchmarks could provide a more robust evaluation. A clarification on why this subset was chosen would improve the overall experiment design.\n\n4. Unclear Necessity of Sparsegen Routing\nThe relationship between Sparsegen routing and the Mixture-of-Experts (MoE) architecture is not fully explained. It is unclear why Sparsegen routing was specifically chosen over other potential methods. Has the paper explored alternatives like Gumbel-Softmax or directly using softmax without Top-K selection? These methods could provide valuable insights and potentially simplify the routing process. A discussion on why Sparsegen is preferred, and a comparison to other discrete or continuous routing methods, would strengthen the theoretical justification for this choice.\n\n5. Computational cost\nThe paper lacks a quantitative evaluation of the additional computational cost introduced by the Sparsegen routing (such as the sorting operation and MLP prediction), and does not provide direct comparisons with baseline methods in terms of training/inference time, FLOPs, or latency.\n\nReferences:  \n[1] Harder Task Needs More Experts: Dynamic Routing in MoE Models.  \n[2] HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts.  \n[3] KASA: Knowledge-Aware Singular-Value Adaptation of Large Language Models.  \n[4] MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)