HiMoLE: Towards OOD-Robust LoRA via Hierarchical Mixture of Experts
Authors: Yinuo Jiang
Yan Xiaodong
Keyan Ding
Deng Zhao
Lei Liang
Qiang Zhang
Huajun Chen
Venue: NeurIPS 2025 poster ()
Keywords: OOD Robustness
Large Language Models
Parameter-Efficient Fine-Tuning
Mixture of Experts.
Primary area: deep_learning

Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have enabled the efficient adaptation of large language models (LLMs) by updating only a small subset of parameters. However, their robustness under out-of-distribution (OOD) conditions remains insufficiently studied. In this paper, we identify the limitations of conventional LoRA in handling distributional shifts and propose $\textbf{HiMoLE}$($\textbf{Hi}$erarchical $\textbf{M}$ixture of $\textbf{L}$oRA $\textbf{E}$xperts), a new framework designed to improve OOD generalization. HiMoLE integrates hierarchical expert modules and hierarchical routing strategies into the LoRA architecture and introduces a two-phase training procedure enhanced by a diversity-driven loss. This design mitigates negative transfer and promotes effective knowledge adaptation across diverse data distributions. We evaluate HiMoLE on three representative tasks in natural language processing. Experimental results evidence that HiMoLE consistently outperforms existing LoRA-based approaches, significantly reducing performance degradation on OOD data while improving in-distribution performance. Our work bridges the gap between parameter efficiency and distributional robustness, advancing the practical deployment of LLMs in real-world applications.

HiMoLE: Towards OOD-Robust LoRA via Hierarchical Mixture of Experts
Yinuo Jiang Yan Xiaodong", Keyan Ding², Deng Zhao
Lei Liang .Qiagg Zhang Huajun Chen
CColeee of Computer Science and Technology, Zhejiang University >ZJU-Ant Group Joint Lab of Knowledge Graph
"ZJU-hang Global Scientific and Technological Innovation Center, Zhejiang University *ZJU-UIUC Institute, Zhejiang University AAt Group
{qiang. zhang. ca, huajunsir (zjjs edu.
Abstract
Parameter-efficient fine-tuning (PEFT) methods, such LoRA, have enabled the efficient adaptation large language models (LLMs) by updating only a small subset of parameters. However, their robustness under out-of-distribution (OOD) conditions remains insufficiently studied. this paper, we identify the limitations of conventional LoRA in handling distributional shifts and propose HiMoLE (Hierarchical Mixture of LoRA Experts), a new framework designed to improve OOD generalization. HiMoLE integrates hierarchical expert modules and hierarchical routing strategies into the LoRA architecture and introduces a two- phase training procedure enhanced by diversity-drie -rivven loss. This design mitigates negative transfer and promotes effective knowledge adaptation across diverse data distributions. We evaluate HiMoLE on three representative tasks in natural language processing. Experimental results evidence that HiMoLE consistently outperforms existing LoRA-based approaches, significantly reducing performance degradation on OOD data while improving in-distribution performance. Our work bridges the gap between parameter efficiency and distributional robustness, advancing the practical deployment of LLMs in real-world applications.
Introduction
Large language models (LLMs) have brought transformative advances across a wide range domains. However, their unprecedented scale incurs substantial computational and storage costs. To address this issue, parameter-efficienl fine-tuning (PEFT) techniques, such as LoRA [1], have emerged as practical solutions. By updating only small subset of model parameters. PEFT methods reduce storage and computational requirements while achieving performance comparable to full-model fine-tuning. This efficiency makes them especially attractive for
deployments.
Despite these advantages, PEFT methods face a critical shortcoming: limited generalization under out-of mut-of-distribution (OOD) conditions. While deep learning models typically perform well in- distribution (ID) data, their performance often degrades when faced with data that deviates from the training distribution 2 3 This issue persists in L.MMs after full fine-tuning and is particularly pronounced in domains characterized by high heterogeneity
5 such as biomedicine and the
*Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.0
NER
1.0
SA
1.0
EQA
E D
000
Rade
OCO
Performance
BLA
B.W
0.2
0.2
0.2
1022
(a)
[b]
(c)
(d)
Figure 1: Robustness analysis of parameter-efficient fine-tuning, (a)-c)); In-distribution (ID) and out-of-distribution (OOD) results for the Base, LoRA, and MixLoRA models on three representative tasks: Named Entity Recognition (NER) in biomedicine, Sentiment Analysis (SA) in social science, and Extractive Question Answering (EQA) in general domain. (d): Impact of routing granularity in Beaiurr-of-LooR-----ee
where "OOD" refers to an OOD validation dataset, while "IDI" "ID2" "ID3" and "ID4" represent the four ID validation datasets. Token-level routing yields better ID performance but fails to generalize to OOD data. In contrast, sentence-level routing improves OOD robustness at the cost of ID accuracy.
social sciences. Surprisingly, although LoRA and related PEFT methods have gained widespread adoption, their robustness distributional shifts remains largely unexplored. Our empirical analysis (Fig. [())(()) reveals that standard LoRA suffers considerable accuracy drops when applied to tasks requiring adaptation across diverse knowledge domains. These findings suggest intrinsic limitations in LoRA's ability to generalize beyond the training distribution, motivating the need for more robust PEFT strategies.
Mixture-of mixture-of-parameete parrameter-fffiiiett--xpe (MoPE) methods attempt improve generalization by integrat- ing the Mfxture-of-Xxpaee
(MoE) framework with PEFT, demonstrating effectiveness in multi-task settings 6 7 8 9. However, their effectiveness under OOD conditions remains underexplored. As shown in Fig. [b),,), both LoRA and MixLoRA suffer notable performance degradation in knowledge-intensive tasks (c.g., Named Entity Recognition (NER) in biomedicine and Sentiment Analysis (SA) in social science) when evaluated OOD data. As shown in Fig. Tc). in general- domain tasks (e.g., Extractive Question Answering (EQA)), the gap between ID and OOD perfor- mance persists, revealing limited robustness. In some cases, MixLoRA even underperforms standard LoRA, suggesting potential overfitting. We identify a key source of this limitation: token-level routing in MoPE models is prone to expert misallocation under distributional shift. Local leven features often fail to capture the global semantics necessary for robust generalization, resulting in brittle routing decisions in unseen contexts. These observations motivate the central question of study: How can parameter-efficient fine-tuning methods be improved to enhance in-distribution performance while also ensuring robustness to distrr-distribution data?
To address this issue, we propose HiMoLE (Hierarchical Mixture of LoRA Experts), a novel framework that introduces structural sparsity and hierarchical design into the LoRA architecture. HiMoLE extends conventional MoPE models via the hierarchical architecture which manifests in two dimensions: hierarchical expert design and hierarchical routing strategy. To further improve knowledge utilization and reduce redundancy, we introduce a two-phase training scheme augmented diversity-promoting promoting loss. In summary, work makes the following key contributions:
with diversity
Empirical diagnosis of OOD limitations in LoRA. We systematically investigate the gen- eralization performance of LoRA under distributional shift, revealing significant weaknesses in its ability transfer across heterogeneous domains.
A novel hierarchical MoPE framework. We propose HiMoLE, which introduces hierar- chical expert architectures and routing strategies into the PEFT paradigm. This structure mitigates negative transfer and promotes positive transfer, offering new direction for improving PEFT robustness under distributional shift.
Theoretical and empirical validation. We provide theoretical insights into the advantages of hierarchical routing under distributional shift. Experiments across multiple domains show HiMoLE improves OOD generalization while maintaining strong ID performance.

---

2 Background and Related Work
2.1 OOD Generalization
Out-of-distribute generalization is essential for deploying language models in real-world scenarios, where data distributions are inherently diverse, non-stationary, and unpredictable [10] [i]. This need especially pronounced in high-stakes domains such as clinical decision support and social science analytics, where knowledge continuously evolves and data often deviate from training distributions. In such contexts, models must demonstrate robustness emergent semantic patterns, novel entity relationships, and shifting contextual dependencies. Despite the remarkable progress of large language models across a wide range of tasks and benchmarks, recent studies [4] [2 [3] have revealed significant vulnerabilities under distributional shifts. These findings expose the limitations of current one-tuning strategies and underscore the urgent need for methods explicitly designed to enhance OOD robustness.
2.2 Mixture of Parameter-efffiient Experts
Parameter-Eflicient Fine-Tuning (PEFT) As the scale of LLMs continues to grow, PEFT has emerged as practical and cost-effective adaptation strategy. PEFT techniques update only a small subset of model parameters such as adapter layers or low-rank matrices-wiie -wile keeping the majority of the pre-trained model frozen [14 15 [6]. widely adopted PEFT method is LoRA [ which inserts trainable low-rank adapters into pre- pre-trained layers and updates them during fine-tuning. LoRA achieves competitive performance with a substantially reduced memory footprint. However, PEFT often struggles to generalize across new distributions. The limited number of trainable parameters can restrict the model's capacity to adapt to distributional shifts or novel task requirements.
Integrating MoE with PEFT (MoPE) To reconcile scalability and efficiency, recent work proposes integrating MoE with PEFT techniques, resulting in the MoPE paradigm. In MoPE, each expert is instantiated using a PEFT configuration (e.g., LoRA), and a routing module dynamically assigns inputs to appropriate experts. This hybrid design seeks to combine the modular adaptability of MoE with the resource efficiency of PEFT. MoPE methods differ routing granularity. Token-level routing operates at the sub-sentence level. For example, MixLoRA (6] combines multiple LoRA experts with a shared FFN and incorporates an auxiliary load balancing loss to mitigate expert usage imbalance. LoRAMoE [7 employs router network to reduce knowledge forgetting. HydraLoRA [ adopts an asymmetric architecture with shared LoRA A matrix and expert-specific B matrices. In contrast, sentence-level routing mechanisms operate at the input sentence level. MOELoRA [[8] performs explicit task- mask-to-exper to-expert assignment using task metadata, deterministically routing input sentences based task identifiers. MOCLE [7] clusters instruction semanties and activates task-specific experts by assigning inputs to their corresponding instruction cluster. Although MoPE frameworks have demonstrated effectiveness in multi-task and in-distribution settings, their robustness under distributional shifts remains largely unexplored. This motivates the need for architectures specifically designed to handle complex domain shifts and enhance OOD generalization.
Method
3.1 Preliminaries
Formulation of OOD Generalization. Let denote the input data and y the output. Out-of- distribution generalization refers to scenarios where the test distribution (es( y) differs from the training distribution Parin(2: (x. y). while preserving core semantic relationships. This work focuses the joint occurrence of two distributional shifts [19:
1. Covariate Shift: The input distribution changes (Prea(()) * Pinii (x)), but the conditional distribution remains invarian((Pessl æ) = Prain (y x)).
2. Concept Shift: The input-conditional distribution changes( Pies (y x) + Pumin (y) x) ). which may arise from label semanties or task definitions evolving across domains.
Robust generalization this setting requires models to: (1) handle divergent input distributions (covariate shift), and (2) adapt to latent conceptual variations (concept shift).
3

---

Identification of OOD Generalization Problem in PEFT Following the protocol established
by 4l. we consider three key criteria for identifying OOD data: (1) diverse data sources, (2) low SimCSE similarity [20] with the in-distribution dataset, and (3) measurable performance degradation in models. However, in practice, we find that the second criterion-low SimCSE similarity not always reliable, especially in knowledge-deng, domains. Fine-tuned LLMs can still generalize effectively even when SimCSE scores are low. As such, we primarily rely on criteria (1) and (3) in our OOD dataset selection. Here, performance degradation refers to reduced model performance on OOD data relative to ID data, which can manifest in two ways: (1) the model performs worse than its pre-finetuunee counterpart, and (2) the performance gain OOD data is substantially smaller than that on ID data. We fine-tune models using LoRA on each task's 's ID dataset and evaluate them on both ID and OOD test sets. As shown in Fig. (c), LoRA fails to significantly enhance OOD robustness. To further illustrate this, we analyze failure cases in a biomedical NER task. The fine-tuned model demonstrates two main types of errors in OOD data: (1) failure to produce outputs in the correct structured format, and (2) mislabeling of general-domain entities for instance, tagging symptoms as diseases (see Appendix A for detailed case studies).
Probing OOD Robustness in MoPE Prior research 21 22 neural network optimization
has shown that sparsely activated architectures often generalize better than dense ones, owing to dynamic parameter specialization and reduced task interference. In LLMs, the MoE framework embodies this principle by leveraging conditional computation through expert routing, enabling scalable and efficient learning. Building on these insights, we investigate the OOD robustness of various MoPE configurations, focusing on token-level and sentence- routing strategies. As illustrated in Fig. d)), our results highlight a key trade-off between routing granularities: Token-level routing attains state-of-the-art performance on ID data by exploiting fine-grained contextual cues but exhibits pronounced OOD performance degradation, suggesting overfitting to surface-level patterns. Sentence-lever routing offers improved OOD robustness by aligning with global semantics but suffers from reduced ID performance due its limited sensitivity to local details. This finding highlights the necessity of developing a unified approach that can effectively integrate both routing granularities, achieving high ID accuracy while maintaining strong generalization across OOD scenarios.
3.2 Architecture of HiMoLE
In this subsection, we introduce HiMoLE, a Hicrarchical Mixture of LoRA Experts model designed to flexibly address the OOD robustness challenges inherent in LoRA- fine-tuning, An overview of the HiMoLE architecture is shown in Fig. 2 a).
Hierarchical Experts Our parameter-efffcient expert architecture is organized into N Knowledge Competition Groups (KCGs), each consisting of M Knowledge Collaboration Experts (KCEs). Since domains with heterogeneous knowledge comprise multiple distinct subdomains, we assign each KCG to unique subdomain and initialize its parameters by fine-tuning on the corresponding sub-dataset (as detailed in Section 3.3 Training Strategy). All KCEs within a KCG share the same initialization, providing a consistent Toundation of subdomain-specific knowledge. Each KCE is implemented as a LoRA module, formally defined
E = B-BBA, A A ER Xr B E Rr < min(dmn dow)).
(I)
The experts interact through three distinct modes. (1) Intra-group collaboration: KCEs within the same KCG specialize in subdomain- specific patterns while leveraging shared knowledge, enabling efficient adaptation and positive transfer. (2) Cross-group competition: Different KCGs compete to route tokens to the most relevant KCEs, thereby reducing interference across subdomains and mitigating negative transfer. (3) Cross-group collaboration: KCEs across different KCGs may cooperate to improve generalization, promoting knowledge reuse and transferability. This structured interplay between competition and collaboration ensures both specialization and synergistic learning across knowledge boundaries.
Hierarchical Routing Strategy The core of HiMoLE lies in its hierarchical routing strategy, which integrates sentence-level and token-level expert selection to achieve domain-awar and adaptive inference. For given input sentence, we first compute a sentence-level representation denoted by hegn via applying average pooling over the token-level hidden states heken- This pooled representation

---

(a) Architecture of HiMoLE
HiMoLE
Hortonlienl
Exparts
m888
Heevorohccal Smatogy
FFN
Router
Sentence Roulet
Eajezi donain
Averago Poeling
Feature
Sentence Festure
(b) Training Strategy
Stage
Stage 2
XCG
Sinilant
88
Freedder
Ilarschiil
Data
Data N
Train KCGs suparatcly on Uheir
Jointly optimize the experts and the compleic datasct
Embeddings
Clastering
comesponndnng
Figure 2: Illustration of the proposed HiMoLE. (a) Architecture of HiMoLE. Unlike MoLE, HiMoLE combines hierarchical experts organized Knowledge Competition Groups (KCGs) and their internal Knowledge Collaboration Experts (KCEs) wiith a hicrarchical routing strategy that performs sentence-level coarse allocation followed by token-level refinement. This architecture is designed to enhance OOD robustness in L-RA-b fine-tuning.
(b) Training strategy of HiMoLE, which adopts two-stage training strategy: first, each KCG is trained independently second, both the expert parameters and hierarchical routing components are jointly optimized.
clustered sub-dataset;
processed by sentence-level router fsen() (-) (implemented as linear layer parameterized by Wsen). to compute the initial allocation scores Gsen over KCGs:
sen Ssen (Nsen) W -iee--
These sentence-level scores serve coarse garseegrained guide, determining which KCGs are most relevant to the input. Subsequently, for each token in the sentence, token-level router fuken (.) (implemented linear layer parameterized by Weoken refines this allocation score by integrating coken-specific hidden state btoken:
Gunken = fioken (Mtnken)
W
Jooken-
The final gating weights matrix Ghie are computed using a softmax-normalian fusion of Gsen and Gusens followed by top-k selection to ensure sparsity:
Gmie KeepTop-k (Softmax (G O Gukken)
The forward process of the HiMoLE layer replaced the traditional FFN layer can be represented as:
NxM
0 Wo hnoken
Σ
G bic
(5)
where Wo the parameter matrix of the original FFN layer of the LLM, and denotes the output. The scalar (() modulates the contribution weight of the i-th expert E,. We provide the definitions of the symbols in Appendix
In summary, the hierarchical routing mechanism enables HiMoLE to balance domain specialization and generalization. Sentence-level routing assigns inputs to suitable KCGs based coarse semantic
5

---

cues, while token-level routing fine-tunes expert selection for dynamic. context-aware feature fusion. This design allows for flexible expert collaboration and competition, ultimately enhancing OOD robustness and domain-adaptive inference.
3.3 Training Strategy
As shown in Fig. 2(b), adopt a two-stage training strategy construct and optimize the HiMoLE framework. The first stage initializes the Knowledge Competition Groups (KCGs) with specialized domain knowledge, while the second stage jointly optimizes both the expert networks and the hierarchical routing mechanisms.
Stage 1: Initializing Knowledge Competition Groups We begin with the assumption that each lask may span multiple subdomains. To capture this diversity, we partition the training dataset into N subsets, each corresponding to a distinct semantic cluster, and train N KCGs in parallel. To perform data clustering, we first use pre- trained encoder to obtain semantic embeddings for each data instance. We then apply the K K-means clustering algorithm to group the data into N clusters. Each cluster is treated as sub-dataset, and a separate KCG is independently trained on it. This process results in N distinct groups of LoRA-based L.LM experts, each specialized specific knowledge subdomain.
Stage Co-optimizing the Experts and Routers After initializing the KCGs, we jointly optimize the expert parameters and the hierarchical routing modules. To encourage diversity among experts and reduce redundancy, we introduce diversity loss Ldiverse. Let en denote the output of the w-th KCG, computed as:
Cn Σ Loken
Em
Buskens
where
Em E KCG,-
(6)
where is very small number such as We normalize each expert output en max [e_.ll..)
10 10-8, 8, We then compute pairwise cosine similarities Sni = (en, et) among all the KCG pairs, and define the diversity loss as the average similarity across all unique expert pairs:
Σ Liiverse N(N 1) 1=1,inn
Smi-
To control computational cost, Laivens is computed every ten layers using sampled subset of expert outputs. The final training objective combines task loss Liss, auxiliary loss L which is employed to mitigate the unbalanced load for experts (following [23,, see Appendix C for details), and the diversity loss:
= Linsk + C.Lauxx + B
3.4 Theoretical Analysis of Generalizability in Sparse Routing Systems
We analyze how hierarchical expert routing reduces generalization error by mitigating gradient conflicts through structured sparsity in expert selection. Let t index tokens and 0 be the parameters of an expert E, which includes the low-rank matrices A and B as defined in Eq. [ Let VoL(-) denote the gradient of the loss with respect to 9. We define the expected pairwise gradient similarity as follows:
SimGrad == Eeys [cos (gt, )e where gt == VoL(h.)
Definition 1 (Gradient Conflict) Let be + by be inputs from distinct tokens. A gradient conflict occurs if cos (VaL(h)), voL(he)) <
In this context, higher value of SimGrad indicates better alignment between token gradients and thus fewer gradient conflicts (24).
Theorem (Hierarchical Routing Mitigates Gradient Conflicts) Let Sim GrnGrad and Sim Gra denote the expected pairwise gradient similarity under hierarchical and token-only

---

routing, respectively. Then hierarchical routing with composition fsen (sen()yen) @ Fuken (Moten) yields:
Sim Grad = Stm Graen + Ao.
(10)
where Ag >= O. Proof. See Appendix D
That is, hierarchical routing reduces the prevalence of conflicting gradients by inducing structural sparsity in expert selection. We then connect gradient alignment to generalization through a bound on the generalization error.
Lemma (Generalization Bound via Gradient Variance (25))
Let 9t == VoL(h)) and define
the gradient variance as Vig) := E [il(= E(g Then, the generalization error of stochastic
gradient descent with additive Gaussian noise satisfies:
R
T
Gen S
(()),
(11)
where R represents a constant related to the properties of the loss function and the data distribution, and b denotes the number of training samples. is the total number of iterations. 1) is the learning rate at step and o- is the standard deviation of the Gaussian noise at step
Theorem (Gradient Conflict Reduction Enhances Model Generalization) Assume hierarchi-
cal routing achieves lower gradient variance such that V (ghie) <= V (gooken), then under the condi- tions of Lemmal 7 hierarchical routing yields tighter generalization bound, i.e., Genaic S GeDgoken-
Proof. By monotonicity of the square root and the inequality on gradient variance, the result follows directly from Lemma See Appendix D for further discussion.
4 Experiments
4.1 OOD benchmark
Name Entity Recognition (NER). To emulate real-world data heterogeneity and enhance the complexity of the NER task, we selected the biomedical domain as our experimental scenario- field characterized by diverse entity types and lexical variations. For the con- struction of the ID dataset, we rigorously curated
knowledge-intene
resources from the BigBio benchmark 26).. with the corpora primarily sourced from PubMed Central (PMC), a premier reposi- tory of peer-reviewed biomedical literature. This process resulted in BigBio-NER, the largest dataset for biomedical NER. For the selection of OOD datasets, we adopted the criteria outlined in Sec- tion 3.1 choosing the rare disease dataset 22 sourced from the National Organization for Rare Disorders database (28)
Sentiment Analysis (SA). To further enhance the complexity of the SA task and better simulate real-world application scenarios, we frame our SA experiments within social science contexts, where affective expressions exhibit heightened subjectivity and domain-specifi connotations. We adopted the sentiment analysis component in SOCIALITEINSTRUCTINN This comprehensive collection of socially-oriented textual interactions contains sentiment labels across various social science scenarios. For the selection of OOD data, we adopted the criteria 30 dataset.
[29] dataset as our dataset.
outlined in Section 3.1 and chose the OPTIMISM
Extractive Question Answering (EQA). Following previous work [4, we chose SQuAD [31 as the ID dataset, which constructs question-answer pairs based on Wikipedia passages. For the selection of OOD data, we chose NewsQA 32, which writes questions for CNN news articles, each of which requires reasoning answer.
4.2 Experimental Settings
Base Model and Data Separation For the NER task, we employ OneKE-13B +R as our base model, which is capable of generalized knowledge extraction across multiple domains and tasks, For
7

---

Table 1: Comparative performance of different LoRA methods under out-of-distribution scenarios. Please refer to Appendix E.3 for the metrics details and Appendix E.5 for the complete results. The best results on ID data and the best results on OOD data excluding the base model are highlighted in boldface and underlined, respectively.
Task
NER
SA
EQA
Datasct Metric
ID
OOD
ID
OOD
ID
OOD
F1
P
F1
P
R
EM
REM
EM
EM
ROUGE-2
EM
ROUGE-2
Base Model
52.5
56.2
51.7
65.8
63.1
76.4
48.8
59.5
56.9
7.2
12.3
8.3
15.8
LoRA
73.9
77.5
73.3
61.9
56.0
77.8
66.7
55.7
55.0
68.2
48.7
35.5
28.2
MixLoRA
76.0
78.2
75.7
61.0
59.1
73.7
69.5
66.5
31.4
69.4
48.6
38.3
27.9
HydraLoRA
77.3
78.4
76.7
62.9
59.4
75.5
70.3
67.2
30.4
68.7
46.4
38.3
26.4
HiMoLE
77.9
78.7
77.3
65.3
64.4
74.3
73.3
68.8
32.8
70.5
49.8
38.9
28.7
the SA and EQA tasks, we utilize L-ama-2-7B [44] as our base model. To separate in-distribution data, use BioBERT [5], BERTweet [66, and DeBERTa-v3-large [37] encoders to extract sentence-level embeddings for the NER, SA and EQA tasks, respectively. Through sentence feature- based K-means clustering, we obtain subsets of sizes 4, 3 and 3 for each task (see the visualization of the clustering results in Appendix E.
Baselines and Settings We compare our HiMoLE method against LoRA and traditional mixture of LoRA experts. In our HiMoLE approach, each Knowledge Component Group consists of Knowledge Component Experts. For the mixture of LoRA experts, we compared state-of-the--a methods, including MixLoRA (6] and HydraL.RRA
p In all mixture of experts methods, we
adopted the tok-k routing and set k in tok- to 2, while setting the LoRA rank to 8. We apply MOE to the FFN layer of every Transformer block. To maintain uniformity in parameter sizes across all methods, we set the rank to 80 in traditional LoRA.
4.3 Primary Results
In Table we provide a comprehensive comparison of HiMoLE with various baselines. There are several observations: (1) Overall, HiMoLE achieves the best performance across all tasks on both ID and OOD datasets compared other LoRA-based methods. Specifically, compared to the best baseline, HiMoLE achieves improvements of up to 3.0% on ID datasets and 5.0% on OOD datasets. (2) Although the recently proposed HydraLoRA and MixLoRA suggest their ability under multi-task learning scenarios, the lack of adequate sentence-level message integration for routing creates bottleneck under outtt--disttibuttin data, even resulting in worse performance in knowledge- -ontesigeinttesive domains. (3) The generalization improvements from HiMoLE demonstrate domain-depended variability, showing more pronounced gains in knowledge-intensse the performance enhancement compared to the best baseline increasing from 0.6% to 5.0% on OOD datasets. (4) All MoPE-based methods still face limitations in enhancing generalization ability when operating under fixed number of parameters and limited data.
domains, with
4.4 Effectiveness of HiMoLE on OOD samples
Hierarchical Experts To substantiate the superiority of our hierarchical expert design, we conduct load balancing evaluations on out-of-distributtion datasets. Through the visualization of expert selec- tion logits from randomly sampled OOD instance (Fig. 4 see the complete result in Appendix [E5).
we observe that MixLoRA exhibits consistently severe load imbalance across layers compared to HiMoLE, with this disparity intensifying at deeper layers- particularly in its final layer where 78.3% of tokens disproportionately select Expert 4 with probabilities exceeding 0.85. Quantitatively, we adopt the Max Viogiobal metric from 38 (see the definition Appendix E..3 to evaluate MoPE-layer balancing. As displayed in Fig. 4 HiMoLE consistently improved the load balancing among ex- perts, with particularly notable improvements in knowledge-intensive domains (20.6% reduction in biomedical entity recognition). These results indicates that HiMoLE's hierarchical architecture effectively reduces expert redundancy through structural sparsity constraints, thereby promoting experts specialization and balanced expert utilization.

---

HIMNLLE Layer
MisLoRA Laver B
Selection
Net c Type
5000 Polforooonm Noformutoo
644
Expert Selection Portern
Laver
2.0
Expar: Solection Rottor
15g
KER RGA
54
Figure 4: Comparison of ex- perts load balance for differ- ent mixture LoRA experts methods. We utilize Max Vio to evaluate the load perfor- mance.
50
100
190
045
139
200
5
10
22
24
20
34
Figure Visualization of expert selection logits produced by HiMoLE and Mi
Table 2: Robustness character-level adversar- ial attack. Reported results are accuracy scores. The Robustness Ratio is defined adversarial sample accuracy divided by original accuracy.
Table 3: Ablation study the stage training strategy and diverse loss. Reported results are F1 scores. Please refer to Appendix E.5 for the complete results.
Metric
Base
HiMoLE
MixLoRA
Dataset
ID1
ID2
ID3
ID4
00D
original 0.488
0.733
0.695
HiMoLE
87.6
64.0
75.3
79.6
65.3
attacked
0.196
0.330
0.280
-wo. Two-stage Training
77.7
32.1
64.1
60.8
53.5
Robustness Ratio
40.2%
45.1%
40.3%
wwo. Diverse Loss
87.0
63.1
74.7
78.2
65.0
Hierarchical Routing Strategy To further investigate the robustness of the hierarchical routing strategy, compared its performance against the token routing method (MixLoRA in the sentiment analysis task. This assessment involved generating adversarial ditoo--distribttion samples using the TextBugger [39] tool. Specifically, randomly sampled 500 instances from the in-distribution test set and injected token-level noise via character-level perturbations.
As shown in Table 2 both routing strategies experienced significant performance degradation under adversarial attacks; however, hierarchical router demonstrated not only a superior absolute perfor- mance over token-level router, with 6.0% improvement, but also exhibited a smaller decline in performance, with a 4.8% improvement over the token routing method. This indicates that hierar- chical router possesses markedly stronger robustness against adversarial samples when compared to the base model and the token-level MoPE. The results further validate that HiMoLE, through its integration of global features, maintains stable performance against local perturbations by selecting appropriate experts via sentence-level semantic analysis, even in adversarial scenarios.
4.5 Ablations
Two-stage Training Strategy We investigated the influence of different training strategies on model performance by conducting the experiment the NER task. As evidenced in Table when using single-stage joint training without preliminary KCG initialization, we observed a dramatic performance degradation of up 31. 31.9% F1 the ID2 dataset. Furthermore, we examined the bad cases and found that the proportion of incorrect formats had increased significantly (see Appendix A). These observations indicate that (1) unstable optimization trajectories were adopted when learning router-expert interactions from random initialization, and (2) the absence first-stage specialization prevents KCGs from developing domain-specifee inductive biases, resulting ambiguous routing signals. Thus, proper KCG initialization is critical for our hierarchical routing mechanism to function.
Diversity Loss We investigated the importance of diverse loss on model performance. As evidenced in Table omitting the diversity loss component leads to statistically significant performance degradation across both ID and OOD data. This consistent pattern reveals that diversity regularization can further mitigate expert group redundancy and improve model robustness.

---

8
M
M
23.3%
Performence
Performence
37.3%
000
00D
ID
OOD
0OD
HIMoLE
MixLoRA
HydroLoRA
HIMGLE
MiXLoRA
HydroLoRA
Figure 5: plyper-parmmeter Analysis on LoRA rank r and Knowledge Collaboration Experts Numbers M. Performance OOD dataset evaluated using REM (Appendix E.3
4.6 Hyper-parameter Analysis
As illustrated in Figure 3 we further conduct hyper-paramest analysis on the sentiment analysis task examine the impact of the LoRA rank r and the number of knowledge collaboration experts M. In experiments, investigating r. The results reveal that HiMoLE demonstrates superior robustness against variations in both parameters, consistently outperforming baseline methods across all configurations. Notably, while competing approaches exhibit significant performance degradation on OOD data with reduced LoRA ranks, HiMoLE achieves remarkable OOD accuracy improvement from 31.3% 60.0%, conclusively validating our method's effectiveness.
fix r 4 when analyzing M, and conversely maintain M 4 when
Conclusion
While PEFT methods like LoRA have significantly lowered the barriers for adapting large language models to downstream tasks, our investigation exposes their critical vulnerability to distributional phitis-particularlly in knowledge-intensive domains. Our proposed HiMoLE framework alleviates this fundamental problem with by integrating hierarchical experts with a hierarchical routing strategy. This approach leverages sentence-level information coarsely allocate experts to relevant subdomains and then refines the routing weights using token-leve information, enabling efficient acquisition of new knowledge while preserving existing knowledge. By theoretically and empirically validating this approach across three representative NLP tasks, we establish a new paradigm for developing adaptable language models that achieve parameter efficiency with enhanced generalization capacity.
Acknowledgement
This work funded by Zhejiang Provincial "Jianbing" "Lingyan" Research and Development Pro- gram of China 2025C01129) National Natural Science Foundation of China (62302433, 62301480, U23A20496), and Hangzhou West Lake Pearl Project Leading Innovative Youth Team Project (TD2023017). This work supported by Ant Group and Zhejiang University Ant Group Joint Laboratory of Knowledge Graph.
References
[1] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, 2021.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning, 79:151-178 2010.
[3] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples neural networks, 2018.
[4] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distriibttio robustness in nip: Benchmark, analysis, and lms evaluations, 2023.
10

---

[5] Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et Evalu- ation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 30(9):2613-2622 2024.
[6] Dengchun Li, Yingzi Ma, Naizheng Wang. Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts, 2024.
[7] Yunhao Gou, Zhili Liu, Kai Chen, Langing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T. Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision- -anguugg instruction tuning, 2024.
[8] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning, 2023.
[9] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024.
[10]
Linyi Yang. Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang. Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. Out-of-distribution generalization in natural language processing: Past, present, and future. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 4533-4559, Singapore, December 2023, Association for Computational Linguistics.
[11] Xuezhi Wang. Haohan Wang, and Diyi Yang. Measure and improve robustness in nlp models: survey, 2022.
[12] Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. On the robustness of chatgpt: An adversarial and out-of-distribution perspective, 2023.
[13] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang. Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts, 2024.
[14] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019.
[15] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning, 2021.
[16] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpattre and Graham Neubig. To- wards unified view of parameter-efffiient transfer learning, 2022.
[17] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1932-1945, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[18] Qidong Liu. Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets llms: Parameter efficient fine- tinettunig for multi-task medical applications, 2024.
[19] Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola. Dive into deep learning, 2023.

---

[20] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia. and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republie, November 2021. Association for Computational Linguistics.
[21] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmant 1989.
[22] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks, 2015.
[23] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022.
[24] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020.
[25] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M. Roy. Information- theoretic generalization bounds for stochastic gradient descent, 2021.
[26] Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Ruisi Su, Wojeiech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Ott, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang, Alison Callahan. Daniel León Periñán. Théo Gigant, Patrick Haller, Jenny Chim, Jose David Posada, John Michael Giorgi. Karthik Rangasai Sivaraman, Mare Pàmies, Marianna Nezhurina, Rohert Martin, Michael Cullan, Moritz Freidank, Nathan Dahlberg, Shubhanshu Mishra, Shamik Bose, Nicholas Michio Broad, Yanis Labrak, Shlok S Deshmukh, Sid Kiblawi, Ayush Singh, Minh Chien Vu, Trishala Neeraj, Jonas Golde, Albert Villanova del Moral, and Benjamin Beilharz. Bigbio: A framework for data-centric biomedical natural language processing, 2022.
[27] Cathy Shyr Yan Hu, Lisa Bastarache, Alex Cheng, Rizwan Hamid, Paul Harris, and Hua Xu. Identifying and extracting rare discases and their phenotypes with large language models. Journal of Healthcare Informatics Research, 8(2):438-461, Jannuaarry 2024.
[28] Claudia Martinez-deMiguee Isabel Segura- Bedmar, Esteban Chacón-Solano, and Sara Guerrero- Aspizua. The raredis corpus: corpus annotated with rare diseases, their signs and symptoms, 2021.
[29] Gourab Dey, Adithya V Ganesan, Yash Kumar Lal, Manal Shah, Shreyashee Sinha, Matthew Matero, Salvatore Giorgi, Vivek Kulkarni, and H. Andrew Schwartz. SOCIALITE-LLAMA An instruction tuned model for social scientific tasks. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisties (Volume 2: Short Pupers), pages 454,468, St. Julian's ss Malta, March 2024. Association for Computational Linguistics.
[30] Xianzhi Ruan, Steven Wilson, and Rada Mihalcea. Finding optimists and pessimists on Twitter. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 320-325, Berlin, Germany, August 2016. Association for Computational Linguistics.
[31] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016
[32] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih, editors, Proceedings of the 2nd Workshop on Repre- sentation Learning for NLP, pages 191-200) Vancouver, Canada, August 2017. Association for Computational Linguistics.
[33] Yujie Luo, Xiangyuan Ru, Kangwei Liu, Lin Yuan, Mengshu Sun, Ningyu Zhang, Lei Liang, Zhiqiang Zhang, Jun Zhou, Lanning Wei, Da Zheng, Haofen Wang, and Huajun Chen. Oneke: A dockerized schema-guided Ilm agent-based knowledge extraction system, 2025.
12

---

[34] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Rohert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[35] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 123-1240, September 2019.
[36] Dat Quoe Nguyen, Thanh Vu, and Anh Tuan Nguyen. BERTweet: A pre-trained language model for English tweets. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, Online, October 2020. Association for Computational Linguistics.
[37] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre- trainininng with cradientddisentangle embedding sharing, 2023.
[38] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-ree load balancing strategy for mixture-of-eeperts, 2024.
[39] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-worid applications. In Proceedings 2019 Network and Distributed System Security Symposium, NDSS 2019. Internet Society, 2019.
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best judgment and
13

---

write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong ussumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well cell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
In Section 3.4 and Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the datasets are open-source, and the code can found in the supplemen- tary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
icy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. htte//ublcccgggudeeeeee htte://////ddee//eeuuuubbiiii /at////ooeSSsiiseeeeeeeeeeeeeeee ississsnns ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental details can be found in Section 4.2 and Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: All results from our experiments are presented as either the best or average outcomes.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information the computer resources can be found in Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///wwiippp.cppppp////iaiaaa
Answer: [Yes]
Justification: We make sure the research is conducted in the paper, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
17

---

Justification:
In Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: No such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The models and the data are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palasets.papersstt paperssi thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This study does not involve any research conducted with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This study does not involve any research conducted with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification:
The LLM is used only for formatting purpose.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httppe///earrrp..ccco///eeeeeeee for what should or should not be described.
20

---

A
Bad case studies
Incorrect Format In the training ID data, the presence of numerous duplicate entity objects in the labels causes LoRA-based fine-tuning methods to inadvertently replicate this pattern. Consequently, when applied OOD data, the model tends generate more entities than necessary, often resulting in repetitive output behavior. This ultimately hinders its ability to produce correctly formatted JSON outputs as instructed. As illustrated T this issue is particularly pronounced with both simple LoRA and token-level MoLE methods. In contrast, sentence-level MoLE and HiMoLE effectively address this problem. Notably, sentence-level MoLE ensures that examples produce correctly formatted JSON outputs.
Table 4: Comparison of the Format Robustness
HiMoLE stage training
Method
Base
LoRA
Token-MoLEE
Sentence-MoLF
HiMoLE
Incorrect Format Ratio %)
1.19
8.33
7.14
0.00
2.38
44.0
Misclassifications Analyzing the erroneous results revealed that models trained on ID data fre- quently mislabel certain symptoms diseases when assessed on OOD data, failing apply the general knowledge that differentiates the two. HiMoLE significantly mitigates this issue. Figure showcases several comparative results between MixLoRA and HiMoLE.
B Definitions of the Symbols
Table 5: Definitions of the Symbols used in the paper
Symbol
Description
m.m.n. n,l B.A
index number
the low-rank matrices
E
the weight matrix of LoRA expert
din doa; damb
the LoRA rank, the dimension of the input and the output of FFN layer, the dimension of the sentence embedding used for dataset clustcring
the number of the Knowledge Competition Groups the number of the Knowledge Collaboration Experts in a Knowledge Competition Group the gating weights matrix derived from the sentence-level router, the token-level router and the hierarchical router, respectively learnable parameters of sentence-level router and token-level router the representation of the sentence- leveec level router and the token-level router
M
Gudi: Osmy Grie
W Weeken sm
fuoens fen
the output of a FFN layer
Auken
loken-level hidden representation sentence-level hidden representation
hye
e
output a Knowledge Competition Group the cosine laritarity between a pair of Knowledge Competition Groups the representations of overall loss. the task loss, the auxiliary loss and the diverse loss the gradient of the parameter 0 with the loss function C expected pairwise gradient similarity from all different tokens
Lua, Luxx Ldiverse
VoL(.)
SimnGadd
V(g)
the variance of gradient
the notation of the expectation the real number space
21

---

Base
MIxLoRA
HIMOLE
Alinstomuni
disented
Alkectonuri
Alteptomuni
choracterized by accumubttion of homoo homogentiscc basty Aflected indivituals lack snough funasional leow of an enryme requined to breeksown homagentise acid.
choractertoed by accumuation homogentiic
character ard by accumuction homogontsit acid the budyy Aflected indiedush last enough functional levels of an entyme required to breskdown homogentiir anid. Affected indiidua's may have dark urne or unine that turns bllck when esposed to elr Howwewe this change may not occur for severa hours after nnstion and often goes unnoticed. Asile from dark unine that present from infarey, affected indivitusls proeesly by do not devwlop infoncy childhoed and adultheod.
busty, Allested individssh lact enough functionaal lestt
of an ennyme required to breakdoww homogentior scid
Affocted individuals may have dark unite or urine that turns bleck when exposse to air. Howsser, this thange may not occur for several hours after untoation and often goes unnot cod, Aside from dark urine that is present from
Afforted individuals may have dark uriee or unine that turns
black when exposed to ait. Howseer this change may not occur for several hours after unination and often goes unnoticed, Aside from dark unine that is present from infarny, affected individuals generall do desw
infenty, affected individuall gensrally do deselop sumptooss (spympto motic) their Infoncy chilchood
symptoms (osymptomet oflen unasvere their condition adultheed.
infoncy chiicoodd
symptoms (osymptomatid oflen uneware their conditian event ly develop ochronasis, which is the blutsh black disooloraton connective and other
oflan
aduithood.
individuats evertually develop ochroross, which
eventually develos op achronosis which
affected
stush slack discoloration connective and other
the blussh slack disoteration connective and other
fisue within the body. Aflested indivdush may deoelop
Iisue within the bodyy Affected indisidon's may deswlop
lixsee within the body Affecced individuads may dewwla
discoloration the skin carthng withis the body such as over part the outer eat In some cases, the wh
the skin everiving cartllgee within the body such as over part the outer ear in some cases, the whites
aiocolorntio
diccolomntoo the son everlying cartlage within the body
such as over part the outer ear. in some cases, the whites
of the eyes (velers) may also herome divesloeed. In
of the eyees (schers) may also berome dianloeed In
of the eyes (sclers) may also become disniired
aduthhood, offected indv duals also develop brognessivel
aoulthood affected indiiduals als doveloo progresiive
aduithced, affected ind viduals also develeo brogresstrel
at of the spins and lange joints. The HGD gune
arthriiis of the spine and lange joints. The HGD gene codos
tiiiiisaf the spine and lange joints. The H&D gene codes
for the enoyme required for breeldown homogenteic HGD Albestonari allects
for the encyme required for breakddow of hamage-tricc HGD
for the
required for breakdowwn of homagentsic Bllaptonutis equel athough symetoms to develop sooner and become
acid. Mutztions
acd, Mutations
illaptonuri
oci. Mutation
gane
equel numbers
Albaplintura alletts
equel numbers
Altaplanune wlleets males
stoough symptoms
develop sconer and become
strough symptoms to develop sooner and become
more severe in males, More than affected incividuals
more severe in males. More than 1,000 affected indivituals
more severe in males, More than 1,000 affected individuals have been reported the mediral literatree The ecart
hase been reported the mediaa literature, The esart
have been reported the med literatrre, The esart
incidence of Heaptnnnng is unknown, in the United States is untimeted to occur in in 250,000-1, 0,000,000 live brith.. Alncttnnrti has been reported in all ethnir groups. Areas with incressed frequencies of the dicorder have been
Inc donce of Mlapoonunt i uninown, in the Untedd States it estimated to occur in 250, 000-00001,,000000 0,000,0000 live listin. Alkaptonura has been reported in all ethniz groups. Areas with increased frequencies of disorder have boen idemiffed in Slowaii, the Dominican Rappbiic and Gurmanyy
Incidonce of Blaasoonuis 5 unknoww, in the United States
il esiim ated to eccur in in 250,000-5,000 1,000,00000000 000 low lirths. Mlesptoruni has been reported in all ethntic groups Aress with increased frequencies of cisorder have been
abintffed in Slovakis, the Duminican lapublic and Carmanyy
idemiified in Sloonkia, the Dominizan Republic and Germany.
Base
MixLoRA
HiMoLE
Ete-Htghh Currie sundromel's a disonder that
Hit- Curt's sindromals disonder that occurs exdusivdyy woman, is (pertoon cum) (per spaliti)) mackle that wpanaiis the stomach
Hit Curtis mandromels is diaorder that occurs by (pertone.m) and tisues surrounding the Ipan inspaiii)) The muscle that wparates the stomach the chest (daphragm) which play an essential in breathing. may alo be affected. Commen symptoms indude uevere pain in the upper right area iqueth of
lmost exdusivvly
character by lining
by
excluively weman, & lning
(ooftoncuu)) tasues surrounding the Genthhpaiiti) macke that wparatus siomach and the chest (dtspheagm). which plas an essental in breathing. may also be affected. Common symptoms inckude sesere pain in the upper right area (quediant)) of the abdomon, fever, chills, headaches, and a general teeling of poor health (malaso). fin Hugh Curls iis syndromelss i
tssues surrounding the
the chest (diapraggm) which play an essential in breathing. may also be affected. Common symptoms inc ude unvere pain in the upper right weea (quedhant) of the abdomen, fever chills, headaches, and a general tealing of poor health (mralassa) fin Hugh Cur tis syndromil &
the abdomen, fever chills, headaches, and a general teel
of poor health (malaise) Fit Hugh curtss pendroms is
complication pulvic ind unmettrry dinees (11)) grceral trast Neiseria thowas
complication ion jubvar inflammttrry diseass (11)) general genibal tract Neisaria sonerrhoeee
complicat ion pabee inflammation
(Pil) general
torm for infection of
torm for Infection of often
torm for Infection the genital trad
infection most often
Infection
Infection
ofan
Neisarria gonenboeee
Chlamydia trachomatis diggnss Bitahuugh
Chiamycia trachomatis diagnods Bitalligh
Chiamydia trachomatt. diagnous Bneeltghhhumm
anddoomels made through the exdusion of other causes
sindromals made through the exclusion of other causes
aindromale is made through the exclutio of other couses
apper right abdominal pain. A lignosis may be confirmed
upoer right abdominal pain. A diagnosi may be confimed
upper right abduminal pain. A daagrosis may be confirmed
with a variety of speriallzed tests inclding eny examination, diag rostic laparoscopy, and certain laboratony ezoms K-ray essmination on may include ultramund, chest or stomach radiographs, and computed tomog (CT)
with a vartety of speraalied bests Inclding vray
with a vortety of specialted test including why
esamination, diag dagnesic laparsscoy, and certain laboratory
esamination, clage asci laparosoops, and cortain laboratory momms X-ney mosmination may include ultroound, cheet or
ecams. K-rny ecemination may inc tide ulrasound, chest or
stomach rachogrash, and computed tomography (CT)
storach rad roolograhh, and computed tomography (CT)
scanning X-lay are used to rule out other possible conditionn reveal characteriste infommatioon the
scanning, X-rays are used to rule oui other posible
scanring. X-raps are used to rule out other posibre
conditions reveal characteritic intiammation the
conditions reveal chameteritiic Infiammtionn of
perthopatic region. During lsporcscopy, small, tube abdurinel cailly through small in stomach. approscopic exam allows physidin to vie the liver suroonnding tisue. Laboratory exams identily infectiin with Chiemyyii trachometis or Neiverri gonorrhceae.
peritepatic region. During lsparcccopy small tube through
opaile reaion. Duringa laparsscopx, small, tube aldlomiiaal
inserted aeduminal
thrtugh al the stomach. A laparostoplc exam slloss physican to vlow the liver surrounding Usaue. Laboratory caams idemtily infection with Chiemydhi teachomatis Neivuetis gonormoese..
stomach. teprrsccopic exem allow physicton to
view the lher sumounding tasue. Laberatory caams ideettly infection with Chismytis trechaments Neisueti gonormosae.
Base
MixLoRA
HiMoLE
Sooneyylsspihhyse dpplase tanday Scof se0t) is rare,
Scanchyospien picphhyyllpplyaallloooo dysplado Stol is a rane, hereditary aerienal dosnie that only affects males Phynical idwartem).
Scordsloup phyooul dysplasia lar SIDE Seo is a rane, hereditary pekeer dhanted hat only affechs males. Physios character include moderote
pereditery skeletal diceted that only affects males. Phylica (dwortom),
choracteriszies include moderate short mudenate--o-sseee spinal deformtie,, barne-hheeped diproportionant
moderate short choracteristic ipinal deformaiis, herrelsheped premature
idwarfam),
diepropotionntely spinal delormiiss, barne-sheped trunk, premature
trunk, and premature
diepreportonately
cotecarthrits, SEDT does not onibi any cthnic predhposition Affected indiciuush hure been described serr European, American, Aston and Australion populations ond tbut not in African Amoricans to date). One escimate suggests that the inzidence is persoes per million
astooarthitti, BEDI does not cdhbtt any ethnic
assooarthviti, BEDI coes eshibit any ethnic
predispositien Affected individues ln' have been descr European, American, Aston, and Australlon populations (but not in ArriianAAmriaass to dana). One exiimae suggests twat the incidence is persons per million.
predispostion Afficted individuels have been deez European, Amerizan, Aston, and Australion populations (out nos in Anrian-Ammrraaa to data), Ona extimate supgests that the incidenee is persons per millen.
Figure 6: Misclassifications Cases. Text highlighted in green represents correct entity annotations, while yellow represents incorrect entity annotations.
22

---

C Auxiliary Loss
Given N experts indexed by i1 N and batch with T tokens. Let G(-) denotes the top-k router, F; is the fraction of tokens dispatched to expert, and P; is the fraction of the router probability allocated for expert The final loss then multiplied by the expert count to keep the loss constant as the number of experts varies, which can be formulated as following:
1 Σ (argmax, R(x)x =)) P. F T
Σ
R(m)),
(12)
T
WEB
N
Laxx = N. P.P
Ps:
(13)
1=1
D Proof
Here provide the proof of Theorem
Lemma Let represent distinct tokens and represent sentences. To simplify, let's assume that the sample gradients set of independent and identically distributed unit vectors. In this case, SimGrad can be expressed as (hh)))l?
SimGrad [[cos( (e)))gg)))))) )) Eg(e ge)
g (B [0c ge (9.]) = E, (9t Egg (gv)) = E.g. (gt) TBg (gtr)
(14)
-11E,,111 |I L(x:))]
For the i-th expert Ei, let Aus denotes (E, E argmaxx (hs) ) 2 Sankon (he)))). Ba. denotes I (E, E argmax; (Frken (he))) For hierarchical router Ghiee and token router Groken their gradient operators can be written as follows:
Proof D.1
0 Caie = I(E, E a (8,[(ha,t)
=
-(Ee E (Psmaxx(Smll O Sinren (b+))) (OiL(har))
(15)
As.t V0,L(h.,))
Vo, Lioken = I(E argmax (Groken)))))) v0,((x,,))
= =I(E, argmaxy, ( (Sukee (I+))) v0,[(ho.))
(16)
BB,t v0,C(ha.))
Combining Lemma 2, Do, can be written as follows:
Ds. SimGradhie SimGradroken
|E(Vo, I 112 ||E Lioken 11 [B(AAct)2 B(Ba,)) (ho,,(h)s))))
(17)
Since A Ba, {0, 1}. we have:
=(A.,))= B(A.))= B(Ba.)
Next, analyze the relationship between As, and Ba,t. The hierarchical router ensures that the effective selection of token router not overlooked by positively adjusting the scores. Additionally, by introducing global information through sentence features, it uncovers experts that neglected under local token features, which guarantee that:
Bat 1 Aa, 1, but not vice versa, As,t 1 + Bar
As a result,
Ao. = [E(A,,)) -(B.
(18)
= [E(A,,;) -(B.
L(VO,
23

---

Table 6: Deseription Datasets used in experiments.
Test
Task
Domain
Train
ID
OOD
NER
Biomedical
35132 11257
2060/1267/2764/2744
684
SA
Social Science
2374
1465
EQA
General
87599
10570
3882
Now we turn to the proof of Theorem
Lemma Let denotes the gradient, E(g) denote the average gradient and m denote the number of tokens, then E(g)² can be written as follows:
E(g)²
Σ 11 + 2 9t gr
(19)
m2
type
Lemma Let (g) represents the gradient variance, then V(g) x -SimGrad:
Vgg) [l[ - [(g:) m
19: - B(g)
(119a12 2g: (8(g) + (z)))²)
(20)
1|9e1 - B(g)²²
Combing with Lemma 3, V(g) can be written
V(0) = m
|[
||
Σ tff
gr -SimGrad
m
2
m²
(21)
Proof D.2 By Theorem I, SimGradnie >= SimGraddoton By lemma 4, higher SimGrad indicates lower gradient variance, hence V (ghie) < V (gooke).
E Experiments Details
E.1 Datasets
Table summarizes the datasets used in our experiments, including their task names, respective domains, the number training and test sets. For Biomedical NER, the ID test data was partitioned into 4 sub-datasets using feature-based K-means clustering. All datasets are downloaded from HuggingFace using the DATASETS library in Python. Additionally, we provide a UMAP visualization of the datasets in Fig. 7
24

---

E.2
Hyperparameters and Implementation Details
Table 7: Hyperparameter configurations of LoRA, MixLoRA/HydraLLRA and HiMoLE for fine-
tuning LLaMA2-7B and OneKE-13B. 13B.
Metric
LoRA
MisLoRA/HyyraLoRAA
HiMoLE
Cutoff Length
1024
1024
1024
Learning Rate
3c-4
3e-4
3 3e-4(stage 1), 3e-5(stage2)
Optimizer
AdamW
AdamW
AdamW
Batch size
16
16
16
Dropout
0.05
0.05
0.05
Where
Up, Down, Gate
Up. Down, Gate
Up. Down, Gate
LoRA Rank
80
LoRA Alpha
160
16
16
Top-K
2
We set a maximum of 10,000 training steps and perform evaluations on the validation sets of all benchmarks every 50 steps. If there is no improvement on the validation set for 10 consecutive evaluations, we will terminate the training early. The best checkpoint, identified by the highest average accuracy across all benchmarks, then selected for evaluation on the set.
All experiments are conducted with GPUs having 24GB memory (RTX 4090) for 7B models, GPUs having 40GB memory (RTX A100) for 13B models, and setup with Python 3.8 and Ubuntu 22.04 on x86-64 CPUs.
E.3 Evaluation Metrics
Performance For the metrics in Table I NER, Fl stands for the average Fl score, stands for the average precision, R stands for the average recall; In SA, EM stands for the exact match, REM stands for the relaxed exact match( we treat the 'positive' label as synonymous with 'optimism' and the 'negative' label synonymous with "pessimism') ); In QA, EM stands for the exact match, ROUGE-2 is employed to captures phrases that hold vital context.
Load Balance Max Vio is used to quantify the degree of load balance of an MoE. layer, defined as Max Vix = max, Load, -Load; where Load; represents the number of tokens assigned to the Load,
i-th expert, and Load, denotes the expected expert load under perfect load balance.In Max Vio global Load; is calculated the whole validation
E,4 Additional Experiments and Analysis
Impact of Two-stage Training Strategy To disentangle the effects of initialization from hierarchical architecture, we add baseline comparisons where MixLoRA and HydraLoRA are trained with the same Stage 1 initialization HiMoLE NER task. As shown in Table 8 although other methods demonstrate improvements, the two-stage training approach combined with the hierarchical mixture of LoRA experts still delivers the best performance across both in-distribution and out-of- distribution settings, with essential improvement under OOD setting.
Table 8: Impact of Training Strategy across mixture of LoRA experts methods. The reported results(%) are F1.
HiMoLE ID
MixLoRA ID
HydraLoRA
Training Strategy
OOD
OOD
ID
OOD
two stage
77.9
65.3
76.4
63.0
77.2
77.3
one stage
61.7
53.5
76.0
61.0
77.3
62.9
25

---

Inference Latency HiMoLE adds lightweight gating networks, maintaining nearly the same parameter count as MixLoRA. For inference. the speed is primarily influenced by the base model. Since the parameters of PEFT modules constitute a small fraction of the total model parameters (ranging from 0.65% 1.05% as shown Table 2 the inference latency differences across are minimal. Table presents the latency and parameter count during inference using Llama2-7B with different Mixture of LoRA Experts methods, evaluated on the OPTIMISM dataset using a single RTX 4090 GPU. Latency was recorded over 50 random samples. The results show nearly equal latency, but HiMoLE exhibits the highest model performance.
E,5
Complete Results
Table T0 presents the complete results of the comparative performance for the NER task in Table Fig. 8 presents the complete results of the routing logits discussed Section 4.3 Table displays the complete results of the ablation experiments in Table B
Training Strategy
Algorithm 1: HiMoLE Two-Stage Training
Input: LLM's frozen weights Wo. training data 2> (composed of (s, y) pairs), pre-trained encoder Encoder(-). cluster count N
Output: Optimized experts and routers
Stage 1: Knowledge Competition Group Initialization
foreach sentence 8; E do
L emb, <- Encoder(s:):
/ //Generate semantic embeddings
(C1, ................................ C.. K-means( (emb.),N). N); For to
//Cluster embeddings
parallel
De f- {ss, y:) Jemb, E C+}:
//Build sub-dataset //Train KOGs in parallel
OKCG, f- arg ming EGNDED Lusk (fa()).))
Stage 2: Joint Optimization of Experts & Routery
Initialize routing parameters Φb load pre-trained {OxCE, M M while not converged do for batch B C D do
foreach (s,
B do g [s)
/ //Routing weights //Weighted combination / //Compute loss
1 Wus + NXM (8);
Compute Lus = l(ÿ. v). Laus, Laivere
Update $ 0, 1/ (Luak aLom Bavvese)
G Limitations and Future work
In this section, we discuss the potential limitations of our proposed method HiMoLE, Firstly, has shown effectiveness in addressing simple OOD scenarios, it still struggles to deal with hard OOD samples(e.g.. in biomedical NER, fails to outperform the base model on the OOD dataset). Future
Table Inference Latency vs Performance across mixture of LoRA experts methods. The reported performance is evaluated using REM.
HiMoLE
MixLoRA
HydraLoRA
Paraam
1.05
1.05
0.65
Latency(s)
127
119
122.0
Performance($)
68.8
66.5
67.2
26

---

Table 10: Complete NER Results
Dataset
IDI
ID2
ID3
ID4
OOD
Metric
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
Base Model
59.1
60.6
59.0
42.4
52.0
40.4
51.3
53,4
51.8
52.3
57.5
51.4
65.8
63.1
76.4
LoRA
85.9
86.7
86.0
61.4
68.8
60.2
78.2
80.0
78.1
66.2
72.1
65.0
61.9
56.0
77.8
MixLoRA
87.4
88.4
87.3
60.6
64.2
$9.9
73.4
75.8
73.0
77.3
79.5
77.0
61.0
59.1
73.7
HydraLoRA
87.8
88.4
63.2
65.2
62.3
75.1
75.9
74.4
78.2
79.4
77.2
62.9
59.4
75.5
HiMoLE
87.6
88.1
87.8
64.0
67.8
63.5
75.3
75.8
73.4
79.6
79.5
79.8
65.3
64.4
74.3
Table 11: Detailed Ablation Results
Dataset
ID1
ID2
ID3
ID4
OOD
Metric
R
HiMoLE
87.6
88.1
87.8
64.0
67.8
63.5
75.3
75.8
73.4
79.6
79.5
79.8
65.3
64.4
74.3
-Ewo stage training
77.7
80.5
77.0
32.1
45.2
30.5
54.3
62.0
53.2
60.8
66.0
60.1
53.5
57.7
59.0
-diverse loss
87.0
87.5
62.7
66.2
62.4
74.9
77.2
74.8
79.8
78.4
65.0
62.8
76.7
work will incorporate data augmentation to handle with hard OOD samples. Secondly, we constrain the model size 13B and limit the number of LoRA experts 16 due to resource and time limitations. As expert numbers scale, the interaction mode among experts would be more intricate and more sophisticated routing topologies like graph can be introduced adapt the nuanced patterns that emerge. Subsequent research be will conducted the larger LLMs and more LoR A experts with more complex interaction mechanisms. This dual-axis expansion (model size + adaptive expert management) could unlock new robustness frontiers without proportional computational overhead.
27

---

15
3
dluster
0-
-10
-15
-10
-5
10
15
UMAP 1
(a) NER
10
duustr
UMAP
0
7.5
12.5
UMAP_1
(b) SA
20
10
cluster
UMAP_2
-20
-10
10
20
UMAP_1
(c) EQA
Figure 7: UMAP visualization of the datasets. Different colors correspond different sub-datasets for KCGs initialization.
28

---

HiMoLE Layer 9
MiXLoRA Layer 9
Expert Selection Pattern
Expert Selection Pattern
1.0
1.0
Token Position
50
0.8
seeeectio Confice
50
0.8
100
100
0.4
0.4
150
150
0.2
0.2
200
200
0.0
4
6
10
12
14
2
4
10
12
14
Expert Index
Expert Index
HiMoLE Layer 19
MixLoRA Layer 19 Expert Selection Pattern
Expert Selection Pattern
Confidence
Token Position
0.8
50
0.8
0.6
0.6
100
100
Selection
150
150
0.2
0.2
200
200
0.0
0.0
Expert Index
Expert Index
HiMoLE Layer 29
MiXLoRA Layer 29
Expert Selection Pattern
Expert Selection Pattem
1.0
1.0
Selection Confidenc
0leco Confidence
Token Position
50
0.8
50
0.6
0.6
100
100
0.4
0.4
150
150
0.2
200
200
0.0
0.0
6
10
12
14
2
4
10
12
14
Expert
HiMoLE Layer 39
MixLoRA Layer 39
Expert Selection Pattern
Expert Selection Pattem
1.0
Confidence
Token Position
50
50
0.6
0.6
1000
100
0.4
Selection
150
0.2
200
0.0
0.0
6
10
12
14
4
10
12
14
Expert
Expert
Figure 8: Expert Logits selection pattern.
29


Reviews:
[{"review_id": "hznySAFnNs", "reviewer": "Reviewer_ADXP", "summary": "This paper is concerned with OOD perfomance of the LoRA  finetuning frameworks. The OOD performance of LoRA is  first shown  empirically to be limited and then  by means of a  mixure-of-experts scheme the  OOD performance is improved.", "questions": "- The nuance parameter ‘k’ in eq(4), how do you choose it? have you performed any  optimization on it to find the optimal value? same for $\\alpha$ and $\\beta$  in the loss function.\n- You have tested your  scheme only on LoRA family, while there are other  PEFT methods which have equivalent performance with fewer trainable params such as BitFit, RoCoFT, (IA)3. Is  your method   extendible  directly to other PEFT methods?", "limitations": "The HiMoLE  scheme  introduced in this paper is only applicable on LoRA family PEFT.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Vulnerability of  LLMs  to  distribution shift in the  data is  an important problem and yet less explored in the  literature, therefore this paper is of high interest to the community. The strength is showing theoretically that  hierarchical routing reduces the prevalence of conflicting gradients by inducing structural sparsity in expert selection and this is directly connected to generalization error. \n\n- in (2)-(3) please mention the  sizes of matrices W,h,G, for better readablity.\n- please proofread the paper for grammatical errors: \"layer replaced the traditional FFN layer can be..\"\n- the connection between  (10) and (11) is  related to lemma 4 in the appecdix. Not including it in the main text is like a missing part. In particular, how SimGrad and V(.) are connected should be clearly discussed in the main text.\n- In algorithm1,  are the parameters  Alpha,beta ,k   fixed and  fed in as the inpute? please  correct this.\n-In the proof of lemma 4, there seems to be  some  approximation which  should be denoted by ~ and not =.", "was_revised": true, "final_justification": "The Authors sufficiently addressed my comments and made it cleat why LoRA is chosen and MoE framework for other PEFT  methods while  doable is not trivial.", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n- W1: better readablity of the the sizes of matrices.\n\n  Thanks for the suggestion, we will add in the revised manuscript. The size is as follows: W ∈ ℝn×d; h ∈ ℝd; G ∈ ℝn.\n- W2: grammatical errors\n\n  The sentence \"layer replaced the traditional FFN layer can be...\" has been revised to: \"The layer replacing the traditional FFN layer can be...\".\n- W3: missing connection in the main text\n\n  Sorry for the inconvenience, we will add in the revised manuscript.\n- W4: grammatical errors\n\n  Yes, the parameters alpha,beta ,k are fixed. We will correct the representations in the revised manuscript.\n\n- Q1: The configuration of alpha, beta ,k\n  We selected the optimal k based on the Silhouette Coefficient which evaluates the quality of clustering results. The hyperparameters α and β were tuned on a validation set: we tried several values (α, β ∈ {0.1, 0.01, 0.001}) and chose the combination that achieved the best performance on the validation set.\n- Q2 & L1: application to other PEFT methods\n\n  Thank you for raising this important point. While our current implementation of HiMoLE (Hierarchical Mixture of LoRA Experts) is tailored to the LoRA family, this design choice stems from the **inherent compatibility between Mixture-of-Experts (MoE) architectures and LoRA’s low-rank adaptation mechanism**. As shown in recent work ( [1]), **most existing MoE-based parameter-efficient fine-tuning (MoPE) frameworks, including ours, leverage LoRA due to its modularity and ease of integration with sparse expert routing**. \n  The hierarchical routing mechanism in HiMoLE relies on dynamically combining task-specific LoRA adapters. Direct extension to methods like BitFit or (IA)³ would require re-engineering their parameter update rules to align with MoE’s sparse activation paradigm, which is non-trivial. For instance, BitFit updates only bias terms, while (IA)³ introduces learned scaling vectors—neither naturally supports the \"expert\" abstraction central to MoE.\n  Despite this limitation, our work **diagnoses a critical, underexplored issue: the out-of-distribution (OOD) generalization vulnerability of existing LoRA and MoE-LoRA methods under distribution shift**. We propose a **novel hierarchical MoPE framework to mitigate gradient conflicts via structural sparsity, advancing the theoretical understanding of MoE’s role in LoRA robustness** .We appreciate the reviewer’s insight and agree that broader compatibility with PEFT methods is an exciting direction. Our work prioritizes rigor in addressing LoRA’s OOD limitations while laying a foundation for future MoPE frameworks.\n\nReferences:\n\n[1]A Survey on Mixture of Experts in Large Language Models"}, {"review_id": "xhXguBSj4p", "reviewer": "Reviewer_aQXK", "summary": "Authors propose a method for enhancing OOD prediction using MoE and PEFT. Their core idea is to use sentence level and token level routing for improving expert selection. The queries are routed to N group of M LoRA experts, done through two FFD networks.\n\nThe method is evaluated in three datasets, and compared to a base un-finetuned model, the simple LoRA, and two other variations of LoRA. A set of experiments are also reported on the selection of the experts and also on the effectiveness of the training algorithm.", "questions": "I have listed a list of references. Please let me know why they were not discussed or included as baseline models", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths:**\n\n-  The paper is easy to read\n-  The experiments are detailed\n-  The method is well explained, illustrated, and theoretically discussed\n\n**Weaknesses:**\n\n- Relevant work to OOD and PEFT [1,2,3] is not discussed, other related work to MoE is missing [4]. These are neither discussed nor used as baselines. Please correct me if I am wrong.\n- The results are not good. In NER, the base OOD model is working better. In SA, on one metric the base OOD model is better. In EQA, the model is working almost identical to the other baselines.\n- The core idea of the model (the sentence level and token level routing) is not new [5]\n\n[1] Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning, 2023\n\n[2] How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models, 2023\n\n[3] SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning, 2024\n\n[4] Mixture of LoRA Experts, 2024\n\n[5] Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference, 2021", "was_revised": true, "final_justification": "I increased my review score from 3 to 4. Please add the discussion regarding those models to the paper.", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n- W1 & Q1: Relevant work to OOD and PEFT\n\n  Due to space constraints of the main paper, we prioritized foundational and highly influential works in OOD generalization and  LoRA while adopting state-of-the-art Mixture of Parameter-Efficient Experts (MoPE) methods as baselines. Below, we clarify the reason:\n\n  1. References [1-2]: These works focus on OOD detection (e.g., model confidence calibration and thresholding for identifying distribution shifts), whereas our work addresses OOD generalization (improving robustness to unseen domains). Specifically:\n    - [1] investigates the correlation between OOD detection performance and pretrained language model (PLM) scale.\n    - [2] studies CLIP-based fine-tuning’s impact on OOD detection in few-shot settings.\n  By contrast, we diagnose OOD generalization limitations in LoRA-based methods and propose a novel hierarchical MoPE framework to alleviate these issues.\n  2. Reference [3]: This CLIP-based method implements sparsity via a masking strategy, which differs fundamentally from our modular, hierarchical design tailored for LoRA domains. Our approach optimizes parameter efficiency and generalization through dynamic expert activation, rather than masking predefined components.\n  3. Reference [4]: only  introduce the MoPE framework to the FFN layer of every Transformer block for all MoPE methods, while [4] integrates the MoPE framework to the entire Transformer layer. HiMoLE and the baselines included in the paper focus on the hierarchical routing strategy and hierarchical experts designwithin each layer, whereas [4] emphasizes architectural design across layers. Extending HiMoLE to full Transformer layers is planned for future work.\n- W2: The results are not good.\n\n  As discussed in Section 3.1 (\"Identification of OOD Generalization Problem in PEFT\") and illustrated in Figure 1(a-c), **standard LoRA exhibits significant performance degradation on OOD datasets** when applied to tasks requiring adaptation across diverse knowledge domains. This degradation manifests in **two key ways**: (1) the model performs worse than\n  its pre-fine-tuned counterpart, and (2) the performance gain on OOD data is substantially smaller than that on ID data. To address these limitations, we propose HiMoLE to **mitigate the OOD generalization issue**.  \n  While HiMoLE still incurs a performance drop on OOD datasets compared to the base model, it outperforms other LoRA-based methods, achieving up to **5.0%** absolute improvement over the best baseline under OOD settings. When the LoRA rank is set to 4, HiMoLE achieves a substantial improvement on the SA task under the OOD setting, increasing **from 30.1% to 59.4%**.This demonstrates its enhanced robustness to distributional shifts. \n  For the Extractive Question Answering (EQA) task, which exhibits similar challenges, we further conduct statistical significance tests to validate the consistency of HiMoLE’s performance gains. Specifically, we use the average of EM and ROUGE-2 as our evaluation metric and conduct a t-test. The results show that HiMoLE’s gains are statistically significant, with **p-values of 1.8×10⁻⁵ on in-distribution data and 2.3×10⁻⁶ on out-of-distribution data**, reinforcing its reliability in diverse OOD scenarios. \n- W3: The core idea of the model is not new\n\n  While sentence-level and token-level routing are established techniques, our key innovation is a hierarchical framework that synergistically unifies them to overcome the shortcomings of LoRA and earlier MoE methods under distribution shifts. In contrast, **existing MoE approaches either rely on mutually exclusive routing [5] without integration or restrict themselves to token-level routing only [6,7,8]**. As reviewer ADXP mentioned, **vulnerability of LLMs to distribution shift in the data** is an important problem and **yet less explored in the literature**. Specifically:\n  1. We first reveal a key trade-off between routing granularities in Section 3.1: Token-level routing delivers top in-distribution accuracy by using fine-grained context, but it overfits and performs poorly out-of-distribution. Sentence-level routing is more robust out-of-distribution through its focus on overall meaning, yet it loses in-distribution accuracy by overlooking local details. \n  2. To improve performance on both in-distribution (ID) and out-of-distribution (OOD) data, we propose a Hierarchical Routing Strategy that goes beyond the mutually exclusive routing in [5] and the purely token-level routing of existing LoRA MoE methods [6,7,8], including a two-stage cascaded mechanism:\n  - Stage 1 (Sentence-Level Routing): Dynamically assigns input sentences to specialized LoRA experts (In our work we refer it as Knowledge Cooperation Group(KCG)) based on global semantic patterns, ensuring robust OOD generalization.\n  - Stage 2 (Token-Level Routing): Within each KCG, fine-grained token-level gating further adapts local feature representations, preserving ID task precision.\n\nReferences:\n\n[1] Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning, 2023\n\n[2] How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models, 2023\n\n[3] SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning, 2024\n\n[4] Mixture of LoRA Experts, 2024\n\n[5] Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference, 2021\n\n[6] HydraLoRA: An Asymmetric LoRA Architecture for\nEfficient Fine-Tuning\n\n[7] MIXLORA: Enhancing Large Language Models\nFine-Tuning with LoRA-based Mixture of Experts\n\n[8] LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin"}, {"review_id": "fYN0yCGaa9", "reviewer": "Reviewer_CeY4", "summary": "This paper finds that vanilla LoRA and token-level MoE LoRA suffer from limited generalization under out-of-distribution (OOD) conditions. And this paper finds that sentence-level routing improves OOD robustness. Thus, it proposes HiMoLE, a hierarchical expert architecture which combines token-level MoE LoRA and sentence-level MoE LoRA, to enhance in-distribution performance while also ensuring robustness to out-of-distribution data. Moreover, it proposes a two-stage training strategy to enhance performance.", "questions": "For HiMoLE, could the authors clarify whether the maximum of 10,000 training steps refers to the combined total of both stage 1 and stage 2, or if it is exclusive to stage 2?", "limitations": "HiMoLE appears to be limited to knowledge-intensive tasks, and there is no experimental evidence showing that it can be extended to reasoning-intensive tasks such as math and code.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The motivation of this paper is relatively clear: token-level routing achieves better in-distribution (ID) performance but struggles to generalize to out-of-distribution (OOD) data. In contrast, sentence-level routing improves OOD robustness at the expense of ID accuracy. To address this trade-off, the paper proposes a hierarchical expert architecture and routing strategy that combines both token-level and sentence-level routing within a PEFT framework.\n\n2. This paper provides a theoretical explanation for why HiMoLE works, from the perspective of gradient conflict reduction enhancing model generalization.\n\nWeakness:\n\n1. Non-zero LoRA initialization can accelerate the training speed of LoRA [1]. The initialization of stage 1 may be the main factor contributing to HiMoLE, rather than the hierarchical expert architectures and routing strategies. The authors should include baseline comparisons that combine stage 1 initialization strategies with MixLoRA and HydraLoRA.\n\n2. Some ablation experiments should be added, such as comparing HiMoLE with the baseline under different LoRA ranks.\n\n3. It seems a bit unusual to me that the experiments use OneKE-13B for the NER task, while Llama2-7B is used for SA and EQA tasks. I wonder why a single model isn't used across all experiments.\n\n4. HiMoLE appears to be limited to knowledge-intensive tasks, and there is no experimental evidence showing that it can be extended to reasoning-intensive tasks such as math and code.\n\n[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. In NeurIPS 2024.", "was_revised": true, "final_justification": "The author has mostly resolved my concerns, especially _weakness 1_ that I was most worried about.", "rebuttal": "Thank you for your thoughtful feedback and constructive suggestions. Below, we address the concerns raised:\n- W1: Impact of Stage 1 Initialization\n\n  We appreciate the observation regarding the potential influence of non-zero LoRA initialization. To disentangle the effects of initialization from our hierarchical architecture, we add baseline comparisons where MixLoRA and HydraLoRA are trained with the same Stage 1 initialization as HiMoLE on NER task. Results is as follows, the metric used in the table is F1:\n\n  |  | Training Setting | ID | OOD |\n  | :--- | :--- | :--- | :--- |\n  | MixLoRA | two stage | 76.4| 63.0|\n  |  | one stage | 76.0 | 61.0 |\n  | HydraLoRA | two stage | 77.2 | 63.4 |\n  |  | one stage | 77.3 | 62.9 |\n  | HiMoLE | two stage | **77.9** | **65.3** |\n  |  | one stage | 61.7 | 53.5 |\n\n  Although other methods demonstrate improvements, the two-stage training approach combined with the hierarchical mixture of LoRA experts still delivers **the best performance across both in-distribution and out-of-distribution settings**, with essential improvement under OOD setting.\n- W2: Ablation Studies on LoRA Ranks\n\n  We agree that LoRA rank analysis is critical. We conduct experiments comparing HiMoLE with baseline methods using another LoRA rank(4) on the Sentiment Analysis (SA) task. As shown in the table below, HiMoLE achieves the best performance across different LoRA ranks, with a particularly significant margin(30.1% -> 59.4%) under the OOD setting, further validating the effectiveness of our method.\n\n  | Accuracy(%)| HiMoLE | MixLoRA | HydraLoRA |\n  | :--- | :--- | :--- | :--- |\n  | ID | 72.7 | 69.5 | 69.2 |\n  | OOD | **59.4** | 23.0 | 30.1 |\n- W3: Model Selection Across Tasks\n\n  The choice of different base models (OneKE-13B for NER, Llama2-7B for SA/EQA) was driven by **task requirements**:\n  - OneKE-13B: Specializes in entity-centric knowledge, making it ideal for NER and easy to adapt to medical NER domain. \n  - Llama2-7B: Strong in general language understanding, suitable for SA and EQA.\n  We note that HiMoLE’s architecture is model-agnostic, and its benefits hold across base models.\n- W4 & L1: Generalization to Reasoning-Intensive Tasks\n\n  Thank you for this valuable observation regarding the current scope of HiMoLE. We acknowledge that the experiments presented in the paper primarily focus on knowledge-intensive tasks involving domain shifts and factual robustness, as this is based on the former study [1] and aligns with our core design motivation: addressing the critical challenge of OOD generalization for knowledge-dependent capabilities. This focus is supported by prior research [2,3] indicating that **knowledge-intensive tasks often exhibit more severe OOD generalization degradation compared to reasoning-intensive tasks**, where LLMs tend to demonstrate stronger inherent generalization abilities. Consequently, methods specifically targeting knowledge robustness, like HiMoLE, are particularly relevant for this class of problems.\n\n  Regarding reasoning-intensive tasks (e.g., math, code),  the state-of-the-art performance on complex reasoning tasks often leverages sophisticated techniques like **verifier-guided reinforcement learning** [4], which operates under a different paradigm than the SFT-based PEFT methods central to our current HiMoLE investigation. Building upon this foundation and your insightful comment, our future work will actively explore integrating HiMoLE's principles with  reinforcement learning to enhance LLM's reasoning skills . \n- Q1: Training Step:\n  It refers to the combined total of both stage 1 and stage 2 for HiMoLE.\n\nReferences:\n\n[1] Revisiting Out-of-distribution Robustness in NLP: Benchmarks, Analysis, and LLMs Evaluations\n\n[2] Generalization vs memorization: Tracing language models’ capabilities back to pretraining data\n\n[3] Quantifying generalization complexity for large language models.\n\n[4] DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning"}, {"review_id": "RijuFxebXP", "reviewer": "Reviewer_Q7d6", "summary": "To address the generative power and robustness of traditional LoRA in OOD scenarios, this paper proposes HiMoLE, an optimized LoRA architecture that integrates hierarchical expert structures (Knowledge Competition Groups (KCGs) and Knowledge Collaboration Experts (KCEs)) with a two-tiered routing mechanism (sentence-level + token-level router). It consists of a two-stage training and diversity loss design, and is empirically evaluated to outperform existing methods on multiple NLP tasks.", "questions": "1. HiMoLE design is complicated, but the depth of ablation study is not deep enough. E.g., is the co-existence of “sentence-level + token-level” routing necessary? Does “cross-group collaboration” between KCEs lead to performance improvements? Is it better to replace diversity loss with negatively correlated regular terms?\n2. During the training process, the experts are fixed first and then tuned later: if there is a bias in the initial KCG, will the subsequent global tuning stage fall into local optimization?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. Robustness of LoRA under OOD (out-of-distribution) conditions is an important research issue.\n2. HiMoLE introduces a two-tier hierarchical structure, more fine-grained expert design, and hierarchical routing to enable the model to better handle semantic heterogeneity.\n3. This paper provides theoretical analysis and experimental evaluation to show the performance of HiMoLE under ID and OOD conditions.\n\nCons:\n1. Missing overhead analysis: HiMoLE introduces hierarchical expert structure + hierarchical routing + two-stage training, which introduces large amount of parameter routing computation, expert switching overhead and pre-clustering tuning.\n2. Lacking sensitivity to the number of clusters N and experts M, i.e., lack of sensitivity analysis for K-Means clustering.\n3. The theoretical part neglects practical analysis and empirical proofs. E.g., Lemma 1 and Theorem 2 belong to the “generalized” SGD generalization framework without combining with the practical structure or the features of the loss function. \n4. The paper piles up too much customized terminology.\n5. For evaluation, the OOD of the NER, SA, and QA is still a controlled division within the domain, although it has been transformed into a distribution. More extreme OOD settings can be added, such as domain transfer (e.g., migrating from legal QA to medical QA), and practical user dataset.", "was_revised": true, "final_justification": "Thanks for your detailed reply. At this stage I think my concerns are mostly addressed.", "rebuttal": "Thank you for your thoughtful feedback and constructive suggestions. Below, we address the concerns raised:\n- W1: Missing overhead analysis\n\n  We acknowledge the lack of explicit discussion on computational overhead and  analyze the costs introduced by HiMoLE’s hierarchical structure as below:\n  - Parameter Routing and Inference scalability: HiMoLE adds **lightweight gating networks, maintaining nearly the same parameter count as MixLoRA**. For inference, the speed is **primarily influenced by the base model**. Since the parameters of PEFT modules constitute a small fraction of the total model parameters (ranging from 0.65% to 1.05% as shown in the following Table), the inference latency differences across are minimal. The following presents the latency and parameter count during inference using Llama2-7B with different Mixture of LoRA Experts methods, evaluated on the OPTIMISM dataset using a single RTX 4090 GPU.  Latency was recorded over 50 random samples. The results show **nearly equal latency, but HiMole exhibits the highest model performance**.\n\n    |  | HiMole | MixLoRA | HydraLoRA |\n    | :--- | :--- | :--- | :--- |\n    | %Param | 1.05 | 1.05 | 0.65 |\n    | Latency(s) | 127 | 119 | 122 |\n    | Performance(%) | 68.8 | 66.5 | 67.2 |\n\n  - Two-Stage Training: The two-phase training incurs extra cost, but Stage 1 is amortized across downstream tasks. Stage 2 converges faster than end-to-end training due to better initialization. Meanwhile，the pre-clustering encoder's parameter count (e.g., 500M params) and FLOPs are orders of magnitude smaller than modern LLMs (e.g., 7B-13B params), **making clustering computationally trivial**. In our experiments, pre-clustering consumed < 2% of total training time (e.g., 0.2 GPU hours vs. 12 hours on NER task ). As validated in [1], **the two-stage training approach is more compute efficient** than training a larger generalist LLM.\n  We acknowledge HiMoLE is not zero-overhead, but **its costs are bounded**. Crucially, these overheads are **dwarfed by HiMoLE’s robustness gains under distribution shift**. \n\n- W2: Lack of sensitivity analysis\n\n  We acknowledge the lack of sensitivity experiments. The number of clusters (N) was determined using the silhouette coefficient describing the clustering quality during the pre-clustering stage.  As validated in the Section 4.5 of HydraLoRA[2], the number k of clusters is NOT a sensitive parameter with a wide range of reasonable number k of clusters performing decently well in all settings; K-means is simple but effective with the  sophisticated hyper-parameter search approaches.\n\n  We further evaluate HiMoLE with M=6 experts and the LoRA rank of 4, as shown in the table below. The results demonstrate that HiMoLE exhibits **stronger robustness to variations in the number of experts and LoRA rank**, while consistently achieving the best performance across all configurations. Notably, under the OOD setting, it delivers a substantial performance gain (31.3% → 60.0%), further validating the effectiveness of our method.\n\n  | Setting | Accuracy(%)| HiMoLE | MixLoRA | HydraLoRA |\n  | :--- |:--- | :--- | :--- | :--- |\n  | M=4 | ID | 72.7 | 69.5 | 69.2 |\n  |  | OOD | **59.4** | 23.0 | 30.1 |\n  | M=6 | ID | 71.8 | 71.0 | 71.6 |\n  |  | OOD | **60.0** | 27.3 | 31.3 |\n- W3: Lack of practical analysis and empirical proofs\n\n  We appreciate the your emphasis on practical alignment. While we employ a generalized SGD framework, we explicitly bridge this theory to HiMoLE’s structure through the following connection:\n  1. We acknowledge that the presentation in the manuscript focuses primarily on establishing the core theoretical generalization guarantees within a generalized SGD framework for our specific problem setting (Lemma 1, Theorem 2). We agree that explicitly incorporating the exact structural details of the loss function into these specific lemmas/theorems would further strengthen the practical connection. However, we respectfully clarify that the core theoretical mechanism governing generalization improvement identified in our analysis – **the reduction of generalization error driven by a decrease in the variance of the stochastic gradients** – is a fundamental and widely applicable principle in SGD analysis, largely independent of the specific loss function structure[3]. This principle justifies the applicability of the generalized framework in our work.\n  2. As shown in Table 3, in the ablation of diverse loss, where we maintain the same loss features, it still outperformers other MOPE methods on OOD dataset.\n- W4: Too much customized terminology\n\n  Thank you for highlighting this issue. We acknowledge that the initial draft overused specialized terminology, which may have hindered readability. To address this, we will simplify the language in the revised manuscript.\n- W5: Evaluation on extreme OOD data\n  \n  Thank you for the suggestion. We further evaluate our model on a challenging out-of-distribution (OOD) dataset from a different domain—specifically, the Politics subset of CrossNER [4]. Results show that while our method still improves performance on these extreme OOD samples, the gains are more modest compared to OOD cases within the domain.\n\n  |  | Base Model | HiMole | MixLoRA | HydraLoRA |\n  | :--- | :--- | :--- | :--- | :--- |\n  | F1(%) | 61.5 | 66.9 | 65.8 | 66.5 |\n\n- Q1: Ablation study\n  \n  As shown in Fig 1.d and illustrated in Section 3.1, there is **a key trade-off between routing granularities** : Token-level routing delivers top in-distribution accuracy by using fine-grained context, but it overfits and performs poorly out-of-distribution. Sentence-level routing is more robust out-of-distribution through its focus on overall meaning, yet it loses in-distribution accuracy by overlooking local details. Hence we integrate sentence-level routing and token-level routing in a hierarchical manner to enhance performance on both in-distribution (ID) and out-of-distribution (OOD) data.\n  \n  To validate that cross-group collaboration among KCEs contributes to performance gains, we conduct an ablation study on the NER task. Specifically, we compare HiMoLE against the single KCG setting(which lacks cross-group collaboration) on the subdataset aligned with the KCG’s distribution. As shown in the following table, the substantial performance improvement confirms that cross-group collaboration enhances model effectiveness. \n  \n  Regarding the loss function, the diverse loss **encourages expert groups to specialize in distinct task clusters**, thereby preventing redundancy and enhancing performance, which has been proved in Section 4.5. While negatively correlated regularization could theoretically achieve a similar effect, we found that **applying it to LLMs introduces computational overhead in practice** due to the cost of calculating parameter-wise negative correlations during training.\n\n  |  | ID1 | ID2 | ID3 | ID4 |\n  | :--- | :--- | :--- | :--- | :--- |\n  | KCG | 85.3 | 47.5 | 69.5 | 71.2 |\n  | HiMoLE | 87.6 | 64.0 | 75.3 | 79.6 |\n\n- Q2: Initialization\n\n  We appreciate the reviewer’s insightful question regarding the sensitivity of our two-stage training approach (KCG initialization and global fine-tuning) to potential bias in the initial KCG. While a severely biased initial KCG could theoretically limit the solution space, our design incorporates global fine-tuning that mitigate this risk and promote convergence toward a robust optimum.  the global fine-tuning stage does not freeze the expert parameters (KCG). All the adapter parameters are updated via backpropagation during global SGD. This end-to-end training allows the model to: Adjust the parameters of the KCG to better align with the integrated task objectives; Refine how information flows between experts based on the global loss signal. As validated in [1], **the two-stage training manner outperforms other training methods across various tasks**. Meanwhile, we further conduct two-stage training on MixLoRA and HydraLoRA, both exhibits improvement:\n\n  |  | Training Setting | ID | OOD |\n  | :--- | :--- | :--- | :--- |\n  | MixLoRA | two stage | 76.4 | 63.0 |\n  |  | one stage | 76.0 | 61.0 |\n  | HydraLoRA | two stage | 77.2 | 63.4 |\n  |  | one stage | 77.3 | 62.9 |\n  | HiMoLE | two stage | 77.9 | 65.3 |\n  |  | one stage | 61.7 | 53.5 |\n\n[1]Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM\n\n[2]HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning\n\n[3]Exploring Generalization in Deep Learning\n\n[4]Evaluating Cross-Domain Named Entity Recognition"}]

Author rebuttal:

Meta-review: This paper introduces a framework (HiMoLE) to improve the out-of-distribution (OOD) robustness of LoRA. HiMoLE proposes a hierarchical mixture of LoRA experts and includes sentence-level and token-level routing. Authors also provides a specialized two-stage algorithm to train this model. As highlighted by the reviewers, the problem studied in the work is important and clearly motivated, and the empirical evidence is promising.

Reviewer CeY4 worried that the performance gains were due to the two-stage initialization rather than the architecture; the authors addressed this by providing new ablation study. The same reviewer also highlighted that authors do not study reasoning tasks and the authors concurred that HiMoLE is motivated with knowledge tasks and it may not work for reasoning tasks. Reviewer aQXK questioned the novelty and results, but was convinced after the authors clarification. Reviewer Q7d6 called for deeper analysis of overhead, hyperparameters, and more extreme OOD settings, all of which the authors provided with new experiments. Finally, Reviewer ADXP's concern about applicability beyond LoRA was addressed by explaining the unique architectural compatibility, which the reviewer accepted. Authors are highly encouraged to include these in the next revision.
Decision: Accept (poster)