Title: HiMoLE: Towards OOD-Robust LoRA via Hierarchical Mixture of Experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: OOD Robustness

Large Language Models

Parameter-Efficient Fine-Tuning

Mixture of Experts.
Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have enabled the efficient adaptation of large language models (LLMs) by updating only a small subset of parameters. However, their robustness under out-of-distribution (OOD) conditions remains insufficiently studied. In this paper, we identify the limitations of conventional LoRA in handling distributional shifts and propose $\textbf{HiMoLE}$($\textbf{Hi}$erarchical $\textbf{M}$ixture of $\textbf{L}$oRA $\textbf{E}$xperts), a new framework designed to improve OOD generalization. HiMoLE integrates hierarchical expert modules and hierarchical routing strategies into the LoRA architecture and introduces a two-phase training procedure enhanced by a diversity-driven loss. This design mitigates negative transfer and promotes effective knowledge adaptation across diverse data distributions. We evaluate HiMoLE on three representative tasks in natural language processing. Experimental results evidence that HiMoLE consistently outperforms existing LoRA-based approaches, significantly reducing performance degradation on OOD data while improving in-distribution performance. Our work bridges the gap between parameter efficiency and distributional robustness, advancing the practical deployment of LLMs in real-world applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the generative power and robustness of traditional LoRA in OOD scenarios, this paper proposes HiMoLE, an optimized LoRA architecture that integrates hierarchical expert structures (Knowledge Competition Groups (KCGs) and Knowledge Collaboration Experts (KCEs)) with a two-tiered routing mechanism (sentence-level + token-level router). It consists of a two-stage training and diversity loss design, and is empirically evaluated to outperform existing methods on multiple NLP tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. Robustness of LoRA under OOD (out-of-distribution) conditions is an important research issue.\n2. HiMoLE introduces a two-tier hierarchical structure, more fine-grained expert design, and hierarchical routing to enable the model to better handle semantic heterogeneity.\n3. This paper provides theoretical analysis and experimental evaluation to show the performance of HiMoLE under ID and OOD conditions.\n\nCons:\n1. Missing overhead analysis: HiMoLE introduces hierarchical expert structure + hierarchical routing + two-stage training, which introduces large amount of parameter routing computation, expert switching overhead and pre-clustering tuning.\n2. Lacking sensitivity to the number of clusters N and experts M, i.e., lack of sensitivity analysis for K-Means clustering.\n3. The theoretical part neglects practical analysis and empirical proofs. E.g., Lemma 1 and Theorem 2 belong to the “generalized” SGD generalization framework without combining with the practical structure or the features of the loss function. \n4. The paper piles up too much customized terminology.\n5. For evaluation, the OOD of the NER, SA, and QA is still a controlled division within the domain, although it has been transformed into a distribution. More extreme OOD settings can be added, such as domain transfer (e.g., migrating from legal QA to medical QA), and practical user dataset.\n\nQUESTIONS:\n1. HiMoLE design is complicated, but the depth of ablation study is not deep enough. E.g., is the co-existence of “sentence-level + token-level” routing necessary? Does “cross-group collaboration” between KCEs lead to performance improvements? Is it better to replace diversity loss with negatively correlated regular terms?\n2. During the training process, the experts are fixed first and then tuned later: if there is a bias in the initial KCG, will the subsequent global tuning stage fall into local optimization?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper finds that vanilla LoRA and token-level MoE LoRA suffer from limited generalization under out-of-distribution (OOD) conditions. And this paper finds that sentence-level routing improves OOD robustness. Thus, it proposes HiMoLE, a hierarchical expert architecture which combines token-level MoE LoRA and sentence-level MoE LoRA, to enhance in-distribution performance while also ensuring robustness to out-of-distribution data. Moreover, it proposes a two-stage training strategy to enhance performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The motivation of this paper is relatively clear: token-level routing achieves better in-distribution (ID) performance but struggles to generalize to out-of-distribution (OOD) data. In contrast, sentence-level routing improves OOD robustness at the expense of ID accuracy. To address this trade-off, the paper proposes a hierarchical expert architecture and routing strategy that combines both token-level and sentence-level routing within a PEFT framework.\n\n2. This paper provides a theoretical explanation for why HiMoLE works, from the perspective of gradient conflict reduction enhancing model generalization.\n\nWeakness:\n\n1. Non-zero LoRA initialization can accelerate the training speed of LoRA [1]. The initialization of stage 1 may be the main factor contributing to HiMoLE, rather than the hierarchical expert architectures and routing strategies. The authors should include baseline comparisons that combine stage 1 initialization strategies with MixLoRA and HydraLoRA.\n\n2. Some ablation experiments should be added, such as comparing HiMoLE with the baseline under different LoRA ranks.\n\n3. It seems a bit unusual to me that the experiments use OneKE-13B for the NER task, while Llama2-7B is used for SA and EQA tasks. I wonder why a single model isn't used across all experiments.\n\n4. HiMoLE appears to be limited to knowledge-intensive tasks, and there is no experimental evidence showing that it can be extended to reasoning-intensive tasks such as math and code.\n\n[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. In NeurIPS 2024.\n\nQUESTIONS:\nFor HiMoLE, could the authors clarify whether the maximum of 10,000 training steps refers to the combined total of both stage 1 and stage 2, or if it is exclusive to stage 2?\n\nLIMITATIONS:\nHiMoLE appears to be limited to knowledge-intensive tasks, and there is no experimental evidence showing that it can be extended to reasoning-intensive tasks such as math and code.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
Authors propose a method for enhancing OOD prediction using MoE and PEFT. Their core idea is to use sentence level and token level routing for improving expert selection. The queries are routed to N group of M LoRA experts, done through two FFD networks.

The method is evaluated in three datasets, and compared to a base un-finetuned model, the simple LoRA, and two other variations of LoRA. A set of experiments are also reported on the selection of the experts and also on the effectiveness of the training algorithm.

STRENGTHS AND WEAKNESSES:
**Strengths:**

-  The paper is easy to read
-  The experiments are detailed
-  The method is well explained, illustrated, and theoretically discussed

**Weaknesses:**

- Relevant work to OOD and PEFT [1,2,3] is not discussed, other related work to MoE is missing [4]. These are neither discussed nor used as baselines. Please correct me if I am wrong.
- The results are not good. In NER, the base OOD model is working better. In SA, on one metric the base OOD model is better. In EQA, the model is working almost identical to the other baselines.
- The core idea of the model (the sentence level and token level routing) is not new [5]

[1] Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning, 2023

[2] How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models, 2023

[3] SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning, 2024

[4] Mixture of LoRA Experts, 2024

[5] Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference, 2021

QUESTIONS:
I have listed a list of references. Please let me know why they were not discussed or included as baseline models

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with OOD perfomance of the LoRA  finetuning frameworks. The OOD performance of LoRA is  first shown  empirically to be limited and then  by means of a  mixure-of-experts scheme the  OOD performance is improved.\n\nSTRENGTHS AND WEAKNESSES:\nVulnerability of  LLMs  to  distribution shift in the  data is  an important problem and yet less explored in the  literature, therefore this paper is of high interest to the community. The strength is showing theoretically that  hierarchical routing reduces the prevalence of conflicting gradients by inducing structural sparsity in expert selection and this is directly connected to generalization error. \n\n- in (2)-(3) please mention the  sizes of matrices W,h,G, for better readablity.\n- please proofread the paper for grammatical errors: \"layer replaced the traditional FFN layer can be..\"\n- the connection between  (10) and (11) is  related to lemma 4 in the appecdix. Not including it in the main text is like a missing part. In particular, how SimGrad and V(.) are connected should be clearly discussed in the main text.\n- In algorithm1,  are the parameters  Alpha,beta ,k   fixed and  fed in as the inpute? please  correct this.\n-In the proof of lemma 4, there seems to be  some  approximation which  should be denoted by ~ and not =.\n\nQUESTIONS:\n- The nuance parameter ‘k’ in eq(4), how do you choose it? have you performed any  optimization on it to find the optimal value? same for $\\alpha$ and $\\beta$  in the loss function.\n- You have tested your  scheme only on LoRA family, while there are other  PEFT methods which have equivalent performance with fewer trainable params such as BitFit, RoCoFT, (IA)3. Is  your method   extendible  directly to other PEFT methods?\n\nLIMITATIONS:\nThe HiMoLE  scheme  introduced in this paper is only applicable on LoRA family PEFT.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}