HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts
Authors: Mengqi Liao
Wei Chen
Junfeng Shen
Shengnan Guo
Huaiyu Wan
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Multi-Task Learning
Parameter-Efficient Fine-tuning
Mixture of Experts
Primary area: foundation or frontier models, including LLMs

Abstract: Recent studies have combined Mixture of Experts (MoE) and Parameter-Efficient Fine-tuning (PEFT) to fine-tune large language models (LLMs), holding excellent performance in multi-task scenarios while remaining resource-efficient.  However, existing MoE approaches still exhibit the following limitations: (1) Current methods fail to consider that different LLM layers capture features at varying levels of granularity, leading to suboptimal performance. (2) Task-level routing methods lack generalizability to unseen tasks. (3) The uncertainty introduced by load imbalance loss undermines the effective specialization of the experts. To address these challenges, we propose HMoRA, a Hierarchical fine-tuning method that combines MoE and LoRA, employing hybrid routing that integrates token-level and task-level routing in a hierarchical manner. This hierarchical hybrid routing allows the model to more efficiently capture both fine-grained token information and broader task contexts. To improve the certainty of expert selection, a novel routing auxiliary loss is introduced. This auxiliary function also enhances the task router's ability to differentiate tasks and its generalization to unseen tasks. Additionally, several optional lightweight designs have been proposed to significantly reduce both the number of trainable parameters and computational costs. Experimental results demonstrate that HMoRA outperforms full fine-tuning across multiple NLP benchmarks, while fine-tuning only 3.9\% of the parameters. The code is  available on: https://github.com/LiaoMengqi/HMoRA.

Published as a conference paper at ICLR 2025
HMORA: MAKING LLMs MORE EFFECTIVE WITH HI- ERARCHICAL MIXTURE OF LORA EXPERTS
Mengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo & Huaiyu Wan' Department of Computer Science
Beijing Jiaotong University
{maliao, w_chen, j fsshen, guoshn, hywan)ynnn ) jtu. wwwp:tt.uuuu... edu.cn
ABSTRACT
Recent studies have combined Mixture of Experts (MoE) and Parameter-Effiiien Fine-tuning (PEFT) to fine-tune large language models (LLMs), holding excellent performance in multi-task scenarios while remaining resource-effiiient However, existing MoE approaches still exhibit the following limitations: (1) Current meth- ods fail to consider that different LLM layers capture features at varying levels of granularity, leading to suboptimal performance. (2) Task-level routing meth- ods lack generalizability to unseen tasks. (3) The uncertainty introduced by load imbalance loss undermines the effective specialization of the experts. To address these challenges, we propose HMoRA, a Hierarchical fine-tuning method that combines MoE and LoRA, employing hybrid routing that integrates token-level and task-level routing in hierarchical manner. This hierarchical hybrid routing allows the model more efficiently capture both fine-grained token information and broader task contexts. To improve the certainty of expert selection, a novel routing auxiliary loss is introduced. This auxiliary function also enhances the task router's s ability to differentiate tasks and its generalization to unseen tasks, Addi- tionally, several optional lightweight designs have been proposed to significantly reduce both the number of trainable parameters and computational costs. Ex- perimental results demonstrate that HMoRA outperforms full fine-tuning across multiple NLP benchmarks, while fine-tuning only 3.9% of the parameters. The code is available on: htttp:://iithu. com/ / Li MaiaMMMggi/HHooR
INTRODUCTION
Large language models (LLMs) have made significant strides and achieved impressive capabilities various natural language processing (NLP) (Liu et al. 2023 Touvron et al. 2023 Team et al.) 2024) tasks, such machine translation, text generation, and question answering. However, with models reaching tens or hundreds of billions of parameters, the computational and memory costs for training have increased substantially (Kaplan al. 2020). Therefore, reducing these costs while maintaining performance has become a critical challenge.
In recent years, Mixture of Experts (MoE) (Jacobs et al. 1991 Ct 2024 has attracted growing interest in the research community. The MoE is a neural network architecture that activates a subset of expert modules for each input, utilizing routing mechanism to determine which experts en- gaged. MoE models enable the expansion of model capacity without significantly increasing compu- tational costs. Recent studies have successfully integrated MoE into LLMs, significantly improving scalability and efficiency (Lepikhin et al.) 22200] Du et al., [2022] Jiang et al. (2024). enhancing gen- eralization in multi-task learning (Fedus et al. 2022). Furthermore, Shen et al. (2023) demonstrates additional performance gains in LLMs by combining MoE with instruction fine-tuning. However, due to the presence of multiple expert modules, standard MoE models have an extremely large number of parameters, which poses challenges for storage, deployment, and practical application. Recent studies have combined Low-Rank Adaptation (LoRA) (Hu et al. (02T) with MoE for large language model fine-tuning leveraging LoRA's parameter efficiency alongside MoE's ' ability to en-
"Corresponding author.

---

Published as a conference paper at ICLR 2025
hance performance in complex and multi-task scenarios Zadouri al. 2023 Li al.
2024a
Tian
et al. 2025).
However, existing MoE models still face several challenges. (1) Combining MoE routing at differ- ent granularities, such as token-level and task-level, allows the model to more effectively capture information across diverse representational hierarchies (Kudugunta et al. 2021). However, existing MoE LLMs that employ multi-granular routing (Ren et al.] 2023) fail to account for the fact that different layers of LLM capture features distinct granularities (Geva et al. 2021). Consequently, the efficiency of capturing multi-granular information remains suboptimal. (2) What's more, exist-
ing task-level routing MoE methods rely on task labels, which restricts their capacity to generalize to unseen tasks (Ren et al.
[0223] Feng et al., [2024] Liu et al. (024). Some methods (Kudugunta 2023) eliminate reliance on task labels by using sentence representa- tions for routing, a technique known as sentence-level routing. However, experiments show that sentence-level routing leads to suboptimal performance. (3) Additionally, existing load balancing
jet al., 2021; Zadouri et al.
loss functions (Shazeer al., 2017: Fedus et al. 2022) often result in a lack of certainty in routing results, leading to unstable routing and undermining the specialization of experts.
To address these challenges, we propose a hybrid routing method that combines task-level and token- level routing in a hierarchical manner, allowing shallow layers of LLMs capture fine anee-griied token- level nuances, while deeper layers focus on broader task-level understanding. To tackle the issue of uncertainty in existing routing methods, a novel auxiliary function is proposed to enhance the certainty of expert selection and maintain balanced experts selection, improving expert specializa- tion. Moreover, although our task routing is based on sentence representations, the auxiliary loss enhances the ability of task router to distinguish between different tasks in an unsupervised manner and demonstrates generalization to unseen tasks. Additionally, some lightweight designs are offered to reduce both the trainable parameters and computational costs without significantly compromising performance. Due to the high cost of training a standard MoE model from scratch, we validate the effectiveness of our proposed method in the context of fine-tuning. Specifically, we implement our approach as a PEFT method that integrates MoE and LoRA, which we refer to as HMoRA. Our contributions can be summarized as follows:
We propose a hierarchical hybrid routing mechanism that more efficiently captures infor- mation at different granularities across various layers of LLMs.
We introduce novel auxiliary function that enhances the certainty of routing methods while maintaining balance in the experts selection, thereby improving expert specialization.
By incorporating our auxiliary loss, the task router can learn to differentiate tasks in an unsupervised manner and generalize to unseen tasks.
We provide several optional lightweight designs that further reduce both trainable parame- ters and computational costs.
We train on a multi-task dataset and evaluate performance across various NLP benchmarks. With only 3.9% of the parameters compared to full fine-tuning, our method outperforms full fine-tuning on multiple benchmarks.
2 PREILIMINARY
Our work builds widely adopted causal decoder L.MMs (Radford et al.) 2019 Touvron al. 2023 Yang et al.) 2024). functioning as a plugin integrated into the dense layers of these models. Below, we will briefly introduce the dense layers in causal decoder LLMs and the mutil-task fine-tuning.
Dense layers in Causal Decoder LLMs. Causal decoder-based LLMs are built on the Transformer decoder architecture. Each layer contains self-attention mechanism and feed-forward network (FFN). The self-attention mechanism can vary across different LLMs, such as the multi-head at- tention or the multi-query attention (Ainslie al. 2023). In general, involves three dense layers (W,, q. Wt. W:) computing the query, key, and value, and a fourth dense layer (W.) aggregating the attention heads. FFN typically consists of two dense layers with non-linear activation function, as described by the following equation:
FFN(X) 6(WW,,)) W Wown.
2

---

Published as a conference paper at ICLR 2025
Here, X is the imput to the FFN. While some LLMs may include bias terms in these dense layers, they are omitted here for simplicity. If the activation function ø is a gated activation function like SwiGLU (Shazeer 2020), it introduces an additional dense layer, W gate. Our work focus these dense layers (Wg. WK. W., Wo. Wap, Wdown. and W Wgate) where HMoRA will be integrated.
Supervised Fine-Tuning Supervised fine-tuning is method specifically tailored to adapt LLMs to generate outputs that align with given instructions or prompts. Multi-task fine-tuning trains the acquire general problem-solving skills
model on a variety of tasks and input formats, enabling rather than being specialized to single dataset (Wei et al. 202T; Chung et al. 20224) During fine- tuning, each training sample consists of an input token sequence ' and corresponding target token sequence 7 The loss function for fine-tuning is defined as:
LIM Σ log( T :T)))))
Here, is the length of the target sequence TH, PLM(-) represents the predicted probability of the i-th target token T, conditioned on the input sequence T and all predicted tokens in the target sequence. The model is optimized to maximize the likelihood of generating the correct target sequence.
3 THE HMORA METHOD
In this section, will elaborate on HMoRA illustrated in Figure I We first introduce how to combine LoRA and MoE, leveraging the strengths of both approaches (Section 3.1). Then, we present the hybrid routing mechanism which combines both token-level and task-level routing in hierarchical manner to capture fine-grained and broader contextual information (Section Next, introduce a novel auxiliary loss aimed enhancing routing certainty while maintaining bal- anced selection of experts, thereby improving expert specialization. The combination of our auxil- iary loss with hybrid routing enhances the task router's 's ability distinguish between tasks and even generalize to unseen tasks, improving overall performance in multi -tak scenarios (Section 3.3
3.1 MIXTURE OF LORA EXPERTS
We insert MoRA blocks (mixture LoRA expert blocks), which consist of set of LoRA experts and a router Rt as plugins into the dense layers of LLMs. This combination of LoRA and MoE leverages the parameter efficiency of LoRA while benefiting from the strong multi-task performance of MoE. The forward pass for each expert is defined as:
E = WW..WB.
where WA. E Rr and WB, E Brrd are low-rank matrices, with < dmn doa- WA. is randomly initialized, while WR, is set to zero, ensuring consistency with the pre-trained state at the start of fine-tuning. X is the input to the dense layer, with dimension of din, and is processed in batches. The forward pass of the dense layer, with the MoRA block inserted, is defined as:
Y Σ E,E,
(4)
2 = XW +Y,
(5)
where E, the output of the i-th expert, 9s represents the gate value computed by the router for the i-th expert, is the number of experts, and W is the original dense layer. The final output Z denotes the combination of the expert outputs Y with the output of original dense layer. During training and inference of LLMs, we replace the original dense layer output with Z. Here, freeze W and only update the parameters of experts and router, significantly reducing the parameter count compared to full fine-tuning and standard MoE approaches.
The router R can be viewed as a function fa(-) Rd + R that maps the input X to a distribution g E R where g represents the gate values used to select experts. In this paper, we investigate soft routing and top-k routing, combined with different auxiliary losses, as the routing methods for the routers. A detailed explanation of routing methods is provided in Appendix A
3

---

Published as a conference paper at ICLR 2025
Transformtt Ducodie in LAMs
Moli
Tiak
Takes
PFN
plo
pectralned wettght
Task R
Tokes
Self Actention
Tokan
Taakllval Ronter
Token-kevel Router
Honee
Bncodar
Clask mpposabattioe currapuraaing 211)
PEN
Bmbbed "an tmbbadytt
(DFihlee state cotteemmaatttcccnnaaaata cornepeenling tucens]
Self Attention
Embeddinns:
Proau
Tisk
Fast
Bmbeeky
Embesig
Imbbud Yark
Token &
The HMoRA architecture combines token-level and task-level routing, utilizing a task encoder and task embedding obtain task representations. MoRA blocks integrated into the dense layers of LLMs.
Figure 1:
3.2 HIERARCHICAL HYBRID ROUTING
Most previous MoE methods use token-level routing, which helps experts learn token-level features but fails capture task-level information. Methods like MoELoRA (Liu et al. (024) and MoA (Feng et al. 2024) use task-level routing but overlook fine-grained token-level details, crucial for capturing data subtleties. Although Ren et al. (2023) and Kudugunta al. 2021) incorporate both token-level and task-level routing, they overlook the fact that shallow layers in LLMs primarily cap- ture token-level information, while deeper layers focus more on semantic-level information (Geval it al. (021I. To address these limitations, we propose a hierarchical hybrid approach that more effectively exploits multi granular information for routing.
To perform level routing, we first need to derive task representation from the input. We define task embedding fusk. which concatenated with the embeddings of the input tokens 7 and processed by task encoder, TaskEncoder ). This process is formalized as
(6) where TaskEncoder(-) can be single or multi-layer Transformer encoder. The output of the task encoder corresponding to Eusk serves as the task representation. Embed(-) is the LLM's embed- ding layer, with fusk being trainable while the rest of the embeddings remain frozen. We initialize Cask using the question mark symbol's 's embedding, providing meaningful starting point for task differentiation.
Husk Task
Cusk))
To combine token-level and task-level routing, we merge a task router R, and a token router R, into unified router. The task router computes the task-level routing results, denoted as glsk - JR, (mssk), ) which is calculated once for each input T The token router computes the token-level routing results, denoted as Iteken fR, (X), which is calculated for each token in both T and T We then combine the task-level and token-level routing results to form the final gate values for each token,
g a Pask (1 a gtokens
where (1)) represents the proportion of the two types of routing results and / indicates that the router is at Layer of the LLM. We define (1) as follows:
1 ( (-e+2xEx) +
Here, a(-) represents the sigmoid function, and L the total number of layers in the LLM. and I are hyperparameters that flexibly control the variation of (1) as changes. This allows different

---

Published as a conference paper at ICLR 2025
deep layer
2000
Trundats
Attention
Router
Time
shallow layer
Noun Verh?
750
Self Attention
500
Routee
Startoo
Router
(a)
(b)
Figure 2: (a) More hierarchical LLMs by applying hierarchical hybrid routing, where shallow layers focus on fine-grained token-level distinctions and deeper layers shift towards broader task-level understanding. (b) A more lightweight architecture by router sharing, applying single LoRA to Wo, Wown and using hydra LoRA. (c) A comparison the time required for Ik steps of training and the proportion of trainable parameters across models with different lightweight designs.
layers to capture information at varying granularities, rather than uniformly combining both types of information across all layers. Further explanations and examples are provided in Appendix B Experiments in Appendix E.5 demonstrate that increasing a! with / improves model perfor- mance. Under this setup, shallow layers focus more on token-level information, while deeper layers emphasize task-level information, as shown in Figure 2(a)
Additionally, in Appendix C we introduce series of lightweight designs to further reduce the number of learnable parameters and enhance computational efficiency. Figure((b) provides brief overview of these designs. comparison their training time and parameter efficiency is shown in Figure 2(c)
3.3 ENHANCE CERTAINTY AND MAINTAIN BALANCE OF ROUTING RESULTS
In this section, will discuss the limitations of soft routing and top-k routing introduced in Ap- pendix A and propose novel auxiliary function to address these issues. Moreover, we find that combining the auxiliary loss with the task router enhances its ability differentiate between tasks and allows it to generalize to unseen tasks.
We begin by introducing the concept of entropy (Shannon 1948), which measures the certainty of probability distribution. Given the gate values produced by router, the entropy of is calculated
H Σ 99 log(g;).
where e is the number of experts. The maximum entropy loge indicates maximum routing uncertainty (uniform distribution), while the minimum entropy of 0 represents complete cer- tainty in expert selection (point distribution). Consider a router performing N routing opera- tions in a batch, resulting in a set of distributions = =(g")....,gNN)) The average distribution N. E (1) reflects the balance in routing decisions. The closer this average is to uniform distribution, the more balanced the selection of experts, with the entropy approaching loge.
We conduct experiments to assess the certainty and balance of top-k and soft routing (see Sec- (... In soft routing, the shallow layers of LLMS exhibit near-uniform gate values, with entropy approaching log e, indicating random expert selection and lack of specialization. In deeper layers, while certain experts are more clearly preferred, the selection becomes imbalanced, with a small subset of experts overused, leading to underutilization of others and reduced model performance. A similar issue is observed in top-k routing (Figure (3(b).. Although the load balancing loss helps mitigate imbalance, it comes the cost of reduced routing certainty.
5

---

Published as a conference paper at ICLR 2025
To address these issues, propose novel auxiliary loss function that not only ensures balanced expert selection but also promotes greater certainty in routing results, thereby encouraging more effective specialization of experts across all layers.
To introduce our auxiliary loss, we first present the Generalized Jensen-Shannon Shannon (GJS) divergence (Nielsen & Nock 2009), extension of the JS divergence (Lin 1991), which measures the similar- ity across multiple probability distributions. In GJS, each distribution assigned a weight, with the sum of these weights equal I. We assign weight of T each gate values. The GJS divergence for gate values is computed as:
1 E g
GJS(G) = H
H)
(10)
> encourages the average distribution to approximate uniform distribution, promoting the balancing of experts selection. Minimizing H (g) increases certainty by driving each individual distribution toward point distribution. Thus, the GJS divergence, used as an auxiliary loss, promotes both load balancing and more decisive experiments show that directly optimizing this auxiliary loss can reduce model performance by overly constraining model's flexibility. To mitigate this, we propose the Constrained GJS (CGJS) divergence:
where H.) denotes entropy. Maximizing
routing decisions. However,
CGJS(G) min
(
78 loge - max
Σ ), N
1
(11)
N
Here, T6 and % are hyperparameters in [0, 1]. controls routing balance, with values closer to 1 promoting more balanced experts selection. % regulates routing certainty, with values closer to 0 increasing certainty. Fine-tuning " and % preserves model flexibility, mitigating performance degradation while maintaining expert specialization and balanced experts selection. The definition of the auxiliary loss is as follows:
max ((76 - loge CG)
Canx
(12)
log
The loss is normalized by dividing by log e, ensuring consistency as the number of experts e changes. We apply the auxiliary loss separately to the gate values generated by each task token router within each batch. The auxiliary function essentially performs a clustering- effect. When applied to task routers, the routing results for similar tasks are brought closer together in the latent space, while dissimilar tasks are driven further apart. This clustering- like approach enhances the task router's ability differentiate tasks in an unsupervised manner and generalizes well to unseen tasks. In Appendix D we explain in detail, from the perspective of clustering theory, why our auxiliary loss achieves this effect.
Finally, we only optimize the parameters associated with the experts, router, task encoder, and task embedding to minimize the combined language model loss and auxiliary loss. The loss function is defined as:
L = LIM
Σ
(
(13)
GuX
RES
where À is hyperparameter that adjusts the weight of the auxiliary loss in the overall optimization process. S represents the set of task routers and token routers, and ( denotes the auxiliary loss computed for each individual task or token router R.
4 EXPERIMENTS
We fine-tune our model on Flan v2 (Chung et al. 20244 Longpre et al. 2023 dataset designed for instruction fine-tuning across 1,836 tasks such as natural language inference, question answering, translation, and sentiment analysis, among others. Fine-tuning this diverse multi-task dataset enables the model to acquire general problem-solving capabilities rather than simply fitting to a specific dataset.

---

Published as a conference paper at ICLR 2025
Benchmarks and Metrics. To evaluate multitask performance, we test several NLP bench- marks, including MMLU (Hendrycks et al. 2020), MMLU-Pro (Wang et al. 2024). ARC-Easy, ARC-Challenge (Clark et 2018), OpenBookQA (Mihaylov et al. 2018), SWAG (Zellers et al, 2018), and Commonsense (Talmor et al., 2018). These benchmarks consist of multiple-choiee questions, assessing various aspects of the model's natural language understanding. For MMLU and MMLU-Pro, we use macro accuracy, which averages accuracy across all tasks, while for the other benchmarks, we use accuracy as the evaluation metrie. More detailed information about the training data and benchmarks is provided in Appendix E.
Base Model and Baseline. We utilize Qwen2 1.5B (Yang et al., 2024) our base models. For baseline comparisons, we compare HMoRA with full fine-tuning (Full FT), LoRA (r = 8 and r 64 and methods incorporating mixtures of LoRA experts. These methods include MoL.RA (Zadouri et al.) 2023). MixLoRA (Li et al.) 2024a) and HydraLoRA (Gao et al.) 2024). These models were selected due to their similarity training setup to HMoRA. In contrast, other methods may require predefined task-specific datasets pretrained LoRA modules, which differ significantly in setup and assumptions. We provide a brief introduction to the baselines in Appendix E.2
Training and Evaluation Sctup. We limit the maximum number of training steps to 10,000, con- ducting evaluations every 200 steps on the validation sets of all benchmarks. If there is no improve- ment the validation set for 10 consecutive evaluations, best checkpoint, determined by the highest averaged accuracy across all benchmarks, is selected for evaluation on the test set. Each experiment is repeated times, and report the mean of the evaluation metrics.
will terminate the training early. The
4.1 ROUTING METHODS COMPARISON
We compare the performance of soft routing and top-k routing, as well as the impact of our auxiliary loss (Laux) and load balancing loss (LMe) mentioned in Appendix A
RM
Laux
Lots
MMLU
MMLU-Pro
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
55.16
24.81
69.07
85.44
81.33
53.91
70.05
62.83
Soft
yes
54.85
26.40
70.12
85.36
81.68
56.15
70.99
63,65
54.16
25.55
69.41
85.15
81.58
53.82
70.41
62.87
Top-k
yes
54.09
25.44
69.12
85.56
82.02
54.63
71.42
63.19
yes
54.79
26.01
69.67
85.48
82.36
55.76
71.99
63.72
Table I: Performance comparison for different routing methods (RM). We calculate the average accuracy (Avg) across seven benchmarks as measure of the model's capability in multi-task sce- narios. The best result for soft or top-k routing on each dataset is highlighted in bold.
Implementation Details. For the MoRA block, we set r 8 and e 8. applying MoRA to all dense layers of Qwen2, including Wy. W1, W,, Wa, Wgate- Wap and W Woown- We only use token-level routing in this section. For Loke, used the recommended X = 0.01. For Laux, set À = 0.003, To = 0.4, and "/- 1. For top-k routing, we set k = 2. More training and implementation details as show in Appendix
Main Results. As shown in Table [ incorporating Laux consistently enhances performance, with both soft routing and top-k routing achieving significant accuracy improvements. To further analyze routing certainty and balance, we visualize the entropy of the gate values across different layers in Figure 3 Figure 3 demonstrates that our auxiliary function enhances routing certainty while maintaining relative balance. Although Lote improves routing balance, it reduces routing certainty and does not achieve the same performance gains as Laux- Notably, top-k routing with Laux achieves the best average accuracy. Further experimental results on the hyperparameters of Laxx along with additional analysis of the balance and certainty of routing results, are provided Appendix
4.2 BASELINE COMPARISON
Implementation Details. We compare HMoRA with other fine-tuning methods, employing top- 2 routing and the auxiliary function Cuxx, along with the hierarchical hybrid routing mentioned in Appendix B The hyperparameters for hierarchical hybrid routing set as follows: = 4,

---

Published as a conference paper at ICLR 2025
Entrop
1.5
0.5
(a) Soft routing
2.00
cerieity
1.50
1.00
0.75
0.50
10
Router ind different LLMs
(b) Top-k routing
Figure 3: Visualizing the entropy of the gate values produced by routers across different layers. The base model has 28 layers, but for clarity, we start at layer 4 and sample every layers, resulting in layers. Each layer contains routers (corresponding Wg. Ws. W,, W.. Wamee Wup, and W Wdowa). totaling 49 routers. The dotted lines represent
indicating the balance of expert -
selection -
higher values suggest more balanced routing. The solid lines show NV E Hg)()), reflecting the certainty of routing decisions lower values indicate greater certainty.
Method
TP
MMLU
MMLU-Pro
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
100%
54.12
25.09
69.13
85.10
82.35
55.11
71.16
63.15
LoRA r
0.60%
52.89
23.77
67.52
83.92
79.96
50.51
66.72
60.76
r
4.78%
24.24
68.60
80.93
53.33
70.64
62.28
MoLoRA
3.82%
53.95
25.26
69.10
85.44
81.98
54.43
70.98
63.02
MixLoRA
3.97%
53.95
24.81
68.94
85.09
81.15
52.44
70.31
62.38
HydraLoRA
3.20%
54.10
25.08
69.32
85.11
81.29
53.66
70.33
62.70
HMoRA LW
3.90%
$4.02
25.61
70.73
85.63
82.20
56.40
72.59
63.88
HMORA w/o LW
6.31%
54.63
26.59
71.47
85.87
83.23
55.28
72.08
64.16
Table 2: Results of baseline comparison experiments across multiple NLP benchmarks. TP refers to the percentage of trainable parameters relative to full tin-ttnnigg The best result for each benchmark is highlighted in bold. "w LW" and "w/o LW" refer to using and not using lightweight designs, respectively.
22 1row
0.2, and Shigh 0.8. All other hyperparameters are consistent with those outlined
in Section 4.1
We also compare HMoRA with lightweight designs mentioned in Appendix C
setting 1/8
For LoRA, conducted experiments with both r 8 and 64. For other
mixtures of LoRA experts models, we fixed = 8 and hyperparameter search for these baselines and report the best results.
8. Additionally, we performed
Main Results. As shown in Table HMoRA w LW designs outperforms full fine-tuning out of benchmarks, while requiring only 3.9% of the trainable parameters. Even on the two benchmarks where HMoRA w LW slightly lags behind, the performance gap is minimal. HMoRA w/o LW surpasses full fine-tuning across all benchmarks. Moreover, HMORA w/o LW significantly out- performs LoRA and other mixture of LoRA experts methods, demonstrating superior performance across all benchmarks. HMoRA w/ LW also achieves higher average accuracy. These results demonstrate the effectiveness of HMoRA in efficiently fine-tuning LLMs in multi-task setting. We also conducted baseline comparison experiments on LLaMA 3.2 1B, with the results and analysis provided in Appendix E.7
8

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDY
Ablation Study on the Hyperparameters of the Auxiliary Function. In Appendix E.4 we con- duct ablation experiments primarily on the hyperparameter Te. finding that setting To around 0.4 yields better model performance.
Ablation Study on the Hyperparameters and μ for Hierarchical Hybrid Routing. In Ap- pendix E.5 we perform ablation experiments on the hyperparameters e and We find that setting < > O. Lee, increasing a generally leads to better performance and the model's performance not sensitive μ..
Ablation Study on Lightweight Designs. In Appendix E.6 we examine the impact of each lightweight design on model performance.
Quantitative Study on the Ability of Task Routers to Differentiate Unseen Tasks. We con- duct quantitative study on the performance of task router on unseen tasks in Appendix [E.8 The experimental results show that the task router using Laux is able to effectively differentiate 42 out of 57 sub-tasks (73.68%) in MMLU. Without any auxiliary function, none of these tasks can be distinguished, while using Late can differentiate just 7 tasks (12.28%)
high school compuier computer science
poycharegy
(a) Visualization of gate values from task router with auxiliary loss.
hip schiol computer scunce
collgge computer science
high school mathematice
collgge nathenatis
high school peycho ogy
profession poychelogy
-30
-10
20
(b) Visualization of gate values from task router without auxiliary loss.
Figure 4: t-SNE visualization of the gate values computed by the final task router in LLMs for different tasks in the MMLU dataset.
Qualitative Study on the Ability of Task Routers to Differentiate Unseen Tasks. To more intu- itively reveal the impact of the auxiliary loss on the task router, we selected 3 pairs of tasks from MMLU, with each pair containing 2 similar tasks and 100 samples per task. We routed the ques- tions of these tasks and visualized the routing results from the task router in the last laye of LLM with t-SNE (Van der Maaten & Hinton 2008). As shown in Figure 4(a) when the task router uses Lauxs each task forms a distinct cluster, with similar tasks (represented by the same shape) positioned closer together, while dissimilar tasks are clearly separated. In contrast, the visualization of the rout- ing results from task router without Laux is shown in Figure (bb The routing results for these tasks, although forming some clusters, are noticeably less distinguishable compared to those using Lax- This demonstrates that using Laux enhances the task router's ' ability to differentiate tasks, leading to more accurate and distinct routing decisions. Notably, no explicit task labels were provided during training, and the tasks from MMLU were unseen during training. This suggests that the task router learned to differentiate tasks in unsupervised manner and can generalize to unseen tasks.

---

Published as a conference paper at ICLR 2025
MMLU
MMLU-Pro
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
HMoRA w/ /o Lanx for Task Router
54,63
26.59 25.33
71.47
85.87 85.43
83.23 82.13
55,28 53.48
72.08
64.16
53.83
70.96
71.09
63.18
Table 3: Ablation study results on the impact of the auxiliary loss for the task router.
As shown in Table 3 when the auxiliary loss is not applied to the task router, the model's perfor- mance significantly declines all benchmarks. This indicates that the auxiliary loss plays crucial role in the task router's performance.
RELATED WORK
Mixture of Experts (MoE). MoE was introduced by Jacobs et al. (1991) as a framework that divides complex problems into simpler tasks, each handled by specialized expert. (Shazeer et al. (2017) im- proved the efficiency of this approach by activating only a subset of experts for each input using the Sparsely- -ated MoE layer. Building on this, Lepikhin et al. (2020) scaled MoEs to 600-billion parameter multilingual Transformer, enhancing scalability. Du et al. (2022) further scaled this to 1.2 trillion parameters with GLaM, activating small fraction of the model for each input, resulting in significant computational savings while outperforming GPT-3 on 29 NLP benchmarks. |Fedus et al. (2022) simplified the MoE routing algorithm, allowing each token select one expert, speeding up training without sacrificing quality. Shen et al. (2023) demonstrated that combining MoEs with instruction fine-tuning in L.MSs improves performance on task-specific benchmarks while main- taining computational efficiency. Furthermore, MoE is not limited to language modeling but is also extensively utilized in other domains. For instance, Mao et al. (2024) employ MoE to capture travel time uncertainty in road segments under dynamic traffic conditions and Li et al. (2024b) integrates MoE to handle inputs from various modalities.
Combining MoE with LoRA. Combining MoE with LoRA enhances LLMs for multi-tas task learning while significantly reducing the number of trainable parameters. MoLoRA (Zadouri et al.) 2023
integrates LoRA with MoEs, achieving performance comparable full fine-tuning. Luo al. (024)
employs contrastive learning to promote expert diversity, while Dou et al.] (2024) introduces an auxiliary function to specialize experts in either world knowledge or downstream tasks, enhancing overall effectiveness. Xu et al. (2024), Feng et al. (2024), and Zhao et al. (2024) dynamically compose independently trained LoRA experts for different tasks, albeit the cost of labor-intensive training. Architecturally. MixLoRA A (Li et al. 2024a) applies MoE to the feed-forwarr network and LoRA self-attention, whereas HydraLoRA (Tan al., (025) uses an asymmetric design of LoRA experts to reduce parameters. [Gao et al. (2024) demonstrates that assigning more experts to deeper layers improves performance. 'lask-leve routing, as used Lii al. 2024) and/Feng al. I (2024), assigns tokens based tasks, improving expert specialization in multi-tast scenarios.
6 CONCLUSION
This paper introduces HMoRA, an approach that enhances LLMs by integrating a mixture of LoRA experts with hierarchically combined token-level and task-level routing. This design enables the model to capture both fine-grained and global information across different layers of the LLM. We propose novel auxiliary loss that enhances routing certainty while maintaining a balanced selec- tion of experts, thereby improving expert specialization. It also strengthens task differentiation and enhances generalization to unseen tasks. Additionally, our lightweight designs reduce the parameter size and computational cost, increasing the model's 's practicality. Experimental results demonstrate that HMoRA outperforms both full- harlmparaeeter fine-tuning and other LoRA-based approaches across multiple benchmarks. Our auxiliary loss and hybrid routing have the potential to enhance the perfor- mance of standard MoE architectures; however, efficiency in industrial-scat applications remains a critical aspect that requires further refinement. This opens promising avenues for future research.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4895 4901, 2023.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling intructiionfinetuna lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 0ee.//003..5555 2018.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1932-1945, 2024.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547- 5569. PMLR, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(12)):1-39, 2022.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An effi- cient multitask tuning for large language models. arXiv preprint SttV:22003.034322, 2024.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint 2xXx:22402.0866,, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. arXiv preprint arXh::2402.1235.. 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03322 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0960 09685, 2021.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8)) 1991.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040088 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.

---

Published as a conference paper at ICLR 2025
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Khan-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, 2021.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen. Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 100066/66666 16668, 2020.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint wrXiv:2204.151555 2024a.
Yunxin Li, Shenyuan Jiang. Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal ilms with mixture of experts. arXiv preprint arXiv:2405.11275 2024b.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151) 1991.
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets 1lms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1104-1111 1114, 2024.
Xiao Liu, Yanan Zheng. Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648, PMLR, 2023.
I Loshchilov. Decoupled weight decay regularization, arXiv preprint urXiv:1711.051000 2017.
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts parameter-efficien fine-tuning for large lan- guage models, 2024.
Xiaowei Mao, Yan Lin, Shengnan Guo, Yubin Chen, Xingyu Xian, Haomin Wen, Qisen Xu, Youfang Lin, and Huaiyu Wan. Dutytte: Deciphering uncertainty origin-destination travel time estima- tion. arXiv preprint arXh::2408. 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, arXiv preprint arXiv:1809.02700 2018.
Frank Nielsen and Richard Nock. Sided and symmetrized bregman centroids. IEEE transactions Information Theory. 55(6):2882-290) 2009.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. Pangu-('(Sigme To- wards trillion parameter language model with sparse heterogeneous computing. arXiv preprint arXiv:2303. 10845, 2023.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-42) 1948.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05222, 2020.
12

---

Published as a conference paper at ICLR 2025
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of saree-of-Eaae of-exxssts Layer, January 2017. URL
06538v1
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Bar- ret Zoph, William Fedus, Xinyun Chen, et al. Mixture-of- oo-eeeestseesseeeeeooeeeee meets instruction tuning: A winning combination for large language models. arXiv preprint 4771 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXi:: 2018.
GLM Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatgim: A family of large language models from glm-130b to glm-4 all tools. arXiv e-prints, pp. arXiv 2401, 2024.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, Advances in Neural Information Processing Systems, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint Stff::2400001577. 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jingwei Xu, Junyu Lai, and Yunpeng Huang. Meteora: Multiple-tasks embedded lora for large language models. arXiv preprint arXiv:2405.1 130553 2024.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. 2023.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.08. 05326, 2018
Ziyu Zhao Leilei Gan, Guoyin Wang. Yuwei Hu, Tao Shen, Hongxia Yang. Kun Kuang, and Fei Wu. Retrieval-auggeente mixture of lora experts for uploadable machine learning. arXiv preprint arXiv:2406. 16989, 2024.
A ROUTING METHODS
The routing methods fx(() used in existing MoE methods can be broadly divided into two cate- gories: soft routing and top-k routing. The soft routing is formulated as follows:
fx(x = softmax(h(X))
(14)
Here, h(-): : d + R represents the routing function of the router. can be implemented using simple dense layer multi-layer perceptron (MLP). While soft routing is simple, it requires the activation of all experts. This approach is only applied in the mixture of LoRA experts method. In standard MoE models, using soft routing would result in substantial computational overhead. To
13

---

Published as a conference paper at ICLR 2025
address this, standard MoE models typically activate only few experts sparsely, necessitating the use of top-k routing. The corresponding formula is as follows:
g fx(X) softmax (eenanxxXKKepppppoosffssaaaa///
(15)
is among the top k in w.
(c; = -oo otherwise.
(16)
This approach retains only the top experts with the highest gate values, setting the rest zero. Sparse activation is achieved by activating only experts with non-zero gate values. However, top-k routing can easily lead to certain experts being selected frequently, while others are rarely or never selected. This imbalance can degrade the model's generalization ability.
To address this issue, studies such as Shazeer al. (017) and |edus et al. (0022) propose various load balancing techniques, which employ auxiliary loss to ensure a more even distribution in the selection of experts, thereby improving the performance of MoE models. The load balancing loss introduced in Fedus al. (2022) encourages uniform distribution of tokens across the experts. The loss function in the original work applicable to top-1 routing. We extend it to top-k routing. The auxiliary loss for batch of gate values G is formulated
Lole
Σ P. P.P
(17)
where F; is the fraction of tokens routed expert given by:
F
1
Σ 1 [a in top-k of
(18)
k
G
BEG
and P. is the average router probability for expert i, defined as:
P
Σ
(19)
9EG
This auxiliary loss encourages uniform token routing across experts by minimizing the difference between F; and P., ensuring balanced selection of expert.
B MAKING HYBRID ROUTING HIERARCHICAL
In Equation 8 there are two hyperparameters, and μ. As increases from 0 to L, the value of (!) transitions from a-- + μ4) to oe + p). If positive, (!)) increases with I; conversely, if is negative, (1) decreases / increases. The larger the value of e, the more rapid this transition. The parameter can be adjusted to control whether more layers employ token routing or task routing. To provide better understanding, we present examples various configurations of and μ in Table
C
1
a
(1)
(2)
a
(3)
(4)
(5)
2
0.12
0.23
0.40
0.60
0.77
0.88
-2
0.88
0.77
0.60
0.40
0.23
0.12
10
0
0.12
0.88
1
10
0
0.12
0.88
1
0
0.5
0.5
0.5
0.5
0.5
0.5
Table 4: Example of different setups of ( Here, we set L = 5.
Additionally, if (9) is too small, the task router contributes very little the final gate values. Con- versely, if (0) two thresholds, Slow and Shigh- When ạ < Slow, only token routing, and when a Bnigh we only use task routing. This allows HMoRA to achieve higher efficiency, as the computational overhead of task routing is significantly lower than that of token routing.
large, the token router has minimal influence on the final gate values. So set
14

---

Published as a conference paper at ICLR 2025
C LIGHTER AND MORE EFFECTIVE HMORA
The inclusion of task encoders and task routers significantly increased the number of trainable pa- rameters. In this section, we explore several designs to reduce the parameter count and computa- tional costs of HMORA, making it more lightweight and efficient.
Single LoRA. As shown in Figure we observed that the mean entropy of the gate values at Wo and W down was relatively high without Laux- Even with Laxs the mean entropy of the gate values at W. remained higher than the others. This suggests that the inputs to these two dense layers do not demonstrate a clear preference for different experts. As a result, we opted to apply a single LoRA to these dense layers instead of using MoRA.
Router Sharing. Whether to assign a router to each dense layer or to a larger module is a question worth considering. Given that the inputs W W1, and W, are identical, and W gate. it is reasonable to assign the same router to these dense layers. In contrast, assigning separate routers to each dense layer may increase the complexity of subsequent integration among their outputs. Therefore, we assign a shared router to Wq. W1. and Wv, and another shared router to Wap and W gate-
those W
Hydra LoRA (Tian et al. 2025) found that the LoRA matrix WA. when trained on different tasks, tends to show similarities across those tasks, with the primary differences man- ifesting in matrix WB. As result, matrix WA can be shared among different experts, capturing aank-sgi knowledge, while matrix W8 specializes learning domain-specific expertise. Addi- tionally, LoRA+ (Hayou al. 2024) proposed assigning a higher learning rate (e.g.. 7n times, where "/s > 1) to matrix WB compared to WA, which accelerates convergence and improves performance. In HMoRA, integrate these approaches.
Hydra LoRA+.
D ENABLING UNSUPERVISHD LEARNING THROUGH AUXILIARY LOSS
In this section, we offer an explanation from the perspective of clustering theory, detailing how ap- plying Lax enables the task router distinguish between different tasks in unsupervised manner and how this capability generalizes to unseen tasks.
From the perspective of clustering theory, the role of our auxiliary function in the task router be understood as driving the model to automatically discover and distinguish the underlying structures of different tasks. One of the core objectives of clustering theory is maximize the distance between different clusters (i.e., different tasks), ensuring significant separability between clusters. Another key objective is to minimize the distance or increase the similarity between data points within the same cluster, so that the data points within a cluster are more tightly grouped. Our auxiliary function achieves these goals in the following ways:
Maximizing H EN ()): The auxiliary function encourages the routing decisions to be distributed more uniformly. This ensures that, global scale, routing is not overly concentrated on a few experts but instead utilizes all experts in a balanced manner, ensuring greater differentiation between tasks when selecting experts. This is analogous the clus- tering objective of "maximizing cntererluster distance," where different tasks are assigned to distinct expert clusters.
Minimizing 1 Ei= ()) The auxiliary function drives the entropy of each individual routing decision toward zero, making the routing decisions more deterministic. This en- sures that different inputs from the same task are consistently routed to the same expert, thereby reducing intra-task variability. This is akin to the clustering objective of "mini- mizing intra-cluster variance," ensuring that all instances of the same task are routed to the same or similar experts.
N
With this design, the task encoder and task embedding map the task information from the input into a feature space, where similar tasks are positioned closer together, while dissimilar tasks are pushed further apart. The router then utilizes the routing function to assign different task clusters in the feature space corresponding combinations of experts. This process resembles unsupervised clus- tering, where the model, by optimizing the auxiliary function, automatically discovers the structural relationships between tasks.
15

---

Published as a conference paper at ICLR 2025
Even when faced with unseen tasks, the task encoder captures the relevant task information from the input and maps it into the feature space. The router then routes the task to the most similar expert cluster based its position in this space. This capability is analogous to how clustering algorithms assign new data points to the most appropriate cluster by evaluating their proximity to existing clusters.
In summary, the auxiliary function promotes intra-cluster consistency and inter-cluster separability within the framework of clustering theory, enabling the task router automatically differentiate between tasks in an unsupervised manner. Furthermore, this differentiation capability generalizes to unseen tasks. This mechanism not only enhances the model's performance in multi-task scenarios but also improves its adaptability and robustness in dynamic, evolving environments.
E ADDITIONAL EXPERIMENT INFORMATION, RESULTS, AND ANALYSIS
E.1 DATASET STATISTICS
For training, we utilize the Flan v2 dataset, which is derived from 473 individual datasets, spanning 146 task categories and encompassing total of 1,836 tasks. The dataset is divided into four mix- tures: TO-SF, Muffin, SNI, and COT. We use a i--miliion-sampl subset of Flan vz and adjust the proportions of the four mixtures, shown in Table
TO-SF
Muffin
SNI
CoT
Proportion (%)
52.5
31.5
10.4
5.6
Table 5: Proportion of different mixtures in Flan v2.
To evaluate multitask performance, we test on several well-established NLP benchmarks:
MMLU (Massive Multitask Language Understanding) This benchmark covers 57 tasks across various domains, assessing a model's 's ability to generalize across different subjects.
MMLU-Pro: An extension of MMLU, this variant increases the difficulty by expanding the number of multiple-choice options ten, making the evaluation more challenging.
ARC (AI2 Reasoning Challenge): This benchmark tests grade-schor science questions, divided into two subsets: ARC-Easy, which involves straightforward questions, and ARC- Challenge, which requires more complex reasoning and deeper knowledge.
OpenBookQA: A benchmark focused on science questions that require the combination of provided facts with external common knowledge, testing the model's 's applied reasoning abilities.
SWAG (Situations With Adversarial Generations): This benchmark evaluates common- sense reasoning by asking the model to predict the most plausible next action given scenario.
Commonsense A multiple-choice dataset that tests commonsense reasoning, requiring models to apply implicit world knowledge to answer questions correctly.
Each of these benchmarks is designed to prohe different dimensions of generalization, reasoning. and knowledge application in natural language understanding, providing a comprehensive evaluation of multitask learning performance. Detailed information about these datasets shown Table 6
For each evaluation, testing on full datasets would be highly time-consuming, even exceeding the training time. To address this, we split these benchmarks into smaller subsets. For MMLU, we randomly sample 300 examples from the validation set as the new validation set and 2,000 examples from the test set as the new test set. For MMLU-Pro. we randomly sample 2,000 examples from the test set as the new test set. For SWAG, we randomly sample 300 examples from the original validation set to create new validation set, and 2,000 examples form new test set. For Com- monsenseQA, randomly sample 300 examples from the original validation set to create new
16

---

Published as a conference paper at ICLR 2025
Dataset
train
validation
test
Number Options
Metries
MMLU
99842
1531
14042
4
Macro accuracy
MMLU-Pro
70
12032
10
Macro accuracy
ARC-C
1119
299
1172
4
Accuracy
ARC-E
2251
570
2376
4
Accuracy
OpenBookQA
4957
500
500
4
Accuracy
SWAG
73546
20006
20005
5
Aceuracy
CA
9741
1221
1140
4
Accuracy
Table 6: Information of the original datasets.
Dataset
validation
test
MMLU MMLU-Pro
300
2000
70
2000
ARC-C
299
1172
ARC-E
570
2376
OpenBookQA
500
500
SWAG
300
2000
CommonsenseQA
300
900
Table 7: Number of samples in validation and test splits from the subsets of original datasets.
validation set, and 900 examples to form a new test set. The number samples in the subsets we created for each benchmark is shown in Table 7 All of our experiments are evaluated on this newly created subset.
E.2 BRIEF INTRODUCTION OF BASELINE
In this section, we introduce three approaches that combine Moï with LoRA: MoLoRA, MixLoRA. and HydraLoRA. These methods, like our own, do not require datasets with specific task labels or pre-trained LoRA experts, making them suitable for broad range of tasks. This is the primary reason we selected them as baselines. Below is brief overview of each method:
MoLoRA: MoLoRA employs token-level soft routing and applies MoRA only to the dense layers within the FFN.
MixLoRA: MixLoRA utilizes token-level top-k routing and applies MoRA to the dense layers within the FFN, while simultaneously fine-tuning each dense layer in the self- attention module using LoRA. Additionally, MixLoRA incorporates load-balan loss, Cale.
HydraLoRA: HydraLoRA introduces novel structure for LoRA experts, where shared LoRA matrix WA is used among all experts, while each expert has its own unique LoRA matrix Wn., This design significantly reduces the number of parameters by sharing part of the LoRA components, which makes the model more parameter-efficient Hydral employs token-level soft routing.
These methods combine the strengths of MoE and LoRA, making them highly relevant for compar- ison with our proposed method.
E.3 TRAINING IMPLEMENTATION DETAILS
General Experimental Setup. We set the learning rate to 2 10-5, with warm-up period of 500 steps during which the learning rate increases linearly. We use the Adam Loshchilov (2017) optimizer, with dropout rate of 0.1 and label smoothing set 0.1. We simulate base size of 12 using gradient accumulation. The maximum input length is set to 1024, and the maximum output length is set to 512. All the experiments conducted NVIDIA A40 GPUs.
17

---

Published as a conference paper at ICLR 2025
HMoRA Implementation Details. For the router. implement it using single dense layer and apply dropout to its imputs during training. We also apply dropout to the inputs the experts. For the task encoder, we use single-layer Transformer encoder, ensuring that the model dimension of task encoder consistent with the L.M. The number of attention heads is set to 16, and the hidden layer size in the FFN is set twice the model dimension.
E.4 ANALYSIS OF THE CERTAINTY AND BALANCE OF DIFFERENT ROUTING METHODS
In this section, we further analyze the impact the auxiliary loss on the determinism and balance of the routing results. The experimental setup is consistent with that in Section 4.1 except that adjust the parameter % for Lauz- We primarily experiment with Yes which affects the certainty of routing results. We fix 7 1 we aim for the routing be balanced possible. If the training data itself is imbalanced, setting "/ to a lower value may yield better results. We select "Ye values from {0, 0.2, 0.4, 0.6, 0.8). The experimental results shown in Table We observe that for both the 2 routing and soft routing, setting "e to 0.4 yields better results, while setting "e too high or low negatively affects the performance the auxiliary function.
Routing Method
%
MMLU
MMLU-Pro
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
0.0
53.87
25.00
69.18
84,90
81.21
57.14
70.59
63.13
0.2
54.47
25.57
69.12
85.00
81.14
53.44
70.25
62.71
Top-2
54.79
26.01
69.67
85.48
82.36
55.76
71.99
63.72
25.13
69.49
85.44
82.76
56.14
71.39
0.8
54.39
24.25
68.89
85,25
81.08
54.41
70.70
62.71
54.81
24.72
69.78
85,79
82.09
55.42
70.70
63.33
0.2
53.64
25.96
69.52
85.14
81.75
56.02
70.93
63.28
Soft
0.4
54.85
26.40
70.12
85,36
81.68
56.15
70.99
63.65
53.19
25.04
69.09
85.17
81.41
53.29
70.16
62.48
0.8
53,94
25.23
69.78
85,59
81.82
53.56
71.16
63.01
Table 8: Performance comparison of Top-2 and Soft routing methods across various % settings on multiple NLP benchmarks.
gammaa gamma 0.4 gamma_ -.8
0.75
0.25
Figure 5: t-SNE visualization of gate values from the top-2 routing with our auxiliary loss dif- ferent "e values. Dark blue points (low entropy) represent high certainty, while yellow points (high entropy) represent low certainty.
Furthermore, to intuitively illustrate the impact of %e on the certainty of routing, we randomly sam- pled a data point from the training set and input it into the model. We then visualized the gate values, computed by the final router over the first 200 tokens, using t-SNE. As shown in Figure 5 dark blue points represent gate values with very low entropy, indicating a high level of certainty, as they approach single-poine distribution. In contrast, yellow points represent higher entropy values, indicating lower certainty. The eight dark blue clusters in the figure correspond to eight distinct ex- perts. It can be observed that when %e 0, most of the gate values are highly concentrated, forming
18

---

Published as a conference paper at ICLR 2025
near single-point distributions. As % increases, the certainty of the routing decreases, leading to more dispersed gate values.
2.00
lop-2 uss
1.75
10
1.00
-10
-20
0.25
-10
Figure 6: t-SNE visualization of gate values from different routing methods. The plot compares four routing methods: top-2 routing without any auxiliary loss (rhombic), top-2 routing with Lote (triangles), and top-2 routing with Laux at (circles) and 0.4 (squares). The color scale represents entropy, with darker colors indicating lower entropy (higher certainty) and lighter colors indicating higher entropy (lower certainty).
We compare the certainty of standard top-2 routing, top-2 routing with Laux, and top-2 routing with Loke- The results are shown in Figure 6 In top-2 routing without any auxiliary loss, despite the high certainty of the gate values, there biased toward a single expert (as seen in the upper-middle part of the figure). This causes the MoE model effectively degrade into a non-MoE model. Conversely, top-2 routing with load balancing loss results in a lack of certainty, with most gate values approaching uniform distribution, which undermines expert specialization. In contrast, top-2 routing with our auxiliary loss achieves both a relatively balanced selection of experts and greater determinism, thereby enhancing expert specialization.
severe imbalance, with nearly all routing decisions
0.5
gemme -
gamma_c 0.4
parmaac 0.8
0.4
loed balancing
Inp-2 only
Proportions
0.2
0.0
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Figure 7: Proportion activations for each expert in top-2 routing across different configurations. The plot compares the impact of Laux with varying % values (0, 0.4, and 0.8), and load balancing Lote on the selection of experts (1-8).
We further analyze the balance of these routing methods by counting the number of experts activated by the top-2 routing for the first 200 tokens of input and calculating the proportion of activations for each expert relative to the total number of expert activations. Since top-2 routing activates both the expert with the highest probability and the expert with the second-hii -ighhest probability, the maximum activation proportion for any single expert can reach 50%. The results are shown in Figure With- out any auxiliary loss, t routing is the most imbalanced, with Expert 7 being activated nearly 50% of the time, while Expert 1, 4, and are almost never used. A similar imbalance occurs when
19

---

Published as a conference paper at ICLR 2025
Laux is applied with 7o O. However, when "e is set to 0.4 0.8, this imbalance is significantly reduced. The most balanced routing is achieved with top-2 routing using Late- However, as noted in the previous analysis, this balance comes at the cost of reduced certainty. Furthermore, given that the data itself is imbalanced, enforeing strict balance can even hinder the specialization of the experts. The experimental results in Section 4.7 demonstrate that using our auxiliary loss leads to better overall performance.
E.5 EXPLORING THE EFFECTS OF HYBRID ROUTING
In this section, we investigate how combining task-level and token-level routing can enhance the performance of LLMs. In this set of experiments, do not apply the lightweight designs. For the 1, and À 0.003. The upper and lower bounds for hybrid
auxiliary loss, we set 0.4, routing are set Sdown = 0.2 and Sup = 0.8, respectively. We conduct experiments by varying e and p.
Manner
P
50
TP
MMLU
MML.U-Pro
ARC-C
ARC-E
OpenBonk
SWAG
Comm
Avg
5.04%
54.79
26.01
69.67
85.48
55.76
-135
0.2
6.42%
54.62
25.57
69.29
84.91
81.68
56.98
70.93
63.43
-0.4
54.76
85.22
Constant
0.4
0.6
6.42%
54,26
25.42
69.55
85.44
82.42
$3.27
70.22
62,94
1.35
6.42%
53.43
25.09
69.18
85,20
82.36
54.48
71.31
63.01
2
6.16%
53.96
26.11
69.21
85,07
82.15
54.87
70.93
63.19
-4
0
6,29%
54,30
25.47
69.55
85.69
82.42
$4.78
71.05
63.32
Hierarchical
6.29%
85.58
83.43
72.59
63.70
-2
6.31%
$4.63
26.59
71.47
85.87
83.23
$5.28
72.08
64.16
6.26%
54,50
69.89
85.66
83.42
55.89
72.23
63.96
Table 9: The performance on 7 benchmarks when (1) is set to constant or hierarchical. For the
constant group of experiments, (1) is set to a fixed constant across all layers. The value of (! shown in the table represents the approximate value under the corresponding settings of and μ.. For the hierarchical group of experiments, indicates that ( increases with layer 1, while indicates that it decreases with layer
As shown in Table9 we observe that increasing (1) across layers significantly boosts the model's performance compared both decreasing a (!) and constant (1) The configuration where (! in- creases progressively across layers achieves relatively better average performance, with the setting 0 and μ 2 yielding the best results. This outcome underscores the effectiveness of hierar- chical hybrid routing, which enables the model to shift its focus from token-level details in the early layers to broader task-level understanding in the deeper layers. Setting e >> 0 (with (3) increasing) generally yields better performance. The model appears be less sensitive to the parameter μ..
E.6 EFFICIENCY ANALYSIS OF LIGHTWEIGHT DESIGNS
In this section, evaluate the lightweight designs discussed in Appendix C
All methods were tested under the same hardware configuration. and we measured the time required to train for 1,000 steps to estimate the computational cost of each lightweight design. For hicrarchical hybrid routing, we set = 4 and μ = 2, with all other settings consistent with those outlined in Appendix E.S We set the actual batch size 4, but accumulate gradients over 3 steps simulate batch size of 12.
As depicted in Figure 2 router sharing does not significantly reduce the number of trainable parameters, but it substantially lowers computational cost. Conversely, both Hydra LoRA and Single LoRA greatly decrease the number of trainable parameters. When these designs are combined, we observe nearly one-third reduction in both computational cost and trainable parameters.
In terms of performance, as shown Table I0 router sharing not only maintains performance levels comparable to the base method but even improves the model's performance on the SWAG bench- mark. However, both Hydra LoRA and Single LoRA introduce some performance degradation. The combination of Hydra LoRA and LoRA+ helps mitigate the performance drop when 7)8 is ap- plied. Integrating all of the lightweight designs results in only a minimal decrease in performance, demonstrating that these approaches effectively reduce both computational and memory costs with-
20

---

Published as a conference paper at ICLR 2025
Method
TP
MMLU
MMLU-Pro
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
Base
6.31%
54.63
26.39
71.47
85.87
83.23
55.28
72.08
64.16
Router Sharing
6.22%
54,52
25.68
71.27
85.41
82.83
$6.38
72.08
64.05
Single LoRA
4.77%
53.41
25.23
69.81
85.18
81.35
56.13
70.88
63.14
Hydra LoRA
4.46%
53.75
26.27
70.01
85,41
81.55
55.64
71.11
63.39
Hydra LoRA+
4.46%
1.2
53.78
25.39
69.35
85,41
82.82
54,30
71.36
63,20
Hydra LoRA+
4.46%
1.4
53,53
25.27
69.15
85,38
81.68
54.85
71.16
63.00
Hydra LoRA+
4.46%
53.17
25.08
69.47
85.58
81.35
54.55
71.93
63.02
Hydra LoRA+
4.46%
1.8
53.79
24.99
68.89
85.52
83.50
54.78
71.65
63,30
Hydra LoRA+
4.46%
2.0
53,40
26,81
70.89
85,35
82,76
55.68
71.97
63.84
All
3.90%
2.0
54.02
25.61
70.73
85.63
82,20
$6.40
72.59
63.88
Table 10: Performance comparison of various lightweight design strategies, including Router Shar- ing, Single LORA, Hydra LoRA, and Hydra LoRA+ with different 7/ settings. The table shows the percentage of trainable parameters (TP), as well as performance across multiple benchmarks. The "+ All" row represents the integration of all lightweight designs.
out significantly compromising the model's accuracy benchmark tests. These findings highlight the practicality of adopting lightweight designs, especially in resource-constraine environments, as they enable substantial efficiency gains while preserving competitive performance.
E.7 BASELINE COMPARISON ON LLAMA 3.2 1B
We conducted baseline comparison experiments on LLaMA 3.2 1B. We set % to 0.8 and f)в to 4. We increased the maximum training steps to 20,000, and evaluated on the validation set every 1 1,000 steps. other parameters are consistent with those described in Section 4.2 results are shown in Table II]
The experimental
TP
MMLJU
MML.--Pr
ARC-C
ARC-E
OpenBook
SWAG
Comm
Avg
full fine-tuning
100%
27.42
12.05
27.51
41.22
50.91
47.75
33.61
LoRA
0.45%
31.4
12.21
33.42
46.63
51.82
37.62
45.81
36.99
LoRA r=64
3.64%
33.82
$7.57
55.15
39.44
47.59
40.68
MoLoRA
2.67%
25.02
9.67
26.31
26.74
27.07
25.31
22.51
23.23
MixLoRA
2.81%
34.18
12.76
38.26
54.07
50.91
42,08
46.2
39.78
HydraLoRA
2.37%
34.39
12.56
36.68
58.63
58.99
39.82
45.81
40.98
HMoRA w/o LW
6.61%
32.69
13.34
37.26
56.01
59.07
38.29
50.78
41.06
HMoRA w LW
4.63%
35.19
12.91
38.6
58.84
61.72
39.29
54.25
42.97
Table 11: Bascline comparison based on LLaMA 3.2 IB.
We observed that full fine-tuning resulted significantly poorer performance compared PEFT methods. Through our analysis, we summarized two potential reasons for this outcome. First, the Flan dataset does not encompass the knowledge contained the benchmarks. Consequently, for knowledge-intensive benchmarks such as MMLU and MMLU-Pro, the performance of LLMs on these benchmarks heavily relies on their inherent knowledge. Second, our experiments revealed that training steps compared to Qwen gradually improve its average accuracy across all benchmarks. For full fine-tuning, more fine-tuning steps could lead to greater forgetting of the pre-trained knowledge. In contrast, for PEFT methods, this issue is mitigated the original model parameters remain frozen during training. Moreover, MoLoRA appeared to be completely ineffective, which we suspect is due the fact that MoLoRA does not fine-tune the attention layers. Surprisingly, HMoRA w LW achieved the highest average accuracy and performed best on 4 out of 7 benchmarks, even surpassing HMoRA w/o LW.
LLaMA requires
E.8 QUANTITATIVE EXPERIMENTS AND ANALYSIS ON TASK ROUTER
In the ablation experiments presented in the paper, we tested unseen tasks from MMLU and used visualizations verify the task router's ability to differentiate between unseen tasks. Here, further explore this capability through quantitative analysis. We sampled 100 examples from each of the 57 tasks in MMLU and analyzed the task router's routing results for these samples. First,
21

---

Published as a conference paper at ICLR 2025
recorded the proportion of expert activations relative to the total number of activations, shown in Table 12
is important to note that, since we use top-2 routing, each sample activates two experts (an expert pair). The results show that, without any auxiliary function, all tasks are routed exclusively to experts and 2.
Expert
1
2
3
4
6
7
with Laux without any loss with Lbke
10.21
0.45
5.21
34.11
10.67
1.72
31.25
6.34
0.5
0.5
0
8.47
2.11
18.37
27.87
5.62
37.55
0
Table 12:
Proportion of expert activations across different settings, showing the distribution of expert usage with and without auxiliary loss functions.
Further, we analyzed the expert pairs activated for each task. For each task, we identified the most frequently activated expert pair from the 100 samples the main expert pair (MEP). As show in Table 13, we calculated the proportion of tasks in which the activation of the main expert pair exceeded certain proportion threshold.
Threshold
2 0.7
0.8
2 0.9
with Laux
78.94%
73.68%
54.38%
14.03%
without any loss
100%
100%
100%
100%
with Lase
21.05%
12.28%
3.5%
1.75%
Table 13: Proportion of tasks where the main expert pair activation exceeds different thresholds.
We define tasks with a MEP proportion exceeding 0.8 as recognizable by the router, as this indicates consistent and reliable routing, Although the proportion of MEP activation consistently reaches 100% without any auxiliary function, it merely routes all tasks to single pair of experts (1, 2), which we do not regard as a valid indicator of its capability to differentiate between tasks.
Main Expert Pair
(3, 6)
(2, 7)
(0, 4)
All
All Tasks MEP Proportion 2 0.8 Tasks
44
5
8
57
34
3
5
42
Ratio
77.27%
60%
62.5%
73.68%
Table 14: Statistics of main expert pairs using Laux. showing the number of tasks for each main expert pair and the proportion of tasks where the main expert pair activation exceeds 0.8.
Main Expert Pair
(0,
(1, 3)
(2, 3)
(2, 4)
(2, 5)
(3, 5)
All
All Tasks
2
3
1
4
11
36
57
MEP Proportion 2 0.8 Tasks
0
0
Ratio
0%
33.33%
0%
0%
0%
16.67%
12.28%
Table 15: Statistics of main expert pairs using Lees showing the number of tasks for each main expert pair and the proportion of tasks where the main expert pair activation exceeds 0.8.
We further analyzed the MEP for all tasks. The statistics of main expert pairs using Caux and Late are shown Table and Table [5] respectively. As shown Table task router grouped MMLU tasks into three clusters, with 42 tasks (73.68% % being effectively recognized. In contrast, when using Cote. although the number of primary expert pairs increased, only two clusters, (1, 3) and (3, 5), were significant, and only 7 tasks (12.28%) were effectively recognized overall. The above quantitative analysis demonstrates that our auxiliary function enhances the task router's ability to differentiate between tasks, even for those that were not encountered during training.
22


Reviews:
[{"review_id": "Eydz17fOJv", "reviewer": "Reviewer_XbfV", "summary": "This paper presents HMoRA, a hierarchical fine-tuning method that combines Mixture of Experts (MoE) and LoRA for large language models. The key innovation lies in its hybrid routing mechanism, which hierarchically combines token-level and task-level routing. This allows the model to capture information at different granularities, enhancing its understanding capabilities. Additionally, a novel routing auxiliary loss is introduced to improve the certainty of expert selection and maintain a balanced selection of experts. The method also incorporates several optional lightweight designs that significantly reduce the number of trainable parameters and computational costs without sacrificing much performance.", "questions": "-How well does the method perform on a more diverse set of unseen tasks? Are there any specific characteristics of tasks that might affect its generalization ability? Could the authors provide more details on how the model's performance on unseen tasks was evaluated? What metrics were used and how were the unseen tasks selected?\n\n-Regarding the hierarchical routing in Figure 2(a), what are the criteria for determining the optimal division between shallow and deep layers? How sensitive is the model's performance to this division?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The combination of hierarchical routing and the auxiliary loss function is a novel contribution. The hybrid routing that integrates token-level and task-level routing in a hierarchical manner allows the model to capture different granularities of information, which is a significant improvement over existing methods that focus on only one level of routing.\n\nThe experimental results demonstrate that HMoRA outperforms full fine-tuning across multiple NLP benchmarks while fine-tuning only a small percentage (3.9%) of the parameters. This shows the effectiveness of the proposed method in achieving good performance with limited computational resources.\n\nThe proposed optional lightweight designs are a practical addition as they significantly reduce both the number of trainable parameters and computational costs without significantly compromising performance. This makes the method more applicable in resource-constrained environments.", "weaknesses": "The paper claims that the method can generalize to unseen tasks, but the experiments and analysis related to unseen tasks could be more in-depth. There is a need for more experiments that closely mimic real-world scenarios where the model encounters truly unseen tasks.\n\nWhile some ablation studies are presented, they could be more comprehensive. For example, for aspects like the hierarchical routing shown in Figure 2(a), more investigation could be done on where exactly to best divide the shallow and deep layers for optimal performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GKtuMUIvk5", "reviewer": "Reviewer_nMMJ", "summary": "The paper introduces HMoRA (Hierarchical Mixture of LoRA Experts), presenting a significant advancement in fine-tuning LLMs by effectively combining MoE and LoRA with innovative hierarchical routing and auxiliary loss mechanisms. This approach not only improves multi-task performance and expert specialization but also ensures parameter and computational efficiency, making it a promising technique for scalable and versatile natural language processing applications.", "questions": "See weaknesses. Since I am not a researcher in this field, I cannot be certain that the questions I have raised are correct", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "-  HMoRA successfully combines Mixture of Experts (MoE) with Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA. This integration enables the model to outperform traditional fine-tuning approaches when handling a wide range of tasks. By utilizing specialized experts, HMoRA enhances the model’s ability to generalize across multiple tasks, resulting in outstanding performance on various natural language processing benchmarks.\n\n- One of the standout features of HMoRA is its ability to achieve high performance by fine-tuning only 3.9% of the model’s parameters. This significant reduction in trainable parameters greatly decreases the computational and memory overhead compared to full parameter fine-tuning. Consequently, HMoRA proves to be an exceptionally efficient method for deploying large-scale models in environments with limited resources.\n\n- The paper is well-written, presenting its concepts and methodologies clearly. Additionally, it showcases sufficient innovation, contributing novel ideas and approaches to the field of large language model fine-tuning.", "weaknesses": "- The experiments presented in the paper were primarily conducted using the Qwen2 1.5B model and did not extend to larger-scale models (such as those with hundreds of billions of parameters) or to different architectural frameworks. As a result, the scalability and effectiveness of HMoRA on larger or alternative types of LLMs remain unclear.\n\n- Furthermore, although the introduction of lightweight designs is commendable, HMoRA still requires additional computational resources to manage hybrid routing and auxiliary loss functions. However, this increase is understandable given the overall efficiency improvements, and I would not place undue criticism on this aspect.\n\n- Additionally, the auxiliary loss function involves setting multiple hyperparameters (such as γb and γc), which adds complexity to effective tuning across various tasks and datasets. Has there been any investigation into how these hyperparameters impact the ablation studies, and what parameter tuning techniques were employed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uOvgNGT2m6", "reviewer": "Reviewer_qmpt", "summary": "This paper introduces a hierarchical method, HMoRA, for effective fine-tuning. Specifically, HMoRA integrates token-level and task-level routing to combine MoE and LoRA and utilizes a routing auxiliary loss to improve the certainty of expert selections. Experimental results show that HMoRA outperforms full fine-tuning and some other baseline methods.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to read with clear explanations of the technical details.\n2. The approach is straightforward but performs good performance. The hybrid routing that combines token-level and task-level routing is innovative and allows the model to capture multi-granularity information. \n3. It is intuitive to apply Generalized Jensen-Shannon divergence for router selection. The experiment results also demonstrate the effectiveness of the auxiliary loss.\n4. The overall experimental results show the effectiveness and the potential of the proposed method.", "weaknesses": "1. Yet the method shows promising performance in this paper, its novelty is limited in my opinion. The task-level MoE and GJS divergence is not a new topic in MoE or noisy learning areas.\n2. The description of task-level routing is somewhat vague. In Figure 2, the inputs include T^in and T^tg. Are these two distinct tasks? How are they set up during training and evaluation?\n3. The task representation is encoded based on a task embedding, how can it expand into new tasks? In addition, task routing is a crucial method component, yet there appears to be no ablation experiment focused on task routing within the experiments conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qOPrvTS3zC", "reviewer": "Reviewer_wyMT", "summary": "In this paper, authors target at the routing problem in existing MoE methods: data granularity used by MoE layer and the generalization to unseen task and expert specialization. The authors introduced HMoRA, a method combining MoE and LoRA used to fine tune LLMs. It includes a hybrid routing strategy that incorporates token-level and task-level routing. They redesign the routing auxiliary loss, and employ lightweight designs to reduce parameters and computational costs.", "questions": "1. It seems all the components are somehow disconnected. A lot of efforts are put on introducing the new gating algorithm. How is related to apply MoE to LoRa and enable MoE to both token and tasking routing. I feel like the gating algorithm is a separate problem.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiments to validate the idea are well designed and thoroughly covers most of the scenarios.", "weaknesses": "1. Presentation could be further improved. I feel like the appendix C and D are also important contributions for this idea. However, the organization of the paper make it difficult to identify what is the key contribution and insight of the proposed approach.\n2. The idea of mixture of lora and using MoE for both token routing and task routing is not new. I do think a highlight of clear difference between HMoRA and this line of work should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)