Glider: Global and Local Instruction-Driven Expert Router

## Abstract

Abstract The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. This has enabled the creation of powerful and adaptive routing-based “Model MoErging" (Yadav et al., 2024 ) methods with the goal of using expert modules to create an aggregate system with improved performance or generalization. However, existing MoErging methods often prioritize generalization to unseen tasks at the expense of performance on held-in tasks.
This limitation adversely impacts practical applicability, as real-world deployments require robust performance across both known and novel tasks.
We observe that current token-level routing mechanisms neglect the global semantic context of the input task. This token-wise independence hinders effective expert selection, particularly for held-in tasks, as routing decisions fail to incorporate the holistic semantic properties of the task.
To address this, we propose a novel method, G lobal and L ocal I nstruction D riven E xpert R outer ( 𝙶𝙻𝙸𝙳𝙴𝚁 𝙶𝙻𝙸𝙳𝙴𝚁 \mathtt{GLIDER} typewriter_GLIDER ) that integrates a multi-scale routing mechanism, encompassing a semantic global router and a learned local router. As recent LLMs demonstrate advanced reasoning capabilities for semantic-related contexts, the global router leverages this ability to enhance expert selection. By utilizing the input query and an LLM, the router generates semantic task instructions that guide the retrieval of the most relevant experts across all layers. This global guidance is complemented by a local router that facilitates token-level routing decisions within each module, enabling finer control and enhanced performance on unseen and challenging tasks. Our experiments using T5-based expert models for T0 and FLAN tasks demonstrate that 𝙶𝙻𝙸𝙳𝙴𝚁 𝙶𝙻𝙸𝙳𝙴𝚁 \mathtt{GLIDER} typewriter_GLIDER achieves substantially improved held-in performance while maintaining strong generalization on held-out tasks. Additionally, we perform ablations experiments to dive deeper into the components of 𝙶𝙻𝙸𝙳𝙴𝚁 𝙶𝙻𝙸𝙳𝙴𝚁 \mathtt{GLIDER} typewriter_GLIDER and plot routing distributions to show that 𝙶𝙻𝙸𝙳𝙴𝚁 𝙶𝙻𝙸𝙳𝙴𝚁 \mathtt{GLIDER} typewriter_GLIDER can effectively retrieve the correct expert for held-in tasks while also demonstrating compositional capabilities for held-out tasks.
Our experiments highlight the importance of our multi-scale routing that leverages LLM-driven semantic reasoning for MoErging methods. Our code is available at https://github.com/UNITES-Lab/glider .

## 1 Introduction

The emergence of highly capable large language models (LLMs) has marked an increased attention in downstream task specialization. This specialization often leverages parameter-efficient fine-tuning (PEFT) techniques, such as LoRA (Hu et al., ), which introduce minimal trainable parameters (“adapters") to adapt pre-trained LLMs for specific tasks. The compact size of these specialized PEFT modules enables easy sharing of these modules, which has led to the distribution of an evergrowing number of adapters on various platforms.

This proliferation of expert models, i.e. specialized adapters, has led to the development of methods for re-using such experts to improve performance or generalization (Muqeeth et al., ; Ostapenko et al., ; Huang et al., ). Central to these approaches are routing mechanisms that adaptively select relevant experts for a particular task or query. These routing methods have been referred to as “Model MoErging” (Yadav et al., ) since they frequently share methodologies and ideas with mixture-of-experts (MoE) models (Shazeer et al., ; Fedus et al., ; Du et al., ) and model merging (Yadav et al., ; ; Ilharco et al., ). However, MoE methods that train experts jointly from scratch (Gupta et al., ) while MoErging utilizes a decentralized, community-sourced pool of pre-trained experts. Furthermore, it departs from traditional model merging techniques by dynamically and adaptively combining these experts, optimizing performance at the query or task level.
MoErging methods offer three key advantages: ($1$) They support decentralized model development by reusing and routing among independently trained experts, reducing reliance on centralized resources. ($2$) They facilitate modular capability expansion and “transparency" in updates as they either add or modify specialized expert models.
3) They allow for compositional generalization by recombining fine-grained skills from various experts, extending the system’s abilities to new unseen tasks beyond the capabilities of the individual expert models.

Most existing methods for MoErging often prioritize performance on either known expert tasks (held-in) or generalization to unseen tasks (held-out) depending on their use cases (Chronopoulou et al., ; Muqeeth et al., ; Zhao et al., ).
This specialization limits practical applicability, as real-world deployments demand robust performance across both held-in and held-out tasks. Consequently, existing methods exhibit suboptimal performance when evaluated on both held-in and held-out tasks, often leading to suboptimal overall performance.
For example, while Phatgoose (Muqeeth et al., ) demonstrate strong performance on held-out data, they do not perform well on held-in tasks.
We hypothesize that this gap arises from the model’s token-level routing mechanism. We show that for the held-in tasks the independent routing decisions at each layer, based solely on individual token embeddings, lack sufficient global context to retrieve the correct expert for all token at every module.
This leads to suboptimal routing which may propagate noise through the network, further hindering accurate expert utilization in deeper layers.
This highlights a critical limitation of token-level approaches to handling both held-in tasks, which hence falls short of the goal of building a routing system that seamlessly handles arbitrary queries.
We believe that adding a global routing mechanism based on semantic task information can further aid the token level router for correct retrieval for held-in tasks. Hence, we ask the question.

This paper addresses the challenges by investigating the potential of leveraging the inherent reasoning and generalization capabilities of LLMs to guide the routing process in an MoE-like model composed of specialized LoRA modules. We introduce, Global and Local Instruction Driven Expert Router ($\mathtt{GLIDER}$) that hinges on a multi-scale routing mechanism that contains both local and global routers as shown in Figure 1. The global router leverages LLM-generated, semantics-aware instructions (see Appendix A.2) to select the top-$2$ expert models for each input query across all the layers. This high-level guidance is then complemented by a learned local router, which makes token-level routing decisions at each module, enabling fine-grained control and improving performance on the challenging held-out tasks. Through this framework, we highlight the crucial role of LLM reasoning in unlocking the compositional generalization capabilities of MoE models.

To test the effectiveness of our $\mathtt{GLIDER}$ method, we follow Phatgoose (Muqeeth et al., ) and use T5 models (Raffel et al., ) to create expert models for T0 held-in (Sanh et al., ) and FLAN tasks (Longpre et al., ) and test performance on T0 held-in & held-out (Sanh et al., ) and big-bench lite (BIG-bench authors, ) & hard tasks (Suzgun et al., ).
Our key contributions and findings are:

- •
We introduce $\mathtt{GLIDER}$, which employs LLM-guided multi-scale global and local attention. Our experiments show that $\mathtt{GLIDER}$ outperforms previous methods, significantly improving performance on held-in tasks (e.g. $6.6\%$ over Phatgoose on T0 held-in) while also enhancing zero-shot held-out compositional generalization (e.g. $0.9\%$ over Phatgoose on T0 held-out).
- •
We find that without LLM assistance, MoE models underperform individual specialized models on held-in tasks by $8.2\%$. Incorporating semantic-aware instructions enables $\mathtt{GLIDER}$ to achieve comparable performance, demonstrating the LLM’s capacity to effectively infer task identity and guide module selection without explicit task labels.
- •
$\mathtt{GLIDER}$ also maintains strong performance on held-out tasks, showcasing its adaptability and generalization capabilities. Our work highlights the critical role of LLMs in enhancing MoE models’ compositional generalization, advancing the development of more robust and versatile AI systems capable of handling both familiar and novel tasks.

## 2 Related Works

#### MoErging Methods.

The abundance of specialized expert models has spurred the development of techniques to leverage “experts" models for enhanced performance and generalization. Yadav et al. () in their recent survey called such techniques as “MoErging" (^*^**See e.g. [https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)) methods which rely on adaptive routing mechanisms to select relevant experts for specific tasks or queries.
These methods can be broadly classified into four categories based on the design of their routing mechanisms.

$\mathtt{Embedding-Based~{}Routing:}$ This category encompasses methods that derive routing decisions from learned embeddings of expert training data. These methods typically compare a query embedding against the learned expert embeddings to determine the optimal routing path. Examples include AdapterSoup (Chronopoulou et al., ), Retrieval of Experts (Jang et al., ), Token-Level Adaptation (Belofsky, ), LoraRetriever (Zhao et al., ), Mo’LoRA (Maxine, ), the embedding-based approach of Airoboros (Durbin, ), and Dynamic Adapter Merging (Cheng et al., ).

$\mathtt{Classifier-Based~{}Routing:}$ This category consists of methods that train a router to function as a classifier. This router is trained to predict the optimal routing path based on features extracted from expert datasets or unseen data. Representative methods in this category include Zooter (Lu et al., ), Branch-Train-Mix (Sukhbaatar et al., ), Routing with Benchmark Datasets (Shnitzer et al., ), Routoo (Mohammadshahi et al., ), and RouteLLM (Ong et al., ). The key distinction between embedding-based and classifier-based routing lies in the router’s architecture and training methodology. While embedding-based routing often employs a nearest neighbor approach, classifier-based routing typically relies on logistic regression or analogous classification techniques.

$\mathtt{Task-Specific~{}Routing:}$ This category focuses on methods tailored to enhance performance on specific target tasks. These methods learn a task-specific routing distribution over the target dataset to optimize performance for the given task. Methods in this category include LoraHub (Huang et al., ), LoRA-Flow (Wang et al., ), AdapterFusion (Pfeiffer et al., ), $\pi$-Tuning (Wu et al., ), Co-LLM (Shen et al., ), Weight-Ensembling MoE (Tang et al., ), MoLE (Wu et al., ), MeteoRA (Xu et al., ), PEMT (Lin et al., ), MixDA (Diao et al., ), and Twin-Merging (Lu et al., ).

$\mathtt{Routerless~{}Methods:}$ This final category encompasses methods that do not rely on an explicitly trained router. Instead, these methods often employ alternative mechanisms, such as heuristics or rule-based systems, for routing decisions. Examples include Arrow $\nearrow$ (Ostapenko et al., ), PHATGOOSE (Muqeeth et al., ), the “ask an LLM" routing of Airoboros (Durbin, ) and LlamaIndex (Liu, ).

#### Model Merging.

Model merging (Yadav et al., ; Choshen et al., ; Wortsman et al., ; Ramé et al., ; Matena & Raffel, ; Ilharco et al., ; Tam et al., ; Jin et al., ; Yang et al., ) consolidates multiple independently trained models with identical architectures into a unified model that preserves individual model capabilities. While simple parameter averaging suffices for models within a linearly connected low-loss parameter space (McMahan et al., ; Stich, ; Frankle et al., ; Wortsman et al., ), more sophisticated techniques are necessary for complex scenarios. For instance, task vectors facilitate merging expert models trained on diverse domains (Ilharco et al., ). Additionally, methods like weighted merging using Fisher Importance Matrices (Matena & Raffel, ; Tam et al., ) and TIES-Merging, which addresses sign disagreements and redundancy (Yadav et al., ) offers improved performance. As a non-adaptive expert aggregation method, merging serves as a fundamental baseline for numerous Model Editing with Regularization (MoErging) techniques.

#### Multitask Learning (MTL).

research offers valuable insights for decentralized development. Notably, investigations into task-relatedness (Standley et al., ; Bingel & Søgaard, ; Achille et al., ; Vu et al., ; Zamir et al., ; Mou et al., ) provide guidance for designing routing mechanisms, while MTL architectures addressing the balance between shared and task-specific knowledge (Misra et al., ; Ruder et al., ; Meyerson & Miikkulainen, ; Zaremoodi et al., ; Sun et al., ) offer strategies for combining expert contributions in a decentralized manner.

#### MoE for Multitask Learning.

Recent research has extensively investigated mixture-of-experts (MoE) models for multitask learning, achieving promising results in unseen task generalization. These approaches generally fall into two categories: ($1$) Example Routing: Studies like Muqeeth et al. (); Zadouri et al. (); Wang et al. () train routers to dynamically select experts for each input, while Caccia et al. () demonstrate the efficacy of routing at a finer granularity by splitting expert parameters into blocks. ($2$) Task Routing: Ponti et al. () employs a trainable skill matrix to assign tasks to specific parameter-efficient modules, while Gupta et al. () leverages task-specific routers selected based on domain knowledge. Ye et al. () proposes a layer-wise expert selection mechanism informed by task representations derived from input embeddings. Such approaches leverage task-specific representation to allow the router to effectively select the most suitable experts for unseen tasks. While these studies differ from our setting by assuming simultaneous data access, they offer valuable insights applicable to our exploration of creating routing mechanisms over expert models.

Figure: Figure 2: We present routing heatmaps for $\mathtt{GLIDER}$ and Phatgoose on two held-in and two held-out tasks. For held-in tasks, oracle experts are marked with red dashed lines. $\mathtt{GLIDER}$ selects oracle experts more frequently than Phatgoose for held-in tasks, leading to improvements of $3.3\%$ on CommonGen and $6.5\%$ on PAWS. For held-out tasks, $\mathtt{GLIDER}$ also tends to select the most relevant experts across most LoRA modules, resulting in improvements of $2.2\%$ on COPA and $5.8\%$ on StoryCloze.
Refer to caption: x2.png

## 3 Problem Statement

In our work, we aim to build a routing mechanism capable of performing well on diverse queries from various tasks, including both seen and unseen tasks. For each query/token and module, this routing mechanism dynamically selects a model from a large pool of specialized expert models to achieve high performance.
To facilitate modular development, we adopt a contributor-aggregator framework (Yadav et al., ) where individual contributors create specialized expert models from a generalist model for their respective tasks and distribute these models to others for public usage. The aggregator builds a routing mechanism over the expert models that shared by the contributor to direct queries to the most relevant experts.
Following recent works (Muqeeth et al., ; Ostapenko et al., ), we use parameter-efficient finetuning (PEFT) (Liu et al., ; Sung et al., ; Poth et al., ) methods like LoRA (Hu et al., ) to train the expert models. Since PEFT typically has lower computational and communication costs than full-model finetuning (Hu et al., ; Liu et al., ), the use of PEFT makes it easier to participate and contribute. PEFT methods introduce modules throughout the model – for example, LoRA (Hu et al., ) introduces a low-rank update at every linear layer in the model. We refer to each of these updates as a module.
Subsequently, the trained expert models and additional information are shared with the aggregators. The aggregator’s job is to collect these expert models and the additional information and design the post-hoc routing mechanism. This mechanism will effectively direct incoming queries to the most appropriate expert model for each token and at each module to ensure optimal performance on both seen and unseen tasks. This approach allows for the seamless integration of new capabilities by adding expert models to the existing pool. Next, we formally define our contributor-aggregator framework.

Let us assume that there are $N$ contributors, $\{c_{1},c_{2},\ldots,c_{\mathtt{N}}\}$, and each contributor $c_{i}$ has access to a task-specific datasets $\mathcal{D}_{i}$. Each contributor, $c_{i}$, follows the predefined training protocol $\mathcal{T}$ provided by the aggregator.
The training protocol ($\mathcal{T}$) takes in a base model ($\bm{\theta}_{\textrm{base}}$) and a dataset ($\mathcal{D}_{i}$). It returns the expert model parameters ($\phi_{i}$) along with any additional information ($\Psi_{i}$) that needs to be shared with the aggregators, for example, the gate vectors described in Section 4.1. Specifically, $\{\phi_{i},~{}\Psi_{i}\}\leftarrow\mathcal{T}(\bm{\theta}_{\textrm{base}},
\mathcal{D}_{i})$.
All contributors share this information with the aggregator, which creates a pool of models containing $\{(\phi_{i},\Psi_{i})\}_{i=1}^{\mathtt{N}}$. The aggregators ($\mathcal{A}$) then uses these expert models and the auxiliary information to create a routing mechanism $\mathcal{R}(.)$ that takes the user query $q$ as the input and return routing path describing how the information is routed through the given set of expert models. Formally, $\mathcal{R}(.)\leftarrow\mathcal{A}(\{(\phi_{i},\Psi_{i})\}_{i=1}^{\mathtt{N}})$. The function $\mathcal{R}(.)$ describe the full path of input query by making various choices about 1) expert input granularity, choosing to route per-token, per-query, or per-task, 2) expert depth granularity, opting for either per-module or model-level routing, and 3) selecting between sparse or dense routing. Finally, the aggregator uses the routing mechanism to answer incoming queries.

## 4 Methodology

To recap, our goal is to build a MoErging method that dynamically routing queries to a diverse pool of specialized expert models, addressing the challenge of effectively handling queries from various tasks and ensuring both held-in and held-out performance. Our proposed method, Global and Local Instruction Driven Expert Router ($\mathtt{GLIDER}$), leverages a combination of local and global routing vectors to achieve this goal. Specifically, contributors train task-specific routing vectors, while a large language model (LLM) generates a global semantic task instructions which are then converted to global instruction routing vectors. During inference, these local and global routing vectors are combined to perform top-k discrete routing, directing queries to the most suitable expert model. This process is visualized in Figure 1 and described in detail below.

### 4.1 Expert Training Protocol

Our expert training protocol $\mathcal{T}$ takes as input the base model parameters, $\theta_{\textrm{base}}$, and a dataset $\mathtt{d}$ and performs three steps to obtain the required output. First, we train the LoRA experts ($\phi$), then train the local routing vectors ($\mathtt{l}$) while keeping the LoRA experts fixed. Finally, we train obtain the global routing vector ($\mathtt{g}$) by using an LLM and an embedding model. Formally, in our case, $\phi,~{}\Psi=\{\mathtt{l},\mathtt{g}\}\leftarrow\mathcal{T}(\theta_{\textrm{
base}},\mathtt{d})$ which are then shared with the aggregators to create the routing mechanism. We described these steps in detail below.

#### PEFT Training of Expert Model.

$\mathtt{GLIDER}$ is compatible with expert models trained using parameter-efficient finetuning methods (e.g. LoRA (Hu et al., ), Adapters (Houlsby et al., )) that introduce small trainable modules throughout the model. We focus on PEFT experts because they typically have lower computational and communication costs than full-model finetuning (Yadav et al., ), making it easier to train and share expert models. Following Phatgoose (Muqeeth et al., ), this work specifically focuses in LoRA (Hu et al., ) due to its widespread use.
LoRA introduces a module comprising the trainable matrices $\mathtt{B}\in\mathbb{R}^{\mathtt{d}\times\mathtt{r}}$ and $\mathtt{A}\in\mathbb{R}^{\mathtt{r}\times\mathtt{n}}$ in parallel to each linear layer with parameters $\mathtt{W}\in\mathbb{R}^{\mathtt{d}\times\mathtt{n}}$. Given the $\mathtt{t}^{\text{th}}$ input token activation $\mathtt{u_{i}}$, LoRA modifies the output of the linear layer from $\mathtt{Wu_{i}}$ to $\mathtt{Wu_{i}+\frac{\alpha}{r}*BAu_{i}}$ where $\mathtt{\alpha}$ is a constant and usually is set to $\mathtt{1}$. During training, the matrices $\mathtt{A}$ and $\mathtt{B}$ are trainable while the original linear layer $\mathtt{W}$ is kept frozen. We denote the final trained expert parameters with $\phi=\{(\mathtt{A_{1}},\mathtt{B_{1}}),\ldots,(\mathtt{A_{m}},\mathtt{B_{m}})\}$, where $\mathtt{m}$ is the number of modules in the model.

#### Training Local Routing Vectors.

Following Phatgoose (Muqeeth et al., ), after training the PEFT modules on their dataset, a local router is introduced before each PEFT module. This router, employing a shared vector across all queries and tokens, dynamically determines the utilization of the PEFT module based on the input token activations. The router is trained for a small number of steps using the same dataset and objective as the PEFT module, while keeping the expert PEFT parameters fixed. This process effectively learns to associate the token activation patterns with the learned expert model. For LoRA, the local router, represented by a trainable vector $\mathtt{v}\in\mathbb{R}^{\mathtt{d}}$, controls the contribution of the PEFT module to the final output. This results in a modified linear layer of the form $\mathtt{Wu_{i}+\frac{\alpha}{r}*BAu_{i}*\sigma(v^{\mathsf{T}}u_{i})}$, where $\mathtt{\alpha}$, $\mathtt{W}$, $\mathtt{B}$, and $\mathtt{A}$ are frozen, and the local router $\mathtt{v}$ is learned. We denote the final local routing vectors as $\mathtt{l}=\{\mathtt{v_{1}},\ldots,\mathtt{v_{m}}\}$ where $\mathtt{m}$ is the number of modules in the model.

#### Creating LLM-Aided Global Routing Vector.

The local routing vectors capture the intricate relationships between token activations and expert models, enabling efficient query routing in cases where no dedicated expert is available. Conversely, for queries corresponding to held-in tasks, direct retrieval of the relevant expert model is preferred to process the full query. For this purpose, we create a global routing vector that utilizes an LLM to generate a semantically-informed instruction, termed as task description, which effectively captures the essence of the kind of queries the expert can handle. We prompt an LLM with three randomly selected in-context examples to generate this task description. We used the gpt-4-turbo model along with the prompt provided in Appendix A. The resulting task description is then embedded using an off-the-shelf embedding model, specifically the nomic-embed-text-v1.5 model, to produce a global routing vector for the task. We denote the global routing vector as $\mathtt{g}\in\mathbb{R}^{\mathtt{d_{g}}}$.

### 4.2 𝙶𝙻𝙸𝙳𝙴𝚁 𝙶𝙻𝙸𝙳𝙴𝚁 \mathtt{GLIDER} typewriter_GLIDER : Inference Expert Aggregation Phase

Following training, all contributors share their expert models along with the auxiliary information comprising of the local and global routing vectors, $\{\mathtt{\phi^{t}},~{}\mathtt{l^{t}},~{}\mathtt{g^{t}}\}_{\mathtt{t=1}}^{
\mathtt{N}}$ with the aggregators. The $\mathtt{GLIDER}$ method subsequently leverages this information to perform inference on arbitrary queries.

#### Local Router.

Before each input module $\mathtt{m}$, a separate local router $\mathtt{L}_{\mathtt{m}}\in\mathbb{R}^{\mathtt{N\times d}}$ is inserted to make local per-token, per-module routing decisions. For a given module $\mathtt{m}$ and expert model $\mathtt{c}$, we first standardize the task-specific local routing vectors $\mathtt{v_{m}^{c}}$ by subtracting its mean and dividing by the standard deviation to obtain $\bar{\mathtt{v}}_{\mathtt{m}}^{\mathtt{c}}$. Next, we obtain the local router for module $\mathtt{m}$ by stacking these standardised local routing vectors as $\mathtt{L}_{\mathtt{m}}=[\bar{\mathtt{v}}_{\mathtt{m}}^{\mathtt{1}};\ldots;
\bar{\mathtt{v}}_{\mathtt{m}}^{\mathtt{N}}]\in\mathbb{R}^{\mathtt{N\times d}}$. Next, for each token $\mathtt{i}$ with activation $\mathtt{u_{i}}$ coming into module $\mathtt{m}$, we standardise it to obtain $\bar{\mathtt{u}}_{\mathtt{i}}$. We then compute the local affinity scores, $\mathtt{s}^{\mathtt{loc}}_{\mathtt{m}}\in\mathbb{R}^{\mathtt{N}}$ between the local router $\mathtt{L}_{\mathtt{m}}$ and $\mathtt{u_{i}}$ as $\mathtt{s}^{\mathtt{loc}}_{\mathtt{m}}=\texttt{cos-sim}(\mathtt{L}_{\mathtt{m}
},\mathtt{u_{i}})$.

#### Global Router.

The global router aims to capture task semantics to retrieve relevant experts for any given input query. We create the global router $\mathtt{G}\in\mathbb{R}^{\mathtt{N\times d_{g}}}$ by stacking the global routing vectors from all the expert models as $\mathtt{G}=[\mathtt{g}^{\mathtt{1}};\ldots;\mathtt{g}^{\mathtt{N}}]$. This router is not a part of the base model and is added before the model to independently process the fully query. Given an input query $\mathtt{u}$ along with three few-shot input-output pairs of similar queries, we prompt an LLM (gpt-4-turbo) using the template provided in Appendix A to obtain a task description for the query. We then embed this task description using the same embedding model (nomic-embed-text-v1.5) to obtain the vector $\mathtt{q_{u}}\in\mathbb{R}^{\mathtt{d_{g}}}$. We then compute the global affinity score, $\mathtt{s}^{\mathtt{glob}}\in\mathbb{R}^{\mathtt{N}}$, by computing the cosine similarity as $\mathtt{s}^{\mathtt{glob}}=\texttt{cos-sim}(\mathtt{G},\mathtt{q_{u}})$.

#### Combining Global and Local Router.

At each module $\mathtt{m}$, we have the global and local affinity scores $\mathtt{s}^{\mathtt{glob}}$ and $\mathtt{s}^{\mathtt{loc}}_{\mathtt{m}}$ respectively. Following Phatgoose (Muqeeth et al., ), we scale the local scores with a factor of $1/\sqrt{\mathtt{N}}$. However, the global router’s main goal is to retrieve the correct expert for the held-in tasks. Therefore, we first check if the expert with the highest global affinity score ($\texttt{max}(\mathtt{s}^{\mathtt{glob}})$) is above a threshold ($\mathtt{p}$). If such experts exist, then we set a high $\alpha$ to enforce retrieval and vice versa. Hence, we propose to scale the global scores with $\alpha$, where $\alpha=\gamma*\mathbb{I}_{\{\texttt{max}(\mathtt{s}^{\mathtt{glob}})-\mathtt{p
}>0\}}+\beta$, where $\mathtt{p}$ is the cosine similarity threshold, and $\gamma$ and $\beta$ are scaling hyperparameters. Using our ablation experiments in Section 5.4, we set $\mathtt{p}=\mathtt{0.8}$, $\gamma=100$ and $\beta=3$.
We then obtain the final affinity score $\mathtt{s}\in\mathbb{R}^{\mathtt{N}}=\alpha*\mathtt{s}^{\mathtt{glob}}$ + $\mathtt{s}^{\mathtt{loc}}_{\mathtt{m}}/\sqrt{\mathtt{N}}$. Then $\mathtt{GLIDER}$ selects the top-$\mathtt{k}$ experts after performing softmax over the final affinity score $\mathtt{s}$ as $\mathcal{E}_{\mathtt{top}}$ = top-$\mathtt{k}(\texttt{softmax}(\mathtt{s}))$.
Finally, the output of the module for token activation $u_{i}$ is computed as $\mathtt{Wu_{i}}+\sum_{\mathtt{k}\in\mathcal{E}_{\mathtt{top}}}\mathtt{w_{k}*B_
{k}A_{k}u_{i}}$.

## 5 Experiments

### 5.1 Setting

#### Dataset.

Our experiments utilize the multitask prompted training setup introduced by Sanh et al. (), which has become a standard benchmark for evaluating generalization to unseen tasks (Chung et al., ; Longpre et al., ; Jang et al., ; Zhou et al., ). Following Phatgoose (Muqeeth et al., ), we employ LM-adapted T5.1.1 XL (Lester et al., ) as our base model which is a 3B parameter variant of T5 (Raffel et al., ) further trained on the C4 dataset using a standard language modeling objective. For held-out evaluations, we follow Phatgoose (Muqeeth et al., ) and use three held-out benchmark collections. We use the T0 held-out (T0HO) datasets used in Sanh et al. () and the two subsets of BIG-bench (BIG-bench authors, ). Specifically, we use BIG-bench Hard (BBH) (Suzgun et al., ), consisting of 23 challenging datasets, and BIG-bench Lite (BBL) (BIG-bench authors, ), a lightweight 24-dataset proxy for the full benchmark. Similar to Muqeeth et al. (), we exclude certain BIG-bench datasets due to tokenization incompatibility with the T5 tokenizer.

#### Expert Creation.

To create the pool of expert module for routing, we follow Muqeeth et al. () and use two distinct dataset collections: ❶ T0 Held-In (Sanh et al., ) consisting of the 36 held-in prompted datasets for tasks from the T0 training procedure. ❷ The “FLAN Collection" (Longpre et al., ) which significantly expands the T0 tasks by incorporating prompted datasets from SuperGLUE (Wang et al., ), Super Natural Instructions (Wang et al., ), dialogue datasets, and Chain-of-Thought datasets (Wei et al., ). Following Muqeeth et al. (), we create $\mathtt{166}$ specialized models from the FLAN Collection. For each dataset in these collections, we train Low-Rank Adapters (LoRAs) (Hu et al., ) modules resulting in pools of $\mathtt{36}$ and $\mathtt{166}$ expert models for T0 Held-In and FLAN, respectively. Similar to Phatgoose, we use a rank of $r=16$ and train for $\mathtt{1000}$ steps using the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $5\times 10^{-3}$ and a warmup ratio of $0.06$. After training the LoRA module, we freeze it and train the local routing vectors for an additional 100 steps with the same hyperparameters. Finally, following prior work (Shazeer et al., ; Du et al., ; Lepikhin et al., ), $\mathtt{GLIDER}$ performs top-$k$ routing with $k=2$.

### 5.2 Baselines

Expert Merging.
Model Merging (Yadav et al., ; Choshen et al., ) involves averaging the parameters of multiple models or modules to create a single aggregate model. We merge by multiplying the LoRA matrices and then taking an unweighted average of all the experts within the pool. It is important to note that this merging strategy requires homogeneous expert module architectures; in contrast, $\mathtt{GLIDER}$ can accommodate heterogeneous expert modules.

Arrow.
Following Ostapenko et al. (), we employ a routing mechanism where gating vectors are derived from LoRA expert modules. Specifically, the first right singular vector of the outer product of each module’s LoRA update ($BA$) serves as its gating vector. Input routing is determined by a probability distribution based on the absolute dot product between the input representation and each gating vector. We utilize top-$k$ routing with $k=2$.

Phatgoose. Phatgoose (Muqeeth et al., ) first learn the LoRA modules for each, followed by learning a sigmoid gating vector similar to our local router. During inference, they make routing decisions for each token independently for all modules. Specifically, they first standardize the input token activations and gating vectors from all experts and then perform similarity-based top-2 routing.

LoRA Hub.
LoraHub (Huang et al., ) method performs gradient-free optimization using few-shot task samples to learn mixing coefficients for different expert models while keeping them fixed. Once the coefficients are learned, they merge the experts with the learned weight and route through the merged expert.

Multi-task Fine-Tuning.
While multitask training, a proven method for enhancing zero-shot generalization (Sanh et al., ; Wei et al., ), is infeasible given our problem setting and data access limitations, we include it as a baseline using publicly available models. Specifically, we utilize the T0-3B model (Sanh et al., ) for the T0 Held-In datasets, given its training on a matching dataset collection. For FLAN, a directly comparable publicly available model is unavailable; therefore, we report results for FLAN-T5 XL, trained on a different, undisclosed dataset mixture, while acknowledging the limitations of this indirect comparison.

Oracle.
Following Jang et al. () and Muqeeth et al. (), we employ an Oracle routing scheme as a performance upper bound. This scheme selects the expert exhibiting optimal performance on a given evaluation dataset, thus representing a non-zero-shot approach.

### 5.3 Main Results

Table 1 presents the comparison results among our $\mathtt{GLIDER}$ and six baselines on both held-in and held-out settings. To further illustrate the performance, we also include the results of Oracle Expert, which has extra access to the task identities of expert modules and evaluated datasets and can be regarded as an upper bound.

#### T0 Setting.

In the T0 task set, the following observations can be drawn: ❶ For the held-in tasks, i.e. T0-HI, $\mathtt{GLIDER}$ significantly outperforms other baselines and almost matches the performance of Oracle Expert upper bound. ❷ For T0-HO and BBL tasks, $\mathtt{GLIDER}$ achieves the best performance among all the methods, including Oracle Expert upper bound. ❸ $\mathtt{GLIDER}$ has negligible lower performance, i.e. $0.01\%$, compared to the Expert Merging baseline in BBH but outperforms it by around $12\%$ on T0-HO and $1.5\%$ on BBL. Besides Expert Merging, $\mathtt{GLIDER}$ outperforms all other methods on BBH, including the Oracle Expert upper bound.

**Table 1: Performance evaluated on the T0 set and FLAN set. We present the performance on both held-in tasks (i.e. T0-HI) and held-out tasks (i.e. T0-HO, BBH, and BBL). We compare the following methods: ($1$) performance upper bound, i.e. Oracle Expert; ($2$) zero-shot baselines, i.e. Multi-Task Fine-Tuning, Expert Merging, Arrow, and Phatgoose; ($3$) few-shot baselines, i.e. LoRA Hub and $\mathtt{GLIDER}$. We mark the best performance besides the upper bound (i.e., Oracle Expert) in bold.**
| Method | T0 | FLAN |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| T0-HI | T0-HO | BBH | BBL | BBH | BBL |  |
| Oracle Expert | $69.60$ | $51.60$ | $34.90$ | $36.60$ | $38.90$ | $45.40$ |
| Multi-Task Fine-Tuning | $55.90$ | $51.60$ | $34.90$ | $36.60$ | $\mathbf{38.90}$ | $\mathbf{45.40}$ |
| Expert Merging | $30.73$ | $45.40$ | $\mathbf{35.30}$ | $36.00$ | $34.60$ | $34.00$ |
| Arrow | $39.84$ | $55.10$ | $33.60$ | $34.50$ | $30.60$ | $29.60$ |
| Phatgoose | $61.42$ | $56.90$ | $34.90$ | $37.30$ | $35.60$ | $35.20$ |
| LoRA Hub | $31.90$ | $46.85$ | $31.35$ | $31.18$ | $34.50$ | $30.54$ |
| $\mathtt{GLIDER}$ | $\mathbf{68.04}$ | $\mathbf{57.78}$ | $35.29$ | $\mathbf{37.46}$ | $35.07$ | $35.52$ |

Figure: Figure 3: Global routing scores for tasks in the T0 set. The red horizontal line indicates our design threshold of $0.8$. Each column represents an evaluated task from T0-HI, T0-HO, BigBench using T0 held-in experts. All global routing scores for each task are plotted, corresponding to the $35$ experts in total.
Refer to caption: x3.png

### 5.4 Ablation Study and Further Investigation

**Table 2: Ablation on the instruction coefficient $\alpha$. We mark the best performance in bold and the performance corresponding to the selected $\alpha$ by $\mathtt{GLIDER}$ in blue.**
| $\alpha$ | T0 |  |  |  |
| --- | --- | --- | --- | --- |
| T0-HI | T0-HO | BBH | BBL |  |
| $1$ | $62.20$ | $57.04$ | $35.05$ | $\mathbf{37.79}$ |
| $3$ | $63.40$ | $57.78$ | $\mathbf{35.29}$ | $37.46$ |
| $10$ | $65.52$ | $\mathbf{57.98}$ | $34.80$ | $37.04$ |
| $100$ | $\mathbf{68.04}$ | $53.22$ | $31.73$ | $34.97$ |
| $1000$ | $66.88$ | $52.91$ | $30.71$ | $34.31$ |
| $3000$ | $66.69$ | $52.37$ | $30.03$ | $33.24$ |

#### Ablation on the global routing scale α 𝛼 \alpha italic_α .

To illustrate how the specialization and generalization abilities change as we scale the coefficient $\alpha$ of the global routing score, we conduct the ablation study of $\alpha$ ranging $\{1,3,10,100,1000,3000\}$. As shown in Table 3, we present experimental results of the T0 task set on both held-in and held-out tasks. For held-in tasks, i.e. T0-HI, $\mathtt{GLIDER}$ can select the optimal $\alpha$ to scale the global routing score. For held-out tasks, i.e. {T0-HO, BBH, BBL}, $\mathtt{GLIDER}$ produce either the optimal $\alpha$ (for BBH) or the sub-optimal $\alpha$ with slightly lower performance to the optimal ones (for T0-HO and BBL).

#### Ablation on the routing strategy.

There exists a trade-off between performance and efficiency when using different $\mathtt{top}\texttt{-}\mathtt{k}$ routing strategies (Ramachandran & Le, ). To investigate the impact of routing strategy in $\mathtt{GLIDER}$, we evaluate $\mathtt{top}\texttt{-}\mathtt{k}$ routing of $\mathtt{k}$ in $\{1,2,3\}$. Moreover, we further evaluate the $\mathtt{top}\texttt{-}\mathtt{p}$ routing (Huang et al., ; Zeng et al., ) of $\mathtt{p}$ in $\{25\%,50\%,75\%\}$, where each token selects experts with higher routing probabilities until the cumulative probability exceeds threshold $\mathtt{p}$. As shown in Table 3, we can draw the following conclusions: ($1$) For $\mathtt{top}\texttt{-}\mathtt{k}$ routing, $\mathtt{k}=2$ shows comparable or better performance than $\mathtt{k}=3$, particularly for T0-HO and BBH, while offering improved efficiency. ($2$) For $\mathtt{top}\texttt{-}\mathtt{p}$ routing, higher $\mathtt{p}$ values consistently yield better performance at the cost of efficiency. Therefore, we use $\mathtt{top}\texttt{-}\mathtt{2}$ routing in $\mathtt{GLIDER}$ by default.

#### Investigation on the threshold design of global scores.

As described in Section 4, we compute the scale $\alpha$ for global scores using the formula $\alpha=\gamma*\mathbb{I}_{\{\texttt{max}(\mathtt{s}^{\mathtt{glob}})-0.8>0\}}+\beta$, where we establish a threshold of $0.8$ to differentiate evaluated tasks. Figure 3 presents the global routing scores for each task in the T0 set to motivate the rationale behind this design. For all held-in tasks (i.e., T0-HI), at least one expert (typically the oracle expert trained on the evaluated task) achieves global routing scores exceeding $0.8$. Consequently, $\mathtt{GLIDER}$ applies a higher $\alpha=100$, enabling effective identification of tasks corresponding to a specifically trained expert and enhancing retrieval of this oracle expert. For nearly all held-out tasks (i.e., T0-HO and BigBench), no global routing score surpasses $0.8$, prompting $\mathtt{GLIDER}$ to utilize a lower $\alpha=3$. Two exceptions among the held-out tasks are bbq_lite_json and strange_stories in BigBench, as shown in the figure, where one score marginally exceeds $0.8$ in each case. For these two, $\mathtt{GLIDER}$ employs the higher $\alpha=100$, resulting in performance improvements of $1.3\%$ and $2.9\%$ respectively over $\alpha=3$, thus showing the effectiveness of our design.

## 6 Conclusion

This paper introduces $\mathtt{GLIDER}$, a novel multi-scale routing mechanism that incorporates both global semantic and local token-level routers. By leveraging the semantic reasoning capabilities of LLMs for global expert selection and refining these choices with a learned local router, $\mathtt{GLIDER}$ addresses the limitations of existing methods that often perform poorly on held-in tasks. Our empirical evaluation on T0 and FLAN benchmarks, using T5-based experts, demonstrates that $\mathtt{GLIDER}$ achieves substantial improvements in held-in task performance while maintaining competitive generalization on held-out tasks. These findings suggest that incorporating global semantic task context into routing mechanisms is crucial for building robust and practically useful routing-based systems.

## Appendix

## Appendix A LLM for Task Instruction Generation.

### A.1 Prompt Template

We use the following prompt with $3$ randomly selected samples for each task to generate its description. The prompt is then fed into the gpt-4-turbo OpenAI API to get the generated task descriptions.

### A.2 Examples of the Generated Instructions

We provide several examples of LLM-generated instructions in this section.

WikiBio (Lebret et al., ) (T0 Held-In):

- •
Create a short biography using the provided facts, demonstrating knowledge in historical and biographical writing.
- •
Write a short biography based on the given factual bullet points, demonstrating proficiency in summarizing and transforming structured data into coherent narrative text.

CommonGen (Lin et al., ) (T0 Held-In):

- •
Generate a coherent sentence using all the given abstract concepts, requiring the skill of concept integration to form a meaningful sentence.
- •
Generate a coherent sentence by creatively combining a given set of abstract concepts.

COPA (Huang et al., ) (T0 Held-Out):

- •
Identify the most logically consistent sentence from two given options based on the provided context, demonstrating reasoning and causal relationship skills.
- •
Generate the most likely outcome for a given scenario by choosing between two provided options based on contextual clues and causal reasoning.

Date Understanding (Srivastava et al., ) (BigBench-Hard):

- •
Calculate the date based on the given information and present it in MM/DD/YYYY format, ensuring that you accurately account for day, month, and year changes.

Hindu Mythology Trivia (Srivastava et al., ) (BigBench-Lite):

- •
Generate the correct answer by making use of your knowledge in Hindu mythology and culture.

## Appendix B Demonstrating Compositional Generation

In addition to significant improvements on held-in tasks, $\mathtt{GLIDER}$ demonstrates strong performance on held-out tasks, showcasing its generalization capability. To further examine this ability to handle unseen tasks by composing experts, we provide specific task examples illustrating the association between selected experts and the evaluated task. As Figure 2 shows, $\mathtt{GLIDER}$ primarily selects two experts for the COPA (T0 held-out) task, corresponding to CosmosQA and QuaRel. The following three examples from these tasks demonstrate their close semantic relationship:

- •
COPA:
–
Question: Everyone in the class turned to stare at the student.
Select the most plausible cause:
- The student’s phone rang.
- The student took notes.
–
Answer: The student’s phone rang.
- •
CosmosQA:
–
Question: That idea still weirds me out . I made a blanket for the baby ’s older sister before she was born but I completely spaced that this one was on the way , caught up in my own dramas and whatnot . Luckily , I had started a few rows in white just to learn a stitch ages ago , and continuing that stitch will make an acceptable woobie , I think .
According to the above context, choose the best option to answer the following question.
Question: What did I make for the baby .
Options:
A. I made a carseat .
B. None of the above choices .
C. I made a crb .
D. I finished a pair of booties .
–
Answer: D.
- •
QuaRel:
–
Question: Here’s a short story: A piece of thread is much thinner than a tree so it is (A) less strong (B) more strong.
What is the most sensical answer between "Thread" and "Tree"?
–
Answer: Thread.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: Inovation and contribution:
1. The core of this paper combines the scores from the Global Router and the Local Router, then performs a top-k operation based on the combined scores. a. Line 85: The descriptions of the algorithm in the preceding and following contexts are clearly conflicting, making it contradictory to describe the algorithm effectively. b. Relying on GPT-4 turbo to gather information from the full text means the process heavily depends on GPT-4's capabilities. Furthermore, as seen in line 346 and the subsequent ablation experiments, the value of $\alpha$ is significant, indicating a high weight assigned to the Global score.
2. The paper extensively references the work of Muqeeth but fails to explain the rationale behind these references, raising concerns about the originality of the article.
3. Relying on GPT-4-turbo for global semantic information raises some questions about the overall workload and originality of the research. From the formula in line 346 and the subsequent ablation experiments, it appears that the global score heavily influences the final affinity score, making the expert selection largely dependent on GPT-4-turbo. Additionally, the potential latency issues introduced by using GPT-4-turbo during inference are not addressed, which is a significant concern. Overall, placing so much emphasis on GPT in the paper could weaken its persuasive impact.

Format and Typo Issues:
1. 60) Formatting issue: The left parenthesis is missing.
2. 67-68) The sentences are semantically repetitive.
3. 140) What does the arrow ($\rightarrow$) signify here?
4. 216) This figure is drawn in a hard model; it’s unclear what it is trying to convey.
5. 279) The explanation of $\Psi$ is missing.
6. 289-290) The notation used here is confsued.
7. 301) There is no explanation for the newly introduced $\sigma$.
8. 319) Suddenly introducing the variable 𝑡 t from line 289 is confusing.
9. 323) Why isn’t the normalization process presented in a formula? Section 4.2 is poorly written, relying solely on descriptive language, lacking organization.
10. 354) The T0 held-in dataset is omitted.
soundness: 2
strengths: 1. The insights of the paper are to be praised.
2. Very interesting topic and focus on the router optimization. 
3. Use the big model and small model together to solve the problem 
4. Experimental results are good.
summary: This paper focuses on addressing the trade-off between performance improvement and generalization ability in expert modules. It assumes that this issue arises from the lack of global semantic context in token-level routing strategies, it then seeks to resolve this problem by combining global semantics with token-level information through the use of both global and local expert routers during routing.
weaknesses: 1. Writing and Presentation:The paper could benefit from some polishing. There are a number of typos and semantic issues, and the overall formatting could be improved for better readability. Additionally, some figures are a bit challenging to interpret. For instance, Figure 1 is only referenced in Appendix B but appears as the first figure in the Related Works section, which can disrupt the flow and clarity for the reader.

2. Clarity of Background and Concepts: The background and explanation of key concepts in the paper could be clearer. While there are many references to ideas and works by Yadaav, the connections and explanations aren’t sufficiently detailed, which may leave readers a bit confused. In my initial reading, I found myself questioning whether the discussion pertained to a Mixture of Experts (MoE) scenario or Model Merging.

3. Logical Flow and Mathematical Details: The paper seems to lack some logical coherence, especially regarding mathematical descriptions and derivations. There are no thorough mathematical proofs provided, and the modeling of the scenario feels a bit scattered across different sections. Some variable explanations are incomplete, which can be frustrating. Moreover, discussions around problems and solutions could be more precise. For instance, when mentioning that routing strategy issues stem from a lack of global semantics, it would be helpful to have more rigorous mathematical reasoning or experimental evidence to support this claim.

4. Inconsistencies: There are some notable inconsistencies in the paper. For example, in line 85, it mentions that the Global Router selects the top-2 experts based on global semantics. However, the description of the Global Router algorithm starting from line 329 doesn’t reference any top-2 (or top-k) selection process. The top-k expert selection is only brought up later around line 347, based on the final score calculated from the weighted sum of global and local affinity scores. Clarifying these points would enhance the overall coherence of the paper.

5. The experimental design could use some improvement. The main experiments lack detailed explanations, and Figure 3 is somewhat unclear. Many of the experimental configurations seem to mirror those from Muqeeth's work without providing enough context, which might raise questions about the originality of this study. Additionally, the ablation experiments focus on relatively trivial variables, while more significant factors—such as the differences between excluding and including the global semantics generated by GPT-4-turbo—are overlooked. Addressing these points could enhance the depth and rigor of the research.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Could you explain why T5 was chosen as the primary architecture for evaluation? Have you conducted any preliminary experiments with decoder-only models e.g. (LLama3-8B etc)?
* Could you provide more details about the evaluation metrics used in Table 1? What exactly do these numbers represent?
* Since this design integrates routing into the LoRA inference procedure, could you provide a detailed analysis of the additional computational overhead? How much will the inference latency be affected by such design?
soundness: 3
strengths: * The paper leverages LLMs to generate semantic task descriptions, providing global context for routing decisions which is a unique approach not explored in previous routing methods
* The paper well address the limitations of current approaches (focusing on either held-in or held-out tasks) and provides a novel solution integrating both.
summary: This paper presents GLIDER, a method that combines global semantic and local token-level routing for LLMs. The key innovation is using an LLM to generate task instructions that guide expert model selection globally, while using token-level routing locally. Tested on T5-based models with T0 and FLAN benchmarks, GLIDER improves performance on held-in tasks by 6.6% while maintaining strong generalization on held-out tasks. This shows that incorporating LLM-driven semantic understanding helps achieve better routing compared to pure token-level approaches.
weaknesses: * The experiments focus solely on T5, an older encoder-decoder architecture. The effectiveness of GLIDER on modern decoder-only models (like GPT family, LLaMA, etc.) remains unproven, which is crucial given these are now the mainstream architectures for LLMs.
* Table 1 lacks clarity on evaluation metrics and methodological details. Without clear metric definitions and evaluation protocols, it's difficult to fully assess and compare the reported improvements.
* The routing design will bring extra computational overhead, how will GLIDER's inference latency change compare with the normal LoRA decoding methods (vLLM's lora inference module).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: # incremental addition of new experts

One question that is not addressed is the incremental addition of new experts/datasets pair, and what are the consequences (on normalization, etc.). It seems it should be trivial to do this incrementally, but a discussion in the appendix (and an example use-case when, say a previously held-out task becomes held-in so that the router now uses global instead of local weights) would certainly improve the quality of the paper

# limit of global router

the global router is constructed using a sample of 3 questions, which may be ok for very simple and low-diversity datasets, but not for more complex  and diverse tasks,. a more in-depth study of global router across a wider range of datasets with quantitative assessment of global router policies seems necessary
soundness: 2
strengths: this work introduces a routing mechanism to tradeoff between local and global experts, to increase performance on held in tasks, without compromising capability to handle held out tasks.

the goal is clear and the approach is simple (as it is heuristic in nature).
summary: the paper studies ensembles of LLM models (model MoErging), proposing a technique for selecting experts to route tokens to at global (for selection of experts for in profile tasks)  and local levels (to have more flexibility to handle out of distribution tasks)

the paper is incremental in nature (with small differences with respect to Phatgoose, from architectural design choice, to experimental settings and way too many details, to the point it feels it should be named Phatgoose++ instead of Glider)

although appealing, the approach is heurisitic in nature: given this, one would have expected a significantly larger experimental part, including a wider range of tasks (and possible comparison points beyond those adopted in Phatgoose), and a statistical relevant comparison of improvements
weaknesses: # evaluation
as this work has no theoretical basis,  one would have expected a significantly larger experimental part to convince the reader of the generality of the approach on  
- a significant wider range of tasks (and possible comparison points beyond those adopted in Phatgoose),
- further exhibiting a statistical relevant comparison of improvements

this is not the case, so the paper execution is far from being convincing.

Additionally, while the main advantage of this work is to increase performance of held-in tasks, authors additionally point out advantages that are too thin to be worth noting; and they do so in a disturbingly biased manner.  For instance, authors claim 0.9% over held out tasks over Phatgoose (in bold), but the 0.9% (actually 0.88%) is the maximum observed across 5 held out datasets in Tab 3 (for the other 4 is 0.39%, 0.16% -0.53% and 0.3%) 

# empirical evidence of global router

the work is motivated to find *semantical* resemblance beyond tasks. however, the approach on held out tasks seems to leverage *syntactical* resemblance . Fig  1 shows held out tasks to systematically select two experts (one of which seems to be further common to a couple of tasks).  Yet appendix B just shows the tasks to be syntactically similar: i..e, the Q&A pair has a cloze format, which is rather typical of  simple benchmarks. As such, I am little reassured that the performance generalization will be maintained on more complex tasks, and this work is far from fully elucidating the robustness of the proposed method.

Quantitative assessment of global experts over a wider range  of diverse tasks  (say few tens of datasets per type of answer) would have allowed to get true insights about the nature of global experts (e.g., whether the identified expert triggers "cloze" type answers, irrespectively of the semantic of the question)

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: My first question relates to the core motivation of improving held-in performance and the necessity of doing so given that for any held-in task, we always have access to the expert specialized to that task. Could the authors explain scenarios in which using GLIDER is preferable to simply selecting the specialized expert for a known held-in task? 

My second question is to what extent is GLIDER more of a novel component to local routing schemes that aims to encode global context, as opposed to an entirely new model. If GLIDER is more a novel component than an entire model, then I think the authors should include ablation studies on the local router choice, in particular using Arrow.
soundness: 2
strengths: 1) 
The core idea -- that incorporating global information of the specialization of finetuned expert models into local routing schemes can improve expert aggregation algorithms -- is intuitive and persuasive.


2) 
The use of an LLM to encode global semantic information of the overall expert specialization is a creative method for effectively integrating the required global context
summary: The authors propose GLIDER, which is a token-level and module-level hierarchical routing strategy for combining pools of parameter-efficient finetuned expert models, incorporating both local and global information over the specialization of the expert models. The local component learns a gating between LoRA modules which selects the best layer module from the pool of experts for each token. The global component uses an LLM to encode the overall task expertise of each expert, which is then incorporated into the routing scheme to enhance the routing such that it is sensitive both to local module-wise expertise and overall global expertise of the aggregated models. This scheme hopes to maintain strong generalizability to unseen tasks without sacrificing performance on held-in tasks.
weaknesses: 1) 
My first concern relates to the overall problem setting of the paper and its core motivation of improving performance on held-in tasks. The authors claim that existing MoErging methods often prioritize generalization to unseen tasks at the expense of performance on held-in tasks, and indeed in Table 1 the authors report as one of their main results that GLIDER significantly outperforms baselines on held-in tasks. However, performance on held-in tasks is deemed unimportant precisely because we already have access to the specialized expert trained on that exact held-in task, and so we can always retain performance on any given held-in task by simply using the expert specialized to that task. Indeed, this is the reason why Phatgoose does not report results for held-in tasks. For this reason, the 'Oracle Expert' recorded in Table 1 for held-in tasks is not an Oracle but an attainable result for which we always have access - it is just the expert specialized to that given task. 

So in this sense, given that for held-in tasks GLIDER still underperforms the expert specialized to that given task, I'm not yet convinced of one the proposed main benefits of GLIDER, since for any given held-in task we could easily get better performance by just selecting the corresponding expert specialized to that task. 

Fundamentally, I'm not yet persuaded that the performance gains on held-in tasks justify the claim that GLIDER is an overall superior model, since by the problem setting these are not tasks we need to optimize for. If the authors can provide justification for why performance on held-in tasks is indeed important and why selecting GLIDER over the corresponding specialized expert for a given held-in task would be preferable, then I would be happy to change my score, but as it stands I'm concerned that a large proportion of GLIDER's performance gains are on tasks that we need not optimize for.

2) 
A second concern is that GLIDER's architecture, by the authors' own acknowledgement, is basically identical to Phatgoose for the local component of the hierarchical router. This being the case, the contribution of the paper is more so appending a global context component to Phatgoose, rather than an entirely new model. It would therefore be informative to consider alternative backbones for the local component of the router, for example Arrow. This could help isolate the contribution of the global routing component and help to demonstrate the robustness of potential improvements brought about by the proposed inclusion of global information.

3) 
Some grammar / spelling related issues:

Lines 53-54: 'However, MoE methods that train experts from scratch while MoErging utilizes  a decentralized...' -> delete 'that'

Line 72: 'retrieve the correct expert for all token at every module' -> tokens should be plural

Line 264: 'our goal is to build a MoErging method that dynamically routing queries' -> should be 'routes queries'

Line 286: 'this work specifically focuses in LoRA' -> focuses 'on' LoRA

Lines 288-289 'Given the $t^{th}$ input token activation $u_i$ -> should be $u_t$ I'm assuming?

Line 332: 'added before the model to independently process the fully query' -> process the 'full' query

Line 348: 'the output of the module for token activation $u_i$ is computed as $Wu_i + \sum_{k \in \xi_{top}} w_k * B_kA_ku_i$ -> It looks like you've forgotten to actually define $w_k$, I'm assuming it's the softmaxed affinity score, but you've left it undefined.