Venue: NAACL 2024

# Unlocking Emergent Modularity in Large Language Models

\({}^{12}\)Zihan Qiu\({}^{*,+}\) , \({}^{3}\)Zeyu Huang\({}^{*}\), \({}^{1}\)Jie Fu\({}^{\dagger}\)

\({}^{1}\)CSE, HKUST  \({}^{2}\)IIIS, Tsinghua University,

\({}^{3}\)ILCC, University of Edinburgh

qzh1628@gmail.com, zeyu.huanged.ac.uk, jiefu@ust.hk

Equal contributionWork done while interning at HKUST.Corresponding author

###### Abstract

Modular Neural Networks (MNNs) demonstrate various advantages over monolithic models. Existing MNNs are generally _explicit_: their modular architectures are pre-defined, with individual modules expected to implement distinct functions. Recent works reveal that there exists _implicit_ modularity in standard pre-trained transformers, namely _Emergent Modularity_. They indicate that such modular structures spontaneously exhibit during the early pre-training phase. Despite the benefits of modularity, most Language Models (LMs) are still treated as monolithic models in the pre-train and fine-tune paradigm, with their emergent modularity locked and underutilized. In this work, focusing on unlocking the emergent modularity in LMs, we showcase that standard LMs could be fine-tuned as their Mixture-of-Expert (MoEs) counterparts without introducing any extra parameters. Such MoEs are derived from emergent modularity and are referred to as Emergent MoEs (EMoE). Our experiments demonstrate that fine-tuning EMoE effectively improves downstream in-domain and out-of-domain generalization compared with vanilla fine-tuning. Our analysis and ablation studies further illustrate that it is robust to various configurations and can scale up to Large Language Models (i.e., Llama2-7B and Llama-30B). Code is available at this repo.

## 1 Introduction

Modularity attracts considerable attention from the artificial intelligence community [5]. Neural networks with modular designs, termed Modular Neural Networks (MNNs), have exhibited a wide range of advantages, including adaptation [4], data efficiency [9], and better generalization abilities [10, 11]. Typical MNNs are usually _explicitly_ modular. They have a pre-defined modular structure and are expected to achieve a divide-and-conquer solution for the given task. Among various MNNs, Mixture-of-Experts (MoEs) employ a conditional computation strategy where different submodules - so-called experts - are expected to be activated by different types of inputs. MoEs see substantial success in various domains [11, 4, 4, 12, 13] in the era of large-scale transformers, and they are therefore a widespread modular neural architecture.

Apart from _explicit_ MNNs, some research finds that modular structures spontaneously emerge during training, not only in small-scale CNNs or LSTMs [12, 13], but also large-scale pre-trained transformer models. Zhang et al. [23]; Li et al. [24] reveal notable sparse activation patterns within the FeedForward Networks (FFNs) in pre-trained transformer models. They find that in T5-Base [14] and ViT-B16, only 3.0% and 6.3% neurons are activated during one forward process, respectively. Furthermore, Zhang et al. [22] utilize handpicked semantic and knowledge-intensive tasks to probe the nature of neurons in FFNs. They observe a strong correlation between neuron activation and specific tasks, further discovering clear function-based neuron grouping of the pre-trained T5 model (neurons with similar functions are usually co-activated). They summarize such phenomenon as _Emergent Modularity_ (EM).

Though modularity emerges, pre-trained language models are generally treated as monolithic models in the standard pre-train and fine-tune paradigm. It is natural to ask _whether their EM and the potential improvements brought by EM are locked in this process_.

In this paper, we advocate unlocking the EM in the pre-trained language models could bring gen

eralization improvements for downstream tasks. Specifically, we split certain FFNs layers of the original model into MoEs layers. The MoEs is derived according to the EM in that layer and can be regarded as the externalization of the EM. Hence, the obtained MoEs model is called Emergent MoEs (EMoE). We then fine-tune the obtained EMoE model to investigate whether unlocking EM encourages downstream task performance.

We validate our empirical findings with various models, evaluation benchmarks, fine-tuning methods (parameter-efficient tuning and full fine-tuning). We find that fine-tuning EMoE achieves stronger generalization performance than vanilla fine-tuning across various experimental settings, demonstrating that unlocking the EM of LMs boosts the models' downstream generalization abilities. We provide a comprehensive analysis for EMoE: 1) We first validate that EMoE indeed unclocks EM in pre-trained language models by showcasing its task-specific expert choice. 2) We then reveal that EMoE ameliorates the parameter updating during fine-tuning and can even be abandoned afterward. We want to highlight that this property further improves the practicality of EMoE as the model architecture does not change before and after fine-tuning. Meanwhile, our ablation studies show the EMoE's robustness to various hyper-parameter configurations. 3) We also conclude that EMoE could mask neurons with negative transfer effects. We hope our research discoveries can bring novel insights and serve as an example attempt towards further unlocking the EM of LLMs.

## 2 Methodology

### Preliminaries

**Transformer FFNs are Key-Value Memories.** The FFNs layer in the transformer block typically includes weights \({\bf K}\in\mathbb{R}^{h\times d}\), \({\bf V}\in\mathbb{R}^{d\times h}\), where \(h\) is embedding size and \(d\) is the dimension of the hidden layer (usually \(d=4h\)), and a non-linear activation function \(\sigma(\cdot)\). For an input \({\bf x}\in\mathbb{R}^{h}\), the output \({\bf y}\in\mathbb{R}^{h}\) can be calculated as Eq. 1:

\[{\bf y}={\rm FFN}({\bf x};{\bf K},{\bf V})=\sigma({\bf x}\cdot{\bf K})\cdot{ \bf V}. \tag{1}\]

More precisely, for each column \({\bf K}_{:,i}\) and row \({\bf V}_{i,:}\), Eq.1 can be rewritten as:

\[{\bf y}=\sigma({\bf x}\cdot{\bf K})\cdot{\bf V}=\sum_{i=1}^{h}\sigma({\bf x} \cdot{\bf K}_{:,i})\cdot{\bf V}_{i,:} \tag{2}\]

Following Geva et al. (2021, 2022); Huang et al. (2023), we regard columns in \({\bf K}\) as key vectors and rows in \({\bf V}\) as value vectors, the output of an FFNs network can be viewed as a weighted sum of value vectors based on the activation scores \(\sigma({\bf x}\cdot{\bf K})\). For the rest of this paper, we refer to one key-value memory pair using _neuron_ and the co-activation of neurons using _modularity_.

**Mixture-of-Experts** In transformers, MoEs is often implemented by replacing the original FFNs with a group of parallel FFNs and introducing a gating module. Supposing there are \(N\) experts: \(\left\{{\rm FFN}^{n}(\cdot;{\bf K}^{n},{\bf V}^{n})|\,n\in[1,N]\right\}\), the gating module \(g(\cdot\,;{\bf G},k)\), defined with its parameters \({\bf G}\) and an integer \(k\), is to map input \({\bf x}\) to a score distribution of experts \(g({\bf x};\,{\bf G},k)\in\mathcal{R}^{N}\). Typically, \(g\) is implemented with a simple linear layer followed by a softmax function and a Top-k function. Given \({\bf x}\in\mathbb{R}^{h}\), the output \({\bf y}\in\mathbb{R}^{h}\) of can be summarized as the weighted sum of the output from all experts:

\[{\bf y}=\sum_{n\in N}g_{n}({\bf x};{\bf G},k)\,{\rm FFN}^{n}({\bf x};{\bf K}^{ n},{\bf V}^{n}) \tag{3}\]

When \(k\) for Top-K is smaller than \(N\), only a subgroup of experts is involved in the computation, termed sparse MoEs.

### Emergent Mixture-of-Expert

According to Eq. 2 and Eq. 3, we find that FFNs internally resemble MoEs if we consider keys as the gating module and values as the group of experts, which inspire us to transform existing FFNs into sparse MoEs to unlock its modular potentials. Since our research goals mainly focus on EM, a preferred approach to externalizing EM into sparse MoEs should not introduce additional parameters, training, and data, which may result in impractical or undesired biases. Therefore, after splitting original _neurons_ into different groups to construct different experts, each group's average of key vectors is calculated to form the gating module. Details are described and illustrated in Fig. 1.

**Clustering-based Experts Construction.** We aim to ensure that _neurons_ that tend to be co-activated are divided into the same group. Since _neurons_ with similar key vectors tend to be co-activated according to Eq. 2, we split them into separate experts by clustering their key vectors. Specifically, given a FFNs layer \({\rm FFN}(\cdot;{\bf K},{\bf V})\), we perform constrained clustering Malinen and Franti (2014) (more details are in Appendix A.0.1) to partition all key vectors \({\bf K}\) into \(N\) experts on 

average. Denoting the indices of keys in the \(i\)-th group as \(E_{i}\subset[d]\), for \(\forall j\in E_{i}\), we extract _neuron_\((\mathbf{K}_{:,j},\mathbf{V}_{j,:})\) to form the \(i\)-th expert \(\mathrm{FFN}(\cdot;\mathbf{K}^{i},\mathbf{V}^{i})\) as depicted in Fig. 1(b). After that, the computation of each expert follows Eq. 1.

**Avg-k Gating.** As discussed, we do not want to introduce extra trainable parameters when externalizing EM. Therefore, we construct the gating module by averaging each expert's keys, which should route the input \(\mathbf{x}\) to the experts who tend to have larger activation scores and thus contribute more to the model's output. The gating function is usually implemented by a single layer \(\mathbf{G}\in\mathcal{R}^{h\times N}\), in the avg-k gating's case, the weights in \(i\)-th column \(\mathbf{G}_{:,i}\) can be calculated as follows:

\[\mathbf{G}_{:,i}=\mathrm{Avg}(\mathbf{K}^{i},\text{dim=0}). \tag{4}\]

And then the gating score for \(i\)-th expert is:

\[g_{i}(\mathbf{x};\mathbf{G},k)=\left\{\begin{array}{rl}&1~{}~{}\text{if}~{} i\in\mathrm{Top-K}(\mathbf{x}\cdot\mathbf{G};k)\\ &0~{}~{}\text{else}\end{array}\right. \tag{5}\]

where \(\mathrm{Top-K}(\cdot;k)\) returns indices of \(k\) largest element of the given input along. As the gating score is the average of activation scores (before \(\sigma(\cdot)\)) of neurons in that expert:

\[\mathbf{x}\!\cdot\!\mathbf{G}_{:,i}=\mathbf{x}\!\cdot\!\mathrm{Avg}(\mathbf{ K}^{i})=\frac{N}{d}\sum_{j\in E_{i}}\mathbf{x}\mathbf{K}^{i}_{:,j}=\frac{N}{ d}\sum_{j}a_{j}. \tag{6}\]

a larger value of gating score \(g_{i}\) implies more activated keys within the corresponding expert. Consequently, the expert could potentially contribute more to the output \(\mathbf{y}\) for input \(\mathbf{x}\). During downstream tuning, gating weights are tied with the FFNs parameters using Eq. 4.

## 3 Experiments

**Configurations**: We first evaluateEMoE using BERT and GPT2 series models. We employ GLUE Wang et al. (2019) and GLUE-X Yang et al. (2023) for benchmarking in-domain (ID) and out-of-domain (OOD) performance of the fine-tuned model, respectively. We mainly present the experimental results when employing LoRA Hu et al. (2022) to fine-tune the pre-trained language models for two reasons: (1) with the increasing scale of pre-trained models, parameter-efficient tuning Houlsby et al. (2019) can scale up to very large language models and thus becomes more practical. (2) standard LoRA weights are added in each self-attention block, and the parameters in FFNs will not be updated, allowing us to investigate whether leveraging EM, even without fine-tuning the parameters of EMGe, can yield improvements. We present EMoE full fine-tuning results in the Appendix B.1. Besides, we scale up EMoe on Llama2-7B and Llama-30B Touvron et al. (2023) for further validation. We instructuning models on Alpaca and testing them on the MMLU benchmark Hendrycks et al. (2021). For more details about datasets, evaluation metrics and computation cost, please refer to the Appendix D and E.

**Baselines** Our baselines include (1) _vanilla LoRA-tuning_: addLoRA weights to the q and v projections in attention layers; (2) _GMoE_ Li

Figure 1: (a) Existing literatures Geva et al. (2021, 2022) suggest that the FFNs in transformers can be viewed as key-value memories. They regarded the input as a query, the first layer as keys, and the second as values. Given an input, keys are sparsely activated (marked in red). Most of the values don’t impact the output. (2) The FFNs block can be partitioned into experts by clustering keys. (3) Afterward, experts’ key averages are used as the gating weights. The inner product between \(\mathbf{x}\) and gating weights are used to select experts.

 

et al., 2023): Instead of splitting, GMoE replicates FFNs layer and train new gating layers to introduce MoE structure from the two-to-last and fourth-to-last transformer blocks in the original pre-trained models. Since GMoE copies the FFNs of the pretrained model to form the MoEs, it is ineffective if introduced MoEs aren't tuned. Thus, we conduct experiments with LoRA tuning for GMoE and tune the transformer block where the original FFNs are replaced. (3) _EMoE-learn_: an ablation method, where the gating function is learned (same as GMoE) during fine-tuning. This helps us better understand the effect of avg-k gating.

**Hyper-parameters**: those unrelated to MoEs (e.g., learning rate, batch size) remain consistent with the baselines. Following Li et al. (2023), we replace original FFNs with EMoE layer in {last two even layers, last one even layer}. Comparable hyper-parameter searches are conducted for both GMoE and EMoe for the number of experts \(N\) and top-k: GMoE explores \(N\) within {4, 8} and top-k within {1, 2}; for EMoE, \(N\) is fixed at 64, with top-k explored within {16, 32}. The underlying reason for different \(N\) and top-k in GMoE and EMoE is that EMGE decomposes the original FFNs into experts, and the total parameters of all experts remain the same as the original FFNs. On the contrary, GMoE replicates the original FFN from a MoEs architecture; the larger \(N\) and top-k they choose, the more parameters and computational overhead they introduce. Our ablation study indicates that while more careful hyper-parameter searches may yield superior performance, adhering to a top-k/\(N=0.25\) or \(0.5\) for EMoE consistently brings improvement over vanilla fine-tuning.

**Evaluation Metrics** All experiments except the scale-up ones are repeated three times, and the average is presented in the main section. Full results are in Appendix F.1. For the _OOD metric_, we follow GLUE-X Yang et al. (2023) and employ the Friedman rank Friedman (1940)1 rank\({}_{f}=\frac{1}{n}\sum_{i=1}^{n}\text{rank}_{i}\). For each method under the same backbone, rank\({}_{i}\) is produced based on each dataset's best and average results. For the 13 OOD tasks used in GLUE-X, each method generates 26 rank\({}_{i}\) values. The OOD results presented in Table 1 represent the mean of all these rank\({}_{i}\) values. The original results of each task can be found in the Appendix F.1.

Footnote 1: Our evaluation includes nine methods detailed, demonstrating the relationships among these methods. Therefore, these values can’t provide a direct comparison with the results in the GLUE-X paper

**Results with BERT and GPT2** According to Tab. 1: (1) EMAoE demonstrates enhancements compared to vanilla LoRA-tuning. Notably, EMoE also achieves results comparable to GMoE on BERT-large and outperforms GPT2-XL with much fewer learnable parameters. (2) While EMoE-learn's results are better than EMoE in several tasks (STSB, QNLI), EMoE exhibits higher stability than EMoE-learn and delivers superior overall results. (3) MoEs structures improve OOD performance (GMoE, EMoE, EMoE-learn vs. LoRA).

**Results with Llama** Unlike GMoE, EMoE does not introduce any additional trainable parameters or procedures so that we can scale EMoE up to models of 7B and 3OB sizes to validate its effectiveness further. From Tab. 2: 1. EMoE consistently yields improvements compared with LoRA tuning with negligible additional computation. 2. Because EMoE does not introduce any additional trainable parameters, the choice of K and N proportions remains large (can be even larger) and is applicable in larger-scale models. While the performance varies when employing different \(N\) and \(K\), EMoE consistently outperforms the vanilla LoRA tuning. This

\begin{table}
\begin{tabular}{l|c c c c c c c c|c} \hline \hline
**Algorithm** & **MRPC** & **CoLA** & **RTE** & **STSB** & **SST2** & **QNLP** & **MNLI** & **ID-Avg\(\uparrow\)** & **ood\(\downarrow\)** \\ \hline \multicolumn{10}{c}{BERT-Large (340M Parameters)} \\ \hline LoRA & 89.97 & 63.40 & 72.92 & **90.51** & 93.16 & 92.20 & 87.21 & 85.40 & 84.35 & 4.86 \\ GMoE & 89.45 & 63.80 & 72.56 & 90.29 & **93.85** & 92.32 & **87.99** & **85.92** & 84.52(+0.18) & **4.04** \\ EmoE-learn & 89.87 & 64.00 & 71.36 & 90.48 & 93.65 & **92.40** & 87.55 & 85.62 & 84.37(+0.02) & 4.66 \\ EMGE & **90.85** & **65.33** & **75.21** & 90.43 & 93.50 & 92.23 & 87.74 & 85.43 & **85.09(+0.74)** & 4.37 \\ \hline \hline \end{tabular} 
\begin{tabular}{l|c c c c c c c c|c} \hline \hline
[FOOTNOTE:]Footnote †: footnotemark: [ENDFOOTNOTE]

\end{table}
Table 1: ID and OOD performance of EMoE and baseline models. All the reported results are obtained from 3 independent experiments. OOD Metrics (averaged over 14 OOD tasks, lower is better) provide additional information for OOD generalization. The best result is highlighted in **bold**.

 

suggests that although additional hyperparameters \(N\) and \(K\) are introduced, they do not lead to usability challenges. Please refer to Sec. 5 for more ablation results on hyperparameters.

**Additional Results** Moreover, we conduct experiments with full fine-tuning under comprehensive evaluation settings: 1. Vision OOD benchmark Domained Gulrajani and Lopez-Paz (2021), full fine-tuning ViT-Small (22M) and ViT-Base (86M). 2. GLUE benchmark, fully fine-tuning BERT-Base, BERT-Large, and GPT2-Small. 3. Full fine-tuning Llama2-7B. We present detailed results and analyses in Appendix B.1. According to the results, EMoE brings consistent improvements over standard full finetuning across various tasks and model scales. On the vision OOD benchmark Domainbed, which strictly controls evaluation metrics, EMoE achieved results comparable to the state-of-the-art GMoE. Furthermore, when full-finetuned with Alpaca, EMoE exhibited a notable improvement of **1.58** on MMLU to the standard tuning baseline. These findings underscore the effectiveness of EMoe in enhancing model performance across various architectures and tasks.

## 4 Analysis

### Does ENoE Unlock Emergent Modularity?

We first investigate whether simple key-vector-based clustering partitioning could capture the modular pattern of the neuron activation. The activation scores of different neurons on different inputs are visually shown in the Fig. 2 left and middle. Before clustering (Fig. 2 left), the activation of neurons seems random. After rearranging those activation scores according to the EMoE partition (Fig. 2 middle), we observed that only a part of the neurons are frequently employed in this task, and clear activation clusters emerge. This demonstrates that **key-vector-based clustering can decompose modular components within the standard model**. We then delve into expert usages across different tasks in the EMoE. The heatmap between tasks and experts (Fig. 2 right) showcases that the expert utilization differs between tasks, while some crucial experts are nearly used by all 8 tasks. For example, expert 19, that heavily used in QQP and MNLI, is merely activated in MRPC and RTE. On the contrary, expert 33 is frequently activated by all 8 tasks.

### How does EMoE improve fine-tuning performance?

Though EMoE achieves notable improvements in both ID and OOD scenarios, it is non-trivial that simply transforming the pre-trained FFNs into MoEs before fine-tuning can yield such benefits, especially when using LoRA tuning, where the gates and experts part are not updated. We investigate the mechanism behind the enhanced performance. We use BERT-Large as the backbone model.

**EMoE benefits LoRA weight learning instead of influencing inference.** EMoE and the standard

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline Experts (N) & topic (k) & MMLU score & times (s) & FLOPS (\(10^{17}\)) \\ \hline \multicolumn{5}{c}{Llama2-7B} \\ \hline
1 & 1 & 46.96 & 1396 & 6.92 \\
64 & 16 & **47.58** & 1545 & 7.03 \\
64 & 32 & 47.37 & 1521 & 7.13 \\ \hline \multicolumn{5}{c}{Llama-3OB} \\ \hline
1 & 1 & 56.18 & 6943 & 22.5 \\
256 & 64 & **57.11** & 6955 & 22.5 \\
256 & 128 & 56.64 & 6974 & 22.4 \\ \hline \end{tabular}
\end{table}
Table 2: Instruction LoRA-tuned Llama models evaluated (in accuracy) on the MMLU benchmark. Times are wall-clock computation times for training 3 epochs on Alpaca using the same GPU devices. When \(N=K=1\), it represents the vanilla LoRA tuning.

Figure 2: **Left**: Activations of neurons (z-axis denotes activation value) in FFNs of a pre-trained transformer models. **Middle**: By clustering the keys in the FFNs layer and rearranging the activation scores accordingly, modular patterns of neuron activation emerge. **Right**: The heat map between experts and tasks. It is observed that the activation of experts is task-dependent, while some experts are generally shared across different tasks.

 

model only differ in FFNs activations. Such differences might (1) directly impact outputs during inference and (2) influence the parameter updating during training. In light of this, we propose two variants and compare them with vanilla LoRA tuning and EMoE: (a) LoRA2EMoE: Using LoRA to fine-tune the original model and split it into EMoEs during inference. If it surpasses the vanilla LoRA tuning, we can infer that EMGE mainly impacts model inference. (b) EMoE2LoRA: Using LoRA to fine-tune an EMoE model and merge experts into original FFNs during inference. If no changes occur, it implies that EMoE mainly ameliorates the parameter updating of the fine-tuning stage.

According to Fig. 3, doing sparse activation during testing does not contribute to better generalization on average (LoRA2EMoE). However, when merging ENoE back into the original FFNs after fine-tuning, the performance remains significantly better than the vanilla LoRA tuning and is almost identical to EMoE (EMoe2LoA and EMoE). Please refer to Appendix F.1 Tab. 16 for full results. Thanks to this property, we can use EMoE during fine-tuning and then convert the models to standard ones. _This allows the model to enjoy the benefits of EM without any alterations to its deployment, which improves EMoE's practicability in the LLMs era._ We further validate this on the Llama series models and the findings are consistent. Thus, in Tab. 2, we report the results using EMoE to fine-tune and standard Llama architecture for benchmarking. And we highlight that this property can also be applied in the full fine-tuning setting, as illustrated in Appendix B.1 Tab. 7.

**EMoE masks neurons with negative transfer impacts.** The only difference between EMoE and vanilla LoRA tuning is EMoE blocks some activated neurons during training by Top-k expert selection. Based on these, we hypothesize that the effects of EMoE stem from preventing negative knowledge transfer from blocked neurons. Therefore, we investigate whether there are such negative transfers. Specifically, we study the following expert selection variants: (1) Bottom-k: select \(k\) experts with the _lowest_ scores; (2) Not-top-k: select \(N-k\) experts who are not among the top-k experts. These variants are evaluated across six tasks from GLUE. The averaged outcomes are in Fig. 4. Full results can be found in Appendix F.1 Tab. 17. LoRA tuning results with Bottom-k and Not-top-k expert selections are worse than vanilla LoRA tuning, while Top-K outperforms it. We also examine the neuron activation ratio (the number of activated neurons in selected experts versus the number in the FFNs) across these variants. Full results can be found in Appendix F.2 Tab. 21. The activation ratios for these three variants are approximately 0.43 for Top-k, 0.57 for Not-top-k, and 0.12 for Bottom-k, respectively. Notably, Not-top-k significantly lags behind Top-k, even though it involves and activates more neurons, suggesting the performance drop is more related to the neurons' property. This further corroborates that masked neurons have negative transfer effects.

Inspired by this, we choose a multi-task learning setting where negative transfer might be more pronounced. We adopt the codebase2 from ATTEMPT (Asai et al., 2022). For the ID scenario, we follow ATTEMPT and select six small tasks. For the OOD scenario, we train the models on two larger natural language inferen

Figure 4: Sparse activated training accuracies with different expert selections.

Figure 3: ID and OOD accuracies compared with LoRA for validating EMoE’s training & inference effects.

\begin{table}
\begin{tabular}{c c|c|c} \hline \hline Experts (\(N\)) & topk (\(k\)) & ID avg & OOD avg \\ \hline Baseline & - & 65.73 & 51.65 \\
8 & 2 & 67.98 & 52.19 \\
16 & 4 & 68.14 & **53.23** \\
32 & 8 & **73.29** & 53.20 \\ \hline \hline \end{tabular}
\end{table}
Table 3: T5-base Multi-tasks ID and OOD accuracies across different EAOE settings.

 

and test them on four NLI datasets from different domains. All hyperparameters unrelated to MoEs are consistent with the baseline. We list the MoEs-related hyperparameters and average results in Tab. 3. More details about experiment setting and results are in Appendix B.2. We can observe that EMoE exhibits a more substantial improvement compared to the baseline. In the ID setting, the highest improvement reaches **7.56**, even considering the average performance across the six tasks. In the OOD setting, the highest average OOD result across the four datasets also improves by 1.58.

## 5 Ablation studies

**Sparsity and Modularity** To provide further evidence that the EMoE's improvements stem from leveraging modular features rather than just sparse activation or MoEs architecture, we compare the results of (1) _key-vector-based clustering_ expert construction and (2) _random_ construction. We employ the same setting of Sec. 4.2. The relative changes in averaged outcomes compared to the vanilla fine-tuning are shown in Tab. 4, while full results can be found in the appendix F.1, Tab. 17. It's noteworthy that while cluster top-k exhibits a significant improvement over the standard, random top-k is conversely worse than vanilla fine-tuning. This suggests that random construction can negatively impact gating, and only MoEs structure itself can't bring improvement. Moreover, when selecting weights with negative transfer under bottom-k selections, it's observed that cluster bottom-k also achieves lower results. In summary, **clustering-based methods can externalize implicit modularity within the pre-trained language model**. Within suitable frameworks like MoEs, such modularity can facilitate downstream tuning.

**MoEs Constructing Methods** In EMoEs, the expert construction methods and gating methods are pivotal. To further understand splitting versus replicating FFNs, avg-k gating versus learned gating, we visualize expert selections of modularized GPT2-XL during tuning on 6 tasks with 16 experts. In Fig. 5, we showcase the results for the largest dataset QNLI among them. Full results are available in the Appendix G. Our observations are: (1) Both avg-k gating and learned gating converge, as indicated by the lower halves of the plots. (2) avg-k gating is more stable than learned gating. As shown in Fig. 5 (a) and (b), the expert selection merely changes during fine-tuning. This might mitigate data inefficiency from gating inconsistencies across different training stages Zuo et al. (2022). (3)EMoE, with its differently initialized experts parameters, exhibits better load balancing than GMoE (expert 11 in Fig. 5 c is way more frequently selected than other experts). In GMoE, all experts share identical initialization; in EMoE, the experts are derived from FFNs with EM. This also suggests a good initialization can facilitate MoEs learning, as indicated by Nie et al. (2021).

Fig. 5 (a) and Appendix G demonstrate that some experts in EMoE are barely selected during fine-tuning. This bears similarity to pruning, leading to the argument that EMoE's improvements may stem from pruning. In this context, continuing our exploration from 4.2 on examining EMoE during training and inference, we supplement two ablations under the same setting: 1. _Training-Pruning_: Training EMoE (N=64, top-k=32), pruning experts with lower selection frequency, and evaluating the pruned model. 2. _Pruning-Training_: pruning lower selection frequency experts of a new model as per step 1, then training and evaluating the pruned model. The experimental outcomes are detailed

\begin{table}
\begin{tabular}{c|c c} \hline \hline  & Top-k & Bottom-k \\ \hline Cluster & +0.92 & -0.48 \\ Random & -0.11 & -0.34 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results for different expert constructions (clustering-based v.s. random) and selections

Figure 5: Expert selections during training with distinct gating functions (avg-k vs. learned) and expert types (splits of FFNs vs. copies of FFNs). The vertical axis illustrates training steps (top-down arrangement signifies begin-end); the horizontal axis represents expert selection frequency within 1K steps (deeper color implies a higher frequency). (a), (b) and (c) correspond to EMoE, EMoE-learn, and GMoE.

 

in Tab. 22. Our primary findings are: 1. When sparsely selected experts are pruned post-EMoE tuning, the performance consistently outperforms the LoRA baseline (even when only 1/16 experts are used). This underscores that EMoE's impact is manifested during the fine-tuning stage, not inference. 2. In the optimal pruning-training setting, the performance surpasses the baseline but falls significantly short of EMoE. This suggests that EMoE's improvements are not solely attributable to pruning. Compared with EMoE selecting experts for each token, pruning masks the same neurons for all tokens. Therefore, pruning can also be considered task-level EMoE, thus less effective. Lastly, it is important to emphasize that in multi-task learning, pruning methods struggle to ascertain the neurons requiring pruning, whereas EMoE demonstrates significant effectiveness in these scenarios.

**MoEs Configurations** Beyond the settings detailed in the main results section, which are based on \(N=64\) experts and top-k \(\in\{16,32,48\}\), we also present specific scenarios where \(N\in\{16,32\}\), and top-k varies within \(\{2,4,8,16\}\) in Fig. 6. Notably, in each of these settings, (1) EMoE consistently surpasses the standard model, illustrating its robustness to hyper-parameters. (2) On average, avg-k gating exhibits superior performance than learned gating. Though learned gating (EMoE-learn) outperforms avg-k gating in a few specific settings (Fig. 6 (b) and (e)). This is consistent with the earlier results in Section 3. Regarding how many EMoE layers should be introduced, our findings align with those discovered in GMoE, indicating that only a limited number of layers can be converted into the EMoE layer. If excessive EMoE layers are introduced, performance deteriorates. Taking GPT2-XL (48 layers) as an example, when introducing EMoE every two layers in the latter half, the performance averaged across 5 GLUE tasks (79.36) matches that of the standard model (78.87). However, when adopting EMoE for every 2-layer for the entire model, the performance lags slightly behind that of the standard model (78.17) but surpasses the EMoE-learn (75.87). For additional configurations, please refer to the appendix F.2 Tab. 19.

## 6 Related Work

Works most related to our work is introducing modularity based on off-the-shelf pre-trained models. For example, GMoe Li et al. (2023) and Upcycling Komatsuzaki et al. (2023) copy theFFNs from a trained transformer model to form the MoEs architecture. Their modular structure is introduced by replicating existingFFNs modules, leaving EM within pre-trainedFFNs unexplored. MoEfication Zhang et al. (2022) and MoEBert Zuo et al. (2022) explores the EM within the model. They seek to improve the inference efficiency by decomposing the originalFFNs layer into sparse MoEs. They utilize the sparse expert activation to reduce inference overhead and do not touch on how EM influences the model's performance in downstream fine-tuning. Our method to splitFFNs is adopted from one simple method in the ablation studies of MoEfication paper Zhang et al. (2022): clustering-based expert construction and avg-k gating. We empirically find that such a simple method can validate the improvements brought by EM and answer our research questions. We thus leave more elaborated methods for future works. Please refer to Appendix C for a detailed comparison of EMoE with those related works.

## 7 Conclusion

In this work, we validate that unlocking the EM in standardLMs improves downstream task ID and OOD performances. EMoE can bring this benefit without adding any parameters, significant training costs, or any alterations to its deployment, which improves its practicability in the LLMs era. One possible reason is the modular structure can allevi

Figure 6: Results of 3 tasks for different experts splittings and expert select fractions (top-k / number of experts). ‘e=16’ and ’e=32’ mean splittingFFNs into 16 and 32 experts, respectively.

 

ate negative transfer effects presented in the LMs. We hope our findings can deepen the understanding of neural networks' modularity, further helping the community develop more sophisticated modular neural architectures and utilize existing LMs.

## 8 Limitations

Our primary objective was to investigate the utility of EM, and thus, we predominantly adopted the techniques from MoEfication for decomposition. We encourage further research to propose improved algorithms for harnessing EM. Our research findings have not been validated on more challenging tasks (e.g., Mathematical Reasoning Imani et al. (2023)). While our analysis was primarily conducted on models with a maximum parameter count of 1.5B, we validate the scaling-up ability of EMoE to Llama-30B.

## References

* A. Agarwala, A. Das, B. Juba, R. Panigrahy, V. Sharan, X. Wang, and Q. Zhang (2021)One network fits all? modular versus monolithic task formulations in neural networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, External Links: Link Cited by: SS4.
* M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz (2019)Invariant risk minimization. CoRRabs/1907.02893. External Links: Link, 1907.02893 Cited by: SS1.
* A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi (2022)ATTEMPT: parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pp. 6655-6672. External Links: Link, Document Cited by: SS1.
* G. Auda and M. S. Kamel (1999)Modular neural networks A survey. Int. J. Neural Syst..9 (2), pp. 129-151. External Links: ISSN 0037-947X, Document, Link Cited by: SS1.
* H. Bao, W. Wang, L. Dong, Q. Liu, O. Khan Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei (2022)VLmo: unified vision-language pre-training with mixture-of-modality-experts. In NeurIPS, Cited by: SS1.
* ECCV 2018
- 15th European Conference, Vol. 11220 of Lecture Notes in Computer Science, pp. 472-489. External Links: ISBN 978-3-319-
 

2013, Sydney, Australia, December 1-8, 2013_, pages 1657-1664. IEEE Computer Society.
* Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. _J. Mach. Learn. Res._, 23:120:1-120:39.
* Friedman (1940) Milton Friedman. 1940. A comparison of alternative tests of significance for the problem of m rankings. _The annals of mathematical statistics_, 11(1):86-92.
* Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_, pages 30-45. Association for Computational Linguistics.
* Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_, pages 5484-5495. Association for Computational Linguistics.
* Goyal and Bengio (2020) Anirudh Goyal and Yoshua Bengio. 2020. Inductive biases for deep learning of higher-level cognition. _CoRR_, abs/2011.15091.
* Goyal et al. (2021) Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Scholkopf. 2021. Recurrent independent mechanisms. In _9th International Conference on Learning Representations, ICLR 2021_. OpenReview.net.
* Gulrajani and Lopez-Paz (2021) Ishaan Gulrajani and David Lopez-Paz. 2021. In search of lost domain generalization. In _9th International Conference on Learning Representations, ICLR 2021_. OpenReview.net.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _9th International Conference on Learning Representations, ICLR 2021_. OpenReview.net.
* Hod et al. (2021) Shlomi Hod, Stephen Casper, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. 2021. Detecting modularity in deep neural networks. _CoRR_, abs/2110.08058.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022_. OpenReview.net.
* Huang et al. (2023) Zeyu Huang, Yakang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformerpacther: One mistake worth one neuron. In _The Eleventh International Conference on Learning Representations, ICLR 2023_. OpenReview.net.
* Imani et al. (2023) Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathpromptter: Mathematical reasoning using large language models. In _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics: Industry Track, ACL 2023_, pages 37-42. Association for Computational Linguistics.
* Kaushik et al. (2020) Divgansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020. Learning the difference that makes A difference with counterfactually-augmented data. In _8th International Conference on Learning Representations, ICLR 2020_. OpenReview.net.
* Komatsuzaki et al. (2023) Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In _The Eleventh International Conference on Learning Representations ICLR 2023_. OpenReview.net.
* Li et al. (2023) Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. 2023. Sparse mixture-of-experts are domain generalizable learners. In _The Eleventh International Conference on Learning Representations, ICLR 2023_. OpenReview.net.
* Li et al. (2017) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. 2017. Deeper, broader and artier domain generalization. _CoRR_, abs/1710.03077.
* Li et al. (2022) Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Kashank J. Reddi, Ke Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, and Sanjiv Kumar. 2022. Large models are parsimonious learners: Activation sparsity in trained transformers. _CoRR_, abs/2210.06313.
* Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24_, pages 142-150. The Association for Computer Linguistics.
* Joint IAPR International Workshop, S+SSPR 2014_, volume 8621 of _Lecture Notes in Computer Science_, pages 32-41. Springer.
* Mustafa et al. (2022) Basil Mustafa, Carlos Riouelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022. Multimodal contrastive learning with limoe: the language-image mixture of experts. In _NeurlPS_ 

Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. 2021. Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate. _arXiv preprint arXiv:2112.14397_.
* Pfeiffer et al. (2023) Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, and Edoardo Maria Ponti. 2023. Modular deep learning. _CoRR_, abs/2302.11529.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In _5th International Conference on Learning Representations, ICLR 2017_. OpenReview.net.
* Shen et al. (2023a) Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. 2023a. Scaling vision-language models with sparse mixture of experts. _CoRR_, abs/2303.07226.
* Shen et al. (2023b) Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfeng Chen, and Chuang Gan. 2023b. Moduleformer: Learning modular large language models from uncurated data. _CoRR_, abs/2306.04640.
* Szymanski and Lemmon (1993) Peter T. Szymanski and Michael D. Lemmon. 1993. Adaptive mixtures of local experts are source coding solutions. In _Proceedings of International Conference on Neural Networks (ICNN'88)_, pages 1391-1396. IEEE.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptise Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Vaghani and Thummar (2023) Nirali Vaghani and Mansi Thummar. 2023. Flipkart product reviews with sentiment dataset.
* Venkateswara et al. (2017) Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. 2017. Deep hashing network for unsupervised domain adaptation. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 5385-5394. IEEE Computer Society.
* Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanneret Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019_, pages 3261-3275.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019_. OpenReview.net.
* Weiss et al. (2022) Martin Weiss, Nasim Rahaman, Francesco Locatello, Chris Pal, Yoshua Bengio, Bernhard Scholkopf, Li Erran Li, and Nicolas Ballas. 2022. Neural attentive circuits. In _NeurIPS_.
* Wu et al. (2022) Chuhan Wu, Fangzaho Wu, Tao Qi, and Yongfeng Huang. 2022. Noisytune: A little noise can help you finetune pretrained language models better. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022_, pages 680-685. Association for Computational Linguistics.
* Yang et al. (2023) Linyi Yang, Shuubai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. GLUE-X: evaluating natural language understanding models from an out-of-distribution generalization perspective. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 12731-12750. Association for Computational Linguistics.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015_, pages 649-657.
* Zhang et al. (2022a) Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2022a. Mixture of attention heads: Selecting attention heads per token. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 4150-4162. Association for Computational Linguistics.
* Zhang et al. (2022b) Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022b. Moefication: Transformer feed-forward layers are mixtures of experts. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 877-890. Association for Computational Linguistics.
* Zhang et al. (2023) Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. 2023. Emergent modularity in pre-trained transformers. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4066-4083. Association for Computational Linguistics.
* Zhu et al. (2022a) Simiao Zhu, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hanny Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022a. Taming sparsely activated transformer with stochastic experts. In _The Tenth International Conference on Learning Representations, ICLR_. OpenReview.net 

Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. 2022b. Moebert: from bert to mixture-of-experts via importance-guided adaptation. _arXiv preprint arXiv:2204.07675_.

## Appendix A Additional Information

#### a.0.1 Constrained clustering

Constrained clustering Malinen and Franti (2014) enhances traditional clustering methods by incorporating additional user-specified constraints. This approach involves balancing the k-means algorithm for clustering, making it more efficient and effective in generating meaningful clusters. The introduction of constraints, such as must-link or cannot-link conditions, guides the clustering algorithm to produce results that align more closely with domain-specific requirements or prior knowledge. This method has improved the relevancy and accuracy of clustering outcomes in various applications.

## Appendix B Additional Experiment Results

### Full Fine-tuning Performance

**We test the EMoE's OOD performance on Domainbed.** Domained provides comprehensive vision OOD evaluations (one result is aggregated with 30 experiments), and the outcomes vary marginally. Moreover, Gulrajani and Lopez-Paz (2021) indicates that vanilla full fine-tuning with fair hyper-parameter search is a strong baseline compared with specifically designed methods like Invariant Risk Minimization Arjovsky et al. (2019). More dataset details are in appendix D.0.1. According to Table 5: (1) Overall, EMoE outperforms ViT and GMoE (except ViT-base Train-validation setting, upper right). (2) Compared with EMoE, EMoE-learn incorporates a learned gate. While it surpasses avg-k gating in certain scenarios (ViT-small Terra Train-validation), it can also lead to a performance drop compared with vanilla ViT. Its overall performance is lower than EMoE. (3) In tasks where the standard model is strong (like Office), EMoE performs better than other MoEs methods. One possible reason is that the avg-k gating reduces the influence of gating weights (\(g_{n}(\mathbf{x};\mathbf{G};k)\) in Equation 3), making it more like the standard model in such scenarios.

In vision tasks, as highlighted by Gulrajani and Lopez-Paz (2021), the hyper-parameter search has a profound impact on outcomes. Consequently, we search with a relatively large scope: for GMoE, \(N\) is searched within {4, 6, 8}, and top-k within {2, 3, 4}; for EMoE, \(N\) is sought within {6, 12, 24}, and top-k within {2, 4, 8}.

**We evaluate EMoE's ID performance on 5 GLUE tasks**. We also include an additional baseline noisy tuning Wu et al. (2022), which improves adaptation for free by adding uniform distribution noise proportional to the standard deviation of the pre-trained weights before tuning. According to Table 6: (1) On average, EMoE and EMGE-learn outperform other baselines. (2) Among the two methods that do not introduce additional parameters, EMoE significantly outperforms noisy tuning. (3) EMoE provides stable improvements over baselines across different settings, demonstrating its generality.

### Mult-tasks Setting

In our analysis 4, we have identified that the improvement brought by EMoE is likely associated with mitigating negative transfer. Inspired by this, we choose a multi-task learning setting where negative transfer might be more pronounced. We adopt the codebase3 from ATTEMPT Asai et al. (2022). For the in-domain (ID) scenario, we follow the settings outlined in ATTEMPT and select six tasks from the Super-GLUE benchmark Wang et al. (2019). For the out-of-domain (OOD) scenario, we take two larger natural language inference (NLI) datasets MNLI and QNLI, from GLUE as our ID training data. We subsequently conducted direct testing on four additional NLI datasets from different domains. All hyperparameters unrelated to MoEs are kept consistent with the baseline, and we have listed the MoEs-related hyperparameters in the result table. Our observations are as follows: 1. EMoE exhibits a substantial improvement compared to the baseline. In the in-domain (ID) setting, the highest improvement reached 7.56, even considering the average performance across the six tasks. In the OOD setting, the highest average OOD result across the four datasets improved by 1.58. 2. Across various settings of N and K, EMoE consistently outperforms the vanilla fone-tuning. Within the hyperparameter search space specified in our paper, EMoE consistently improves at least 2 points over the baseline in the ID setting. This emphasizes the effectiveness of EMoE and EMoE's robustness to the explored hyperparameter range.

 



\begin{table}
\begin{tabular}{l|c|r r r r r r} \hline \hline
**Backbone** & **Algorithm** & **MRPC** & **CoLA** & **RTE** & **STSB** & **SST2** & **Avg** \\ \hline  & baseline & 88.45 & 60.67 & 68.95 & 87.87 & 91.97 & 79.582 \\  & noisy tuning & 88.43 & 61.79 & 71.36 & 88.27 & 92.32 & 80.43(+0.85) \\ BERT & GMoE & 88.63 & 61.25 & 70.28 & 88.63 & 92.28 & 80.21(+0.63) \\ Base & EMoE-learn & 89.05 & **62.46** & **70.40** & 88.47 & 92.58 & **80.59(+1.01)** \\  & EMGoE & **89.45** & 61.55 & 69.68 & **88.71** & **92.89** & 80.46(+0.87) \\ \hline  & baseline & 89.82 & 65.41 & 74.89 & 89.87 & 93.50 & 82.70 \\ BERT & noisy tuning & 90.42 & 64.75 & 73.41 & 90.05 & 93.65 & 82.46(-0.24) \\ Large & GMoE & **91.24** & 64.90 & 74.24 & 90.00 & 93.58 & 82.79(+0.09) \\  & EMG-learn & 90.57 & 65.51 & 74.72 & 90.22 & **93.73** & 82.95(+0.25) \\  & EMGoE & 90.74 & **65.79** & **76.17** & **90.31** & 93.58 & **83.32(+0.62)** \\ \hline  & baseline & 84.46 & 47.07 & 67.15 & 86.29 & 92.13 & 75.42 \\ GPT2 & noisy tuning & 84.15 & 46.16 & 67.51 & 86.09 & 92.13 & 75.21(-0.21) \\ Small & GMoE & 85.07 & 47.77 & 67.51 & 86.57 & 92.35 & 75.85(+0.43) \\  & EMGoE-learn & **85.73** & 47.24 & 67.99 & **86.66** & 92.35 & 75.99(+0.57) \\  & EMGoE & 85.40 & **48.00** & **68.95** & 86.64 & **92.70** & **76.34(+0.92)** \\ \hline \hline \end{tabular} 
\begin{tabular}{l|l l l l l l} \multicolumn{6}{c}{Results with ViT-based (86M) backbone} \\ \hline
**Algorithm** & **PACS** & **VLCS** & **Office** & **Terra** & **Avg** \\ \hline \multicolumn{6}{c}{Train-validation selection criterion} \\ \hline ViT & 89.1 & **80.7** & 77.2 & 47.3 & 73.58 \\ GMoE & **90.0** & 80.4 & 77.0 & **49.2** & **74.15** \\ EMoE-learn & 89.8 & 80.6 & 76.5 & 48.7 & 73.90 \\ ENoE & 89.4 & 80.7 & **77.3** & 48.5 & 73.98 \\ \hline \multicolumn{6}{c}{Leave-one-domain-out selection criterion} \\ \hline ViT & 88.9 & 80.8 & **77.5** & 46.1 & 73.33 \\ GAOeE & 89.3 & 81.0 & 76.7 & 50.1 & 74.28 \\ EAoE-learn & 89.3 & 81.2 & 76.5 & **50.5** & 74.38 \\ EMoE & **89.6** & **81.6** & 77.4 & 50.0 & **74.65** \\ \hline \multicolumn{6}{c}{Test-domain selection criterion} \\ \hline ViT & 88.8 & 79.0 & 77.2 & 46.7 & 72.93 \\ GMoE & 89.7 & 79.0 & 77.0 & **48.8** & 73.63 \\ EMoE-learn & **89.7** & **79.7** & 76.6 & 48.7 & 73.68 \\ EMoE & **89.7** & **79.7** & **77.5** & 48.8 & **73.93** \\ \hline \hline \end{tabular}
\end{table}
Table 6: ID performance on GLUE tasks with different backbones and algorithms. All the reported results are obtained from 3 independent experiments. The average accuracy (Avg) is reported along with the relative improvement compared to the baseline. The best result is highlighted in **bold**.

\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline
**Algorithm** & **PACS** & **VLCS** & **Office** & **Terra** & **Avg** \\ \hline \multicolumn{6}{c}{Train-validation selection criterion} \\ \hline ViT & 86.9 & **79.7** & 73.0 & 44.0 & 70.90 \\ GMoE & 87.7 & 79.6 & 73.1 & 45.4 & 71.45 \\ EMoE-learn & 87.2 & 79.6 & 72.5 & **46.1** & 71.35 \\ EMGoE & **87.8** & 79.5 & **73.1** & 45.9 & **71.58** \\ \hline \multicolumn{6}{c}{Leave-one-domain-out selection criterion} \\ \hline ViT & 86.1 & 79.7 & **73.3** & 45.0 & 71.03 \\ GMoE & 86.5 & 80.5 & 73.1 & 45.3 & 71.35 \\ EMoE-learn & **86.8** & 79.6 & 72.6 & 45.8 & 71.20 \\ EMGoE & 86.8 & **80.6** & 73.3 & **46.1** & **71.70** \\ \hline \multicolumn{6}{c}{Test-domain selection criterion} \\ \hline ViT & 86.5 & 78.2 & 73.1 & 44.0 & 70.45 \\ GAOE & 87.2 & 79.0 & **73.4** & 45.3 & 71.23 \\ EMoE-learn & 87.4 & **79.1** & 72.8 & 45.4 & 71.18 \\ EMGoE & **87.6** & 79.0 & 73.3 & **45.5** & **71.35** \\ \hline \hline \end{tabular} 
\begin{tabular}{l|l l l l l l} \multicolumn{6}{c}{Results with ViT-based (86M) backbone} \\ \hline
**Algorithm** & **PACS** & **VLCS** & **Office** & **Terra** & **Avg** \\ \hline \multicolumn{6}{c}{Train-validation selection criterion} \\ \hline ViT & 86.9 & **79.7** & 73.0 & 44.0 & 70.90 \\ GMoE & 87.7 & 79.6 & 73.1 & 45.4 & 71.45 \\ EMoE-learn & 87.2 & 79.6 & 72.5 & **46.1** & 71.35 \\ EMGoE & **87.8** & 79.5 & **73.1** & 45.9 & **71.58** \\ \hline \multicolumn{6}{c}{Lleave-one-domain-out selection criterion} \\ \hline ViT & 86.1 & 79.7 & **73.3** & 45.0 & 71.03 \\ GMoE & 86.5 & 80.5 & 73.1 & 45.3 & 71.35 \\ EMoE-learn & **86.8** & 79.6 & 72.6 & 45.8 & 71.20 \\ EMGoE & 86.8 & **80.6** & 73.3 & **46.1** & **71.70** \\ \hline \multicolumn{6}{c}{Test-domain selection criterion} \\ \hline ViT & 86.5 & 78.2 & 73.1 & 44.0 & 70.45 \\ GAOeE & 87.2 & 79.0 & **73.4** & 45.3 & 71.23 \\ EMoE-learn & 87.4 & **79.1** & 72.8 & 45.4 & 71.18 \\ EMGoE & **87.6** & 79.0 & 73.3 & **45.5** & **71.35** \\ \hline \hline \end{tabular} 
\begin{tabular}{l|l l l l l l} \multicolumn{6}{c}{Test-domain selection criterion} \\ \hline ViT & 88.8 & 79.0 & 77.2 & 46.7 & 72.93 \\ GAOeE & 89.7 & 79.0 & 77.0 & **48.8** & 73.63 \\ EMoE-learn & **89.7** & **79.7** & 76.6 & 48.7 & 73.68 \\ EMGoE & **89.7** & **79.7** & **77.5** & 48.8 & **73.93** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Overall OOD performances with 3 selection criteria. All the reported results are obtained following the Domanibed codebase



### Performance Across Different Training set Volumes

Previous research has indicated that modular architectures offer improved data efficiency (Bengio et al., 2020). Therefore, we conducted experiments with GPT2-XL on six tasks using varying proportions of original training data, and the results for all tasks are presented in Figure 7. It can be observed that EMoE consistently outperforms the standard across different data stations. EMoE achieves superior results even when using less than 20% of the data. On SST2, only using 50% data, EMoE shows comparable performance to the standard. More details can be found in the Appendix F.2 Table 20. This further underscores the benefits of incorporating modular structures.

## Appendix C Comparison between EMoE, MoEfication, and GMoE

Leveraging modular designing into neural networks has various advantages (Pfeiffer et al., 2023; Chowdhery et al., 2022; Chen et al., 2023a), including superior generalization abilities (Goyal et al., 2021; Li et al., 2023). Most MNNs are _explicitly_ modular. Among them, MoEs (Szymanski and Lemmon, 1993) is currently a standard architecture for developing MNNs (Shen et al., 2023b; Shazeer et al., 2017; Fedus et al., 2022; Zhang et al., 2022a). Apart from explicit MNNs, Hod et al. (2021); Csordas et al. (2021) explore emergent modular structures in CNNs and LSTMs. Some recent works (Zhang et al., 2022b; Li et al., 2022) focus on the sparsity of more complicated pre-trained transformers. Based on their observations, Zhang et al. (2023) recently explore modularity in pre-trained transformer FFNs by employing handpicked semantic and knowledge-intensive tasks to probe the modular nature of pre-trained transformers.

Having observed that FFN layers in pre-trained Transformers are sparsely activated (many neurons are unused for inputs), MoFication splits transforms FFNs into a sparse MoE, aiming to approximate the functionality of the original FFNs to reduce the computational cost, further improving inference efficiency. Besides, the GMoE makes multiple replicates of the original FFN layer and introduces a learned gate to form a MoE architecture. They claim that such architecture could improve OOD performance from their theoretical perspective of algorithmic alignment framework. MoEfication and GMoE do not touch on how emergent modularity influences the training stage. The table below illustrates a detailed comparison of these works. These differences are summarized in Table 10.

## Appendix D Datasets and Evaluation Metrics

#### d.0.1 Domainbed

The four datasets (PACS (Li et al., 2017), VLCS (Fang et al., 2013), OfficeHome (Venkateswara et al., 2017), and Terra Incognita (Beery et al., 2018)) are selected from Domainbed. Each dataset comprises 4 distinct domains(PACS: {art, cartoons, photos,

\begin{table}
\begin{tabular}{c|r r|r|r} \hline Experts (N) & toppk (k) & MMLU score & times (s) & FLOPS (\(10^{16}\)) \\ \hline w/o tuning & - & 46.79 & - & - \\ full tuning & - & 46.5 & 4988 & 8.97 \\
64 & 16 & **48.08** & 5036 & 9.12 \\
64 & 32 & 47.44 & 5041 & 9.24 \\ \hline \end{tabular}
\end{table}
Table 7: Instruction full-tuned Llama2-7B’s MMLU scores. Times are wall-clock computation times. The term ‘times’ in the subsequent tables refers to the same concept. ’w/o’ refers to ’without’.

Figure 7: Average performance of EMoE with different proportions of training data.



sketches}, VLCS: {Caltech101, LabelMe, SUN09, VOC2007}, Office-Home: {art,clipart, product, real}, and Terra Incognita: {L100, L38, L48, L46}). One or two domains' data are sequentially designated within a single training for OOD evaluation. For example, when training on PACS, {art, cartoons} could be selected as ID training data, while {photos, sketches} are designated for OOD testing. This configuration results in \(C_{4}^{2}+C_{4}^{1}=10\) distinct training processes within each dataset. Suppose there are \(d_{tr}\) ID domains, "Train-validation" means selecting OOD test checkpoints based on ID accuracies from the validation subsets of all \(d_{tr}\) ID domains; "Leave-one-domain-out" means leaving one selected ID domain as a validation set, doing training on \(d_{tr}-1\) domains; "Test-domain" means selection based on limited access to test domains and selection based on these results. The final results are aggregated with selection criteria provided by Domainbed4. As a result, even a variation of 0.1 in the benchmark outcomes signifies a significant improvement.

Footnote 4: [https://github.com/facebookresearch/DomainBed](https://github.com/facebookresearch/DomainBed)

In our experiments, all the hyper-parameters, like training steps, learning rates, and weight decay, except those related to MoEs, strictly follow GMoE.

### Glue

Each task involves one to four OOD tasks from GLUE-X [5], resulting in 13 OOD tasks in total. To illustrate, consider the Sentiment Analysis task: we first fine-tune models on SST-2 from GLUE and report the validation results as ID performance, then use the test data from IMDB [11], Yelp [5], Amazon [12] and Flipkart [13] from GLUE-X for OOD testing.

In the _full fine-tuning_, to ensure convergence and reduce randomness, we train all models 10 epochs across 3 random seeds on each task. Each experiment does a hyper-parameter search on learning rates on [2e-5, 3e-5, 5e-5] as suggested by BERT [14]. The training batch size is 32. In the _LoRA tuning_, following LoRA [15] that tunes models with more epochs and larger learning rates than standard full fine-tuning, all models are trained 20 epochs on small and medium datasets and 5 epochs on large ones (like QNLI, MNLI, QQP). The learning rate is searched in [2e-4, 3e-4, 5e-4]. All methods are implemented with LoRA_rank=8 and LoRA_alpha=16. The training batch size is 16 due to a larger model size. Other settings like max_length following the codebase from hugging

\begin{table}
\begin{tabular}{l|l|c c} \hline \hline
**Aspect** & **EMoE** & **MoEfication** & **GMoE** \\ \hline Research Problem & Exploit the emergent modularity during fine-tuning pre-trained transformers. & Approximate FFNs with sparse MoE to improve inference efficiency. & Validate the OOD improvements brought by Sparse MoE architectures. \\ \hline Methods & Split FFNs & Split FFNs & Copy of FFNs \\ \hline Practicality & No additional trainable parameters. Experts can be recomposed into the standard model so that models can be deployed as a standard model. & May need re-training on the original task. May suffer from inference owing to the specific implementation of MoE architectures. & Additional trainable parameters are introduced. May suffer from inference latency owing to the specific implementation of MoE architecture. \\ \hline Contribution & Significant general improvement without adding parameters and not depending on the specific implementation. & Improved inference efficiency (depending on the specific implementation of MoE), but performance drop. & Significant OOD improvement with additional parameters and specific implementation. \\ \hline \hline \end{tabular}
\end{table}
Table 10: Comparison between EMoE, MoEfication, and GMoE

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Dataset** & **PACS** & **VLCS** & **OfficeHome** & **TerraInc** \\ \hline \#Domains & 4 & 4 & 4 & 4 \\ Classes & 7 & 5 & 65 & 10 \\ Images & 9,991 & 10,729 & 15,588 & 24,788 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Used dataset information from Domainbed 

face5. After training on GLUE, we directly test the selected models on GLUE-X with the data from the official repo6.

Footnote 4: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

Footnote 5: [https://github.com/YangLinyi/GLUE-X](https://github.com/YangLinyi/GLUE-X)

## Appendix E Computation Cost and Memory Usage

### Experiments with public MoEs library

Theoretically, EMoE does not introduce additional parameters compared to the standard model. Although it adds computation in the gating portion within the MoEs layer, it omits a substantial amount of computation within the FFNs layer. For instance, the computation in the gating portion is of the order of \(h\times N\), where h represents the model's hidden size, and N is the number of experts. In contrast, the complete computation in the FFNs layer is of the order of \((h\times h\times 4h)\times 2\), and sparse activations can reduce more than a quarter of this computation. Since \(N\ll h\), theoretically, using EMoE within a single block should accelerate the forward pass of the model. However, in real deployment, we have observed that the hardware implementation of MoEs can result in EMoE being, on average, approximately 10% slower than the standard model. The memory usage is also slightly higher, by less than 5%

Each experiment was conducted on a single NVIDIA 40G A100 GPU. The training times for different tasks ranged from just over ten minutes (RTE) to more than ten hours (QQP).

### Experiments with self-implementation

We further find that the increasing wall time and the GPU memory usage come from the public library tutel we used to implement EMoE. We reimplemented our method and observed that EMoE does not require significant additional run time and memory usage. Specifically, we introduce an alternative implementation approach in EMoE where hidden states are used to calculate gate scores after computing the first layer. These scores mask the outputs of unselected experts, mimicking the effect of MoEs. Though this theoretically increases FLOPS compared to traditional MoEs, in practice, the speed is comparable to standard models, as demonstrated in Tab. 7, 2 All experiments about LLMs are conducted on eight NVIDIA 80G A100 GPU.

## Appendix F Tabular Results

### Full Tables in Full Fine-tuning with Standard Deviation

This section presents the mean and variance of experiments conducted with three different random seeds. The Domanibed results are demonstrated in Table 13, full fine-tuning results are in Table 14, LoRA tuning results are in Table 15.

### Full Tables and Figure Data Sources in Analysis

In the Analysis section 4, we have transformed tabular data into graphs or retained only a subset of the results for clarity. The original and complete results corresponding to them are presented in this section. Fig. 3 is summarized from Tab. 16 and Tab. 17. The OOD results in Table 1 are from 15. The ablation studies are from Tab. 19, and Tab.20.

## Appendix G More Visualization Results

In this part, we demonstrate more gating visualization results on SST-2, STSB, MRPC, and RTE in Figure 8. These results are consistent with earlier findings: (1) Both avg-k gating and learned gating converge, as indicated by the lower halves of the plots. (2) avg-k gating is more stable than learned gating. This could mitigate data inefficiency resulting from inconsistencies in gating across different stages of training [22].

 



\begin{table}
\begin{tabular}{l c c c c c} \multicolumn{5}{c}{Results with ViT-small backbone} \\ \hline
**Algorithm** & **PACS** & **VLCS** & **OfficeHome** & **TerralInc** & **Avg** \\ \hline \multicolumn{5}{c}{train-validation selection criterion} \\ \hline ViT & 86.9\(\pm\)0.2 & **79.7\(\pm\)0.4** & 73.0\(\pm\)0.2 & 44.0\(\pm\)1.1 & 70.90 \\ GMoE & 87.7\(\pm\)0.2 & 79.6\(\pm\)0.4 & 73.1\(\pm\)0.3 & 45.4\(\pm\)0.3 & 71.45 \\ EMoE-learn & 87.2\(\pm\)0.4 & 79.6\(\pm\)0.2 & 72.5\(\pm\)0.2 & **46.1\(\pm\)0.4** & 71.35 \\ EMoE & **87.8\(\pm\)0.2** & 79.5\(\pm\)0.4 & **73.1\(\pm\)0.2** & 45.9\(\pm\)0.3 & **71.58** \\ \hline \multicolumn{5}{c}{leave-one-domain-out selection criterion} \\ \hline ViT & 86.1\(\pm\)0.6 & 79.7\(\pm\)0.4 & **73.3\(\pm\)0.1** & 45.0\(\pm\)0.5 & 71.03 \\ GMoE & 86.5\(\pm\)0.3 & 80.5\(\pm\)0.2 & 73.1\(\pm\)0.3 & 45.3\(\pm\)0.6 & 71.35 \\ EMGe-learn & **86.8\(\pm\)0.0** & 79.6\(\pm\)0.3 & 72.6\(\pm\)0.2 & 45.8\(\pm\)0.6 & 71.20 \\ EMGe & 86.8\(\pm\)0.1 & **80.6\(\pm\)0.4** & 73.3\(\pm\)0.2 & **46.1\(\pm\)0.6** & **71.70** \\ \hline \multicolumn{5}{c}{test-domain selection criterion} \\ \hline ViT & 86.5\(\pm\)0.4 & 78.2\(\pm\)0.4 & **73.1\(\pm\)0.2** & 44.0\(\pm\)0.5 & 70.45 \\ GMoE & 87.2\(\pm\)0.4 & 79.0\(\pm\)0.3 & **73.4\(\pm\)0.2** & 45.3\(\pm\)0.4 & 71.23 \\ EmoE-learn & 87.4\(\pm\)0.2 & **79.1\(\pm\)0.3** & 72.8\(\pm\)0.1 & 45.4\(\pm\)0.6 & 71.18 \\ EMoE & **87.6\(\pm\)0.5** & 79.0\(\pm\)0.2 & 73.3\(\pm\)0.0 & **45.5\(\pm\)0.1** & **71.35** \\ \hline \end{tabular} 
\begin{tabular}{l|c c c c} \multicolumn{5}{c}{Results with ViT-base backbone} \\ \hline
**Algorithm** & **PACS** & **VLCS** & **OfficeHome** & **TerralInc** & **Avg** \\ \hline \multicolumn{5}{c}{train-validation selection criterion} \\ \hline ViT & 89.1\(\pm\)0.0 & **80.7\(\pm\)0.1** & 77.2\(\pm\)0.1 & 47.3\(\pm\)0.8 & 73.58 \\ GMoE & **90.0\(\pm\)0.3** & 80.4\(\pm\)0.6 & 77.0\(\pm\)0.1 & **49.2\(\pm\)1.1** & **74.15** \\ EACoE-learn & 89.8\(\pm\)0.2 & 80.6\(\pm\)0.2 & 76.5\(\pm\)0.1 & 48.7\(\pm\)0.5 & 73.90 \\ EMoE & 89.4\(\pm\)0.4 & 80.7\(\pm\)0.2 & **77.3\(\pm\)0.1** & 48.5\(\pm\)0.5 & 73.98 \\ \hline \multicolumn{5}{c}{leave-one-domain-out selection criterion} \\ \hline ViT & 88.9\(\pm\)0.4 & 80.8\(\pm\)0.3 & **77.5\(\pm\)0.1** & 46.1\(\pm\)0.6 & 73.33 \\ GMoE & 89.3\(\pm\)0.6 & 81.0\(\pm\)0.3 & 76.7\(\pm\)0.1 & 50.1\(\pm\)1.1 & 74.28 \\ EMoE-learn & 89.3\(\pm\)0.2 & 81.2\(\pm\)0.1 & 76.5\(\pm\)0.1 & **50.5\(\pm\)0.2** & 74.38 \\ EMoE & **89.6\(\pm\)0.2** & **81.6\(\pm\)0.2** & 77.4\(\pm\)0.1 & 50.0\(\pm\)1.1 & **74.65** \\ \hline \multicolumn{5}{c}{test-domain selection criterion} \\ \hline ViT & 88.8\(\pm\)0.7 & 79.0\(\pm\)0.5 & 77.2\(\pm\)0.0 & 46.7\(\pm\)0.4 & 72.93 \\ GMoE & 89.7\(\pm\)0.5 & 79.0\(\pm\)0.3 & 77.0\(\pm\)0.1 & **48.8\(\pm\)0.4** & 73.63 \\ EMoE-lean & **89.7\(\pm\)0.4** & **79.7\(\pm\)0.2** & 76.6\(\pm\)0.1 & 48.7\(\pm\)0.3 & 73.68 \\ EMoE & **89.7\(\pm\)0.4** & **79.7\(\pm\)0.2** & **77.5\(\pm\)0.1** & 48.8\(\pm\)0.6 & **73.93** \\ \hline \end{tabular} 
\begin{tabular}{l|c c c c} \multicolumn{5}{c}{Results with ViT-base backbone} \\ \hline
**Algorithm** & **PACS** & **VLCS** & **OfficeHome** & **TerralInc** & **Avg** \\ \hline \multicolumn{5}{c}{train-validation selection criterion} \\ \hline ViT & 89.1\(\pm\)0.0 & **80.7\(\pm\)0.1** & 77.2\(\pm\)0.1 & 47.3\(\pm\)0.8 & 73.58 \\ GMoE & **90.0\(\pm\)0.3** & 80.4\(\pm\)0.6 & 77.0\(\pm\)0.1 & **49.2\(\pm\)1.1** & **74.15** \\ EACoE-learn & 89.8\(\pm\)0.2 & 80.6\(\pm\)0.2 & 76.5\(\pm\)0.1 & 48.7\(\pm\)0.5 & 73.90 \\ EMoE-ELM & 89.4\(\pm\)0.4 & 80.7\(\pm\)0.2 & **77.3\(\pm\)0.1** & 48.5\(\pm\)0.5 & 73.98 \\ \hline \multicolumn{5}{c}{leave-one-domain-out selection criterion} \\ \hline ViT & 88.9\(\pm\)0.4 & 80.8\(\pm\)0.3 & **77.5\(\pm\)0.1** & 46.1\(\pm\)0.6 & 73.33 \\ GMoE & 89.3\(\pm\)0.6 & 81.0\(\pm\)0.3 & 76.7\(\pm\)0.1 & 50.1\(\pm\)1.1 & 74.28 \\ EMoE-ELM & 89.3\(\pm\)0.2 & 81.2\(\pm\)0.1 & 76.5\(\pm\)0.1 & **50.5\(\pm\)0.2** & 74.38 \\ EMoE & **89.6\(\pm\)0.2** & **81.6\(\pm\)0.2** & 77.4\(\pm\)0.1 & 50.0\(\pm\)1.1 & **74.65** \\ \hline \multicolumn{5}{c}{test-domain selection criterion} \\ \hline ViT & 88.8\(\pm\)0.7 & 79.0\(\pm\)0.5 & 77.2\(\pm\)0.0 & 46.7\(\pm\)0.4 & 72.93 \\ GMoE & 89.7\(\pm\)0.5 & 79.0\(\pm\)0.3 & 77.0\(\pm\)0.1 & **48.8\(\pm\)0.4** & 73.63 \\ EMoE-Iean & **89.7\(\pm\)0.4** & **79.7\(\pm\)0.2** & 76.6\(\pm\)0.1 & 48.7\(\pm\)0.3 & 73.68 \\ EMoE & **89.7\(\pm\)0.4** & **79.7\(\pm\)0.2** & **77.5\(\pm\)0.1** & 48.8\(\pm\)0.6 & **73.93** \\ \hline \end{tabular}
\end{table}
Table 13: Overall out-of-domain performances with different selection criteria. All the reported results are obtained from three independent experiments following the Domainbed codebase. The best result is highlighted in **bold**.In cases where results are the same, the smallest standard deviation determines the best result. EMoE demonstrates comparable results to GMoE.





\begin{table}
\begin{tabular}{l|c c c c c c c c c c c} \hline \hline
**Algorithm** & **CoLA** & **Gram** & **MRPC** & **Twitter** & **RTE** & **Hans** & **SciTail** & **STSB** & **Sick** & **Avg** \\ \hline LoRA & 60.89\(\pm\)2.55 & 41.77\(\pm\)1.62 & 86.83\(\pm\)0.87 & 75.42\(\pm\)2.71 & 78.70\(\pm\)1.02 & 60.37\(\pm\)1.31 & 77.36\(\pm\)0.73 & 89.22\(\pm\)0.13 & 78.48\(\pm\)0.33 & 72.12 \\ LoRA-to-EMoE & 61.31\(\pm\)2.14 & 41.99\(\pm\)1.56 & 86.83\(\pm\)0.91 & 75.15\(\pm\)3.02 & 78.58\(\pm\)0.95 & 60.39\(\pm\)1.32 & 77.24\(\pm\)0.68 & 89.23\(\pm\)0.13 & 78.53\(\pm\)0.35 & 72.14 \\ EMoGE & 62.69\(\pm\)0.91 & 42.95\(\pm\)0.95 & 87.82\(\pm\)0.17 & 76.07\(\pm\)2.12 & 79.54\(\pm\)0.45 & 61.56\(\pm\)1.65 & 78.09\(\pm\)0.56 & 89.39\(\pm\)0.31 & 78.57\(\pm\)0.67 & 72.96 \\ EMGeL-toLoRA & 62.69\(\pm\)0.91 & 42.94\(\pm\)0.95 & 87.82\(\pm\)0.17 & 76.06\(\pm\)2.12 & 79.54\(\pm\)0.45 & 61.56\(\pm\)1.65 & 78.07\(\pm\)0.58 & 89.39\(\pm\)0.33 & 78.58\(\pm\)0.67 & 72.96 \\   EMoE+LN & 62.69\(\pm\)0.91 & 42.94\(\pm\)0.95 & 87.82\(\pm\)0.17 & 76.06\(\pm\)2.12 & 79.54\(\pm\)0.45 & 61.56\(\pm\)1.65 & 78.07\(\pm\)0.58 & 89.39\(\pm\)0.33 & 78.59\(\pm\)0.67 & 72.96 \\ \hline \end{tabular}
\end{table}
Table 16: ID and OOD results of BERT-L for different settings. “LoRA-to-EMoE” refers to converting a model tuned using standard LoRA into EmoE for testing. On the other hand, ”EMoE-to-LoRA” involves merging a tuned EMoe model back into a standard standard model during testing.

\begin{table}
\begin{tabular}{l|l|c c c c c c c} \hline \hline
**Backbone** & **Algorithm** & **MRPC** & **CoA** & **RTE** & **STSB** & **SST2** & **AVg** \\ \hline \multirow{6}{*}{BERT-B} & baseline & 88.45\(\pm\)0.40 & 60.67\(\pm\)0.54 & 68.95\(\pm\)0.69 & 87.87\(\pm\)0.12 & 91.97\(\pm\)0.19 & 79.582 \\  & noisy tuning & 88.43\(\pm\)0.12 & 61.79\(\pm\)0.16 & 71.36\(\pm\)0.17 & 88.27\(\pm\)0.94 & 92.32\(\pm\)0.25 & 80.43(\(+\)0.85) \\  & GmoE & 88.63\(\pm\)0.53 & 61.25\(\pm\)2.36 & 70.28\(\pm\)0.68 & 88.63\(\pm\)0.65 & 92.28\(\pm\)0.24 & 80.21(\(+\)0.63) \\  & EMoE-learn & 89.05\(\pm\)0.23 & **62.46\(\pm\)1.01** & **70.40\(\pm\)1.28** & 88.47\(\pm\)0.74 & 92.58\(\pm\)0.14 & **80.59(\(+\)1.01)** \\  & EMC & **89.45\(\pm\)0.36** & 61.55\(\pm\)0.67 & _69.68\(\pm\)1.02 & **88.71\(\pm\)0.50** & **92.89\(\pm\)0.19** & 80.46(\(+\)0.87) \\ \hline \multirow{6}{*}{BERT-L} & Baseline & 89.82\(\pm\)1.30 & 65.41\(\pm\)0.47 & 74.89\(\pm\)1.39 & 89.87\(\pm\)0.28 & 93.50\(\pm\)0.24 & 82.70 \\  & noisy tuning & 90.42\(\pm\)0.35 & 64.75\(\pm\)1.31 & 73.41\(\pm\)1.62 & 90.05\(\pm\)0.46 & 93.65\(\pm\)0.11 & 82.46(\(-\)0.24) \\  & GMoE & **91.24\(\pm\)0.25** & 64.90\(\pm\)1.26 & 74.24\(\pm\)1.04 & 90.00\(\pm\)0.64 & 93.58\(\pm\)0.25 & 82.79(\(+\)0.09) \\  & EMoE-learn & 90.57\(\pm\)0.43 & 65.51\(\pm\)0.32 & 74.72\(\pm\)2.13 & 90.22\(\pm\)0.49 & **93.73\(\pm\)0.35** & 82.95(\(+\)0.25) \\  & EMoE & 90.74\(\pm\)0.65 & **65.79\(\pm\)1.16** & **76.17\(\pm\)0.00** & **90.31\(\pm\)0.43** & 93.58\(\pm\)0.32 & **83.32(\(+\)0.62)** \\ \hline \multirow{6}{*}{ GPT2} & baseline & 84.46\(\pm\)0.51 & 47.07\(\pm\)1.60 & 67.15\(\pm\)0.51 & 86.29\(\pm\)0.29 & 92.13\(\pm\)0.30 & 75.42 \\  & noisy tuning & 84.15\(\pm\)0.92 & 46.16\(\pm\)2.79 & 67.51\(\pm\)0.78 & 86.09\(\pm\)0.38 & 92.13\(\pm\)0.27 & 75.21(\(-\)0.21) \\ \cline{1-1}  & GMoE & 85.07\(\pm\)0.45 & 47.77\(\pm\)3.20 & 67.51\(\pm\)0.51 & 86.57\(\pm\)0.29 & 92.35\(\pm\)0.35 & 75.85(\(+\)0.43) \\ \cline{1-1}  & EMoE-learn & **85.73\(\pm\)0.09** & 47.24\(\pm\)1.48 & 67.99\(\pm\)0.17 & **86.66\(\pm\)0.32** & 92.35\(\pm\)0.38 & 75.99(\(+\)0.57) \\ \cline{1-1}  & EMoE & 85.40\(\pm\)0.77 & **48.00\(\pm\)1.50** & **68.95\(\pm\)0.29** & 86.64\(\pm\)0.16 & **92.70\(\pm\)0.22** & **76.34(\(+\)0.92)** \\ \hline \hline \end{tabular}
\end{table}
Table 14: Results on GLUE tasks with different backbones and algorithms. All the reported results are obtained from 3 independent experiments. The average accuracy (avg) is reported along with the relative improvement compared to the baseline. The best result is highlighted in **bold**.

\begin{table}
\begin{tabular}{l|l l l l l l l l l} \hline \hline
**Algorithm** & **MRPC** & **CoLA** & **RTE** & **STSB** & **SST2** & **QNLI** & **QQP** & **MNLI** & **ID-Avg** & **OOD** \\ \hline \multicolumn{8}{c}{BERT-Large (340 Million Parameters) Results} \\ \hline LoRA & 89.97\(\pm\)0.40 & 63.40\(\pm\)0.62 & 72.92\(\pm\)1.64 & 90.51\(\pm\)0.18 & 93.16\(\pm\)0.19 & 92.20\(\pm\)0.13 & 87.21\(\pm\)0.60 & 85.40\(\pm\)0.07 & 84.35 & 4.86 \\ Block & 89.34\(\pm\)0.84 & 62.10\(\pm\)0.91 & 71.96\(\pm\)1.68 & 90.39\(\pm\)0.14 & 93.35\(\pm\)0.43 & 92.04\(\pm\)0.16 & 88.45\



\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \hline
**Algorithm** & **MRPC** & **CoLA** & **RTE** & **STSB** & **SST2** & **QNLI** & **Avg** \\ \hline LoRA & 89.97\(\pm\)0.40 & 63.40\(\pm\)0.62 & 72.92\(\pm\)1.64 & 90.51\(\pm\)0.18 & 93.16\(\pm\)0.19 & 92.20\(\pm\)0.13 & 83.69 \\ Cluster-top & 90.85\(\pm\)0.61 & 65.33\(\pm\)0.40 & 75.21\(\pm\)1.62 & 90.54\(\pm\)0.03 & 93.50\(\pm\)0.33 & 92.23\(\pm\)0.10 & 84.61(+0.92) \\ Cluster-not-top & 89.61\(\pm\)0.76 & 63.21\(\pm\)0.44 & 72.56\(\pm\)1.28 & 90.31\(\pm\)0.07 & 93.12\(\pm\)0.34 & 92.14\(\pm\)0.18 & 83.49(-0.20) \\ Cluster-bottom & 89.21\(\pm\)0.69 & 63.08\(\pm\)1.09 & 71.72\(\pm\)0.34 & 90.15\(\pm\)0.18 & 92.97\(\pm\)0.19 & 92.13\(\pm\)0.31 & 83.21(-0.48) \\ Random-top & 89.88\(\pm\)0.75 & 63.26\(\pm\)0.39 & 72.56\(\pm\)1.06 & 90.33\(\pm\)0.05 & 93.35\(\pm\)0.00 & 92.14\(\pm\)0.20 & 83.59(-0.11) \\ Random-not-top & 90.09\(\pm\)0.75 & 63.35\(\pm\)0.34 & 72.44\(\pm\)1.19 & 90.44\(\pm\)0.07 & 93.31\(\pm\)0.25 & 92.20\(\pm\)0.16 & 83.64(-0.05) \\ Random-bottom & 89.47\(\pm\)0.23 & 63.17\(\pm\)0.98 & 71.96\(\pm\)0.74 & 90.30\(\pm\)0.23 & 93.11\(\pm\)0.25 & 92.10\(\pm\)0.11 & 83.35(-0.34) \\ \hline \hline \end{tabular}
\end{table}
Table 17: ID results of BERT-L for different settings. “Cluster-top” refers to EMoE utilizing avg-k gating. ”Cluster-not-top” represents a scenario where, during gating, the top-k experts are removed. Similarly, “Cluster-bottom” involves selecting the bottom-k experts with the lowest scores during gating. “Random” denotes the approach of randomly selecting key values to construct experts. The terms ”top,” “not-top,” and ”bottom” have the same meanings as in the cluster situations.





\begin{table} \end{table}
Table 18: Raw OOD performances across 13 tasks. Average results with standard deviation and best results are reported separately. Due to the large deviation across seeds overall methods, we use Friedman rank metrics (Friedman, 1940).





\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline \hline
**Data** & **CoLA** & \multicolumn{2}{c}{**MRPC**} & \multicolumn{2}{c}{**RTE**} & \multicolumn{2}{c}{**STSB**} & \multicolumn{2}{c}{**SST2**} & \multicolumn{2}{c}{**QNLI**} & \multicolumn{1}{c}{**Average**} \\
**Fraction** & **EMoE** & **Standard** & **EMoE** & **Standard** & **EMoE** & **Standard** & **EMoE** & **Standard** & **EMoE** & **Standard** & **EMoE** & **Standard** & **Diff** \\ \hline
1.0 & 62.27 & 60.88 & 87.75 & 86.83 & 80.02 & 78.70 & 89.37 & 89.07 & 95.41 & 95.18 & 92.10 & 91.84 & 0.74 \\
0.9 & 61.58 & 60.01 & 87.52 & 86.49 & 79.87 & 77.85 & 89.18 & 89.07 & 95.41 & 95.16 & 92.05 & 91.94 & 0.85 \\
0.8 & 60.89 & 59.28 & 86.58 & 86.35 & 79.22 & 76.77 & 88.98 & 88.99 & 95.34 & 95.06 & 91.94 & 91.83 & 0.78 \\
0.7 & 59.29 & 58.25 & 86.10 & 85.56 & 77.98 & 76.77 & 87.95 & 87.95 & 95.41 & 95.06 & 92.04 & 91.74 & 0.57 \\
0.6 & 58.91 & 57.93 & 85.61 & 84.76 & 76.29 & 75.45 & 86.70 & 86.17 & 95.26 & 95.03 & 91.83 & 91.54 & 0.62 \\
0.5 & 55.18 & 53.89 & 84.91 & 84.76 & 76.53 & 75.21 & 85.63 & 85.59 & 95.19 & 94.91 & 91.23 & 91.12 & 0.53 \\
0.3 & 50.17 & 50.29 & 82.80 & 82.59 & 73.52 & 72.44 & 80.23 & 79.08 & 94.82 & 94.72 & 90.45 & 90.26 & 0.44 \\
0.1 & 46.47 & 45.54 & 78.17 & 77.85 & 63.05 & 63.17 & 62.33 & 60.81 & 94.49 & 94.38 & 88.39 & 88.32 & 0.47 \\ \hline \hline \end{tabular}
\end{table}
Table 20: Comparison of EMoE and Standard Results with Different Training Data Fraction

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline  & & Top selection & Bottom selection \\ Dataset & Activation Ratio & EMGe & EMoE-learn & EMGe & EMoE-learn \\ \hline \multicolumn{5}{c}{SST2} \\ \multirow{3}{*}{Activation Ratio} & 32 & 0.6879 & 0.6796 & 0.3121 & 0.0552 \\  & 16 & 0.4317 & 0.4247 & 0.124 & 0.1293 \\  & 8 & 0.2616 & 0.2558 & 0.0528 & 0.3218 \\ \hline \multirow{3}{*}{Weighted Activation Ratio} & 32 & 0.731 & 0.7234 & 0.269 & 0.2791 \\  & 16 & 0.4953 & 0.4888 & 0.1041 & 0.109 \\  & 8 & 0.3304 & 0.3237 & 0.0442 & 0.046 \\ \hline \multicolumn{5}{c}{CoLA} \\ \multirow{3}{*}{Activation Ratio} & 32 & 0.6815 & 0.6788 & 0.3185 & 0.3212 \\  & 16 & 0.4292 & 0.4239 & 0.1295 & 0.131 \\  & 8 & 0.264 & 0.2582 & 0.0552 & 0.0637 \\ \hline \multirow{3}{*}{Weighted Activation Ratio} & 32 & 0.7341 & 0.4958 & 0.2659 & 0.2744 \\  & 16 & 0.512 & 0.3389 & 0.0468 & 0.1109 \\ \cline{1-1}  & 8 & 0.3589 & 0.7256 & 0.1063 & 0.0488 \\ \hline \hline \end{tabular}
\end{table}
Table 21: Results of top and bottom selection strategies on SST2 and CoLA datasets with different activation ratios. The activation ratio is determined by calculating the proportion of activated neurons that belong to the selected expert among all activated neurons. Meanwhile, the weighted activation ratio is computed by taking the ratio of the sum of activation scores in the selected experts to the sum of the activation scores across the entire FFNs.

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \hline
**Algorithm** & **MRPC** & **CoLA** & **RTE** & **STSB** & **Avg** \\ \hline standard & 86.83\(\pm\)0.87 & 60.88\(\pm\)2.54 & 78.70\(\pm\)0.59 & 89.07\(\pm\)0.11 & 78.87 \\ EMoE & 88.05\(\pm\)0.35 & 63.11\(\pm\)0.21 & 80.02\(\pm\)0.34 & 89.37\(\pm\)0.24 & 80.14 \\ EMoE-learn & 87.93\(\pm\)0.61 & 62.87\(\pm\)0.71 & 79.90\(\pm\)0.61 & 89.40\(\pm\)0.08 & 80.03 \\ EMoE-last-every2 & 87.27\(\pm\)0.47 & 61.60\(\pm\)0.63 & 79.18\(\pm\)0.17 & 89.38\(\pm\)0.24 & 79.36 \\ EMoE-team-every2 & 87.26\(\pm\)0.21 & 61.82\(\pm\)1.10 & 78.46\(\pm\)1.22 & 89.31\(\pm\)0.15 & 79.21 \\ EMoE-every2 & 86.78\(\pm\)0.34 & 59.21\(\pm\)0.79 & 77.38\(\pm\)1.04 & 89.31\(\pm\)0.06 & 78.17 \\ EMoE-team-very2 & 86.71\(\pm\)1.32 & 54.02\(\pm\)0.47 & 74.25\(\pm\)0.74 & 88.51\(\pm\)0.31 & 75.87 \\ \hline \hline \end{tabular}
\end{table}
Table 19: Results of different MoEs Configurations



\begin{table}
\begin{tabular}{c|c c|c c} \hline \hline  & \multicolumn{2}{c}{CoLA} & \multicolumn{2}{c}{SST2} \\ \hline LoRA & \multicolumn{2}{c}{63.40} & \multicolumn{2}{c}{93.16} \\ EMoE & \multicolumn{2}{c}{65.33} & \multicolumn{2}{c}{93.54} \\ \hline Remained Expert & T-P & P-T & T-P & P-T \\ \hline
64 & 65.26 & 63.4 & 93.42 & 93.16 \\
32 & 65.33 & 63.52 & 93.54 & 93.34 \\
16 & 65.33 & 63.42 & 93.45 & 93.27 \\
8 & 64.33 & 63.42 & 93.45 & 93.27 \\
4 & 63.80 & 63.21 & 93.34 & 93.21 \\ \hline \hline \end{tabular}
\end{table}
Table 22: Comparing EMoE and pruning at different states. T-P means training EMoE, pruning experts with lower selection frequency, and evaluating the pruned model. P-T means pruning lower selection frequency experts of an untrained model, then training and evaluating the pruned model.



Figure 8: Expert selections during training with distinct gating functions (avg-k vs learned gate) and expert types (modules from the standard vs repetitions of the standard). The vertical axis illustrates training steps, with top-down arrangement signifying begin-end; the horizontal axis represents expert selection frequency within 1K steps. (a), (b) and (c) correspond respectively to EMoE, EMoE-learn, and GMoE configurations. The subplots from top to bottom are results for SST-2, STS-B, MRPC, and RTE.