Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
Authors: Ted Zadouri
Ahmet Üstün
Arash Ahmadian
Beyza Ermis
Acyr Locatelli
Sara Hooker
Venue: ICLR 2024 poster (2024)
Keywords: Mixture of Experts
Parameter-Efficient Fine-Tuning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The Mixture of Experts (MoE) is a widely known neural architecture where  an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, we push MoE to the limit. We propose extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts.Our MoE architecture outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts -- less than 1\% of an 11B parameters model. Furthermore, our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected to rigorous parameter constraints.

Published as a conference paper at ICLR 2024
PUSHING MIXTURE OF EXPERTS TO THE LIMIT: EXTREMELY PARAMETER EFFICIENT MoE FOR INSTRUCTION TUNING
Ted Zadouri', Ahmet Üstün', Arash Ahmadian', Beyza Ermiş', Acyr Locatelli", Sara Hooker! Cohere For AI, Cohere eted, ahmet, arash, beyza, acyr, sarahookes/@cchere.com
ABSTRACT
The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, push MoE to the limit. We propose extremely parameter-efficien MoE by uniquely combining MoE architecture with lightweight experts. Our MoE architecture outperforms standard parameter-efficien fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts less than 1% of 1 11B parameters model. Furthermore, our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected rigorous parameter constraints. Our code used in all the experiments is publicly available here: https: / thub. com/ for- aer--aappraamete efficient moe
INTRODUCTION
A conventional training paradigm is to apply the weights of model to each input. Arguably, this is efficient since given input may need all of a model's capacity. contrast, MoEs build on the premise that sub-modular components so called experts can specialize in different types of inputs. This emphasis on conditional computation has important efficiency side-effects such as constant inference costs. This has made MoEs an area of significant research and widespread adoption in the era large-scale Transformers where scaling has increased deployment and latency costs (Shazeer
ct al. 2018 Riquelme et al. 2021 Du et al. 2022
Fedus et al. 2022)
Performance vs Parameter Budget
Performance vs Base Model Size
Average Median Accuracy
62.0
08A-15
Average Median Accuracy
Full Fine- Tuning
10V-10
MoV-60
80.0
82.0
IA
Full fine-tuning
MoLoRA-10
58.0
LoRA
86.0
54.0
52.0
84.0
(TA)
50.0
0.01%
0.6%
3%
5%
770M
38
1B
% of Parameters Updated
Base Model Parameters
Figure Left: Our MoV and MoLORA outperform single PEFT methods using a comparable amount of parameters demonstrated for T5-XL (3B). Righr: Our MoV approach scales up to 1B;; with tiny parameter updates, achieving very competitive performance to full fine-tuning.

---

Published as a conference paper at ICLR 2024
While the majority of work to-date has focused MoEs a pretraining strategy,the inherent motivation of MoEs is not confined solely to pretraining. In fact, the merits of MoEs are arguably well suited to an instruction fine-tuning setting where the data is often deliberately structured to represent a diverse set of tasks, often referred to as multi-task finetuning (Chung et al. 2022 Wei
al. 2022 Sanh et al. 2022 Longpre al. 2023 Muennighoft et al. 2023).
In this work, we pose the question can we leverage MoËs for instruction fine-tuning? One of the main drawbacks of Moïīs paradigm is that it introduces an extreme amount of total parameters (Fedus et al.)
2022). Despite the conditional computation, fully fine-tuning MoE architecture is computationally demanding given the need to update all the parameters. For most practitioners, given the scale of
modern LLMs (Brown et al., [2000; Touvron et al. 2023 Kaplan et al. 2020: Anil et al.) (2023) this is
an infeasible computational cost.
Thus, we focus on more realistic setting for everyday practitioners can we successfully apply MoEs to parameter-efficeet fine-tuning (PEFT) methods such as (IA) (Liu et al. 2022) or LORA (Hu al. 202T) which only fine-tune a far smaller number of parameters. This is a significant challenge not only since our aim is to update only a small percentage of all parameters but as we also navigate the optimization challenges inherent to MoEs already noted by prior work (Chen et al.) 2022 in a more constrained environment.
In this work, we propose a new framework that leverages the benefits of MoE in severely constrained computational environment. We introduce Mixture of Vectors (MoV) and Mixture of LORA (MoLORA), a parameter-efficient adaptation of the Mixture of Experts approach. Unlike the standard MoE, our framework can be utilized in a parameter-limieet nature. We evaluated our framework by fine-tuning T5 models (Raffel et al. 2020) ranging from 770M to 1IB, using P3 instruction tuning mixture (Sanh et al. 2022 which includes 12 different tasks from 62 datasets. Remarkably, our method achieves performance parity with full fine-tuning on unseen tasks by updating less than 1 1% of the parameters. It also significantly outperforms base parameter-efffcient techniques like (IA)² 3 or LORA.
setting due to its lightweight
Contributions (1) We present extremely parameter-
MoEs. This architecture leverages MoEs in more realistic setting using modular and lightweight experts. Our MoEs can be used to fine-tune a dense model by updating less than 1% its parameters. (2) Instruction fine-tuning with proposed methods consistently outperforms traditional parameter efficient methods on unseen tasks, while maintaining high parameter efficiency across different scales. The mixture of (IA)" vectors (MoV) achieves up to 14.57% and 8.39% improvements over the standard (IA) at 3B and 1IB model sizes respectively. This superiority holds across different model sizes, types of experts and trainable parameter budgets. (3) We show that our recipe can match the performance of full fine-tuning at large scales while updating tiny fraction of the model parameters. Our results across 8 datasets from unseen tasks show that our MoV which updates just 0.32% and 0.86% of the parameters in the 3B and 1 1B models achieves higly competitive performance to full fine-tuning with a significantly reduced computational cost. (4) Finally, we present an extensive set of ablation studies that systematically evaluate the efficacy of various MoE architectures and PEFT strategies at various model sizes, different adapter types, the number of experts, routing mechanisms, and the importance of optimizing hyperparameters, especially given the sensitivity of MoE.
2 METHODOLOGY
The standard approach for instruction tuning is to fine-tune all model parameters leading to high compute and memory costs. Our method offers an alternative by pushing the mixture of expert architecture to an extreme degree of parameter efficiency with parameter-efficien fine-tuning (PEFT) methods. this section, we describe our framework in detail.
2.1 PARAMETER-EFFICHE FINE-TUNING WITH (IA)3 AND LORA ADAPTERS
PEFT methods address the challenges associated with updating large number of parameters especially emerging at scale when fully fine-tuning an LLM - by restricting weight updates to a limited number of parameters. To show how our method scales with different PEFT techniques,
2

---

Published as a conference paper at ICLR 2024
claas Moy_Layer (nn. modale) e_experts: int number of experts
W2
def call(self. inputs): inputs anape: [oatch, seg, d_dim] batch, seq, h_dim inputs.
(MOV)
VIC
W
Mov scaling vectors: [n_experts, h_dim] moy_vectora (ell:ppaammmmor__calala nn. im.init.ones. [self.n_experts, h_dim))
('mov m'mor_scalers"'
router probs: [batch, sed, e_experts] router_prood = felf.router router (inputs, selL.n_experls, dlype-' (loal32')
Router
combined vector: [batch, seg, h_dim] mov_ muv_combine jup.einsum( .e, ed-2...d
router_proke mov_vectors)
Kayddee
Miypot
return inpute mov_combine
Figure 2: Left: Overview of the MoV architecture highlighting soft-merging where only the vectors and router are updated for each multi-head attention block, as denoted by color. Right: JAX-like pseudo-code illustrating the core implementation of MoV layer.
experiment with both (IA) and LORA. These methods add small number of parameters to the existing pre-trained model. We briefly introduce each PEFT method below:
(IA)² (Liu et al. 2022) introduces three new vectors, 4 R ly e Rd. In Rd which re-scale
key and value activations in self-attention, and intermediate activations in position-wise feed-forward layers:
QQOok)) Vile
softmax
(Le (OOV (4r O (Wiz))W2
((IA))
where Q, K, V are query. key, and value projection matrices for self-attention, and W1. W2 are frozen weights of the feed-forward layers in the pretrained model. Since (IA)3 only updates 4, l,, In rescaling vectors for each Transformer laye it is extremely parameter-efficient For the 3 billion parameter T5 model (Raffel al. 2020), it only updates 0.018% of the total parameters.
Low-Rank adaptation (LORA; Hu et al. 2021) optimizes low-rank decomposition dense layers in LLMs.. For pre-trained weight matri Wo E Xdd and input activation E Rdd, LORA decomposes Wo into two low-rank matrices:
Wax + AWr = Wor + BAr
(LORA)
where B E Rd A E Rrxda, and the rank < min(dm. dz). During fine-tuning, all pretrained weights are frozen, and only A and B matrices are updated.
We use LORA adaptation for all the linear layers in each Transformer block including query Q, key value V. output O of the self-attention and the feed-forward layers W, and W2. To illustrate its parameter efficiency, for T5 3B model, LORA with a rank of 4, updates 0.3% of the model parameters.
2.2 EXTREMELY PARAMETER EFFICIENT MIXTURE OF EXPERTS
We propose extremely parameter-efficient Mixture of Experts (MoE) framework that leverages lightweight "adapters" as experts on top of a pretrained dense model. Concretely, the MoE is a family of neural network architecture that enables conditional computation through multiple experts that activated based on gating mechanism (router). An MoE layer consists of a router network R and a
For encoder-decoder model with I. number of layers in both sides, (IA)3 introduces T.dd. d, + da) new parameters for encoder and L(2dg 2d, dg) for decoder, due the additional encoder-docnder attention.
3

---

Published as a conference paper at ICLR 2024
set of n experts Ei...., E, where each expert E, is parameterized function. Following Fedus et al. (2022), our router network consists of a dense layer with trainable weights Wa E Rd followed by seftmax function which takes intermediate token representation input and combines the output of each expert based on the gating scores 81, ... 8n:
8; R(x); = softmax (
(Router)
y
S1 Ex(
(MoE)
For Transformer models (Vaswani et al. 2023). dense feed-forward layers are replaced by MoE layers where each expert E, corresponds an independent dense feed-forward network. This multiplies the total number of model parameters as each expert size and number of experts increases. However, in our parameter-efficient MoE architecture, we replace each expert with a lightweight PEFT adapter such as (IA) vectors or LORA adapters. During fine-tuning, pretrained weights of dense layers remain fixed, while experts and router layers are trained from scratch. Unlike the standard MoE, our lightweight experts learn to adapt the pretrained Transformer layers the fine-tuning time. this way, our MoE framework requires limited number of parameter updates and does not introduce a huge model size total.
In addition parameter efficiency, (IA)² vectors enable routing computation with soft merging that leads reduction in FLOPs without information loss. Concretely, since (IA)² vectors used as linear functions, we compute a weighted average of experts first and then apply a PEFT transformation using the combined expert Emiz in Muqeeth et al. (2023):
Emin Σ
81 Es
Emin (x)
(Soft Merging)
For LoRA experts, unlike (IA)² vectors, we compute a weighted average across outputs of LoRA adapters to not lose the memory advantage of low-rank projection.
We call the variants of our method as Mixture of Vectors (MoV) and Mixture of LORA (MoLORA)
that leverage (IA)s vectors LORA adapters as experts respectively, both demonstrating consistent gains over the corresponding PEFT method. 2 the architecture of a MoV layer together with the corresponding pseudo-code Only updating small fraction of parameters through MoV and MoLORA has multiple practical benefits not only to training but to inference time, with the latter being unique to MoE architectures. We provide a brief overview of these gains below:
P
Efficiency in training Our parameter-efffiient MoE formulation leads to a significant reduction in memory. The freezing of most parameters during training reduces the computational overhead of calculating gradients for model parameters but also reduces the memory requirements of storing the optimizer states for the model.
Efficiency at inference The inherent structural modularity of our MoV and MoL.RA methods allows for significant memory gains at inference time. For traditional MoE models, many copies of the full-fledged feed-forward blocks need to be stored in memory at inference time which is an expensive undertaking. With our methods, only a single copy of the model backbone needs to be stored in memory in addition lightweight parameter-efficiees experts. This leads to significant reduction in the memory requirements at inference time.
EXPERIMENTS
Dataset We conduct instruction-tuning experiments using comprehensive set of prompt instructions from the Public Pool of Prompts (P3) dataset Sanh et al. (2022). We follow the same procedure as Raffel et al. (2020). where each task is converted into the format provided templates in (Sanh et al.) 2022). P3 is collection of 62 datascts covering wide variety of tasks. For evaluation, we follow the same zero-shot evaluation presented in TO (Sanh et al. 2022). which includes 8 datasets from different unseen task categories.
The corresponding pseudo-code for MoI.ORA is given in Appendix A.9
4

---

Published as a conference paper at ICLR 2024
Experimental Setup For the base pretrained models, use T5 v1. 11.1+LM adaptation (Lester et al. 2021) that includes T5 models of different sizes ranging from 770M to 11B parameters. For all experiments, fine-tune using Adafactor optimizer (Shazeer & Stemn 2018) with learning rate of 3e 3e_1. We set the sequence length to 1024 for the input and 256 for the target following Sanh et al. (2022). For all parameter-efficient MoE variants, fine-tune T5 models using batch size of 32 over 500K steps.
Baselines We compare our mixture of parameter-efficient experts against both TO baseline as the fully fine-tuned model and the standard purameter-efficient fine-tuming methods (IA)² and LORA with rank 4 for all model sizes. Additionally, for the 3B parameter base model, we evaluate LoRA with ranks and 16. For TO baselines, based on experiments with different hyperparameters, find that a larger batch size and learning rate result in better performance; thus, we replicated TO by fine-tuning for 10k steps with a batch size of 256 and learning rate of le following Phang et al. (2023) these hyperparameters achieve significantly higher results, as shown in Table For (IA)* and LORA, we use the same training hyperparameters, such as a learning rate of 3e -4 and batch of 32 over 500k steps.
Infrastructure All experiments were conducted on TPU v4 machines up to 256 pod slices. We used SeqIO and T5X (Roberts al. 2022) frameworks for training, evaluation, and inference of all the models experimented.
3.1 ABLATIONS
Given that no work to date has studied MoE extremely parameter-efficient settings at scale, we also seek to understand the key characteristics of our proposed methodology by running rigorous ablations. We detail both briefly, along with the experimental setup below:
Routing Input: Token vs Sentence Embeddings Do sentence embeddings in routing lead to higher performance? our main MoV and MoLORA methods, router layers take intermediate embeddings of input tokens input, similar to other MoE architectures (Shazeer et al. [017]] Fedus et al., (022). However, as an alternative, a sentence embedding can be computed for each instruction (prompt with corresponding input) and used as input for the router (Ye et al. (022). To compare both, sentence embeddings that are derived using the Sentence-TSs (Ni et al., 2022), which was initialized from the pretrained T5 and trained on diverse data sources. Without additional fine-tuning, we pass each instruction to Sentence-TT T5 to obtain an embedding with dimension of 768.
Soft Discrete What is the best routing strategy parameter-efficient MoEs? In our MoE framework, we use soft merging of experts as the routing strategy. Soft merging refers to a weighted average of all the experts computed within a specified routing block. As an alternative, we also test a discrete top-k routing strategy as used in standard MoE architectures (Shazeer et al.) 2018. Fedus et al., 2022). In the top-k routing approach, rather than considering all experts for decision, only the top 'k' experts, determined by the router, are chosen for the computation which introduces the sparsity and decreases the amount of computation Note that, although the computation conditional to the top-k experts, the required memory depends on the total number of experts.
Routing Strategy:
We evaluate top-k selection with = {1,2] as they were proposed by previous work (Shazeer et al. 2017 Fedus et al. (022). Additionally, assess discrete routing with top-k using load balancing tollowing to Feduu et al. (2022) which promotes balanced top-k selection through an auxiliary loss, aiming for equitable workload distribution among experts.
4 RESULTS AND DISCUSSION
Parameter efficient MoEs vs PEFTs How does our MoE recipe compare to a single expert PEFT? Table I compares zero-shot performance of PEFTs methods ((IA) and LORA), and our variants of parameter-efficient MoE (MoV and MoL.ORA), using T5-3B as the hase model. We observe that our MoE variants (MoV and MoLORA) present significant performance boost over the standard (IA)² vectors and LORA adapters.
MoV using 30 experts achieves a 14.57% performance improvement compared to its dense counterpart (IA)3. This improvement is consistent across all unseen tasks and is achieved at marginal increase in
5

---

Published as a conference paper at ICLR 2024
Zero-shot Results at 3B Scale
Model
% Params.
ANLI
CB
RTE
WSC
WIC
Copa
WNG
HS
Average
TD-3B (Sanh
50.39
27.51
51.91
FullFFT
TO-3B (our replication)
100%
41.08
80.36
76.17
53.37
53.92
88.94
57.46
29.19
60.06
(TA)
0.01%%
34.08
50.0
66.43
56.25
55.41
79.08
52.09
29.91
52.90
PEFT
LORA (rank 4)
0.3%
37.5
75.57
73.53
61.02
51.25
83.6
54.33
25.32
$7.51
LORA (rank 8)
0.6%
37.5
75.0
77.98
62.5
51.49
83.67
55.72
27.3
58.89
LORA (rank 16)
1.2%
38.5
80.36
76.71
63.46
$1.02
84.5
54.7
27.11
59.54
MoV-10
0.32%
38.92
75.0
78.88
62.5
52.19
85.77
55.96
30.24
59.93
MoV-30
0.68%
38.7
78.57
80.87
63,46
51.1
87.25
56.27
28.63
60.61
Our Method
MoV-60
1.22%
38.83
76.79
74.55
60.1
52.66
55.49
30.47
59.83
MoLORA-2 (rank 4)
0.75%
39.2
82.14
80.32
62.5
50.39
80.58
57.38
28.47
60.12
MoL-ORA-10 (rank
3.18%
78.
63.46
50.86
86.5
55.41
26.72
59.77
MoLORA-15 (rank
4.69%
40.0
80.36
80.51
62,98
50.86
89.0
55.33
27.3
60.79
Table 1: Average median results on unseen tasks for full model fine-tuning (TO), parameter-efficient fine-tune methods ((IA)² and LORA) and our mixture of parameter-efficient experts (MoV and MoLORA), using T5-3B base model (Raffel et al., (020). Note that our TO scores are significantly
higher than the original TO confirming previous work (Phang et al. 2023 Ivison et al. 2023
Our Methods Standard PEFTs
770M Model Size
3B
Full Fine-Tuning
80
Average Median
Full Fine-Tuning
03
Average
14
(1A)*
MoLoRA-10
(1A)
McLoRA-15
Figure 3: Comparison of the top per-performing variants from our proposed mixture of PEFT experts versus their dense counterparts across T5-Large (Left) and T5-XL (Right).
the number of updated parameters only additional 0.018% parameters per expert. In the context of LORA at rank 4, our MoLORA equipped with 15 experts, achieves an average median score increase of 5.70%. This improvement is notably less significant when compared to MoV. We attribute this disparity to the difference in updated parameter count in LORA adapters and (IA)² vectors (0.3% vs 0.018%). Finally, comparing LoRA rank and 16 with MoV-30 covering similar parameter budgets, Mo V-30 outperforms LoRA of both ranks. Overall, learning mixture for both MoV and MoLORA as opposed to a single dense model leads to notable gains in zero-shot performance.
MoV outperforms MoLORA given same parameter budget Between methods, MoV achieves a better performance- parameter cost trade-off at the 3B parameters base model. As shown the left plot in figure Mo MoV with 30 experts, only updating 0.68% of all parameters, achieves nearly the same performance as MoLORA with 15 experts that updates 4.6% 69% of parameters. This shows the effectiveness of MoE approaches even with tiny experts a large base model scale.
Parameter efficient MoFs vs full fine-tuning How does MoE compare to updating all parameters during finetuning? As shown in Table [ when compared fully fine-tuned TO-3B, our proposed methods, MoV and MoLORA both with 10 experts, are on par with full fine-tuning. This is impressive as MoV-10 only updates 0.32% of all model parameters. Furthermore, when increasing the number of experts from 10 to 15 and 30 for MoV and MoLORA respectively, our both methods outperform the full fine-tuning by a small margin.
4.1 HOw DO PARAMETER- EFFICIENT MOES SCALE WITH BASE MODEL SIZE?
Figure (right) shows the scaling characteristic of MoV with 60 experts compared with (IA)3 and full fine-tur toninn for 770M, 3B and 1 IB parameters base models. We find that across all model sizes
We provide comparison for fine-tuning efficiency between MoV and (IA)* in Appendix A.4

---

Published as a conference paper at ICLR 2024
Token vs Sentence Embeddings Routing Effectiveness Expert Count Model Sizes
Top-K Selection
Avernge Median Anturacy
MoV-10
Sentence embeddiig
Moden Accurscy
Moy- (cep-2
Mov-10
700
33 770
MOV-1 1EA)
Routing Stralegles
Base Model Componints
Number Experts
Figure 4: Left: sero-shot performance of passing embedding of the token sequence the router vs. passing tokens to the router. Middle: Zero-shot performance across T5 model sizes (Large, XL, XXL) as the number of experts increases. Right: The effectiveness activating top-k experts.
we evaluate, our parameter-efficien MoEs consistently maintain higher performance compared to standard PEFTs and achieve comparable results with full fine-tuning.
MoV benefits from scaling At all model sizes, MoV-60 significantly outperforms standard (IA)². It is also far closer in performance to full fine-tuning than single expert. For example, at 770M parameters, there a 12.34% performance gap between (IA)² and full fine-tuning vs 5.56% for MoV-60. As the base model scales up. MoV becomes more competitive with full fine-tuning. For 3B and 11B parameter models, MoV-60 achieves performance approximately on par with the full fine-tuning, despite updating less than 1.3% of the total parameters.
MoLORA outperforms MoV in smaller model size regimes As discussed in the main results, at larger model sizes MoV achieves better performance-paramett efficiency trade-off compared to MoLORA. Conversely, at the 770M scale, MoLORA with 10 experts that update 3.18 3.18% of total parameters, performs better compared to MoVV60 and nearly matches the performance of full fine- tuning (Figure Finally, similar to MoV, MoLORA archives higher performance than LORA at both 770M and 3B scales.
4.2 HOw DOES THE NUMBER OF EXPERTS IMPACT THE DOWNSTREAM PERFORMANCE?
The center plot of Figure shows the performance of MoV with different numbers of experts at all model sizes. We find that increasing the number of experts generally improves unseen task performance. However, this improvement is contingent upon the specific number of experts and the base model size. For both 770M and 1 IB parameter base models, our MoV method achieves its best performance by using 60 experts. To illustrate, when the number of experts is increased from 10 to 60, the average median accuracy improves from 52.47 to 53.63 for the 770M model and from 62.3 to 64.08 for the 1IB model. However, the 3B model, using just 30 experts, updating 0.68% of the parameters, reaches peak accuracy with a score of 60.61 at this scale, as performance stagnates when 60 experts are used. This trend of performance improvement by scaling more experts is further endorsed in the context of MoLORA; when scaling experts from sets of (5, 10, 15), there was a corresponding elevation in the average median score, registering at 58.6. 59.77, and 60.79, respectively.
4.3 WHAT IS THE BEST ROUTING STRATEGY IN PARAMETER-EIFICISS MoEs?
In Figure the rightmost plot shows the overall unseen task performance when using different routing strategies for MoV. Specifically, we compare the soft merging of 10 experts (dashed line) with discrete top-2 and top-1 routing. We observe that soft merging significantly outperforms discrete routing in the MoV-10 setting- Specifically, for discrete routing with top-k experts, where 2, the MoE achieves average median accuracy of 54.92 and 57.45 respectively. In contrast, using the soft merging approach, where all experts are activated, we observe an accuracy of 59.93.
and
Furthermore, to understand if recover the performance loss of top-k routing by using load balancing, we integrated the loss balancing following to Fedus et al. 2022). However, we find that the top-k selection of with load balancing loss leads to further decrease in performance 1.5 average median score. Together, these results show that in extremely parameter-efficiem MoE settings, soft merging enables superior performance. Note that top-2 and top-1 routing strategies (among 10
7

---

Published as a conference paper at ICLR 2024
Training Tasks
Evaluation Tasks
0.5
quii
super_ gle
rotten_tom Lommatees
super _ whogrande
Aug Probabilty
044
0.5-
0.
0.3
8
00
Dpent D
Espont
Pxpert
Figure 5: Mean expert routing probabilities for intermediates activations at the last feedforward layer. Values are averaged across tokens and batch. Experts weighted differently in soft merging depending on the task. Left: Measured on tasks seen during training. Right: Measured on unseen evaluation tasks.
experts) perform better than MoV with only 2 experts and a single expert (IA)² respectively, showing that soft merging performs better when larger number of experts are used.
4.4 DOES SENTENCE EMBEDDINGS IN ROUTING LEAD TO HIGHER PERFORMANCE?
To understand the effects of pronounced inductive bias towards task representations in our MoE framework, we compare using sentence embeddings of instructions with token embeddings for the routing imput. These sentence embeddings are obtained offline using an external sentence embedding model. Here, we aim to evaluate how pronounced task information affects the router's s decision and the subsequent generalization capabilities of the model downstream tasks. Figure 4 leftmost plot shows performances of token routing and sentence routing at all model sizes. We find that the token routing exhibits superior performance with 3.03%, 8.86% and 0.94% improvement for 770M, 3B, and IIB base model sizes, respectively. These results suggest that a higher degree of inductive bias for task datasets is not necessarily beneficial, as our approaches can acquire a diverse set of task knowledge directly from the hidden representations of tokens. Furthermore, token routing enables the use of learned experts and routing layers without any prior task information for unseen tasks.
4.5 DO EXPERTS SPECIALIZE IN DIVERSE KNOWLEDGG ACROSS DIFFERENT TASKS?
To understand how expert routing differs for different tasks, we take a closer look how experts are activated for variety of tasks. Figure 5 shows the mean expert probabilities for MoV with 5 experts that are located feed-forward layers in the last decoder block at the 770M parameter T5 model. We selected the last decoder block it has been shown that deeper layers learn more task-specific information (Rogers et al.) 2020). We plot the mean routing probabilities for both training tasks and evaluation tasks that unseen during training understand cross-task generalization through the lens of experts if skills learned at training time generalize to unseen tasks at evaluation time. Intuitively, if experts have indeed learned different skills, expect that they contribute in different degrees to tasks that are different in nature. The amount of contribution is directly reflected in the routing probability of each expert since use soft merging, i.e., summation of expert vectors weighted by the routing probability as described in Figure 2 As such, the mean routing probabilities plotted in Figure [5 provide an overall picture of the contribution of each expert, depending on the downstream task.
Specialization across unseen vs seen tasks As depicted in Figure 5 both evaluation and training tasks lead to the activation of experts at different magnitudes. For example, both quail 1l and tuper-glue.co activate Expert the most out of the 5 experts, followed by Expert 4, but are different both in terms of the relative contribution of each expert and the ordering of the remaining 3 experts based on routing probability. A similar pattern can be observed for common_ge -gen & winogrande, they both activate Expert the most but otherwise different. Overall, the fact that routing specialization seems to oceur regardless whether the downstream task was trained on, suggests that expert specialization is inherent and transferable from tasks to unseen tasks.
8

---

Published as a conference paper at ICLR 2024
4.6 HYPERPARAMETER, SENSITIVITY
Given the widely documented sensitivity of MoE-style architecture to hyperparameters (Fedus et al. 2022, Shazeer et al. 2017), we ran extensive ablation studies to uncover the idiosyncrasy of PEFT methods in the context of MoF. We experimented with batch sizes of 32, 128, 256, and 2048 and we found that the larger the batch size, the more likely our MoEs to collapse to a single expert. Our empirical finding resonates with [Shen et al. (2023) which also finds that small batch is necessary for stable training. For instance, by experimenting with a batch size of 2048 and evaluating every 5K steps up to 20K, we observed that the performance of our after 5K steps, converging to performance levels akin to their dense counterparts. Additionally, we experimented with varying learning rates from 3e-3 Ge We find that smaller learning rate 3e stabilizes training in parameter-effiiiet experts by preventing rapid, imbalanced updates that can suppress diversity and lead suboptimal solutions.
MoEs deteriorates
5 RELATED WORK
Mixture-of- Expertss (MoE) MoE has been deeply explored in Natural Language Processing (Shazeer
et al. 2017 Lepikhin et al. 2020
Kudugunta et al. 2021 Lou et al. 2022 Mustafa et al. 2022
Fedus et al. 2022; Du al. [022]
Zoph et al.] [022] Clark et al. 2022 Zhou et al. 2022 Zuo et al.
Komatsukkka et al. (003)) It effectively increases the model's parameter capacity, activating
specific parts while maintaining similar computation its dense equivalent. Notably, efforts have
been made to improve routing (Eigen et al.] 2014, Hazimeh et al.,
|02T] Lewis et al. 2021 Roller
jet al.] [02T] Zhou et al., 2022] Zuo et al. [022]] Fedus et al. [022
Du et 2022) MoE's ' utility
also has been extended to multi-task scenarios (Hazimeh et al. 202
Kudugunta et al. 2021). Unlike
these, our research scales both data volume and task count in MoE, targeting its training instability
while prioritizing efficient fine-tuning. Recently, Shen al. 2023)
discussed instruction fine-tuning to address the generalization challenges tied to MoE models. We explore its efficacy, emphasizing a unique PEFT component counter the high memory demands of traditional MoE.
Parameter- Parfiitte-Effficientt Fine-tuning, HHoulsy et al. (0199) established "adapters" in the NLP domain to fine-tune BERT. Many adapter variants exist with varied design choices (Bapna et al., 2019 Pfeiffer et al.
2021 1|
Li Liang 2021 Hu et al. 2021] Zaken et al. 2022: Liu et al. 2022). We have chosen
(IA) (Liu et al. 2022) and LORA (Hu et al. 2021) as PEFT components in our MoE because they
offer an optimal balance between performance and parameter efficiency (Mahabadi al. 10211)
Several studies have explored PEFT in relation to MoE, with nuances. Wang et al. (2022) studied single-task fine-tuning by employing random routing and few-shot evaluation. In contrast, our research focuses on instruction-tunine
across multiple tasks, using up to an 11B parameter T5 model Raffel et al. (2020). We emphasize its potential by evaluating unseen tasks through instruction tuning. In Ponti et al. (2022). they learn 'skills' and a binary matrix to determine tasks and skills relationship. Building on this, Caccia et al. 2023) decomposes task-skill matrix introducing 'multi-head' routing function, exploring supplementary inductive biases, and showing effectiveness. Unlike the two mentioned studies, we fine-tunes without specific task identifiers. enabling evaluation of unseen tasks without further few-shot fine-tuning. Contemporaneously. Maxine (2023) uses routing method based on cosine distances between the embedding of input sentences and the average embedding of examples in each dataset used to train experts separately, unlike our end-to-end training of all experts.
6 CONCLUSION
This work introduces parameter-efficient MoEs, Mixture of Vectors (MoV) and Mixture of LoRAs (MoLORA), to address the challenges of standard MoEs in strictly computationally limited environ- ment. Our method outperforms parameter-efficien techniques and achieves performance parity with full fine-tuning on unseen tasks by updating less than 1% of the 3B and 1BB model parameters. Our extensive experiments, including rigorous ablations across model sizes, routing input and routing strategy, confirm the effectiveness of our approach across diverse unseen tasks, highlighting its superior accuracy and computational efficiency. Furthermore, framework's 's versatility seamlessly integrates with other parameter-efficiee strategies and remains compatible with effcciency-enhaccig techniques such as quantization (Dettmers et al. 2023).

---

Published as a conference paper at ICLR 2024
REFERENCES
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernic,, Kevin Robinson, Sebastian Ruder. Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Chool Aakanksha Chowdhery. Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang. John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerrerrtVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization, 2023.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces, 2023.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experr layer in deep learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /ttpss///pppeeeieieeeee net/ htt///oormidddifem (oorrmm?i==Ma/uugDnnn
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellssern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin
10

---

Published as a conference paper at ICLR 2024
Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkum Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction restuutioo-finetuued language models, 2022.
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche. Eliza Rutherford, Tom Hemnigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprins arXiv:2305.14314 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstera. Toju Duke, Lucas Dixon, Kum Zhang, Quoe V Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of mixture-of-experts, 2022.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in deep mixture of experts, 2014.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Lu Wang. Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 18.586/3/ 1DD195544/ UR/jitppss//claaatooooooy.ooo
5409
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, and Matthew Peters. Hint: Hypernetwork instruction tuning for efficient zero-and few-shot generalisation. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11272-11288, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee- teoTThop, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of- experts from dense checkpoints, 2023.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Thang Luong, and Orhan Firat. Beyond distillation: Task-level eexture-of-experts for efficient inference, 2021.
11

---

Published as a conference paper at ICLR 2024
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising grquence-to-sequemee pre-training for natural language generation, translation, and comprehension, 2019.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training large, sparse models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6265 6274. PMLR, 18-24 Jul 2021. URL
mlr. h...ppees//v13//// / 21 html
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin Raffel. Few-shot parameter-efficien fine-tuning is better and cheaper than in-context learning, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Cross-token modeling with conditional computation, 2022.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks, 2021.
Maxine. Llama-2, mo. lora, 2023. 1lama-2-men mo ora-rrr---s ora-f55909444711
URL https://crunie medium. com/
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsoureing instructions, 2022.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng- Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2023.
Mohammed Mugeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing, 2023.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts, 2022.
Jianmo Ni. Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874, 2022.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückléé Kyunghyun Cho, and Iryna Gurevych, Adapter- fusion: Non-destruetive task composition for transfer learning. 2021.
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back-propagation In International Conference on Machine Learning, pp. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2024
Edoardo M. Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining modular skills in multitask learning, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2020.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583- 089558855555,, 2021.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery. Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thore Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t55x and seqio. arXiv preprint arXiv:2203. 17189, 2022. URL htt.t:ttt///rt.t
org abs/ 2203. 17189
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works, 2020.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely mixture-of-experts layer, 2017.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young. Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers, 2018
Sheng Shen, Le Hou, Yangi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou, Mixture-of-experts meets instruction tuning:a winning combination for large language models, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-aute of-adaptations for pfffmeter-efffcient model tuning, 2022.
13

---

Published as a conference paper at ICLR 2024
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetumed language models are zero-shot learners, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thougle prompting elicits reasoning in large language models, 2023.
Qinyuan Ye, Juan Zha, and Xiang Ren. Eliciting and understanding cross-task skills with task-level mixture-of-experts 2022.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-officient fine-tuning for transformer-based masked language-models, 2022.
Yangi Zhou, Tao Lei. Hanxiao Liu, Nan Du, Yanping Huang. Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Le, and James Laudon. Mixture-of-expert with expert choice routing, 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts, 2022.
A APPENDIX
A.1 EXTENDED RELATED WORK
Mixture-of-Experts The Mixture-of of (MoE) has been investigated thoroughly Natural
Language Processing (Lou et al.] 2022 Mustafa et al. 2022 Shazeer et al. 2017 Lepikhin et al. 20200
Fedus et al. 2022 Du et al. 2022 Zoph et al. 2022] Clark et al. [022 Zhou et al. | Komatsuzaki
et al. 2023 Kudugunta et al.] 2021 Zuo et al. 2022) an effective way of increasing the model's capacity in parameter size where certain parts of the model are activated while computation is kept
the same close to its dense counterpart. In the context of MoE, there is body of work focusing on improving the routing (Hazimeh et al.
2021 Lewis et al. 2021 Roller et al.) 2021 Zhou et al.) 2022)
including random routing (Zuo et al. 2022) activating all expert through weighted average (Eigen
et al. 2014) to sparsely select a single or l. experts (Fedus et al., 2022 Du et al.) 2022). MoE has also been invested in multi-tast task settings including multilingual neural machine translation Hazimeh itt al. 2021 Kudugunta et al. (021). Unlike these studies, our research addresses MoE by scaling both the volume of data and the number of tasks, aiming to mitigate the instability inherent in training the MoE models. But our primary emphasis remains on achieving efficient fine-tuning. Recently, (Shen al. (2023) highlighted how instruction fine-tnning with scaled tasks can counteract the generalization challenges tied to MoE models. In distinction from this, our study scrutinizes the efficacy of instruction fine-tuning in the MoE domain, specifically concentrating on a unique ensemble of the PEFT components, considering the memory cost of the traditional MoE can be prohibitive for many practitioners. Similar to the aforementioned work, Ye et al. (2022) utilized MoE in multi-task context, employing BART Lewis al. (2019) as their pre-trained model. However, they limited their experimental scope to a smaller scale and used replicas of each transformer layer as experts, simply multiplying the model by the number of experts. Our work, on the other hand, presents an extreme parameter efficiency with small experts at large scale up to 1 IBB parameter base model.
Instruction Tuning Instruction tuning, as elucidated in (Sanh et al. 2022 Wei et al. 2022 Mishra
et al. 2022). is a technique where language model fine-tuned over collection of tasks using paired prompts and responses. The primary goal of this technique is to enable the model to predict responses accurately based on the provided prompts, thereby augmenting its ability to understand and execute instructions effectively. The method has gained considerable attention due to its pronounced success in enhancing zero-shot performance on tasks to which the model has not been previously exposed. Additionally, instruction tuning has led to breakthroughs such as Chain of Thought Prompting (Weil et al. 2023) where a breakdown of complex problems into smaller steps to produce intermediate reasoning along with the final solution, PaLM (Chowdhery et al. 2022). FLAN (Wei et al.) (022). In our work, we explore the use of instruction fine-tuning with the intention of harnessing its benefits that enable the model to learn from diverse set of inputs where the mixture of expert style models
14

---

Published as a conference paper at ICLR 2024
suits well, for enhanced evaluation performance unseen tasks. Our objective remains to optimize computational efficiency without compromising zero-shot performance.
Crrameter-Efficieee Fine-tuning. Houlsby et al. (0199) established "adapters" in the NLP domain to fine-tune BERT. There are many variants of adapters with different design choices (Bapna et al. 2019
Pfeiffer et al. 2021 Li & Liang] (2021) proposed updating soft prompts concatenated to embeddings
or layer outputs instead of adapters. Zaken et al. (022) show that just updating only a small subset of parameters during fine-tuning (e.g. just biases) very effective. (uu et al. (2021) proposed LORA based on low-rank decomposition matrices of transformer layers. They show superior performance with a smaller parameter budget and no inference cost as LORA parameters can be applied offline to the baseline model. Liu et al. (2022) proposed (IA)², task- vectors to modify attention activation. Instead of using feedforward layers inserted in transformer layers as adapters, they learn vectors to update (by broadcast multiplication) key, value, and linear layer weight matrices. Unlike the other PEFT methods, (IA)² does not induce any additional inference cost and enables mix-batches (from different datasets). The multiplicative nature of the (IA)* creates an interesting opportunity for the mixture-of-expert type of modeling without parallelization overhead. Chen al. (2023) experiment with different design spaces (essentially a hyperparameter search) for PEFT. They suggest four phases: 1) grouping layers into different sets; 2) adding trainable parameters towards each group: 3) deciding which group should be trained; 4) assigning groups with different training strategies. Their finding is that different architectures have different best settings. We have chosen (IA)) and LORA as our PEFT components because they offer an optimal balance between performance and parameter efficiency (Mahabadi et al. 2021 Liu et al. 2022).
Several studies have explored PEFT in the context of MoE or in a similar fashion, albeit with certain distinctions. For instance, Wang et al. (2022) focused on single-task fine-tuning employing a mixture of adapters for BERT base with 110M parameters (Devlin et al. 2019) and RoBERT RTaaaagg with 355M parameters (Liu et al. 2019). incorporating random routing, and adopting a few-shot evaluation. In divergence from this, our work centers instruction-tuning with multiple tasks present during fine-tuning. We underscore the efficacy of this approach by rigorously testing up to 11B parameter text-to-text model Raffel et al. (2020), implementing token routing, and strictly emphasizing evaluation on a set of unseen (held-out) tasks to underscore the potential of instruction tuning. In another work, |Pntt et al. (2022) introduced Polytropon, which involves learning adapters (termed as 'skills') ) specific to each task and employing task-skills binary matrix to determine the skill set associated with each task. In their method, input examples dictate the selection of adapters. These adapters are then aggregated, and the resultant single adapter is integrated into the overall architecture. Extending upon the Polytropon framework, Caccia et al. (2023) implemented distinct skill set for every layer in their variant named Polytropon-S. They introduce a deterministic routing function, delve into supplementary inductive biases, show effectiveness up to 3B models, and they don't employ MoE style architecture. Our research presents departure from these two studies. Specifically, our primary experimental setup employs MoEs that do not require any specific task evaluate
identifier during fine-tuning by the use of the token routing strategy. In this way, our instruction- -otunede MoFs on unseen tasks without any further task-specific few-shot fine-tuning. Contemporaneously. Maxine (2023) uses routing method based on cosine distances between the embedding of input sentences and the average embedding of examples in each dataset used to train experts separately, unlike our end-to-end training of all experts. We showed the scaling property of our MoEs in this setting by fine-tuning models up to IIB parameters.
A.2 ZERO-SHOT EVALUATION FOR P3 DATASET
In our study, we conducted a comprehensive evaluation of the variants of our proposed methods in comparison to our established baselines. This evaluation encompassed various sizes of the T5 model, specifically 770M, 3B, and IIB. These results are given in Table 2 B and Both mean and median scores were reported for every evaluation set derived from the P3 dataset, which covers range of tasks. For further details and a more in depth exploration. please refer to the following URL: http://hhggggaaccaaceec/oo
https: / ng ct www./ataatess.///eeoooooo bici ence//3
15

---

Published as a conference paper at ICLR 2024
La-Lerge (770M)
Model
Params.
Metric
ANLA
Copa
WNG
IS
Average
Full-FT
T0-770M (ours)
100%
median
35.6
71.43
75.63
57.21
51.41
77.0
53,04
26.78
56.01
mean
35.57
57.74
75.88
52.31
52.52
74.6
52.93
26.74
53.54
(lAY'
0.036%
median
33.5
42.86
67.87
62.02
52.35
67.0
51.22
26.33
50.39
PEFT
33.27
45.12
67.08
58.17
52.74
66.63
51.35
26.32
50.09
LoRA trank 4)
0.497%
modian
35.0
55.36
57.4
63.46
50.24
77.0
53.28
26.67
52.3
35.26
51.67
59.35
62,98
50.66
76.5
52.41
27.24
52.0
LoRA trank 40)
5%
modian
35.75
64,29
58.84
63,46
50.31
78.50
$2.94
26.81
53.86
mean
35,54
51.55
59,68
63,56
50.63
76.61
52.47
26,91
52.13
MOV-5
0.27%
median
33.6
41.07
71.48
61.54
50.86
76.5
51 .6
26.02
51.57
meam
33.51
71.26
51.14
73.8
51.55
26.01
51.36
MoV-10
0.55%
median
33.9
42.86
74.19
62.5
50.31
77.0
$2.64
26.34
52.47
33.68
42.38
50.74
74.82
52.2
51.78
Our Method
MoV-20
1.10%
median
33.7
41.U7
73.83
63.46
50.94
75.46
$1.14
25.48
51.89
moan
33.98
45.12
73.36
51.33
73.47
51.3
25.45
51.64
MoV.30
1 1.66%
median
33.75
41.07
72.92
55.77
51.25
77.0
$1.46
26.55
$1.22
33.81
44.88
72.56
56.15
51.29
77.43
51.81
26.52
51.81
MoV-60
3.32%
median
34.0
53.57
75.81
57.69
50.55
77.96
53.12
26.33
$3.63
34.24
52.26
75.02
58.37
50.78
77.06
52.87
26.74
53.42
MoLoRA-10
5.60%
median
33.2
67.86
68.41
64.9
50.39
80.0
52.64
52.64
55.52
mean
33.37
56.31
68.88
63.37
51.55
79.35
52.31
52.31
53.99
Table 2: Zero-shot evaluation of the 770M parameter model across all unseen tasks, comparing different numbers of experts for both MoV and MoLoRA.
T5-XL (3B)
Model
% Params.
Metrie
ANLI
CB
RTE
WSC
WIC
Copa
WNG
H8
Average
Full-FT
T0-38 (Sanh et
2022
meun
33.42
45.36
64.55
65.10
50.69
72.40
50.97
27.29
$1.22
TD-3B (our ceplication)
100%
median
41.08
80.36
76.17
53.37
53.92
88 94
57.46
29. 19
60.06
40.73
74:52
76.82
52.21
53.84
88.99
56.83
29.2
59.14
(LA)
0.018%
median
34.08
50.0
66.43
56.25
58.41
79.08
52.09
29.91
$2.90
PEFT
LoRA (rank 4)
0.3%
median
mean
34.56
37.5
$1.077
75.57
73.53
61.02
54.9
55.61
51.25
83.6
$4.35
25.32
$7.51
52.98
37,85
66.9
77.04
56.73
52,29
82.83
55. 64.64
26.79
57.01
frank
0.6%
37.5
62.5
55.72
meun
37.64
64.05
77.91
$6.71
51.77
82.84
55.23
26.83
$7.03
LoRA (rank 16)
1.2%
median
38.5
80.36
776.7
63.46
51,02
84.5
14.77
27.11
59. 54
mcan
37.11
60.18
51.49
55.14
$7.05
MoVi2
0.18%
median
54.7
46.43
66.06
$6.25
54.86
85.42
$3.75
29.25
53.34
35.14
54.4
MoV-5
0.23%
median
37.1
76.79
78.16
57.69
52.27
86.77
53.99
29.31
59.01
mean
37.66
62.14
78.3
$8.46
53.54
86.52
54.54
28.3
$7.43
MoV-10
0.32%
median
$8.92
62.5
52,
59.95
meun
38.83
29.11
MoV-20
0.50%
median
39.2
75.0
76671
$7.60 89
$3.45
89.01
55.64
30.89
59.7
39.25
611.05
76.53
56.63
$3.45
86.93
56.24
29.36
57.81
38.7
63.46
56.27
mean
38.9
67.5
81.23
59.9
52.43
86, 28
56.
27.57
58.77
MoV-60
1.22%
mediaa
38.83
76.79
74.55
60.1
$2.66
89.79
55.49
30:47
59.83
Our
MoV-10 (top-1)
0.32%
median
micn
38.97
33.9
75.0
71.12
61.06
$7.79
50.71
70.0
51.7
29.24
25.89
51.92
mean
34,31
60.6
71.41
$8.94
51.24
68.39
51.79
25.99
$2.82
MoV-10 (top-2)
0.32%
median
mean
38.89
69.76
74.95
47.69
53.5
79.89
53.83
26.91
55.67
MeLORA-2 (rank
0.75%
median
39.2
82.14
80.32
62.5
50.39
80.58
57.38
28.47
60.12
38.86
80.0
60.0
76
MoL.ORA-5 (mank 4)
1.66%
36.75
$6.25
55.17
55.8
58.6
mean
37.52
62.14
80.22
$2.6
55. 34
84.05
56.04
26.62
56.82
MoLORA-10 10 frank 4)
3.18%
median
38.5
78.57
78.16
63.46
50,86
86.5
55.41
26.72
59.77
mean
38.49
59.9
56.1
57.71
McLORAA-IS (rank 4)
4.69%
median
40.0
80.36
80.51
62.00 98
50.86
89.0
55.33
27.3
60.79
39.73
69.52
80.97
60.67
51.54
86.5
55.03
27.25
58.9
Table 3: In most comprehensive experimental setup, we conducted zero-shot evaluation across all unseen tasks using a 3B parameter model. We compared varying numbers of experts for both MoV and MoLoRA and experimented with top-k selection routing strategy
A.3 TOKEN VS. SENTENCE EMBEDDINGS FOR ROUTING
We present the mean and median results for our routing strategies in Table Specifically, we assessed performance by either passing tokens directly to the router or by passing sentence embeddings.
16

---

Published as a conference paper at ICLR 2024
T5-XXL (1IB)
Model
% Params.
Metric
ANLI
CB
WSC
WIC
Copa
WNG
HS
Average
10-118 Sanh
2022
78.57
90.79
33.65
FulLFT
TO-lIB four replication)
100%
man
median
41.16
47.1
70.12
80.36
81.41
80.83
61.45
60.1
56.58
56.27
96.08
90.02
67.32
59.94
31,61
33.58
65.03
61.70
mean
72.62
58.17
56.66
96.0
30.95
DA)
86.27
mean
63.27
75.31
55.49
52.27
85.74
55.06
30.09
57.41
MoN-10
0.143%
median
45.83
76.79
78:52
53.85
51.88
94.23
63.77
33.5
mann
44.73
70.12
78.88
54,23
53,26
93.64
63.57
33.59
61.5
MoV-20
0.2879%
median
44.5%
76.79
73,83
55.77
$2.98
95.0
62.27
32.92
61:77
Our Method
43.54
69.17
74.1
52.88
54.5
93.93
62.95
32.85
60.53
76.79
53.84
93.62
31.34
mean
43.32
69. 29
77.22
53.56
56.03
95.65
63.52
31.32
61199
MoV-60
0.862%
median
45.17
75.0
83.03
60.1
53.68
95.42
65.82
34,38
61,08
Table We evaluated the largest available model size from the original T5 pre-trained checkpoint, T5-XXL with IIB parameters, to demonstrate the efficacy of our proposed mixture of PEFT experts this scale.
Our findings indicate that, particularly for the T5-XL (3B) model, token routing consistently yields better performance in terms of both mean and median values. The Anli dataset is excluded from our embedding dataset.
MoV Token vs. Sentence Embedding
Model
Metric
CB
RTE
WSC
WIC
Copa
WNG
HS
Average
MoV-10 (Token) 777M
median 42.86
74.19
62.5
52.64
52.64
77.0
26.34
55.12
mean
42.38
74.51
59.23
52.2
52.2
74.82
26.72
54.37
MoV-10 (Embedding)) 770M
median
48.21
67.15
62.98
51.8
50.99
67.0
26.38
53.5
mean
51.67
67.29
58.37
51.79
50.99
65.8
26.57
53.21
MoV-10 (Token) 3B
median
75.0
78.8
62.5
52.19
55.96
85.77
30.24
62.94
mean
63.45
79.49
60.19
53.04
56.27
86.41
29.11
61.14
MoV-10 (Embedding)- 3B
median
$7.14
67.15
61.06
55.33
52.49
82.5
29.08
57.82
mean
51.07
68.81
58.65
55.28
52.57
80.53
28.51
56.49
MoV-10 (Token) 11B
median
76.79
78.52
$3.85
51.88
63.77
94.23
33.5
64.65
mean
70.12
78.88
$4.23
53.26
63.57
93.64
33.59
63.9
MoV-10 (Fmbedding) IIB
median
75.0
78.7
$7.69
54.0
57.85
92.0
33.08
64.05
mean
66.19
79.1
58.37
54.83
58.78
91.17
32.7
63.02
Table 5;
The above results demonstrate the effectiveness of token routing in comparison imposing a strong inductive bias, such as sentence embedding across various model parameters.
A.4 FINE-TUNING AND INFERENCE EFFICIENCY
We have curated a comprehensive table detailing the efficiency metrics, which include parameter number, training and inference time, memory consumption, and overall performance for various methods. By using these metrics, we aim to provide a more nuanced understanding the performance- to-cost ratio.
Table 6 shows the updated parameter count and training time ratios for Mixture of Vectors (MoV) and aIA)² with respect full-fine tuning. (IA)3 trains fastest but with the worst performing method. MoV (10 and 30 experts) offers balance of a large reduction in training speed and higher performance, outperforming LoRA in training time (1.3x) and overall zero-shot performance.
Table shows the inference time metrics for T5-3B. In inference, (TA)² and LoRA add no extra time when their weights are integrated into the base model, while MoV increases it by only 1.06x to 1.15x depending the sequence length and the number of experts. Furthermore, during the inference time, we experimented using only the top-K experts that are determined by the router probabilities to decrease the inference cost. We found that when large number of experts are trained (e.g. 30), using only a top-K K (c.g. 15) subset of these experts during inference maintains high performance. Concretely MoV-30 with 15p-15 experts achieves average performance of 60.25. This is very
17

---

Published as a conference paper at ICLR 2024
Base model
Model
% Params. updated 4
Training time reduction
Average zero-shot performance t
size
(IA)*
0.018%
38%
52,90
3B
MoV (10 Experts)
0,32%
31%
59,93
MoV (30 Experts)
0.68%
27%
60.61
LoRA (rank4)
0.3%
7%
57.51
LoRA (rank 8)
0.6%
58.89
LoRA (rank 16)
1.2%
3%
59.54
(IA)
0.0098%
36%
59.12
11B
MoV (10 Experts)
0.143%
27%
62.30
MoV (30 Experts)
0.431
23%
62.22
Table 6: Fine-tuning efficiency metrics for Mixture of Vectors (MoV) and its dense PEFT counterpart, (IA)". We compare the ratio of updated parameters and the training times with respect to full fine- tuning (with the same batch size), across different scales. MoV exhibits marginal difference in training time compared (IA)² while demonstrating significant improvement in performance
competitive with using all experts (60.61), however, decreases the inference latency by 1% (1.14x) for the sequence length 128.
Finally, Table shows the peak memory usage during fine-tuning for MoV and the baselines using T5-3B as the base model. In terms of updated parameters vs performance trade-off, our MoV method outperforms others, while balancing parameter count and performance effectively. For the maximum memory use since we use use float32 computation for the router, MoV uses slightly more peak memory, however, it offers optimal performance without significant efficiency costs during training.
Seq length: 1024 MoV-10
Method
(IA)*
LoRA 1x
MoV-30 1.07x
Increase in latency
1x
1.06x
Seq length: 512 MoV-10
Method
(IA)*
LoRA
MoV-30
Increase in latency
1x
1x
1.08x
1.11x
Seq length: 128 MoV-10 1.10x
Method
(IA)
LoRA 1x
MoV-30 1.15x
Increase in latency
1x
Table 7: Relative increase in increase time using different sequence lengths for fine-tuning T5-3B model, measured TPU v4-32 with per-replica batch size of 8)
Method
(IA)
LoRA (r=4)
LoRA (r=8)
LoRA (r=16) 9.05
MoV-10 10.44
MoV-30 10.62
Peak Mem (GB)
7.88
8.96
9.01
Table 8: Peak memory usage for fine-tuning T5-3B model, measured in TPU v4-32
A.5 MoLoRA vs. LORA
is important to clarify that multiple parallel LoRA in our MOE-style architecture differs in terms of optimization from single higher-rank LoRA due to the of routers. By conditioning via the router that sensitive to different input data, small LoRA weights the MoLoRA can exploit different low-rank properties. Therefore, MoLoRA trained with MoE-style routing enables better "specialization" for different tasks in each smaller LoRA.
Unlike MoLoRA, single higher-ranke rank LoRA that lacks such router conditioning exploits general properties already emphasized in the base model. This limits specialization and can be suboptimal, especially when fine-tuning data is highly heterogeneous.
18

---

Published as a conference paper at ICLR 2024
MoLoRA vs. LoRA
Model
Metric
ANLI
CB
RTE
WSC
WIC
Copa
WNG
HS
Average
LoRA (rank 40) 770M
median
35.75
64.29
58.84
63.46
50.31
78.50
52.94
26.81
53.86
mean
35.54
51.55
59.68
63.56
50.63
76.61
52.47
26.91
52.13
MoLORA-10 (rank 4) 770M
median
67.86
68.41
50.39
80.0
52.64
52.64
55.52
33.37
56.31
68.88
63.37
51.55
79.35
52.31
52.31
53.99
LoRA (rank 8) 3B
median 37.5
75.0
77.98
51.49
83.67
55.72
27.3
58.89
37.64
64.05
77.91
56.71
51.77
82.84
55.23
57.03
MoLORA-2 (rank 4) 3B
median
39.2
82.14
80.32
62.5
50.39
80.58
57.38
28.47
60.12
mean 38.86
65.71
80.0
60.0
50.8
82.17
56.51
28.03
57.76
Table 9: Comparison of MoLoRA and LoRA for the same parameter budget.
To validate that these are different approaches and outcomes, trained new LoRA with a rank of 40 the TS-Large model (parameter size 770M) to compare with our MoLoRA (rank of 4, 10 experts). Additionally, for the T5-3B model parameters, provide comparison between MoLORA (rank of 4, experts) and a single LoRA with rank Results are given in Table A.5
These results show that our MoLoRA achieves better performance compared to single higher-rank LoRA (similarly parameter-sized) on both scales. This underscores the benefits of MoE-style multiple parallel LoRAs in handling heterogeneous fine-tuning data.
A.6 GENERATIVE TASK PERFORMANCE
To validate our methods' efficacy in an evaluation setup that includes auto-regressive generation, we conduct an additional evaluation on summarization dataset, namely SAMSum (Gliwa et al,, (201)). Table A shows the comparison between our MoV, MoLoRA, PEFT baselines ((74), LoRA) and full fine-tuning using the T5-3B base model. Note that this not held-out task, given that the training split of the dataset is included in the P3 dataset. However, we believe it may be valuable data point for the generalization of our methods in-distribution. We calculate the rouge scores for each model.
These results confirm that MoV and MoLoRA achieve very competitive results with full fine-tuning in more complex generative task, similar to the trend we showed in the paper's 's main evaluation setting.
Comparison of Different Models
Base T5 3B model
Rougel
Rouge2
RougeL.
Avg
IA3
43.6
21.4
36.8
33.9
MoV. experts
47.8
24.8
40.3
37.6
MoV, 30 experts
48.5
25.3
41.1
38.3
LoRA, rank
46.1
22.4
38.0
35.5
LoRA, rank
46.2
22.5
38.1
35.6
LoRA, rank 16
46.3
23.0
38.5
35.9
MoLoRA, experts, rank 4
47.9
24.8
40.1
37.6
Full fine-tuning (TO-3B)
48.6
25.8
41.2
38.5
Table 10: Comparison of T5 3B model variations and their Rouge metric scores.
A.7 CROSS-TASK TRANSFER BETWEEN TRAINING AND EVALUATION TASKS
MoE-style architectures can leverage cross-task relationships between tasks and such knowledge to achieve strong performance in unseen tasks. To validate that there are such cross-task relationships between multiple training tasks and held-out evaluation tasks, we fine-tuned task-specific (1A)3 vectors for each task category in training and evaluated them for held-out tasks. As shown in Table A.7 different training tasks perform best different held-out tasks. For example, the multiple- choice_qa task is most beneficial for super seper_guue_cb and ayperr-lue.copp. Conversely, paraphrase and structure to text tasks work best on super glue rtc. As validated by our experimental results, MoE-style architecture better exploits such latent relationships with expert specialization compared
19

---

Published as a conference paper at ICLR 2024
to other parameter-effcient fine-tuning methods (IA)², and LoRA, resulting in higher performance with similar efficiency.
Model Performance Metries
Task
QA Closed Book
Paraphrase
QA Extractive
QA Multiple Choice
Sentiment
Strueture Text
Summarization
Tople
Classification
RTE (NLI)
57.07
59.81
52.92
35
53.64
47:25
47.90
47,97
WSC (Coreference)
45.00
62,40
40.28
43.84
43,46
62,30
61.44
55.96
WNG (Coreference)
49.59
50.19
51,53
50.93
49.80
50,19
49.86
50.22
HS (Sentence comp.
21.14
25.68
26.48
27.39
25.32
25.44
21.76
27.22
COPA (Sentence comp.)
49.60
$4,59
51.87
66.95
57.51
55.15
53,86
$1.98
Table 11: Performance metrics for different tasks on the large 770M model.
A.8 LIMITATIONS
primary constraint of our experimental framework is its pretrained model architecture of encoder- decoder. Based on the universality of our approach, we believe that our findings extend to decoder- only models. However, we leave this as the subject future work. Additionally, our assessment is exclusively within the context of instruction fine-tuning. Exploration of its efficacy during the pre-training phase remains an avenue for future research.
A.9 MoLORA IMPLEMENTATION
class MoyoRA_Layer (nn. modulc))
n_experts: int # number of exports rank: int # low-rank dimension h_out: int # output dimension
def call (self, inputs): inputs shape: Ibatch, seq, dimi batch, seq, dim inputs.shape
MOLORA In_experts, h rank] molora_A = self. param ('molora_A', mn.init.normall (self.n_experts, h_dim, self.rank))
MoLORA expertss rank, h_out] molora self, wel..param parppaaaaaaaaaaa ('moloraa'" nn. in.init.zerosil (self.n rank,rank, self.b_out))
Ax: [batch, seq, n_experts, rank]
no inp.einne einsum dinsum("..., edr.d..rr.....rr
inputs, molcra_A)
BAX: [batch, seq, expeerss, o_uut]
molora BAx inp. eapseei .'... eer, ero-..........
aclora_Ax, melcra_B)
router probs: [batch, seq, n_experts] router_probs self. router (inputs,, self.n_expertar dtype=' flcat332')
combined LoRAs' outputs: [batch, seg, h_out]
colori_commine inp. inp.einsum(' .'...,, .0..
couter_probs. molcra_HAX)
return melora_combine
20


Reviews:
[{"review_id": "f4nrMNYF6j", "reviewer": "Reviewer_s33v", "summary": "The paper addresses how to leverage MoEs for instruction fine-tuning and presents MoV and MoLORA. The use of MoE in fine-tuning leverages the fact that conditional computation is efficient while avoiding the disadvantage that MoE requires relatively large storage by combining MoE with (IA)^3 and LoRa. In-depth experimental studies demonstrate that the proposed method achieves competitive results with full fine-tuning, only 1% of the parameters involved, and no limitation to the model scale, and can also adapt the model to unseen tasks.", "questions": "The article mentioned that the larger the batch size, the easier it is for MoE to collapse to an expert. However, the difference between fine-tuning and pretrain is that the model has entered a relatively stable and better-performing landscape before training begins. Why under such conditions, the more common gradient direction brought by a larger batchsize will still cause a certain expert to dominate the gradient descent direction of the parameter space?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tLeverage the advantages and avoid the disadvantages of the MoE structure when combined with PEFT methods.\n2.\tIn-depth ablation study demonstrates the capabilities and scalabilities of the proposed method.", "weaknesses": "1.\tMoE takes advantage of the inherent heterogeneity of the data, allowing different parameters to handle different distributions in the data. However, fine-tuning usually focuses on one or a few downstream tasks. In this case, the motivation for using ten or even dozens of experts for learning requires further justification.\n2.\tLoRa exploits the inherent low-rank properties of large models. However, multiple parallel LoRa matrices are mathematically equivalent to higher rank LoRa. The effectiveness of MoLoRa proposed in this article cannot be proven without comparison with higher-rank lora.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZbLxH45fi", "reviewer": "Reviewer_EZ7y", "summary": "The authors propose a parameter-efficient Mixture of Experts (MoE) training approach. By combining MoE architecture with lightweight experts, the paper introduces Mixture of Vectors (MoV) and Mixture of LORA (MoLORA), optimized for parameter efficiency training.  In particular, each expert is replaced with a lightweight PEFT adapter such as $(IA)^3$ vectors or LORA adapters. The proposed MoV and MoLORA, are highly efficient in terms of parameters. By updating less than 1% of the model’s parameters, MoV and MoLORA consistently maintain higher performance compared to standard PEFTs and achieve comparable results with full fine-tuning.", "questions": "The authors show that token routing performs better than sentence routing across various model sizes. An alternative input to the router could be the representation derived from the last encoder layer of T5. How does this encoded representation perform in comparison? This could also offer considerable computational advantages, particularly in situations involving a large number of experts, as it would allow for the pre-computation of the weighted average of experts immediately following the generation of the encoder representation. On the other hand, it would be interesting to see if adding or concatenating the encoder's representation to the representation of tokens further improve performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, presenting a clear and sound approach. The subject matter is highly relevant to the ML community, addressing a critical challenge: while MoEs holds great potential, its practical application has been notably hindered by prohibitive computational expenses and training instabilities, rendering it inaccessible to many researchers. Showing that parameter-efficient methods such as $(IA)^3$ or LORA can substantially improve the feasibility and effectiveness of training MoEs is a significant and valuable contribution to the field.\n\nThe authors have provided extensive ablation studies, systematically evaluating the effectiveness of their proposed MoV and MoLORA approaches against parameter-efficient fine-tuning (PEFT) strategies, and full fine-tuning. The evaluations cover multiple model sizes from the T5-family, adapter types, the number of experts, and routing mechanisms.", "weaknesses": "- All experiments in this paper have been conducted exclusively on T5 model family (encode-decoder architecture). Showing that the proposed light-weight MoEs additionally works for decoder-only architectures would significantly strengthen the paper’s contributions and findings.\n\n- The zero-shot evaluation tasks considered in this paper are mainly classification/multiple choice selection tasks and require generating a single token. The paper does not clearly articulate the adaptability of the proposed method to more complex tasks, such as summarization, translation, or coding, which necessitate the auto-regressive generation of longer sequences. It would be beneficial to explore and elucidate how the experts and routers operate in such long-seq generation scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4M5qet7kFo", "reviewer": "Reviewer_hesi", "summary": "This paper aims at conducting efficient instruction tuning with language models equipped with mixture of expert layers, while to manage the large computational cost, the authors propose to consider each single copy of PEFT parameters as an expert instead of a copy of MLP, the common definition of experts, for better usage. Extensive experiments demonstrate the superiority of the proposed MOV and MOLORA architectures.", "questions": "- Implementation details:\n  - Soft vs. discrete merging: There are two things to decide, including how many experts to choose and how to combine these experts. According to the equation in Sec. 2.2, does soft merging suggest that you first ensemble different expert weights according to the router output and then conduct forward propagation for only once? If so, does this manner only apply for (IA)3 or for LoRA with discrete merging also, since the previous manner is different from our common modeling of MoE architecture? Would you mind further explain how you conduct this ablation?\n  - What is the \"average median score\", the main evaluation metric used by the authors, first mentioned in the first line of Page 6?\n\n[1] Shen, Sheng, et al. \"Mixture-of-experts meets instruction tuning: A winning combination for large language models.\" *arXiv preprint arXiv:2305.14705* (2023).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors utilize extensive accuracy vs. compute figures to demonstrate the superior efficiency.\n- The authors emphasize the scalability of the proposed methods, which is important for large models.\n- The paper writing and architecture is clear with easy to read.", "weaknesses": "- About expert specialization and load balancing: \n  - One of the fundamental reasons to utilize MoE-style architecture is expert specialization.\n  - In Sec. 3, the authors mention to apply load balance loss specifically for discrete merging. Is load balancing applied also for soft merging? And does soft merging rely on load balancing loss to prevent expert collapse? \n  - If not, how to specilize the abilities of each expert to obtain the results in Fig. 5 with simple token-level routing without any explicit control?\n  - If indeed we can get specialized experts, does the similar distribution shown in Fig. 5 suggest that the evaluated tasks still have connection with the several training tasks, otherwise it is hard to understand why a linear router can generate such a low entropy distribution on the evaluated tasks with a zero-shot setting?\n  - In other words, if the evaluated tasks differ from the training tasks a lot, the reasonable routing distribution should be close to uniform, especially for a linear router.\n- About parameter efficient fine tuning (PEFT):\n  - Another way to view this work is another way to increase amounts of PEFT parameters.\n  - But one perspective of current development of PEFT is that the amount of parameters is the most important, while how you deploy the parameters does not make much differ.\n  - In Tab. 1, comparing LoRA (r=4) & MoV-10, LoRA (r=8) & MoV-30 and LoRA (r=16) & MoV-60 (all pairs with similar learnable parameters), the performance gaps are 2.42%, 1.72% and 0.29%, greatly shrinking with respect to the number of parameters, doubting about the scalability of the proposed method. \n- Overall, \n  - From the perspective of MoE, this paper shares similar architecture with [1], and unfortunately still cannot answer the basic question including why the specialized architecture can benefit instruction tuning on novel tasks.\n  - From the perspective of PEFT, the methods can still not convince me that it is non-trivial to other PEFT methods by simply adding more parameters for better results.\n\n[1] Shen, Sheng, et al. \"Mixture-of-experts meets instruction tuning: A winning combination for large language models.\" *arXiv preprint arXiv:2305.14705* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMOca3rJyj", "reviewer": "Reviewer_Bdev", "summary": "The authors propose a new Parameter Efficient Fine-Tuning (PEFT) method which uses Mixture-of-Experts (MoE) architecture. In particular, the authors target two pre-existing PEFT methods which are (IA)3 and LoRA, and replace the vectors in (IA)3 and LoRA components in LoRA with MoE modules - MoV (Mixture of Vectors) and MoLoRA (Mixture of LoRA) respectively. The authors show the effectiveness of the proposed methods by comparing the downstream task fine-tuning performance. MoV and MoLoRA outperforms (IA)3 and LoRA performance and show similar performance as full weight fine-tuning. The authors also add various ablation studies including different routing strategies, varying number of experts, and expert usage on different tasks.", "questions": "- 'efficiency side-effects': side-effect usually means negative, but here constant inference cost is a positive thing.\n- page 2, Contributions (3): 'higly' -> 'highly'\n- Do you have any insights why more number of experts doesn't always give better results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors show that a simple combination of two well-known methods (PEFT and MoE) can achieve better quality.\n- Overall, the proposed method is well described and the paper is easy to follow", "weaknesses": "- Training and inference efficiency analysis is missing. Table 6 briefly touched on the training of MoV case, but it would be good to have more comprehensive analysis such as memory consumption and training time. Also, LoRA has an advantage that the LoRA components can be added to the original weight and no additional inference cost is incurred. On the other hand, MoE might introduce some overheads. As this paper is about an efficient method, this kind of analysis will make the paper more comprehensive.\n- Even though MoV and MoLoRA use small fraction of the original model parameters, they are using quite more parameters than (IA)3 and LoRA. This is important because we are handling quite larger models these days and those information would be useful to consider. Therefore, the information about how much memory consumption and the number of parameters will increase (not full fine-tuning experiments) would be useful to have.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)