Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models
Authors: Sheng Shen
Le Hou
Yanqi Zhou
Nan Du
Shayne Longpre
Jason Wei
Hyung Won Chung
Barret Zoph
William Fedus
Xinyun Chen
Tu Vu
Yuexin Wu
Wuyang Chen
Albert Webson
Yunxuan Li
Vincent Y Zhao
Hongkun Yu
Kurt Keutzer
Trevor Darrell
Denny Zhou
Venue: ICLR 2024 poster (2024)
Keywords: MoE
Instruction Tuning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Mixture-of-Expert models benefit more from instruction tuning than dense models

Abstract: Sparse Mixture-of-Experts (MoE) is a neural architecture design that adds learnable parameters to Large Language Models (LLMs) without increasing computational complexity (FLOPs). Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instruction tuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (in the second and third scenarios), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MoE-32B, surpasses the performance of Flan-PaLM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied by FLAN-MoE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

Published as a conference paper at ICLR 2024
MIXTURE-OF-EXPERTS MEETS INSTRUCTION TUN- ING: A WINNING COMBINATION FOR LARGE LAN-
GUAGE MODELS
Sheng 1.2*
Le Hou
Yanqi Zhou'
Nan Du
Shayne Longpre*.
Jason Wei
Hyung Won Chungl*
Barret Zoph
William Fedus
Xinyun Chen
Tu Vul.*
Yuexin Wu²
Wuyang Chen 5.
Albert Webson'
Yunxuan Li Vincent Zhao'
Hongkun Yu² Kurt Keutzer² Trevor Darrell Denny Zhou'
Google University of California, Berkeley Massachusetts Institute of Technology
University of Massachusetts Amherst
Thhe University of Texas Austin
ABSTRACT
Sparse Mixture-of-Experts (MoE) a neural architecture design that adds learnable parameters to Large Language Models (LLMs) without increasing FLOPss Instruc- tion tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning individual downstream tasks devoid instruction tuning; (ii) Instruction tuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tun- ing supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computa- tional capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (in the second and third scenarios), used independently or in conjunction with task-specific finetuning. Our best model, FLANN MAN-MME22a. surpasses the performance of FLAN-PALMM20 on four benchmark tasks, while using only third of the FLOPs. The advancements embodied by FLAN-MoE inspire re-evaluation the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.
find that MoE models benefit more from
INTRODUCTION
The development of increasingly large and sophisticated deep learning models drives the recent advancements in the field of natural language processing (NLP). Among these models, transformer- based language models Vaswani al. (2017) have emerged the de facto standard for wide range of NLP tasks, owing to their unparalleled capabilities in capturing complex linguistic patterns and generalizing across diverse contexts. One particularly successful paradigm for training such models is instruction-tunng -tning Sanh et al. (2022); Wei et al. (2022a); Chung et al. (2022); Longpre et al. (2023); Muennighoff et al. (2022a); Ouyang et al. (2022a), which enhances their performance on specific tasks by adapting their pre-tee representations to follow natural language instructions.
While the benefits of Large Language Models (LLMs) (Chowdhery et al., 2022; Anil al., 2023; Touvron et al., 2023a;b; OpenAl, 2023; Brown et al., 2020; Scao et al., 2022) are indisputable, their rapidly growing size and computational requirements pose significant challenges in terms of training efficiency, and deployment costs. Thus, there is pressing need for developing scalable techniques that can harness the power of these models without incurring prohibitive computational overheads. On the other hands, models with sparsely activated Mixture of Experts (MoEs) significantly reduce the computational complexity of LLMs. However, we show that conventional, task-specific finetuning MoE models lead suboptimal performance, often worse than finetuning dense models with the same computational cost. One possible reason is that MoE models were prone to overfitting on
Work done Google; 'Correspondence: lehou gongle.com

---

Published as a conference paper at ICLR 2024
task-speciff datasets which have different data distributions compared to the general pretraining data (Zoph et al., 2022). We propose to use instruction-tune tonnnn to alleviate this problem, since it adds training losses from multiple tasks, which can be viewed regularization terms.
We demonstrate this through a two-fold analysis: (I) we expand on the known benefits of instruction- tuning for task-specific downstream finetuning Longpre al. (2023), illustrating its significantly larger impact when applied to MoE models compared to their dense equivalents. (2) we emphasize the necessity of an instruction-tuningg stage for MoE models Shazeer et al. (2017) 1); Du et al. (2022); Fedus et al. (2021); Lepikhin et al. (2020) to surpass the performance of dense models on downstream and held-out tasks. Our unique amalgamation, FLAN-MOE, an instruction tunee model built on the Flan mixture Chung et al. (2022), which successfully harnesses the strengths of both instruction- tuning and the sparse MoE technique. Compared to instruction-tune -unnng dense models, FLANN- MOE effectively and efficiently scales up language models, without necessitating rise in carbon footprint.
We subject our model, FLAN-MOE, to tests across an array of tasks encompassing natural language understanding, reasoning, and question answering. Our evaluation consists of three distinct setups: (i) Direct finetuning of the model on individual downstream tasks; (ii) Instruction tuning succeeded by in-context, few-shot, or zero-shot generalization on downstream tasks; and (iti) Instruction tuning enhanced with subsequent finetuning on individual downstream tasks. The results show FLAN- MoE's marked superiority over its dense counterparts in the second and third settings. Notably, these advancements materialize without the need for augmented computational resources or memory requisites. Our best model manages eclipse the performance of a FLAN-PALM equivalent, requiring only a third of the computational cost per token on four benchmarks. To summarize:
We establish the critical role of instruction-tune -uninngttng in the efficacy of MoE models:
In the absence of instruction tuning, MoE models fall short in performance when compared dense models on downstream tasks.
With instruction tuning, MoE. models exceed the performance of dense models on downstream tasks, well as on held-out zero-shot and few-shot tasks.
We analyze the performance of various MoE models subjected to instruction-tuning
2
METHOD
We leverage sparsely activated Mixture-of- Experts (MoE) Lepikhin et al. (2020); Fedus et al. (202 1); Zhou et al. (2022) in FLAN-MOB models. Similar to the Switch Transformer Fedus et al. (2021), we replace the feed-forward component of every other Transformer layer with MoE layer. Each MoE layer consists of a collection of independent feed-forward networks as the 'experts'. A gating function then uses a softmax activation function to model probability distribution over these experts. Each MoE layer's 's learnable gating network is trained to use its input to activate the best one/two experts for each token of input sequence. During inference, the learned gating network dynamically picks the two best experts for each token. For an MoE layer with E experts, this essentially provides a collection of O(E²) different combinations of feed-forward networks instead of one in the classic Transformer architecture, enabling greater computational flexibility. The final learned representation of a token will be the weighted combination of the outputs from the selected experts.
We fine-tune FLAN-MoE using the language model objective on the FLAN collective dataset Chung al. (2022); Longpre et al. (2023). Each FLAN-MOE will inherit the auxiliary loss setting during pre-training. All the model parameters will be updated. We adapt the sequence length of each FLAN-MME to 2, 0448 for input and 512 for output based on the relative position embedding. The dropout rate is 0.05 and the expert dropout rate is 0.2. The learning rate is le-1 and the batch size is 32. The optimizer setting follows Chung et al. (2022) using AdaFactor Shazeer & Stern (2018). All the FLAN-MoE are pretrained with the same objective and data as Raffel et al. (2020), except that ST-MoE uses GLaM Du et al. (2022); Chowdhery et al. (2022) dataset.
3 EXPERIMENT
We study FLAN-MoE in the context of instruction tuning. We first perform a controlled comparison of FLAN-MOE to equivalent "standard" dense encoder-decoder Transformer (T5), across range
2

---

Published as a conference paper at ICLR 2024
FLOPs per token
Total # Params
MMLU
BBH
Reasoning
QA Direct
Model
Norm, Avg.
Direct
CoT
Direct
CoT
CoT
T5MMALL
0.06G
80M
26.7
7.2
26.7
5.6
10.3
33.8
26.3
FLAN-TDEWAL
0.06G
80M
28.7
12.1
29.1
19.2
15.0
40.9
28.7 (+2.4)
T5
0.3G
250M
25.7
14.1
14.6
14.7
35.3
26.2
FLAN-TORASE
0.3G
250M
35.6
33.3
30.3
26.8
16.4
48.8
33.9 (+7.7)
1.0G
780M
25.1
15.3
27.7
16.2
11.9
36.4
25.7
FLAN-TULAGE.
780M
44.7
38.9
34.7
28.5
22.2
64.6
42.0 (+16.3)
T5x
3,6G
3B
25.3
14.1
27.4
19.3
14.2
38.2
25.9
FLAN-T5x.
3.6G
3R
50.3
46.1
40.2
35.9
33.9
74.1
48.0 (+22.1)
T5xML
13.9G
11B
26.1
19.1
29.5
19.3
21.4
47.4
27.7
FLAN-T5SSI
13.90
1IB
52.6
47.9
45.6
41.6
46.3
80.4
51.7 (+24.0)
PaLM
12.6G
24.3
30.8
30.1
24.9
47.6
27.1
FLAN-PaLM
12.6G
36.4
36.9
75.1
47.5 (+2004)
PaLM
91.6G
62B
55.1
49.0
37.4
43.0
50.6
70.4
51.0
FLAN-PaLM
91.6G
62B
59.6
56.9
47.5
44.9
59.7
85.3
57.6 (+6.6)
PaLM
847G
540B
71.3
62.9
49.1
63.7
72.6
86,0
66.2
FLAN-PaLM
847G
540B
73.5
70.9
57.9
66.3
76.5
89.9
70.3 (+4.1)
Switchasst
0.3G
3.5B
28.3
13.6
1.4
5.2
35.8
20.2
PLAN-Swithhasse
0.3G
3.5B
38.0
34.2
33.2
29.4
18.6
58.0
36.8 (+16.6)
SwitchLaGGE
1.0G
24.0
23.1
0.2
7.2
12.4
33.7
17.7
46.1
40.3
36.3
28.0
25.3
66.5
43.5 (+25.8)
Switchas
13.9G
395B
24.6
15.1
0.0
6.7
9.2
32.5
17.8
FLAN-Switcher.
13.9G
395B
55.6
50.1
47.9
43.5
46.6
78.8
54.2 (+36.4)
GSAMALL
0.06G
0.3B
23.9
0.0
0.2
0.8
0.8
24.1
16.7
FLAN-GA FLAN-GSAMAL
0.06G
0.3B
32.6
26.9
29.6
20.9
16.1
48.9
31.8 (+15.1)
GSEASE
0.3G
26.8
17.6
FLAN-GSRASE
0.3G
1.3B
39.9
33.6
33.7
25.1
22.0
57.9
38.3 (+20.7)
GSLAKOR
1.0G
9.2B
26.4
12.8
0.2
14.3
13.0
31.9
19.2
FLAN-GSSARG
1.0G
9,2B
47.8
35.0
29.2
27.6
69.5
44.5 (+25.3)
GSxl
03.6G
17.4B
25.7
10.0
0.0
0.0
10.4
35.0
18.7
FL
3.6G
17.4B
51.1
42.3
40.1
31.4
34.3
73.9
48.7 (+30.0)
BC
0.06G
0.3B
25.3
1.2
0.1
2.3
0.8
36.0
18.1
FLAN-EC,MAIL
0.06G
0.3B
34.1
25.1
29.2
22.1
16.6
58.1
33.1 (+15.0)
ECRASE
1.3B
25.0
25.9
35.7
18.5
FLAN-ECRASS
33.0
34.0
61.5
40.3 (+21.8)
ECLARGE
1.0G
9,2B
23.4
12.6
0.0
8.6
6.7
40.1
17.3
FLAN-ECIANIE
1.0G
9,2B
48.3
44.5
37.9
32.0
32.2
73.1
46.4 (+29.11)
BC..
3.6G
17.4B
26.7
11.0
0.0
1.9
12.4
34.2
19.4
FLAN-EC..
3.6G
17.4B
52.1
41.4
40.3
33.2
38.1
74.3
49,4 (+30.0)
STAAN
0.3G
1.3B
25.2
17.7
14.0
12.6
25.7
18.1
FLAN-STEAE
0.3G
1.3B
42.4
35.5
34,9
26.4
22.5
61.5
40.4 (+21.8)
ST
32.1G
259B
25.5
15.1
0.0
5.5
9.8
32.1
18.4
FLAN-STEE
32.1G
65.4
63.0
54.4
47.4
66.3
63.9
63.6 (+45.2)
Table I: MoE models improve instruct tinitttning performance on top of dense counterparts. The benchmark suites are MMLU (57 tasks), BBH (23 tasks), Reasoning (4 Tasks), and QA (4 Tasks). The evaluation metric across all benchmarks few-shot prompted accuracy, specifically the exact match. To calculate this metric, we take an unweighted average across all tasks. For comprehensive evaluation, we report the normalized average of MMLUpinet. BBHpirest Reasoningcor. and QApireat- The MMLU and BBH evaluation benchmarks are held-out (not included in the finetuning data.) while the Reasoning and QA evaluation benchmarks are held-in. (Noted that FLAN-STT -T outperforms FLAN-PAR PA-LLM while being <30% of the FLOPS.)

---

Published as a conference paper at ICLR 2024
Held-Out Eval
Held-Our Bud--Out Eval
90
90
Avg Eval Metrics (%)
80
Avg Eval Metrics (%)
80
70
1.1..2
14.4
+9.7
+15.0
+11..6
+13.3
+13.9
60
60
+
-7.1
50
0
89
282
682
1.836
16
32
64
128
# Tasks Instruction Finetuning
Experts for Flan MoE
T5 FT
Flan T5 FT
MoE FT
Flan-MoE FT
Figure 1: The effect of instruction tuning MoE models versus dense counterparts for base-size models (same flops across all models in this figure). We perform single-task finetuning for each model held-out benchmarks. Compared to dense models, MoE models benefit more from instruction-tuning, and are more sensitive to the number of instruction-tunin the performance of MoE models scales better the number of tasks, than the number of experts.
tasks. Overall,
of model sizes in Section 3.2.
We subsequently demonstrate Section 3.3 that scaling up our model, referred to FLAN-MOE, can attain remarkable performance levels. Our most extensive model, 3228-ST328- surpasses the performance of FLAN-PALMe28 while utilizing less than 30% FLOPs per token. We further ablate the various design decisions the Section 4.
3.1 SETTINGS
Traning Data. By default, all models are trained on the 1,836 finetuning tasks introduced by Chung al. (2022). Specifically, Muffin comprises 80 tasks from Wei al. (2022a) and 26 dialog/program synthesis tasks; TO-SF comprises 193 tasks from Sanh et al. (2022); NIV2 comprises 1554 tasks from Wang al. (2022b); CoT comprises reasoning tasks.
Evaluations. We conduct both zero-shot and few-shot evaluations on held-out tasks as in Chung al. (2022) which
included part of the finetuning data. We MMLU Hendrycks et al. (2020), BigBench Hard (BBH) Srivastava et al. (2022), and 4 reasoning benchmarks: GSM8K Cobbe al. (2021), SVAMP Patel et al. (2021), ASDIV Miao et al. (2020), and StrategyQA Geva al. (2021)
For MMLU and BBH, we evaluate both the ability of directly predicting the answer via direct prompting, as well as via chain-of-thought (CoT) prompting Wei et al. (2022b). For reasoming tasks, we only measure CoT prompting accuracy. For all benchmarks except for QA we use the exact evaluation prompts used in prior work: five-shot for MMLU, three-shot for BBH, eight-shot for reasoning tasks, and zero-shot for QA. For a given model we also report a single "normalized average" metric, following the "normalized preferred metric" in BIG-Bench Srivastava et al. (2022). Our normalized average metric is the macro-average over four normalized scores: MMLUDireat BBHp Reasoningcor. and QApirect- Results for every subtask in each benchmark are reported in Appendix B.
3.2 CONTROLLED STUDY ACROSS SCALES
We instruction finetune a range of FLAN-MOE models at batch size 32 and sequence length 2048 for 200k steps. This matches the number of training examples used for FLAN-TS Chung al. (2022). We re-finetuning our own FLAN-T5 variants for fair comparisons.
We use 64 experts for SMALL, RASE, 32B, XI. and 128 experts for all the other model sizes following Fedus (2021); Zhou et al. (2022); Zoph (2022)
4

---

Published as a conference paper at ICLR 2024
Arch:
small
base
large
x
xtd
Type:
Flan- 15
Flan Flan-MoE
MMLU-Direct OShot
BBI--Drect 0Shot
45
50
Avg. Held-Out Score
Avg. Held-Out Score
40
40
35
30
25
20
10
100
104
10t4
10a
10a
GFlops Per Token Prediction
GFlops Per Token Prediction
Figure 2: Average zeroshot performance of FLAN-MOE models versus FLAN-TS dense models for similar effective FLOPs per token over the 57 MMLU tasks and 23 BBH tasks."
Dense Model Size. Figure shows the performance of each model (dense and sparse) against forward-pasd -pss FLOPs. The cost cost-performance Pareto frontier for FLAN-MoE dominates the dense models by wide margin, indicating FLAN-MOE offers strong improvements across all scales from small, to xxl. The effect is particularly large on zero-shot and few-shot MMI.UDirett with absolute performance improvements of 7. 7. For challenging tasks BBH FLAN-MoL offers a strong boost at small scales, while at larger scales the gains are more modest but still significant.
Expert Number. The performance of FLAN-MOE models has been observed to scale with the number of experts included in the architecture, but it tends to saturate beyond certain threshold. Initially, as the number of experts increases in Figure 3, the model benefits from a richer repertoire of specialized sub-networks. This diverse ensemble enables the MoF model demonstrate enhanced adaptability and efficiency in processing complex tasks, leading to improved performance overall. However, the number of experts continues to grow, the performance gains begin to diminish, eventually reaching a point of saturation for BASE-sized model.
Routing Strategy Routing strategy an essential component of Mixture-of Mixture-of-Experts (MoE) models, playing pivotal role in determining the effectiveness and efficiency of these models. This distribution process is crucial for maximizing the utilization of the model's capacity while minimizing the risk of overfitting. An effective routing strategy not only ensures that the appropriate experts are selected for a given input, but also that resources are allocated optimally, leading to enhanced computational efficiency and faster training times. Consequently, there have been trending strategies, token- choice Lepikhin et al. (2020) which lets the token select the top-K experts, and expert-choice Zhou al. (2022) which lets the experts select the top-K tokens.
We presented a detailed study about how different routing decisions affect the instruct fine-tuning performance in Figure and Table which includes the checkpoints from Switch Transformer top-1 loken-choice gating (FLAN-Swice GShard top-2 token-choice gating (FLAN-GS) and expert-choice top-2 gating (FLAN-EC) models pre-trained on the same T5 Raffel al. (2020) dataset. Among these benchmarks, the MMLUpied model shows the most significant improvement, with an increase from 38.0% 39.9% for BASE/LARGE- sized models. Although the gains at the extra-large scale are more modest, they remain noteworthy and meaningful. It's noteworthy that instruction- amplifies the performance both held-out MMLU, BBH, and held-in QA and reasoning benchmarks for MoE models versus dense models of equivalent capacity. The advantages amplified even further for larger MoE models. For instance, instruction-tuniig enhances the performance of ST by substantial 45.2%, while the improvement observed for FLAN-PALMeaE is comparatively modest at around 6.6%.
significantly
Furthermore, the FLAN-EC strategy consistently outshines the FLAN-GS approach for the given model across various scales and tasks. is noteworthy that the performance gap between the token-choice and expert-choice models can be bridged when incorporate advanced auxiliary loss
5

---

Published as a conference paper at ICLR 2024
122
Are 100-0 Scm
12.5
12.9
Fan Mar-ME
TlanksE Seiih
Ran
Ran
Kot
12.
42
02
80
120
120
40
02
30
100
123
24
24
26
0.
10
02
04
28
Expert
Expott
Pub Taker Trodctes
Tokes Prodictin
(a) Scaling (MMLU)
(b) Scaling (BBH)
(c) Routing (MMLU)
(d) Routing (BBH)
Figure 3: Average few-shot performance of FLAN-Men MoE models over the 57 MMLU tasks and 23 BBH tasks. (orange, blue, green stands for small, bsae, large model sizes.)
and pre-training strategy as exhibited in ST-MoE Zoph et al. (2022). This integration led to the development of our FLAN-ST models. Considering that the largest ST-MoE set the benchmark in a variety of NLP tasks when appropriately fine-tuned, we have also decided to scale up FLAN-ST, employing instruction fine-tuning. We presented learnining efficiency experiments in Appendix A.1.
3.3 SCALING UP FLAN-MOE
We increase the architecture size to assess the performance of FLAN-MOF in the large-scale regime. As discussed above, we instruction fine-tune the largest ST-MoE328 Zoph et al. (2022) model with 12 expert layers in encoder, and decoder, respectively; these are non-uniformly distributed, with 64 experts per layer, and K 2 activated per token. It was trained at a batch size of 32 and sequence length of 2048 for 200k steps. We average checkpoints towards the end of training. The model FLAN-ST:2R. comprising a total of 32 billion parameters, only utilizes 32.1 GFLOPs per token, which amounts to merely one-third of the computational power required by FLAN-PALMen model. Additionally, all the routers combined account for less than 4 million parameters. Table illustrates the performance of this model alongside current state-of-the-ee instruct fine-tuned models.
FLAN-ST 32B achieves 65.4% few-shot MMLU benchmark accuracy and 54.4% few-shot BBH benchmark accuracy, with a relatively modest architectural size and training count. Notably, FLAN- ST surpasses the performance of FLAN-PALMens which consumes nearly triple the compute resources, by a substantial margin across all four benchmarks. However, it is important to ac- knowledge the considerable performance gap that persists between the largest FLAN-PALMsa0p and FLAN-ST:2n models.
DISCUSSION & LIMITATIONS
4.1 ABLATION STUDIES
Sparse models have performed remarkably well in the regime of large datasets, but have sometimes performed poorly when finetuning data is limited Zoph et al. (2022); Fedus et al. (2021). Instruction tuning can also be viewed as a continual finetuning stage, so we present a detailed study on how different factors impact the instruct finetuning performance of FLAN-MoE and offer practical recipe. All the discussion is based on instruction tuning BLLAN-ECAAA//LLANN-. BASE for 100k steps.
Auxiliary Loss. The incorporation of auxiliary loss Lepikhin et al. (2020); Zoph et al. (2022) helps mitigate the risk of overfitting by promoting the diversification of the experts' knowledge and improving the model's generalization capabilities for sparsely gated mixture-ou cotuur--o--xpperr models. Fur- thermore, auxiliary losses can be employed to address specific issues, such as load balancing among experts or preventing expert collapse, which can further enhance the model's overall performance. We experiment with both balancing loss that is used in Lepikhin et al. (2020) and router Z-loss that is used in Zoph et al. (2022) in Table 2. The implementation of balancing loss contributed to enhanced performance on MMLU, BBH, and GSM8K for FLAN-EC deterioration of performance. Conversely, for FLAN-ST BASE observed a contrasting trend. We conjecture that the discordance between the auxiliary loss during pre-training and instruction-tuniin -entiiiin--unnggggg could potentially disrupt the optimization process, thereby leading to suboptimal FL.AN-MOF.
whereas Z-loss resulted in

---

Published as a conference paper at ICLR 2024
40
Avg. Held-Out Score
Avg. Held-Out Score
35
35
30
30
25
25
Baseline
Frezze MoE
+- Baseline
Frezze MoE
20
Freeze-Gan Gate
Z-loos
20
Freeze: Gate
Z-loss
Preeze-Expert
Balanco-los
Freeze- Expert-Expprt
Balancodloss
15
50
100
150
200
50
100
200
Number of Steps (k)
Number Steps (k)
(a) Different Tuning Strategies for FLAN-STRAN (b) Different Tuning Strategies for FLAN-EC
Figure 4: Average few-shot performance of FLAN-MOE with different finetuning strategy.
Finctuning
Strategy
MMLU Direct
BBH Direct
GSMBK CoT
Avg.
Finetuning Strutegy
MMLU Direct
BBH Direct
GSMHK CoT
Avg.
Baselinep BCu
Baselineguaw STaw
40.2
33.9
6.6
38.0
Preeze-Galén.ae
40.6
33.5
6.4
38.2
Freeze- Expertri RCaoe
38.3
32.5
5.4
36.2
Freeze Experin ST...
39.6
32.9
37.3
Freeze Mo MLossLLBECe
Freeze-MaFFe PLASSEESTTTT..
Z-losspuas
38.9
32.8
5.7
36.8
Z-losSm Balance collance-lsssp... STiuus
40.6
33.4
6.5
38.1
Balance-
40.8
33.4
38.3
38.8
31.3
36.2
Table 2: Ablations on different finetuning strategies of FLAN-ECRASA and FLAN-ST
Expert/Gating Freeze. In an effort to enhance the generalization capabilities of sparse models and combat overfitting. researchers have discovered that finetuning subset of model parameters results in improved generalization performance for ST-Moo models, as noted in the study by ST-MoE Zoph et all (2022). Interestingly, it was observed that updating non-MoE parameters yields similar outcomes to updating all parameters, while updating only expert parameters performs slightly better. We conducted experiments by freezing the gating function, expert modules, and MoE parameters of the given model, presented in Table 2. The results indicate that freezing either the expert or MoE components negatively impacts performance. Conversely, freezing the gate slightly improves performance, albeit not significantly. We postulate that this observation is related to the under-fittin fitting of the FLAN-MOF,, as in Figure 4, which depicts the finetuning data efficiency ablation study.
Finetuning v.s. Instruction tuning. To compare the gap between finetuning MoE directly and FLAN-MOF,, experiment with single-task finetuned MoE, single-task finetuned FLANN-MOË, and dense counterparts Figure 5. We perform Held-Out tasks, we observed that the improvement of FLAN-MoE over finetuning MoE noticeably larger compared to the improvement of FLAN-TS over finetuning T5. This difference becomes even more pronounced when there is a scarcity labeled data or when the model size is increased. This suggests that FLAN-MOE mitigates the overfitting issue associated with directly finetuning MoE. Despite their advantages such as increased adaptability and efficiency managing complex tasks, MoE architectures are prone to overfitting during the finetuning process, as discussed in (Zoph et al., 2022; Artetxe et al., 2022), which may be attributed to the additional hyperparameters for stabilizing MoEs and the aforementioned increased sizes. This can be seen in Figures 5 and where single-task fine-tuned MoE models sometimes underperform their dense T5 counterparts. Interestingly, compared dense models, MoE models benefits more from instruction-tume better with respect to the number of tasks rather than the number of experts. We hypothesize this is due to the specialized nature of individual experts, which can lead heightened sensitivity noise and limited generalization capabilities when exposed to unseen data. Noted that we follow (Chung al., 2022) for the speciici fine-tuning datasets, which could be domain -ppaiiffc and present extra challenges for MoE models and therefore suboptimal performance. The findings are consistent with those in (Artetxe al., 2022), which observed mixed fine-tuning performance MoE models
search for each finetuning setting. On
In addition, MoE models scale
Appendix A shows details hyperparameter sensitivity, LM adaptation, and decoder-only MoF.
7

---

Published as a conference paper at ICLR 2024
Held-Out Eval
Held-Out Eval
90
90
+6,6
+120
80
80
Eval Metrics (%)
Eval Metrics (%)
+19.6
+1344+194
70
+12.7
117.8
+33.2
+22.3
+7.8
+-14.9
60
60
50
CondaQA
CxC
PubmedQA SearchQA
CondaQA
CxC
PubmedQA
SearchQA
(a) FLAN-ECRASE V.S. FLAN-T5
(b) FLAN-ECCARGR V.8. FLAN-T5, AS
FT
Flan-T5-FI
MoE FT
Flan-MoE FT
Figure 5: FLAN-MoE Outperforms MoE (the pretrained MoE) on Single-Task Finetuning. In other words, adding an instruction-tune tutionnt-tunnggggg stage significantly improves the performance. We compare single- task finetuned MoE, single-task finetuned FLAN-MoB and dense counterparts. The performance gap between FLAN-MoE and MoE is noticeably larger than that between FLAN-T5 and T5.
across 8 tasks, both with and without fine-tuning. Notably, in all instances, MoE models produced inferior results compared to their Dense model counterparts.
4.2
TRAINING AND INFERENCE COST OF FLAN-MOE
4.3
LIMITATIONS
Expert Specialization. As the size of a FLAN-MOE model increases in Figure 6, notable rise in expert specialization tends to occur. Larger models entail higher number of parameters and more complex structures, which inherently provide a broader scope for each expert to specialize specific facets of the problem space. This increased specialization can be understood as form of division of labor, where each expert sub-network becomes adept at handling certain type of task or data pattern. Consequently, the overall model can demonstrate a higher degree of adaptability and precision in tackling diverse and complex tasks. We also observe that after instruction tuning, the MoE models exhibit better expert usage, which may help prevent the expert collapse for generalization after Instruction-tuning as in Zuo et al. (2021).
Failure Cases. The fine-grained specialization of FLANN-ME models, particularly when fine-tuned on English-only instructions, can inadvertently lead to a narrowing of the model's capacity to effectively pro-
Expert Usage (%)
65
cess and generate content in multiple languages. We found all the FLAN-Mes Moo perform poorly multi- lingual benchmarks including TyDiQA and MGSM. Even the largest FLAN-ST 32w only achieves 15.5% on MGSM and 25.1 % on TyDiQA, which only comparable to the vanilla PaLM62B with 18.2% on MSGM, and PaLM with 25.0% on TyDiQA. It also underperform LAAN-AALMvariane ses that this issue may stems from the model's over- optimization towards the specificities of the English language during finetuning, which can impede its ability to navigate the complexities of other languages. Consequently, while MoE models offer significant benefits in terms of task task-specific adaptability and efficiency, their potential shortcomings multilinguality highlight the importance incorporating diverse linguistic data during the training process to ensure broad and effective language coverage.
63
100
200
Number of Steps (k)
We hypothe-
Figure 6: Expert usage of FLAN-EC small / base large during instruction tuning, where larger models entail smaller expert usage.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Instruction Tuning. Instruction tuning has evolved as strategy to enhance the functionality and interactivity of large language models (LLMs) for dialogues and complex tasks. Prior studies, including Raffel et al. (2020); Liu et al. (2019); Aribandi et al. (2021), have delved into large-scale multi-task fine-tuning to enhance the downstream single target fine-tuning, albeit without instruction prompts. Initiatives such as UnifiedQA Khashabi et al. (2020); McCann et al. (2018), 20; Keskar et al. (2019) have amalgamated a multitude of NL.P tasks into singular generative question answering format, utilizing prompt instructions for multi-task fine-tuning and evaluation. Efforts like Natural Instructions Mishra et al. (2021), Flan 2021 1 Wei et al. (2022a), and P3 (the Public Pool of Prompts, Sanh et al. (2022)) have collated vast NLP task collections, templatizing them with instructions for fine-tuning models to enhance their adaptability to unseen instructions. Some studies, such as Super-Natural Instructions Wang et al. (2022b) and OPT-IML Iyer et al. (2022), took this step further by combining numerous datasets and tasks into a single resource. In the meantime, others like xP3 Muennighoff al. (2022b) introduced multilingual instruction tuning and Flan 2022 [4] employed Chain-of- Thain-of-Thought training prompts. Recently, there has been move towards expanding task diversity more assertively using synthetic data generation, particularly for creative and open- ended dialogue Wang et al. (2022a); Honovich et al. (2022); Zhou et al. (2023). Some researchers have also tried to provide human feedback on language model responses Ouyang et al. (2022b); Glaese et al. (2022); Nakano et al. ); Bai et al. (2022b;a), or bridge the modality gap with multi-modal instruction fine-tuning Liu et al. (2023); Dai et al. (2023); Li et al. (2023).
Sparse Mixture of Experts models. The foundation of our work is built on the concept of deep sparse cexture-of-Experfr (MoEs), topic that has been independently explored in both Computer Vision Riquelme et al. (2021); Lou et al. (2021); Mustafa et al. (2022); Shen et al. (2023) and Natural Language Processing Lou et al. (2021); Mustafa et al. (2022); Shazeer et al. (2017); Lepikhin et al. (2020); Fedus et al. (2021); ); Du et al. (2022); Zoph et al. (2022); Clark et al. (2022); Zhou et al. (2022); Komatsuzaki al. (2022); Kudugunta al. (2021); Zuo et al. (2021); Artetxe al. (2022). The idea revolves around conditional computation, which aims to enhance the number of model parameters without a corresponding rise in computational expense. MoE models leverage learned gating mechanism that triggers only select subset of k experts out of total of E for given input. This approach allows an input either select all experts Eigen et al. (2013) or merely a sparse mixture of them, as observed in recent massive language models Fedus et al. (2021); Du et al. (2022). While a number of studies have sought to enhance the gating mechanism itself Hazimeh et al. (2021); Lewis et al. (2021); Roller et al. (2021); Zhou et al. (2022), MoE models have also been explored in the context of multitask learning Hazimeh et al. (2021): Kudugunta et al. (2021); Ma et al. (2018). This essentially permits input to choose the most relevant expert(s) for given task, thereby optimizing the processing and results. Nevertheless, the instability of MoE models during fine-tuning or multitask learning has consistently been challenge. Our study aims to investigate whether instruction fine-tuning with scaled tasks might contribute to mitigating the generalization issues inherent MoE models.
6 CONCLUSION
In this work, we have introduced FLAN-MOE, an innovative method to amplify the scalability of instruction-tumm language models by employing the sparse Mixture-of- Ex-extss (MoE) technique. Our strategy amalgamates the merits of instruction instruction-finctuinng, which bolsters task-specific performance, and MoE, which provides computational efficiency coupled with diminished memory requirements. We have substantiated the effectiveness of FLAN-MoE through comprehensive experiments across a wide spectrum of Natural Language Processing (NLP) tasks, such as natural language understanding. question answering, and reasoning. Our results consistently underscore the superior performance of FLAN-MoL over current sate--of-the-ae art methods, marking substantial advancements in both accuracy and efficiency. Notably, compared to dense models, these advancements attained without necessitating an increase in computational resources or memory usage during training and inference, often even reducing the resource requirements in the process.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. /0V:: 2023.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint wrxiv:211..000 10952, 2021.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11699-11732 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint DxXiv:2204.0566. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiw:2212.0807., 2022b..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. preprint arXiv:2005. 14 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling centruction-inntunne language models. arXiv preprint arXiv:2210.1 11416, 2022.
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann. Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In ICMI., pp. 4057-4086. PMLR, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10: 14168, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305. 06500, 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In ICML., pp. 5547-5569. PMLR, 2022.
David Eigen, Mare Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXiv: 13/2.431, 2013.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.039611 2021. URL peattss:/arxhhe org/ abs/ / 2201.111 03961.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint 220. 14375, 2022.
10

---

Published as a conference paper at ICLR 2024
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, 2021. URL https::tp//ccc /ttp:///////eeeeeeee ngs neuri meurip..cc.. wwtpiiss.cccppppeeeee cc/paper/////ppccc 2021 0021/hash//
Aosillacbbbatattto htmm.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:221..0968, 2022.
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, al. Tutel: Adaptive mixtureoof-expert scale. arXiv preprint 2rXiv:2206.03382 2022.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasumuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering, text classification, and regression via span extraction. arXiv preprint arXiv:1904.09286, 2019.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system. arXiv preprint arXiv:2005.00700 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of- experts from dense checkpoints. arXiv preprint arXiv:2212.05055. 2022.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixtureoof-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, 2021.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16663 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104. 000.0000..1...... 2021.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE lay- ers: Simplifying training of large, sparse models. In ICML. PMLR, 2021. URL http:
//proceedings.m.. mir press petres/vv39//e http////1///eaaa.lbb.ammm
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 230v:2330408488 2023.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv: 15504, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In ICML, 2023.
11

---

Published as a conference paper at ICLR 2024
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Cross-token modeling with conditional computation. arXiv preprint arXiv:2109.02 00000, 2021.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships multi-gate gate mixture-of-expert. In Proceedings of the 24th ACM SIGKDD International Conference
in multi task learning with multi
Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. ACM, 2018. doi: 10. 1 01144/221889932200000 URL https: / .rg// 10. 10..115// /01115/2111 321 981 9. 3220007.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1oXiv:8806.00 2018.
Shen- Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, 2020.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross- Cross-task generalization via natural language crowdsoureing instructions. arXiv preprint arXiv:2104.08773, 2021.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin -in Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint 2rXiv:211..0188. 2022a.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint
2022b.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.02777 022770 2022.
Reichhroo Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assistee question-answeringg with human feedback. arXiv preprint
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, 2022a.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022b.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.0719,, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 11:140:1-140:67, 2020. URL http:://mir.orr// papers//22120-072
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2024
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Advances in Neural Information Processing Systems 34: Annual Con- ference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 2021. URL.ttts:: /ttp:e//o/cooeeoooooooooo wew.eeiinnnnnnnnnnnnnereeee neurips...... ccc htte/ppprr2221111 /hashhhh htmll
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler. Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICIR, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint ar arXiv:2211.05100 2022.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.
arXiv preprint arXiv: 04045,4355 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR. OpenReview.net, 2017. URL https:ttp//./...tt /oop/rppeerreeeeeeee eew net forum? di BickMDqlg.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrlll, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts, arXiv preprint arXiv:2303.07226, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Marriga-Alonsee et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 0:X11:2206.04615, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether ofain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6998999 6008, 2017. URL http::/papeet nips oc httppaapr//// 71181 www.aaattttt..... is all aawaaalllllleeeee need
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint axXiv:2212.10566 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super- naturalinstruccions:: Generalization via declarative instructions on 1600+ nlp tasks. In EMNLP, 2022b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2022a.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NewrIPS, 2022b.
13

---

Published as a conference paper at ICLR 2024
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. arXiv preprint arXiv:2305.11206, 2023.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew M Dai, Zhifeng Chen, Quoe V Le, and James Laudon. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems, 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906 08906, 2022. doi: 10.48500ARXXV:222.00000 URL https: /ttp:i/aaxiivoooo.a org/ abs/ /bbs/220..088 08906.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint 10.42:21 2021.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RESULTS
A.1
LEARNING EFFICIENCY
We present a detailed learning efficiency experiment in Figure 7 across number of steps. It shows that MoE starts outperform Dense counterparts right after 25k steps with instruction tuning.
A.2 HYPERPARAMETER SENSITIVITY
Following ST-MoE Zoph et al. (2022), we further experiment with expert dropout (0.0, 0.1, 0.5), varying the learning rate (le-4, (1e-,,5-- -e- 1) and batch size (16, 32, 64) to examine the hyperpa- rameter sensitivity of FLAN-MOE. We found that the performance varies in different tasks but not significantly with all the hyperparameters. but lower learning rate and small batch size lead to more stable instruction finetuning process of the model at extra-large scales.
Noted that we conduct instruct-tuning for 100k steps following (Chung et al., 2022). The instruction- tuned models introduce 10% of the pre-training cost, or 10x of the single-task finetuning cost regardless of Dense or MoE models.
A.3 DECODER-ONLY MOE
We perform further ablation on the effects of instruction tuning on decoder-only MoE models (Du et al., 2022) as shown in Figure 8 at xl scale. can be seen that decoder-only model benefits more from instruction tuning. which shows the potential of BLAN-MoBE at more generalized architecture and objective setting. We leave the study of scaling decoder-only FLAN-MoB to future works.
A.4 LANGUAGE MODEL ADAPTATION
Another possible effect of why instruction-tuine could be effective is because the additional steps of language model objective pretraining, which previous studies Lester al. (2021) found could make T5 more adept at handling task-specific challenges. In Figure 9, we ablate this factor and show detailed analyses regarding token dropping ratio. We can see that the scalabity of MoE largely improved after Im adaptation, but the gaps persists from Dense couterparts. Also, the large token dropping rate presented using common capacity factor during Oshot evaluation is improved after Im adapation. After instruction-tuning we can see the token dropping behavior is much similar to which shown the pretraining. Scalable generalization gains on Oshot MMLU can promised here as well. We also try to increase the capacity factor to 64 which alleviates the token dropping ratio but yields worse performance even compared to activating two experts in our default experiment setting. We attribute this to the significant discrepancy of capacity factor in pretraining and evaluation, which may
MMLU Few ShatShat
BBH Direct Few Few-Shot
50
35
Avg. Held-Out Score
Avg. Held-Out Out Score
40
30
30
25
20
20
15
10
50
100
150
200
50
100
150
200
Number of Steps (k)
Number Steps (k)
Figure 7: Learning efficiency comparison. Average zero-shot, and few-shot performance of FL.AN- SWITCH models versus FLAN-T5 dense models more tokens processed during training FLAN Tasks. (the colors blue, orange, and green correspond to small, base, and large models respectively, while the shapes dots represent Dense models and triangles represent MoE models.)
15

---

Published as a conference paper at ICLR 2024
Few-shot Eval
50
15
Flan-Ts
45
Switch
Flan Switch GLaM
GLaM
Metrics (%6
35
30
25
15
10
MMLU
BBH
Figure 8: Impact of Instruction Tuning on Decoder-only MoEs.
Taken Dropping Ratio
MMLU-Direct 0Shot
MML--iiret 0Shot
16
50
50
14
45
Score
10
35
10
10a
Gflops Per Token Prediction
GFlops Per Token Prediction
GFlops Per Token Prediction
(a) Token Dropping Ratio
(b) Oshot after LM Adaptation
(c) Oshot after Pretrain
Figure 9: Effect of Language Model Adaptation and Instruction Tuning on token dropping ratio and Oshot MMLU.
cause the behavior of MoE deviating drastically. We leave the further study towards understanding the effect of token dropping and evaluation performance to future works.
Qualitatively, we noted that post-instruction tuning, the MoE models exhibited reduced tendency to drop formatting tokens (such as "In" and certain stop words). This change is crucial, particularly for multiple-choice questions and other evaluation benchmarks employed in our study.
A.5 INFERENCE AND TRAINING OVERHEAD OF MOE
We've conducted comparative analysis of disk memory, GPU memory and throughput under optimal batch sizes on 16 A100 DGX, using different engineering techniques and public libraries. Our analysis shows that disk memory scales linearly with the number of experts in MoE models. However, employing a proper parallelism strategy. like expert parallelism, can substantially reduce GPU memory usage. For instance, 16-expert model using expert parallelism maintains the same GPU memory footprint as a dense model but can achieve a 28% increase in throughput. This can be further optimized, reducing the difference 13% with optimizations outlined in (Hwang et al., 2022; Rasley et al., 2020). It's also worth noting that when GPUs are limited, inference costs may increase due to less efficient data locality, as each GPU processes more data for expert parameters. We plan to expand on this discussion in the final version, benchmarking additional model variants. Regarding training, we utilize 4x8x8 TPU Pods and internal infrastructure with carefully annotated tensor, model, and expert parallelism strategy. The overall overhead in step time for 128 expert MoE models can be kept within range of 13%-27% depending on the base model size to dense counterparts, when optimal batch sizes used. In summary, while FLOPs for MoE and dense models comparable, throughput and GPr memory can also be similar with appropriate optimization and batch size. However, the increase in disk memory usage is unavoidable cost factor.
16

---

Published as a conference paper at ICLR 2024
B SUBTASK RESULTS IN EACH BENCHMARK
B.1 MMLU
In the case of five-shot MMLU, we employ the "dev" as the small sample exemplars. The performance of individual tasks in MMLU on the "validation" set is detailed in this section (refer https: / /wwww for more information). Please note, all MMLU findings
conssts/oommnniii talitty_aasaaa catmuuuttt_attlllooooeoo catalog/
buggingface wwwpee/hendccsssseeeeeeeeee presented in this paper correspond to the "validation" set. We employ the prompts in Chung et al. (2022).
Table 3: MMI.U[:00] individual task performance.
MMLU
Abdract Algobri
Asatomy
Astronemyy
Business Fities
Clinical Knowlodge
College Biology
College Chemiety
College Camp,
College Math
Collage Medicine
Model
Dives
CaT
Dinas
CuT
Diras
CaT
Dinest
CuT
Dinect
CUT
Direct
CuT
Direct
OUT
Direct
CoT
Daett
CoT
Dixat
Col
davinci
27.3
27.3
50.0
42.9
25.0
31.2
45.5
36
31.0
34.5
43.8
25.0
12.5
18.2
36.4
27.3
9.1
36.4
31.8
IeNi-davinci-002
9.1
27.3
57.1
28.6
62.5
56.2
63.6
72.7
51.7
55.2
68.8
43,8
12.5
37.5
636
36.4
54.5
164
636
54.5
test-davinci-Oor
18.2
36.4
50.0
57.1
62.5
62.5
63.6
63.6
62.1
65.5
62.5
81.2
25.0
28.0
54.5
45.5
8188
72.7
72.7
68.2
code davinci-O00 002
18.2
27.3
71.4
35.7
68.8
56.2
54.5
63.6
69.0
65.5
62.5
50.0
25.0
45.5
27.3
72.7
45.5
77.3
86.4
80M
T5-Small
18.3
31.2
0.0
27.6
18.8
D.0
37.5
72.7
0.0
27.3
18.2
0.0
Slan-IS-Small
27.3
9.1
42.9
7.1
18.8
6.2
18.2
27.3
34.5
20.7
31.2
18.8
12.5
0.0
18.2
0.0
36.4
9.1
50.0
18.2
250M TS. B5-B
18.2
18.2
28.6
0.0
37.5
12.5
45.5
0,0
34.5
6.9
18.8
6.2
62.5
25.0
45.5
9.1
18.2
18.2
18.2
18.2
Flan TS Base
18.2
18.2
42.9
35.7
37.5
37,5
36.4
36.4
34,5
27.6
37,5
18.8
12.5
25.0
27.3
36.4
18.2
0.0
40.9
22.7
780M
T5-Large
18.2
0.0
21.4
0.0
25.0
18.8
45.5
9.1
6.9
10.3
18.8
0.0
37.5
37.5
45.5
18.2
18.2
9.1
18.2
9.1
Fan--r--Large
18.2
27.3
35.7
28.6
37.5
31.2
36.4
45.5
44.8
37.9
43.8
43.8
25.0
12.5
27.3
36.4
45.5
27.3
45.5
45.5
3B TS-XI.
18.2
0.0
14.3
0.0
31.2
0.0
9.1
0100
10.3
17.2
31.2
12.5
25.0
12.5
45.5
0.0
9.1
9.1
18.2
0.0
Flan-T5-XI.
25.3
36.4
35.7
50.0
62.5
45.5
45.5
55.2
55.2
56.2
50.0
25.0
45.5
27.3
50.0
50.0
11B
TS-XXL
27.3
0.0
21.4
0.0
31.2
0.0
0.0
10.3
31.0
43.8
0.0
50.0
36.4
0.0
9.1
54.5
0.0
Ran-T5-XXL
36.1
54.5
56.2
36.4
36.4
9.1
Pla-PaLM
36.4
18.2
42.9
35.7
43.8
50.0
36.4
45.5
48.3
41.4
56.2
50.0
25.0
25.0
54.5
63.6
18.2
27.3
50.0
18:2
62B
PaLM
27.3
9.1
50.0
21.4
50.0
43.8
63.6
81.8
51.7
62.1
68.8
31,2
37.5
25.0
$1.5
18.2
36.4
9.1
59.1
45.5
Ran-PaLM
18.2
18.2
57.1
42.9
68.8
68.8
63.6
54.5
51.7
55.2
68.8
75.0
12.5
37.5
54.5
27.3
36.4
45.5
81.8
63.6
540R
Pal
27.3
18.2
78.6
42.9
68.8
81.2
63.6
72.7
72.4
75.9
87.5
62.5
50.0
25.0
54.5
36.4
36.4
27.3
77.3
77.3
Plan-Pal.M
0.0
9.1
50.0
71.4
81.2
75.0
63.6
54.5
79.3
62.1
87.5
62.5
62.5
62.5
81.8
63.6
36.4
63.6
86.4
86.4
250M
Switches
9.1
18.2
14.3
21
43.8
36.4
0.0
10.3
37.5
37.5
37.5
36.4
0.0
36.4
18.2
40.9
0.0
Switdhwit
50.0
43.8
36.4
6.2
36.4
36.4
22.7
Switch sner
27.3
21.4
26.1
0.0
45.5
FLAN Swildhs
18.2
18.2
35.7
35.7
37.5
25.0
36.4
45.5
48.3
41.4
43.8
37.5
12.5
37.5
45.5
36.1
27.3
9.1
54.5
50.0
11B Switchar
18.2
0.0
50.0
18.8
6.2
45.5
0.00
10.3
6.9
18.8
6.2
37.5
12.5
45.5
18.2
36.4
18.2
9.1
22.7
FLAV-Switchye
45.5
9:1
42.9
42.9
56.2
56.2
54.5
45.5
55.2
44.8
68.8
56.2
0.0
12.5
45.5
27.3
36.4
27.3
54.5
36.4
80M
FLAN-OSWAALL
18.2
18.2
35.7
35.7
12.5
18.8
27.3
9.1
31.0
34.5
25.0
12.5
25.0
12.5
36.4
9.1
9.1
18.2
50.0
27.3
FLAN-GSDASt
18.2
35.7
50.0
63.6
18.8
36.4
27.3
50.0
45.5
FLAN-GS
35.7
56.2
27.5
51.7
43.8
36.4
59.1
50.0
80M
ECANLE
18.2
9.1
35.7
28.6
31.2
18.8
36.4
18.2
34.5
31.0
31.2
12.5
37.5
0.0
54.5
0.0
18.2
18.2
40.9
22.7
250M BLA ECA
27.3
18.2
5000
42.9
43.8
37.5
27.3
45.5
48.3
24.1
37.5
43,8
0.0
12.5
45.5
36.4
27.3
18.2
36.4
31.8
9.1
35.7
28.6
50.0
63.6
50.0
36.4
36.4
45.5
PLAN-ECa
17.7
37.0
44.0
58.1
37.7
45.0
36.4
17.7
58.6
49.6
250M
STaNs
18.2
18.2
21.4
31.2
12.5
45.5
45.5
10.3
6.9
12.5
37.5
25.0
37.5
45.5
45.5
36.4
18.2
18.2
9.1
STA
11.5
9.1
45.3
28.6
21.1
31,2
47.9
36.4
47.2
31.0
27.4
37.5
52.4
25.0
56.9
18.2
20.6
18.2
56.9
22.7
32B
STuR
27.3
0.0
35.7
0.0
37.5
18.8
18.2
18.2
27.6
6.9
12.5
25.0
37.5
25.0
18.2
9.1
18.2
0.0
13.6
18.2
SLAN-ST:aD
18.2
18.2
50.0
714
68.8
81.2
72.7
81,8
79.3
65.5
87,5
68.8
25.0
25.0
51.5
9.1
18.2
18.2
68.2
72.7
17

---

Published as a conference paper at ICLR 2024
Table 4: MMLU[10:20] individual task performance.
MMLUU
College Phynics
Computer Security
Conceprual physics
Foonometics
Elecrrical Engineering
Elementary Mathematics
Dormal Logic
Global Facts
Hlgg School
Ihology
High School
Chemistry
Model
Dinazi
CaT
Dimat
CaT
Diva
CaT
Diectt
Cal
Direct
CiT
Dinact
CaT
Direct
CiT
Dinect
CiT
Dinat
CiT
Dinani
CaTl
davinci
45.5
36.
72.7
54.5
38.5
46.2
25.0
33.3
25.0
50.0
24.4
29.3
14.3
14.3
20.0
20.0
28.1
34.4
31.8
13.6
test davinci 002
$4.5
818
$18
$1.8
53.8
61.5
58.3
50.0
50.0
37.5
$6.1
73.2
7.1
28.6
50.0
70.0
71.9
71.9
18.2
36.4
davinci 003
36.4
45.5
818
63.6
42.3
57.7
58,3
58.3
50.0
56.2
48.8
75.6
42.9
40.0
50.0
71.9
75.0
36.4
36.4
code davie--vvinii-000 002
80M T5-Small
18.2
18.2
38
4,9
20.0
0.0
0.0
Flan-T--Small
36.
9.1
26.9
30.8
16.7
29.3
17.1
50.0
20.0
36.
22.7
26.9
20.0
20.0
27.3
Flan-TS-Bar Base
72.7
45.5
27.3
27.3
19.2
26.9
41.7
33.3
25.0
37.5
26.8
14.6
28.6
42.9
40.0
20.0
37.5
28.1
45.5
31.8
780M
T5-Large
18.2
18:2
18.2
18.2
26.9
23.1
25.0
0.0
37.5
12.5
29.3
19.5
7.1
0.0
0.0
20.0
9.6
40.9
9.1
Laangge-Large
$1.5
36.4
$4.5
54.5
26.9
23.1
16.7
16.7
37,5
37,5
36.6
17.1
42.9
35.7
40.0
20.0
40.6
25.0
27.3
27.3
38
TS-XL
18.2
9.1
9.1
18.2
19.2
23.1
41.7
0.0
37.5
25.0
30.0
17.1
42.9
0.0
30.0
10.0
$1.2
0.0
27.3
4.5
Plan-TS-XI.
72.7
36.4
36.4
36.4
38.5
46.2
33.3
16.7
56.2
25.0
54,1
24,4
28.6
14.3
20.0
30.0
57.5
34.4
31.8
36.4
11B
TS-XXL
18.2
18.2
27.3
45.5
23.1
34.6
16.7
0.0
31.2
25.0
26.8
19.5
42.9
0.0
20.0
10.0
15.6
0.0
31.8
0.0
XXX
$4.5
27.3
27.3
34.6
42.3
25.0
16.7
43.8
48.8
36.6
35.7
30.0
40.0
53.1
31.8
40.9
8B
PalM
18.2
36.4
36.4
27.3
26.9
30.8
16.7
33.3
12.5
18.8
24.4
24.4
14.3
0.0
30.0
20.0
15.6
21.9
18.2
22.7
Flan-PaLM
45.5
27.3
72.7
38.5
38.5
33.3
25.0
34.1
34.1
30.0
20.0
18.2
18.2
62B
PaLM
$4.5
45.5
63.6
42.3
42.3
16.7
33.3
62.5
56.2
24.4
51,2
21.4
30.0
40.0
59.4
31.2
36.4
31.
40.9
540B
36.4
Flan-PaLM
63.6
72.7
90.9
81.8
69.2
65.4
66.7
58.3
75.0
58.5
70.7
60.0
70.0
71.9
68.2
40.9
250M
Switchaee
9.1
9.1
18.2
9.1
23
26.9
16.7
0.0
43.8
50.0
26.8
17.1
28.6
0.0
30.0
10.0
12.5
25.0
51.8
0.0
FI Switthhee
56.4
36.4
27.3
18.2
42.3
42.3
16.7
25.0
31.2
31,2
9.8
31.7
35.7
7.1
30.0
20.0
25.0
18.8
22.7
18.2
790M
Switch
27.3
36.4
36.4
18.2
30.8
26.9
25.0
25.0
18.8
0.9
26.8
24,4
7.1
28.6
30.0
10.0
37.5
25.0
22.7
36.4
FLAN- Switchseen
63.6
45.5
45.5
36.4
42.3
26.9
41.7
25.0
37.5
31.2
43.9
19.5
35.7
42.9
20.0
30.0
40.5
27.3
13.6
11B
Switchuu
9.1
9.1
18.2
26.9
19.2
25.0
0.0
31.2
31.2
22.0
14.6
21.4
14.3
10.0
0.0
21.9
36.4
9.1
Switcheen.
FLAN-GS
11.5
40.0
40.0
250M
PLAN-GS
63.6
45.5
18.2
27.3
23.1
23.1
41.7
33.3
18.8
25.0
22.0
14.6
35.7
35.7
40.0
40.0
25.0
18.8
13.6
27.3
790M
FLAN-GS,
$4.5
45.5
45.5
36.4
30.8
35.5
41.7
50.0
43.8
50.0
29.3
34.1
50.0
14.3
40.0
20.0
50.0
43.8
18.2
18.2
80M
FLAN BC
72.7
27.3
63.6
27.3
26.9
15.4
25.0
16.7
17.1
31.7
30.0
40.0
34,4
12.5
31.8
40.9
EC a
780M
FLAN-EC, LA
36.4
45.5
36.4
462
34.6
33.3
33.3
37.5
31.2
36.6
36.6
35.7
14.3
30.0
40.0
53.1
5000
27.3
22.7
38
FLAN-BCS.
54.0
47.3
35.9
37.4
41.8
26.3
41.2
24.3
37.0
30.9
50.7
20.7
13.8
43.1
49.5
31.0
$2.6
45.0
17.7
144
250M
ST
9.1
45.5
18.2
18.2
26.9
154
25.0
0.0
31.2
25.0
14.6
26.8
35.7
14.3
10.0
10.0
21.9
6.2
40.9
27.3
FTAN-ST..oM
47.9
38.5
44.1
34.1
16.0
409
328 STiza
$4.5
0.0
27.3
27.
23.1
42.3
41.7
0.0
24.4
12.2
50.0
20.0
12.5
13.6
22.7
PLAN-STYZ
57.7
51.2
40.0
40.9
18

---

Published as a conference paper at ICLR 2024
Table 5: MMLU[20:30] individual task performance.
Schrol Pampon liieory
Hligh Marnasnnmmio
High Math
High
Cimp. So.
Gongraphy
Gome Poliiics
Micsezzoomizs
Prycios
Poycholegy
Statiatie
Model
Disci
Cul
Diruzi
Call
Direcl
Col
Direct
CuT
Direct
CuTl
Dinat
OfT
Dreai
CuT
Disci
Col
Dinzi
OiT
Dinas
Col
davinci
dast-davinci-lll
77.3
74.4
73.1
test
95.5
77.3
80.8
23.5
52.2
colo-davin--0002
83.9
77.8
90.9
86.4
67.4
67.4
51.7
83.5
80.8
23.5
29.4
95.0
65.2
80M
Small
22.2
33.3
0.0
16.4
0.0
28.6
33.3
25.6
4.7
13.8
13.8
34.5
3.8
35,3
0.0
25.0
34.8
17.4
Flan T5-Small
222
0.0
27.3
18.2
38.1
4,8
32.6
7.0
10.3
26.9
7.7
47.1
118
28.3
0.0
250M
TS-Buse
33.3
810
27.8
0.0
4.5
13.6
38.1
$2.4
27.9
25.3
17.2
13.8
23.1
25.1
17.6
23.5
20.0
11.7
34.8
34,8
Flan-T5-Ra Raae
44.4
22.2
5000
55.6
50.0
50.0
66. 7
47.6
25.3
92.6
13.8
17.2
42.3
38.5
11.8
17.6
300
18.3
10.4
17.4
26.9
Large
36.9
57.7
38
TS-XI
22.2
0.00
33.3
5.6
27.3
31.8
23.8
52.4
30.2
32.6
20.7
3.4
26.9
15.4
17.6
17.6
15.0
15.0
34.8
13.0
Plan-T5 TS-XL
66.7
33.3
77.8
77.8
63.6
63.6
71.4
47.6
34.9
46.5
24.1
13.8
46.2
53.8
17.6
29.4
78.3
63.3
43.5
26.1
TS-XXL.
11.1
38.9
0.0
23.7
40.9
57.1
30.2
37.3
27.6
344
26.9
42.3
17.6
17.6
38.3
21.3
4.3
Plan T5 XXL
64.4
55.6
72.2
72.2
72.7
68.2
81.0
66.7
44.2
39.5
34.5
27.5
50.0
26.9
17.6
17.6
86.7
78.3
34.8
34.8
8R
PaLM
16.4
27.3
18.6
24.1
26.1
Flan-PuLM
45.5
44.2
55.3
62B
PaLM Flan PALM
66.7
66.7
61.1
55.6
63.6
72.7
47.6
57.1
41.9
51.2
27.6
34.5
57.7
65.4
29.4
17.6
83.3
75.0
47.8
52.2
55.6
55.6
38.9
72.2
S18
77.3
76.2
71.4
58.1
60.5
17.2
34.5
09.2
69.2
23.5
29.4
88.3
85.0
52.2
30.4
540B
PALM
100.0
88.9
77.8
00.9
81.0
R1.4
74.4
06.2
76.8
Flan-PalMM
100.0
72.2
95.5
90.9
05.2
85.7
79.1
72.8
44.8
100,0
88.5
5.9
29.4
93.3
47.8
250M
Switchaer Soit
28.6
11.6
59.1
68.2
42.9
92.6
6.9
57.7
42.3
29.4
20.4
60.0
99.1
780M
Switch
22.2
33.3
27.8
16.7
27.5
18.2
9.5
33.3
25.6
30.2
10.3
24.1
34.5
38.5
41,2
17.6
21.7
15.0
26.1
54.5
23.5
11B
Swithen FLAN-Switchse
44.4
0.0
27.8
27.8
18.2
27.3
$2.4
4.8
20.9
16.3
41A
0.0
23,
0.0
17.6
5.9
15.0
13.3
43.5
26.1
55.6
44.4
72.2
72.2
72.7
81.8
85.7
76.2
62.8
46.8
34.5
20.7
53.8
$3.8
23.5
20.4
85.0
78.3
19.1
34.8
50M
PLAN-GSensn
22.2
0.0
33.3
15.7
50.0
27.3
38.1
23.8
50.2
27.9
24.1
10.3
23.1
36.6
23.5
41.2
38.3
28.3
21.7
30.4
250M
FLAN FLAN-GSESer
24.4
11:1
50.0
38.9
50.0
54.5
92.4
36.
34.0
25.3
20.7
17.2
46.2
15.
58.8
17.6
46.7
350
10.1
34.8
780M
PLAN GRann
44.4
22.2
61.1
27.8
72.7
59.1
$1.0
76.2
41.9
32.6
27.6
31.0
61.5
$0.0
29.4
41.2
80.0
66.7
30.4
34.8
60M
FLAN-BCessn
44.4
11:1
33.3
22.2
45.5
36.4
42.9
38.1
30.2
18.6
27.6
13.8
19.2
15.4
23.5
23.5
46.7
30.0
19.1
21.7
250M
FLAN BE
44.4
22.3
61.1
22.2
63.6
90.1
57.1
42.9
44.2
97.3
11.0
31.0
50.0
42.1
20.4
17.6
63.3
56.7
26.1
30.4
7W0M
FLAN DC
66.7
61.4
61.1
22.2
77.3
96.4
57.1
$7.1
37.2
37.2
27.6
27.5
50.0
55.8
41.2
17.6
83.3
75.0
30.4
30.4
30
PLAN DC2.
55.1
57.7
71.7
29.4
$1.3
$3.9
80.5
62.2
55.3
47.4
20.2
14,9
64.9
47.5
17.1
23.7
91.
56.5
38.6
40.6
250M
STrS
33.3
0.0
33.3
11.1
18.2
0.0
47.6
25.6
18.6
30.2
46.8
24.1
19.2
0.0
29.4
17.6
15.0
233
26.1
3488
PLAN PLAN-STUAR
58.0
33.3
63.5
55.6
61.5
36.4
54.8
57.1
32.6
27.9
30.0
31.0
60.1
46.2
31.8
55.3
64.1
51.7
24.1
39.1
ST
2788
23.8
28.6
32.6
23.3
23,1
15.4
23.5
118
26.7
17.4
PLAN-STOR
66.7
7788
77.8
95.5
81.8
95.2
90.5
76.7
69.8
37.9
41.4
76.9
76.9
17.6
118
95.0
86.7
60.9
19

---

Published as a conference paper at ICLR 2024
Table 6: MMLU[30:40] individual task performance.
MMLU
Schhol
US Histrey
School
World History
High School
Human Aging
Human Secuality
Intemntionna Law
Barisprdance
Logical Fallacios
Machine Learsing
Mansgement
Marketting
Model
Direct
Coll
Direct
Col
Dirrer
Col
Direct
Coll
Direct
Coll
Direct
CrT
Direct
CoT
Direct
CoT
Direct
Coll
Direct
Corl
duvinet
30
60.9
84-6
38.5
18.2
9.1
55.6
50.0
27.3
56.0
64.0
text-davinci-002
86.4
72.7
69.2
73.1
78.3
87.0
66.7
58.3
92.3
84.6
63.6
45.3
77.8
66.7
45.5
36.4
72.7
22.7
80.0
80.0
leat-davins--00
81.8
81.8
80.8
76.9
78.3
73.9
66.7
58.3
84.6
84.6
63.6
$1.5
83.3
83.3
45.5
$4.5
81.8
72.7
84.0
76.0
oode-davinci-002
100.0
77.3
76.9
84.6
78.3
78.3
75.0
58.3
100.0
92.3
63.6
72.7
83.3
72.2
54.5
63.6
90.9
81.8
80.0
80.0
S0M TS Small
40,9
0.0
30.8
0.0
34.8
13.0
41.7
25.0
30.8
0.0
27.3
27.3
33.3
0.0
27.3
0.0
18.2
9.1
24.0
4.0
Plan- Small
50.0
31.8
15.4
7.7
4.3
13.0
33.3
16.7
23.1
7.7
27.3
9.1
22.2
16.7
18.2
0.0
18.2
9.1
41.0
20.0
BS-Base 250M
18.2
0.0
30,8
0.0
30.4
30.4
33.3
25.0
7.7
7.7
27.3
18.2
33.3
27.8
36.4
27.3
18.2
0.0
20.0
24.0
Plan IS Base
59.1
50.0
50.0
50.0
30.4
30.4
50.0
33.3
38.5
46.2
18.2
44.4
66.7
18.2
36.4
36.4
18.2
64.0
60.0
780M TS-Large Plan- Plaage--Laree
13.6
0.0
30.8
0.0
47.8
39.1
41.7
41.7
7.7
D.D
18.2
0.0
33.3
22.2
36.4
9.1
18.2
27.3
20.0
16.0
54.5
51.7
42.3
52.2
56.5
41.7
53.8
30.8
45.5
77.8
55.6
18.2
84.0
68.0
3B
TS-XI.
18.2
0.0
30.8
7.7
21.7
30.4
41.7
33.3
7.7
30.8
27.3
9,1
27.8
27.8
27.3
0.01
18.2
27.3
28.0
20.0
Flan-TS-XI.
56.5
47.8
75.0
54.5
66.7
45.5
84.0
84.0
11B
TS-XXL
22.7
34.6
0.0
8.7
43.5
25.0
25.0
46.2
0.0
27.3
9.1
22.2
44.4
9.1
0.01
54.5
45.5
20.0
60.0
XIXI
77.8
76.0
SB
PaLM
36.4 12.7
31.8
15.4
23.1
47.8
34.8
16.7
16.7
53.8
46.2
27.3
9.1
16.7
22.2
18.2
18.2
18.2
36.4
32.0
24.0
Plan-PaLM
52.2
56.5
66.7
38.5
72.7
61.1
72.2
45.5
45.5
72.0
68.0
62B
PaLM
27.3
40.9
51.7
38.5
69.6
65.2
58.3
25.0
76.9
61.5
45.5
27.3
61.1
66.7
45.5
81.8
84.0
80.0
Flan-PalM
63.6
27.3
72.0
72.0
540B
PaLM
78.3
73.9
91.7
61.5
63.6
83.3
66.7
27.3
84.0
84.0
Switchaee
7.7
27.8
32.0
8.0
FLAN- Switch
50.0
36.4
46.2
19.2
47.8
47.8
25.0
25.0
46.2
30.8
36.4
18.2
55.6
50.0
18.2
45.5
45.5
54.5
68.0
56.0
Switchance
11.5
21.7
30.4
27.3
22.2
27.8
27.3
27.3
32.0
16.0
PLAN-Swilch,.ser
59.1
36.4
42.3
50.0
47.8
60.9
41.7
33.3
61.5
53.8
45.5
45.5
66.7
50.0
9.1
18.2
72.7
72.7
80.0
76.0
11B Switchan
30.8
30.8
26.9
26.1
8.7
16.7
7.7
0.0
27.3
27.8
22.2
27.3
18.2
27.3
20.0
0.0
PLAN- PLAN-Switchee
68.2
59.11
65.44
61.5
52.2
69.6
66.7
41.7
100.0
76.9
27.3
27.3
77.8
66.7
36.4
36.4
63.6
72.7
92.0
80.0
SUM
FLAN- Goeenrs
50.0
27.3
38.5
19.2
30.4
30.4
16.7
25.0
30.8
30.8
27.3
18.2
38.9
33.3
45.5
9.1
36.4
18.2
64.0
40.0
250M
FLAN-GS: Grast
54.5
36.4
57.7
34.6
34.8
34,8
66.7
66.7
46.2
46.2
36.4
61.1
61.1
9.1
27.3
36.1
15.5
61.0
52.0
snce
PLAN-BCouaL
21.7
26.1
64.0
250M
FLAN-EC
72.7
27.3
57.7
26.9
52.2
43.5
25.0
41.7
53.8
45.5
36.4
77.8
61.1
18.2
18.2
18:2
76.0
48.0
790M
FLAN-BCLSSEE
68.2
45.5
65.4
38.5
56.5
60.9
41.7
50.0
61.5
23.1
36.4
18.2
66.7
55.6
36.4
18.2
72.7
72.7
80.0
68.0
3B
BLAN-ECA.
76.8
38.4
61.0
50.7
73.4
60.9
66.2
35.2
68.7
53.7
25.0
47.1
71.7
51.9
26.8
19.7
12.2
73.1
95.5
78.1
250M ST
13.6
31.8
30,8
19.2
26.1
13.0
41.7
7.7
0.0
27.3
0.0
27.8
22.2
27.3
18.2
18.2
15.5
21.0
0.0
FLAN-STAAE
32B
ST
31.8
26.9
11.5
34.8
13.0
33.3
25.0
15.4
27.3
22.2
22.2
27.3
27.3
54.5
18.2
12.0
16.0
73.9
72.7
83.3
77.8
20

---

Published as a conference paper at ICLR 2024
Table 7: MMLU[40:50] individual task performance.
MMLAI
Medical Geneties
Misc.
Moral Disputes
Moral Socmutio
Nutitionn
Philosophy
Prehistccy
Professional Accouating
Prodessional Law
Professional Medicine
Model
Direct
Direct
Col
Direct
CoT
Direct
CoT
Direct
CoT
Direct
Col
Direct
Coll
Direct
CoT
Direct
Direct
CoT
davinci
72.7
90.9
50.0
65.1
$7.9
39,5
24.0
34,0
$4.5
45.5
44.1
61.8
45.7
42.9
29.0
35,5
31.2
26.5
32.3
38.7
00t--bavinc--00
90.9
90.9
79.1
81.4
632
6588
46.0
40..
75.8
69.7
67.6
67.6
60.0
65.7
64.5
41,9
45.3
38.8
64.5
71.0
1e0i-davinsi-000
1000
52.6
76.5
76.5
38.7
74.2
67.7
davinci-002
10000
50.0
79.4
76.5
51.6
TS-Small
6.5
Flan TS Small
18.2
9.1
34.9
19.8
21.1
5.3
23.0
19.0
33.3
12.1
26.5
11.8
42.9
20.0
32.3
22.6
32.4
14.1
12.9
16.1
BS-Base
20.6
35.5
3.2
Flan-TS Base
27.3
54.5
36.0
29.1
34.2
42.1
24.0
21.0
39
33.3
35.3
35.3
45.7
28.6
19.4
35.5
27.6
23.5
22.6
25.8
780M T5-Large
27.3
26.7
29.1
15.8
0.0
24.0
14.0
33.3
0.0
23.5
23.5
17.1
11.4
32.3
12.9
23.5
29.0
0.0
Laagge
45.5
72.7
47.7
51.2
50.0
39.5
24.0
27.0
45.5
42.4
52.9
$2.9
15.7
40.0
35.5
19.4
32.4
30.0
41.9
29.0
38
TS-XL
18.2
0.0
27.9
24.4
15.8
7.9
24.0
27.0
33.3
9.1
17.6
29.4
20.0
8.6
22.6
6.5
23.5
1.2
32.3
0.0
Plan-TS-XI.
72.7
72.7
60.5
61.6
42.1
94.2
33.0
18.0
60.6
$4.5
55.9
52.9
45.7
51.4
25.8
41.9
57.1
27.6
48.4
45.2
11B
TS-XXL
18.2
36.4
34.9
43.0
18.4
7.9
31.0
0.0
30.3
24.2
23.5
44.1
17.1
45.7
16.1
22.6
23.5
0.0
29.0
0.0
XXL
90.9
72.7
62.8
68.6
39.5
37.0
32.0
42.4
61.8
64.7
41.9
38.7
35.9
32.9
58.1
51.6
8B
PaLM
54.5
27.3
30.2
32.6
34.2
39.5
22.0
23.0
21.2
15.2
26.5
26.5
28.6
28.6
12.3
25.8
25.9
22.9
9.7
19.4
Flan-PaLM
63.6
68.6
39.5
36.8
25.0
33.3
61.8
35.5
45.2
51.6
35.5
62B
PaLM
10000
100.0
68.6
63.2
57.9
31.0
41.0
72.7
60.6
61.8
61.8
45.2
29.0
40.0
26.5
64.5
58.1
71.0
540B
61.5
Flan-PaLM
90.9
100.0
83.7
76.3
71.1
54.0
71.0
75.8
79.4
79.4
64.5
61.3
90.3
77.4
250M
Switchuur
45.
18.2
25.6
17.4
7.9
2.6
24.0
5.01
30.3
27.3
29
8.8
11.4
28.6
19.4
0.01
24.1
0.0
35.5
0.0
FL Swithhiit
364
45.5
41.9
47.7
36.8
94.2
32.0
53.0
48.5
27.3
38.2
79.4
40.0
31.4
19.4
32.3
26.5
17.1
29.0
38.7
790M
Switchusee
0.0
9.1
27.9
21.1
26.3
21.1
22.0
20.0
21.2
21.2
29.1
11.8
18.6
22.9
32.3
32.3
27.6
4.1
16.1
19.1
FLAN-SwitchLueds
$4.5
54.5
53.5
59.3
47.4
28.9
24.0
23.0
60.6
50.3
41.2
35.3
42.9
60.0
38.7
25.8
36.5
25.3
51.6
38.7
11B
Switchaa Swithhw
364
27.3
22.1
26.7
18.4
0100
21.0
24.0
15.2
15.2
35.3
3A.2
20.0
25.7
32.3
29.0
25.3
22.9
19.4
25.8
50.0
27.0
60.6
61.8
58.8
35.5
PLAN-GS
50.0
29.0
12.9
250M
PLAN-GS
$4.5
63.6
46.5
46.5
44.7
39.5
27.0
25.0
45.5
30.3
38.2
47.1
34.3
25.7
16.1
19.4
24.7
24.7
45.2
25.8
790M
FLAN Gaaa
$1.8
72.7
66.3
61.6
31.6
42.1
35.0
28.0
48.5
51.5
55.9
52.9
51.4
34.3
19.4
29.0
34.7
20.0
54.8
29.0
80M
FLAN-EC BC
9.1
45.5
38.4
39.5
39.5
44.7
30.0
17.0
48.5
54.5
14.7
29.4
31.4
17.1
16.1
32.3
27.1
24.1
38.7
23.6
FLAN-BC
45.5
$2.3
56.4
41.2
25.8
39.5
47.1
29.0
3B
PLAN-EC,
90.4
56.4
68.1
60.7
52.1
31.4
24.5
25.7
66.2
32.3
55.4
35.5
59.5
61.4
35.0
27.8
43.6
26.2
41.4
40.6
250M
ST
27.3
0.0
26.7
20.9
15.8
0.0
23.0
0.0
24.2
12.1
29.4
5.9
17.1
5.7
35.5
6.5
23.5
1.2
19.4
29,0
STAN
47.9
54.5
41.9
31.3
36.8
22.4
25.0
36.4
40.6
50.0
15.3
28.6
21.8
16.1
31.2
25.3
47.6
3233
32B
18.2
27.9
36.0
36.8
2.6
29.0
0.0
24.2
36.4
14.7
11.8
14.3
25.7
25.8
9.7
24.7
22,6
3.2
FLAN-ST
21

---

Published as a conference paper at ICLR 2024
Table 8: MMLU[50:57] individual task performance.
MMLU
Professional Psychology
Public Relations
Security Studies
Sociology
US Foreign Policy
Virology
World Religions
Average
Model
Direct
CoT
Direct
CaT
Direct
CoT
Direct
CoT
Direct
CoT
Direct
CoT
Direct
CoT
Direct
CoT
davinci
37.7
43.5
50.0
50.0
44.4
40.7
59.1
45.5
33,3
27.8
63.2
68.4
39.7
40.5
text text-davinci-O00
65.2
58.0
50.0
50.0
77.8
48.1
90.9
86.4
81.8
81.8
44.4
33.3
84.2
78.9
63,1
60.0
text davinci 003
68.1
63.8
50.0
50.0
70.4
63.0
86.4
95.5
81.8
90.9
50.0
50.0
84,2
84.2
64.8
64.6
code-davinci-002
76.8
66.7
50.0
58.3
74,1
51.9
86.4
90.9
90.9
72.7
50.0
44.4
84.2
78.9
68.2
64.5
80M
T5 Small
20.3
4.3
33.3
16.7
18.5
0.0
22.7
0.0
27.3
9.1
27.8
5.6
21.1
15.8
26.7
5.6
250M
T5-Base
21.7
13.0
41.7
16.7
37.0
7.4
18.2
4,5
18.2
18.2
33.3
11.1
21.1
21.1
25.7
14.5
Flan-T's -S-Base
39.1
40.6
41.7
33.3
29.6
29.6
54.5
59.1
36.4
45.5
44.4
33.3
31.6
15.8
35.6
780M TS-1
18.8
23.2
25.0
16.7
14.8
18.2
22.7
18.2
18.2
33.3
27.8
31.6
26.3
25.1
15.0
Flan- Larrge
56.5
56.5
58.3
50.0
22.2
29.6
68.2
59.1
54.5
27.3
61.1
38.9
47.4
52.6
44.7
38.8
TS-XL
24.6
20.3
33.3
41.7
27.3
27.3
16.7
27.8
47.4
31.6
14.5
Flan-TS-XL
$6.5
52.2
58.3
50.0
44.4
48.1
77.3
59.1
54.5
72.7
38.9
50.0
73.7
63.2
50.3
46.1
T5-XXL
17.4
30.4
8.3
16.7
18.2
16.7
16.7
68.4
18.7
Flan-TS-XXL
68.1
58.0
58.3
41.7
59.3
44.4
86.4
63.6
54.5
45.5
44.4
50.0
31.6
63.2
52.6
47.9
PaLM
17.4
31.9
33.3
25.0
22.2
25.9
31.8
40.9
36.4
18.2
16.7
27.8
21.1
10.5
24.3
24.1
Flan-PaLM
46.4
43.5
50.0
41.7
40.7
72.7
31.8
63.6
44.4
27.8
68.4
73.7
62B
PaLM
58.0
58.0
58.3
58.3
40.7
40.7
81.8
68.2
81.8
72.7
61.1
44.4
73.7
78.9
55.1
49.0
PaLM
71.0
63.8
55.6
73.7
540B PaLM
73.9
60.9
66.7
58.3
74_1
40.7
95.5
81.8
100.0
100.0
61.1
44.4
89.5
89.5
71.3
62.9
Flan-PaLM
76.8
79.7
58.3
66.7
55.6
95.5
90.9
100.0
100.0
50.0
44.4
89.5
89.5
73.5
70.9
250M
Switchasse
34.8
13.0
16.7
25.9
13.6
18.2
18.2
22.2
5.6
36.8
26.3
28.3
13.6
FLAN- Swith
42.0
39.1
50.0
50.0
18.5
22.2
68.2
72.7
63.6
45.5
44.4
33.3
42.1
$2.6
38.0
34.1
780M
Switch, AROE
23.2
33.3
26.3
FLAN-Switche Switc
58.0
46.4
41.7
25.0
51.9
48.1
72.7
54.5
63.6
54.5
44.4
44.4
57.9
73.7
46.0
40.3
11B Switchen
26.1
17.4
16.7
25.0
29.6
22.7
18.2
18.2
18.2
27.8
16.7
26.3
15.8
24.6
15.1
FLAN Switchxu.
65.2
62.3
50.0
50.0
66.7
55.6
90.9
63.6
81.8
90.9
55.6
44.4
84.2
78.9
55.6
50.1
80M
FLAN GAN--GSMML..
31.9
26.1
58.3
33.3
37.0
44.4
54.5
54.5
36.4
45.5
44.4
38.9
31.6
31.6
32.5
26.8
250M
FLAN-GSEASE
50.7
42.0
41.7
33,3
29.6
40.7
63.6
40.9
36.4
36.4
55.6
50.0
42.1
36.8
39.9
33.6
780M
FLAN- FLAN-GS,.sor GSAARo
62.3
53.6
50.0
50.0
25.9
33.3
72.7
50.0
45.5
45.5
38.9
27.8
52.6
68.4
47.8
40.8
80M
PLAN-ECAMAL
31.9
31.9
33.3
25.0
33.3
29.6
45.5
$0.0
36.4
36.4
33.3
16.7
21.1
26.3
34.1
25.1
250M
BC
52.2
39.1
780M
FLAN-BC, MANN-CCARCE
52.2
52.2
50.0
58.3
40.7
25.9
77.3
68.2
63.6
$4.5
55.6
55.6
73.7
68.4
48.3
43.4
FLAN-ECS AC.-
61.8
47.6
49.5
24.9
47.9
85.9
55.5
81.3
56.2
49.5
43.4
67.9
74.9
52.1
41.4
250M
ST
26.1
15.9
33.3
27.8
31.6
25.2
17.7
FLAN STANS
44.4
34,8
60.7
41.7
32.0
40.7
43.3
27.3
47.9
36.4
41,3
38.9
44.5
42.1
42.4
35.5
32B
ST3B
18.2
16.7
26.3
FLAN-ST
72.5
63.8
50.0
58.3
70.4
55.6
90.9
86.4
100.0
100.0
44.4
44.4
84.2
84.2
65.4
63.0
22

---

Published as a conference paper at ICLR 2024
B.2 BBSH
BBH refers to subset of difficult tasks from BIG-Bench, handpicked by Suzgun et al. (2022) in 2022, where the model proposed by Srivastava al. (2022) in the same year outperformed the average human rater. Suzgun et al. (2022) mentions 23 tasks, two of which consist of three subtasks each. For ease of interpretation, we treat these subtasks as standalone tasks and calculate an unweighted average. We utilize the prompts provided in Suzgun et al. (2022)'s study.
Table 9: BBH[[9] individual task performance.
BBH
Boolean Expressions
Causal Judgement
Date Understandng
Disambigration QA
Dyck Languages
Formal Fallacies
Geometric Stapes
Hypecbatio
Logical Deduction Five Ubjects
Model
Diroet
CuT
Direct
CoT
Direet
CaT
Disect
CoT
Disect
CoT
Dicei
CoT
Direet
CuT
Divert
CaT
Divect
CaT
davinci
$4.0
69.2
57.8
48.1
37.6
52.4
40.0
40.8
28.0
0.0
47.2
52.8
10.4
10.8
49.6
47.6
24.4
34.4
1ext-davinci-002
90.0
87.6
57.8
56.1
55.6
81.6
66.4
70.8
42.0
32,0
52.4
58.4
35.2
56.0
67.2
72.4
31.6
51.2
test-davinsi-0::
58.0
36.8
60.4
code-davinci-002
88.4
92.8
63.6
54.0
63.6
87.2
67.2
76.0
46.8
56.8
52.4
50.4
32.0
54,4
60.4
66.4
32.4
54.8
80M
TS- Small
10.0
0.0
51.3
2.7
20.0
10.8
34.8
14.0
2.4
0.0
52.8
0.0
8.4
0.0
$2.0
0.0
17.2
7.6
Flan- T5-Small
$4.0
39.6
48.1
42,8
22.4
20.4
31.2
2.0
0.0
0.0
53.2
46.8
8.8
4.0
65.2
13.2
22.0
19.2
250M TS Base
46.0
45.6
51.9
38.0
20.0
19.6
33.6
30.8
1.6
0.0
46.8
31.2
22.0
0.0
$1.2
0.0
19.2
9.5
Flan-TS. Ta--ee-Baaee
48.4
46.4
52.4
47.1
18.0
20.4
54,8
44.8
7.6
53 2
49.2
0.4
12.8
67.6
58.8
27.2
22.0
7H0M
TS-Large
20.8
20,0
34,8
46.8
0.0
0.0
19.6
14.8
aane
64.0
58.0
56.1
20.9
74.4
26.8
67.6
61.2
22.8
8.0
56.0
47.6
3B
TS-XL
55.2
47.2
52.4
26.7
21.6
22.4
32.4
4.8
6.0
0.0
17.2
7.2
8.4
0.0
$2.0
0.0
22.0
22.8
Han-TT-XL
52.4
56.0
62.0
56.1
46.8
48.8
70.0
70.4
0.0
0.0
56.4
48.0
15.2
4.4
55.6
56.8
54.0
32.4
11B
TS-XXL
19.6
65.2
52.4
1.6
2
$1.0
35.2
0.D
2.0
0.0
52.4
0.0
15.6
0.0
55.6
0.0
18.0
37.2
Flan TS-XL
56.8
60.8
60.4
53.5
69.6
53.6
71.2
71.2
0.8
0.4
55.6
46.4
14.0
24.8
71.6
53.2
55.6
46.4
8B
Flan-PaLM
48.8
52.8
60.4
54.0
10.8
28.8
58.0
55.6
20.8
0.0
52.0
50.8
15.6
4.0
65.66
36.8
25.2
22.4
PaLM
70.8
39.2
58.8
53.2
34.4
9.6
72.8
PaLM
66.8
73.6
64.2
62.6
42.8
34.4
69.2
39.2
13.2
0.0
55.6
49.2
18.0
13.2
74.4
59.2
54.0
42.8
540B
M
83.2
80.0
61.0
59.4
53.6
79.2
60.8
67.6
28.4
28.0
53.6
$1.2
37.6
0.0
70.8
901.
39.6
49.2
Flan PaLM
86.0
83.2
65.2
63.1
58.0
74,0
76.8
69.6
29.2
23.6
62.4
52.8
40.0
43.6
67.6
88.8
54.4
52.4
250M
Switchwas FLAN-Swilchaan
0.0
51.2
42.8
55.1
55.6
18.8
18.4
63.6
53.6
0.0
56.8
54.8
9.6
8.8
64.3
62.0
34.8
22.0
780M
Switch
26.0
5.3
5.3
0.0
10.8
0.0
0.0
0.0
0.0
0.0
15.2
0.0
8/1
48.1
0.0
0.0
Swin-Switch
$4.0
22.0
56.7
50.8
25.2
24.0
67.2
59.2
0.8
0.0
54.8
43.6
11.6
3.6
56.8
30.0
47.2
28.0
11B
Switchese
0.0
3.2
0.0
37.4
0.0
2.4
0.0
8.8
0.0
0.0
0.0
21.6
0.0
0.4
0.0
30.4
0.0
0.4
PLAN-Swichhas
56.2
57.3
65.5
61.4
60.9
55.3
70.4
66.4
0.8
57.3
47.7
12.8
8.8
58.
58.0
54.9
8IM
PLAN-GS
46.0
21.2
21.6
28.4
54.8
16.4
250M
FLAN-GSSASE
48.0
34.0
53.5
51.9
27.6
11.2
65.2
26.0
0.0
0.0
53.2
$1.6
9.6
18:4
59.6
1.2
35.6
20.4
780M
FI LLAC
46.8
41.2
53.5
50.8
5.6
37.2
68.8
66.0
2.0
0.0
51.2
12.4
19.2
12.8
$4.0
50.8
47.6
28.4
ELAN-ECa
39.2
21.6
31.0
36.4
54.4
250M PLAN-EC..o.
57.6
43.6
50.3
50.8
34.4
24.8
67.6
34.4
0.8
0.0
53.6
17.2
9.6
7.6
72.0
44.0
33.6
24.0
780M
FLAN-EC,
58.8
48.0
58.8
50.8
35.6
43.2
69.2
70.0
0.0
0.0
53.2
30,8
4.5
5.6
66.4
52.8
41.6
21.6
3B
ELAN-EC..
51.3
49.7
59.9
56.2
18.1
37.4
69.0
32.9
-1.3
53.0
50.0
9.9
4,0
40.1
50.1
38.9
STEAST
9.2
0.0
14,4
0.0
0.0
0.0
0.4
18.8
PLAN-ST
48.0
49.3
59.6
54.1
11.6
36.1
66.1
64.2
1.0
0.0
50.0
44.2
19.5
12.1
51.4
49.9
49.6
21.4
328
ST.S
0.0
0.0
0.0
0.0
0.0
32.8
0.0
D.4
00
0.0
0.0
010
0.0
1.2
0.0
0.4
0.0
6.4
SLAN-ST22
63.6
67.6
67.9
65.8
66.1
62.0
70,8
71.8
15.2
0.0
58.8
42.0
22.8
5.2
60.0
51.1
61.0
19.6
23

---

Published as a conference paper at ICLR 2024
Table 10: BBH[9:18] individual task performance.
Legical Doluction Sovcn Objictc
Legical Dediction Thrre Objocts
Movie Rezommennhiio
Multistpp Antlonztic
Navigate
Object Countigg
Pragrins ina Table
Recooning about Culoral Ohyaets
Namos
Ruin
Model
Uirect
Col
Direct
CdT
Direct
Cull
Direct
LoT
Direct
CiT
Direct
Coll
Direct
Col
Direct
Coll
Direct
Cirl
davinci
20.0
27.3
71.3
49
28.1
35.6
13.3
18.4
33.2
text davinci-OO 002
26.8
38.0
45.2
87.5
72.0
78.8
1.2
53.2
68.0
88.8
44.0
772
47.3
81.5
47.6
78.4
65.6
62.8
lest-din tntt-davinci-00
40.0
52.4
62.0
88.0
79.2
K3.6
1.2
19.6
53.2
94.4
33.2
82.0
$2.1
83.6
67.2
96.8
82.0
58.8
code davinci 4022
26.0
38.8
52.8
87.5
84.8
90.4
1.2
47.6
50.4
96.4
45.2
93.2
66.4
79,5
67.6
91.6
75.2
68.4
S0M
TS-Small Flan T5 Small
13.2
5.2
31.6
14.0
26.0
14.8
0.0
0.0
55.2
40.0
10.0
0.0
21.9
19.2
16.0
11.2
22.4
1.6
16.8
112
30.8
30.0
43.2
20.4
0.0
1.6
58.0
58.0
5.6
3.2
21.9
10.3
17.2
10.8
13.2
0.8
250M TS Base Flan Base
14.8
2.4
29.6
22.4
27.6
0.4
0.4
0.0
48.0
42.0
8.8
0.0
21.9
19.2
15.6
12.4
28.0
2.4
24.4
19.2
42.8
40.8
39.6
32.4
0.4
0.0
02.8
32.4
22,8
112
17.8
9.6
22.4
23.6
13.6
10.4
750M T5-1 arp Fla--T5-Lan
13.2
8.0
32.4
26.0
24.8
23.2
0.4
0.0
42.0
42.0
9.6
6,4
21.9
23.3
10.4
14.8
23.6
0.4
46.8
22.4
53.2
36.8
41.6
28.0
0.4
0.1
41.8
34,0
32.8
16.8
22.6
22.6
43.6
35.1
25.8
25.6
3B TS-XL
13.6
15.2
35.2
35.6
23.6
0.8
0.8
42.0
38.0
6.4
25.2
21.2
25.3
12.8
14.8
26.0
0.8
Flan XL
53.6
25.2
66.0
50.5
46.4
36.4
0.5
0.4
45.4
46.4
42.4
30.8
37.7
35.6
50.8
46.0
42.0
28.4
11B
T5-XXL
18.0
18.0
36.8
42.8
46.0
45.2
0.0
41.6
37.2
31.6
33.2
21.2
24.7
16.4
22.8
20.8
0.0
FX-X-T5-XXL
54.8
48.8
76.0
58.8
53.3
53.2
0.4
0.4
60.4
54.0
50.8
34.0
39.0
39.0
58.8
46.8
52.4
53.2
SB
PaLM
13.2
14.8
35.6
36.4
28.4
26.4
0.8
58.0
58.0
36.8
18.8
25.3
19.9
18.0
18.8
21.2
24.4
Plan-PaLM
25.6
12.8
47.6
40.8
72.8
43.6
0.8
0.8
58.4
55.6
30.0
24.8
26.7
30.1
28.4
34.0
36.8
32.0
62B
PaLM
19.6
20.0
36.8
52.4
60.8
70.8
56.4
55.2
41.6
50.4
24.0
37.0
17.2
48.0
50.4
54.0
Flan-PaLM
48.8
34.0
74.0
56.0
82.0
72.8
1.2
1.6
60.4
49.2
50.4
51.3
37.0
49.3
50.4
46.0
63.6
54.8
540R
P
24.8
78.0
92.0
62.4
44.5
65.1
38.0
76.0
61.6
Plan-PaLM
50.8
48.4
85.6
87.2
85.6 6
82.4
0.8
20.6
68.4
78.0
54.0
85.8
55.5
72.6
66.4
82.4
$1.2
68.0
250M
Switchass
0.0
0.4
0.0
1.2
0.0
3.5
0.5
0.0
0.0
0.0
0.0
0.0
0,0
0.0
0.0
6.1
0.0
0.0
PLAN-Swich,ar
38.4
23.2
47.2
41.6
41.6
33.2
0.0
0.0
59.2
54,0
30.8
18.4
34,9
19.9
36.8
24.8
12.4
10.4
750M
Switch
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.4
0.0
0.0
0.4
0.0
0.0
17.8
0.0
4.0
0.0
0.4
Fl SlAN-Sulich.
44.8
22.8
57.2
42.0
61.2
47.2
0.4
0.8
45.6
43.2
41.6
53.2
38.4
29.5
42.0
32.4
11.6
10.8
11B
Switchoc
0.0
0.0
0.0
4,0
0.0
1.2
0.5
0.0
0.0
0.0
0.0
1.6
0.0
6.8
0.0
2.0
0.0
2.0
SwinhSuischeu
61.1
46.9
80.6
70.5
58.5
54.1
1.5
0.4
58.4
58.2
47.2
40.3
47.6
44.2
62.8
56.7
66.4
50.4
80M
FLAN-GS
16.8
12.4
33.6
34.4
42.8
13.2
0.0
0.1
62.4
40.0
20.0
9.2
13.0
15.8
25.6
19.2
9.2
6.4
250M
FLAN-GS
34.0
10.8
FLAN CS
46.8
21.6
31.5
25.3
25.6
29.6
32.4
80M ANA--C
148
29.6
64
57.6
19:6
13.7
17.8
21.6
8.8
8.0
250M PLAN-ECaSO
35.2
24.0
50.8
34.8
24.8
34.0
0.4
0.4
62.0
50.4
32.8
24.8
31.5
26.0
53.2
26.0
18.0
15.2
750M
FLAN-EC
50.0
22.8
57.2
30.0
50.8
45.2
0.0
0.8
58.8
59.6
38.
31.2
33.6
27.4
34.4
39.6
20.0
26.4
3B
PLAN-ECo:
53.4
48.6
60.8
56.5
48.6
38
66.7
35.1
0.0
0.4
53.6
49.2
11.0
4.5
61.4
40.3
$3.0
37.9
250M
STEGE
0.0
13.2
0.0
28.8
0.0
4,0
0.0
0.0
42,0
0.0
6.4
0.0
15.8
0.0
6.4
0.0
0.8
FI PLAN-ST
43.5
22.7
3IB
STAN
0.0
1.6
0.0
20.8
0.0
0.4
0.0
0,4
3.2
0.0
0.0
0.0
10.1
0.0
0.0
PLAN-ST
6244
48.0
61.6
55.5
78.0
72.8
64.4
24

---

Published as a conference paper at ICLR 2024
Table 11: BBH[18:27] individual task performance.
BBR
Salien Encr Detectioo
Snarfo
Unlastazzing
Sequerses
Objoda (5)
Trucking Staffled Otjeca (?)
Oljoco (3)
Web
Lics
Weed Sortiz
Average
Model
Direci
OoT
Diect
Cot
Durett
CuT
Direci
CeT
Direct
Cor
Direci
Cu7
Dxrest
CvT
Diect
OcT
Diatt
CoT
Doest
Cot
davinci
22.4
5.2
52.2
47.8
54.4
94.0
22.8
22.4
32.0
18.0
13.6
14.8
33.6
32.0
48.85
59.2
11.2
6.0
33.5
38,3
test davinci 0022
61.6
62.4
65.2
60.7
31.6
92.0
33.6
67.2
25.2
60.3
17.2
59.6
34.8
02.8
51.5
92.0
36,8
46.4
48.5
67.2
text davinci 1003
68.0
04.88
67.4
74.2
12.4
96.0
37.6
58.0
18.0
80.8
16.0
31.2
30.4
08.4
55.2
100.0
45.6
41.6
50.9
7007
code dasinci--00 400
62.0
60.8
61.3
99.6
72.8
976
716
96.8
20.4
89.6
14.4
35.6
37.6
78.
51.6
95.2
50.
4044
52.8
75/7
80M
SS-Small Flat Smuil
12.0
46.4
35.6
0.0
0,0
0.0
22.4
46.6
54.8
15.2
25.6
250M
TS Base
22.0
0.5
46.1
46.4
38.4
28.4
29.4
2004
56
15.2
5.6
31.6
9,8
51.6
22.4
088
322
27.8
14.6
Flat 15 Buse
11.6
18.0
42.7
46.1
92.8
46.4
18.4
20.4
16.8
19.2
10.4
11.2
33.2
32.0
52.4
47.2
4.0
2.0
30.3
26.8
760M
TS4
22.4
0.0
46.1
14.6
46.3
48.4
75.0
28.4
22.0
16.4
152
9.3
12.0
22.8
49.2
22.8
3.2
0.0
27.5
161
La--ne-
41.6
25.6
$7.9
52.8
52.0
45.2
23.2
12.4
11.2
84
10.4
35.6
31.6
51.2
48.4
0.8
2.4
34.7
28.5
47.2
32.4
2.4
19.2
Flu--TS-XL
72.5
31.2
15.6
29.6
11B TSXXXL
15.2
0.0
53.9
25.3
47.2
60.0
19.2
17.2
18.4
1.6
10.0
0.0
33.2
30.0
48.8
4.4
3.2
2.0
29.5
19.3
FIn--I--XXL
45.4
50.0
74.7
76.4
64.4
06.0
75.6
21.2
16.0
12.0
9.5
16.8
78.8
24.8
$4.0
53.2
7.2
4.4
45.6
41.6
62R
PAM
26.0
21.6
53.8
48.3
78.4
05.6
21.2
26-4
19.6
188
13.6
1366
10.4
16.4
48.5
80.8
7.6
84
57.4
42.3
Flan PaLM
45.2
40.4
53.1
75.1
79.2
91.2
30.8
36.0
21.2
18.0
15.2
18.0
22.0
29.0
48.4
92.0
11.2
10.0
47.5
41.9
540B
PaLM
54.0
78.0
30.4
98.0
16.8
57.6
13.6
28.4
58.8
62.0
Flan PalM
$5.4
85.2
50.8
71.6
35.2
66.3
250M
Swid
0.0
0.00
0.0
0.0
0.0
0.0
0.0
13.5
10.0
0.0
0.0
0.0
0,0
0.0
0.0
0.0
0.0
0.0
0.1
1.4
PLAN Switchus
27.2
25.6
39.3
29.9
53.2
56.4
10.4
15.5
11.6
15.2
14.4
14.4
32.0
33.6
49.5
53.2
2.4
1.2
33.2
29.4
SwiRch SALE
JuCK
11B Swisdhmn
0.0
6.8
0.0
0.0
12.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
39.6
0.0
0.0
0.0
6.7
FLAN- SwichSw
51.7
41.1
$1.1
74.3
68.5
74.3
40.0
36.4
19.5
18.0
21.0
14.0
20.3
25.7
50.3
49.7
8.3
4.7
47.9
43.4
W/M FLAN-GS
20.8
46.6
54.0
52.8
22
21.4
23.6
16.0
12.4
14.4
32.0
51.6
0.0
29.6
209
250M
FLAN-GS...
47.8
0.4
25.1
750M
16.8
61.8
59.2
20.5
12.4
5.6
34.0
19.2
1.5
FLAN-EC....
48.3
54.0
30.0
0.0
250M
FLAN-EC..e
41.6
14.8
34.0
750M
55.6
59.2
12.4
12.8
33.6
3B
DC
17.0
2.3
250M STaue
0.0
10.8
0.0
44.4
0.0
47.2
0.0
2.0
10.0
0.0
0.0
0.0
0.0
0.0
0.0
21.2
0.0
6.0
0.0
14.0
FLAN-ST
13.3
11.6
61.0
58.1
56.0
522
15.4
20.2
12.2
12.3
7.9
12.2
33.9
34.5
52.5
48.6
3.3
2.2
34.7
26.6
FLAN-STyE
B.3 REASONING
The four reasoning tasks are held-in, which means we perform instruction finetuning on the training set while evaluating on the "validation" set in sew-shot way. The detailed performance is presented here.
25

---

Published as a conference paper at ICLR 2024
Table 12: Reasoning[:4] individual task performance.
Reasoning
GSM8K
ASDIV
StrategyQA
SVAMP
Average
Model
CoT
CoT
CoT
CoT
CoT
80M
T5-Small Tl-mall-Small
1.1
1.7
37.1
1.3
10.3
2.1
2.8
53.2
2.1
15.0
250M
T5-Base Flan-T5-Base
2.0
1.8
52.8
2.0
14.7
3.9
4.9
53.3
3.5
16.4
780M
T5-Large Flan-T5-Large
1.6
2.0
42.8
1.0
11.9
8.6
14.5
54.2
11.6
22.2
3B
T5-XL
2.7
5.2
45.9
2.9
14.2
Flan-T5-XL
16.9
28.2
64.6
25.9
33,9
11B
T5-XXL
2.5
15,0
55.0
12.9
21.4
Tlan-T5-XXL
26.7
47.4
69.9
41.4
46.3
8B
Plan-PaLM
21.4
37.5
65.5
23.1
36.9
62B
Flan-PaLM
47.5
64.5
76.4
50.2
47.7
540B
Ml
73.0
77.7
83.0
72.2
76.5
250M
Switch FLAN-SWitcheAEE
0.6
1.0
17.5
1.5
5.2
6.4
8.4
53.3
6.3
18.6
780M
Switch FLAN-Switch,ARGE
1.9
2.4
43.2
2.0
12.4
12.7
19.0
56.3
13.0
25.3
11B
SwitchxxL
0.4
36.2
0.1
9.2
FLAN-Switche
27.0
47.8
70.1
41.7
46.6
80M
FLAN-GS GSSMALL
3.7
5.0
53.3
3.3
16.1
250M
BLAN-GSBASE
11.1
13.9
53.7
9.9
22.2
780M
FLAN-GSLARGE
16.7
22.2
54.6
17.0
27.6
80M
FLAN-EC SMALL
5.2
5.6
53.3
5.4
16.6
250M
FLAN-EC BASE
10.7
13.7
53.3
10.5
22.0
780M
FLAN-EC, LAAGE
15.9
25.7
65.5
21.7
32.2
3B
FLAN-EC
21.3
33.6
67.2
30.3
38.1
250M
ST STRASE
2.0
1.9
45.0
1.3
12.6
FLAN-ST
11.2
11.1
59.8
8.0
22.5
ST
2.7
18.4
1.7
16.2
9.8
FLAN-ST:28
51.1
65.3
80.8
68.1
66.3
26

---

Published as a conference paper at ICLR 2024
Table 13: QA[:5] individual task performance.
QA
UnifiedQA
ARC easy
ARC challlenge
BoolQ
Average
Elementary Science
Model
Direct
Direct
Direct
Direct
Direct
80M
Flan-T5-Se T5-Small
27.6
40.4
31.9
63.7
40.9
250M
T5-BaT
34.1
46.1
38.7
76.2
48.8
780M
La-age
43.9
76.3
53.2
84.0
64.4
3B
F5-n-T5-XL
53.7
88.4
66.2
88.0
74.1
11B
T5XXLL
63.4
94.2
74.6
89.3
80.4
8B
Flan-PaLM
72.4
83.4
61.7
83.0
75.1
62B
Flan-PaLM
85.4
92.0
77.3
86.3
85.3
540B
Plan-PaLM
92.7
95.2
88.7
83.0
89.9
250M
FLAN-S SLiN-SSitch
48.1
61.4
43.2
79.3
58.0
780M
FLAN-S MAANN-SSthhhAAR
50.3
70.3
61.7
83.8
66.5
L1B
FLAN-Switchae
60.2
73.7
91.7
89.7
78.8
80M
FLAN-CG FLAN-GSSSAA SMALL
39.0
48.5
36.0
72.0
48.9
250M
FLAN-C BASE
43.9
59.3
45.9
82.5
57.9
780M
FLAN-GS,, LARGE
53.7
69.4
66.7
88.2
69.5
80M
FLAN-EC SMALLL
37.4
61.4
50.0
83.4
58.1
250M
FLAN-EC BASE
51.2
61.4
50.0
83.4
61.5
780M
FLAN-EC, LARGE
59.3
71.8
71.3
90.1
73.1
3B
FLAN-EC XL
60.1
71.8
75.3
90.1
74.3
250M
FLAN-ST BASE
47.2
58.3
57.7
82.6
61.5
32B
ST
31.7
25.8
30.1
40.6
32.1
FLAN-ST
69.9
99.2
90.8
92.1
88.0
B.4 QA
We perform evaluation on four held-out QA tasks and the results are summarized in this section.
27


Reviews:
[{"review_id": "MNplU4nrUS", "reviewer": "Reviewer_8EEX", "summary": "This paper studies the role of instruction tuning in whether mixture of experts models outperform dense models on language tasks. It turns out that instruction tuned mixture of experts performs better than dense models.", "questions": "1) The authors state that MoE can be used to add learnable parameters to LLMs \"without increasing inference cost.\" I think this is somewhat confusing. Increased memory usage is as much of a \"cost\" as increased FLOPs or latency.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) This paper does not really present a new algorithm or theory, but is mostly a large collection of experiments showing under what conditions the proposed Flan-MoE model performs well. With that being said, the number and thoroughness of the experiments and ablations is quite impressive. \n2) The authors acknowledge limitations of MoE models and show to mitigate them, i.e. using auxiliary loss to mitigate overfitting", "weaknesses": "I don't see any", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRpLFvPK9R", "reviewer": "Reviewer_nXAt", "summary": "MoE model is a sparse model architecture that can be utilized to scale the number of parameters without significantly increasing the computation cost. In this research, the authors conducted experiments comparing dense models with MoE models using instruction tuning. The results indicate that combining sparse MoE models and instruction tuning leads to a significant enhancement in model performance, surpassing dense models across various datasets.", "questions": "1. I understand the performance of instruction tuning on MoE models, but can you please provide any analysis or insight about the reasons behind such good performance? Does it help improve the routing strategy, expert specialization, or something else?\n\n2. The author claims that these advancements are attained without necessitating an increase in computational resources or even reducing the resource requirements. I am confused about how it can reduce resource requirements in the training and inference time. Can you discuss more about the details of the process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ This paper tries to apply the instruction tuning to the context of MoE models for downstream tasks. The experimental results demonstrate the combination has great potential to improve the performance of large language models.\n\n+ The authors conducted comprehensive experiments on various sparse and dense models to support their claim, and in most cases, the combination of instruction tuning and MoE models shows strong performance over other models.", "weaknesses": "+ Lack of clear motivation. The motivation behind the combination of MoE with instruction tuning requires further discussion. While it is acknowledged that instruction tuning and MoE models can outperform dense models or fine-tuning MoE, it will be beneficial to provide some insights into why these approaches were chosen.\n\n+ The presentation of this paper needs some improvements. Some grammar things could be improved in the explanation and discussion of the key component of this paper. \n\n+ The impact of the combination design (instruction tuning and MoE) on training and inference time should have more discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jzGs1ulbt", "reviewer": "Reviewer_YDYY", "summary": "This paper studies the benefits of applying instruction-tuning to MoE models. It presents a series of instruction fine-tuned MoE models, called FLAN-MoE, which have shown superior performance over task-specific fine-tuned MoE and their corresponding dense models.", "questions": "1. Some statements made by the paper are rather confusing. For example, the paper states, “However, we show that conventional, task-specific finetuning MoE models lead to suboptimal performance, often even worse than finetuning dense models with the same computational cost. One of the possible reasons is the discrepancy between general pretraining and task-specific finetuning.” However, regardless of whether the model architecture is dense or sparse, isn't there always a discrepancy between pre-training and task-specific fine-tuning?\n\n2. When the paper says the FLAN-MoE “does not come from increased computation resources or memory requirements,” what does it mean? Does it refer to computation and memory requirements during training/inference compared to compute-equivalent dense/MoE models? \n\n3. The paper says, “We demonstrate that in the absence of instruction tuning, MoE models fall short in performance when compared to dense models on downstream tasks.” However, this seems to be contradictory to some prior studies. For example, https://arxiv.org/pdf/2112.10684.pdf shows that MoE models can outperform compute-equivalent dense models on supervised fine-tuning tasks. \n\n4. What is the difference between FLAN-MoE and MoE in Section 4.1? \n\n5. The paper does not seem to describe the model architecture of FLAN-MoE adequately. The only one mentioned is ST-MoE-32B. It would be interesting to see how different pre-trained MoE models would affect the conclusion.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is a timely study, given that fine-tuning large-scale pre-trained MoE models for specific tasks is quite challenging.\n\n2. The paper provides relatively comprehensive studies of MoE models with instruction tuning, demonstrating that MoE models can benefit from additional instruction tuning.\n\n3. The findings are well-documented, including a range of MoEs sizes and discussions on limitations and failure cases.", "weaknesses": "1. While several MoE models have been tested, the conclusion about the necessity of the instruction-tuning stage is not convincingly demonstrated. For instance, the addition of this instruction-tuning stage can introduce additional training and tuning costs, e.g., in comparison to using just task-specific fine-tuning. Is it possible that the performance improvement can also come from this extra training cost?\n\n2. Related to the training cost, the paper claims that its improvement does not stem from increased computational resources or memory requirements. However, this is a bit confusing because instruction fine-tuning in this paper clearly uses a large set of datasets for training, which incurs training costs. Yet no direct report in terms of the training cost is included in the paper. To be more convincing, a detailed report on how the proposed method affects training costs should be included.\n\n3. Some parts of the paper lack clarity. See detailed questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQjouS75C1", "reviewer": "Reviewer_SYmE", "summary": "This paper demonstrates that large scale instruction tuning (using the FLAN data) of sparse mixture-of-expert (MoE) models before finetuning on downstream task data is crucial for MoE to beat comparable dense models (in terms of inference FLOPs).  Merely performing finetuning with MoE on downstream task data without instruction tuning beforehand underperforms directly finetuning a dense model (without instruction tuning), whereas the addition of the instruction tuning stage to the MoE model causes it to outperform dense models with the equivalent training procedure.  At all model scales, MoE outperforms comparable dense models whenever the instruction tuning phase is present, whereas MoE without instruction tuning underperforms.", "questions": "*Figure 1 (right):* It is not a big deal but can be a bit confusing that the T5 and Flan-T5 green and blue bars are included for each number of experts as these are independent of the number of experts.\n\n*Table 1:* Why not show Switch and GS performance at the 32G FLOP scale?\n\n*Figure 3:* Why not label the orange and blue curves by model size, at least in the caption?\n\n*Figure 6:* How is expert utilization measured?\n\n**Notes and minor details:**\n\n*Typo:* “benefits from a richer repertoire of specialized sub-networks .” (extra space before period)\n \n*Figure 2 caption:* “Average zero performance” --> “Average zero shot performance?”\n \nAdd period after paragraph title “Routing Strategy” for formatting consistency.\n \n\nTypo near bottom of page 8: “issue may stes” → “issue may stem”\n\n*Appendix p. 15*\n“We present a detailed learning efficiency experiment in Figure 7 across number of steps. It shows that MoE starts to outperform Dense counterparts right after 25k steps with instruction tuning.”\n\n* There are no labels for the lines in the figure, thus it’s impossible to tell which is the dense model and which is the MoE model\n\n*Appendix p. 15:* “We leave the study of scaling decoder-only FLAN-MOE as future works.” --> “We leave the study of scaling decoder-only FLAN-MOE to future work.”\n\n*Appendix p. 15:*\n\n“but yield worse performance” → “but yields worse performance”", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Originality:** As far as I know the significance of instruction tuning for MoE has not been studied extensively in the manner this paper puts forward.\n\n**Quality:** The claims are plausible and well supported.  The authors conducted comprehensive ablations across model scales, # of tasks for instruction tuning, MoE expert selection method, # of experts, etc.  There is no reason to question the central claim.\n\n**Contribution:** The contribution is to provide high quality data towards the effect of large scale instruction tuning for MoE models in relation to dense models with a comparable number of inference FLOPs.  They demonstrate that instruction tuning may be essential for MoE models to succeed.", "weaknesses": "**Weaknesses:**  There is little insight into what is causing the failure of direct finetuning of MoE models on downstream task data.  It could be that MoE models have higher capacity to overfit, however it is unclear if instruction tuning is preventing this or if there are other factors at hand.  More conceptual insight would be nice, however I do not view this as a major weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)