Finetuned Language Models are Zero-Shot Learners
Authors: Jason Wei
Maarten Bosma
Vincent Zhao
Kelvin Guu
Adams Wei Yu
Brian Lester
Nan Du
Andrew M. Dai
Quoc V Le
Venue: ICLR 2022 Oral (2022)
Keywords: natural language processing
zero-shot learning
language models

Abstract: This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning—finetuning language models on a collection of datasets described via instructions—substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction tune it on over 60 NLP datasets verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

Published as a conference paper at ICLR 2022
FINETUNED LANGUAGE MODELS ARE ZERO-SHOT LEARNERS
Jason Wei", Maarten Bosma", Vincent Y. Zhao", Kelvin Guu", Adams Wei Yu. Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le
Google Research
ABSTRACT
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction seetinn-finettnnng language models on a collection of datasets described via instructions-suusaaattlll improves zero- shot performance on unseen tasks.
We take a 137B parameter pretrained language model and instruction tune on over 60 NLP datasets verbalized via natural language instruction templates. We evaluate this instruction-tune model, which we call FLLAN, on unseen task types. FL.NN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate. FLAN even outperforms few-shot GPT-3 by large margin ANLI, RTE, BoolQ, A12-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number finetuning datasets, model scale, and natural language instructions are key the success of instruction tuning.
Finetune on many tasks ("Instruction-tuningg)
Input [Commonsense Reasoning)
Input (Translation)
Here is a goal: Get coal steap on summer days.
Translate this sentence to Spanish:
Inference on unseen task type
Input (Natural Language Inference)
How would you accomplish this goaf?
office building was built lass than throe monthe.
Premiee: my age you wil probably have leamt one lssson.
Kaap
Midge Keep slack of pillow oven.
Hypothesis: It's not certain how many lessons you'll learn by your thirtes.
Target
Target
nuevo ediido de oficinas construyó tres moses.
Does the premise entai the hypothesis?
keep stack of pillow fridge
OPTIONS:
(yee
<t nol possible to tell
-no
Sentiment analysis tasks
FLAN Reaponse
Coreference
it not possible to tell
GPT-3 175B zero shot
GPT-3 175B few-shot
FLAN 137B zero-shot
77.4
Performance unseen task types
53.2
56.2
72.6
55.7
56.6
42.9
63.7
49.8
Natural language inference
Reading Comprehension
Closed-Book QA
Figure 1: Top: overview of instruction tuning and FLAN, Instruction tuning finetunes a pretrained language model on mixture of tasks phrased instructions. At inference time, evaluate an unseen task type; for instance, we could evaluate the model on natural language inference (NLI) when no ILI tasks were seen during instruction tuning. Bottom: performance of zero-shot FLAN, compared with zero-shot and few-shot GPT-3, on three unseen task types where instruction tuning improved performance substantially out of we evaluate. NLI datasets: ANLI RI-R3, CB, RTE. Reading comprehension datasets: BoolQ, MultiRC, OBQA. Closed-book QA datasets: ARC-easy, ARC-challenge, NQ. TriviaQA.
"Leadd contributors. Author contributions listed end of paper

---

Published as a conference paper at ICLR 2022
INTRODUCTION
Language models (LMs) at scale, such as GPT-3 (Brown et al. 2020). have been shown to perform few-shot learning remarkably well. They are less successful at zero-shot learning, however. For example, GPT-3's zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference. One potential reason is that, without few-shot exemplars, it is harder for models to perform well on prompts that are not similar the format of the pretraining data.
In this paper, we explore a simple method to improve the zero-shot performance of large language models, which would expand their reach to a broader audience. We leverage the intuition that NLP tasks can be described via natural language instructions, such as "Is the sentiment of this movie review positive or negative?" ?" or "Translate 'how are you' into Chinese." We take a pretrained language model of 137B parameters and perform instruction tuning finetuning the model on a mixture of more than 60 NLP datasets expressed via natural language instructions. We refer to this resulting model as FLAN, for Finetuned Language Net.
To evaluate the zero-shot performance of FLAN on unseen tasks, we group NI.P datasets into clusters based their task types and hold out each cluster for evaluation while instruction tuning FLAN on all other clusters. For example, as shown in Figure T to evaluate FLAN's ability to perform natural language inference, instruction tune the model commonsense reasoning, translation, and sentiment analysis. As this setup ensures that FLAN has not any natural language inference tasks in instruction tuning, then evaluate its ability to perform zero-shot natural language inference.
range of other NLP tasks such as
Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model. FLAN's zero-shot also outperforms 75B-parameter GPT-3's zero-shot on 20 25 datasets that we evaluate, and even outperforms GPT-3's few-shot by large margin on ANLI, RTE, BoolQ, A12-ARC, OpenbookQA, and StoryCloze. In ablation studies, we find that increasing the number of task clusters in instruction tuning improves performance on unseen tasks and that the benefits of instruction tuning emerge only with sufficient model scale.
Instruction tuning is simple method that, as depicted in Figure 2 combines appealing aspects of both the pretrain-finetune and prompting paradigms by using supervision via finetuning to improve language model's responses inference-time text interactions. Our empirical results demonstrate promising abilities of language models perform tasks described purely via instructions. Source code for loading the instruction tuning dataset used for FLAN is publicly available at research/Elee
https: /ttt:://gihuule /ttptttttttt..o/gggggggg....e www.
(A) fretrain-finetune (BERT, T5)
Pretrained LM
Finetune task A
Inference on task A
(C) Instruction tuning (FLAN)
Typically requires many lask specific examples One specialized model for each taak
Pretrained LM
Instruction-tune on many tasks: D,
Inference on task A
Model learns to perform many tasks via natural language instructions
Inference unseen task
(B) Prompting (GPT-3)
Improve performance via few-shol prompling prompt engineering
Pretrained
Inference
Figure Comparing instruction tuning with pretrain-finetune finetune and prompting.
2 FLAN: INSTRUCTION TUNING IMPROVES ZERO-SHOT LEARNING
The motivation of instruction tuning is to improve the ability of language models to respond to NLP instructions. The idea is that by using supervision to teach an LM to perform tasks described via instructions, the LM will learn to follow instructions and do performance on unseen tasks, we group datasets into clusters by task type and hold out each task cluster for evaluation while instruction tuning all remaining clusters.
for unseen tasks. To evaluate
2

---

Published as a conference paper at ICLR 2022
2.1 TASKS & TEMPLATES
As creating an instruction tuning dataset with many tasks from scratch would be resource-intensive, transform existing datasets from the research community into instructional format. We aggregate 62 text datasets that are publicly available on Tensorflow Datasets, including both language understanding and language generation tasks, into a single mixture. Figure shows shows these datasets each dataset is categorized into one of twelve task clusters, for which datasets in given cluster are of the same task type. Descriptions, sizes, and examples of each dataset are shown Appendix G
Natural language inference (7 datasets)
Commonsense (4 datasets)
Sentiment (4 datasets)
Paraphrase
Closed-book Q4 (3 datasets)
Struct to text (4 datasets)
Translation dafasets)
(4 datasets)
ANLI (R1-R3)
RTE
CoPA
IMDB
(nanytchal)
CommanGen
HellaSwag
Sent140
QQP
NQ
DART
ParaCrowl EVES
MNLI
WNLI
PIDA
SST-2
TQA
E2ENLG
ParaCrusl ENFR
StoryCloze
Yelp
STS-8
WMIT-1 ENDD
Reading comp,
Read, comp, wi
Coreforence (3 datasets)
Miss.
Summarization
WMT-18
(5 dafasets)
commonsense datasets)
(7 datasets)
(11 datasets)
Boola
OBQA
DPR
CoGA
TREC
AESLC
News
SamSum
WMT ENIRO
DROP
SQuAD
CosmosQA
Winogrande
QuAC
Wiilingw
WhiT-16 ENURU
WIC
Math
CNN.DM
SulvvAti: Debar
XSum
MultiRC
ReCoRD
WSC273
Gigpword
Figure 3: Datasets and task clusters used in this paper (NLU tasks in blue; NLG tasks in teal).
For each dataset, we manually compose ten unique templates that use natural language instructions describe the task for that dataset. While most of the ten templates describe the original task, to increase diversity, for each dataset we also include to three templates that "turned the task around," (e.g., for sentiment classification we include templates asking to generate movie review). We then instruction tune a pretrained language model on the mixture of all datasets, with examples in each datasst formatted via randomly selected instruction template for that dataset. Figure shows multiple instruction templates for a natural language inference dataset.
Premise
Template 1
Template
Russian cosmonaut Valery Polyakov set the record for the longest continucus amount of time spent in space, a staggering 438 days, 1994 1995.
<premise>
Read the following and determine the hypothesis can be inferred from the premise:
paragraph
above, can we conclude that <hypothesis>?
Premise: <premise>
<options>
Hypothesis: <hypothesis> <options>
Hypothesis
Template 2
Russians hold the record for the longest stay in space.
<premise>
Target
Options:
Can infer the following?
Template 4....
<hypothesis>
Entailment
yes
Not entaliment
no
coptions>
Figure 4: Multiple instruction templates describing natural language inference task.
2.2 EVALUATION SPLITS
We are interested in how FLAN performs on tasks not seen in instruction tuning, and so it is crucial to define what counts an unseen task. Whereas some prior work defines unseen tasks by disallowing the same dataset to appear in training. we use a more conservative definition that leverages the task clusters from Figure In this work, we only consider dataset D unseen at evaluation time if no datasets from any task clusters that 'D belongs to were seen during instruction tuning. For instance, if 'D is an entailment task, then no entailment datasets appeared in instruction tuning, and we instruction-tune tunde on all other clusters! Hence, to evaluate zero-shot FLAN on e task clusters, instruction tune e models, where each model holds out a different task cluster for evaluation.
When evaluating on the read, comp. with commonsense cluster, both read. comp. and commonsense reasoning were dropped from instruction tuning. Conversely, the read. comp. with commonsense cluster was not used for instruction tuning when evaluating read. comp. commonsense reasoning. We also drop the paraphrase cluster from instruction tuning when evaluating on NI.I tasks and vice-versa.
3

---

Published as a conference paper at ICLR 2022
2.3 CLASSIFICATION WITH OPTIONS
The output space for a given task is either one of several classes (classification) or free text (generation). As FLAN is an instruction-tune sarnidd version of a decoder-only language model, it naturally responds in free text, and so no further modifications are needed for generation tasks.
For classification tasks, prior work (Brown et al., 2020) used a rank classification approach where, for example, only two outputs ("yes" and "no") are considered and the higher probability one is taken the model's prediction. Though this procedure is logically sound, it is imperfect in that the probability mass for answers may have an undesired distribution among ways of saying each answer (e.g., large number of alternative ways of saying "yes" may lower the probability mass assigned to "yes"). Therefore, we include an options suffix, in which we append the token OPTIONS to the end of a classification task along with a list of the output classes for that task. This makes the model aware of which choices are desired when responding to classification tasks. Example use of options shown in the NLI and commonsense examples in Figure]
2.4 TRAINING DETAILS
Model architecture and pretraining. In our experiments, we use LaMDA-PT, dense left-to-right, decoder-only transformer language model of 137B parameters (Thoppilan et al. (022). This model pretrained on collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with 32k vocabulary using the SentencePiece library (Kudo & Richardson 2018). Around 10% of the pretraining data was non-English. Note that LaMDA-PT only has language model pretraining (c.f. LaMDA, which was finetuned for dialog).
Instruction tuning procedure. FLAN is the instruction-tun -unne version of LaMDA-PT. Our instruc- tion tuning pipeline mixes all datasets and randomly samples from each dataset. To balance the different sizes of datasets, we limit the number of training examples per dataset to 30k and follow the ssamples-proportionn
mixing scheme (Raffel al. 2020) with mixing rate maximum of 3k [ We finetune all models for 30k gradient steps with a batch size of 8,192 tokens using the Adafactor Optimizer (Shazeer & Stern 2018) with learning rate of 3e-5. The input and target sequence lengths used in finetuning are 1024 and 256, respectively. We use packing (Ralfel et al. 2020) to combine multiple training examples into single sequence, separating inputs from targets using special EOS token. This instruction tuning takes around 60 hours on TPUv3 with 128 cores. For all evaluations, we report results on the final checkpoint trained for 30k steps.
3 RESULTS
We evaluate FLAN on natural language inference, reading comprehension, closed-book QA, transla- tion, commonsense reasoning, coreference resolution, and struct-to-text. As described in $2.2 we evaluate on unseen tasks by grouping datasets into task clusters and holding out each cluster for evaluation while instruction tuning on all remaining clusters (i.c., each evaluation task cluster uses different checkpoint). For each dataset, we evaluate the mean of performance on all templates, which proxies the expected performance given typical natural language instruction. As a dev set is sometimes available for manual prompt engineering (Brown et al., 2020), for each dataset we also obtain the test set performance using the template with the best dev set performance.
For comparison, we report zero and few-shot results for LaMDA-PT using the same prompts as GPT-3 (as LaMDA-PT is not suitable for natural instructions without instruction tuning). This baseline provides the most direct ablation of how much instruction tuning helps. Instruction tuning significantly improves LaMDA-PT on most datasets.
We also show the zero-shot performances of GPT-3 175B (Brown et al. 2020 and GLaM 64B/64E (Du et al. 202T), as reported in their respective papers. With the best dev template, zero-shot FLAN outperforms zero-shot GPT-3 on 20 of 25 datasets and even surpasses GPT-3's few-shot performance on 10 datasets. With the best dev-template, zero-shot FLAN outperforms zero-shot GLaM on 13 of 19 available datasets and one-shot GLaM on 11 of 19 datasets.
II this mixing scheme, mixing rate maximum of 3,000 means that dataset does not receive additional sampling weight for examples in excess of 3,000.
4

---

Published as a conference paper at ICLR 2022
Overall, observe that instruction tuning very effective tasks naturally verbalized as instruc- tions (e.g., NLI, QA, translation, struct-to-text) and is less effective on tasks directly formulated as language modeling, where instructions would be largely redundant (e.g,, commonsense reasoning and coreference resolution tasks that are formatted as finishing an incomplete sentence or paragraph). Results natural language inference, reading comprehension, closed-book QA, and translation summarized in Figure 5 and described below.
Natural language inference
ANLI R2
ANLI R3
ANLI R1
CB
RTE
Reading comprehension
MultiRC
OBQA
BoolQ
Closed-book QA
NQ
ARC-c
TQA
ARC-e
Translation
EN to RO
FLAN 137B LaMDA-PT1378
EN to DE EN to FR
GPT-3 175B GLaM 64B/64E
FR to EN
RO to EN
Supervised model
DE to EN
20
40
60
80
100
Zero-shop performance
Figure 5: Zero-shot performance of FLAN compared to LaMDA-PT 137B, GPT-3 175B, and GLaM 64B/64E natural language inference, reading comprehension, closed-book QA, and translation. Performance of FLAN is the mean of to 10 instructional templates per task. Supervised models were either T5, BERT, translation models (specified Table 2 and Table ii the Appendix).
Natural language inference (NLI). On five NLI datasets, where a model must determine whether a hypothesis is true given some premise, FLAN outperforms all baselines by large margin. As noted by Brown et al. (2020), perhaps one reason why GPT-3 struggles with NLI is that NLI examples are unlikely to have appeared naturally in unsupervised training and thus awkwardly phrased as a continuation of sentence. For FLAN, we phrase NLI as the more natural question "Does <premise> mean that <hypothesis>? ?", achieving much higher performance.
Reading comprehension. On reading comprehension, where models are asked to answer a question about provided passage, FLAN outperforms baselines for MultiRC (Khashabi et al. 2018) and OBQA (Mihaylov et al., 2018). On BoolQ (Clark et al. 2019a), FLAN outperforms GPT-3 by large margin, though LaMDA- already achieves high performance on BoolQ.
Closed-book QA. For closed-book QA, which asks models to answer questions about the world without access to specific information containing the answer, FLAN outperforms GPT-3 on four datasets. Compared to GLaM, FLAN has better performance ARC-e and ARC-c (Clrr et al.) 2018), and slightly lower performance on NQ (Lee et al. 2019 Kwiatkowski et al. 2019) and TQA (Joshi et al., 2017).
Translation. Similar to GPT-3, the training data for LaMDA-PT is around 90% English and includes some text in other languages that not specifically used train the model to perform machine translation. We also evaluate FLAN's performance on machine translation for the three datasets evaluated in the GPT-3 paper: French-English from WMT'14 (Bojar al. 2014). and German-
5

---

Published as a conference paper at ICLR 2022
English and Romanian- EnglishEn from WMT 16 (Bojar et al. 2016). Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases. Similar to GPT-3, FLAN shows strong results for translating into English and compares favorably against supervised translation baselines. Translating from English into other languages, however, was relatively weaker, might be expected given that FLAN uses an English sentencepiece tokenizer and that the majority of pretraining data is English.
Additional tasks. Although see strong results for the above task clusters, one limitation with instruction tuning is that it does not improve performance for many language modeling tasks (e.g.. commonsense reasoning or coreference resolution tasks formulated as sentence completions). For seven commonsense reasoning and coreference resolution tasks (see Table in the Appendix), FLAN only outperforms LaMDA-PP on three the seven tasks. This negative result indicates that when the downstream task is the same the original language modeling pre-training objective (i.e., in cases where instructions are largely redundant), instruction tuning is not useful. Finally, we report results for sentiment analysis, paraphrase detection, and struct-to-text, as well as additional datasets for which GPT-3 results are not available, in Table 2 and Table iin the Appendix. Generally, zero-shot FLAN outperforms zero-shot LaMDA-PT and is comparable with or better than few-shot LaMDA-PT.
ABLATION STUDIES & FURTHER ANALYSIS
4.1 NUMBER OF INSTRUCTION TUNING CLUSTERS
As the core question of paper asks how instruction tuning improves model's zero-shot perfor- mance on unseen tasks, in this first ablation we examine how performance is affected by the number of clusters and tasks used in instruction tuning. For this setup, we hold out NLI, closed-book QA, and commonsense reasoning as evaluation clusters, and use the seven remaining clusters for instruction tuning We show results for one to seven instruction tuning clusters, where clusters are added in decreasing order of number of tasks per cluster.
||
shows these results. As expected. we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis cluster), confirming the benefits of our proposed instruction tuning approach zero-shot performance the seven clusters we test, the performance does not appear to saturate, implying that performance may further improve with even more clusters added instruction tuning. Of note, this ablation does not allow us to draw conclusions about which instruction tuning cluster contributes the most to each evaluation cluster, although see minimal added value from the sentiment analysis cluster.
Figure S
novel tasks. It is further interesting to see that, for
on held-out cluster
90
Held-out clusters
Performance (%)
Commonsense
55.0
59.3
59.2
60.8
61.9
Average
NLI
50
Closed-book QA
# clusters:
summarization" (1))
[# datasats);
translation (20)
(20)
(30)
(34)
(9P)
(39)
read, comp.
sentiment
data to text
coreference
conv. QA
Clusters used for instruction tuning
Figure 6: Adding additional task clusters to instruction tuning improves zero-shot performance on held-out task clusters. The evaluation tasks are the following, Commonsense: CoPA, HellaSwag, PiQA, and StoryCloze. NLI: ANLI RI-R3, QNLI, RTE, SNLI, and WNLI. Closed-book QA: ARC easy, ARC challenge, Natural Questions, and TriviaQA.
We do not use the paraphrase reading comprehension with commonsense clusters for instruction tuning in this ablation because they are too similar NI. and commmonsense reasoning, respectively.

---

Published as a conference paper at ICLR 2022
4.2
SCALING LAWS
Performance on held-out tasks
70
Average zero-shot accuracy
Instruction tuning
As Brown et al. (2020) shows that zero and few-shot capabilities language models sub- stantially improve for larger models, next explore how the benefits of instruction tuning affected by model scale. Using the same cluster split as in the previous ablation study, evaluate the effect of instruction tuning on models of size 422M, 2B, 8B, 68B, and 137B parameters.
13 held-out tasks (%)
60
Untuned model
50
30
0.48
2B
8B
68B
1378
Figure 7 shows these results. We see that for the two models on the order of 100B pa- rameters, instruction tuning substantially im- proves performance on held-out tasks, as is expected given the prior results in our pa- per. The behavior on held-out tasks for the 8B and smaller models, however, is thought- provoking instruction tuning actually hurts performance on held-out tasks. One potential explana- tion for this result could be that for small-scale models, learning the ~40 tasks used during instruction tniig fills the entire model capacity, causing these models to perform worse on new tasks. Under this potential explanation, for the larger scale models, instruction tuning fills up some model capacity but also teaches these models how to follow instructions, allowing them to generalize to new tasks with the remaining capacity.
Model Size (# parameters)
Figure Whereas instruction tuning helps large models generalize
tasks, for small models it actually hurts generalization to unseen tasks, poten- tially because all model capacity is used to learn the mixture of instruction tuning tasks.
4.3 ROLE INSTRUCTIONS
In final ablation study, explore the role of in- structions during finetuning, as one possibility is that performance gains come entirely from multi-task fine- tuning and the model could perform just as well without instructions. We hence consider two finetuning setups without instructions. In a no template setup, only inputs and outputs were given to the model (e.g., for transla- tion the input would be "The dog runs." and the output would be "Le chien court."). In a datasel name setup, each input is prepended with the name of the task and dataset (e.g., for translation to French, the input would be "[Translation: WMT'14 to French] The dog runs.").
no instruction Eval: instruction
37.3
FT: dataset name Eval: instruction
46.6
FT: dataset name Eval: dataset name
47.0
instruction Eval: instruction (FLAN)
20
30
40
50
60
Zero-shot performance (4 task cluster avg.)
We compare these two ablations to FLAN's finetun- ing procedure, which used natural instructions (e.g.. " "Please translate this sentence to French: 'The dog runs."). We perform evaluations for four held-out clus-
Figure 8: Ablation study result using mod- els with instructions removed from finetun- ing (FT).
ters from Figure 5 For the no template setup, we used the FLANN instructions during zero-shot inference (because if we used no template, the model would know what task to perform). For models finetuned on dataset name only, we report zero-shot performance for FLAN instructions as well as using the dataset name. Figure X shows the results-bothh ablation configurations performed substantially worse than FLAN, indicating that training with instructions is crucial for zero-shot performance on unseen tasks.
4.4 INSTRUCTIONS WITH FEW-SHOT EXEMPLARS
So far, we have focused on instruction tuning the zero-shot setting. Here, we study how instruction tuning can be used when few-shot exemplars are available at inference time. The format for the few-shot setting builds on the zero-shot format. For some input x and output y, let instruct(x)) denote the zero-shot instructions. Then, given few-shot exemplars (æ,, y:) and new input Jr, the instruction format for the few-shot setting is "instruct(21) @ yn @ instruct(x2)) y2
7

---

Published as a conference paper at ICLR 2022
instruct(24)) yı instrruc((z)"" where + denotes string concatenation with delimiter token inserted in between. At both training and inference time, exemplars are randomly drawn from the training set, and the number of exemplars is capped 16 and such that the total sequence length is less than 960 tokens. Our experiment uses the same task splits and evaluation procedure as $ such that few-shot exemplars for unseen task only used at inference time.
As shown in Figure 9 few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN. Exemplars are especially effective for tasks with large/complex output spaces, such as struct to text, translation, and closed-book QA, potentially because exemplars help the model better understand the output format. addition, for all task clusters, standard deviation among templates is lower for few-shot FLAN, indicating reduced sensitivity to prompt engineering.
80
Zero-shot FLAN
Few-shat
Performance
60
54.7
593
59.6 80.0
63.8
53.7
49
30.2
40
33,0
20
Task Cluster:
Read. Closed Book
Confferance
Transiation
Struct
datasets:
Adding few-shot exemplars FLAN is complementary method for improving the performance of instruction-tun tunded models. The orange bars indicate standard deviation among templates, averaged at the dataset level for each task cluster.
Figure 9:
4.5 INSTRUCTION TUNING FACILITATES PROMPT TUNING
Untuned model
As we've seen that instruction tuning improves the ability of model respond instructions, follows that, if FLAN is indeed more amenable to performing NLPP tasks, then it should also achieve better performance when performing inference using soft prompts, represented by prepended continuous variables optimized via prompt tuning (Li & Liang 2021 Lester et al. 2021). As further analysis, we train continuous prompts for each of the SuperGLUE (Wang et al., 2019a) tasks in accordance with the cluster splits from $2.2 such that when prompt-tune ttning task T, no tasks in the same cluster as T were seen during instruction tuning. Our prompt tuning setup follows the procedure of Lester et al.] (2021) except that we use a prompt length of 10, weight decay of Ic-4, and did not use dropout on the attention scores; we found in preliminary experiments that these changes improved the performance of LaMDA-PT.
Instruction-tuned model
Performace after
100
prompt tuning
79.1
78.1
75
50
25
32 training Full training examples
set Figure 10: Instruction-tunel models respond better contin- uous inputs from prompt tuning. When prompt tuning on given dataset, no tasks from the same cluster as that dataset were seen during instruction tuning. Perfor- mance shown the average on the SuperGL dev set.
Figure 10 shows the results of these prompt tuning experiments for both using supervsuee training set and in low-resource setting with only 32 training examples. We see that in all sce- narios, prompt tuning works better with FLAN than LaMDA-PT. In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the LaMDA-PT. This result exemplifies in another way how instruction tuning can result in checkpoint that is more desirable for performing NLP tasks.
RELATED WORK
Our work relates to several broad research areas including zero-shot learning, prompting, multi-task
learning, and language models for NLP applications (Radford et al. 2019 Raffel et al.) 2020 Brown al. 2020 Efrat & Levy 2020 Aghajanyan et al.) 202 & Liang] 2021 inter alia). We describe prior work for these broad areas in an extended related work section (Appendix D). and here we describe two subareas narrower in scope that perhaps relate most closely to work.
8

---

Published as a conference paper at ICLR 2022
The way we ask model to respond to instructions is similar QA-based task formulation (Kumar jet al. [2166] McCann et al. (2018). which aims to unify NLP tasks by casting them as QA over a context. Though these methods are very similar ours, they mostly focus multi-task learning instead of zero-shot learning, and-as as noted by Liu et al. - are generally not motivated by using existing knowledge in pretrained LMs. Moreover,
work supercedes recent work such as Chai et al. (2220) and Zhong et al. (2021) in terms of both model scale and scope of tasks.
The success of language models has led to nascent research on the ability of models follow instructions. Most recently. Mishra et al. (202T) finetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks-this is similar to our few-shot instruction tuning result from $4.4 This promising result (as well as one from Ye et al. (2021), which does not emphasize instructions as much) suggests that finetuning on collection of unseen tasks, even at smaller model scale. [Sanh et al. (202T) finetune '55 in a setup similar to ours, finding that zero-shot learning can be improved in a model of 11B parameters. At model scale similar to ours, OpenAl's InstructGPT models are trained via both finetuning and reinforcement learning to produce outputs that are more preferred by human raters (Ouyang al. 2022
tasks improves few-shot performance
6 DISCUSSION
Our paper has explored simple question in zero-shot prompting: does finetuning model collection of tasks phrased as instructions improve its performance on unseen tasks? We operationalize this question via instruction tuning, simple method that combines appealing aspects of both the pretrain-finetuee and prompting paradigms. Our instruction-tungt model, FLAN,, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on. Ablation studies reveal that performance on unseen tasks improves with the number of instruction tuning task clusters, and, interestingly, that performance improvements from instruction tuning emerge only with sufficient model scale. Moreover, instruction tuning can be combined with other prompting methods such as few-shot prompting and prompt tuning.
The diverse capabilities of language models at scale have drawn attention to the tradeoffss between specialist models (one model per task) and generalist models (one model for many tasks: /Arivazhagan et al. 2019 Pratap et al.] 2020), for which our study has potential implications. Although one might expect labeled data to have the most natural role in improving specialist models, instruction tuning demonstrates how labeled data can be used to help large language models perform many, unseen tasks. In other words, the positive effect of instruction tuning on cross-task generalization shows that task-specific training is complementary to general language modeling and motivates further research on generalist models.
As for limitations of our study, there is degree of subjectivity in assigning tasks to clusters (though we try to use accepted categorizations in the literature), and we only explore the use of relatively short instructions of typically single sentence (c.f. detailed instructions given wooke-w A limitation for our evaluation is that individual examples might have appeared in the models' pretraining data, which includes web documents, though in post-hoc analysis (Appendix ( do not find any evidence that data overlap substantially impacted the results. Finally, the scale of FLAN 137B makes it costly to serve. Future work on instruction tuning could include gathering/generattinn even more task clusters for finetuning, cross-lingual experiments, using FLAN to generate data for training downstream classifiers, and using finetuning to improve model behavior with respect to bias and fairness (Solaiman & Dennison 2021).
7 CONCLUSIONS
This paper has explored simple method for improving the ability of language models at scale to perform zero-shot tasks based purely on instructions. Our instruction-tunel model, FLAN, compares favorably against GPT-3 and signals the potential ability for language models at scale to follow instructions. We hope that our paper will spur further research on instructions- based NLP, zero-shot learning, and using labeled data to improve large language models.

---

Published as a conference paper at ICLR 2022
ETHICAL CONSIDERATIONS
This work uses language models, for which the risks and potential harms are discussed in Bender & Koller (2020), Brown al. 2020), Bender al. (2021), Patterson al., (2021), and others. As our contribution in this paper is not a pretrained language model itself but rather an empirical study of how instruction tuning affects the zero-shot performance of language model on unseen tasks, we additionally highlight two relevant ethical considerations. First, labeled datasets such as those we use for finetuning can contain undesirable biases, and these biases can be propagated into zero-shot applications of the model on downstream tasks. And second, instruction-tuned models can potentially require less data and expertise use; such lower barriers to access could increase both the benefits and associated risks of such models.
ENVIRONMENTAL CONSIDERATIONS
We use the same pretrained language models as Austin et al. (202I). The energy cost and carbon footprint for the pretrained models were 451 MWh and 26 (CO2e, respectively. The additional instruction tuning gradient-steps for finetuning FLAN is less than 2% of the number of pretraining steps, and so the estimated additional energy cost is comparatively smaller.
AUTHOR CONTRIBUTIONS
Maarten Bosma conceived the original idea and implemented the first version of FLAN. Vincent Zhao prototyped the training and evaluation pipelines, well as rank classification. Kelvin Guu proposed and implemented the idea of task clusters and evaluation using inter-cluster splits. Jason Wei, Maarten Bosma, Vincent Zhao, and Adams Wei Yu implemented the NLP tasks. Jason Wei, Vincent Zhao, and Adams Wei Yu conducted and managed most of the experiments. Jason Wei designed and ran the ablation studies. Jason Wei, Maarten Bosma, and Quoc V. Le wrote most of the paper. Jason Wei, Maarten Bosma, and Nan Du obtained the zero and few-shot baselines. Vincent Zhao and Kelvin Guu designed, implemented. and conducted the few-shot FLAN experiments. Maarten Bosma and Jason Wei ran the data contamination analysis. Brian Lester ran the prompt tuning experiments. Quoc V. Le and Andrew M. Dai advised, provided high-level guidance, and helped edit the paper.
ACKNOWLEDGEMENTS
We thank Fd Chi, Slav Petrov, Dan Garrette, Ruibo Liu, and Clara Meister for providing feedback on manuscript. We thank Adam Roberts, Liam Fedus, Hyung Won Chung, and Noam Shazeer for helping debug some of our models. We thank Ellie Pavlick for feedback on the study design during the middle stages of the project. We thank Daniel De Freitas Adiwardana for helping initiate the project, large language model advising, and giving us access to some computational resources. Finally, thank the team involved in pretraining LaMDA-PT: Daniel De Freitas Adiwardana, Noam Shazeer, Yanping Huang, Dmitry Lepikhin, Dehao Chen, Yuanzhong Xu and Zhifeng Chen.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre- fe-einetuning. arXiv preprint arXiv:2101. 2021.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine 2019. URL
translation in the wild: Findings and challenges. arXiv preprint
httpss:/aaxiia wttg/a///ggg// 9977 05019
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07700 2021. URL htttttts:/// /ttp//aavvvv///////////////aaav/ org/ /bs/
2108.07732
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp- 355-362 2011. /aLLattttp////clanhoooorrr wrt/ 1033
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ráncrez- Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. ParaCrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp.
2020. J / c www.hhooo...orggoooooo
-ma i 417
Emily M. Bender and Alexander Koller. Climbing towards NLU: meaning, form, and under- standing in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5198, 2020. URL bettss:/accanttant
org/
22200.ac1 maaac--main.. main. 463
Emily M. Bender, Timnit Gebru, Angelina SaMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency, FAceT 221, pp. 610-623. Association for Computing Machinery, 2021. URLhttps:///oi..oooo org.ii.rgg/1 11111011114111111111
3445922]
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The Fifth PASCAL Recognizing Textual Entailment Challenge. In TAC, 2009. URL http:t/t/////eeerrr 232 123 wet..:/11rre=eppptyyeeeeeee
ist psu. edu/
v (tweddd:///owlooddddoooooo
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Thirty- AAAI Conference on Artificial Intelligence, 2020. URL /tttts://aaxiiii org/ abs/ /191l 11641
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia (eds.). Proceedings of the Ninth Workshop Statistical Machine Translation, 2014. URL
org/wl 3300
Ondicj Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Verspoor, Jörg Tiedemann, and Marco Turchi (eds.). Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, 2016. wttptttps//acccloo
org. (argg//16220000 hww.lllyyyyooggogooooo
Rishi Bommasani, Drew A. Hudson, E, Adeli, R. Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, E. Brynjolfsson, S. Buch, D. Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, S. Ermon, J. Etchemendy, Kawin Ethayarajh, L. Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Good- man, S. Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho,

---

Published as a conference paper at ICLR 2022
Jenny Hong, Kyle Hsu, Jing Huang. Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, G. Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, M. Krass, Ranjay Krishma, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, J. Leskovec, Is- abelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, D. Narayanan, Ben Newman, Allen Nie, J. C. Niebles, H. Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, C. Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, D. Sadigh. Shiori Sagawa, Keshav Santhanam, Andy Shih, K. Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, M. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunitics and risks of foundation models. arXiv preprint arXiv:2108.07233 2021. URL https: wrg/ abs / 2108. 07258
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015. URL https
/aclanthology org/D15 1075
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel -osbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 1877-1901, 2020. URL httpps::/ppoceeedinggeeee/ igss neur
neur 1 ps 1371-1382. PMLR, 2020. URL ottp://0000000......160.001.1a1. 02 Chai20a.pdF.
Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. Description based text classification with reinforcement learning. In Proceedings of the Infernational Conference Machine Learning, pp.
ngs. wtt.s...r..rrr/////.. //tee httpp/////1caaaaaaaa/a
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021. URL httthtttps://aatt www.aaawwiv.rrarraaaaaoaoo abs/ /2107
03374
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184, 2018. URL https
//aclantho wow..hooooyy..ooo www.llloyooooogoooo org/D18 1241
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936. 2019a.
pnEtts://accanteee www.eaaahhoooo.oo.oooooooo 9-000
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le.
BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5931-5937, 2019b. URL ttttps://aclantho.oo. wwwpsogg.orrg..oo. wttp:ooggpp..o..ooo 1595
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URLhtttps: /ttp::////xvvvv org/ aba/aaas/100 05457
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning
12

---

Published as a conference paper at ICLR 2022
Research, collobertilaaccoloberrtt pdf
2011. URL https://wwws http://wwww..iiro/rappppptt/r///
Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. Hybrid emoji-
based masked language models for zero-shot abusive language detection. Findings of the Association for Compulational Linguistics: EMNLP 2020, pp. 943-949, 2020. URL https
//aclanthology.o wwt.:/loooo..ooaaooooooooo/ 2020. Bee.eeiiiiniggge Lindinngsseenpp....
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL Recognising Textual Entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entail- ment, MLCW"OS 05. pp. 177-190, 2005. URL htttps://do/ http:///oooooogg//./ 10000.100011111
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Proceedings of the Confer- ence on Neural Information Processing Systems, 2015. URL htttp:://papezzs:: nips/ eC/ paper/2015/
95300
95301
pdd.
Carie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, pp. 107-124, 2019. URL https://ojs ubo uni -ookkkonttttnnn de / sub/ index. phpp sub/ piricle/view. 601
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019. URL
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005. URL https://acllantho..n
org/ I05-002
Nan Du Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mfxture--f-exxpets. arXiv preprint arXiv:2112.069005 2021. URL https://
arriv.ore/paf/212200000
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.
2368-2378, 2378, 2019. Utt.httss: //acl:://ccccc antho ogy. org/Nl 9- 124
Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for WMT-14. In Proceedings of the Ninth Workshop Statistical Machine Translation, pp. 97-104, 2014. (ttht:::: /at/aaaaannttttco .rr.oorgggooo.ooo /W14///14--- 3309
Ondrej Dušek, David M. Howcroft, and Verena Rieser. Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 421-426, 2019. URI. http:::tt:::/aat. http://cllattttt log. 9- 8652
Sergey Edunov, Myle Out, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 489-500, 2018. LRL httppsss//ccaaattoooooo.oo org/ 1045
Avia Efrat and Omer Levy. The Turking Test: Can language models understand instructions? arXiv 11982
preprint arXiv:2010.1 10.00.199980 1982, 2020. https::///
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale summarization dataset and abstractive hierarchical model. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, 2019. URL https: / antho lot..org/P19-110000//
13

---

Published as a conference paper at ICLR 2022
Fast.AL. Yelp Sentiment Classification Dataset.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter
models with simple and efficient sparsity. arXiv preprint arXiv:2101.03990 2021. URL/ttps
//arxiv. org/ awwpaasssoo1 03961
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1126-11135 2017. Utttttss::/arrt
.rq/ htspg//bb111033 11t.. 03400
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3816 1886-3830, 2021. URL httttt::/tclaaataaaet /ttps//aac..attoooo..org//////// acl-long 295
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data, In Proceedings of the 10th International Conference on Natural Language Generation, pp. 124-133, 2017. URL (ttthtts:: h/aaa/ccaanttooooooooo org/
1177-5118
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jermite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Berez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank San- thanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp- 92-120, 2021. URI. httph:tp::.. //tc//acc antho og...og../...................... 2etp//2221...mm11...
ggm- 1.01 10
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1-9, 2007. Utt|https: h/tp://allattta ogy . .arg///gggoo /000-//0--14
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summurization, pp. 70-79, 2019. URLhtttpa http::s///aaaaahhlll
org/
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009. URL htttp:://wwwwww/ cs stan ford
pdf
Dan Goldwasser and Dan Roth. Learning from natural instructions. Machine learn-
ing, 94(2):205-232, 2014. htlthttps:/1iimm http::::/iinkksspiiig.g..o wtt.o///ariclee1.110000 610994-013-5407-1
Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 708-719, 2018. htl.attts.//aaaatttclli/ www.aaattiooooyoooo/---------- www./ooogg/oooooo
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The Second PASCAL Recognising Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. URL
http:// /wwww//www///////////////ww Cs bwu.bi..... wtt....1/sssppeees /ttppppptt//pppaa///RR//... watpaa/s//tteooggniiiaa wrwpmoooogggzzzzziieie
14

---

Published as a conference paper at ICLR 2022
Luheng He, Mike Lewis, and Luke Zettlemoyer. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp.
2015. URL https
V/aclantho 1 ogy. wwtp/ogg/////0/0/
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn' 't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. URL https
//aclantho wgw.oogg.o.ooooo...o............ 2021. emn p ma m..mann.566 564
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. To- ward semantics-based answer pinpointing. In Proceedings of the First International Confer- ence on Human Language Technology Research, 2001. httttp::/www..cllltt.
aathology/HO1-1065
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 328-339, 2018. URL https: / //a/aclanntt www.annhhlogy...oooooooooooooo org PPs//1111111111111
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNP)), pp. 2391-2401, 2019. URL https://
acc aclanthology. 0rg/D19-12499
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's s multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. URL httips:/aclaathoolory
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, 2017.
Logy. wtt./orgpppp.... wwwpaanhhwooooo.ooooo.oeo11..1
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 252-262, 2018.
URL https: / /ttpa/aaaaanooaooooooaaaaoooaa.a www.y.org/N113 //gg/N115 1023
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 199,
URL
pp.171 171
Dimitrios Kotzias, Misha Denil, Nando de Freitas, and Padhraic Smyth. From group to individual labels using deep features. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015. URL htttps://dl. acm. org 2os//0ii11
1145/278325 0015558.78333 2783380
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pp. 66-71. Association for Computational Linguisties, 2018. doi: 10. 10.118633/1/d11--200 URL
Ankit Kumar, Ozan Irsoy, Peter Ondruska. Mohit Iyyer, James Bradbury, Ishaan Guirajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for the International Conference on Machine Learning. pp. 1387. PMLR, 2016. htI.httss: /wt.xvivvv./.. http:////aww.ooo/aa///////////// 07285
natural language processing. In Proceedings the
15

---

org/ Published as a conference paper at ICLR 2022
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 2019. URI.httss: //acl //ap://aaaaaaaoaaoaaaaa logy
erg/Q133 0-9-1111111 1026
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pp. 4034 4048, 2020. URL https? emn //aclanthology wgw.looy..org//2000/ 1o220.finnce w
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 951-958. IEEE, 2009. URL htttps://ieeexp/leet
Bocument/52065.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4483-4499, 2020. URLLAttts///ccaaachooooo .wg. wwt.o//g//222 otg.gyy.......2222.mmm...mmm....
emnlp-main. 363
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 6086-6096 2019. URL holhtts:: / /tt//aaaantttt http://aaaaaatttt........ org/
P19-1612
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, net
2020. URL / http://wwww.eeettoteeeeeeeooo-/e forummid-u 7XHImbb
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for -fficcien prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2021. htt.httss///axiv...et awtpr///////aa//eeee awl.aaab...00000 08691
Hector Levesque, Emest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. 2012. URL TRLhttps: //dl. acm. org/ do1/ / 10. 5555/ 005555/333355 3055//33114333 000031883333319000 3031909
Omer Levy, Minjoon Seo, Hunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333-342, 2017. URL https:://clantthooogg
org/K17-103
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequee teuuueneeeeeeeeeeeeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020. URL https: bettps:/acclatthnn http:///aaaantohoooooooo wwwp:/ttww.ooo..ooooo22/o00200
wwt.:ogg./oooo.. acl main.. 703
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In
Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the
I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4582-4597, 2021. URL http:////...ttttta//aaaaaaatttoa
org/ 2021 acc long.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 3849-5859, 2020. URL https
7/aclantho ogy. ogy.org//000.ooom acl -ma main..59
16

---

Published as a conference paper at ICLR 2022
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Confer- ence on Computational Linguistics, 2002. (tlllattps:://w..all. www.w/oowwwwe////ooooo.ooo. /ntpoloohhoo
C02-1150
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823-1840, 2020. URL httph:tt///et aaaaanaooooooo wwtp:ogg...o.... findings emn 1p. 165
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert Y.S. Lam. Reconstructing capsule networks for zero-shot intent classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4799 4809, 2019a. URL 1486
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2 13586, 2021. Jttpp::::://arx orgg abs 2107 13586
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, 2019b. URLhtttss / acl aaatoo org/
P19-14H 1441
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre- pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726 742, 2020. (thhttps
7/aclanthi www.anntooooyyyooooooo ogy. org. /2lggg/22202mm tacl-11.7
Minh-Thang Luong, Quoe V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. Proceedings of ICLR, 2016. URL https::/arxiv.or
abs/1511. 06114
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint wxXiv1806.008330, 2018.
URL http:::://axxv..org...o/
08730
John McCarthy. Programs with common sense. RLE and MIT computation center, 1960. URL pdd
http://
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct
electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018. URL Bttts:://aclanthooooy...o org/D18 1260
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaiel: Learning to learn in context. arXiv preprint 1rXiv:2110.1594:: 2021. URL htttttttp///xxit/ org/ /bs/////////// 2210
15943
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Natural Instructions: Benchmarking generalization to new tasks from natural language instructions. arXiv preprint arXiv:2104.08773, 2021. httl/httts///rxiiiee
httpg//bs/210
08773
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, 6-1--199
2016. htRl/https/:/acaatttoootoo
17

---

Published as a conference paper at ICLR 2022
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan. Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. DART: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 432-447, 2021. URL https:t/////..m battts://aclanthoooo org/
2021
Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. Annotated Gigaword. In Pro- ceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC- WEKEX), pp. 95-100, 2012. URLLattps//accaanhh..oo
org/w12-3011
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018. URL https: /tap:://aclle antho I Logy wrg.ooggg//...oo 1-18--1206
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial
NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4885-4901, 2020. URL https
//aclanthology wwcoogy.oooo/oooooooooooooooo.oo org. 2020 acl -aaimmain.. 441
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Preprint, 2022. URL https::///// //cdn. cdn
pdf loco to fol ow www.instruetiinnsesseeese wi www.thhhhha__eeeeebeeeeee
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2227-2237, 2018. https!
V/aclanthology:
Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alexander Waibel. Improving zero-shot translation with language iraguagg-indeendeen constraints. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 13 2019. URL /tLLtttps///aaantoool.
org/ /1////133 9- 5200
Mohammad Taher Pilchvar and Jose CollchosCollados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Hu- Language Technologies, Volume I (Long and Short Papers), pp. 1267-1273, 2019, URL 1128
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Massively multilingual ASR: 50 languages, I model, billion parameters. arXiv preprint arXiv:2007.03001, 2020. URL
03001
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computationall Lingusticss Human Language Technologies (NAACL-HLT), pp. 5203-5212, 2021.
UtR.b/tt:://c. http:://cs..hueeee///s/s////////
-tw.s//tt-nnnnnnn--n--n----- sner-
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving openai com/
language understanding by generative pre-training.
larquage-unsuperrisse 2018.
18

---

Published as a conference paper at ICLR 2022
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever.
Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL
h /v2////002.///EEEE....
Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 777-789, 2012. URL
httDs::/acllattoooogyyooooo wtt.hhhooooo.oor/1--/100000
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 2016.
http:////aaaaatttoooooo org/
D16-1264
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018. URL https: patttp::/aacannth..oo
org/P18-212
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249 2019. URL
ftttss://aLaattooloy.. wrg/ Q19 1016
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. A recipe for arbitrary text style transfer with large language models, arXiv preprint arXiv:2109.03910, 2021. PtLpttts: org/ abs 221./.1119999110 03910
Melissa Roemmele, Cosmin Bejan, and Andrew Gordon. Choice plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium Series, 2011. URL index. https//s////SS////pppe//////////
https: httpp::/wwww... www.. www.aaaa.....oooaaaaaaaeeaoooooe wtt//////////..e.e.o./////////e.
Bernardino Romera-Paredes and Philip Tor. An embarrassingly simple approach zero-shot
learning. In Proceedings of the International Conference on Machine Learning, pp. 2152-2161, 2015. (ttp:ttts://ppooeeeee
ngs ml wptess/v37. 3
Sebastian Ruder. An overview of multiitask learning in deep neural networks. arXiv preprint arXiv: 2017. URL http:///tt/////rrrar..o/////////
05098
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adver- sarial winograd schema challenge at seale. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.
2020. URL
org/ abtp//////111110 10641
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. Proceedings of the International Conference on Learning Representations, 2021. URL htttttt////xxiii org/ abs 2110. 08207
David Saxton. Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. Proceedings of the International Conference on Learning Representations, 2019,
Timo Schick and Hinrich Schütze. Exploiting cloze cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021. URL
https: / //acl antho www.anhwwwoooooo/ao.... eacl -ma i n.20
19

---

Published as a conference paper at ICLR 2022
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generate networks. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 1073-1083, 2017. URL https:7/
ac sellanhholgge ogy. www.s/oorg//1111999900
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. Proceedings of the First Conference on Machine Translation; Volume 2. Shared Task Papers, pp. 371-376 2016.
ltttps://aclanthoooooy. org/ -
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018. 8. URL.
https://arxiie www.wxiww..vv.o.ooooo wrggabs/ 1804 04235
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, anthology wrgg//3-1g/111
pp. 1631-1642, 2013. URL
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with talues-targeted datasets. arXiv preprint vrXiv:2106.10322 2021. URL URL/attps://rxxxvvoor
Shashank Srivastava, Igor Labutov, and Tom Mitchell. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 306-316, 2018. URL https://
aclanthology org/P18 10299
Derek Tam, Menton Rakesh R., Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. URL httptttttss//....ttttttr//
2103.11951
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzi -ze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint ar (xXi:22001008333 2022. URL bttps::/arxix...... org/
22C1.
Joaquin Vanschoren. Meta-learning A survey, arXiv preprint arXiv:1810.03548 2018. URL 03548
Marc Velay and Fabrice Daniel. Seq2seq and multi-task learning for joint intent and content extraction for domain specific interpreters. arXiv preprint arXiv:/808.004 004233 2018. URL https
Sarxiv.orrg/abs/1800. 00423
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 355-355, 2018. URL htthhtt::/// htttps://aclanthol.oo 0rw.://--.r///1-5555555
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Conference on Neural Information Processing Systems (NeurIPS), 2019a.
URL https: http:://axxvv.. org/ abs/ 1905. 00537
Lu Wang and Wang Ling. Neural network- nesed abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 47-57, 2016. URL https
// aclweb. org/ htwpb/ogggaatthoggggo..o anthology N1 6-1666100 6-1007
Yiren Wang, Yingee Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Multi- agent dual learning. In Proceedings of the International Conference on Learning Representations (ICLR) 2019, 201 9b.
20

---

org/ anthology/ Published as a conference paper at ICLR 2022
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-labl language learning. arXiv preprint arXiv:2109.0911 2021. URL
09193
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics. 7:625-641, 2019. doi: 10.1162/ tacl a 00290.
org Q19 1040
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 19902, 2022, htthttss: //apps://aaae http://axxvvvv/.. www.aiiwwiggaiaaaaaeeaaaa wwtppr/gppp.....o//////// 11000.111900 11903
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, 2018. URL URLhttp: chntt:://cclla
N18-1101
Joseph Worsham and Kalita, Multi-task learning for natural language processing in the 2020s: where are we going? arXiv preprint arXiv:2007.1000 2020.
org/
abs/2007.16000
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.1086.
2021. URL https://:///at wrw/ 09. 10862
Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6953-6963, 2020.
org/
2020. acl main. 622.
Qinyuan Yc, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. URL htttt::///taii...ttp////
abs / 22104.100000 08835
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJNLP),, pp. 3914-3923, 2019. URL
D1 D19-1404
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4800, 2019. URL httph:://///
/ http://aaaannnooo.ooor.o.//
P19-1472
Rui Zhang and Joel Tetreault. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. URL hattps://aclanee http:///wwaaatttloo.ooorog////// orgp:1111...//99001//
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. CoRR, abs/ 118.810..8....... 12885, 2018. URL http: /tRLl//tp::/axxx..orz///////////
12885
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level comvoluttoooioo networks for text clas- sification. In NIPS, 2015. URL
Yuan Zhang, Jason Baldridge, and Lubeng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
pp. 1298-1308, 2019. httt:ttss: peattp::/accannhoologe www.hhoooooo..ooo/ooooo
wwtp//111///////3000111 1131
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Meta-tuning language models to answer prompts better. arXiv preprint arX1v22104.0460, 2021. URL https: //tpx//avvv///////////e/////aav www. abs/ 21 04670)
21

---

Published as a conference paper at ICLR 2022
A ADDITIONAL RESULTS
This section shows the full results for all datasets we evaluate, Results for translation and struct to text are shown in Table [
and the results for eight NLU task clusters are shown in Table
We show FLAN's performance using the best of up to ten instruction templates as well as the template with the best performance on the dev set. For LaMDA-PT, we the templates from [Brwnn al. (2020), which were optimized for GPT-3, without performing any prompt engineering to optimize them our model. For simplicity. we use greedy search for all generative tasks (compared with beam search used in Brown et al. 2020).. Unlike GPT-3, which chooses the number of few-shot exemplars & via best dev set performance, for few-shot LaMDA-PT we choose the highest ie that fits in the context length of 1024 tokens, from k E (1,3, 10).
For DROP (Dua et al. 2019) and SQuADv2 (Rajpurkar et al. 2018), based on email correspondence with (rronn et al. (2020), their definition of zero-shot differs from ours in that they actually use exemplars, but only from the same passage as the inference question (each passage has more than one question). Hence, GPT-3 zero-shot results not directly comparable with ours for DROP and SQuADv2. We mark these results using the symbol. Moreover, it is unclear how to parse the end of answer for these datasets, and use curly bracket delimiters and ), where expect to indicate the end of the answer.
For struet to text, reported T5/mT5 results are from the GEM benchmark paper (Gehrmann al. 2021). though we do not report their results for DART (through correspondence with authors, we confirmed that their results for DART were incorrect). Though we summarization task cluster during instruction tuning, we leave evaluation of summarization for future work, as the mean input of most summarization datasets exceeds FLAN's input length of 1024 tokens.
FLAN 137B
LaMDA-PT GPT-3 175B
zero-shot
few-shot
Metric
Supervised ZCTO- fow- Model
shot shot [k] shot shot (k]
Zero- few-
average template
best dev template
average template
template [k]
best dey
TRANSLATION
WMT '14 En-Fr
BLEU
35.0
11.2
31.5
151 25.2 32.6
32.9
33.9
33.9
33.8
WMT Fr- BLEU
45.6
7.2
34.7
[S] 21.2
39.2
35.5
35.9
38.0
37.9
[9]
WMT "16 En-Dee BLEU
38.6'
7.7
26.7
15 24.6
29.7
(64]
25.4
27.0
26.8
26.1
[1]]
WMT 16 De-*En BLEU
41.2"
20.8
36.8
27.2 40.6
38.9
38.9
40.6
40.7
[I]]
WMT '16 En-Ro BLEU
39.99
3.5
22.9
1s1 14.1
21.0
16.7#16
18.9
20.5
20.5
[9]
WMT 116 Ro->E BLEU
38.5
9.7
37.5 1S1
19.9
39.5
(64)
36.8
37.3
38.2401
191 3
STRUCT TO TEXT
CommonGen
Rouge-1
64.0"
3.9
56.7 131
54.6+23
56.6403
56.4
[16] 6
Rouge-2
29.4
1.5
29.6
28.8
27.6
30.9
29.9
[16]
Rouge-L
54.5
3.2
48.5
48.4
48.7
50.7
51.0
[16]
DART
Rouge-1
11.3
56.0
45.5+42
48.9
57.9416
59.2
|11]
Rouge-2
1.5
29.6
25.0+337
30.0
35.8+10
36.2
111
Rouge-L
3.2
48.5
38.4+33
43.4
48.5
48.2
E2ENLG
Rouge-1
72.6"
6.2
56.7 BJ
44.8139
51.4
59.141.
59.7
[12]
Rouge-2
47.5"
2.5
31.4 []
24.2433
30.1
33.221.1
33.6
[12] 9
Rouge-L
56.4a
4.9
41.1 []]
37.0+35
42,4
44.9108
45.1
[12]
WebNLG
1ouge-1
83.5a
13.9
68.3 1I
50.6447
57.7
68.5422
71.2
[10]
Rouge-2
63.6"
6.9
46.0 [3]
29.8+42
35.4
48.0+1.5
49.8
[10]
Rouge-L 71.0"
11.8
56.5 [I]
43.4+45
49.7
58.8+11
60.2
[10)
Table 1: Results for translation and struct-to-text tasks. [M] indicates the number of few-shot
exemplars. # indicates the number of templates that FLAN evaluated on. TT511BB (Edunov et al.
"T5-11BB_Edunoo
(2018). Durrani et al. (2014). <Wng et al. (2019)), Sennrich al. (2016). Liu al. (2020).
22

---

Published as a conference paper at ICLR 2022
FLAN 137B
GLaM
LaMDA-PT GPT-3 175B
zero-shot
few-shot
Random Supervised zero- one- zero-
Guess
Model
shot shot shot shot
few-
zero-
shot
fow- shot [4]
average template
template
average template
template [k] At
dev
NLI
ANLI
33.3
57.4
40.9
42.4 39.6
15]
34.6
36.8
(50]
47.7
46.4
44.2.12
47.9
[6]
8
ANLI R2
33.3
48,3
38.2
40.0
39.9
37.5 1S
35.4
34.0
[50]
43.9
44.0
41.6114
41.1
[6]
ANLI
33.3
43.5
40.9
40.8
39.3
40.7 151
34.5
40.2
(50)
47.0+11
48.5
42.8+22
46.8
[6]
CB
33.3
93.6"
33.9
73.2
42.9
34.4 15]
46.4
82.1
(32)
64.1
83.9
82.614.
82.1
[7]
10
MNLI-m
33.3
92.2"
35.7
43.7
51.1
61.2
60.8
63.5
MNLI-mm
33.3
91.9"
37.0
43.8 151
$1.0#65
62.4 61.0+3s
63.5
[10]
QNLI
50.0
96.9"
50.6
55.7
15]
59.6249
66.4
62.0+17
63.3
[12]
RTE
50.0
92.5
68.8
71.5
73.3
70.8
15]
63.5
72.9 (32)
78.3
84.1
79.9
84.5
[8]
SNLI
33.3
91.3*
33.3
54.7
15
43.0+74
53.4
62.3+24
65.6
[15] 9
WNLI
50.0
94.5"
56.3
64.8
61.0 10.6
74.6
11.4
70.4
[14] 10
REAdING COMP.
BoolQ
50.0
91.2"
83.0
82.8
81.0
80.0
60.5
77.5 (32)
80.2
82.9
83.6108
84.6
[4]
9
DROP
80.5
54.9
55.2
3.8
10.3
23.6
36.5 120)
21.9199
22.7
22.3111
23.9
MultiRC
88.1"
45.1
62.0
60.0
59.6 [5)
72.9
74.8
(32)
74.5
69.2
72.1
[I]
8
OBQA
25.0
85.4"
53.0
55.2
41.8
50.6 [10)
57.6
65.4 (100)
17.4
78.4
77.2+13
78.2
[16]
7
SQuADvl
96.2"
22.7
50.2 [3]
79.5
80.1
82.1
82.7
SQuADv2
83.4
68.3
70.0
11.1
34.9 (8)
59.51 69.8 (16)
40.9
44.2
40.8
43.1
[3] 10
CLOSED-BOOK QA
ARC-c
25.0
81.1"
48.2
50.3
42.0
49.4 [10]
51.4
51.5
[50]
61.7
63.1
63.7+a6
63.8
[13]
ARC-e
25.0
92.6"
71.9
76.6
76.4
80.9
68.8
70.
[50]
79.5
79.6
80.5
80.7
[14]
NQ
36.6"
21.5
23.9
3.2
22.1 [5]
14.6
29.9
[64] 18.6
20.7
27.24a5
27.6
[16]
TQA (wik))
60.5a
68.8
71.5
21.9
63.3
[10)
64.3
71.2
(64]
66.5
68.1
66.5+10
67.3
[16] 10
TQA (Ldds dev)
51.0"
18.4
55.1
[10)
55.0
56.7
57.2
57.8
[16]
COMMONSENSE
COPA
50.0
94.8*
90.0
92.0 90.0 89.0 [10)
91.0 92.0 (32) 90.6+20
91.0
88.5+38
87.0
HellaSwag
25.0
47.3
77.1
76.8 57.0 58.8 (10)
78.9 79.3 120) 56.4
56.7
59,4
59.2
[3]
PIQA
50.0
66,8
80.4
81.4
80.3*
80.2" [0)
81.0 82.3
[50] 80.9 208
80.5*
82.1 200
81.7"
[10]
StoryCloze
50.0
89.2
82.5
84.0 79.5
83.7 (10)
83.2 87.7
(30) 92.2+1.
93.3+a9
94.7
[10]
SENTIMENT
IMDB
50.0
95.5
76.9 83.3 (1]
94.1 1:04
94.3
94,8103
95.0
Sent140
50.0
87.0
41.4 63.3
12.
73.5
68.7111
69.3
[16] 6
SST-2
50.0
97.5"
51.0 92.3 [S]
71.6 95.6 (8)
92.6#17
94.6
94.44as
94.6
[16] 8
Yelp
50.0
98.1*
84.7 89.6
97.8+02
98.1
97.9
98.0 [4]
PARAPHRASE
MRPC
50.0
90.4
53.7 64.0 [5]
69. 113
69.1
67.5
67.2
[10]
QQP
50.0
90.6"
34.9 58.9 [3]
72. 16.
75.9
73.5
75.9
[16] 7
PAWS Wiki
50.0
91.9"
45.5 53.5
61.5
69.4
66.24a9
70.2
[10]
COREFERENCE
DPR
50.0
8
54.6 57.3 [5]
60.3
66.8
62.4116
63.3
[16]
Winogrande
50.0
65,8
73.4
73.0 68.3 68.4 [10)
70.2
77.7 (50)
67.3
71.2
72.3109
72.8
[16]
WSC273
50.0
70.0*
86.8
83.9
81.0 61.5
88.3 88.5 (32)
80.8
[-]
READ. COMP. W/ COMMONSENSE
CosmosQA ReCoRD
25.0
1
34.1 33.8 [S]
58.4
60.6
56.7
56.0
93,4" 90.3 90.3 87.8" 87.6"
90.2 89.0 (32) 67.8 110 72.5*
77.0" =10 79.0"
Table Results for eight ULU task clusters. All values shown are for accuracy (or exact match) except DROP, MultiRC, and SQuAD v1 and v2. which are F1. indicates the number of few-shot exemplars. # indicates the number of templates that FLAN is evaluated on. TT.""T5-113 'BERT-large. "see data contamination (Appendix C WSC273 (Levesque et al. 2012 does not have training validation sets, and so we do not compute few-shot results for FLAN. For Trivia QA (TQA), we report exact match (EM) both the wikipedia subset of the dev compare with GPT-3, well as the full TFDS dev set.
23

---

Published as a conference paper at ICLR 2022
B FURTHER ABLATION STUDIES AND ANALYSIS
B.1 DATASETS PER TASK CLUSTER & TEMPLATES PER DATASET
Our primary hypothesis is that instruction tuning on diverse set of tasks improves performance on unseen tasks. $4. showed that adding more task clusters improves performance; here, we further explore whether adding additional datasets improves performance when the number of task clusters is held constant. We use the same split as in $4.1] where the NLI, commonsense reasoning, and closed-book QA clusters are held-out, and seven other task clusters remain for instruction tuning. For these seven task clusters, we instruction tune models using just one dataset per task cluster and using four datasets per task cluster (for task clusters that did not have four tasks, we just used all available tasks). In addition, we simultancously explore the role of the number of instruction templates per dataset; as mentioned in $2.1 for each dataset we manually composed ten instructional templates for instruction tuning. Here, instruction tune models using 4, and 10 templates per dataset.
Figure I1 shows these results. Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters. Using more templates per dataset, however, had a comparatively negligible effect on performance when there was one task per cluster, which disappeared when there were four tasks per cluster. The small effect of templates is striking given original motivation that composing ten templates per task would mitigate overfitting to any particular template. This results serves underscore, however, the unpredictability finetuning large language models, one hypothesis is that models at such scale do not easily overfit to finetuning single task.
75
Datasets per task
70.1
70.5
69.9
cluster
Datasets per task cluster 4
Performance
65
61.6
61.9
61
60
Templates per dalaset
Figure 11: Effect of datasets per task cluster and templates per dataset on performance on three held-out clusters: NLI, commonsense reasoning, and closed-book QA. Adding more datasets per task cluster substantially improves performance. Using more templates per dataset, however, only had a very small effect on performance, which disappeared when there were sufficient dataset per task cluster.
B.2 ROLE OF INSTRUCTIONS DURING FINETUNING
The per-cluster results for the ablation study from are shown in Table 3
B.3 FURTHER ANALYSIS: INSTRUCTION TUNING FACILITATES PROMPT TUNING
The per-dataset results for the analysis in $4.3 are given Table 4 As the above tasks are all classification, further work in this direction might include tasks such summarization question answering, or try to finetune the model using the supervised datasets.
C DATA CONTAMINATION ANALYSIS
One reasonable concern is that since the pretraining corpus of FLAN has more than trillion tokens, it is possible that examples from given evaluation dataset may have already been seen verbatim by the model during pre-training, hence inflating the performance of our purported zero-shot model. To this end, like GPT-3 (Brown et al. 2020), we perform post-hoc data contamination analysis to
24

---

Published as a conference paper at ICLR 2022
sero-shot performance on unseen task cluster
Finetuning prompt
Inference prompt
NLI
Read. Comp.
Closed- Book QA
Translation
Four-Tase
Average
Natural instructions
Natural instructions
56.2
77.4
56.6
30.7
55.2
(=FA)N)
No template Task/dataset name Task/dataset name
Natural instructions
50.5
58.2
25.5
15.0
37.3
Natural instructions
52.8
63.0
44.8
25.9
46.6
Task/dataset name
60.2
64.9
40.8
21.9
47.0
Ablation study result using models where instructions are removed from the finetuning process. In "no template," only inputs and outputs are given, which does not distinguish among tasks during multi-task finetuning. In "task/dataset name", inputs during multi-task finetuning are prepended with the name of the task and dataset (e.g., "/Translation: WMT' 14 French] The dog runs") NLI datasets: ANLI R RI-R3, CB, and RTE; reading comprehension datasets: BoolQ, MultiRC, and OpenbookQA: closed- book QA datasets: ARC-c, ARC-e, NQ, and TQA; translation datasets: WMT' 14 Fr++En, WMT 16 Det>En, and WMT 16 Ro+-n. Notably, training with task/dataset name achieved high NIJ score largely because achieved score of 83.9 on the CB dataset, for which the validation set only has 56 examples (FLAN also gets 83.9 with the best dev template, but the average template was only 64.1).
Table 3:
PROMPT TUNING ANALYSIS
Prompt tuning train. examples
BoolQ
CB
CoPA acc.
MultiRC F1
ReCoRD acc.
RTE
WiC
WSC
acc.
acc.
aсc.
acc.
acc.
LaMDA-PT
55.5
55.4
87.0
65.4
78.0
52.4
51.6
65.4
32
FLAN
77.5
87.5
91.0
76.8
80.8
83.0
57.8
70.2
LaMDA-PT
full dataset
82.8
87.5
90.0
78.6
84.8
82.0
54.9
72.7
FLAN
86.3
98.2
94.0
83.4
85.1
91.7
74.0
86.5
Table 4: FLAN (instruction tuning) responds better to continuous inputs attained via prompt tuning than LaMDA-PT (no instruction tuning). When prompt tuning on a given dataset, no tasks from the same cluster that dataset were seen during instruction tuning.
investigate whether the performance of the model is in fact inflated by evaluating on examples that occurred in the pretraining dataset.
Our data contamination procedure follows the setup of Brown et al. (2020) which, for each bench- mark, produces a "clean" version that removes all potentially leaked examples, defined as examples for which any n-gram (n varies per dataset but is roughly 13) overlapped with anything in the pretraining corpus. We use the same n per dataset as Brown et al. (2020) and also split on spaces. We then evaluate our model on this clean subset, comparing against model performance on the original dataset (clean + dirty). Lower performance on the clean subset would suggest that data contamination leads to inflated results.
summarizes these results, with the exact numbers given in Table 5 We see several trends very similar to those in the GPT-3 paper: (1) many datasets had substantial number of examples that overlapped with the pretraining data, (2) across all datasets, we do not see correlation that evaluating on clean data does worse than evaluating on the total dataset, and (3) as datasets had fewer clean examples, there was higher variance in the percent change in performance (likely due to smaller number of clean examples).
Figure 12
Like GPT-3, we also found that DROP and SQuADv2 had almost total overlap with the pretraining data. We follow their procedure of manually inspecting the data, and find that most overlapping n- grams were only in the contexts of examples (99.6% for DROP and 97.2% for SQuADv2). Overlaps never occurred in both the question and answer for DROP, and only occurred for both the question and answer for SQuADv2 in 5 of the 11,153 evaluation examples. Hence, for these datasets, the
25

---

Published as a conference paper at ICLR 2022
60
DROP
40
only clean data did better
Percent change in performance of FLAN (ecouracy,
20
ANLI R1
BLEU)
SQUAD v2
ReCoRD
PIGA
eval all data (including dirly) did better
-20
50
75
100
Percent of Data Clean in Datasel
Figure 12: Like GPT-3, we also measured performance on cleaned versions of our datasets, which had high confidence to be unseen in the pretraining data of FL.AN. We do not see a correlation that FLAN performed better on evaluation sets which examples occurred more often in the pretraining data. When the percent of clean data is very small, there are fewer examples for computing the clean performance, which leads to high variance.
model gains only background information and cannot memorize the answer to any specific questions (aside from the five examples in SQuADv2).
ANLI RI and R2 (Nie et al. 2020) also had almost complete data contamination, to much higher degree than GPT-3. Upon further inspection, we see that most overlaps oceur example contexts and not hypotheses (97.3% for ANLI RI and 98.2% for ANLI R2). As ANLI RI and R2 are based entirely from Wikipedia examples (R3 not), we posit that this higher degree of contamination in pretraining dataset compared with GPT-3's is potentially due to using more-recent version of Wikipedia that includes the contexts used in ANLI RI and R2 (which were collected in 2019). Because seeing particular context in pretraining does not help with the NLI task given new, unseen sentence, we think it is unlikely that these overlaps affected performance on the two datasets.
Of the remaining datasets, only ReCoRD and PIQA had clean subset performance that was lower than the overall evaluation set performance by more than 1 %.. These two datasets are language modeling (i.c., "what's the best continuation this sentence?"), and so it is more likely compared with previous tasks that seeing a complete sentence in the pretraining data could help the model predict the right answer in downstream evaluations. For PIQA, both the goal and solution had overlaps in 93 of the 1,838 evaluation examples, and for ReCoRD, the query had overlaps in 2,320 of 10,000 training examples. We hence mark these results with an asterisk in Table Brown et al. (2020) also reported substantial contamination rates for these two datasets (61 % dirty for ReCoRD and 29% for PIQA), and also mark PIQA with an asterisk.
As this overlap analysis follows that performed in Brown et al. (2020). reiterate the same caveats: the conservative nature our n-gram matching procedure likely introduces additional false positives; there guarantees that the clean subset is drawn from the same distribution the overall subset; and, accurately detecting test contamination is relatively new research area without established best practices. Moreover, our pretraining corpus is almost five times larger than that used for GPT-3 (which was 500B tokens), it is possible that there are more false positives in detecting dirty data.
26

---

Published as a conference paper at ICLR 2022
% Diff
Total count
Total acc/FI/BLEU
Clean count
Clean acc/FI/BLEU
Dataset
Metric
clean
(clean
overall)
DROP
F1
9,536
22,4
61
33.0
0.6
47.4
SQuADv2
11,873
41.3
38.7
0.9
-6.2
ANLI R1
acc 1,000
48.1
14
57.1
1.4
18.8
ANLI R2
1,000
42.9
21
38.1
2.1
-11.2
ReCoRD
10,000
4.6
3,203
4.5
32.0
-2.7
MultiRC
acc
1.972
75.7
40.7
0.5
PIQA
acc
1,838
23.7
896
23.3
48.7
-1.7
ANLI R3
acc
1,200
44.2
718
45.3
59,8
2.5
HellaSwag
10,042
28.5
6,578
28.7
65.5
0.7
RTE
2.77
84.1
183
84.2
66.1
0.0
WMT' En->Fr
BLEU
3,003
2,243
31.5
74.7
0.9
WMT' 14 Fr-+En
BLEU
3,003
34.0
2,243
34.1
74.7
0.2
BoolQ
acc
3,270
76.5
2,515
76.3
76,9
-044
TQA (tfds-dev)
F1
11.313
62.2
8,731
62.0
77.2
0.2
ARC Easy
acc
2,365
1.888
79.0
79.8
0.66
ARC Challenge
acc
1,165
983
64.2
84.4
1.7
OpenbookQA
acc
500
74.6
425
74.8
85.0
0.3
WMT' 16 En
BLEU
2,999
22.7
2,569
23.0
85.7
1.4
WMT' Dn
BLEU
2,999
38,6
2.569
38.7
85.7
0.2
WMT 16 En
BLEU
1,999
1.752
15.4
87.6
-0.7
WMT' 16 Ro-+En
BLEU
1,999
36.7
1,752
36.8
87.6
0.1
COPA
acc
100
88.0
91
87.9
91.0
0.1
CB
56
41.1
53
41.5
94.6
1.1
NQ
F1
3,610
3,495
24.3
96.8
-0.5
StoryCloze
acc
1.871
1,864
92.1
99.6
Winogrande
acc
1,267
39.4
1,265
39.4
99.8
0.2
Table 5: Overlap statistics for the subset of datasets that also used in GPT-3, sorted from dirtiest to cleanest. An evaluation example was dirty if it had any r-gram collision with the pretraining corpus. We computed these results for FLAN's performance using only single template for each dataset, so they differ slightly compared with the results for average performance over all templates.
27

---

Published as a conference paper at ICLR 2022
D EXTENDED RELATED WORK
D.1 LANGUAGE MODELS AND MULTI-TASK LEARNING
Our work is broadly inspired by long line of prior work language models for NLP applications
(Dai & Le 2015 Peters et al.) 2018: Howard & Ruder 2018 Radford et al., 2018
2019 inter
alia). Instruction tuning can be
a formulation of multitask learning (MTL), which is an
established area within deep learning (Collobert ct al. 2011 Luong et al. 2016 , Ruder 2017: Velay
& Daniel, 2018: Clark et al. 2019b Liu al. 2019b inter alia)-sea Worsham & Kalita (2020) for a recent survey on MTL for NLP. Differing from prior MTL work which focuses on performance
improvements across training tasks (Raffel al. 2020, Aghajanyan et al. (20TT) or to new domains
(Axelrod et al.) (201T) our work is motivated by improving zero-shot generalization to tasks that were not seen in training.
D.2 ZERO-SHOP LEARNING AND
Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among set of unseen categories (Lampert et al., 2009
Pareees 'oor 2015; Srivastava al. 2018 Yin al. 2019 inter alia). In NLP, zero-shot
learning work also includes translating between unseen language pairs (Johnson et al.) 2017: Pham (et al.) 2019), language modeling on unseen languages (Lauscher et al. (0020), as well as various NLP applications (Liu et al. 2019a Corazza et al. 2020: Wang et al. (021). Most recently, the emergent ability of language models (Brown et al. 2020) has led to increased interest how models generalize to unseen tasks, the definition of zero- shot learning used in our paper. In addition, meta-learning (Finn et al. 2017 Vanschoren, 2018 inter alia) also broadly tries to train models that adapt quickly to unseen tasks, typically based few examples.
D.3 PROMPTING
Instruction tuning leverages the intuition that language models at scale contain substantial world knowledge and can perform a range of NLP tasks (Brown et al. 2020, see also [Bommasani et al. (02TT). Another line of work that shares this goal prompts models with continuous inputs optimized via backpropagation to substantially improve performance (Li & Liang 2021 Lester et al. 102T Qin & Hisner 2021 well as work that prompts models produce specialized outputs (Wei et al. 2022). Although the success of these approaches depends heavily on model scale (Lester et al. 2021), for which large models can be costly to serve, the ability of a single large model to perform many tasks slightly eases this burden. As shown by our experiments in $4.5 prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance. Reif et al. (202T) is similar to our work in that they also use related tasks to improve zero-shot learning, though they differ by only using related tasks in the context (and not finetuning), and focus on the application of text style transfer.
Our work shares similar motivations with prompting that we use inference-time text interactions to prompt single model, without creating separate checkpoints for each task. Whereas prompting work such as GPT-3 uses prompt engineering to write prompts that intentionally mimic text that is likely to be seen during pretraining (e.g., for MultiRC GPT-3 tries prompt that mimics test with an answer key), we hope that finetuning models to respond to natural language instructions instead of completing sentence will make such large models more accessible to non-technical users.
D.4 FINETUNING LARGE LANGUAGE MODELS
Finetuning pretrained language models is a well-established method in NLP, with much of the work
so far occurring on models in the range of 100M to 10B parameters (Dai & Le 2015: Devlin et al..
2019 Raffel et al. 2020 Lewis et al., 2020 inter alia). For models of O parameters, recent work has finetuned task specific models for program synthesis (Austin et al. 2021 Chen et al. 2021), ) summarization (Wu al. 2021). as well as improved bias and faimess behavior (Solaiman & Dennison 202T). In addition to the traditional "dense" models, sparse mixture of experts (MoE)
models of up more than IT parameters have been trained and finetuned (Lepikhin al.
2020)
Fedus
28

---

Published as a conference paper at ICLR 2022
et al. 2021). Compared with this prior work that finetunes and evaluates the same downstream task, our setup studies the effect of instruction tuning on ability to perform unseen tasks.
D.5 MUSK QUESTION ANSWERING
The instructions we use for instruction tuning are similar to QA-based task formulation research, which aims to unify NLP tasks by casting them as cression-answeringg over context. For instance, McCann et al. (2018) cast ten NLP tasks as QA and train model on a collection of tasks formulated with natural language prompts; they report transfer learning gains on finetuning tasks well as zero-shot domain adaptation results on SNLI (Bowman al. 2015) and Amazon/Yelp Reviews (Kotzias et al., 2015). While McCann et al. (2018) does not leverage unsupervised pre-training and only reports zero-shot transfer to unseen domains, our work uses a pretrained LM and focuses on zero-shot performance
unseen task clusters. UnifiedQA (Khashabi et al. 2020) shows similar transfer learning gains as McCann et al. (0018) across 20 datasets and reports good generalization to unseen tasks across four types of QA. Focusing on binary text classification, [Zhong et al. (2021) finetune T5-770M on 43 tasks phrased as yes/no questions and study the zero-shot performance on unseen tasks. In comparison, our paper is much larger in scope, empirically demonstrating the idea on a wide range of tasks with much larger model. Other work has used QA-based task formulation for more-targeted applications including semantic role labeling (He et al.) 2015), relation extraction (Levy et al., 2017), coreference resolution (Wu et al.) 2020) and named entity recognition (Li et al. 2020) as question answering.
D.6 INSTRUCTIONS- BASED NLP
Recent improvements in the capabilities of language models have led to increased interest in nascent
area of Instructions-based NLP (Goldwasser & Roth 2014] and see McCarthy (960)). Schick &
Schütze (2021) (also see Gao et al.] [021]] Tam et al.) (02T)) use task descriptions in cloze-style
phrases to help language models assign soft labels for few-shot and semi-supervised learning, though
this line of work finetunes new checkpoints for each downstream task. Efrat & Levy (202)) evaluated
GPT-2 (Radford et al., 2019) on simple tasks ranging from retrieving the nth word of sentence to generating examples for SQuAD, concluding that GPT-2 performs poorly across all tasks.
In terms of the setup of finetuning on a large number of tasks and evaluating on unseen tasks, two recent papers are similar to ours. Mishra et al. (20TT) finetune BART (Lewis al., 2020) using instructions and few-shor examples for tasks such as question answering, text classification, and text modification, and find that this few-shot finetuning with instructions improves performance on unseen tasks. Ye et al. 2021) introduce setup for cross-task few-shot learning, finding that multi-task meta-learning using MAML (Finn et al. 2017) improves the few-shot capabilities of BART on unseen downstream tasks. Our work differs from these two papers in that we focus aero-shot learning, for which we observe the crucial importance of model scale (FLAN is 1,000x larger than BaRT-basc).
Perhaps the papers most related to are the recent Sanh et al. 1021 and Min et al. (202I) which were released after our initial preprint. Min et al. (2021) finetunes GPT-2 Large (770M parameters) to be a few-shot learner, which is the same approach as experiment in Section 4.3. Similar to our conclusions, they also observe that including few-shot exemplars and instruction tuning are complementary ways to improve performance. [Sanh al. (202T) propose to finetune 15-11B to respond to prompts, and they also report performance improvements on zero-shot learning. These two papers and our work all study finetuning with instructions. but, as noted by Min et al. (2021). it is hard to directly compare results, due to differing model sizes, model types (decoder-only vs encoder-decoder), pretraining data, task mixtures, and type of instructions (Sanh et al. (202)) say that their instructions are more diverse).
Finally, OpenAl has a model called InstructGPT (Ouyang et al., 2022). InstructGPT uses human anntations to guide desired model behavior, both via finetuning and reinforcement learning, finding that InstructGPT is preferred by human rathers compared with unmodified GPT-3.
29

---

Published as a conference paper at ICLR 2022
E FREQUENTLY ASKED QUESTIONS
How do the FLAN instructions differ from GPT-3 or T5 prompts?
GPT-3 prompting is done in way such that the prompt looks like data that the model has been pretrained on, and the model finishes the continuation. T5 prompts are mostly just tag for the dataset, which would not work in the zero-shot setting, In contrast, the prompts that we use for FLAN are similar to what would be used to ask human to perform the task.
For instance, given an input for an NLI task, these would be the prompts.
T5 prompt:
cb hypothesis: At my age you will probably have learnt one lesson. premise: It's not certain how many lessons you' learn by your thirties.
prompt::
At my age you will probably have learnt one lesson.
question: it's not certain how many lessons you'11 learn by your thirties. true, false, neither? answer:
FLAN prompt:
my age you will probably have learnt one lesson. Rypothesis: It's not certain how many lessons you' 11 learn by your thirties.
Premise:
Does the premise entail the hypothesis?
So because FLAN prompts are formulated responding instruction, they do not work well for pretrained language models without finetuning. Performance was near zero for most generation tasks. For instance, given the input "Thhe dog runs.' Translate this sentence French.". LaMDA-PT continues with "The dog runs after the cat" instead of actually translating the sentence. Hence, we used the established GPT-3 prompts for our LaMDA-PT baselines.
What are some limitations/failure cases FLAN?
While we qualitatively find that FLAN responds well to most tasks, it does fail on some simple tasks. For instance, as shown in Figure 22 FLAN fails the very simple task of returning the second word in sentence, and also incorrectly translates a question to Danish when asked to answer the question Danish. Additional limitations include context length only 1024 tokens (which is not enough for most summarization tasks), and that the model mostly trained English data.
Can FLAN be used when large amounts of training data available?
In this work, we focus on cross-task generalization to zero-shot tasks, but we also believe that instruction tuning could result in positive task transfer among seen tasks, depending on the mixture of tasks (though we leave this for future work). In $4.5) where we apply prompt tuning to the FLAN checkpoint, we see promising results that indicate positive task transfer in a supervised setting.
Are the unique templates per dataset or per task cluster?
The ten unique templates are for each dataset and not for task cluster. This is because datasets in the same task cluster often differed slightly (e.g., "is this movie review positive" vs "is this yelp review positive") ").
In Figure 7A why does the untuned LaMDA-PT model parameters for reading comprehension and sentiment analysis?
performance with more
For context, Figure 7A is check of correctness for Figure Figure confirms that scale improves performance for tasks that were seen during instruction tuning, as expected. The untuned LaMDA-PT model performance in Figure 7 is shown just for completeness.
30

---

Published as a conference paper at ICLR 2022
Nonetheless, the fact that scale does not always improve zero-shot performance of untuned LaMDA- PT is an interesting artifact. Initially, we were surprised, because Brown et al. (2020) shows that scale improves performance across large number of tasks in aggregate.
turns out that scale does not improve performance for certain tasks. This is especially true for zero-shot learning, and we think that this happens be the case for the reading comprehension and sentiment analysis tasks we evaluate. The GPT-3 paper itself similarly reports that zero-shot performance on BoolQ and DROP decreases from 13B 175B parameters. The GPT-3 paper does not show results on sentiment analysis, but Holtzmann et al. 2021) find that zero-shot performance on SST-2 also gets worse from 13B to 175B parameters. Hence, this artifact is consistent across both GPT-3 and the models we use.
This artifact is certainly worth further study, but is outside the scope of instruction tuning, Ideally, we would have performed the Figure F abllttion with cross-validation instead of single split, which likely would have smoothed that artifact.
QUALITATIVE EXAMPLES
This section shows qualitative examples of FLAN responding to various prompts.
Input:
Input:
Movie review: This movie is the best RomCom since Pretty Woman.
Movie review: This movie is the best RamCom since Pretty Woman.
Did this critic like the movie?
Did this critic dislike the movie?
OPTIONS -yes
OPTIONS -yes -no
-no
FLAN output:
FLAN output:
no
Figure 13: For sentiment analysis, FLAN changes the answer appropriately when the question is flipped.
Input:
Input: What is Barack Obama's profession?
What Barack Obama's profession?
Answer Chinese
FLAN output:
politician
FLAN output:
lawyer
senator
president
Figure 14: For question answering, FLANN can answer a question in another language when instructed to do Multiple FLAN outputs generated via random sampling with temperature of 0.9 and top k of 40.
31

---

Published as a conference paper at ICLR 2022
Input:
Input:
John is reading the paper "Finetuned Language Models are Zero-Shot Learners."
John is reading the paper "Finetuned Language Models are Zero-Shot Learners."
Change to past tense.
Change to passive voice.
FLAN output:
FLAN output:
John was reading paper "Finetuned Language Models Shrot-Shtt Learner.."
The paper "Finetuned Language Models Zero-Shot Learners" being read by John.
Input:
Input:
John reading the paper "Finetuned Language Models Zero-Shot Learners."
John reading the paper "Finetuned Language Models Zero-Shot Learner."" "
Change to the verb eat.
Change the verb to eat and the subject to Noam Chomsky.
FLAN output:
FLAN output:
John is eating the paper "Finetuned Language Models are Zero-Shot Learners."
Noam Chomsky is eating the paper "Finetuned Language Models are Zero-Shot Learners."
Figure 15: FLAN can perform zero-shot rule-based manipulations.
Input:
Input:
Recommend fun activities do rainy night in Seattle.
Recommend activities to do on sunny weekend in Copenhagen.
FLAN output:
FLAN output:
Visit Christiania
to pacific science center
Copenhagen Zoo
Go out to eat
Visit the Tivoli Gardens
Go to comedy club
Take canal tour
Build blanket fort
Museums
Go to the Space Needle
Visit The Little Mermaid Statue
Go to underground tours
Visit Copenhagen Zoo
Find karaoke bar
Visit the Danish Maritime Museum
Figure 16: FLAN can make zero-shot recommendations. Multiple FLAN outputs are generated via random sampling with temperature of 0.9 and top & of 40.
32

---

Published as a conference paper at ICLR 2022
Input:
Input:
Generate utterances with the intent "get COVID vaccine"
Generate utterances with the
"where vote the state Senate election'
FLAN output:
FLAN output:
wauld get the covid vaccine,
Tell me where to vote state
would like COVID vaccine
senate election
soon possible
the state senate election
Find where
Can get COVID vacoine
preventative measure?
Where supposed to the state senate election
Get COVID vaccine for five
old
Where get information
Where is center for COVID
where vote the state Senate election?
vaccine testing?
Figure 17: FLAN can be used in zero-shot fashion to generate data, such as utterances that are consistent with a given intent. Multiple FLAN outputs are generated via random sampling with temperature of 0.9 and top k of 40.
Input:
Input:
Suggest related search term "Novak Djokovic"
Suggest search to "watermelon"
FLAN output:
FLAN output:
novak djokovic girffriend
Cantaloupe
serena williams
Cucumber
rafael nadal
Melon
novak djokovic wife
Fruit
roger federer
Watermelon juice
andy murray
Watermelons
novak djokovic news
Lemon
noval djokovic serbia
Apple
novak djokovic wimbledon
Figure 18: FLAN can be used for zero-shot query expansion. Multiple FLAN outputs generated via random sampling with temperature of 0.9 and top 1e of 40.
Input:
On the way to Yosemite, our front tire blew.
Finish writing the story, and include a happy ending.
Input:
"I guess need to do laundry."
Rewrite above sentence in
FLAN output (cherrypicked):
Shakespearean style.
We got our tire replaced. The mechanic was honest, kind and helpful. Then we had a smooth ride for the rest the trip. Our trip was filled with and excitement.
FLAN output:
To mind, verily. suppose 'tis time, I wash my clothes.
We waited three hours the tow truck. had great picnic. The tow truck driver helped us put spare on, and we arrived Yosemite our reservations
Mine eyes and ears and tongue declare me idle: and yet it is true guess, By my garments, that need to do laundry
Lo my duty now calls To thee, to wash this body's 's clothes.
dinner, with
spare.
Figure 19: FLAN can perform zero-shot tasks relevant to assisted-writing applications. Multiple FLAN outputs are generated via random sampling with a temperature of 0.9 and top 40.
33

---

Published as a conference paper at ICLR 2022
Input:
Make word that means "when bananas shower from the sky"
Input:
Make up word means pipedream"
FLAN output:
FLAN output (cherrypicked):
Bananation
Banana shower
Pipedreamic
Banania
Pipedreamish
Bananafall
Pipedreamlike
Bananada
Phantasmagoric
Figure 20: FLAN can be used for zero-shot word formation. Multiple FLAN outputs are generated via random sampling with temperature of 0.9 and top & of 40.
Input:
Input:
John moved mountain view. He spent friday night in San Francisco. He got COVID vaccine. He voted in the state senate election.
Write sad story about carrots.
FLAN output:
Write story that follows the above plot.
The sun had barely risen when the carrots were taken beds brought market. They't time to gather their strength. They smelled of earth, their greens still wet from dew. interested in buying them. But one man finally reached out picked one!" he cried out. "It's for my mother! I'll pay you well it." The took the carrol to the cashier, eager to get home to his mother. was careful to wrap cerefully that it wouldn't get bruised cirty. He home, when saw mother, gave her the carrot. "Is this one of the ones grew?" she wavering
FLAN output (cherrypicked):
John arrived the with his vaccine, feeling prepared enough for the state senate election of San Francisco. He had voted for the first time last night and was feeling good about it. He felt more prepared than ever to win, and win big, because he was getting COVID vaccine. He knew that his opponent had not gotten COVID vaccine, and he knew he had already defeated him before the election began.
the carrots. 'Illl take this
emotion. "Yes, he replied. "It is, dear." The man left the carrot with his mother and hurried
market. His mother very sick, and he was afraid she wouldn't last much longer. He neaded that none would be left when she was gone.
all the carrots market
Figure 21:
Open-endee generation tasks by FLAN, The carrot story was from sampling sixteen outputs with a minimum length of 150 and choosing the highest probability output.
Input:
Input:
Conan
buy
What is Barack Obama's profession?
some pencils.
What the second word in this sentence?
Answer Danish
FLAN output:
FLAN output:
pencils
Hvad er Barack Obamas
Conan
beskceftigelse?
store
Hvad er Barack Obamas erhverv?
buy
Figure 22: Example failure cases for FLAN. Left: FLAN fails to perform a simple task returning the nith word. Right: FLAN translates question instead of answering it. Multiple FLAN outputs generated via random sampling with a temperature of 0.9 and top k of 40.
34

---

Published as a conference paper at ICLR 2022
CHANGES FROM V4 TO V5
Replaced the tables in the main figure with figure, which takes up less space and focuses
zero-shot performance.
Added GLaM 64B/64E as a baseline.
Moved the ablation about the role of instructions, as well as prompt tuning, into the main paper (and condensed the figures).
CHANGES TO V4 FROM V3
We added Frequently Asked Questions section (Appendix E).
We added a section with qualitative examples (Appendix) F
We added an additional ablation study on the role instructions during finetuning (Ap- pendix B.2).
We updated the related work (Appendix D with manuscripts posted on arxiv since our initial preprint.
CHANGES TO V3 FROM V2
The number of tokens used in pretraining corrected from 2.81T to 2.49T tokens.
CHANGES TO V2 FROM VI
We updated the terminology to "datasets" and "task clusters."
We renamed the previous "open-domain QA" task cluster to "closed-book QA."
We extended the related work section and moved it to the Appendix D using shorter version in the main body.
We added FLAN and LaMDA-PT results for additional datasets for which GPT-3 results were not reported.
For TriviaQA, vI reported results the tfds dev set of 11,313 examples. GPT-3 actually evaluates on the wikipedia dev set of 7,993 examples, so we ran an additional evaluation that dev set order compare with GPT-3's performance. Zero-shot FLAN now beats zero-shot GPT-3 on that task (and therefore on 20 of 25 tasks). We still show the original though there no GPT-3 result to compare with.
result in Table [
We moved commonsense reasoning and coreference resolution from the main body to the Appendix.
We moved prompt tuning from the main body to 4.5
We added data contamination analysis (Appendix
We added few-shot instruction tuning ($4.4).
We cited additional datasets in Appendix G
The number of tokens used in pretraining was corrected from 2.81T to 2.49T tokens.
35

---

Published as a conference paper at ICLR 2022
G TASKS AND DATASETS
This appendix further details the datasets that we use in this paper. We group datasets into one of the following task clusters:
Natural language inference concerns how two sentences relate, typically asking, given first sentence, whether a second sentence is true, false, or possibly true. We use the following datasets:
1. ANLI (Nie et al. 2020
2. CB (De Marnefle et al.
2019
3. MNLI (Williams et al.
2018
4. QNLI Rajpurkar al.
2018
5. SNLI Bowman al.
2015)
6. WNLI (Levesque al.
2012)
7. RTE DDgan et al. 2005 Haim et al. 2006
Giampiccolo et al. 2007 Bentivogli al., 2009)
Reading comprehension the answer. We use the following datasets:
the ability answer question when given passage that contains
1. BoolQ Clark et al.
(2019a)
2. DROP Dua et al.
2019)
3. MultiRC (Kashhabi al.
2018
4. OBQA [Mihaylov al. 2018
5. SQuADvl (Rajpurkar al.
2016)
6. SQuADv2 (Rajpurkar al. 2018)
Commonsense reasoning evaluates the ability perform physical scientific reasoning with an element of common sense. We use the following datasets:
1. COPA (Roemmele et al. 2011
2. HellaSwag (Zeleers et al. 2019)
3. PiQA (Bisk et al.) 2020)
4. StoryCloze (Mostafazadeh al. 2016)
Sentiment analysis is classic NLP task aims to understand whether a piece of text is positive or
negative. We use the following datasets:
1. IMDB (Maas al. 2011)
2. Sentiment140 (Go al.
2009
SST-2 (Socher et al. 2013)
4. Yelp (Fast.Al)
Closed-book QA asks models to answer questions about the world without specific access to information that contains the answer. We use the following datasets:
1. ARC (Clark et al., 2018)
2. NQ (Lee et al.
2019
Kwiatkowski et al. 2019
3. TriviaQA Joshi et al.
2017
Paraphrase detection asks model to determine whether two sentences are semantically equiva-
lent We use the following datasets:
1. MRPC (Dolan & Brockett 2005
2. QQP (Wang et al.) 2018 see)
3. Paws Wiki (Zhang et al. 2019)
Coreference resolution
the ability identify expressions of the same entity some given text. We use the following datasets:
1. DPR (Rahman & Ng 2012)
2. Winogrande (Sakaguchi et al. 2020)
lthough paraphrasing can he seen positive entailment in both directions, it has heen distinct from NI.I the academic literature.
36

---

Published as a conference paper at ICLR 2022
3. WSC273 Levesque et al. 2012)
Reading comprehension with commonsense combines elements of both reading comprehension with commonsense. We use the following datasets:
1. CosmosQA (Huang al. 2019
ReCoRD (Zhang et al. 2018)
Struct text tests the ability describe some structured data using natural language. We use the following datasets:
1. CommonGen (Lin et al.
2020
2. DART (Nan et al. 2021
3. E2ENLG (Dušek et al. 2019)
4. WebNLG (Gardent et al.
2017)
Translation is the task of translating text from one language into different language. We use the following datasets:
1. En from WMT 14 [Bojar et al.
2014)
2. En-De, En-Tr, Tr, En- Cs, En-Fi, En Ro, and En In-Ru from WMT (Bojar et al. 2016)
3. En-Es from Paracrawl (Bañón et al. 2020)
Summarization asks models to read a piece of text and generate an abbreviated summary of
We use the following datasets:
1. AESLC (Zhang & Tetreault
2019)
2. CNN-DM (See et al.
2017)
3. Gigaword (Npolees al.
2012
4. MultiNews (Fabbri et al.
2019)
5. Newsroom Grusky et al.
2018)
6. Samsum (Gliwa et al.
2019)
7. XSum (Naraynn et al.
2018
8. AG News (Zhang al.
2015)
9. Opinion Abstracts Rotten Tomatoes (Wang Ling 2016
10. Opinion Abstracts iDebate (Wang & Ling 2016)
11. Wiki Lingua English (Ladhak et al. 2020)
Additional datasets that we assign to a miscellaneous task cluster include:
Conversational -eession-answwring: QuAC (Choi al. 2018) and CoQA (Reddy al. 2019)
2. Evaluating context-sentence word meanings: WiC (Pilehvar & Camacho Camacho-Collados 2019
3. Question classification: TREC (Li & Roth 2002 Hovy et al. 2001)
4. Linguistic acceptability: CoLA WWrrstadt al. 2019
5. Math questions (Saxton al. 2019)
For all tasks, our finetuning and evaluation code uses tensorflow datasets (TFDS) load and process datasets. Regarding the number of training examples per dataset, we limited the training set size per dataset 30,000 so that no dataset dominated the finetuning distribution. When test set with labels was available in TFDS, we used it: otherwise, we used the TFDS validation set as our test set, splitting the training set into a train and dev set.
On the following pages, we show inputs and outputs for evaluation tasks where we compared with GPT-3. See the attached supplementary material for the templates for all other datasets.
37

---

Published as a conference paper at ICLR 2022
G.1
NATURAL LANGUAGE INFERENCE
INPUT
Joey Heindle (born 14 May 1993 in Munich) is German singer. He is best known for winning the seventh season of the game show Ich bin ein Star Holt mich hier raus! and finishing in 5th place in season 9 of Deutschland sucht den Superstar, despite universally negative reviews from the jury each week.
Based on the paragraph above can we conclude that "Joey Heindle was highly disliked by people television." "?
OPTIONS:
Yes
It's impossible say
-No
TARGET
Yes
Table 6: Example input and target for Adversarial NLI (ANLI). ANLI (Nie et al. 2020)
large-scale NI.I benchmark with adversarial examples collected iteratively with human and model the loop. The task is to determine whether a hypothesis entailed by a premise (entailment, not entailment, impossible to say). There are three rounds, RI-R3. Of the three training sets with 16,946, 45,460, and 100,459 examples, we use 16,946, 30,000, and 30,000 for train and 200 from each of the three TFDS validation sets for dev. We use the TFDS "test" sets of 1,000, 1,000, and 1 1,200 examples as our test set for reporting numbers.
INPUT
A: so watch the fish, you know. Whatever can do to keep myself occupied. like to have the TV on, because that usually keeps me, um, more occupied. It kind of takes the time away and I don't realize, that's 's really the only time I ever watch TV, is when I'm on the bike. and then usually after I'm done riding the bike, just to cool myself down, usually take walk, you know, and that just kind of uh, gets me, you know, to where I' I'm not quite tired guess. But it's definitely a task. B: You think so? A: I can't 't say that really enjoy it.
Based on the paragraph above can we conclude that "she really enjoys it"?
OPTIONS:
Yee
-No
It's impossible say
TARGET
No
Table 7: Example input and target for Commitment Bank (CB). CB (De Marneffe et al.) 2019) is corpus of texts in which a hypothesis extracted from a premise, and the task is to determine whether the hypothesis is entailed by the premise (entailment, not entailment, training set with 250 examples, we use 200 for train and 50 for dev. We use the TFDS validation set of 56 examples as test for reporting numbers.
impossible to say). Of the
38

---

Published as a conference paper at ICLR 2022
INPUT
After years of study, the Vatican's doctrinal congregation has sent church leaders confidential document concluding that "sex-change" procedures do not change a person's 's gender in the eyes of the church.
Based on the paragraph above can we conclude that "Sex-change operations become more common."? "?
OPTIONS:
yes
TARGET
no
Table 8: Example input and target for Recognizing Textual Entailment (RTE), RTE (Dagan et al. 2005 Haim et al. 2006; Giampiccolo al., 2007 Bentivogli al. 2009) asks whether second sentence is entailed by first (binary, either entailed or not entailed). Of the training set with 2490 examples, we 2,290 for train and 200 for dev. We use the TFDS validation of 277 examples as our test set for reporting numbers.
39

---

Published as a conference paper at ICLR 2022
G.2 READING COMPREHENSION
INPUT
There are four ways an individual can acquire Canadian citizenship: by birth on Canadian soil; by descent (being born to Canadian parent); by grant (naturalization); and by adoption. Among them, only citizenship by birth is granted automatically with limited exceptions, while citizenship by descent or adoption acquired automatically the specified conditions have been met. Citizenship by grant, on the other hand, must be approved by the Minister of Immigration, Refugees and Citizenship.
Can we conclude that can i get canadian citizenship if my grandfather was canadian?
OPTIONS:
-no
yes
TARGET
no
Table Example input and target for Boolean Questions (BoolQ). BoolQ/llar Clark al. (20T9a) asks yes/no question based on passage and question. Of the training set with 9,427 examples, we use 9,227 for train and 200 for dev. We use the TFDS validation set of 3,270 examples as our test set for reporting numbers.
INPUT
Imagine you are standing in farm field in central Illinois. The land is so flat you miles and miles. On clear day, you might see a grain silo 20 miles away. You might think to yourself, it sure is flat around here. If you drive one hundred miles to the south, the landscape changes. In southern Illinois, there are rolling hills. Why do you think this is? What could have caused these features? There are no big rivers that may have eroded and deposited this material. The ground is capable of supporting grass and trees, so wind erosion would not explain it. To answer the question, you need to go back 12,000 years. Around 12,000 years ago, giant ice sheet covered much of the Midwest United States. Springfield, Illinois, was covered by mile ice. Its hard to imagine a mile thick sheet ice. The massive ice sheet, called a glacier, caused the features the land you today. Where did glaciers go? Where can you see them today? Glaciers are masses of flowing ice.
for
Question: "How big were the glaciers?"
Response: "One mile"
Does the response correctly answer the question?
OPTIONS:
no
yes
TARGET
yes
Table 10: Example input and target for Multi-Sentence Reading Comprehension (MultiRC), MultiRC
Khashabi al. (2018) asks an open-ended question given a paragraph that contains the training set with 27, 27,243 examples, we use 27,043 for train and 200 for dev. We use the TFDS validation set of 4,848 examples as our for reporting numbers.
the answer. Of
40

---

Published as a conference paper at ICLR 2022
INPUT
soil is renewable resouree for growing plants
A plant that needs to expand will be able to have endless resource in
OPTIONS:
dirt
pesticides
pay
beans
TARGET
dirt
Table 11: Example input and target for Openbook Question Answering (OBQA). OBQA (Mihaylov ct al. 2018) asks 4-way multiple choice questions based facts. Of the training set with 4,957 examples, we use all for train and 200 in the TFDS validation set of 500 examples for dev. We use the TFDS test set 500 examples as our test set for reporting numbers.
41

---

Published as a conference paper at ICLR 2022
G.3 COMMONSENSE REASONING
INPUT
packed up my belongings. What is the cause?
OPTIONS:
was hunting for a new apartment.
I was moving out of my apartment.
TARGET
was moving out of my apartment.
Example input and target for Choice of Plausible Alternatives (COPA). COPA (Roemmele) et al. 2011) is causal reasoning task that asks to infer either cause of effect of premise from two choices, Of the training set with 400 examples, we use 350 for train and 50 for dev. We use the TFDS validation of 100 examples our test for reporting numbers.
Table 12:
INPUT
What happens next in this paragraph?
Once the rope is inside the hook, he begins moving up the wall but shortly after stops and begins talking. The male then begins talking about the clip again and goes back up the wall,
he
OPTIONS:
progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety.
changes time, an instant replay of his initial move is shown a second time.
continues to talk, another male speaks about the move and shows another closeup of the plex by the male.
continues, other people start to arrive and begin hang out with him as he makes a few parts of the rope.
TARGET
progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of for support and safety.
Table 13: Example input and target for Commonsense Sentence Completion (HellaSwag). HellaSwag
(Zellers et al. 2019) tests for sentence completion that requires common sense, asking for the most probable ending given four contexts. Of the training set with 39,905 examples, we use 30,000 for train and 200 for dev. We use the TFDS validation set of 10,042 examples as our test set for reporting numbers.
42

---

Published as a conference paper at ICLR 2022
INPUT
Here goal: Remove smell from garbage disposal.
How would you accomplish this goal?
OPTIONS:
Create soda ice cubes and grind through disposal.
Create vinegar ice cubes and grind through disposal.
TARGET
Create vinegar ice cubes and grind through disposal.
Example input and target for Physical Question Answering (PiQA). PIQA (Bisk al. 2020) is a commonsense QA benchmark for naive physics reasoning, where a solution to goal must be selected from two choices. Of the training set with 16,113 examples, we use 16,013 for train and 100 for dev. We use the TFDS validation set of 1 1,838 examples as our test set for reporting numbers.
Table 14:
INPUT
Caroline never drinks carbonated beverages. Her friends pick on her because of it. One day they challenged her drink soda. Caroline wanted to win the challenge.
Predict the next sentence.
OPTIONS:
Caroline refused to open the soda.
Caroline opened the soda and drank it all in one gulp!
TARGET
Caroline opened the soda and drank all one gulp!
Example input and target for The Story Cloze Test (StoryCloze). StoryCloze (Mostafazadeh et al. 2016) is commonsense reasoning framework for story generation, where system chooses the correct ending to four-sentence story. We use the 2016 version on TFDS. Of the validation set with 1,871 examples (no training set is available), TFDS test set of 1,871 examples as our test set for reporting numbers.
Table 15:
use 1,671 for train and 200 for dev. We use the
43

---

Published as a conference paper at ICLR 2022
G.4
CLOSED-BOOK QA
INPUT
What season the Northern Hemisphere experiencing when it is tilted directly toward the Sun?
OPTIONS:
fall
wiinter
-spring
summer
TARGET
summer
Table 16: Example input and target for The A12 Reasoning Challenge (ARC). ARC (Clark al. 2018) asks grade-school level 4-way multiple choice science questions. There is a challenge set and an easy set, where the challenge set questions were answered incorrectly by both retrieval-based algorithm and a co-occurrence algorithm. Of the training sets with 1,1 examples (challenge) and 2,251 (easy), we use we use 919 and 2,051 respectively for train and 200 each for dev. We use the TFDS test sets of 1,172 and 2,376 examples respectively our test set for reporting numbers.
INPUT
Question: who the girl in more than you know??
Answer:
TARGET
Romi Van Renterghem.
Table 17:
Example input and target for Natural Questions (Open) (NQ). NQ (Lee et al. 2019 Kwiatkowski et al., 2019) asks for an open-ended answer given question, where all questions can be answered using the contents Wikipedia. Of the training set 87,925 examples, we use 30,000 for train and 200 for dev. We use the TFDS validation set of 3,610 examples our test set for reporting numbers.
INPUT
Please answer this question: Henry Croft, orphan street sweeper who collected money for charity, is associated with what organised charitable tradition of working class culture in London, England?
TARGET
pearly kings and queens
Table 18:
Example input and target for Trivia Question Answering (TriviaQA). TriviaQA Joshi et al. (017)) includes question-an question-ansser answer pairs authored by trivia enthusiasts. Of the training set of 87,622 examples, we use 30,000 for train and 200 for dev. We use 7,993 examples from Wikipedia of the 11,313 examples in the TFDS validation set, which the same validation set used in (Brown et 2020 as test set for reporting numbers.
44

---

Published as a conference paper at ICLR 2022
G.5
COREFERENCE RESOLUTION
INPUT
How does the sentence end??
Elena wanted to move out of her parents fast but Victoria wanted to stay for while,
OPTIONS:
Elena went to school.
Victoria went to school.
TARGET
Victoria went to school.
Table 19:
Example input and target for Adversarial Winograd Schema Challenge (Winogrande).
Winogrande (Sakaguchi et al. 2020) tests for coreference resolution by asking model to fill in masked token in sentence by choosing an entity from two options. Of the 40.4k examples in the XL training set, we use 30,000 for train and 200 for dev. We use the TFDS validation set of 1,267 as our test set for reporting numbers.
INPUT
Jane knocked Susan's door, but there
OPTIONS:
Jane out.
Susan was out.
TARGET
Susan was out.
Table 20: Example input and target for Winograd Schema Challenge (WSC273). WSC273 (eveequee et al. 2012) tests for coreference resolution by asking model to complete the sentence in fashion that requires understanding the entities in the sentence. Of the examples the training set (WSC273 test-set only), we none for train and none for dev. We use the TFDS test set our test set for reporting numbers.
45

---

Published as a conference paper at ICLR 2022
G.6
READING COMPREHENSION WITH COMMONSENSE
INPUT
Complete the passage.
(CNN) - At first glance, "The Flat" might seem like an episode of "Hoarders, " Istaeli-style. The documentary film opens after an elderly woman dies in Tel Aviv. Her grandchildren assemble to clean out her apartment, packed with dusty books, vintage clothing (dozens of pairs of fancy gloves, for instance), enough purses to stock a department store, jewelry, mementoes and closets full of knickknacks. But buried among the detritus they chance upon something remarkable mysterious papers linking the grandparents to an important Nazi figure. How could such ardent Zionists, who left their native Germany in the early 1930s, have been involved with an SS official like Leopold von Mildenstein?
What I found out was this journey, the Nazi (
OPTIONS:
Amon Goldfinger) and his wife were accompanied by my grandparents, " Goldfinger told CNN.
CNN) and his wife were accompanied by my grandparents, Goldfinger told CNN.
Germany) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Israeli) and his wife were accompanied by my grandparents. Goldfinger told CNN.
Leopold von Mildenstein) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Nazi) and his wife were accompanied by my grandparents, Goldfinger told CNN.
SS) and his wife were accompanied by my grandparents. " Goldfinger told CNN.
Tel Aviv) and his wife were accompanied by my grandparents, Goldfinger told CNN.
The Flat) and his wife were accompanied by my grandparents." Goldfinger told CNN. Zionists) and his wife were accompanied by my grandparents," Goldfinger told CNN.
TARGET
Leopold von Mildenstein) and his wife were accompanied by my grandparents." Goldfinger told CNN.
Table 21: Example input and target for Reading Comprehension with Commonsense Reasoning (ReCoRD). ReCoRD (Zhang et al.) 2018) asks for the answer to a cloze-style question where an entity is masked out. Of the the training of 100,730 examples, dev. We use the TFDS validation set 10,000 examples as our test set for reporting numbers.
30,000 for train and 200 for
G.7 TRANSLATION (7 LANGUAGES)
INPUT
Here the largest town the district is located: Nordenham, lying opposite to Bremerhaven at the Weser mouth.
Translate German
TARGET
An der B 211 befindet sich in Loyermoor der so genannte "Geest-Abbruch" der eine Höhendif- ferenz von gut 30 Meter überbrücck..
Table 22:
Example input and output for translation. This example is from 1M English English-Germant German; all languages use the same translation templates.
46


Reviews:
[{"review_id": "g-U7KRV0UCv", "reviewer": "Reviewer_DjKD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The problem addressed has high practical value: it tries to make large pre-trained language model more accessible to a range of NLP tasks. The \"instruction tuning\" idea will significantly reduce the cost for task-specific fine tuning, labeled data and prompt engineering compared to other approaches. \n2. The method is simple and easy to implement. Authors carefully design the experiment to minimize the leakage between the fine-tuning and inference data. Given that, it still shows superior performance on different types of NLP tasks. The result on specific task can be further improved when adapting with \"prompt tuning\" on labeled data, which shows that the instruction-tuning process does not drop much task-specific knowledge from the original pretrained model.\n3. The analysis presented in the main paper and the appendix is thorough enough. Authors also discussed about the limitation of model when downstream tasks are more similar to language modeling tasks.\n\nCons:\nThere are still a few questions that can be addressed to make the analysis comprehensive.\n1. Have authors try to use the FLAN prompts on GPT3 or BaseLM and how does the performance look like?\n2. Since instruction tuning will adjust all the parameters in the original pre-trained language model, there is a question what about what is the potential impact of this tuning process? Will it drops any knowledge of any tasks, which will be a disadvantage when the task's labeled data is available? In the Analysis C in the appendix, it will be good to have results for tasks other than classification such as summarization or question answering; and also to have a baseline where the BaseLM model is fine-tuned directly with the task labeled data (without prompt/soft-prompt).\n", "summary_of_the_paper": "The paper explores a simple and effective method to improve zero-shot performance of pretrained language models. Authors take a 137B parameter pretrained model and finetune it on multiple tasks verbalized via natural language instruction templates. As the result, the instruction-tuned model performs well on un-seen tasks with the zero-shot setting.", "summary_of_the_review": "Overall, the paper proposed an interesting idea and showed strong empirical results, hence I vote for accepting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G8KXj5zrhxo", "reviewer": "Reviewer_K8bM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "----\nDetailed comments:\n----\n\n- \"For each dataset, we manually compose ten unique templates\":  Why not have templates per task cluster instead of per dataset?  it is likely a relatively minor effect given the results from Appendix B but it seems like it could slightly prevent overfitting\n\n- The ablation in 4.1 was great (number of clusters).  Nit: I would have tried to move the (datasets per cluster/templates per dataset) ablation to the main body as well and shortened Section 3\n\n- The 4.2 (scaling laws) ablation is perhaps the most interesting of all.\n\n- In figure 6A, why was performance not increasing for untuned models w.r.t model size?  This seems to contradict findings from Brown et al where larger models did better on essentially all tasks.  Were there perhaps some poor datasets that happened to be in the held-in split (since the held-out tasks don't seem to have the same trend)?\n\n----\nAppendix:\n----\n\nI liked the section B ablations (as implied above).\n\nThat more templates per dataset didn't help is particularly interesting and suggests some questions. You hypothesize that more templates doesn't help because \"models at such scale do not easily overfit to a finetuning single task\" - but my intuition is for an opposite explanation -- that the models at such scale easily memorize a small number of templates!  One may even wonder if the instruction nature of the templates is helping at all.  \n\nFrom what I can tell, Appendix C on prompt tuning (which is very interesting) is maybe the primary evidence the instructions are important.  I think more could be done here, some ideas, probably there are better ways to test:\n- Have templates that leave out \"instructions\":  I would guess it wouldn't affect held-in task performance much, but would affect held-out tasks.  \n- Consider HellaSwag/PiQA/etc, where FLAN underperformed few-shot and even zero-shot.  One might hypothesize that if using a (subotimal) template that is less natural for language modeling, that zero-shot performance would suffer, but that FLAN performance wouldn't\n- One might hypothesize that the \"turn the task around\" templates help more than the other more straightforward templates that don't swap information between the prompt and response.\n- Easy but probably not great thing to try:  held-out tasks with wrong/useless templates\n\nA final thought:  It's not obvious that using as many training examples per dataset as possible is optimal, given that the model could overfit to dataset-specific spurious correlations.  This could be another area to investigate\n\n----\nMisc: \n----\n\n- UnifiedQA seems potentially worth citing as prior work", "summary_of_the_paper": "The paper creates a dataset of over 60 NLP tasks described via instructions (using templates for each task) and finds this boosts zero-shot performance on unseen tasks.", "summary_of_the_review": "Overall, the paper's idea is powerful (but of somewhat limited novelty) and the results are good (but not great).  Its greatest strength IMO was the ablations.  My biggest complaint is that it's not completely clear the instructions themselves are important at all - I suggest a few more experiments, though they don't seem crucial.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EVEmcRlRkAc", "reviewer": "Reviewer_BGTp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the method is a simple and straightforward scaling up of concepts and ideas from prior works (e.g. Zhong et al, Adapting ...; Mishra et al, cross-task generalization ...), the empirical results are thorough and impressive (outperforming GPT-3 with a slightly smaller model). The analyses also helps us understand when this method would work and inform us about future research directions.\n\nBelow are my concrete questions and comments: \n\n**Additional Tasks Results (3.4)**\n\nIn Appendix A.1, the paper mainly draws conclusions based on comparisons between GPT-3 and FLAN, which I do not think are fair: GPT-3 and FLAN differ in model size and pre-training data distribution. Instead, I think Base LM vs. FLAN might be a better comparison between “Off the shelf LM” and “instruction-tuned” model (though it won’t change the conclusion).\n\nIt is also worth pointing out in the main paper that for most of the additional tasks, even though it does not lead to higher accuracy, the performance of FLAN is still at least comparable (e.g. <1% worse and difference generally negligible) to Base-LM 0-shot. The only outlier seems to be ReCorD where the performance drops significantly after instruction-tuning, and this probably deserves some discussion. \n\nAlso I might have missed it - for the Base LM 137B zero-shot result, is it on the average template, or the best template? \n\n**Number of Task Clusters (section 4.1)**\n\nFor Figure 5 can you add the untuned model to the curve with the x-axis=0 (0-task cluster)? This can help us understand how much even 1 cluster (e.g. summarization) may help. \n\n**Explanation for scaling (section 4.2)** \n\nIt is an insightful empirical result that instruction tuning only works when model size reaches 68B. However, I am not entirely sure about the potential explanation of “model capacity”. There might be two potential explanations to this phenomena: 1) “model capacity”: as the paper has mentioned, smaller pre-trained models do not have enough model capacity and underfit the instruction tuning data, and 2) “better OOD generalization”: better quality pre-trained models have higher OOD generalization ability (and OOD accuracy) and they are less likely to “overfit” to in-distribution data. \n\nI personally find the second explanation more convincing. For example, Sahn et al. (https://arxiv.org/abs/2110.08207) finds that even models with only 11B parameters can generalize to unseen tasks, using T5 (MLM) and a larger set of prompts. The use of MLM objectives (might) improve the pre-training quality, while more prompts reduce the “overfitting to in-domain data” issue. \n\nI appreciate the fact that the author explicitly states the model capacity hypothesis more as a conjecture rather than a solid explanation. It’d be great if the authors can support the explanation further with more empirical evidence. On the other hand, however, since the results from Sanh et al came out only 2 weeks ago,  I would not change the score based on the response to this question. \n\n**In-context Few-shot vs. Fine-tuned Few-shot (Section 4.3)**\n\nCan the authors compare “fine-tuning/prefix-tuning an instruction tuned model with 16 examples” (appendix C but with only 16 examples) with “in-context prompting” (in 4.3 of the main paper), similar to Chen et al. (https://arxiv.org/abs/2110.07814 )? This would further inform us how we should use the few-shot learning examples for larger language models: put it in-context, or fine-tune? Again, since the comparison of Chen et al. came out only 2 weeks ago and the paper limit is 9 pages, I would not change the score based on the response to this question. \n\n**Others**\n\nResults of Appendix C are interesting and potentially impactful - this might imply that instruction-tuned models will become the new “base” model for the pretraining-finetuning paradigm. Is it possible to briefly mention it in the main paper as well (and redirect the readers to the appendix to see the full results)?\n\nIt might be too late to change the name, but “Finetuned LAnguage Net” (FLAN) is uninformative, since it does not capture any unique aspect of this method. What does “LAnguage” mean here, \"natural language instruction\" or \"language model\"? If it is the former, then directly including the word “instruction” might be better; and hopefully it’s not the latter, since even fine-tuned BERT on SST-2 counts as a fine-tuned language model ...\n\n**Typo**\nIntro: \nInstruction tuning is “a” simple method that, as ….\n\nConclusion:\nMoreover, our work supercedes recent work such “as” \n", "summary_of_the_paper": "The paper proposes a simple method, \"instruction-tuning\", to improve the zero-shot learning capability of large language model, which 1) annotates prompts for a wide range of tasks and then 2) fine-tunes the model to \"answer/respond to\" those prompt. The empirical results are impressive: after instruction-tuning, the 0-shot performance is better than GPT-3 (0-shot, sometimes few-shot) on a wide range of datasets; nevertheless, on datasets with formats already similar to language modeling, the performance gain is negligible or even negative.\n\nThe paper also made a few other observations 1) performance benefits from the number of task clusters 2) instruction-tuning is only beneficial when the model size is larger enough, and 3) few-shot learning still helps. ", "summary_of_the_review": "While the method is not new, the empirical results are strong and comprehensive. Though I disagree on the interpretation of some empirical results, overall the additional analyses bring us further insights on what method works for very large language models (i.e. > 100B dense model). I highly recommend the paper to be accepted to ICLR 2022. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-YgUOpXUim", "reviewer": "Reviewer_4nJJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall well-written with compelling results, this paper describes a new language model (FLAN) and shows how it improves upon the zero-shot task performance of previous language models such as GPT-3. While the paper is lacking some additional analysis, I am hesitant to recommend extremely compute-intensive ablations due the large size of the model (137B parameters).\n\nStrengths:\n - Considers a reasonably wide set of 62 datasets; although the inherent arbitrariness in dataset clustering was listed as a limitation, the clusters look quite reasonable to me, and the removal of overlapping datasets (e.g., \"Reading Comprehension w/ Commonsense\") seems appropriate.\n - Results are better than a strong Base LM baseline, as well as existing state-of-the-art models (GPT-3)\n - Overall the approach is intuitive and conceptually compelling\n - Highly relevant to ongoing work on language modeling, prompt tuning, and zero-shot learning\n\nWeaknesses:\n - From these experiments, it is unclear whether models are actually \"learning to follow instructions\" or just learning a very large space of tasks from the fine-tuning procedure. In other words, even though prompt variance is reported at inference time, the models could potentially perform just as well with nonsense or missing prompts during fine-tuning. As far as I can tell, no experiments that rule out this possibility exist.\n - Although qualitatively useful, the analysis in 4.1 does not conclusively show that the number of instruction tuning clusters aids performance, or that this trend is likely to continue with more clusters. Most of the gain could be acquired by tasks which are most difficult, or most similar to the heldout task, and this analysis cannot disprove such an interpretation. A proper analysis would consider more heldout tasks and permutations of training data, but presumably this is prohibitively expensive.\n - The paper is missing important details about hardware usage and training time\n - Some possible issues which might be resolved by the additional questions below\n\nAdditional Questions:\n - \"For each dataset, we manually compose ten unique templates that use natural language instructions to describe the task for that dataset.\" Do you have unique prompts for each dataset or only for each dataset cluster? Based on a cursory look at the supplementary material, I would assume the latter.\n - I didn't fully understand the justification for the OPTIONS token. Are the fine-tuned models successfully putting (almost) all of their probability mass on the corresponding options? How is the Base LM evaluated (if it's not fine-tuned, presumably it doesn't learn how to handle these options)? \n - Figure 6A: why does the untuned model see worse performance with more parameters?\n\nNits:\n - Figure 1 (Bottom) is possibly misleading, since AFAICT zero-shot FLAN underperforms few-shot GPT-3 on the majority of tasks\n - Not clear what \"turning the task around\" means for some tasks, or why this is a useful type of prompt diversity", "summary_of_the_paper": "This paper describes an approach to fine-tuning large language models which can improve zero-shot accuracy on unseen tasks. ", "summary_of_the_review": "I give this paper a strong recommendation, in spite of some missing ablations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the extent to which a large language model (LM) can generalize to unseen tasks via "instruction tuning", a process that fine-tunes the LM on a large number of tasks with natural language instructions.  At test time, the model is evaluated zero-shot on held out tasks.  The empirical results are good, and the 137B FLAN model generally out performs the 175B untuned GPT-3 model.

All reviewers voted to accept with uniformly high scores, despite two commenting on the relative lack of novelty.  The discussion period focused on questions raised by two reviewers regarding the usefulness of fine-tuning with instructions vs. multi-task fine-tuning without instructions.  The authors responded with an ablation study demonstrating that providing instructions at during tuning led to large gains.

Overall the paper's approach and detailed experiments will be useful for other researchers working in this fast moving area in NLP.
Decision: Accept (Oral)