Year: 2022

Abstract:
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces EXMIX (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using EXMIX, we study the effect of multi-task pre-training at the largest scale to date, and analyze cotraining transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose EXT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised EXMIX. Via extensive experiments, we show that EXT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of EXMIX. EXT5 also significantly improves sample efficiency while pre-training.

heading: 1 INTRODUCTION
text: Transfer learning (Schmidhuber, 1987; Pratt et al., 1991; Caruana et al., 1995) has been the cornerstone of recent progress in natural language processing (Ruder et al., 2019; Devlin et al., 2019; Raffel et al., 2020). While self-supervised pre-training has been shown to be highly effective at exploiting large amounts of unlabeled data without relying on human annotation, there is still much to explore regarding transfer learning in a multi-task co-training setup.
Prior seminal works like T5 (Raffel et al., 2020) and MT-DNN (Liu et al., 2019a) have demonstrated a degree of promise in the paradigm of multi-task co-training (Caruana, 1997). However, the challenge of catastrophic forgetting remains. Tasks often have to be carefully selected in order to demonstrate positive affinity with regards to downstream transfer. In many cases, it is not unreasonable to expect negative transfer (Rosenstein et al., 2005; Vu et al., 2020). This makes the process of empirically curating a set of tasks to include in a transfer learning setup both computationally prohibitive and specific to downstream tasks.
While standard pre-training typically employs a variant of the self-supervised language modeling objective (Raffel et al., 2020), certain types of skills such as commonsense knowledge are only acquired at a slow rate even using massive amounts of unlabeled data (Zhang et al., 2021). As ever larger models are trained, the development of much more sample-efficient pre-training settings becomes thus more important, and could be addressed via multi-task learning.
For the first time, we explore and propose Extreme Multi-task Scaling — a new paradigm for multitask pre-training. Compared to the largest prior work (Aghajanyan et al., 2021), our study doubles the number of tasks and focuses on multi-task pre-training rather than fine-tuning, which enables a direct comparison to standard pre-training. Our proposal is based on the insight that despite negative transfer being common during fine-tuning, a massive and diverse collection of pre-training tasks is generally preferable to an expensive search for the best combination of pre-training tasks.
∗Google AI Resident. †Equal contribution. Sebastian is now at Google Research. Sanket returned to CMU.
To this end, we introduce EXMIX: a massive collection of 107 supervised NLP tasks to be included in a multi-task pre-training setup. We process all tasks in an encoder-decoder friendly format to readily support the sharing of all parameters across all tasks. We postulate that an ensembling effect across as many tasks, distributions and domains as possible results in a consistently net-positive outcome. This echoes early multi-task learning results (Caruana, 1997; Baxter, 2000) indicating that a bias that is learned on sufficiently many tasks is likely to generalize to unseen tasks drawn from the same environment. Moreover, our experiments verify that our EXMIX mixture outperforms a best-effort mixture of manually curated tasks.
Finally, we propose EXT5: a T5 model (Raffel et al., 2020) pre-trained on a mixture of supervised EXMIX and self-supervised C4 span denoising. EXT5 outperforms state-of-the-art T5 models on well-established benchmarks such as SuperGLUE (Wang et al., 2019a), GEM (Gehrmann et al., 2021), and Rainbow (Lourie et al., 2021); as well as Closed-Book QA (Roberts et al., 2020) tasks. Notably, our experimental findings also suggest that including EXMIX may reduce the number of pre-training steps required to achieve strong performance, bringing about substantial sample efficiency benefits.
To summarize, this paper contributes the following:
– We propose EXMIX (§2): a collection of 107 supervised NLP tasks for Extreme Multi-task Scaling, formatted for encoder-decoder training. EXMIX has approximately twice as many tasks as the largest prior study to date (Aghajanyan et al., 2021), totaling 18M labeled examples across diverse task families.
– Given this large collection of tasks, we conduct rigorous empirical studies evaluating transfer between common task families (§2.1). Our experiments show that curating a pre-training mixture based on fine-tuning transfer is not straightforward (§2.2). Hence, efficiently searching for the best subset of tasks to include in a multi-task pre-training setup is challenging and prohibitive.
– Using EXMIX, we pre-train a model alongside the C4 span-denoising objective introduced by Raffel et al. (2020), resulting in a new pre-trained model which we call EXT5 (§3). EXT5 outperforms state-of-the-art T5 on well-established benchmarks such as SuperGLUE, GEM, Rainbow, Closed Book Question Answering, and several other tasks that are outside of EXMIX (§3.2), while also being more sample-efficient (§2.6).
2 THE EXMIX TASK COLLECTION
To explore the Extreme Task Scaling paradigm, we introduce EXMIX (Extreme Mixture), a collection of 107 diverse English NLP tasks totaling 18M examples. Following Raffel et al. (2020), we format all tasks as text-to-text examples to readily allow for multi-task training. This unified format also enables simple implementations without the need for taskspecific heads/losses, loss scaling, or explicit gradient accumulation for heterogenous batches as in prior works (Liu et al., 2019a; Aghajanyan et al., 2021). When selecting examples from EXMIX, ex-
amples from each dataset are sampled proportionate to the individual dataset’s size, with each dataset’s sampling rate capped at 3 × 105 maximum effective examples to ensure a balance between large and small datasets. We refer readers to Appendix A for a comprehensive breakdown of EXMIX. Additionally, we discuss future multilingual variants of EXMIX and EXT5 in §5.

heading: 2.1 TRANSFER RELATIONS BETWEEN EXMIX TASKS
text: As discussed in §1, our goal is to pre-train a model on EXMIX to improve downstream performance. One natural question to ask is which tasks have a negative impact on downstream performance? Specifically, is there a subset of EXMIX that leads to better representations when used for multitask pre-training? Obviously, testing all possible 2|EXMIX| combinations is impractical since the
pre-training process is expensive to run. Instead, we experiment with the less expensive co-training procedure (i.e., multi-task fine-tuning), using representative subsets of similar tasks. Later, in §2.2, we explore whether these results can be used to inform the task selection for multi-task pre-training.
To study transfer amongst task-families in EXMIX, we construct subsets (Table 1) of 3 tasks each that are partitioned along their task family. Using these subsets, we evaluate transfer among task families in a multi-task learning (co-training) setup. While other types of tasks are available in EXMIX, we did not include them because they were not diverse enough to be representative of a task family, and would scale the number of models needing to be trained at a quadratic rate.
Experimental Setting We fine-tune a model on each pair of task families (i.e., 6 datasets at a time). To ensure a fair balance of tasks, we sample tasks proportional within their family, but uniformly between task families. For example, while evaluating how classification tasks and NLI tasks transfer amongst each other, the sampling ratio of MNLI:ANLI will be proportional (approximately 2.4:1), but the overall ratio of NLI examples to classification examples will be 1:1. For reference, we also train a model on each individual task family using proportional example mixing (Sanh et al., 2019).
In total, this results in F+ ( F 2 ) models trained, where F is the number of task families. Our experiments use F = 8
as shown in Table 1, resulting in 34 models trained in total. Each model is fine-tuned on top of the released T5.1.1BASE checkpoint for 200k steps using a batch size of 128 and a constant learning rate of 10−3.
Observations We summarize our results in Table 2. We observe that although there exist particular task-family pairs that show positive transfer (e.g., co-training with NLI helps most other tasks), negative transfer is more common when training across task families compared to intra-family training. 21 out of the 56 inter-family relationships perform worse than intra-family models with the same data budget, which grows to 38 out of 56 for a fixed compute-budget. While the abundance of negative transfer among diverse task families is an expected result, interesting trends manifest in the individual relationships. For example, summarization tasks generally seem to hurt performance on most other task families; and CBQA tasks are highly sensitive to multi-task fine-tuning.
We also report correlations for intra-family datasets in Figure 2 using the same models as in Table 2. In most cases, we see positive correlations between datasets in the same family. In a few cases, however, we observe an opposite trend. For example, fine-tuned models that performed better on the GEM schema-guided dialog dataset achieved lower scores on KILT Wizard-of-Wikipedia.
This initial exploratory analysis highlights both the potential of EXMIX as a tool to systematically study task relationships, as well as the potential challenges in leveraging multi-task learning naively on top of pre-trained representations.

heading: 2.2 CAN FINE-TUNING TASK RELATIONSHIPS HELP CURATE A PRE-TRAINING MIXTURE?
text: Our observations in §2.1 showed that multi-task co-training on top of existing pre-trained checkpoints is not straightforward, and often results in negative transfer. However, the uncovered task relationships might help efficiently search for an ideal subset of EXMIX for multi-task pre-training. To this end, we select a set of the most promising task families to be included in a multi-task pretraining setup, ranking task families by the average relative improvement they provide to other target families (the last column in Table 2). Specifically, we include NLI, commonsense, classification, and closed-book QA tasks from EXMIX to form a mixture of 48 tasks to include in a multi-task
pre-training setup. We then fine-tune the resulting model on SuperGLUE, comparing it to T5.1.1 and EXT5 in Table 3.
While this model narrowly outperformed T5.1.1, it did not yield better results than including all of EXMIX in the multi-task pre-training mixture, as we report in §3.2. Moreover, it did not outperform a random selection of 55 tasks on average, as we report in our task scaling experiments (§2.5).
We conclude that the negative transfer during multi-task fine-tuning does not necessarily inhibit pre-training. While we cannot directly conclude that an ideal subset of EXMIX does not exist to be mixed with self-supervised pre-training for specific downstream tasks, our experiments show that randomly including more diverse pre-training tasks generally improves downstream performance. It must also be noted that the end-goal is to find a mixture that leads to a general pre-trained model that can be used for a large variety of downstream tasks, and that a setup to find a pre-training mixture tailored for SuperGLUE would be different.

heading: 2.3 MULTI-TASK PRE-TRAINING VS PRE-FINETUNING
text: Instead of pre-training on EXMIX, another way to leverage multi-task learning is as an intermediate step between pre-training and fine-tuning. This is referred to as pre-finetuning by Aghajanyan et al. (2021). We conduct controlled experiments to compare pre-training with pre-finetuning. We begin with a standard T5 base checkpoint and pre-finetune it with EXMIX. After this phase, we fine-tune on SuperGLUE.
Table 4 compares pre-finetuning and our proposed multi-task pre-training. We also report the total compute (in total number of tokens processed) by the model in both schemes. The results show that multitask pre-training is significantly superior to prefinetuning. A potential hypothesis is that multi-task pre-training narrows the gap between pre-training and finetuning data distributions, as the pre-training stage more closely resembles fine-tuning. Conversely, segregating pre-training and pre-finetuning
into two different stages may induce catastrophic forgetting of the pre-training task. Hence, in EXT5, we opt for multi-task pre-training over pre-finetuning.

heading: 2.4 HOW MUCH LABELED DATA SHOULD BE MIXED?
text: In this section, we explore how the ratio of C4 to EXMIX examples during massive multi-task pretraining affects performance. As mentioned later in §3, this is controlled by a hyperparameter R, where a pre-training batch will have approximately R times as many C4 examples compared to EXMIX. From our results in Figure 3, we find that despite EXMIX improving downstream performance when mixed with self-supervised C4 pre-training at many rates, a model trained with R = 0 suffers greatly in comparison. This result is significant, as it shows that while EXMIX improves the pre-training process, self-supervised training over a large unstructured corpus is still crucial.

heading: 2.5 DOES ADDING MORE TASKS HELP? TASK SCALING EXPERIMENTS
text: In this section, we explore how model performance changes as the number of tasks included in a massive multi-task pre-training setup is scaled up. We choose random sets of 30, 55, and 80 tasks (each a superset of the last), pre-train a BASE-sized model for 524k steps, and fine-tune them on SuperGLUE. We train our models with batch sizes of 128 and 512 and R = 2 (the ratio of C4 to EXMIX examples) as this configuration worked best for our BASE-sized models (§2.4). We repeat this over three random seeds (for random subset selection), and report average scores in Figure 4.
Overall, with large batches, we can see that increasing the number of tasks being mixed generally helps downstream performance. This reinforces our intuition that task scaling indeed helps. With small batches, there is less of an upward trend, signifying that large batches are essential for a large number of tasks. This is intuitive, given that multi-task learning may cause gradients to be noisy (Yu et al., 2020). Another explanation as to why this happens is that large-batch training can offer benefits even for single-task models (Smith et al., 2018) — a trend formalized by McCandlish et al. (2018).

heading: 2.6 IMPROVING SAMPLE EFFICIENCY WITH EXMIX
text: We hypothesize that extreme multi-task scaling also improves the sample efficiency of pre-training. To test this, we exclude SuperGLUE from EXMIX, pretrain a large model for 200k steps, and fine-tune it on SuperGLUE at several intervals during early pretraining stages. We find that EXMIX pre-training is significantly more sample-efficient than vanilla selfsupervised pre-training. Note that at only 20k pretraining steps, our ExT5 model already achieves 75.8 SuperGLUE score, which outperforms a fully pretrained BERT large model by about +4% (Wang et al., 2019a).

heading: 3 THE EXT5 MODEL
text: To overcome the challenges of multi-task co-training at scale, i.e. negative transfer and catastrophic forgetting explored in §2.1, the rest of this paper revisits the multi-task pre-training paradigm introduced by Raffel et al. (2020) via extreme multi-task scaling. This section introduces EXT5: a pre-trained sequence-to-sequence Transformer encoder-decoder model (Vaswani et al., 2017) based on the popular T5 framework.

heading: 3.1 TRAINING EXT5
text: Pre-training We pre-train on a mixture of C4 and EXMIX (§2), and combine them with a hyperparameter R that is the ratio at which C4 examples are sampled with respect to EXMIX examples. The C4 objective we use is the same as that used by Raffel et al. (2020), and every task optimizes the standard sequence-to-sequence cross-entropy loss. We pre-train EXT5 on the same number of steps as T5, and EXT5 sees an identical number of tokens to the released T5 models. Concretely, we pre-train our models for 1M total steps with a batch size of 2048 and sequence length 512, resulting in a total of approximately 1T tokens seen by the model during pre-training (both unsupervised and supervised inclusive). We use the T5.1.1 architecture (Shazeer, 2020) for all of our experiments — which uses GEGLU-activated layers instead of ReLU in classic Transformer models (Vaswani et al., 2017). For optimization, we use Adafactor with an inverse square root learning rate schedule that kicks in after a a constant phase of 0.01 for 10k steps. EXT5 also uses the same tokenizer as T5.
Fine-tuning We follow the same fine-tuning procedure for T5 and EXT5 for fair comparison, although we found that EXT5 generally benefitted from a smaller learning rate while fine-tuning (10−4 worked well for EXT5 vs 10−3 for T5 variants). Fine-grained details can be found in Appendix B.

heading: 3.2 EXPERIMENTAL SETUP
text: Our experiments consider both within-mixture and out-of-mixture tasks (i.e., whether a task is included in EXMIX). Within-mixture tasks measure the amount the task benefits from multi-task pre-training and extreme task scaling. Similar to the co-trained models in Raffel et al. (2020), we continue to fine-tune on the target task from a pre-trained ExT5 checkpoint. For out-of-mixture tasks, we consider possibly new unseen tasks or collections that were not included in the EXMIX mixture to test the effect of generalizing to unseen tasks. For the sake of brevity, the fine-grained details of these experimental setups can be found in the Appendix.

heading: 3.3 EXPERIMENTAL RESULTS
text: WITHIN-MIXTURE RESULTS
We report results on SuperGLUE (Table 5), GEM (Table 6), Rainbow (Table 7), MsMarco (Table 8) and CBQA datasets (Table 9). On the whole, we observe that ExT5 consistently outperforms strong T5 baselines across a range of model sizes. On SuperGLUE, we achieve +5%, 2.3% and
+0.7% gain on BASE, LARGE and XL respectively. On GEM, ExT5 outperforms T5 on 6 out of 9 collections while remaining on-par on the other 3 collections. Notably, the gain on datasets such as WebNLG are approximately +11% ROUGE for the large model and generally range from +1% to +6% on different collections. On Rainbow, ExT5 outperforms our own run of T5 by +0.7% on average and +4.6% improvement over the best multi-task (sequential) setup in Lourie et al. (2021). Finally, on question answering and ranking, ExT5 substantially outperforms T5 at two different sizes.
OUT-OF-MIXTURE RESULTS
We are also interested in evaluating EXT5 on tasks outside of EXMIX, and hypothesize that the extreme multi-task pre-training of EXT5 will lead to better performance on new unseen settings. Concretely, we fine-tune and evaluate on Machine Translation: translating sentences from English to other languages (Bojar et al., 2014; 2015; 2016); Reasoning: answering scientific questions on ARC (Clark et al., 2018); and Named Entity Recognition: extracting all entities from sentences on the CoNLL-2003 NER dataset (Tjong Kim Sang & De Meulder, 2003).
Table 10 summarizes the results on the out-of-mixture tasks. Across all tasks, we see that EXT5 outperforms upon T5 baselines. The largest improvement is on the ARC scientific reasoning task, perhaps due to the large amount of QA tasks in EXMIX. Though, the trend is consistent also with the NER and MT tasks that do not have any similar dataset in EXMIX. This suggests that the representations learned by EXT5 are more general adaptable to a new objective, even when the output is in a new language.
This improved generalization of EXT5 is very encouraging from a practical stand point, since pretraining again with EXMIX ∪ {t} for any new target task t would be very expensive. Instead, we see that the extreme multi-task pre-training of EXT5 already provides improved results. Therefore, it might only be worth repeating pre-training when the collection of training datasets grows by a significant amount (see §2.5).

heading: 4 RELATED WORK
text: Improving NLP models with Multi-task Learning Collobert & Weston (2008) leverage multitask learning for relatively simple tasks like Part-of-Speech tagging. Phang et al. (2019) use an intermediate fine-tuning stage using four tasks with large datasets for Natural Language Understanding. Similarly, Liu et al. (2019a) proposed MT-DNN, which uses a setup at a scale of around 30 tasks and up to 440M parameters. Most recently, Aghajanyan et al. (2021) use around 50 tasks and models of sizes upto 440M parameters. Gururangan et al. (2020) take an alternative approach, which is to continue pre-training a model but use domain-specific data as an intermediate step. McCann et al. (2018) proposed a unified framework similar to that of T5. Recently, Wei et al. (2021) also illustrated how a multi-task learning stage can greatly improve the zero-shot prompting performance of large language models at the scale of ~137B parameters. Efforts have also been made to tailor pre-training objectives to specific tasks, e.g., question answering (Ram et al., 2021; Jia et al., 2021), dialogue (Li et al., 2020), and span selection tasks (Joshi et al., 2020).
Relationships amongst different tasks Bingel & Søgaard (2017) conducted a study similar to ours in §2.1 but for more traditional NLP tasks like chunking, CCG tagging, POS tagging, etc. More recently, Vu et al. (2020) conducted an in-depth study of relationships between various classification/regression, question-answering, and sequence-labeling tasks, and proposed a task-embedding framework to predict such relationships. Khashabi et al. (2020) also conducted similar experiments but specific to question-answering datasets/formats, resulting in a strong QA model known as UnifiedQA that is also based on the T5 framework. Outside of NLP, Zhang & Yeung (2010) introduced a convex optimization objective for learning task relationships, and Li et al. (2018) explore and exploit task relationships on a variety of diverse datasets.
Choosing which tasks to transfer from Our experiments in §2.2 attempted to empirically select a set of tasks to transfer from. Along these lines, Ruder & Plank (2017) use a Bayesian Optimization method with similarity measures to automatically select relevant data from different domains. Similarly, Guo et al. (2019) use multi-armed bandits to select tasks and a Gaussian Process to control the mixing rates for the selected tasks. Another strand of recent work selects appropriate transfer languages based on manually defined features (Lin et al., 2019; Sun et al., 2021). Aside from the NLP domain, Fifty et al. (2021) proposed a method to select which tasks to transfer to based on task gradients. All of the aforementioned works select data tailored to a downstream task of interest. If a general pre-trained model was attempted to be trained in a similar fashion, computational bottlenecks similar to those motivating §2.1 and §2.2 would arise.
Pre-trained Transformers Transformer models (Vaswani et al., 2017) such as T5 (Raffel et al., 2020), BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) rely on large unlabeled corpus for self-supervised learning. Given the wild success of the pre-train-finetune paradigm, the search for suitable pre-training tasks has also become an active area of research (Lewis et al., 2019; Lan et al., 2019; Chang et al., 2020; Zhang et al., 2019; Lourie et al., 2021). While there has been evidence that supplementary pre-training tasks can help improve performance, this work is the first massive-scale multi-task pre-trained model.
Scaling Laws Scaling laws for Transformers have attracted much attention recently, especially pertaining to model size (Kaplan et al., 2020; Zhai et al., 2021; Tay et al., 2021a). In Kaplan et al. (2020), the authors further investigate scaling with respect to dataset size (on the same pre-training corpus). To this end, this work can be interpreted as an attempt of scaling up with respect to the number of high quality, diverse labeled tasks that can be used for pre-training.

heading: 5 EPILOGUE
text: Limitations Despite our best efforts to evaluate on as many representative tasks as possible while also maintaining a balance among task partitions for a given set of transfer learning experiments, any study that explicitly abstracts datasets into “task families” is highly dependent on nuances pertaining to the nature, domain, and expressiveness of the task family’s representative datasets. For this paper, the subsets were constructed so as to include a diverse set of datasets to evaluate on, and we tried to partition task-families to be as mutually exclusive as possible. However, it must be acknowledged that no dataset is perfectly isolated, and any set of them only a proxy for a larger “task family”. On a separate note, lexical metrics like BLEU/ROUGE are useful but do not paint the full picture of how well a model truly performs on text-generation tasks.
Future Work We believe that a multilingual version of EXT5 would be a natural extension of this work. Such a model will require extra care with regard to balancing not only task families, but also task languages. A multilingual version of EXMIX could provide a more robust foundation for the analysis of task families in existing works that analyze how multilingual NLP models transfer amongst different languages (Kudugunta et al., 2019; Hu et al., 2020; Wang et al., 2021). For example, it would be interesting to understand whether our results in §2.1 hold across different languages (and language families), and to explore cross-lingual cross-task generalization. We also hypothesize that modeling innovations that introduce inductive biases designed to exploit multi-task learning setups (Ha et al., 2016; Tay et al., 2021b) can push the boundary of the strong performance displayed by EXT5. Other solutions like gradient manipulation (Yu et al., 2020; Wang et al., 2021) might also further improve extreme multi-task scaling, albeit at the cost of more complex implementations.
Conclusion This paper explores how supervised multi-task learning at a massive scale can be used to improve existing self-supervised pre-training strategies for NLP models, and does so by introducing EXMIX (§2) and EXT5 (§3). Our experiments showed that while negative transfer is common when fine-tuning on diverse tasks (§2.1), scaling up the number of tasks to include in a multi-task pre-training setup enables strong downstream performance (§3.2) with better sampleefficiency (§2.6). We hope that this paper motivates future research on how existing labeled datasets can be used to further improve NLP models within the pre-train/fine-tune paradigm.

heading: ACKNOWLEDGEMENTS
text: The authors would like to thank Mostafa Dehghani and Adhi Kuncoro for valuable comments and insights. We would also like to thank the authors of Mesh Tensorflow (Shazeer et al., 2018) and T5 (Raffel et al., 2020), as their high-quality code and paper enabled this work.
AUTHOR CONTRIBUTIONS
Vamsi co-led the project and was primarily responsible for the design, implementation, and engineering effort behind it. Vamsi proposed and ran key experiments including (but not limited to): task-family transfer analysis, early proof-of-concepts for extreme task-scaling and EXT5, taskscaling analysis, etc. Vamsi also wrote most of the paper, and was (jointly) responsible for the overall framing of the paper.
Yi served as the co-lead of the paper and proposed the initial idea. Yi was responsible for most of the downstream EXT5 experiments, including SuperGLUE, Rainbow, CBQA, and Machine translation, along with large-scale pre-training of EXT5. Yi also wrote large portions of the paper.
Tal was (jointly) responsible for the overall framing of the paper, and wrote large portions of it. Tal contributed the Vitamin C task to the EXMIX mixture, along with running out-of-mixture experiments for Named Entity Recognition.
Jinfeng contributed the GEM benchmark to our mixture and was heavily involved in running experiments.
Steven contributed the DialoGLUE tasks to EXMIX, helped run a substantial number of experiments and contributed substantially to discussions around the paper’s framing.
Sanket was responsible for and ran experiments on GEM, and contributed the Yahoo Answers and Argument mining tasks.
Honglei contributed the MsMarco task to EXMIX and helped with benchmarking ExT5 on MsMarco. Honglei also contributed scripts and pipelines for obtaining reranking results on MsMarco.
Vinh contributed NewsQuiz, AgreeSum, TweetQa and TweetEval to EXMIX and contributed substantially to paper writing and the framing of the paper.
Dara contributed GPT DeepFake tasks and low-resource tasks (which were not used in the end).
Jianmo contributed the DocNLI task and helped with running experiments for out-of-mixture tasks.
Jai contributed the Race and MultiRC Eraser tasks to the EXMIX and helped edit the paper.
Kai contributed several retrieval tasks to EXMIX, which were not included eventually.
Sebastian helped substantially with the paper narrative, writing of the paper and brainstorming.
Donald (along with Yi and Vamsi) was heavily involved in framing the early vision of the project. Donald was also deeply involved in brainstorming sessions, and provided critical feedback that helped to steer the project in the right direction.
All authors contributed to brainstorming and discussion sessions.
ETHICS STATEMENT
Large language models have been shown to capture certain biases about the data they have been pre-trained on (Bender et al., 2020). While a comprehensive analysis of such biases is outside of the scope of this work, it is a compelling direction to investigate to what extent the inclusion of supervised data during pre-training can help mitigate such biases. An alternative consideration is the addition of diverse values-targeted data (Solaiman & Dennison, 2021) during pre-training in order to instill beneficial biases in a model.
Another challenge when training large models is their energy consumption and environmental impact (Strubell et al., 2019). To ablate different task combinations, we performed experiments using
the more computationally efficient fine-tuning setup. We have shown that EXMIX leads to more sample-efficient pre-training compared to standard self-supervision, which we hope will save compute in future experiments.
REPRODUCABILITY STATEMENT
All of the modeling and training code used for EXT5 and its variants is already open-sourced as a part of the Mesh Tensorflow1 (Shazeer et al., 2018) and T52 (Raffel et al., 2020) Libraries. Additionally, EXMIX is composed of datasets that are already publicly available.

heading: A DATASETS
text: Table 11 summarizes the 107 datasets included in EXMIX. Some of the lines in the table represent existing benchmarks that group several tasks together. From each collection, we use the datasets that include English training data:
– GLUE: CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QQP, STS-B (Cer et al., 2017), MNLI (Williams et al., 2018), QNLI (Converted from Rajpurkar et al. (2016), RTE (Dagan et al., 2006), WNLI (Sakaguchi et al., 2020).
– SuperGLUE: BoolQ (Clark et al., 2019), CB (De Marneffe et al., 2019), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al., 2018), RTE (Dagan et al., 2006), WiC (Pilehvar & Camacho-Collados, 2019), WSC (Levesque et al., 2011).
– KILT: FEVER (Thorne et al., 2018), AIDA (Yosef et al., 2011), WNED (Guo & Barbosa, 2014), T-REx (Guo & Barbosa, 2014), NQ (Kwiatkowski et al., 2019), HoPo (Yang et al., 2018), TQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), WoW (Dinan et al., 2019).
– Rainbow: αNLI (Bhagavatula et al., 2020), CosmosQA (Huang et al., 2019), HellaSWAG (Zellers et al., 2019), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021).
– GEM (en): Wiki-Lingua (Faisal Ladhak & McKeown, 2020), WenNLG (Gardent et al., 2017; Castro Ferreira et al., 2020), CommonGEN (Lin et al., 2020), E2E (Dušek et al., 2019), DART
(Radev et al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang et al., 2020), TurkCorpus(Xu et al., 2016)
– DialoGLUE: Banking77 (Casanueva et al., 2020), HWU64 (Liu et al., 2019b), CLINC150 (Larson et al., 2019), SGD (Rastogi et al., 2020), TOP (Gupta et al., 2018).
– TweetEval: Emotion Recognition (Mohammad et al., 2018), Emoji Prediction (Barbieri et al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identification (Zampieri et al., 2019), Sentiment Analysis (Rosenthal et al., 2017), Stance Detection (Mohammad et al., 2016).

heading: B EXPERIMENTAL DETAILS
text: This section describes the experimental details
B.1 IMPLEMENTATION DETAILS
Our models were trained using Mesh Tensorflow (Shazeer et al., 2018) using the T5 library (Raffel et al., 2020).
B.2 DATASET EXPERIMENTAL SETUP
This section reports the dataset and experimental setup on each individual target task/dataset.
SuperGLUE We finetune on the entire SuperGLUE as a mixture with proportionate sampling in similar fashion to (Raffel et al., 2020). We finetune for a total of 200k steps with a batch size of 128. When selecting checkpoints on SuperGLUE, we follow the same convention as Raffel et al. (2020) in selecting the best checkpoint for each task for a fair comparison to models that are fine-tuned on the individual tasks instead of co-training on all of them.
GEM We report test set results on all datasets except CommonGen and ToTTo, on which we report validation scores. We sweep over learning rates of 10−3, 5 × 10−4 and 10−4. All results are computed using GEM-metrics3. For each dataset, we select the best model checkpoint using average of BLEU, ROUGE-1, ROUGE-2 and ROUGE-L scores on the validation set. We use the greedy decoding strategy to be consistent with the original GEM paper (Gehrmann et al., 2021).
CBQA We report validation set results, and sweep over learning rates of 10−3 and 10−4.
Rainbow We multi-task co-train on all datasets, and sweep over learning rates of 10−3 and 10−4.
WMT Machine Translation We finetune our models on three collections of WMT, namely EnDe, EnFr and EnRo. We use a constant learning rate of 10−3 and dropout of 0.1. We train with a batch size of 4096 for a maximum of 400k steps and report peak validation BLEU score. We use a beam size of 4 and a length penalty of 0.6.
ARC We report scores on the Challenge set, and train with a batch size of 32 and sweep over learning rates of 10−3 and 10−4.
CoNLL-03 NER We convert NER to seq2seq by writing the target as the ordered sequence of tags and entities (for example “When Alice visited New York”→ “[PER] Alice [LOC] New York”). Accuracy is measured on a sentence level, considering a prediction to be correct only if it exactly matches the reference sequence.

heading: C DETAILED EXPERIMENTAL RESULTS
text: Many of our experiments in §2 used the average SuperGLUE score of a model for evaluation. We report the full results on all datasets below.
3https://github.com/GEM-benchmark/GEM-metrics

Review summaries:
This paper revisits the idea of multi-task learning for natural language processing (NLP) and scales it up to 107 supervised NLP tasks as EXMIX (Extreme Mixture) across diverse domains and task families. It extensively analyzes the co-training effect between the different families of NLP tasks and proposes a model pre-trained using a multi-task objective of self-supervised span denoising and supervised EXMIX named EXT5. They show that a massive and diverse collection of pre-training tasks is generally preferable to an expensive search for the best combination of pre-training tasks. It further shows the effectiveness of EXT5 on a range of supervised NLP tasks. 

The paper presents an analysis of multi-task learning with the T5 model at a large scale. The tasks are formatted as text-to-text tasks and trained alongside the span denoising objective on C4, same as the multi-task strategy used in the original T5 paper. The paper evaluates this approach for a few different sizes of the T5 model on various downstream datasets, compared with vanilla T5 model. The paper shows improvements on average accuracy over multiple downstream benchmarks. In addition, there are some ablations on the number of tasks, the ratio of C4 vs multi-task, and comparing multi-task with pre-finetuning. 

This paper introduces "ExMix," a collection of 107 pre-existing NLP datasets across 8 identified task-families. ExMix is proposed as a pre-training dataset, to be mixed with the typical self-supervised pre-training prevalent in NLP. The authors propose to pre-train a T5 model on a mixture of ExMix and C4 (the self-supervised pre-training dataset for T5), which they term ExT5. To do this, they convert every task to a text-to-text problem, so that all tasks, supervised and self-supervised, use standard sequence-to-sequence cross-entropy loss. They pre-train ExT5 for the same number of steps and on the same number of tokens as T5. They then evaluate how ExT5 and T5 perform when fine-tuned to individual tasks. They find that ExT5 largely outperforms T5 on the majority of downstream evaluated tasks, including tasks whose training data was included in ExMix as well as tasks which were not seen during pre-training.  The authors perform an experiment to identify whether or not a manually curated subset of ExMix can provide better pre-training generalization than the full set of tasks, due to negative transfer between tasks. To do this, they examine negative transfer between pairs of task families during fine-tuning. They select the task-families which exhibited the highest amount of fine-tuning transfer across all task-pairs, and use only 48 tasks from these task-families as a pre-training objective. They compare this model to a model trained on a slightly larger subset (55) of randomly selected tasks, as well as ExT5 (trained on all 107) tasks. The authors find that their method of task-selection is largely unhelpful for selecting pre-training tasks - the model trained on a random subset of 55 tasks outperforms the manually curated 48-task model, and ExT5 significantly outperforms both. This suggests that, for pre-training, selecting tasks that complement one-another with respect to downstream performance is not straight-forward, but increasing the number of tasks seems to alleviate negative transfer, rather than exacerbate it.  Finally, the authors perform some additional analysis of the training decisions made around ExT5. They find that: (a) pre-finetuning, which introduces multi-task learning as an intermediate step between self-supervised pre-training and fine-tuning, is not better than mixing all tasks for pre-training when using ExT5, (b) ExMix without self-supervised pre-training is significantly worse than self-supervised training without ExMix, suggesting that self-supervised pre-training is still crucial for pre-training, (c) downstream performance generally increases as the number of tasks scales up, and (d) pre-training with ExMix is significantly more sample-efficient than self-supervised pre-training along, i.e. it achieves strong downstream results very early, even 20k steps into pre-training.

This paper presents a corpus (ExMix) and a model (ExT5) which extend work on multi-task pretraining for large language models.   Their ExMix covers 8 families of supervised natural language processing tasks (eg. summarization, classification) with three individual tasks in each of them. These tasks are all converted into one text-to-text task (like prior work does) in order to allow uniform training on mutliple tasks. This is an impressively comprehensive collection of multi-task data sets, twice as large as the previously biggest multi-task corpus.   They also perform thorough experiments on a variety of intersting research questions.  They look at how to use cheaper fine-tuning to select a sub-set of tasks to use for pre-training - and they show that it is best to use all the supervised tasks in pretraining, not just the subset which worked best in fine-tuning.  They looked at whether adding and additional fine-tuning step before the final GLUE task fine-tuning would be better than pre-training with all this supervised multi-task data, and they show that in fact it is best to actually pre-train with them.  In their pre-training they mix masked-unsupervised learning with the task-supervised learning and they explore what is the right mix of the two, showing that most mixture settings are good, but that without the unsupervised signal, the model performs much worse. Also increasing the number of tasks generally helps downstream performance.   They also look at the efficiency of their approach. Training on ExMix allows the model to reach the performance of the T5 model trained only on plain text with many fewer (approximately half) the number of steps.   Finally they build a large pre-trained model ExT5 on which they perform really extensive evaluations and the demonstrate that the multi-task pretraining clearly helps and improves on a strong baseline across many tasks, including tasks which have not been included in the ExMix training data like translation.